Information processing device, information processing method, and program

Abstract

This invention provides a technology that reduces the workload of users involved in testing. [Solution] The system accepts the selection of a specific answer image that serves as a model answer from among answer images containing answers to the test, and scores the answers within the answer image based on the selected specific answer image.

Description

Technical Field

[0006] ,

[0007] , ,

[0001] The present invention relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] In recent years, as part of the digital transformation (DX) of education, the automation of grading work has been progressing. For example, a school teacher collects answer sheets answered by examinees, digitizes them with a scanner, and compares them with pre-stored answer data (answers) to perform grading.

[0003] Patent Document ¹ discloses a system that automatically grades by collating an image of an answer sheet read by a scanner with a pre-stored model answer.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Disclosure of the Invention

Problems to be Solved by the Invention

[0005] However, when a teacher (user) creates a test, the user may have to solve the test questions himself / herself to create a model answer. When preparing test questions such as questions obtained by partially modifying commercially available questions, for which answers are not prepared, it was necessary to create a model answer and register it in the system, which was a cumbersome task.

[0006] An object of the present invention is to reduce the workload of tasks related to tests by users.

Means for Solving the Problems

[0007] A selection receiving means that accepts the selection of a specific answer image that will serve as a model answer from among answer images on which answers to an exam have been written, and a scoring means that scores the answers in the answer image based on the specific answer image selected by the selection receiving means, It is characterized by being equipped with [the following features]. [Effects of the Invention]

[0008] According to the present invention, the workload of users involved in testing can be reduced. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows an example of a system configuration according to the embodiment. [Figure 2] This figure shows an example of the hardware configuration according to the embodiment. [Figure 3] This figure shows an example of a flowchart illustrating the processing performed by the processing unit (CPU) of the information processing device 120 in the embodiment (first embodiment). [Figure 4] This figure shows an example of a screen for selecting how to obtain model answers. [Figure 5] This figure shows an example of a screen for selecting answer images from the list of model answer candidates. [Figure 6] This figure shows an example of a screen for selecting answer images from the list of model answer candidates. [Figure 7] This figure shows an example of a screen for selecting answer images from the list of model answer candidates (a modified version). [Figure 8] This figure shows an example of a model answer candidate answer image or a correction screen for scoring results. [Figure 9] This figure shows an example of a flowchart illustrating the processing performed by the processing unit (CPU) of the information processing device 120 in the embodiment (second embodiment). [Figure 10] This figure shows an example of a display screen showing the results of evaluating the description of intermediate calculation steps or calculation procedures. [Figure 11] This figure shows an example of a display screen showing the results of evaluating the description of intermediate calculation steps or calculation procedures. [Figure 12] It is a diagram showing an example of a display screen of the result of scoring the answers of the examinee. [Figure 13] It is a diagram showing an example of a display screen of the result of scoring the answers of the examinee. [Figure 14] It is a diagram showing an example of a database stored in the information processing apparatus 120 of the embodiment. [Figure 15] It is a diagram showing an example of a screen for allowing a user to select whether to score an answer image.

Mode for Carrying Out the Invention

[0010] Hereinafter, embodiments will be described in detail with reference to the drawings.

[0011] First, referring to FIG. 1, an example of the configuration of an information processing system in an embodiment of the present invention will be described. FIG. 1 is a schematic diagram showing an example of a system configuration according to the embodiment.

[0012] The image data providing apparatus 110 is an apparatus for providing image data to the information processing apparatus 120 described later.

[0013] The image data providing apparatus 110 includes a scanner (reading device) 111, a mobile terminal 112, and a management server 113.

[0014] When the scanner 111 reads a model answer sheet and an answer sheet on which an examinee has answered and filled in the answers, and acquires image data, the scanner 111 can transmit the image data to the information processing apparatus mediated by the communication network 130.

[0015] The mobile terminal 112 can use its camera function to photograph a model answer sheet and an answer sheet on which an examinee has answered and filled in the answers, and transmit the acquired still image or moving image data to the information processing apparatus 120. For example, a smartphone, a tablet terminal, a wearable terminal such as a smartwatch, etc. are assumed, but other devices may also be used.

[0016] The management server 113 can transmit the following data to the information processing device 120 via the communication network 130: the examinee's student ID number data, an image of the answer sheet with the examinee's answers filled in (hereinafter referred to as the answer image), the data extracted from the answer image (for example, part 1, intermediate calculations, calculation results, etc.) (hereinafter referred to as the answer information data), and the results of grading the examinee's answers (hereinafter referred to as the grading results).

[0017] The information processing device 120 may also function as the image data providing device 110, and may be equipped with a camera function that can acquire still images and video image data, such as a mobile terminal 112.

[0018] Network 130 is a network that enables communication between hardware components according to the embodiment (for example, the image data providing device 110 and the information processing device 120).

[0019] The communication method of the communication network 130 may be wireless or wired. It may also be a WAN (Wide Area Network) or a LAN (Local Area Network).

[0020] Figure 2 shows an example of the hardware configuration according to the embodiment.

[0021] An example of hardware according to this embodiment includes at least an image data providing device 110 and an information processing device 120.

[0022] 201 is the CPU, which controls each device connected to the system bus 204 by loading programs stored in the program ROM 202 or external memory 211 into RAM 203 and executing them.

[0023] Furthermore, the font ROM of ROM 202 stores font data, and the data ROM of ROM 202 stores various other data. 203 is RAM, which functions as the main memory and work area of ​​the CPU 201. 202 is the keyboard controller (input controller), which controls input from 209 and pointing devices (not shown), such as a touchpad. The data table, which will be described later, is also stored in ROM 202.

[0024] The input controller 205 controls the input device 209. The input device 209 also includes a touch panel (touchscreen), in which case the input device 209 also functions as the display unit 210 (a display device for showing a display screen for displaying video). The input device 209 is not limited to the above and can utilize any known device. For example, it could be an audio input device or a device that acquires user gestures using a camera or tracker and converts them into input instructions.

[0025] In the flowchart-based explanation described later, the display destination will be the display unit 210 unless otherwise specified.

[0026] 206 is the video card (VC) which controls the display of the display unit 210. The display unit 210 can utilize various existing output devices. For example, screen displays and retinal projection displays are conceivable, but any output device that allows the user to acquire visual information will suffice.

[0027] 207 is the memory controller, which controls access to external memory 211 (hard disk, etc.). 208 is the network interface card (NIC), which controls communication on the communication network 130. As external memory, external storage devices (hard disk), flexible disks (FD), or CompactFlash® memory connected to the PCMCIA card slot via an adapter can be used to store boot programs, various applications, font data, user files, editing files, and various other data.

[0028] The image data acquisition unit controller 220 is a controller that controls the image data acquisition unit 221. The image data acquisition unit 221 is a unit that combines the mechanisms necessary to acquire image data, and can take various forms to suit the characteristics of the device. For example, in the scanner 111, it is a scanning unit that scans images, and in the mobile terminal 112 and the information processing device 120, it is a camera unit that takes still images and videos.

[0029] Furthermore, for the management server 113 and the information processing device 120, an external memory 211 for acquiring and storing image data and a network interface card 208 for controlling the connection to the communication network 130 for acquiring image data from the outside are expected. (First Embodiment)

[0030] First, the processing flow in the first embodiment will be explained using Figure 3.

[0031] The process shown in the flowchart of Figure 3 is the process in which the CPU 201 of the information processing device 120 reads a predetermined control program and executes it.

[0032] In S301, the information processing device 120 displays a selection screen for the method of obtaining the model answer, as shown in Figure 4. The information processing device 120 accepts the user's selection of either the model answer reading mode or the examinee's answer reading mode.

[0033] In this embodiment, we describe a case in which image data is acquired from the scanner 111, but the information processing device 120 may also request the mobile terminal 112, which is connected to the device for communication, to transmit the image data obtained by photographing the paper, and then acquire the image data.

[0034] The information processing device 120 may also request the management server 113, which is connected via communication, to send image data of the paper it is managing, and obtain the image data.

[0035] Figure 4 illustrates an example of a selection screen for obtaining model answers.

[0036] The screen in Figure 4 includes a button 401a that accepts the selection of the model answer reading mode (S316) and a button 401b that accepts the selection of the examinee's answer reading mode (S302). When the user presses either button, the paper is read. Once the reading is complete, the screen transitions to Figure 5.

[0037] Let's return to the explanation of Figure 3. First, let's explain the process when the selection of the model answer reading mode is received. The model answer reading mode is used when model answers are prepared in advance (when model answers have been created in advance by the teacher).

[0038] In S316, when the information processing device 120 receives a selection of the model answer reading mode from the user (S301: Model Answer Reading Mode), it acquires image data of the model answer (hereinafter referred to as the model answer image) from the scanner 111. Specifically, the information processing device 120 issues a model answer reading instruction to the scanner 111. Upon receiving the instruction, the scanner 111 reads the answer sheet and transmits the read model answer image to the information processing device 120. The information processing device 120 receives the model answer image transmitted from the scanner 111. Here, it is assumed that the model answer image also includes the test questions. The information processing device 120 generates a data table (not shown) for managing the read image data. This data table is generated and managed as a separate file for each test date and test subject. The information processing device 120 saves the acquired image data to the data table.

[0039] In S317, the information processing device 120 acquires image data (answer image) of the answer sheet with answers filled in by the examinee from the scanner 111. Specifically, the information processing device 120 issues a reading instruction to the scanner 111. Upon receiving the instruction, the scanner 111 reads the answer sheet and transmits the scanned answer image to the information processing device 120. The information processing device 120 receives the answer image transmitted from the scanner 111. Here, it is assumed that the answer image also includes the test questions. The information processing device 120 stores the acquired answer image data in the data table shown in Figure 14.

[0040] Figure 14 shows an example of a database stored in the information processing device 120. Data table a in Figure 14 consists of student ID number 1401, answer image 1402, answer information 1403, and scoring result 1404.

[0041] Student ID number 1401 is registered as the student ID number of the examinee. The answer image 1402, answer information 1403, and scoring result 1404 are stored linked to the corresponding examinee's student ID number 1401. For example, data table a stores data linked to examinees with student ID numbers 0001 to 0005.

[0042] Answer image 1402 stores the answer image data of examinees read by S302. For example, data table a stores answer image data (touan01 to touan05) associated with examinees with student ID numbers 0001 to 0005.

[0043] The answer key (1403) and scoring results (1404) will be explained later.

[0044] In S318, the information processing device 120 performs optical character recognition (OCR) on the answer image acquired in S317. It acquires answer information, including the characters contained in the answer image. The answer information to be acquired may include not only the answer result, but also the scene diagram, calculation steps, and calculation result contained in the answer field. The information processing device 120 saves the acquired answer information in a data table (Figure 14). The processing in this step is the same as the processing in S303, so the details will be explained in S303.

[0045] Let's return to the flowchart in Figure 3.

[0046] Once processing in S318 is complete, the information processing device 120 transitions to S310. The processing from S310 onward is the same as when the selection of the examinee answer reading mode, which will be explained next, is accepted.

[0047] Next, we will explain the case where the examinee's answer reading mode is selected. The examinee's answer reading mode is a mode that executes the process of selecting an appropriate answer as a model answer from among the examinee's answer sheets.

[0048] In S302, when the information processing device 120 receives a selection of the examinee answer reading mode from the user (S301: examinee answer reading mode), it acquires the answer image from the scanner 111. Specifically, the information processing device 120 issues a reading instruction to the scanner 111 for the answer sheet. Upon receiving the instruction, the scanner 111 reads the answer sheet and transmits the read answer image to the information processing device 120, which then acquires it. Here, it is assumed that the answer image also includes the test questions. The information processing device 120 generates a data table (Figure 14) for managing the read image data. This data table is generated and managed as a separate file for each test date and test subject. The information processing device 120 saves the acquired image data to the data table.

[0049] In S303, the information processing device 120 performs OCR on the answer image acquired in S302. It acquires answer information, including the characters contained in the answer image. The answer information to be acquired may include not only the answer result, but also the scene diagram, calculation steps, and calculation result contained in the answer field. The information processing device 120 stores the acquired answer information in a data table (Figure 14).

[0050] In this embodiment, the entire process, from character detection to character recognition, is referred to as optical character recognition (OCR).

[0051] The OCR process is explained below. First, the information processing device 120 performs character detection on the entire image to be OCR. Specifically, the information processing device 120 finds the outlines of characters using methods such as template matching and edge detection, and detects the areas containing characters. From the detected areas, the information processing device 120 further divides them into individual areas, taking into account the distance between characters and the shape of the characters. From the detected individual areas, the information processing device 120 extracts features for character identification and performs character detection.

[0052] Subsequently, the information processing device 120 performs character recognition on the detected character. Specifically, the information processing device 120 uses template matching, pattern recognition methods, etc., to compare the shape and pattern of the character, identify the correct character, and recognize it as text. The information processing device 120 then links the results of the character recognition to each individual area.

[0053] In S304, the information processing device 120 performs an evaluation of the characters contained in the answer image that was OCR'd in S303 (evaluation means).

[0054] The evaluation of characters refers to the appearance and form of the characters, such as their beauty, balance, size, and lightness (or darkness), and is primarily based on whether the characters are easy to read.

[0055] Regarding the evaluation results, the evaluation from each perspective may be quantified, and the average of the evaluation results from all perspectives may be used as the evaluation value, or the sum of the results may be used as the evaluation value. Furthermore, quantification will allow for comparison between multiple answer images.

[0056] Regarding the evaluation method for characters, it may be evaluation using pattern matching or other image analysis techniques, evaluation using character evaluation algorithms, or evaluation using AI (a pre-trained model).

[0057] Specifically, regarding the aesthetics of handwriting, for example, one possible method would be to prepare example handwriting data, compare the handwriting of the examinee with that data, quantify the difference between the two, and judge the handwriting as aesthetically pleasing if the value is below a threshold, and as not aesthetically pleasing if it is above the threshold.

[0058] Regarding font size, one possible method is to quantify the size of the margin that remains when the outline of the letters written by the test-taker is drawn around it. If this value is above a threshold, the letters are judged to be small; if it is below the threshold, the letters are judged to be large.

[0059] Regarding the balance of the characters, one possible method is to quantify the size of the margin outside the outline when each character written by the test-taker is circled. If the numerical values ​​are similar for all characters, the balance is judged to be good; if there is a large variation in the numerical values, the balance is judged to be poor.

[0060] Regarding the darkness of the text, possible methods include determining it based on the RGB values ​​and hexadecimal color codes of the OCR-received text.

[0061] AI-based evaluation can be achieved by using an AI trained on training data consisting of, for example, images of characters and numerical evaluations of those images (e.g., Image A scores 58 points, Image B scores 73 points). An AI trained in this way will evaluate the input characters and output the evaluation as a numerical value.

[0062] In S305, the information processing device 120 selects candidate answer images from among the examinees' answer images based on the evaluation results evaluated in S304. Specifically, it selects a predetermined number of images (for example, 10 images) in descending order of evaluation value. It may also combine this with other conditions, such as the evaluation value being above a threshold.

[0063] In this way, by selecting answers from test-takers with high handwriting scores as model answer candidates, users no longer need to choose a model answer from all the test-takers' answer images, making it possible to select a model answer efficiently.

[0064] Furthermore, by selecting model answer candidates based on the appearance and form of the characters, such as the neatness of the handwriting, it becomes possible to select an easy-to-read answer. This allows users to efficiently select a model answer from among the candidate options.

[0065] Furthermore, since the model answer can be determined from a selection of model answer candidates based on the appearance and form of the characters, users can focus solely on the answer result (whether it is appropriate as an answer, etc.) without having to consider the appearance and form of the characters when deciding on a model answer, thus reducing the effort required from the user.

[0066] In S306, the information processing device 120 divides the answer images of the examinees selected in S305 as model answer candidates into sections for each question and displays them for each question (Figures 5 to 7).

[0067] The display of each question is determined by recognizing strings such as "Question 1" and "Question 2," or numbers enclosed in squares, and separating them at these points. Alternatively, questions can be displayed by separating them into sub-questions. That is, the system recognizes strings indicating sub-questions, such as "(1)," and separates the questions based on these strings. By separating questions and displaying one question per page, it becomes easier to select a model answer for each question. Furthermore, further subdividing the question divisions increases the flexibility in creating model answers.

[0068] In this embodiment, the evaluation of handwriting, such as the neatness of the writing, is determined by the examinee (for the same examinee, the evaluation of handwriting does not change regardless of the answer to any question, and the evaluation of handwriting does not usually change significantly from question to question). Therefore, the procedure involves selecting examinees with high handwriting evaluations and dividing their answers by question to create model answers for each question. However, it is also possible to configure the system to select examinees with high handwriting evaluations for each question.

[0069] Figures 5 and 6 will be used to explain the screen for selecting the answer image of the model answer. Figures 5 and 6 show examples of the screen for selecting the answer image of the model answer candidate, where the answer image is displayed. Figure 5 shows an example of the screen displaying the answer image of the first page (question 1) of the six pages, for examinee with student ID number 0002, who was selected in step S305. Figure 6 shows an example of the screen displaying the answer image of the second page (question 2) of the six pages, for examinee with student ID number 0003, who was selected in step S305.

[0070] As shown in Figures 5 and 6, the selection screen displays the answer image display area 501, the model answer selection button 502, the student ID number display counter 503, the counter advance button 504, the counter advance button 505, the display advance button 506, the display advance button 507, the answer page display button 508, and the correction button 509.

[0071] The answer image display area 501 is the area that displays the acquired answer images, and the answer image corresponding to the page displayed in the answer page display 508 is displayed there. For example, in Figure 5, "1 / 6 pages" indicates that the first page (question 1) of the six pages of answer images is displayed in the answer image display area 501. On the other hand, in Figure 6, "2 / 6 pages" indicates that the second page (question 2) of the six pages of answer images is displayed in the answer image display area 501.

[0072] The model answer selection button 502 is a button that accepts instructions to select the answer image displayed in the answer image display area 501 as the model answer.

[0073] When a user presses the model answer selection button 502, the display of the model answer selection button 502 changes (for example, its color changes), and the answer image displayed in the answer image display area 501 is selected as the model answer. Screen (A) in Figure 5 shows the state when the model answer selection button 502 has not been pressed, and the answer image of examinee with student ID number 0002 has not been selected as the model answer. When the model answer selection button 502 is pressed, the display of the model answer selection button 502 changes, as shown in screen (B), indicating that the answer image of examinee with student ID number 0002 has been selected as the model answer. Furthermore, for example, if the answer image of examinee with student ID number 0003 is selected as the model answer in Figure 5, and then the answer image of examinee with student ID number 0002 is selected as the model answer, the selection of examinee with student ID number 0003's answer image as the model answer will be canceled.

[0074] The student ID number display counter 503 is an area that displays information (student ID number in this embodiment) that identifies the examinee whose answer image is displayed in the answer image display area 501. For example, since the student ID number display counter 503 in Figure 5 displays "0002", the user can recognize that the answer of examinee with student ID number 0002 is displayed in Figure 5. On the other hand, since the student ID number display counter 503 in Figure 6 displays "0003", the user can recognize that the answer of examinee with student ID number 0003 is displayed in Figure 6.

[0075] Counter feed 504 and counter feed 505 are buttons that accept instructions to switch to the answer images of other examinees selected in S305.

[0076] When the counter feed button 504 is pressed in the state shown in Figure 5, the closest student ID number among the examinees selected in S305 that is smaller than the student ID number currently displayed in the student ID number display counter 503 will be displayed, and the answer image displayed in the answer image display area 501 will also switch in accordance with the student ID number. For example, if "0003" is displayed in the student ID number display counter 503, and the answer image of examinee "0002" has been selected as a model answer candidate, it will switch to "0002", otherwise it will switch to a number lower than "0001". The answer image displayed in the answer image display area 501 will display the answer image associated with the switched number.

[0077] On the other hand, in the state shown in Figure 5, when the counter feed button 505 is pressed, the closest student ID number among the examinees selected in S305 that is greater than the student ID number currently displayed in the student ID number display counter 503 is displayed, and the answer image displayed in the answer image display area 501 is also switched in accordance with the student ID number. For example, if "0003" is displayed in the student ID number display counter 503, and the answer image of examinee "0004" has been selected as a model answer candidate, it will switch to "0004", otherwise it will switch to a number from "0005" onwards. The answer image display area 501 will display the answer image associated with the switched number.

[0078] In this embodiment, pressing the counter feed buttons 504 and 505 switches the student ID numbers on the student ID number display counter 503 in descending order of numerical value, but they may be switched in other orders. For example, they may be switched in descending order of the average score in the tests taken so far, which allows for quick selection of answer images of examinees who have provided excellent answers.

[0079] The display advance buttons 506 and 507 are buttons that, when pressed by the user, receive instructions from the user to switch between the answer page display 508 and the page (problem) of the answer image displayed in the answer image display area 501.

[0080] When the display advance button 506 is pressed, the answer page display 508 and the answer image display area 501 switch to the previous page (problem).

[0081] When the display advance button 507 is pressed, the answer page display 508 and the answer image display area 501 switch to the next page (problem). That is, "Page 2 / 6" is displayed as shown in the answer page display 508 in Figure 6.

[0082] The correction button 509, when pressed by the user, switches to a screen that accepts correction instructions from the user (correction instruction acceptance mode). When the system switches to correction instruction acceptance mode, an additional box 803 appears in the lower right corner of the answer image display area 501, allowing the examinee to correct their answer information (Figure 8). When the correction button 509 is pressed again, the additional box 803 disappears, and the acceptance of correction instructions ends. Correction will be explained in step S307.

[0083] Thus, according to this embodiment, it is possible to select a test-taker's answer image to be used as a model answer for each question. For a single question, the user can compare answer images from different test-takers by pressing counter buttons 504 and 505, and decide which test-taker's answer image to select as the model answer based on the content of the answer image.

[0084] Figure 7 illustrates a modified version of the screen for selecting the answer image of the model answer.

[0085] Figure 7 shows a modified version of the answer selection screen for model answer candidates shown in Figures 5 and 6. Figure 7 displays the same question and the same test-taker's answer image as in Figure 6. It consists of an answer image display area 701, an answer field 702, a cursor 703, an answer field advance button 704, and an answer field advance button 705.

[0086] The answer image display area 701 is a screen that displays the acquired answer images, showing the answer images corresponding to the pages displayed in the answer page display 508 (page 2 / 6 in Figure 7). The answer images are separated by the number of the main question and each sub-question, similar to Figures 5 and 6. Answer information (intermediate calculation steps 710a, calculation result 710b) extracted from the answer images using OCR processing is identified and displayed. In addition, although the question text is omitted in the example in Figure 7, the question text may be displayed as the answer image display area 701 is scrolled. For example, scrolling upwards would display the main question number, the question text "Find the angle of X below," and a star-shaped figure, similar to Figure 6.

[0087] If you wish to select the answer image displayed in the answer image display area 701 as the model answer, you can do so by pressing the model answer selection button 508.

[0088] Answer field 702 displays only the answer results for each test-taker, and allows for selections to be made by each test-taker.

[0089] The cursor 703 is a cursor that identifies and displays the answer image selected by the user and displayed in the answer image display area 701. For example, in Figure 7, the answer image of examinee with student ID number 0003 is displayed in the answer image display area 701, and the answer related to the examinee being displayed is identified and displayed by the cursor 703.

[0090] If a user wishes to switch from the answer information currently displayed in the answer image display area 701 to the answer information of another test-taker, they can do so by clicking on the answer result of the desired test-taker in the answer field 702. The cursor position will also change depending on the selected answer.

[0091] The answer field advance buttons 704 and 705 are buttons that receive instructions from the user to switch pages (questions) of the answer images displayed in the answer field 702.

[0092] When the answer field advance button 704 is pressed, the answer field 702 switches to the next page (question).

[0093] When the answer field advance button 705 is pressed, the answer field 702 switches to the previous page (question).

[0094] As shown in the modified version of Figure 7, by displaying a list of answer results from multiple test-takers selected as model answer candidates, users can easily compare the answer results of multiple test-takers at a glance, making it easy for them to select the answer that provides the correct answer from among the candidates as the model answer.

[0095] Note that the answer image display area 701 and the answer field 702 do not necessarily have to be integrated as shown in Figure 7; for example, the answer image display area 701 may be displayed as a pop-up or in a separate window. In other words, it may be possible to switch between displaying the answer image display area 701 and displaying the answer field 702. For example, the default state is to display only the answer field 702, and when the user selects an answer from a test-taker for whom they want to display the answer image display area 701, the answer image display area 701 is displayed as a pop-up or in a separate window. This allows both the answer field 702 and the answer image display area 701 to be displayed larger. If the answer field 702 is displayed larger, fewer operations are required to switch the display of the answer image.

[0096] Furthermore, the calculation procedure evaluation result 1001 in Figure 7 will be explained using the flowchart in Figure 9.

[0097] As described above, by displaying answer images with high ratings for handwriting as model answer candidates, the effort required for users to consider the beauty and balance of handwriting from among numerous answer images is reduced, and they only need to focus on the content of the test-taker's answer when selecting a model answer. In addition, since the test-taker's answer image is selected as a model answer, the burden on users to solve the problem from scratch and prepare a model answer is reduced.

[0098] Let's return to the flowchart in Figure 3.

[0099] In S307, the information processing device 120 determines whether or not it has received an instruction from the user to correct the answer. If it has received the instruction (S307:YES), it proceeds to S308 and corrects the answer image according to the user's instructions. On the other hand, if it has not received the instruction (S307:NO), it proceeds to S309.

[0100] In S308, the information processing device 120 can correct an answer image as a model answer according to the user's operation via a selection screen of model answer candidate answer images (Figures 5 to 7) (correction means). The corrected answer image is saved as a model answer candidate, overwriting the existing answer image.

[0101] The specific method for correcting the answer image is explained in Figure 8.

[0102] Figure 8 shows an example of the correction screen for the answer image and scoring results. It consists of a border 801, a delete button 802, and an add box 803. Figure 8 is based on Figure 5, with some screens omitted for explanatory purposes.

[0103] Figures 8a-8c are simplified versions of the answer images of examinee with student ID number 0002 in Figure 5. When the correction button 509 in Figure 5 is pressed, an additional box 803 appears in the lower right corner of the screen, allowing for corrections.

[0104] Border 801 is a border that surrounds text or shapes. Border 801 has handles that allow the user to adjust the size and position of the object used to specify the area, and it accepts user input via drag and drop, allowing the user to adjust it as desired.

[0105] For example, Figure 8 shows what happens when a user tries to adjust an area by hovering the mouse over the handles indicated by "○" located at the center of the edge and four vertices of the border line 801 and dragging them.

[0106] When the delete button 802 is pressed, the text or shape enclosed by the border 801 is deleted.

[0107] The additional box 803 consists of buttons that allow selection of shapes. When the user presses a corresponding button, shapes, text boxes, etc., can be added. For example, if the user presses the rightmost "circle" button and then drags it anywhere within the answer image display area 501, a circular shape will be added.

[0108] Figure 8 shows selectable shapes such as circles, triangles, lines, crosses, text boxes, and arrows, but other shapes may also be used. Furthermore, the color of the shapes and the thickness of their outlines may be changed.

[0109] This section explains how to correct the answer image of the model answer candidate using Figure 8, but the same method can be used to correct the scoring results.

[0110] Figure 8a shows that student ID number 0002 made a mistake in both the intermediate steps of the calculation and the final answer. Therefore, the incorrect intermediate steps and final answer will be deleted, and the correct intermediate steps and final answer will be added.

[0111] In Figure 8a, a text box containing the correct calculation steps has been added to the right side of the screen. The text box is added when the user clicks the "Text" button 804, the fifth button from the right in the add box 803, and then drags to the desired location. The added text box is surrounded by a border 801. The user enters the correct calculation steps, "3000 ÷ 150 = 20 minutes," into this text box.

[0112] Alternatively, instead of using text boxes, users could enter handwritten text in any location, and that text could then be converted into text boxes. This would allow users to make corrections in a way that mimics grading paper answer sheets.

[0113] In Figure 8b, "3000 ÷ 150 = 20 minutes" (correct calculation steps) and "3000 ÷ 150 = 30 minutes" (incorrect calculation steps) are enclosed in box 801.

[0114] First, the user drags the incorrect calculation "3000 ÷ 150 = 30 minutes" and surrounds it with a border 801. When the delete button 802 on the border 801 is pressed, the incorrect calculation is deleted. Then, when the text containing the correct calculation "3000 ÷ 150 = 20 minutes" is moved to the original location of the incorrect calculation, the screen transitions to the state shown in Figure 8c.

[0115] In Figure 8c, after repeating the above operation, all incorrect calculation steps and the final answer have been deleted and replaced with the correct calculation steps and the final answer.

[0116] When the correction button 509 is pressed again, the correction is completed and the system transitions to Figure 5, 6, or Figure 7.

[0117] By correcting the answer images of the model answer candidates through the above process, users can correct the answer images and use them as model answers. This process is effective, for example, when you want to use an answer as a model answer that has legible text and carefully written intermediate calculations, but contains careless mistakes such as forgetting to include units.

[0118] Let's return to the flowchart in Figure 3.

[0119] In S309, the information processing device 120 registers the answer image selected when the model answer selection button 502 is pressed as the model answer. The selected model answer image and answer information are stored in a data table separate from data table 14 (not shown).

[0120] Once the registration of model answers for all questions is complete, the screen shown in Figure 15 is displayed, and instructions are received regarding whether to grade the examinee's answers. If the grade button 1501 is pressed, the process from S310 onwards is executed. If the back button 1502 is pressed, the process returns to S305 or S306, and the selection of model answers is redone.

[0121] In S310, the information processing device 120 performs scoring by comparing the examinee's answer information obtained in S303 with the model answer information registered in S309. This process reduces the burden on users who would otherwise have to manually score examinee answer sheets.

[0122] Specifically, the system determines whether an answer is correct or incorrect based on whether the examinee's answer information matches the model answer information, and then scores the answer. Alternatively, scoring may be performed using an external large-scale language model (generative AI). Specifically, the system outputs the examinee's answer information, the model answer information, and instructions to score the examinee's answer based on the model answer as prompts to the generative AI, and obtains the scoring results from the generative AI. In addition to determining whether an answer is correct or incorrect, the system may also be instructed to award partial points or generate comments indicating that the calculation process is correct if the answer is incorrect but the calculation process is correct.

[0123] In S311, the information processing device 120 displays the scoring results from S310. Figures 12 and 13 are examples of screens displaying the scoring results.

[0124] As shown in Figures 12 and 13, the scoring result display screen shows the scoring result display area 1201, score 1202, evaluation object 1203, model answer 1204, correction button 1205, exit button 1206, question text 1207, and scoring reason 1208.

[0125] Figure 12 shows the scoring results for the same test-takers' answers as those in Figures 5-7 and 10-11.

[0126] Figure 13 shows the scoring results for Question 2 (a reading comprehension question requiring written answers) of examinee with student ID number 0005, outlining the Japanese language exam which consists of a total of six major questions.

[0127] The scoring result display area 1201 is the area where the information processing device 120 displays the results (scoring results) obtained when it scored the examinee's answers in S310.

[0128] The score 1202 displays the score for each question on the scoring result display screen 1201. For example, in Figure 12, the answer image of examinee with student ID number 0002 is scored 7 points for question 1. In Figure 13, the answer image of examinee with student ID number 0005 is scored 12 points for question 2.

[0129] In addition to the points for each question, the total score for the entire test may also be displayed.

[0130] Scoring symbol 1203 displays the scoring result using scoring symbols. Symbols such as a circle (correct answer for a completely correct response), a triangle (partially correct answer with points awarded), and an X (incorrect answer with no points awarded) are displayed in close proximity to the corresponding score.

[0131] For example, in Figure 12, the answer image of examinee with student ID number 0002 shows the correct number "30," but the unit "minutes" is not indicated, so the scoring symbol 1203 is a triangle, and the score is 7 points.

[0132] On the other hand, in Figure 13, the answer image of examinee with student ID number 0005 is graded with a triangle at the top, giving it 7 points, and a circle at the bottom, giving it 5 points.

[0133] Furthermore, the score 1202 and the scoring symbol 1203 may be displayed in a different color (for example, red) to distinguish them from the answer information and the question text.

[0134] Model answer 1204 is a model answer registered in S309 or S316. Users can compare the model answer with the examinee's answer, allowing them to efficiently check the accuracy of their scoring results.

[0135] The correction button 1205, when pressed by the user, switches to a screen that accepts correction instructions from the user (correction instruction acceptance mode).

[0136] For example, in Figure 12, the examinee with student ID number 0002 is graded "Triangle" because the numerical part of the answer is correct, but the examinee made a mistake in the intermediate steps of the calculation. Therefore, it cannot be said that the examinee correctly understood and solved the problem, and the user may decide that there is no need to award partial points. In such cases, the user can correct the grading result to an incorrect "X" without awarding partial points as needed (S312). The screen for accepting corrections is the same as the screen in Figure 8.

[0137] The Exit button 1206 is used when the user has finished reviewing the scoring results displayed on the scoring results screen 1201 and has finished using the system. When pressed by the user, the scoring results are saved to the data table and the system is terminated.

[0138] Problem statement 1207 displays the problem statement of the question answered by the test-taker. Figure 12 shows a math problem, and Figure 13 shows a reading comprehension problem in Japanese language arts, both of which require written answers.

[0139] The scoring reason 1208 displays the reason (a comment generated by the aforementioned AI) when an answer is graded as partially correct. For example, in Figure 12, the scoring result for examinee with student ID number 0002 is a triangle, meaning partially correct. The reason given is "No credits," indicating that points were deducted because the examinee did not have credits. In Figure 13, the answer of examinee with student ID number 0005 is graded as partially correct with a triangle at the top. The reason given is "No description of 'accepting grace and greatness'," indicating that points were deducted because some of the required content was not included. By displaying the reason for a partially correct answer, users and examinees can see why their answer was only partially correct. Furthermore, by returning the test to examinees with scoring reason 1208 displayed, they can learn how to answer correctly and use that knowledge to improve their next test.

[0140] By displaying the scoring results as described above, users can visually verify the accuracy of their scores. Furthermore, since the scoring results can be corrected and saved as needed, it becomes possible for users to manually rescore their scores or add scoring criteria or comments to test takers (not shown).

[0141] Let's return to the flowchart in Figure 3.

[0142] In S312, the information processing device 120 determines whether or not it has received an instruction from the user to correct the scoring result. If it has received the instruction (S312:YES), it proceeds to S313 and corrects the scoring result according to the user's instructions. On the other hand, if it has not received the instruction (S312:NO), it proceeds to S314.

[0143] In S313, the information processing device 120 can correct the scoring result according to the user's operation (correction means) (Figure 8). The corrected scoring result is saved by overwriting the scoring result before correction.

[0144] In S314, when the information processing device 120 receives a user instruction to end scoring, it registers the scoring result for that question in the data table (Figure 14) and proceeds to check the scoring results for the next examinee. Once the scoring results for all examinees and all questions have been checked (the scoring results have been registered), the process in this flowchart ends.

[0145] As described above, this embodiment reduces the burden on users to create model answers by selecting the test taker's answer as the model answer. It is particularly effective for question formats where creating model answers is complicated, such as calculation problems and proof problems in arithmetic and mathematics tests that require showing intermediate steps, and reading comprehension problems in language tests. It reduces the burden on users to solve problems and register them as model answers. (Second Embodiment)

[0146] Next, as a second embodiment, we will describe an embodiment in which a process (Figure 9) is added to assist in selecting a model answer using AI (a trained model). In the second embodiment, the process shown in the flowchart in Figure 9 is executed between S305 and S306 in Figure 3. Therefore, the explanation of the process in the flowchart in Figure 3 will be omitted, and the added process (the process shown in the flowchart in Figure 9) will be explained.

[0147] The process shown in the flowchart of Figure 9 is for determining whether the model answer candidate selected in S305 is appropriate as a model answer. The process shown in the flowchart of Figure 9 is for the CPU 201 of the information processing device 120 to read and execute a predetermined control program.

[0148] In S901, the information processing device 120 determines whether the problem is a calculation problem based on the results of the OCR processing in S303. For example, if the problem statement contains strings that suggest a calculation problem, such as "calculation" or "expression," or if it contains symbols indicating arithmetic operations or symbols such as "=" or "√," it can be determined to be a calculation problem.

[0149] If the test is a calculation problem (S901:YES), proceed to S902; otherwise, proceed to S904.

[0150] In S902, the intermediate calculations or calculation procedures included in the solution image are evaluated (calculation evaluation means).

[0151] Specifically, the generating AI is output as prompts the problem statement, the examinee's answer, and an instruction to evaluate whether the examinee's answer is appropriate, and the evaluation result from the generating AI is obtained. Alternatively, instead of instructing the generating AI to perform the evaluation, the prompts are output to generate the problem statement and appropriate intermediate calculations and calculation procedures, and the evaluation is performed by comparing the intermediate calculations and calculation procedures generated by the generating AI with the examinee's answer (determining whether they match or not).

[0152] In S903, the system determines whether there are errors in the calculation procedure based on the evaluation results (specifically, whether the intermediate steps of the calculation are not shown or are incorrect). If the intermediate steps of the calculation are sufficiently shown in the answer image and the content is correct (S903: NO), the system outputs an evaluation result indicating that the answer image is appropriate as a model answer.

[0153] If the solution image is evaluated as not showing intermediate calculation steps or as being incorrect (S903:YES), an evaluation result indicating that the solution image is not suitable as a model answer will be output.

[0154] Figures 10 and 11 show examples of display screens showing the results of evaluating the description of intermediate calculation steps or calculation procedures.

[0155] The calculation procedure evaluation result 1001 is the evaluation result of the intermediate calculations or calculation procedures for the examinee's answer displayed in the answer image display area 501.

[0156] For example, in Figure 10, the answer of examinee with student ID number 0002 has an alert "X" displayed in the calculation procedure evaluation result 1001 because the intermediate steps and calculation procedures are not adequately described.

[0157] On the other hand, the answer of examinee with student ID number 0003 in Figure 11 was marked with a circle (○) in the calculation procedure evaluation result 1001 because the intermediate steps and calculation procedures were sufficiently described.

[0158] As described above, by notifying users of the evaluation of the calculation procedure, it becomes easier for users to decide on a model answer from among the candidate answers, taking into account not only aspects such as the neatness of the handwriting, but also the appropriateness of the content of the answer.

[0159] In S904, the information processing device 120 determines whether the problem is a reading comprehension problem based on the results of the OCR processing in S303. For example, it may be determined to be a reading comprehension problem if the string of characters in the problem text contains words such as "modern Japanese" or "reading comprehension," or if the number of characters or lines in the problem text exceeds a predetermined number.

[0160] If the test is a reading comprehension question (S904: YES), proceed to S905; otherwise, proceed to S306.

[0161] In S905, the information processing device 120 acquires the long passage from the reading comprehension question and the test question from the results of the OCR processing performed in S303 (acquisition means).

[0162] In S906, a generative AI (large-scale language model) is used to generate example answers and evaluation metrics for the test questions obtained in S905. Specifically, the generative AI is given prompts consisting of the long passage obtained in S905, the test questions, and instructions to generate answers to the test questions based on the long passage, and the answers generated by the generative AI are obtained. The prompts also include instructions to generate evaluation metrics for the answers (points that serve as the basis for partial credit), thereby causing the generative AI to generate metrics for the answers. Evaluation metrics for answers include, for example, the number of characters in the answer, the endings of the answers, the words and phrases included in the answers, and the gist of the content of the answers. Specifically, in the case of the question shown in Figure 13, the evaluation metrics for the answer include that it describes the author's feelings at the time of underlined section A, that it is written in 30 characters or less, that it describes accepting the blessings and greatness of nature, and that it describes eating without being picky.

[0163] In S907, the information processing device 120 scores the answer images of the model answer candidates based on the answers and evaluation indicators generated in S906. Specifically, the information processing device 120 compares each examinee's answer to each question with the generated answer to determine whether it is correct or incorrect, determines whether partial credit can be awarded based on the generated evaluation indicators, calculates the score for each question based on the determination result, and calculates the examinee's total score by summing the scores for each question. Note that scoring may also be performed using a generation AI.

[0164] In S908, the information processing device 120 selects an answer image whose total score is equal to or greater than a predetermined score, based on the scoring results in S907. The selected answer image is displayed in S306 as a model answer candidate.

[0165] In this embodiment, the candidate model answers were narrowed down based on the total score of the test takers. However, the candidates could also be narrowed down based on the score for each question. Alternatively, the candidate selection could be omitted, and the user could be notified of their score to facilitate the user's decision on the model answer.

[0166] By using AI to evaluate and score the calculation steps of candidate model answers in this way, it becomes easier for users to consider the appropriateness of the answer content when deciding on a model answer. This embodiment is particularly useful for problems such as calculation problems and reading comprehension problems, where the user needs to understand each question and the test-taker's answer individually during scoring, making scoring time-consuming.

[0167] On the other hand, the answer data generated by AI is not always accurate, and the scoring results based on that answer data are not always accurate either. Therefore, by having the user ultimately select a model answer and scoring based on that selected model answer, it is possible to prevent the decrease in scoring accuracy that can occur due to the instability of AI accuracy.

[0168] As described above, the present invention can reduce the workload associated with testing. Specifically, it can reduce the burden on users who have to solve test problems, create model answers, and register them in the system.

[0169] In this invention, as an example, an example is described in which image data is generated by acquiring image data of model answers and answer sheets of each examinee, and provided to an information processing device 120 via a communication network 130.

[0170] However, this is merely one example, and you should appropriately select and use image data that contains the information to be evaluated.

[0171] For example, the source images for the image data may include essays written on composition paper, reports written on report paper, or illustrations drawn in the work entry section. In addition, the image data providing device 110 includes various other devices for acquiring, managing, and supplying content to be evaluated.

[0172] In addition to the above, it is possible to receive and implement an invention by including copyrighted works such as text, drawings, videos, and program code that may be subject to evaluation within the scope of image data.

[0173] A recording medium containing a program that implements the functions of the above-described embodiment may be supplied to a system or device, and the computer (or CPU or MPU) of that system or device may read and execute the program stored on the recording medium.

[0174] At least the following can be used as recording media for supplying the program.

[0175] Specifically, these include, for example, flexible disks, hard disks, optical disks, magneto-optical disks, CD-ROMs, CD-Rs, DVD-ROMs, magnetic tapes, non-volatile memory cards, ROMs, EPROMs, silicon disks, and solid-state drives. Programs may also be downloaded and read from a network server, database, etc., using a communication program.

[0176] Furthermore, it goes without saying that the functions of the aforementioned embodiments are realized not only by the computer executing the program it has read, but also by the operating system (OS) running on the computer performing some or all of the actual processing based on the instructions of that program, thereby realizing the functions of the aforementioned embodiments.

[0177] Furthermore, it goes without saying that this also includes cases where, after a program read from a recording medium is written to the memory of a function expansion board inserted into a computer or a function expansion unit connected to a computer, the CPU or other components of the function expansion board or function expansion unit perform some or all of the actual processing based on the instructions of the program code, and the functions of the aforementioned embodiments are realized through that processing.

Claims

1. A selection acceptance method that accepts the selection of a specific answer image that serves as a model answer from among answer images containing answers to an exam, A scoring means that scores the answers in the answer image based on the specific answer image selected by the selection receiving means, An information processing device characterized by comprising:

2. The aforementioned solution image consists of multiple questions, The information processing apparatus according to claim 1, wherein the selection receiving means accepts the selection of a specific answer image for each of the problems.

3. The information processing apparatus according to claim 1, characterized in that the scoring means evaluates the answer image based on answer information obtained from the string of characters contained in the answer image.

4. The system further comprises correction means for correcting the aforementioned answer information, The information processing apparatus according to claim 3, characterized in that the scoring means scores the answer in the answer image based on the answer information corrected by the correction means.

5. The system includes a first evaluation means for evaluating the characters contained in the answer image, The information processing apparatus according to claim 1, characterized in that the first evaluation means evaluates the beauty, balance, size, or shading of the characters.

6. The system includes a selection means for identifying candidate answer images that are model answers based on the evaluation results from the first evaluation means, The information processing apparatus according to claim 5, wherein the selection receiving means receives the selection of a specific answer image from among the answer images that are candidates for the model answer.

7. A second evaluation means for evaluating the intermediate calculation steps included in the aforementioned solution image, The information processing apparatus according to claim 1, further comprising an output means for controlling the output of an alert for the answer image if the evaluation by the second evaluation means determines that the answer image is not suitable as a specific image.

8. The information processing apparatus according to claim 7, characterized in that the second evaluation means evaluates that an image is not suitable as the specific image if the intermediate calculation steps are not described or if there are errors in the intermediate calculation steps.

9. An output means that outputs instructions to generate answers to descriptive questions and evaluation metrics for those answers, An acquisition means for acquiring the answer and the evaluation index of the answer, which are generated based on the instructions output by the output means. Furthermore, The scoring means scores the answer image based on the answer and the evaluation index of the answer. The information processing device according to claim 1, characterized in that the selection receiving means receives the selection of a specific answer image that is a model answer from among the answer images that have scored by the scoring means and have a score of a predetermined amount or higher.

10. The selection receiving means of the information processing device includes a selection receiving step that accepts the selection of a specific answer image that is a model answer from among answer images on which answers to the test have been written, The scoring means of the information processing device scores the answers in the answer image based on the specific answer image selected by the selection receiving means, An information processing method characterized by comprising:

11. A program for causing at least one computer to function as one of the means described in any one of claims 1 to 9.

Images