Calculation worksheet grading system and program

Abstract

This technology enables automatic grading of calculation worksheets with handwritten answers. [Solution] A calculation worksheet scoring system 10 that performs the following processes for scoring: receiving an image file of an answered worksheet 38 with handwritten answers for each calculation problem printed on the worksheet; performing character recognition on the image file of the answered worksheet 38 to identify the characters that make up each calculation problem and answer and their positions; performing calculations on each calculation problem and determining if it is correct if the calculation result matches the answer value, and if it is incorrect if they do not match, if the answer value does not exist, or if the characters that make up the answer cannot be recognized; identifying the position to place a "○" mark for correct calculation problems; identifying the position to place a "✓" mark for incorrect calculation problems; generating a scored worksheet 52 by placing "○" marks and "✓" marks at the corresponding positions on the answered worksheet 38; and outputting this to an external source.

Description

Technical Field

[0001] This invention relates to a calculation print scoring system and a program, and particularly to a technology for automatically performing scoring processing on calculation prints in which arithmetic learners such as elementary school children fill in answers by handwritten.

Background Art

[0002] When learning arithmetic, since it is effective to actually solve a large number of calculation problems by hand, calculation prints are utilized in elementary school classes, learning schools, or at home. Generally, a large number of calculation problems are printed on one page of a calculation print, and learners solve each calculation problem in order and write the answers by hand in the answer space near the mathematical formula.

Non-Patent Document 1

[0003] In order to make the learning using this calculation print substantial, it is prohibited to leave it unfinished, and the execution of scoring after the answers are completed is required, but this has been a heavy burden for teachers and guardians.

Disclosure of the Invention

Problems to be Solved by the Invention

[0004] This invention has been devised in view of such a conventional situation, and aims to provide a technology that enables automatic scoring for calculation prints with handwritten answers filled in.

Means for Solving the Problems

[0005] To achieve the above objective, the calculation worksheet scoring system described in claim 1 is characterized by the following processes: receiving an image file of an answer sheet with handwritten answers for each calculation problem printed on the calculation worksheet as the target for scoring; performing character recognition on the image file of the answer sheet to identify the characters and their positions that constitute each calculation problem and answer; performing calculations on each calculation problem and determining if it is correct if the calculation result matches the value of the answer, and determining if it is incorrect if they do not match, if the value of the answer does not exist, or if the characters that constitute the answer cannot be recognized; identifying the position to place a correct answer icon for correct calculation problems; identifying the position to place an incorrect answer icon for incorrect calculation problems; generating a scored worksheet by placing the correct answer icon and the incorrect answer icon at the corresponding positions on the answer sheet; and outputting the scored worksheet. Examples of "correct answer icons" include the "○" mark. Examples of "incorrect answer icons" include the "✓" mark (check mark) and the "×" mark. Here, "outputting" broadly includes printing via a printer, displaying on a screen, storing as an image file on a storage device, and sending an image file to another computer via a network (the same applies hereafter).

[0006] The calculation worksheet scoring system described in claim 2 is the system of claim 1, characterized by performing the following steps: cutting out images containing calculation problems and their solutions that have been determined to be incorrect from an completed worksheet; arranging the cut-out images on a plane and generating an evaluation sheet that adds the type of incorrect answer for each calculation problem; and outputting the evaluation sheet.

[0007] The calculation worksheet scoring system described in claim 3 is the system of claim 1 or 2, characterized in that it performs the following steps: when the density of a specific character constituting the answer in the completed worksheet is lower than or equal to a predetermined value compared to the density of other characters in the completed worksheet, it identifies the character as a character that was not erased; it performs the following steps: it cuts out an image from the completed worksheet that includes the calculation problem containing the character that was not erased and its answer; and it places the cut-out image on the evaluation sheet and adds a comment indicating that the character was not erased.

[0008] The calculation worksheet scoring system described in claim 4 is the system of claim 1 or 2, characterized in that, when a user is required to write an equals sign by hand for each calculation problem in the calculation worksheet, if there is no corresponding equals sign for a particular calculation problem in the completed worksheet, the system performs the following steps: a process to determine that the equals sign has been forgotten; a process to identify the position to place an icon indicating the forgotten equals sign; a process to place the icon at the corresponding position in the completed worksheet; a process to cut out an image from the completed worksheet that includes the calculation problem for which the equals sign has been forgotten and its answer; and a process to place the cut-out image on the evaluation sheet and add a comment indicating the forgotten equals sign.

[0009] The calculation worksheet scoring system described in claim 5 is the system of claim 1 or 2, comprising: a review information storage means that stores the correspondence between the identification information of a calculation worksheet and the identification information of a review calculation worksheet for each calculation worksheet; and a calculation worksheet storage means that stores an image file for each calculation worksheet, and is characterized by performing the following: a process of identifying the identification information of an answered worksheet by reading the characters or codes printed on the answered worksheet; a process of referring to the review information storage means to identify the identification information of a review calculation worksheet associated with the answered worksheet if there are a predetermined number of calculation problems that have been determined to be incorrect; and a process of obtaining and outputting an image file of a review calculation worksheet from the calculation worksheet storage means.

[0010] The calculation worksheet scoring system described in claim 6 is characterized by comprising: a learning data storage means for storing learning data consisting of pairs of learning data: an image file of an answered worksheet with handwritten answers written on it for each calculation problem printed on the worksheet, and an image file of a scored worksheet with a correct answer icon indicating a correct answer or an incorrect answer icon indicating an incorrect answer written on it for each answer on the answered worksheet; a means for generating a trained model for scoring calculation worksheets by taking a plurality of learning data from the learning data storage means and inputting it into a model equipped with character recognition functionality for training; a means for generating a scored worksheet in which a correct answer icon is placed for correct answers and an incorrect answer icon is placed for incorrect answers by inputting an answered worksheet into the trained model; and a means for outputting the scored worksheet.

[0011] The program described in claim 7 is characterized by causing a computer to function as a calculation print scoring system described in any one of claims 1 to 6. [Effects of the Invention]

[0012] According to the calculation worksheet scoring system and program of this invention, it is possible to automatically generate scored worksheets with correct answer icons such as "○" and incorrect answer icons such as "✓" placed near each calculation problem on completed worksheets with handwritten answers by learners, thereby significantly reducing the burden on teachers and parents.

[0013] Furthermore, by cutting out images of incorrect calculation problems and their solutions from completed worksheets and arranging them on a flat surface, and generating and outputting an evaluation sheet that includes the type of incorrect answer (calculation error, nonexistent value, unrecognizable value, etc.), it becomes possible to identify the patterns of errors made by learners. If the evaluation sheet also displays instances of forgotten erased characters or missing equals signs, it becomes possible to clearly identify learners' habits that require attention.

[0014] Furthermore, when there are a predetermined number or more of incorrect answers, by presenting a calculation print for review that helps overcome the incorrect answers, learners can quickly tackle review problems.

[0015] By realizing the generation of scored prints by inputting into a learned model that has been trained with learning data consisting of image files of a large number of answered prints and pairs of image files of answered prints, it is possible to improve the efficiency and accuracy of automatic scoring. As a result, it becomes possible to perform automatic scoring not only for relatively simple one-line calculation problems (horizontal calculations) but also for complex written calculations in which a plurality of numbers are arranged vertically for calculation.

Brief Description of the Drawings

[0016] [Figure 1] It is a block diagram showing the functional configuration of a calculation print scoring system according to this invention. [Figure 2] It is a flowchart showing the procedure of processing executed by a server and a client terminal. [Figure 3] It is a diagram showing an initial screen displayed on a client terminal. [Figure 4] It is a diagram showing a state in which an answered print is displayed on the initial screen. [Figure 5] It is a flowchart showing the details of analysis processing executed by an analysis unit. [Figure 6] It is a diagram showing a scored print screen displayed on a client terminal. [Figure 7] It is a diagram showing an evaluation sheet screen displayed on a client terminal. [Figure 8] It is a diagram showing a review print screen displayed on a client terminal. [Figure 9] It is a block diagram showing the functional configuration related to the learning of AI. [Figure 10] It is a flowchart showing the procedure of processing in the learning process. [Figure 11] It is a diagram showing a state in which an answered print of written calculation is displayed on the initial screen. [Figure 12] It is a diagram showing a scored print screen of written arithmetic displayed on a client terminal. [Figure 13] It is a diagram showing an evaluation sheet screen related to written arithmetic displayed on a client terminal.

Best Mode for Carrying Out the Invention

[0017] As shown in FIG. 1, the calculation print scoring system 10 includes a server 11, and this server 11 includes a user processing unit 12, an analysis unit 14, an analysis result presentation unit 16, a user DB 18, a review information storage unit 20, an analysis result storage unit 22, and a calculation print storage unit 24. A client terminal 28 such as a personal computer, a tablet, or a smartphone operated by a user 26 such as a teacher, a cram school instructor, a guardian, or a learner (child) is connected to the server 11 via the Internet.

[0018] The above user processing unit 12, analysis unit 14, and analysis result presentation unit 16 are realized by the CPU of the server 11 operating according to an OS and a dedicated application program.

[0019] Also, the user DB 18, the review information storage unit 20, the analysis result storage unit 22, and the calculation print storage unit 24 are provided in the storage device of the server 11.

[0020] The user DB 18 stores attribute information such as the account name, password, name, and email address of each user 26.

[0021] The review information storage unit 20 stores the correspondence relationship between the identification information of a large number of calculation prints that elementary school students should learn and the identification information of calculation prints (hereinafter referred to as "review prints") useful for reviewing each calculation print. As the "identification information", in addition to the print ID, for example, a combination of the corresponding school year, unit, title, etc. is applicable.

[0022] The calculation print storage unit 24 stores numerous image files of calculation prints, associated with identification information.

[0023] Each client terminal 28 stores image files of calculation worksheets (hereinafter referred to as "answered worksheets") on which learners have handwritten answers. These image files are prepared by scanning pre-answered printouts into the client terminal 28 via a scanner or smartphone camera, and converting them into a predetermined file format (e.g., a PDF file).

[0024] The following describes the processes performed by the server 11 and client terminal 28, following the flowchart in Figure 2. First, when user 26 accesses server 11 via client terminal 28, the user processing unit 12 sends an initial screen (S10), which is displayed on the client terminal 28's display (S12). Figure 3 shows this initial screen 30, which includes a print placement space 32, a print selection button 34, and a send button 36.

[0025] When user 26 clicks the print selection button 34 and specifies the completed printout stored in a designated folder on the client terminal 28, the completed printout 38 is displayed in the print placement space 32, as shown in Figure 4 (S14). If you have made the wrong selection, click the print selection button 34 again and select the correct completed printout. After confirming that the displayed completed printout 38 is correct, user 26 clicks the send button 36 to send the completed printout 38 to the server 11 (S16).

[0026] Upon receiving this, the server 11 starts up the analysis unit 14 and performs analysis on the completed answer sheet 38 (S18). The details of this analysis process will be explained below, following the flowchart in Figure 5.

[0027] First, the analysis unit 14 identifies the identification information of the submitted answer sheet 38 (S18-01). In other words, since a barcode 40 indicating identification information is pre-printed on the top of each calculation sheet, the analysis unit 14 identifies the identification information of the completed worksheet 38 by reading this barcode 40.

[0028] Next, the analysis unit 14 performs character recognition processing (OCR processing) on ​​the completed answer sheet 38 (S18-02). Specifically, the printed characters (typeset) and handwritten characters are extracted from the image of the completed worksheet 38, and their respective character codes and positions (coordinates within the completed worksheet 38) are identified.

[0029] If handwritten characters are written in extremely cursive script or mistakes are crossed out, making character recognition impossible, the analysis unit 14 assigns a code indicating "character recognition impossible" to the relevant characters. Furthermore, if the density of a particular handwritten character in the completed worksheet 38 is lower than a predetermined ratio or more compared to the average density of the handwritten characters in the same worksheet 38 (for example, less than 30% of the average density), the analysis unit 14 assigns a code indicating that the character was "left unerased (a character that was not completely erased with an eraser)."

[0030] Next, the analysis unit 14 extracts the formulas to be graded from the completed answer sheet (S18-03). Here, "mathematical formula" generally refers to a combination of calculation problems, equality signs, and solutions that are arranged within a predetermined distance from each other and on the same line. For example, in the case of the completed worksheet 38 in Figure 4, "(1) 0.2 + 0.3 = 0.5" in the first row of the left column and "(11) 0.6 - 0.1 = 0.5" in the first row of the right column are extracted as typical mathematical formulas. In this process, the analysis unit 14 also determines the coordinates of the four points within the rectangle surrounding each mathematical formula.

[0031] Furthermore, even if the numbers corresponding to the answer are missing, as in the fourth line of the right column, "(14) 1-0.7=", the analysis unit 14 recognizes it as "the answer has been forgotten" and extracts it as a single mathematical expression. At this point, the analysis unit 14 identifies the coordinates of the four points within the rectangle surrounding the equation "1-0.7=".

[0032] Furthermore, even if the density of the "1" that makes up "10.4" on the right side is below a predetermined ratio compared to the average density of other answers in the same completed worksheet 38, as in "(15) 1.4-1=10.4" on the 5th line of the right column, the analysis unit 14 will identify it as a "character that was not erased (a character that was not completely erased with an eraser)," and "1.4-1=10.4" will be extracted as a mathematical formula as is. At this point, the analysis unit 14 identifies the coordinates of the four points within the rectangle surrounding the equation "1.4-1=10.4".

[0033] Furthermore, the expression "(19) 1.6-0.9 0.7" in the 9th row of the right column lacks an equals sign, but the analysis unit 14 identifies it as a "missing equals sign" and extracts it as a single mathematical expression. At this point, the analysis unit 14 identifies the coordinates of the four points within the rectangle surrounding the formula "1.6-0.9 0.7".

[0034] Furthermore, in the case of "(10) 0.8+0.7=(Geometric Shape)" in the 10th row of the left column, a spiral-shaped "unrecognizable character" is written where the answer should be written, but the analysis unit 14 extracts the entire expression, including the unrecognizable character, as a single mathematical formula. At this point, the analysis unit 14 identifies the coordinates of the four points within the rectangle surrounding the equation "0.8 + 0.7 = (figure)".

[0035] Next, the analysis unit 14 performs calculation processing on each of the extracted mathematical formulas (S18-04) and determines whether the answer is correct or incorrect (S18-05).

[0036] First, in mathematical expressions such as "(1) 0.2 + 0.3 = 0.5" or "(8) 0.7 + 0.5 = 1.5", where there is a number, operator, number to the left of the equals sign and a number to the right of the equals sign, the operation to the left of the equals sign is performed. If the resulting value matches the number to the right of the equals sign, it is judged as "correct," and if they do not match, it is judged as "incorrect."

[0037] Furthermore, if there is a "number, operator, number" to the left of the equals sign, as in "(10) 0.8 + 0.7 = (shape)", and an "unrecognizable character" to the right of the equals sign, the operation on the left side is performed, and if the resulting value does not match the number on the right side (i.e., the value does not exist), it is judged as "incorrect".

[0038] Furthermore, in mathematical expressions like "(14) 1-0.7=", where there is a number, operator, number to the left of the equals sign, but no number to the right of the equals sign, the operation on the left side is performed, and the result is judged as "incorrect" if it does not match the number on the right side (i.e., the value does not exist).

[0039] In contrast, in the case of a mathematical expression such as "(19) 1.6-0.9 0.7", where a space of a predetermined width exists where an equals sign should be, and to the left of the space there is a number, operator, number, and to the right of the space there is a number, the operation to the left of the space is performed, and if the resulting value matches the number to the right of the space, it is judged as "correct", and if they do not match, it is judged as "incorrect".

[0040] If the analysis unit 14 determines in S18-05 that the answer is "correct", it identifies the position where the "○" mark indicating the correct answer should be placed (S18-06), and if it determines that the answer is "incorrect", it identifies the position where the "✓" mark indicating the incorrect answer should be placed (S18-07). For example, the coordinates near the right end of each mathematical expression are specified as the positions for placing the "○" or "✓" marks.

[0041] Next, the analysis unit 14 identifies the position where a checkmark (✓) should be placed to indicate the missing equality sign in the mathematical expression (S18-08, S18-09). The position where the checkmark should be placed is, for example, the position where the equals sign should be (the space between a number written in type and a number written by hand).

[0042] Next, the analysis unit 14 determines that a certain number of incorrect answers (for example, 3 or more questions) require review (S18-10 / Y), and refers to the review information storage unit 20 to identify the corresponding review worksheet (S18-11). In contrast, if the number of incorrect answers is less than the specified number, review is deemed unnecessary (S18-10 / N), and the process of identifying worksheets requiring review is skipped.

[0043] The results of the processing by the analysis unit 14 are stored in the analysis result storage unit 22 (S18-12). Examples of analysis results stored here include the following data. (1) Image file of completed worksheet (2) Coordinates of the four points in the rectangle surrounding each extracted formula (3) Incorrect formula (4) Types of errors (calculation errors / forgetting to write the answer / forgetting to erase writing / forgetting to write an equals sign / characters unrecognizable, etc.) (5) Coordinates where the "○" mark should be placed (6) Coordinates where the "✓" mark should be placed (7) Identification information for review worksheets (if review is required)

[0044] Once the analysis processing by the analysis unit 14 is completed, the analysis result presentation unit 16, which refers to the analysis result storage unit 22, generates a scored printout, which is then sent to the client terminal 28 via the user processing unit 12 (S20 in Figure 2). As a result, the graded printout screen is displayed on the client terminal 28's display (S22).

[0045] Figure 6 shows the graded printout screen 50, where the graded printout 52 is displayed in the printout placement space 32. As shown in the diagram, on the graded printout 52, a "○" mark 54 indicating a correct answer and a "✓" mark 56 indicating an incorrect answer or a omission of the equals sign are placed in the corresponding positions on the completed printout 38. Furthermore, the parts identified as mathematical formulas by the analysis unit 14 are enclosed in rectangles 58.

[0046] When user 26 clicks the print button 60, the graded printout 52 is printed out via the printer connected to client terminal 28. Furthermore, when user 26 clicks the save button 62, the PDF file of the graded printout 52 is stored in the storage device of client terminal 28. Furthermore, when user 26 clicks the evaluation button 64 and requests the presentation of the evaluation sheet (S24), the analysis result presentation unit 16 generates the evaluation sheet screen and sends it to the client terminal 28 via the user processing unit 12 (S26). As a result, the evaluation sheet screen is displayed on the client terminal 28's display (S28).

[0047] Figure 7 shows the evaluation sheet screen 70, which includes sections for "Incorrectly Answered Questions," "Things to Watch Out For," and "Reflection." The section titled "Incorrect Answers" contains the following information: (1) Images of the incorrect formulas (marked with a checkmark 56 and a rectangle 58) and comments indicating the type of error, such as "calculation error". (2) A cropped image of the mathematical formula whose answer was determined to be unrecognizable (with a checkmark 56 and a rectangle 58), and a comment indicating the type of error, "Unreadable Characters". (3) Images of the formulas that were determined to have missing answers (marked with a checkmark 56 and rectangle 58), and the comment "missing answer" indicating the type of error. (4) A cropped image of a mathematical formula that was determined to have forgotten to erase characters (with a checkmark 56 and a rectangle 58), and a comment indicating the type of error, "forgotten to erase characters". (5) A cropped image of a mathematical formula that was determined to have a missing equals sign (marked with a circle 54, a checkmark 56, and a rectangle 58), and a comment indicating the type of error, "Missing equals sign".

[0048] The "Things to be careful of" section includes the following phrases, corresponding to different types of errors other than calculation errors, to serve as a warning. Write the letters correctly. Be careful not to forget to write down your answer. Be careful not to forget to erase the writing. Let's write the equals sign (=) correctly. Additionally, if there are more than a specified number of calculation errors, a cautionary phrase such as "There are many calculation errors." can be added to the "Things to be careful about" section.

[0049] The "Let's Review" section includes the title of a review worksheet (Addition with Carrying Over) and the unit (1st Grade, Lower Semester, 40) that will help overcome the calculation errors that occurred in this calculation worksheet.

[0050] When user 26 clicks the print button 60, the contents of the evaluation sheet are printed out via the printer connected to client terminal 28. Furthermore, when user 26 clicks the save button 62, the evaluation sheet PDF file is stored in the storage device of client terminal 28. Furthermore, when user 26 clicks the review print button 72 and requests a review print (S30), the analysis result presentation unit 16 retrieves the corresponding review print from the calculation print storage unit 24, and the review print screen is sent to the client terminal 28 via the user processing unit 12 (S32). As a result, the review printout screen is displayed on the client terminal 28's display (S34).

[0051] Figure 8 shows the review worksheet screen 80, with the review worksheet 82 displayed in the worksheet placement space 32. When user 26 clicks the print button 60, the review worksheet 82 is printed out via the printer connected to client terminal 28 (S36). Then, once the answers to the review worksheet 82 have been completed, the completed worksheet 38 is converted into an image file and sent to the server 11 via the initial screen 30, thereby triggering the automated scoring process through this system 10 again.

[0052] In the above example, a user 26, such as a parent or student, uploads a completed answer sheet 38 to the server 11 and requests automatic grading. However, this invention is not limited to such usage patterns. In other words, users such as teachers and cram school instructors can upload completed worksheets 38 for multiple students they are in charge of to the server 11, and then request grading, evaluation, and review instruction for all of these completed worksheets 38 at once. In this case, server 11 sends a PDF file containing multiple graded printouts, evaluation sheets, and review printouts to client terminal 28.

[0053] Furthermore, while the above example shows the analysis unit 14 identifying the identification information of the calculation worksheet based on the barcode 40 on the completed worksheet 38, it is naturally also possible to identify the identification information by recognizing the characters written on the math worksheet. For example, the analysis unit 14 recognizes a string of characters corresponding to the title of the completed worksheet 38 (for example, "The Mechanism of Decimals and Addition and Subtraction"), and uses this text as a key to search the review information storage unit 20, thereby identifying the identification information of the completed worksheet 38.

[0054] By the way, the mathematical formulas on the completed worksheet 38 consist of a mix of printed calculation problems and handwritten equality signs and answers, so it is possible that non-standard errors such as forgetting to write an equality sign or forgetting to erase numbers may occur. Furthermore, in addition to calculation problems where numbers, operators, and equals signs are arranged in a horizontal line (horizontal calculation), as described above, automatically scoring problems where multiple numbers are arranged vertically (long division) would require even more complex processing. Therefore, it is effective to improve the efficiency and accuracy of processing by having a trained AI (artificial intelligence) model, which has improved the accuracy of character recognition and mathematical formula extraction processing through a learning process using a large number of completed worksheets 38, handle part of the processing performed by the analysis processing unit 14.

[0055] As one embodiment of this, Figure 9 is a block diagram mainly describing the functional configuration related to the generation of a trained AI model, in which the learning unit 102, the training data storage unit 104, and the model 106 are provided within the server 11. After the training of the model 106 is completed, the analysis unit 14 performs the scoring process of the completed answer sheet 38 using this trained model 106 (details will be described later).

[0056] The learning unit 102 is realized by the CPU of the server 11 operating according to the OS and a dedicated application program. Furthermore, the learning data storage unit 104 is located within the storage device of the server 11. Model 106 consists of an LLM (Large-Scale Language Model) equipped with OCR functionality and specialized for symbolic computation.

[0057] The learning data storage unit 104 stores a large amount of learning data (training data), consisting of pairs of completed worksheets 38 with answers written by elementary school students, such as horizontal and vertical arithmetic problems, and graded worksheets 52 corresponding to each completed worksheet 38. The graded printout 38 is used as correct answer data (correct answer labels), and each mathematical formula is marked with a rectangle 58, a circle mark 54, or a checkmark 56. In addition, the relevant formulas are assigned codes indicating "unrecognizable characters," "answer forgotten," "letters not erased," "equals sign forgotten," etc.

[0058] The learning process in this system 10 will be explained below, following the flowchart in Figure 10. First, the learning unit 102 reads a predetermined batch number of learning data stored in the learning data storage unit 104 (S50), and sequentially inputs the completed worksheets 38 for each learning data into the model to be learned (base model) 106 (S52).

[0059] Next, the learning unit 102 obtains the processing result from the model 106 (S54), calculates the error between this processing result and the correct data (S56), and adjusts the parameters of the model 106 according to the backpropagation method to minimize the error (S58). Then, by repeating the processes from S50 to S58 until a predetermined termination condition (for example, the completion of the required number of epochs) is met (S60 / Y), the learning unit 14 completes the trained model 106 (S62).

[0060] After this, when the analysis unit 14 receives the completed answer sheet 38 to be graded from the user processing unit 12, it inputs it into the trained model 106 and processes it, including character recognition, extraction of mathematical formulas to be graded, determination of whether the answers to calculation problems are correct or incorrect, and determination of the presence or absence of equality signs.

[0061] The following describes a specific example of automatic scoring using the trained model 106 on calculation worksheets for written calculations, but the process is basically the same for horizontal calculations. Figure 11 shows the state in which a completed calculation worksheet 38 is displayed on the client terminal 28's screen. The completed calculation worksheet 38 contains multiple printed calculation problems and handwritten calculation processes and answers. Each math problem consists of multiple numbers arranged vertically, an operator (×), and a dividing line. If the multiplier is a single digit, the answer is written directly below the dividing line. In contrast, if the multiplier has multiple digits, the intermediate steps are written in multiple lines, a dividing line is drawn, and the answer is written below it. Additionally, if a carry-over occurs, a relatively small auxiliary digit corresponding to the carry-over is added.

[0062] When user 26 clicks the submit button 36, the completed worksheet 38 is sent to the server 11, and the analysis process is performed by the analysis unit 14 and the trained model 106. The following outlines the steps involved in this analysis process. First, the analysis unit 14 reads the barcode 40 of the submitted answer sheet 38 to identify the identification information of the answer sheet 38.

[0063] Next, the trained model 106 performs the following process to generate the graded printout 52. (1) Perform character recognition processing (OCR processing). (2) For handwritten characters that cannot be recognized, a code indicating "unrecognizable" will be assigned. (3) If the density of a particular handwritten character is lower than the average density of other handwritten characters by a predetermined ratio or more, a code indicating that it is a "character that was not erased" will be assigned. (4) Identify the formula to be graded and enclose it in rectangle 58. (5) Perform the calculations in the mathematical formula and determine whether the answer is correct or incorrect by comparing it with the answer. (6) Mark each calculation problem with a "○" mark 54 for a correct answer or a "✓" mark 56 for an incorrect answer. (7) Assign a code to calculation problems that do not have an answer to indicate that the answer was not written down. *When grading horizontal calculations, if there is no equals sign (=) in the formula, a code indicating "missing equals sign" will be added.

[0064] Upon receiving the output from the trained model 106, the analysis unit 14 stores the following data in the analysis result storage unit 22. (1) Image files of graded printouts (2) Identifying information of the incorrect formula (3) Types of errors (calculation errors / forgetting to write the answer / forgetting to erase writing / forgetting to write an equals sign / characters unrecognizable, etc.) (4) Identification information for review worksheets (if there is a predetermined number of incorrect answers and the analysis unit 14 determines that it "requires review")

[0065] The scored printout 38 stored in the analysis result storage unit 22 is transmitted to the client terminal 28 via the analysis result presentation unit 16 and the user processing unit 12. Figure 12 shows the graded print screen 50 displayed on the client terminal 28's display, with the graded print 52 displayed in the print placement space 32. As shown in the diagram, the graded worksheet 52 has a "○" mark 54 indicating a correct answer and a "✓" mark 56 indicating an incorrect answer placed above the sequential number of each equation. Furthermore, the parts identified as mathematical formulas by Model 106 are enclosed in rectangle 58.

[0066] When user 26 clicks the print button 60, the graded printout 52 is printed out via the printer connected to client terminal 28. Furthermore, when user 26 clicks the save button 62, the PDF file of the graded printout 52 is stored in the storage device of client terminal 28. Furthermore, when user 26 clicks the evaluation button 64 and requests the presentation of the evaluation sheet, the analysis result presentation unit 16 generates the evaluation sheet screen and sends it to the client terminal 28 via the user processing unit 12.

[0067] Figure 13 shows the evaluation sheet screen 70 displayed on the client terminal 28's display, and includes sections for "Questions you got wrong," "Things to be careful about," and "Let's reflect."

[0068] The section titled "Incorrect Answers" contains the following information: (1) Images of the incorrect formulas (marked with a checkmark 56 and a rectangle 58) and comments indicating the type of error, such as "calculation error". (2) A cropped image of a mathematical formula that was determined to have forgotten to erase characters (with a checkmark 56 and a rectangle 58), and a comment indicating the type of error, "forgotten to erase characters". (3) Images of the formulas that were determined to have missing answers (marked with a checkmark 56 and rectangle 58), and the comment "missing answer" indicating the type of error.

[0069] The "Things to be careful of" section includes the following phrases, corresponding to different types of errors other than calculation errors, to serve as a warning. Be careful not to forget to write down your answer. Be careful not to forget to erase the writing.

[0070] The "Let's Review" section contains review worksheets that will help you overcome the calculation errors that occurred in this calculation worksheet.

[0071] When user 26 clicks the print button 60, the contents of the evaluation sheet are printed out via the printer connected to client terminal 28. Furthermore, when user 26 clicks the save button 62, the evaluation sheet PDF file is stored in the storage device of client terminal 28. Furthermore, when user 26 clicks the review print button 72 and requests a review print, the analysis result presentation unit 16 retrieves the corresponding review print from the calculation print storage unit 24, and the review print screen is sent to the client terminal 28 via the user processing unit 12. As a result, the review printout screen is displayed on the client terminal 28's display (illustration not shown).

[0072] The above exemplifies the automatic scoring process for written calculations using the pre-trained model 106. It goes without saying that this automatic scoring process can also be applied to calculation worksheets for horizontal calculations, or to worksheets containing both horizontal and written calculations. Furthermore, the pre-trained Model 106 can handle not only multiplication but also addition, subtraction, and division calculations using the written method, automatically scoring calculation worksheets. [Explanation of symbols]

[0073] 10. Calculation Worksheet Grading System 11 Servers 12 User Processing Unit 14 Analysis Department 16 Analysis result presentation section 18 User Database 20 Review Information Memory Unit 22 Analysis result storage section 24 Calculation Print Storage Unit 26 Users 28 Client terminals 30 Initial screen 32 print placement spaces 34 Print selection button 36. Send button 38 Answered Printouts 40 barcodes 50. Graded printout screen 52 Graded Printouts 54 "○" mark 56 "Re" mark 58 rectangle 60 Print button 62 Save button 64 rating buttons 70 Evaluation Sheet Screen 72 Review Printout Button 80 Review Printout Screen 82 Review Printout 102 Learning Department 104 Learning data storage unit 106 AI Models

Claims

1. The process involves accepting image files of completed worksheets with handwritten answers for each calculation problem printed on the worksheet as the basis for grading. The process involves performing character recognition on the image file of the completed worksheet to identify the characters that make up each calculation problem and answer, and their positions. The process involves performing calculations on each problem, determining that the answer is correct if the calculation result matches the given value, and determining that the answer is incorrect if they do not match, if the given value does not exist, or if the characters constituting the answer cannot be recognized. Regarding the correct answer to the calculation problem, the process involves determining the position where the correct answer icon should be placed, Regarding incorrect calculation problems, the process involves determining the position where the incorrect answer icon should be placed, The process involves placing correct and incorrect answer icons in corresponding positions on the completed worksheet to generate a graded worksheet, The process of outputting graded printouts, A calculation-based print grading system characterized by performing the following actions.

2. The process involves extracting images containing incorrect calculation problems and their solutions from the completed worksheets. The process involves arranging the cropped images on a plane and generating an evaluation sheet with comments indicating the type of incorrect answer for each calculation problem. The process of outputting the evaluation sheet, The calculation worksheet scoring system according to claim 1, characterized by performing the following:

3. A process to identify a character as an unerased character when the density of a specific character constituting an answer in a completed worksheet is lower than a predetermined amount compared to the density of other characters in the same worksheet, The process of cutting out images containing calculation problems and their solutions, including any characters that were left unerased, from completed worksheets, The process involves placing the cropped image onto the evaluation sheet mentioned above, and adding comments indicating any text that has been forgotten to be erased. The calculation worksheet scoring system according to claim 2, characterized by performing the following.

4. In the calculation problems on the above calculation worksheet, if the equals sign is to be written by hand, The process involves determining if a corresponding equals sign is missing in a specific calculation problem within a completed worksheet, and whether the equals sign was simply omitted. The process of determining the position to place the icon indicating that the equals sign was not written, The process involves placing the icon at the corresponding location on the graded printout mentioned above, The process involves cropping images containing calculation problems with missing equals signs and their solutions from completed answer sheets. The process involves placing the cropped image onto the evaluation sheet mentioned above, and adding a comment indicating that an equals sign was forgotten. The calculation worksheet scoring system according to claim 2, characterized by performing the following.

5. A review information storage means that stores the correspondence between the identification information of a calculation worksheet and the identification information of a review worksheet for each calculation worksheet, The system includes a calculation print storage means that stores image files for each calculation printout. The process involves reading the characters or codes printed on the completed answer sheet to identify the identification information of the said answer sheet, If there are a predetermined number of calculation problems that have been determined to be incorrect, the process involves referring to the review information storage means and identifying the identification information of the review calculation worksheet associated with the completed worksheet, The above calculation print storage means retrieves image files of calculation printouts for review and outputs them. The calculation worksheet scoring system according to claim 1, characterized by performing the following:

6. A learning data storage means for storing learning data consisting of pairs of learning data: an image file of an answer sheet with handwritten answers for each calculation problem printed on it, and an image file of a graded sheet with a correct answer icon or an incorrect answer icon for each answer on the answer sheet; A means for generating a trained model for grading calculation worksheets by extracting multiple training data from a training data storage means and inputting them into a model equipped with character recognition functionality for training, By inputting completed worksheets into this trained model, a means is provided to generate graded worksheets in which correct answers are marked with a correct icon and incorrect answers with an incorrect icon. A means of outputting this graded printout, A calculation worksheet scoring system characterized by having the following features.

7. A program characterized by causing a computer to function as a calculation print scoring system according to any one of claims 1 to 6.

Images