Graphic correctness determination device, graphic correctness determination method, program, and recording medium

Abstract

To provide a graphic correctness / incorrectness determination device capable of improving the accuracy of figure recognition.SOLUTION: In a graphic correctness / incorrectness determination device 10 of the present invention, a graphic image acquisition unit 11 acquires a graphic image input for a question, a graphic image recognition unit 12 recognizes the obtained graphic image, and calculates the certainty of one or more recognized graphics and the recognition graphics to generate a certainty list, a correct answer information acquisition unit 13 acquires correct answer information of the question, a correctness / incorrectness determination unit 14 compares the recognition graphic with the correct answer information to generate match / mismatch information, and determines whether the graphic image is correct or incorrect based on the certainty list and the match / mismatch information to generate correctness / incorrectness information, and a correctness / incorrectness information output unit 15 outputs the correctness / incorrectness information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a graphic correctness determination device, a graphic correctness determination method, a program, and a recording medium. [Background technology]

[0002] Patent Document 1 discloses a technique for recognizing handwritten characters using machine learning. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2014-071813 Summary of the Invention [Problem to be solved by the invention]

[0004] However, there are limits to how much advances in machine learning can improve the accuracy of character recognition. There is a strong demand for improved accuracy, particularly in character recognition for quizzes, tests, questionnaires, etc. This problem applies not only to characters but also to general graphic recognition, including numbers and symbols.

[0005] SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide a graphic correctness determination device, a graphic correctness determination method, a program, and a recording medium that can improve the accuracy of graphic recognition. [Means for solving the problem]

[0006] In order to achieve the above object, the graphic correctness determining device of the present invention comprises: The system includes a graphic image acquisition unit, a graphic image recognition unit, a correct answer information acquisition unit, a correct / incorrect determination unit, and a correct / incorrect information output unit, the graphic image acquisition unit acquires a graphic image input in response to a question, the graphic image recognition unit recognizes the acquired graphic image, calculates the recognized one or more recognized graphics and the certainty of the recognized graphics, and generates a certainty list; the correct answer information acquisition unit acquires correct answer information for the question, the correctness determination unit compares the recognized graphic with the correct answer information to generate match / mismatch information, and determines whether the graphic image is correct or not based on the confidence level list and the match / mismatch information to generate correctness information; the correct / incorrect information output unit outputs the correct / incorrect information. It is a device.

[0007] The method for determining whether a figure is correct or incorrect according to the present invention comprises the steps of: The method includes a graphic image acquisition step, a graphic image recognition step, a correct answer information acquisition step, a correct / incorrect determination step, and a correct / incorrect information output step, The graphic image acquisition step acquires a graphic image input in response to a question, The graphic image recognition step recognizes the acquired graphic image, calculates the recognized one or more recognized graphics and the certainty of the recognized graphics, and generates a certainty list; The correct answer information acquisition step acquires correct answer information for the question, the correctness determining step includes: comparing the recognized graphic with the correct answer information to generate match / mismatch information; and determining whether the graphic image is correct or not based on the confidence level list and the match / mismatch information to generate correctness information; The correct / incorrect information output step outputs the correct / incorrect information. It is a method. [Effects of the Invention]

[0008] According to the present invention, it is possible to improve the accuracy of pattern recognition for patterns written in an input format such as handwriting, in which the character style and character shape vary each time the pattern is written. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of a graphic correctness determining device according to the first embodiment. [Figure 2] FIG. 2 is a block diagram showing an example of the hardware configuration of the graphic correctness determining device of the first embodiment. [Figure 3]FIG. 3 is a flowchart showing an example of processing in the method for determining whether a graphic is correct or not according to the first embodiment. [Figure 4] Figure 4(A) is a schematic diagram showing an example of a question and an answer (graphic image) handwritten to that question, and Figure 4(B) is a schematic diagram showing an example of a confidence level list generated from the graphic image shown in Figure 4(A). [Figure 5] Figure 5(A) is a schematic diagram showing an example of a question and an answer (graphic image) handwritten to that question, and Figure 5(B) is a schematic diagram showing an example of a confidence level list generated from the graphic image shown in Figure 5(A). [Figure 6] Figure 6(A) is a schematic diagram showing an example of a question and an answer (graphic image) handwritten to that question, and Figure 6(B) is a schematic diagram showing an example of a confidence level list generated from the graphic image shown in Figure 6(A). [Figure 7] Figure 7(A) is a schematic diagram showing an example of a question and an answer (graphic image) handwritten to that question, and Figure 7(B) is a schematic diagram showing an example of a confidence level list generated from the graphic image shown in Figure 7(A). [Figure 8] Figure 8(A) is a schematic diagram showing an example of a question and an answer (graphic image) handwritten to that question, and Figure 8(B) is a schematic diagram showing an example of a confidence level list generated from the graphic image shown in Figure 8(A). DETAILED DESCRIPTION OF THE INVENTION

[0010] In the graphic correctness determination device of the present invention, for example, The correctness determining unit may determine whether the graphic image is correct or not based on the following criteria (1), (2), and (3), and generate correctness information. (1) Correct answer criteria If the graphic image satisfies either of the following conditions (1a) and (1b), it is determined to be correct. (1a) The recognized figure with the highest confidence in the confidence list matches the correct answer information. (1b) The certainty of the recognized image that matches the correct answer information is equal to or greater than a correct answer threshold. (2) Criteria for determining incorrect answers If the following condition (2a) is met, the graphic image is determined to be an incorrect answer. (2a) In the confidence level list, the confidence level of the recognition pattern that matches the correct answer information is less than the correct answer threshold, and the confidence level of the recognition pattern that does not match the correct answer information is equal to or greater than the incorrect answer threshold. (3) Conditions that cannot be judged If the following condition (3a) is satisfied, the graphic image is determined to be unidentifiable. (3a) In the confidence level list, the confidence level of the recognized image that matches the correct answer information is less than the correct answer threshold, and the confidence level of the recognized figure that does not match the correct answer information is less than the incorrect answer threshold.

[0011] The graphic correctness determination device of the present invention includes, for example, It further includes a learning section, The learning unit may calculate a rate of correctness of the outputted correct / incorrect information, and adjust the correct answer threshold and the incorrect answer threshold in the correct / incorrect determining unit based on the rate of correctness.

[0012] In the graphic correctness determination device of the present invention, for example, the graphic image recognition unit includes a trained model based on machine learning, The graphic image may be recognized using the trained model, and one or more recognized recognized figures and a degree of certainty of the recognized figures may be calculated.

[0013] In the method for determining whether a figure is correct or not according to the present invention, for example, The correctness determining step may be configured to determine whether the graphic image is correct or not based on the following criteria (1), (2), and (3) to generate correctness information. (1) Correct answer criteria If the graphic image satisfies either of the following conditions (1a) and (1b), it is determined to be correct. (1a) The recognized figure with the highest confidence in the confidence list matches the correct answer information. (1b) The certainty of the recognized image that matches the correct answer information is equal to or greater than a correct answer threshold. (2) Criteria for determining incorrect answers If the following condition (2a) is met, the graphic image is determined to be an incorrect answer. (2a) In the confidence level list, the confidence level of the recognition pattern that matches the correct answer information is less than the correct answer threshold, and the confidence level of the recognition pattern that does not match the correct answer information is equal to or greater than the incorrect answer threshold. (3) Conditions that cannot be judged If the following condition (3a) is satisfied, the graphic image is determined to be unidentifiable. (3a) In the confidence level list, the confidence level of the recognized image that matches the correct answer information is less than the correct answer threshold, and the confidence level of the recognized figure that does not match the correct answer information is less than the incorrect answer threshold.

[0014] The method for determining whether a figure is correct or incorrect according to the present invention includes, for example, Furthermore, it includes a learning process, The learning step may be performed by calculating a percentage of correct answers of the outputted correct / incorrect information, and adjusting the correct answer threshold and the incorrect answer threshold in the correct / incorrect determining step based on the percentage of correct answers.

[0015] In the method for determining whether a figure is correct or not according to the present invention, for example, the graphic image recognition step includes a trained model obtained by machine learning, The graphic image may be recognized using the trained model, and one or more recognized recognized figures and a degree of certainty of the recognized figures may be calculated.

[0016] The program of the present invention is a program for causing a computer to execute the steps of the method of the present invention as a procedure.

[0017] The recording medium of the present invention is a computer-readable recording medium on which the program of the present invention is recorded.

[0018] The graphics in the "graphic image" of the present invention include letters, numbers, symbols, terms, words, and the like.

[0019] The present invention is applicable to, for example, subjects for which the answers required of respondents (including answers or responses using multiple-choice options) are known in advance, specifically, questionnaires, tests, quizzes, study drills, etc. Furthermore, the present invention can be used as part of devices such as questionnaire devices, test devices, quiz devices, and the like, as well as corresponding methods and programs.

[0020] Next, embodiments of the present invention will be described with reference to the drawings. The present invention is not limited to the following embodiments. In the following drawings, the same parts are denoted by the same reference numerals. Furthermore, the descriptions of the embodiments can be mutually incorporated unless otherwise specified, and the configurations of the embodiments can be combined unless otherwise specified.

[0021] [Embodiment 1] Fig. 1 is a block diagram showing an example of the configuration of a graphic correctness determination device 10 according to this embodiment. As shown in Fig. 1, this device 10 includes a graphic image acquisition unit 11, a graphic image recognition unit 12, a correct answer information acquisition unit 13, a correctness determination unit 14, and a correctness information output unit 15. Furthermore, this device 10 may further include a learning unit 16, etc., as an optional configuration. The above-mentioned units are connected to each other, for example, by an internal bus.

[0022] The device 10 may be, for example, a single device including the above-described units, or a device in which the units can be connected via a communication network. The device 10 can also be connected to an external device (described later) via the communication network. The communication network is not particularly limited and any known network can be used, for example, a wired or wireless network. Examples of the communication network include the Internet, the World Wide Web (WWW), a telephone line, a Local Area Network (LAN), a Storage Area Network (SAN), a Delay Tolerant Networking (DTN), a Low Power Wide Area Network (LPWA), and a Local 5G (L5G). Examples of wireless communication include Wi-Fi (registered trademark), Bluetooth (registered trademark), Local 5G, and LPWA. Examples of the wireless communication include direct communication between devices (Ad Hoc communication), infrastructure communication, and indirect communication via an access point. The device 10 may be incorporated into a server as a system. Furthermore, the device 10 may be, for example, a personal computer (PC, for example, desktop or notebook type) on which the program of the present invention is installed, a smartphone, a tablet terminal, a wearable terminal, or the like. Furthermore, all or part of the components of the device 10 may be realized on the cloud. Specifically, the device 10 may be in the form of cloud computing or edge computing, for example, in which at least one of the components is on a server (cloud) and the other components are on a terminal.

[0023] FIG. 2 illustrates a block diagram of the hardware configuration of the device 10. The device 10 may include, for example, a central processing unit (CPU, GPU, etc.) 101, a memory 102, a bus 103, a storage device 104, an input device 105, an output device 106, and a communication device 107. Note that these are merely examples, and the hardware configuration of the device 10 is not limited to these as long as it is capable of executing the processing of each of the above-mentioned units. Furthermore, the number of central processing units 101, etc. included in the device 10 is not limited to the example shown in FIG. 2; for example, the device 10 may include multiple central processing units 101. The units in the hardware configuration of the device 10 are connected to each other via their respective interfaces (I / F) and a bus 103.

[0024] The central processing unit 101 is responsible for overall control of the device 10. In the device 10, the central processing unit 101 executes, for example, the program of the present invention and other programs, and also reads and writes various types of information. The central processing unit 101 can then execute the processing of each part of the device 10.

[0025] The bus 103 can also be connected to, for example, an external device. Examples of the external device include an external storage device (such as an external database), an external input device, and an external output device. The device 10 can be connected to an external network (the communication line network) by, for example, a communication device 107 connected to the bus 103, and can also be connected to other devices via the external network.

[0026] The memory 102 may be, for example, a main memory (primary storage device). When the central processing unit 101 performs processing, the memory 102 reads various operating programs, such as the program of the present invention, stored in the storage device 104 (described later), and the central processing unit 101 receives data from the memory 102 and executes the programs. The main memory may be, for example, a RAM (random access memory). Alternatively, the memory 102 may be, for example, a ROM (read only memory).

[0027] The storage device 104 is also referred to as an auxiliary storage device, for example, in contrast to the main memory (primary storage device). As described above, the storage device 104 stores an operating program including the program of the present invention. The storage device 104 may be, for example, a combination of a recording medium and a drive that reads and writes data from and to the recording medium. The recording medium is not particularly limited and may be, for example, an internal or external type, such as a hard disk (HD), CD-ROM, CD-R, CD-RW, MO, DVD, flash memory, or memory card. The storage device 104 may be, for example, a hard disk drive (HDD) or a solid state drive (SSD) in which the recording medium and drive are integrated.

[0028] In the present device 10, the memory 102 and the storage device 104 can also store various information such as log information, information acquired from an external database (not shown) or an external device, information generated by each process of the present device 10, and information used when the present device 10 executes each process. Note that at least a portion of the information may be stored, for example, in an external server other than the memory 102 and the storage device 104, or may be stored in a distributed manner across multiple terminals using blockchain technology or the like.

[0029] The device 10 may further include, for example, an input device 105 and an output device 106. The input device 105 is a device for inputting, for example, letters, numbers, the position of an object displayed on the screen, an image, sound, etc., and specific examples thereof include a digitizer (such as a touch panel), a keyboard, a mouse, a scanner, an imaging device, a microphone, a sensor, etc. The output device 106 is, for example, a display device (such as an LED display or a liquid crystal display), a printer, a speaker, etc.

[0030] Next, an example of the method for determining whether a graphic is correct or not according to this embodiment will be described with reference to the flowchart in Fig. 3. The method for determining whether a graphic is correct or not according to this embodiment is implemented as follows, for example, using the device 10 for determining whether a graphic is correct or not shown in Fig. 1. Note that the method for determining whether a graphic is correct or not according to this embodiment is not limited to use of the device 10 for determining whether a graphic is correct or not shown in Fig. 1.

[0031] In the following, the graphic image acquisition process can be performed, for example, by a graphic image acquisition unit 11, the graphic image recognition process can be performed, for example, by a graphic image recognition unit 12, the correct answer information acquisition process can be performed, for example, by a correct answer information acquisition unit 13, the correct / incorrect judgment process can be performed, for example, by a correct / incorrect judgment unit 14, the correct / incorrect information output process can be performed, for example, by a correct / incorrect information output unit 15, and the learning process can be performed, for example, by a learning unit 16.

[0032] First, the graphic image acquisition unit 11 acquires a graphic image input in response to a question (S11). The input is an "answer" or "answer" to the question. The input (entry) format for the question is, for example, an input format in which the font or character shape changes each time the question is entered, such as handwritten input or eye-gaze input. The question may be, for example, printed on a print medium such as recording paper, or may be displayed on a terminal having a touch display or the like. If the question is printed on a print medium, the graphic image acquisition unit 11 may, for example, acquire an electronic version of a graphic entered on the print medium as the graphic image. The method of electronicization is not particularly limited, and may be scanning or photography with a camera. On the other hand, if the question is displayed on a terminal, the graphic image acquisition unit 11 may, for example, acquire a graphic entered by handwriting or the like as the graphic image. The terminal may be, for example, the device 10.

[0033] Next, the graphic image recognition unit 12 recognizes the acquired graphic image and calculates the recognized one or more recognized figures and their confidence levels to generate a confidence level list (S12). An example of the confidence level list will be described later. The graphic image recognition unit 12 may perform the recognition using, for example, machine learning including deep learning. Specifically, the graphic image recognition unit 12 may include, for example, a trained model based on machine learning. The graphic image recognition unit 12 may then recognize the graphic image using, for example, the trained model and calculate the recognized one or more recognized figures and their confidence levels. The trained model is not particularly limited. For example, the trained model may be a model trained by associating handwriting with attributes (e.g., age, gender, race, nationality, etc.) of respondents (those who input figures in response to the questions), or a model trained by associating the respondents with their respective handwriting levels. In this way, by taking into account the attributes of the respondents or information specific to specific individuals during the recognition process, the accuracy of the recognition by the graphic image recognition unit 12 can be further improved.

[0034] Next, the correct answer information acquisition unit 13 acquires correct answer information for the question (S13). The correct answer information is information indicating the correct answer to the question. If the question has multiple options (e.g., a questionnaire, etc.) and the options are "1," "2," "3," and "4," for example, at least one of the options "1," "2," "3," and "4" is the correct answer. On the other hand, if the question does not have multiple options, for example, all or part of the answer that can be derived from the question may be the correct answer. More specifically, if the question asks about the sum of 2 and 2, the correct answer is "4." In this way, there may be multiple correct answers indicated by the correct answer information for the question. The correct answer information acquisition unit 13 may acquire the correct answer information pre-stored in the memory 102 and the storage device 104, for example, may acquire the correct answer information input from an external device, or may acquire the correct answer information from an external device.

[0035] Next, the correctness determination unit 14 compares the recognized graphic with the correct answer information to generate match / mismatch information, and determines whether the graphic image is correct or not based on the confidence list and the match / mismatch information to generate correctness information (S14). For example, in the case where the question has multiple-choice questions as described above, if the graphic image recognition unit 12 recognizes the graphic image as "9," the correctness determination unit 14 generates match / mismatch information indicating "mismatch" because there is no "9" in the options.

[0036] The generation of the correctness information by the correctness determination unit 14 will be described in more detail below. The correctness determination unit 14 may generate the correctness information by determining whether the graphic image is correct or not, for example, according to the following criteria (1), (2), and (3).

[0037] Criteria (1): Correct answer criteria If the graphic image satisfies either of the following conditions (1a) and (1b), it is determined to be correct. (1a) The recognized figure with the highest confidence in the confidence list matches the correct answer information. (1b) The certainty of the recognized image that matches the correct answer information is equal to or greater than a correct answer threshold.

[0038] Criteria (2): Criteria for determining incorrect answers If the following condition (2a) is met, the graphic image is determined to be an incorrect answer. (2a) In the confidence level list, the confidence level of the recognition pattern that matches the correct answer information is less than the correct answer threshold, and the confidence level of the recognition pattern that does not match the correct answer information is equal to or greater than the incorrect answer threshold.

[0039] Judgment criterion (3): Conditions that cannot be judged If the following condition (3a) is satisfied, the graphic image is determined to be unidentifiable. (3a) In the confidence level list, the confidence level of the recognized image that matches the correct answer information is less than the correct answer threshold, and the confidence level of the recognized figure that does not match the correct answer information is less than the incorrect answer threshold.

[0040] The correct answer threshold and the incorrect answer threshold can be set to any confidence level. The correct answer threshold may be set to, for example, a confidence level smaller than the incorrect answer threshold. Specifically, when the confidence level is expressed as a numerical value from 1 to 100, the correct answer threshold may be a numerical value less than 50, and the incorrect answer threshold may be a numerical value equal to or greater than 50.

[0041] Then, the correct / incorrect information output unit 15 outputs the correct / incorrect information (S15). The output is performed via the output device 106, for example.

[0042] If the device 10 further includes a learning unit 16, for example, after step S15, the learning unit 16 may calculate the accuracy rate of the output correct / incorrect information, and adjust the correct answer threshold and the incorrect answer threshold in the correct / incorrect judgment unit 14 based on the accuracy rate (S16). Note that this is not limiting, and the correct answer threshold and the incorrect answer threshold can be adjusted manually by the user, for example.

[0043] The step S16 is an optional step and may not be executed. The method for determining whether a figure is correct or not according to the present embodiment ends the process (END) after the step S15 or the step S16, for example.

[0044] In conventional OCR (Optical Character Recognition) technology, even when deep learning is used, there are many cases where it is difficult to draw a line at the judgment criteria. Specifically, for example, there are cases where a character to be recognized as a specific character (e.g., "4") by conventional OCR technology appears to a human as a different character (e.g., "9") from the specific character. Furthermore, the judgment criteria may differ depending on the operational status of the OCR technology, and different characters may be recognized depending on the operational status. In contrast, according to this embodiment, the accuracy of graphic recognition can be improved by using, for example, the confidence factor and the correct answer information. Furthermore, when the shape of a product is recognized as a graphic (e.g., circle, triangle, square, etc.), the invention described in this embodiment can also be applied to quality control on a production line.

[0045] [Embodiment 2] The present invention will be described in more detail below by taking the example of recognizing handwritten numbers, although these are merely examples and the present invention is not limited to these.

[0046] FIG. 4(A) is a schematic diagram showing an example of a question and an answer (graphic image) handwritten for that question. In FIG. 4(A), a question asking about the sum of 2 and 2 and an answer field 1 in which the answer to the question is handwritten are displayed on the touch display of the device 10, which is a smartphone. In this example, it is assumed that the correct answer information corresponding to the question is preset to "4" in the device 10. In this case, the graphic image acquisition unit 11 acquires the graphic image entered in the answer field 1. Next, the graphic image recognition unit 12 recognizes the graphic image and generates a confidence level list. An example of the confidence level list in this example is shown in FIG. 4(B). In this confidence level list, the "Recognized Number (Recognized Figure)" column indicates the recognized figures recognized by the graphic image recognition unit 12, and the "Confidence Level (%)" column indicates their confidence levels. In this example, the confidence level is limited to 100%, and the higher the confidence level, the higher the likelihood that the recognized figure is a graphic image. The degree of certainty is not limited to this example. In the following example, it is assumed that the correct answer threshold is preset to a certainty of 10% and the incorrect answer threshold is preset to a certainty of 85%.

[0047] The processing by the correctness determination unit 14 will be described in more detail with reference to FIG. 4 as an example. In this example, since "4" is the correct answer as described above, the correctness determination unit 14 generates match / mismatch information indicating that only the recognition figure "4" matches the correct answer information and that the other recognition figures do not match the correct answer information, as shown in the "Match / Mismatch" column in FIG. 4(B). Next, the correctness determination by the correctness determination unit 14 will be described. In this example, the recognition figure "4" has the highest confidence level in the confidence level list shown in FIG. 4(B) and matches the correct answer information. Furthermore, the confidence level of the recognition figure "4" (98%) is equal to or greater than the correct answer threshold (10%). Therefore, since the recognition figure "4" satisfies both conditions (1a) and (1b) of the correct answer determination condition, which is the judgment criterion (1), the correctness determination unit 14 determines the graphic image entered in the answer column 1 as the correct answer and generates correctness information indicating the correct answer.

[0048] FIG. 5(A) is a schematic diagram showing an example of a question and an answer (graphic image) handwritten for that question. FIG. 5(A) is the same as FIG. 4(A) except that the answer handwritten in answer column 1 is different from the answer shown in FIG. 4(A). In this example, it is assumed that "4" is preset in the device 10 as the correct answer information corresponding to the question. In this case, as described above, the graphic image acquisition unit 11 acquires the graphic image entered in answer column 1, and the graphic image recognition unit 12 recognizes the graphic image and generates a confidence level list. An example of the confidence level list in this example is shown in FIG. 5(B). The confidence level list shown in FIG. 5(B) has the same content as each confidence level list shown in FIG. 4(B) except that the confidence levels for each recognized graphic are different.

[0049] The processing by the correctness determination unit 14 will be described in more detail with reference to FIG. 5. In this example, as described above, since "4" is the correct answer, the correctness determination unit 14 generates match / mismatch information indicating that only the recognition figure "4" matches the correct answer information and that the other recognition figures do not match the correct answer information, as shown in the "Match / Mismatch" column in FIG. 5(B). Next, the correctness determination by the correctness determination unit 14 will be described. In this example, the recognition figure "3" has the highest confidence in the confidence list shown in FIG. 5(B), but does not match the correct answer information. Furthermore, the confidence of the recognition figure "4" that matches the correct answer information is equal to or lower than the correct answer threshold. Therefore, in this example, neither of the conditions (1a) nor (1b) of the correct answer determination condition, which is the judgment criterion (1), is satisfied, and therefore the correctness determination unit 14 does not determine the graphic image entered in the answer column 1 as the correct answer. On the other hand, in this example, the confidence level (2%) of the recognition figure "4" that matches the correct answer information in the confidence level list shown in Figure 5(B) is less than the correct answer threshold (10%), and the confidence level of the recognition figure that does not match the correct answer information (for example, the recognition figure "3" with the highest confidence level: 98%) is equal to or greater than the incorrect answer threshold (85%). Therefore, in this example, the incorrect answer determination criterion (2a), which is the judgment criterion (2), is satisfied, so the correctness determination unit 14 determines that the graphic image entered in the answer column 1 is an incorrect answer and generates correctness information indicating the incorrect answer.

[0050] FIG. 6(A) is a schematic diagram showing an example of a question and an answer (graphic image) handwritten for that question. FIG. 6(A) is the same as FIG. 4(A) except that the answer handwritten in answer column 1 is different from the answer shown in FIG. 4(A). In this example, it is assumed that "4" is preset in the device 10 as the correct answer information corresponding to the question. In this case, as described above, the graphic image acquisition unit 11 acquires the graphic image entered in answer column 1, and the graphic image recognition unit 12 recognizes the graphic image and generates a confidence level list. An example of the confidence level list in this example is shown in FIG. 6(B). The confidence level list shown in FIG. 6(B) has the same content as each confidence level list shown in FIG. 4(B) except that the confidence levels for each recognized graphic are different.

[0051] The processing by the accuracy determination unit 14 will be described in more detail with reference to FIG. 6 as an example. In this example, since "4" is the correct answer as described above, the accuracy determination unit 14 generates match / mismatch information indicating that only the recognition figure "4" matches the correct answer information and that the other recognition figures do not match the correct answer information, as shown in the "Match / Mismatch" column in FIG. 6(B). Next, the accuracy determination by the accuracy determination unit 14 will be described. In this example, in the confidence list shown in FIG. 6(B), the confidence level (9%) of the recognition figure "4" that matches the correct answer information is less than the correct answer threshold (10%), so it does not satisfy the condition (1b) of the accuracy determination condition, which is the judgment criterion (1). However, since the recognition figure "4" has the highest confidence level in the confidence list shown in FIG. 6(B) and matches the correct answer information, it satisfies the condition (1a) of the accuracy determination condition, which is the judgment criterion (1). Therefore, the correctness determining unit 14 determines that the graphic image input in the answer column 1 is the correct answer, and generates correctness information indicating the correct answer.

[0052] FIG. 7(A) is a schematic diagram showing an example of a question and an answer (graphic image) handwritten for that question. FIG. 7(A) is the same as FIG. 4(A) except that the answer handwritten in answer column 1 is different from the answer shown in FIG. 4(A). In this example, it is assumed that "4" is preset in the device 10 as the correct answer information corresponding to the question. In this case, as described above, the graphic image acquisition unit 11 acquires the graphic image entered in answer column 1, and the graphic image recognition unit 12 recognizes the graphic image and generates a confidence level list. An example of the confidence level list in this example is shown in FIG. 7(B). The confidence level list shown in FIG. 7(B) has the same content as each confidence level list shown in FIG. 4(B) except that the confidence levels for each recognized graphic are different.

[0053] The processing by the correctness determination unit 14 will be described in more detail with reference to FIG. 7 as an example. In this example, since "4" is the correct answer as described above, the correctness determination unit 14 generates match / mismatch information indicating that only the recognition figure "4" matches the correct answer information and that the other recognition figures do not match the correct answer information, as shown in the "Match / Mismatch" column in FIG. 7(B). Next, the correctness determination by the correctness determination unit 14 will be described. In this example, the recognition figure with the highest confidence in the confidence list shown in FIG. 7(B) is the recognition figure "5" that does not match the correct answer information, and therefore does not satisfy the condition (1a) of the correct answer determination condition, which is the judgment criterion (1). However, since the confidence (50%) of the recognition figure "4" that matches the correct answer information is equal to or greater than the correct answer threshold (10%), the condition (1b) of the correct answer determination condition, which is the judgment criterion (1), is satisfied. Therefore, the correctness determination unit 14 determines the graphic image entered in the answer column 1 as the correct answer and generates correctness information indicating the correct answer.

[0054] FIG. 8(A) is a schematic diagram showing an example of a question and an answer (graphic image) handwritten for that question. FIG. 8(A) is the same as FIG. 4(A) except that the answer handwritten in answer column 1 is different from the answer shown in FIG. 4(A). In this example, it is assumed that "4" is preset in the device 10 as the correct answer information corresponding to the question. In this case, as described above, the graphic image acquisition unit 11 acquires the graphic image entered in answer column 1, and the graphic image recognition unit 12 recognizes the graphic image and generates a confidence level list. An example of the confidence level list in this example is shown in FIG. 8(B). The confidence level list shown in FIG. 8(B) has the same content as each confidence level list shown in FIG. 4(B) except that the confidence levels for each recognized graphic are different.

[0055] The processing by the accuracy determination unit 14 will be described in more detail with reference to FIG. 8 as an example. In this example, since "4" is the correct answer as described above, the accuracy determination unit 14 generates match / mismatch information indicating that only the recognition figure "4" matches the correct answer information and that the other recognition figures do not match the correct answer information, as shown in the "Match / Mismatch" column in FIG. 8(B). Next, the accuracy determination by the accuracy determination unit 14 will be described. In this example, the recognition figure with the highest confidence in the confidence list shown in FIG. 8(B) is the recognition figure "3" which does not match the correct answer information, and therefore does not satisfy the condition (1a) of the accuracy determination condition, which is the judgment criterion (1). Furthermore, since the confidence (1%) of the recognition figure "4" which matches the correct answer information is less than the correct answer threshold (10%), the condition (1b) of the accuracy determination condition, which is the judgment criterion (1), is also not satisfied. Furthermore, since the confidence level (1%) of the recognition figure "4" that matches the correct answer information is less than the correct answer threshold (10%), and the confidence level of the recognition figure that does not match the correct answer information (e.g., the recognition figure "3" with the highest confidence level: 9%) is less than the incorrect answer threshold (85%), the incorrect answer determination criterion (2a) of the judgment criterion (2) is not satisfied. On the other hand, since the confidence level (2%) of the recognition figure "4" that matches the correct answer information is less than the correct answer threshold (10%), and the confidence level of the recognition figure that does not match the correct answer information (e.g., the recognition figure "3" with the highest confidence level: 9%) is less than the incorrect answer threshold (85%), the condition (3a) of the undeterminable condition of the judgment criterion (3) is satisfied. Therefore, the correctness determination unit 14 determines that the graphic image entered in the answer column 1 is undeterminable. When it is determined that the graphic image is undeterminable as in this example, correctness information indicating an incorrect answer or undeterminable may be generated, for example. Furthermore, if it is determined that the determination is impossible, the correctness determining unit 14 may obtain the result of the user's visual determination of the correctness of the graphic image, and generate correctness information.

[0056] [Embodiment 3] The program of this embodiment is a program for causing a computer to execute each step of the method of the present invention as a procedure. In the present invention, "procedure" may be read as "processing." The program of this embodiment may be recorded, for example, on a computer-readable recording medium. The recording medium is, for example, a non-transitory computer-readable storage medium. The recording medium is not particularly limited, and examples thereof include a read-only memory (ROM), a hard disk (HD), an optical disk, etc.

[0057] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

[0058] <Additional Notes> Some or all of the above embodiments can be described as, but not limited to, the following supplementary notes. (Appendix 1) The system includes a graphic image acquisition unit, a graphic image recognition unit, a correct answer information acquisition unit, a correct / incorrect determination unit, and a correct / incorrect information output unit, the graphic image acquisition unit acquires a graphic image input in response to a question, the graphic image recognition unit recognizes the acquired graphic image, calculates the recognized one or more recognized graphics and the certainty of the recognized graphics, and generates a certainty list; the correct answer information acquisition unit acquires correct answer information for the question, the correctness determination unit compares the recognized graphic with the correct answer information to generate match / mismatch information, and determines whether the graphic image is correct or not based on the confidence level list and the match / mismatch information to generate correctness information; the correct / incorrect information output unit outputs the correct / incorrect information. A device for determining whether a figure is correct or incorrect. (Appendix 2) The correctness determination unit determines whether the graphic image is correct or not based on the following criteria (1), (2), and (3) to generate correctness information. 2. The graphic correctness determination device according to claim 1. (1) Correct answer criteria If the graphic image satisfies either of the following conditions (1a) and (1b), it is determined to be correct. (1a) The recognized figure with the highest confidence in the confidence list matches the correct answer information. (1b) The certainty of the recognized image that matches the correct answer information is equal to or greater than a correct answer threshold. (2) Criteria for determining incorrect answers If the following condition (2a) is met, the graphic image is determined to be an incorrect answer. (2a) In the confidence level list, the confidence level of the recognition pattern that matches the correct answer information is less than the correct answer threshold, and the confidence level of the recognition pattern that does not match the correct answer information is equal to or greater than the incorrect answer threshold. (3) Conditions that cannot be judged If the following condition (3a) is satisfied, the graphic image is determined to be unidentifiable. (3a) In the confidence level list, the confidence level of the recognized image that matches the correct answer information is less than the correct answer threshold, and the confidence level of the recognized figure that does not match the correct answer information is less than the incorrect answer threshold. (Appendix 3) It further includes a learning section, the learning unit calculates a correct answer rate of the output correct / incorrect information, and adjusts the correct answer threshold and the incorrect answer threshold in the correct / incorrect determination unit based on the correct answer rate. 3. The graphic correctness determination device according to claim 2. (Appendix 4) the graphic image recognition unit includes a trained model based on machine learning, Recognizing the graphic image using the trained model, and calculating one or more recognized graphics and the confidence level of the recognized graphics. 4. A graphic correctness determination device according to any one of appendices 1 to 3. (Appendix 5) The method includes a graphic image acquisition step, a graphic image recognition step, a correct answer information acquisition step, a correct / incorrect determination step, and a correct / incorrect information output step, The graphic image acquisition step acquires a graphic image input in response to a question, The graphic image recognition step recognizes the acquired graphic image, calculates the recognized one or more recognized graphics and the certainty of the recognized graphics, and generates a certainty list; The correct answer information acquisition step acquires correct answer information for the question, the correctness determining step includes: comparing the recognized graphic with the correct answer information to generate match / mismatch information; and determining whether the graphic image is correct or not based on the confidence level list and the match / mismatch information to generate correctness information; The correct / incorrect information output step outputs the correct / incorrect information. How to determine whether a figure is correct or incorrect. (Appendix 6) The correctness determination step determines whether the graphic image is correct or not based on the following criteria (1), (2), and (3), and generates correctness information. A method for determining whether a figure is correct or incorrect, as described in Appendix 5. (1) Correct answer criteria If the graphic image satisfies either of the following conditions (1a) and (1b), it is determined to be correct. (1a) The recognized figure with the highest confidence in the confidence list matches the correct answer information. (1b) The certainty of the recognized image that matches the correct answer information is equal to or greater than a correct answer threshold. (2) Criteria for determining incorrect answers If the following condition (2a) is met, the graphic image is determined to be an incorrect answer. (2a) In the confidence level list, the confidence level of the recognition pattern that matches the correct answer information is less than the correct answer threshold, and the confidence level of the recognition pattern that does not match the correct answer information is equal to or greater than the incorrect answer threshold. (3) Conditions that cannot be judged If the following condition (3a) is satisfied, the graphic image is determined to be unidentifiable. (3a) In the confidence level list, the confidence level of the recognized image that matches the correct answer information is less than the correct answer threshold, and the confidence level of the recognized figure that does not match the correct answer information is less than the incorrect answer threshold. (Appendix 7) Furthermore, it includes a learning process, The learning step calculates a correct answer rate of the output correct / incorrect information, and adjusts the correct answer threshold and the incorrect answer threshold in the correct / incorrect determining step based on the correct answer rate. A method for determining whether a figure is correct or incorrect, as described in Appendix 6. (Appendix 8) the graphic image recognition step includes a trained model obtained by machine learning, Recognizing the graphic image using the trained model, and calculating one or more recognized graphics and the confidence level of the recognized graphics. A method for determining whether a figure is correct or incorrect, as described in any one of appendices 5 to 7. (Appendix 9) A program for causing a computer to execute procedures including a procedure for acquiring a graphic image, a procedure for recognizing a graphic image, a procedure for acquiring correct answer information, a procedure for determining whether the image is correct, and a procedure for outputting correct or incorrect information; The graphic image acquisition step acquires a graphic image input in response to a question, The graphic image recognition step recognizes the acquired graphic image, calculates one or more recognized recognized figures and certainty of the recognized figures, and generates a certainty list; The correct answer information acquisition step acquires correct answer information for the question, the correctness determination step includes collating the recognized graphic with the correct answer information to generate match / mismatch information, and determining whether the graphic image is correct or not based on the confidence level list and the match / mismatch information to generate correctness information; The correct / incorrect information output step outputs the correct / incorrect information. (Appendix 10) The correctness determination step determines whether the graphic image is correct or not based on the following criteria (1), (2), and (3), and generates correctness information: The program described in Appendix 9. (1) Correct answer criteria If the graphic image satisfies either of the following conditions (1a) and (1b), it is determined to be correct. (1a) The recognized figure with the highest confidence in the confidence list matches the correct answer information. (1b) The certainty of the recognized image that matches the correct answer information is equal to or greater than a correct answer threshold. (2) Criteria for determining incorrect answers If the following condition (2a) is met, the graphic image is determined to be an incorrect answer. (2a) In the confidence level list, the confidence level of the recognition pattern that matches the correct answer information is less than the correct answer threshold, and the confidence level of the recognition pattern that does not match the correct answer information is equal to or greater than the incorrect answer threshold. (3) Conditions that cannot be judged If the following condition (3a) is satisfied, the graphic image is determined to be unidentifiable. (3a) In the confidence level list, the confidence level of the recognized image that matches the correct answer information is less than the correct answer threshold, and the confidence level of the recognized figure that does not match the correct answer information is less than the incorrect answer threshold. (Appendix 11) Further, it includes a learning procedure, the learning step calculates a correct answer rate of the output correct / incorrect information, and adjusts the correct answer threshold and the incorrect answer threshold in the correct / incorrect determination step based on the correct answer rate; The program described in Appendix 10. (Appendix 12) The graphic image recognition procedure includes a trained model through machine learning, Recognizing the graphic image using the trained model, and calculating one or more recognized graphics and the confidence level of the recognized graphics. 12. The program of any one of appendices 9 to 11. (Appendix 13) A computer-readable recording medium having recorded thereon a program according to any one of appendices 9 to 12. [Industrial Applicability]

[0059] According to the present invention, it is possible to improve the accuracy of graphic recognition for graphics entered using an input format in which the font and shape vary each time the graphics are written, such as by hand. Therefore, the present invention is particularly useful for recognizing graphics entered for a question in which the answer required from the respondent is known in advance. [Explanation of symbols]

[0060] 1 Answer column 10. Graphic accuracy judgment device 11 Graphic image acquisition unit 12 Graphic image recognition section 13 Correct answer information acquisition unit 14 Correctness Judgment Section 15 Correction information output section 16 Learning Department 101 Central Processing Unit 102 memory 103 Bus 104 Storage device 105 Input Device 106 Output Device 107 Communication Devices

Claims

1. The system includes a graphic image acquisition unit, a graphic image recognition unit, a correct answer information acquisition unit, a correct / incorrect determination unit, and a correct / incorrect information output unit, the graphic image acquisition unit acquires a graphic image input in response to a question, the graphic image recognition unit recognizes the acquired graphic image, calculates the recognized one or more recognized graphics and the certainty of the recognized graphics, and generates a certainty list; the correct answer information acquisition unit acquires correct answer information for the question, the correctness determination unit compares the recognized graphic with the correct answer information to generate match / mismatch information, and determines whether the graphic image is correct or not based on the confidence level list and the match / mismatch information according to the following criteria (1), (2), and (3), to generate correctness information; the correct / incorrect information output unit outputs the correct / incorrect information. A device for determining whether a figure is correct or incorrect. (1) Correct answer determination conditions If the graphic image satisfies either of the following conditions (1a) and (1b), it is determined to be correct. (1a) The recognition pattern with the highest degree of certainty in the certainty list matches the correct answer information. (1b) The degree of certainty of the recognized figure that matches the correct answer information is equal to or greater than a correct answer threshold. (2) Criteria for determining incorrect answers If the following condition (2a) is met, the graphic image is determined to be an incorrect answer. (2a) In the confidence level list, the confidence level of the recognition pattern that matches the correct answer information is less than a correct answer threshold, and the confidence level of the recognition pattern that does not match the correct answer information is equal to or greater than an incorrect answer threshold. (3) Conditions that cannot be judged If the following condition (3a) is satisfied, the graphic image is determined to be unidentifiable. (3a) In the confidence level list, the confidence level of the recognition pattern that matches the correct answer information is less than the correct answer threshold, and the confidence level of the recognition pattern that does not match the correct answer information is less than the incorrect answer threshold.

2. It further includes a learning section, the learning unit calculates a correct answer rate of the output correct / incorrect information, and adjusts the correct answer threshold and the incorrect answer threshold in the correct / incorrect determination unit based on the correct answer rate.

2. The graphic correctness determination device according to claim 1.

3. the graphic image recognition unit includes a trained model based on machine learning, Recognizing the graphic image using the trained model, and calculating one or more recognized graphics and the confidence level of the recognized graphics.

3. The graphic correctness determination device according to claim 1 or 2.

4. The steps include a graphic image acquisition step, a graphic image recognition step, a correct answer information acquisition step, a correct / incorrect determination step, and a correct / incorrect information output step, The graphic image acquisition step acquires a graphic image input in response to a question, The graphic image recognition step recognizes the acquired graphic image, calculates the recognized one or more recognized graphics and the certainty of the recognized graphics, and generates a certainty list; The correct answer information acquisition step acquires correct answer information for the question, The correctness determining step includes collating the recognized graphic with the correct answer information to generate match / mismatch information, and determining whether the graphic image is correct or not based on the confidence level list and the match / mismatch information according to the following criteria (1), (2), and (3), to generate correctness information: The correct / incorrect information output step outputs the correct / incorrect information. A method for determining whether a graphic is correct or incorrect, in which each step is performed by a computer. (1) Correct answer determination conditions If the graphic image satisfies either of the following conditions (1a) and (1b), it is determined to be correct. (1a) The recognition pattern with the highest degree of certainty in the certainty list matches the correct answer information. (1b) The degree of certainty of the recognized figure that matches the correct answer information is equal to or greater than a correct answer threshold. (2) Criteria for determining incorrect answers If the following condition (2a) is met, the graphic image is determined to be an incorrect answer. (2a) In the confidence level list, the confidence level of the recognition pattern that matches the correct answer information is less than a correct answer threshold, and the confidence level of the recognition pattern that does not match the correct answer information is equal to or greater than an incorrect answer threshold. (3) Conditions that cannot be judged If the following condition (3a) is satisfied, the graphic image is determined to be unidentifiable. (3a) In the confidence level list, the confidence level of the recognition pattern that matches the correct answer information is less than the correct answer threshold, and the confidence level of the recognition pattern that does not match the correct answer information is less than the incorrect answer threshold.

5. Furthermore, it includes a learning process, The learning step calculates a correct answer rate of the output correct / incorrect information, and adjusts the correct answer threshold and the incorrect answer threshold in the correct / incorrect determining step based on the correct answer rate.

5. The method for determining whether a graphic is correct or not according to claim 4.

6. the graphic image recognition step includes a trained model obtained by machine learning, Recognizing the graphic image using the trained model, and calculating one or more recognized graphics and the confidence level of the recognized graphics.

6. The method for determining whether a figure is correct or incorrect according to claim 4 or 5.

7. A program for causing a computer to execute procedures including a graphic image acquisition procedure, a graphic image recognition procedure, a correct answer information acquisition procedure, a correct / incorrect determination procedure, and a correct / incorrect information output procedure; The graphic image acquisition step acquires a graphic image input in response to a question, The graphic image recognition step recognizes the acquired graphic image, calculates one or more recognized recognized figures and certainty of the recognized figures, and generates a certainty list; The correct answer information acquisition step acquires correct answer information for the question, The correctness determination step includes collating the recognized graphic with the correct answer information to generate match / mismatch information, and determining whether the graphic image is correct or not based on the confidence level list and the match / mismatch information according to the following criteria (1), (2), and (3), to generate correctness information: The correct / incorrect information output step outputs the correct / incorrect information. (1) Correct answer determination conditions If the graphic image satisfies either of the following conditions (1a) and (1b), it is determined to be correct. (1a) The recognition pattern with the highest degree of certainty in the certainty list matches the correct answer information. (1b) The degree of certainty of the recognized figure that matches the correct answer information is equal to or greater than a correct answer threshold. (2) Criteria for determining incorrect answers If the following condition (2a) is met, the graphic image is determined to be an incorrect answer. (2a) In the confidence level list, the confidence level of the recognition pattern that matches the correct answer information is less than a correct answer threshold, and the confidence level of the recognition pattern that does not match the correct answer information is equal to or greater than an incorrect answer threshold. (3) Conditions that cannot be judged If the following condition (3a) is satisfied, the graphic image is determined to be unidentifiable. (3a) In the confidence level list, the confidence level of the recognition pattern that matches the correct answer information is less than the correct answer threshold, and the confidence level of the recognition pattern that does not match the correct answer information is less than the incorrect answer threshold.

8. Further comprising a learning procedure, the learning step calculates a correct answer rate of the output correct / incorrect information, and adjusts the correct answer threshold and the incorrect answer threshold in the correct / incorrect determination step based on the correct answer rate; The program according to claim 7.

9. The graphic image recognition procedure includes a trained model obtained by machine learning, Recognizing the graphic image using the trained model, and calculating one or more recognized graphics and the confidence level of the recognized graphics.

9. The program according to claim 7 or 8.

10. A computer-readable recording medium having recorded thereon a program described in any one of claims 7 to 9.

Images