Information processing device, information processing method, and program

Abstract

An information processing device 1 comprises: an acquisition unit 131 that acquires teacher data in which a recognized character string, which is a character string generated by recognizing a character string included in an image through optical character recognition, and a ground truth character string, which indicates the ground truth of a character string in the image corresponding to the recognized character string, are associated with each other; and a training unit 132 that, on the basis of the teacher data acquired by the acquisition unit 131, trains a pretrained model pretrained to be able to execute a natural language processing task, to thereby generate a trained model trained to output a character string using a recognized character string as an input, the character string that is output corresponding to the recognized character string that is input.　The recognized character string may include a character string included in an image and generated through optical character recognition of a document on which a seal is stamped so as to overlap one or more characters constituting the character string.

Description

Information processing device, information processing method, and program

[0001] The present invention relates to an information processing device, an information processing method, and a program.

[0002] There is known a technique for correcting character strings recognized by OCR by detecting errors and correcting the detected errors (see, for example, Patent Document 1).

[0003] Special table 2018-523188 publication

[0004] In transaction documents, a seal may be placed over a character string in the document, and in such documents, it may happen that multiple characters in the character string cannot be read. In the prior art, when multiple characters in the character string are misrecognized in this way, it is difficult to correct the misrecognition.

[0005] The present invention has been made in view of these points, and has as its object to make it possible to correct the recognition results based on the tendency of character recognition by OCR (Optical Character Recognition).

[0006] An information processing device according to a first aspect of the present invention includes an acquisition unit that acquires training data that associates a recognized string, which is a string generated by recognizing a string included in an image using optical character recognition, with a correct string that indicates the correct string in the image that corresponds to the recognized string, and a learning unit that generates a trained model that is trained to take a recognized string as input and output a string that corresponds to the input recognized string, by training a pre-trained model that has been pre-trained to be able to perform a natural language processing task based on the training data acquired by the acquisition unit.

[0007] The recognized character string may be a character string contained in an image, and may include a character string generated by optical character recognition of a document in which a seal is stamped so as to overlap one or more characters that make up the character string.

[0008] The acquisition unit acquires a recognition character string to be inferred, and the information processing device may further have an inference unit that inputs the recognition character string acquired by the acquisition unit into the trained model and outputs a character string corresponding to the input recognition character string, and a display control unit that displays the character string output by the trained model on a display unit.

[0009] The acquisition unit may further acquire a recognized image in order to generate a recognition character string to be inferred, and the information processing device may further have a determination unit that determines whether or not a character string in the image overlaps with a seal impression, and when the determination unit determines that a character string and a seal impression overlap in the image, the inference unit may input the recognition character string acquired by the acquisition unit into the trained model and output a character string corresponding to the input recognition character string.

[0010] The training data may further be associated with recognition means identification information for identifying the recognition means that performed optical character recognition on the image corresponding to the recognition string, and the trained model may take the recognition string and the recognition means identification information as input and output a string corresponding to the input recognition string, the acquisition unit may acquire the recognition string to be inferred and the recognition means identification information corresponding to the recognition string, and the inference unit may input the recognition string and recognition means identification information acquired by the acquisition unit into the trained model and output a string corresponding to the input recognition string.

[0011] An information processing method of a second aspect of the present invention includes a step executed by a computer to acquire training data that associates a recognized string, which is a string generated by recognizing a string included in an image using optical character recognition, with a correct string that indicates the correct string in the image that corresponds to the recognized string, and a step of training a pre-trained model that has been pre-trained to be able to perform natural language processing tasks based on the acquired training data, thereby generating a trained model that has been trained to take a recognized string as input and output a string that corresponds to the input recognized string.

[0012] In a third aspect of the program of the present invention, a computer is caused to execute the steps of acquiring training data that associates a recognized string, which is a string generated by recognizing a string included in an image using optical character recognition, with a correct string that indicates the correct string in the image that corresponds to the recognized string, and training a pre-trained model that has been pre-trained to be able to perform natural language processing tasks based on the acquired training data, thereby generating a trained model that has been trained to take a recognized string as input and output a string that corresponds to the input recognized string.

[0013] The present invention has the effect of being able to correct recognition results obtained by OCR that have a predetermined tendency.

[0014] FIG. 1 is a diagram for explaining an overview of an information processing system S. FIG. 2 is a block diagram showing the configuration of an information processing device 1. FIG. 3 is a diagram showing an example of training data acquired by an acquisition unit 131. FIG. 4 is a diagram showing a part of an image for which a recognition character string is to be generated. FIG. 5 is a flowchart showing the flow of processing in the information processing device 1. FIG. 6 is a diagram showing an example of training data acquired by an acquisition unit 131.

[0015] 1 is a diagram illustrating an overview of the information processing system S. The information processing system S is a system for outputting the results of optical character recognition of character strings included in an image. The information processing system S includes an information processing device 1, a recognition means 2, and an information terminal 3.

[0016] The information processing device 1 is a device for correcting a character string generated by optical character recognition. The recognition means 2 performs optical character recognition on an image to output a character string contained in the image. The information processing device 1 may be configured integrally with the recognition means 2. In other words, the information processing device 1 may be configured to generate a character string as a recognition result based on an image to be recognized, correct the character string as a recognition result, and output the corrected character string.

[0017] The information terminal 3 is a terminal used by a user who uses the information processing system S. The information terminal 3 is, for example, a smartphone, a tablet, or a personal computer.

[0018] First, learning in the information processing device 1 will be described. The information processing device 1 acquires training data D1 that associates a recognized character string with a correct character string. A recognized character string is a character string generated by recognizing a character string included in an image using optical character recognition. A correct character string indicates the correct character string included in the image from which the recognized character string is extracted. The training data includes multiple pairs of a recognized character string and a correct character string. The recognized character string may include a character string different from the correct character string (i.e., a character string generated as a result of OCR incorrectly recognizing a character string), or may include a character string identical to the correct character string.

[0019] The information processing device 1 trains a pre-trained model based on training data to generate a trained model M. When a recognition character string to be inferred is input, the trained model M outputs a character string corresponding to the recognition character string to be inferred. The pre-trained model is a trained model trained to understand the meaning and context of a sentence using a large amount of text data as a training dataset. As an example, the pre-trained model is a trained model trained to predict subsequent words based on a given context by solving Masked Language Model (MLM) and Next Sentence Prediction (NSP) tasks based on the training dataset.

[0020] An overview of the inference process in the information processing system S will be described. The information processing device 1 acquires a recognition character string corresponding to image data D2 to be inferred. The image data D2 is image data obtained by imaging a document including a character string. More specifically, the recognition means 2 acquires the image data D2 to be inferred, performs optical character recognition on the image data D2 to generate a recognition character string to be inferred, and outputs the generated recognition character string to the information processing device 1. The information processing device 1 inputs the recognition character string into a trained model M, and outputs a character string corresponding to the recognition character string.

[0021] The information processing device 1 configured in this manner has the effect of being able to correct the recognition result based on the OCR recognition tendency. That is, by learning the relationship between the OCR recognition error result and the actual character string contained in the image, it is expected that the recognition result can be corrected even when it is difficult to detect an error due to a difference between the number of characters read by the OCR and the actual number of characters contained in the image.

[0022] 2 is a block diagram showing the configuration of the information processing device 1. The information processing device 1 has a communication unit 11, a storage unit 12, and a control unit 13. The control unit 13 has an acquisition unit 131, a learning unit 132, an inference unit 133, a display control unit 134, and a determination unit 135.

[0023] The communication unit 11 is a communication interface for transmitting and receiving data to and from other devices via a network. The storage unit 12 is a storage medium including a read-only memory (ROM), a random access memory (RAM), a solid-state drive (SSD), a hard disk drive, etc. The storage unit 12 stores in advance a program to be executed by the control unit 13.

[0024] The control unit 13 is a processor such as a CPU (Central Processing Unit), etc. The control unit 13 executes the programs stored in the storage unit 12, thereby functioning as an acquisition unit 131, a learning unit 132, an inference unit 133, a display control unit 134, and a determination unit 135.

[0025] The acquisition unit 131 acquires teacher data. As an example, the acquisition unit 131 acquires the teacher data from the information terminal 3. Fig. 3 is a diagram showing an example of the teacher data acquired by the acquisition unit 131. As shown in Fig. 3, in the teacher data, a recognition character string is associated with a correct character string corresponding to the recognition character string.

[0026] The learning unit 132 generates a trained model that is trained to take a recognition string as input and output a character string corresponding to the input recognition string by training a pre-trained model that has been pre-trained to be able to execute a natural language processing task based on the training data acquired by the acquisition unit 131. As an example, the learning unit 132 trains the pre-trained model M by taking a recognition string as input and causing the pre-trained model to execute a task of outputting a character string, and updates parameters based on an error between the output character string and a correct string associated with the input recognition string, thereby training the pre-trained model M and generating the trained model M.

[0027] By configuring the information processing device 1 in this way, it is possible to provide a means for correcting the recognition result based on the tendency of recognition errors made by OCR.

[0028] Correcting the recognition results provided by the information processing device 1 is particularly suitable when some element has been overwritten so as to overlap the character string to be read at the time of optical character recognition. As an example, the information processing device 1 is suitable for correcting a character string in which a recognized character string has been generated by optical character recognition of a document in which a seal has been placed so as to overlap the character string to be recognized. More specifically, this is the case when correcting a character string on an invoice or the like that has been issued with a seal placed so as to overlap the issuing company's name. That is, the recognized character string is a character string contained in an image, and includes a character string generated by optical character recognition of a document in which a seal has been placed so as to overlap one or more characters that make up the character string. FIG. 4 illustrates a portion of an image of a stamped invoice. In the image illustrated in FIG. 4, a seal has been placed so as to overlap the portion of the character string "ABC Co., Ltd." that originally appeared on the document, corresponding to "Company A." In this case, the recognition character string is expected to be, for example, a character string such as "Stock #BC," and the learning unit 132 generates a trained model M by pairing the recognition character string "Stock #BC" with the correct character string "ABC Co., Ltd." as training data.

[0029] The following describes the inference process in the information processing device 1. The acquisition unit 131 acquires a recognition character string to be inferred. The inference unit 133 inputs the recognition character string acquired by the acquisition unit 131 into the trained model M, and outputs a character string corresponding to the input recognition character string. The acquisition unit 131 may acquire the recognition character string from a recognition means, or may acquire the character string from the information terminal 3.

[0030] The display control unit 134 causes the display unit to display the character string output by the trained model M. As an example, the display control unit 134 causes the character string output by the trained model M to be displayed on the display unit of the information terminal 3. The information processing device 1 may also have a display unit, in which case the display control unit 134 causes the display unit of the information processing device 1 to display the character string output by the trained model M.

[0031] The display control unit 134 may cause the display unit to display, in association with each other, the recognition character string acquired by the acquisition unit 131 and a character string corresponding to the recognition character string that the inference unit 133 has output to the trained model M. The display control unit 134 may also cause the display unit to display a screen configured to allow the user to select either the recognition character string or the character string that the inference unit 133 has output. The display control unit 134 may determine whether the recognition character string matches the character string that the inference unit 133 has output to the trained model M, and, if the recognition character string does not match the character string that the inference unit 133 has output to the trained model M, may cause the display unit to display information that allows the user to recognize that the character string has been corrected.

[0032] The information processing device 1 is configured to input a recognized character string into a trained model M that has learned the tendency of OCR errors and output a corresponding character string, making it possible to recognize a character string taking into account the tendency of OCR misrecognition.

[0033] The information processing device 1 may be configured to perform the above correction process when there is a possibility that the character string is difficult to read.

[0034] The acquisition unit 131 acquires an image that has been recognized in order to generate a recognition character string of an inference target. As an example, the acquisition unit 131 acquires the recognition character string and the image that has been recognized in order to generate the recognition character string in association with each other.

[0035] The determination unit 135 determines whether or not a character string in an image overlaps with a seal impression. As an example, the storage unit 12 stores a determination model, which is a trained model for determining whether or not a character string in an image overlaps with a seal impression. The determination model is trained to receive an image as input and output a flag indicating whether or not a character string included in the image overlaps with a seal impression. As an example, the determination model is trained based on training data that associates an image including a character string with a label indicating whether or not the character string in the image overlaps with a seal impression.

[0036] The determination unit 135 inputs the image acquired by the acquisition unit 131 into the determination model and outputs a flag indicating whether or not there is overlap between the character string included in the image and the seal impression. The determination unit 135 determines whether or not there is overlap between the character string and the seal impression in the image based on the flag output by the determination model.

[0037] When the determination unit 135 determines that a character string and a seal impression overlap in the image, the inference unit 133 inputs the recognition character string acquired by the acquisition unit 131 into the trained model M and outputs a character string corresponding to the input recognition character string. The display control unit 134 may associate the character string output by the trained model M with the image of the recognition target acquired by the acquisition unit 131 and display them on the display unit. This configuration allows the user to determine the validity of the character string output by the information processing device 1.

[0038] By configuring the information processing device 1 in this way, it becomes possible to correct the reading result based on the tendency of erroneous recognition when there is a possibility that reading of the character string is difficult. As a result, it becomes possible to prevent unnecessary correction of the character string, and the accuracy of character string recognition can be improved.

[0039] The information processing device 1 may further include a recognition unit for performing optical character recognition on an image and generating a recognition string. In this case, the acquisition unit 131 acquires an image to be subjected to optical character recognition and inputs the acquired image to the recognition unit. The acquisition unit 131 acquires the recognition string generated by the recognition unit performing optical character recognition on the image.

[0040] [Processing Flow in Information Processing Device 1] Fig. 5 is a flowchart showing the processing flow in the information processing device 1. The flowchart shown in Fig. 5 starts from the point in time when an instruction to start learning is received.

[0041] The acquisition unit 131 acquires teacher data (S01). The learning unit 132 generates a trained model M based on the acquired teacher data (S02). As an example, the learning unit 132 generates the trained model M by fine-tuning a pre-trained model based on the acquired teacher data.

[0042] The acquisition unit 131 acquires the recognition character string to be inferred (S03). The inference unit 133 inputs the recognition character string acquired by the acquisition unit 131 into the trained model M, and outputs a character string corresponding to the input recognition character string (S04). The display control unit 134 displays the character string output by the trained model M on the display unit (S05). Then, the information processing device 1 ends the processing.

[0043] <Modification> Each type of OCR may have different tendencies in recognizing character strings. Therefore, the information processing device 1 may be configured to train the trained model M based on training data that further includes the type of OCR that generated the recognized character string.

[0044] The training data is further associated with recognition means identification information for identifying the recognition means that performed optical character recognition on the image corresponding to the recognition character string. Fig. 6 is a diagram showing an example of training data acquired by the acquisition unit 131 in this case. In the training data shown in Fig. 6, recognition means identification information is further associated with pairs of a recognition character string and a correct answer character string. The recognition means identification information indicates, for example, the OCR vendor, the software version, or the model.

[0045] The learning unit 132 trains a pre-trained model based on training data including recognition means identification information, and generates a trained model M. In this case, the trained model M receives a recognition string and recognition means identification information as input, and outputs a string corresponding to the input recognition string.

[0046] The acquisition unit 131 acquires a recognition character string to be inferred and recognition means identification information corresponding to the recognition character string. That is, the acquisition unit 131 acquires the recognition character string to be inferred and the recognition means identification information indicating the recognition means that generated the recognition character string in association with each other.

[0047] The inference unit 133 inputs the recognition character string and recognition means identification information acquired by the acquisition unit 131 into the trained model M, and causes it to output a character string corresponding to the input recognition character string.

[0048] In the information processing device 1 according to the modified example, it is possible to provide corrections to the recognition results based on the recognition tendencies that differ for each type of OCR.

[0049] The present invention has been described above using embodiments, but the technical scope of the present invention is not limited to the scope described in the above embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. For example, all or part of the device can be configured by functionally or physically distributing or integrating in any unit. Furthermore, new embodiments resulting from any combination of multiple embodiments are also included in the embodiments of the present invention. The effects of the new embodiments resulting from the combination also have the effects of the original embodiments.

[0050] REFERENCE SIGNS LIST 1 Information processing device 2 Recognition means 3 Information terminal 11 Communication unit 12 Storage unit 13 Control unit 131 Acquisition unit 132 Learning unit 133 Inference unit 134 Display control unit 135 Determination unit

Claims

1. An information processing device having: an acquisition unit that acquires training data that associates a recognized string, which is a string generated by recognizing a string included in an image using optical character recognition, with a correct string that indicates the correct string in the image that corresponds to the recognized string; and a learning unit that generates a trained model that is trained to take a recognized string as input and output a string that corresponds to the input recognized string, by training a pre-trained model that has been pre-trained to be able to perform natural language processing tasks based on the training data acquired by the acquisition unit.

2. The information processing device according to claim 1, wherein the recognized character string is a character string contained in an image and includes a character string generated by optical character recognition of a document in which a seal is stamped so as to overlap one or more characters that make up the character string.

3. The information processing device according to claim 1, wherein the acquisition unit acquires a recognition character string to be inferred, and the information processing device further comprises: an inference unit that inputs the recognition character string acquired by the acquisition unit into the trained model and outputs a character string corresponding to the input recognition character string; and a display control unit that displays the character string output by the trained model on a display unit.

4. The information processing device according to claim 3, wherein the acquisition unit further acquires a recognized image in order to generate a recognition character string to be inferred, the information processing device further has a judgment unit that judges whether or not a character string in the image overlaps with a seal impression, and when the judgment unit judges that a character string and a seal impression overlap in the image, the inference unit inputs the recognition character string acquired by the acquisition unit into the trained model and outputs a character string corresponding to the input recognition character string.

5. The information processing device of claim 3, wherein the training data is further associated with recognition means identification information for identifying the recognition means that performed optical character recognition on the image corresponding to the recognition string, the trained model receives the recognition string and the recognition means identification information as input and outputs a string corresponding to the input recognition string, the acquisition unit acquires the recognition string to be inferred and the recognition means identification information corresponding to the recognition string, and the inference unit inputs the recognition string and recognition means identification information acquired by the acquisition unit into the trained model and outputs a string corresponding to the input recognition string.

6. An information processing method executed by a computer, comprising: a step of acquiring training data that associates a recognized string, which is a string generated by recognizing a string included in an image using optical character recognition, with a correct string that indicates the correct string in the image that corresponds to the recognized string; and a step of training a pre-trained model that has been pre-trained to be able to perform natural language processing tasks based on the acquired training data, thereby generating a trained model that has been trained to take a recognized string as input and output a string that corresponds to the input recognized string.

7. A program that causes a computer to execute the steps of: acquiring training data that associates a recognized string, which is a string generated by recognizing a string included in an image using optical character recognition, with a correct string that indicates the correct string in the image that corresponds to the recognized string; and generating a trained model that has been trained to take a recognized string as input and output a string that corresponds to the input recognized string, by training a pre-trained model that has been pre-trained to be able to perform natural language processing tasks based on the acquired training data.

Images