Information processing device, information processing method, and program

Abstract

An information processing device 1 that includes: an acquisition unit 131 that acquires first teacher data in which a plurality of image data including character strings, and correct answer character strings that are character strings included in each of the plurality of image data are associated with each other, and second teacher data in which the plurality of image data, recognition target character strings that are character strings to be recognized, and flags indicating whether or not the correct answer character strings corresponding to each of the plurality of image data match the recognition target character strings, are associated with each other; and a learning unit 132 that causes a task for outputting the character strings included in the image data to be learned by using the image data associated with the first teacher data as inputs, causes a task for determining whether the correct answer character strings corresponding to the image data and the recognition target character strings match to be learned by using the associated image data in the second teacher data and the recognition target character strings as inputs, and generates a trained model for outputting predicted character strings that are character strings predicted to be included in the image data using the image data as inputs.

Description

Information processing device, information processing method, and program

[0001] The present invention relates to an information processing device, an information processing method, and a program.

[0002] A method has been proposed for recognizing character strings contained in an image using a language model (see, for example, Patent Document 1).

[0003] Darwin Bautista and Rowel Atienza. Scene text recognition with permuted autoregressive sequence models. In ECCV, pages 178-196, 2022.

[0004] However, when recognizing a character string contained in a scene, it is necessary to recognize various fonts, colors, shapes, or backgrounds, and there are cases where the recognition accuracy does not improve with conventional techniques.

[0005] The present invention has been made in consideration of these points, and aims to improve the accuracy of recognizing character strings contained in images.

[0006] An information processing device according to a first aspect of the present invention includes: (1) first teacher data associating multiple image data pieces each including a character string with a correct string that is a character string included in each of the multiple image data pieces; and (2) second teacher data associating (a) the multiple image data pieces, (b) a target string that is a character string to be recognized, and (c) a flag indicating whether the target string matches the correct string corresponding to each of the multiple image data pieces; and a learning unit that generates a trained model that has been trained with (1) a string recognition task, which is a task of taking image data associated in the first teacher data as input and outputting a character string included in the image data based on the first teacher data; and (2) a true / false determination task, which is a task of taking image data associated in the second teacher data and the target string as input and determining whether the target string matches the correct string corresponding to the image data, based on the second teacher data.

[0007] The character string to be recognized may include either (1) a correct character string corresponding to image data associated with the character string to be recognized, or (2) a character string that is incorrectly recognized based on the image data.

[0008] The learning unit may train the character string recognition task and the correctness determination task in a single learning process to generate the trained model.

[0009] The trained model may take target image data, which is image data including a character string to be recognized, as input, and output a predicted character string and a correct / incorrect flag indicating whether the predicted character string is correct, and the acquisition unit may further acquire target image data, which is image data including the character string to be recognized, and the information processing device may further have an output unit that inputs the target image data acquired by the acquisition unit into the trained model and outputs a predicted character string for the target image data and a correct / incorrect flag for the predicted character string.

[0010] The system may further include a display control unit that, when the correct / incorrect flag output by the output unit indicates that the predicted character string is incorrect, displays a message indicating that the prediction result may be incorrect.

[0011] If the correct / incorrect flag output by the output unit indicates that the predicted character string is incorrect, the optical character recognition means may be configured to output characters contained in image data when image data is input, and may further include a recognition means control unit that inputs the target image data to the optical character recognition means that is different from the trained model, and may further include a display control unit that causes the optical character recognition means to obtain the recognized character string by optical character recognition of the target image data, and display the obtained character string.

[0012] The system may further include a generation unit that generates the second training data by associating a string output by the trained model in the process of the learning unit learning the string recognition task, which string is different from a string included in the input image data, with the image data input to the trained model as the target string to be recognized.

[0013] An information processing method according to a second aspect of the present invention includes a first acquisition step executed by a computer to acquire first teacher data associating multiple pieces of image data each containing a character string with a correct string that is a character string included in each of the multiple pieces of image data; a second acquisition step to acquire second teacher data associating (a) the multiple pieces of image data, (b) a target string that is a character string to be recognized, and (c) a flag indicating whether the target string matches the correct string corresponding to each of the multiple pieces of image data; and a learning step to generate a trained model that has been trained using (1) a string recognition task, which is a task of taking image data associated in the first teacher data as input and outputting a character string included in the image data, based on the first teacher data, and (2) a true / false determination task, which is a task of taking image data associated in the second teacher data and the target string as input and determining whether the target string matches the correct string corresponding to the image data, based on the second teacher data, and that takes image data associated in the second teacher data and the target string, based on the second teacher data, and that outputs a predicted string that is a character string predicted to be included in the image data.

[0014] In a third aspect of the program of the present invention, a computer is caused to execute a first acquisition step of acquiring first teacher data that associates multiple image data items each containing a character string with a correct string that is a character string included in each of the multiple image data items; a second acquisition step of acquiring second teacher data that associates (a) the multiple image data items, (b) a recognition target string that is a character string to be recognized, and (c) a flag indicating whether the correct string corresponding to each of the multiple image data matches the recognition target string; and a learning step of generating a trained model that has been trained based on (1) a string recognition task that is a task of taking image data associated in the first teacher data as input and outputting a character string included in the image data, and (2) a true / false determination task that is a task of taking image data associated in the second teacher data and the recognition target string as input and determining whether the correct string corresponding to the image data matches the recognition target string, the trained model receiving image data containing a character string as input and outputting a predicted character string that is a character string predicted to be included in the image data.

[0015] According to the present invention, it is expected that the accuracy of recognizing character strings contained in images can be improved.

[0016] FIG. 1 is a diagram for explaining an overview of an information processing system S according to an embodiment. FIG. 2 is a diagram showing an example of the data structure of first teacher data D1. FIG. 3 is a diagram showing an example of the data structure of second teacher data D2. FIG. 4 is a block diagram showing the configuration of an information processing device 1. FIG. 5 is a diagram showing an example of the processing of a learning unit 132. FIG. 6 is a diagram showing an example of the processing of a learning unit 132. FIG. 7 is a diagram showing an example of a screen displayed by a display control unit 134. FIG. 8 is a flowchart showing the flow of processing in the information processing device 1.

[0017] [Overview of Information Processing System S] Fig. 1 is a diagram illustrating an overview of an information processing system S according to an embodiment. The information processing system S is a system for providing OCR (Optical Character Recognition) that extracts characters included in an image. The information processing system S includes an information processing device 1 and an information terminal 2.

[0018] The information processing device 1 is an OCR device for recognizing character strings contained in an image. The information processing device 1 trains a trained model, which is a machine learning model for character recognition based on an image. The information processing device 1 also inputs image data to be inferred into the trained model and outputs characters contained in the image data.

[0019] The information terminal 2 is a terminal used by a user of the information processing system S. The information terminal 2 is, for example, a smartphone, a tablet, or a personal computer. The information terminal 2 transmits an instruction to start learning to the information processing device 1 in response to a user operation, and transmits training data to be used for learning. The information terminal 2 also transmits an image of a target for character string recognition, and acquires and displays the character string recognition result obtained by the information processing device 1.

[0020] An overview of the processing in the information processing device 1 will be described. The information processing device 1 trains a learning model on a string recognition task and a true / false judgment task to generate a trained model. The string recognition task is a task for predicting a string of characters contained in image data. The true / false judgment task is a task for predicting whether a string of characters contained in image data matches an input string, based on the image data and the string. The learning model is a pre-trained model that has been trained to be able to execute natural language processing tasks based on a large dataset.

[0021] The information processing device 1 acquires first teacher data D1 and second teacher data D2. The first teacher data D1 is teacher data mainly used for learning a string recognition task. FIG. 2 is a diagram showing an example of the data structure of the first teacher data D1. In the first teacher data D1, multiple image data D11 are associated with correct strings D12 corresponding to each of the multiple image data D11. A string of characters is captured in the image data. The correct string D12 indicates the correct string of characters contained in each of the multiple image data D11.

[0022] The second teacher data D2 is teacher data used for learning the true / false judgment task. FIG. 3 is a diagram showing an example of the data structure of the second teacher data D2. The second teacher data D2 is data in which a plurality of image data D21, a judgment target string D22, and a true / false flag D23 corresponding to each of the plurality of image data are associated with each other. The judgment target string D22 of the second teacher data D2 includes either (1) a correct answer string corresponding to the image data D21 associated with the judgment target string D22, or (2) a string that was incorrectly recognized based on the image data D21 in the learning process or the inference process. The true / false flag is a flag that indicates whether the correct answer string and the judgment target string match.

[0023] The image data (D11 and D12) in the first teacher data D1 and the second teacher data D2 are, for example, images that include a character string indicating a brand or the like attached to a product or the like. The correct character string in this example is the character string attached to the product or the like that appears in the image. The first teacher data D1 and the second teacher data D2 may be composed of common images, or may each include different images.

[0024] By learning in this way, the trained model takes image data containing a character string as input and outputs a predicted character string, which is a character string predicted to be contained in the image data.

[0025] Such a configuration of the information processing system S is expected to improve the accuracy of recognizing character strings contained in images. In particular, by having the information processing device 1 learn a true / false judgment task based on second teacher data D2 including character strings (character strings different from correct character strings) that have been misrecognized by the information processing device 1 during the learning or inference process and image data of the objects from which the character strings have been recognized, it is possible to learn the tendency of misrecognition, and it is expected that the accuracy of recognizing character strings will improve.

[0026] 4 is a block diagram showing the configuration of the information processing device 1. The information processing device 1 has a communication unit 11, a storage unit 12, and a control unit 13. The control unit 13 has an acquisition unit 131, a learning unit 132, an output unit 133, a display control unit 134, a recognition means control unit 135, and a generation unit 136.

[0027] The communication unit 11 is a communication interface for transmitting and receiving data to and from other devices via a network. The storage unit 12 is a storage medium including a read-only memory (ROM), a random access memory (RAM), a solid-state drive (SSD), a hard disk drive, etc. The storage unit 12 stores in advance a program to be executed by the control unit 13.

[0028] The control unit 13 is a processor such as a CPU (Central Processing Unit), etc. The control unit 13 executes the programs stored in the storage unit 12, thereby functioning as an acquisition unit 131, a learning unit 132, an output unit 133, a display control unit 134, a recognition means control unit 135, and a generation unit 136.

[0029] The acquisition unit 131 acquires the first teacher data D1 and the second teacher data D2. As an example, the acquisition unit 131 acquires the first teacher data D1 and the second teacher data D2 from the information terminal 2. The acquisition unit 131 may acquire the first teacher data D1 and the second teacher data D2 from an external device (not shown).

[0030] The learning unit 132 causes learning of a string recognition task based on the first teacher data D1. The learning unit 132 also causes learning of a true / false determination task based on the second teacher data D2. The processing of the learning unit 132 will be described with reference to FIG. 5 . The learning unit 132 receives image data D11 associated in the first teacher data D1 as input and outputs character strings included in the image data D11. As an example, in the learning model, the image data D11 is divided into multiple patches D31 and converted into a vector by taking the projection of each patch. As an example, the end of the input data includes data indicating the end of the input data ([SEP] in FIG. 5 ).

[0031] The learning unit 132 inputs the prediction result output based on the input data to the trained model in an autoregressive manner. The learning unit 132 concatenates a vector corresponding to the patch D31 with the prediction result D41 output by the training model based on the most recently input vector, inputs the concatenated vector to the training model, and outputs the prediction result D41. The training model repeats the process in an autoregressive manner until a vector indicating the end of prediction ([EOS] in FIG. 5 ) is output.

[0032] When the prediction is completed, the learning unit 132 updates the parameters of the learning model based on the difference between the character string output by the learning model and the correct character string D12 associated with it in the first teacher data D1. As an example, the learning unit 132 calculates the cross-entropy error between a vector indicating the character string output by the learning model and a vector corresponding to the correct character string D12, and updates the parameters of the learning model based on the calculated cross-entropy error.

[0033] Learning of the true / false judgment task will be described with reference to FIG. 6 . The learning unit 132 inputs the image data D21 and the character string D22 to be judged, which are associated in the second teacher data D2, into the learning model and outputs a true / false flag D42. The learning unit 132 converts the character string D22 to be judged into a vector and constructs input data D32 by concatenating the vector corresponding to the patch into which the image data D21 is divided with the vector corresponding to the character string D22 to be judged. In this case, the input data D32 includes data indicating a data separator ([SEP] in FIG. 6 ) between the end of the patch and the character string to be judged. The learning unit 132 updates the parameters of the learning model based on the difference between the true / false flag D42 output by the learning model and the true / false flag D23 associated in the second teacher data D2. The learning unit 132 updates the parameters of the learning model based on the cross-entropy error between the true / false flag D42 output by the learning model and the true / false flag D23 associated in the second teacher data D2.

[0034] By configuring the information processing device 1 in this way, it is possible to improve the accuracy of recognizing character strings contained in an image.

[0035] The learning unit 132 trains the string recognition task and the correctness determination task in a single learning process to generate a trained model. The learning unit 132 trains the learning model on both the string recognition task and the correctness determination task from the start to the end of learning. The learning unit 132 may train the string recognition task and the correctness determination task sequentially or in parallel.

[0036] By learning the predicted results of the strings output during the learning process as the strings to be judged in the second training data D2, it is possible to learn the tendency of the trained model to make incorrect recognitions.

[0037] The generation unit 136 generates second training data D2 in which the target string to be recognized is a string output by the trained model during the process in which the learning unit 132 trains the string recognition task, and which is different from a string contained in the input image data, and which is associated with the image data input to the trained model.

[0038] The information processing device 1 may be configured to determine whether the recognized characters are correct or incorrect in a single prediction process.

[0039] In this case, in addition to the above learning, the learning unit 132 causes the learning model to learn a true / false judgment task based on the first teacher data D1. In other words, the learning unit 132 inputs the character string output by the learning model in accordance with the input image data D11 into the learning model as a character string to be judged, and causes the learning model to learn the true / false judgment task.

[0040] The processing of the learning unit 132 in this case will be described with reference to FIG. 7 . In this case, the learning unit 132 inputs a vector corresponding to a patch D31 obtained by dividing the associated image data D11 in the first teacher data D1 into the learning model, and outputs a prediction result. The learning unit 132 autoregressively inputs the output prediction result into the learning model. When the learning model has finished predicting a character string, it outputs information indicating the end of the character string ([SEP] in FIG. 7 ). When the learning model receives the information indicating the end of the character string, it outputs a true / false flag based on the input patch and the predicted result of the character string. The learning unit 132 updates the parameters of the learning model based on the difference between the character string and true / false flag output by the learning model and the correct character string D12 associated with the image data D11 in the first teacher data and the true / false flag with a value of "TRUE" to generate a trained model. The trained model receives target image data, which is image data including a character string to be recognized, as input, and outputs a predicted character string and a true / false flag.

[0041] The acquisition unit 131 acquires target image data. The target image data is image data that is to be subjected to character recognition by the information processing device 1, and includes a character string to be recognized. The acquisition unit 131 may acquire the target image data from the information terminal 2. The acquisition unit 131 may also acquire the target image data from an external device (not shown).

[0042] The output unit 133 inputs the target image data acquired by the acquisition unit 131 into the trained model and outputs a predicted character string for the target image data and a true / false flag for the predicted character string. The display control unit 134 may display the predicted character string and the true / false flag output by the trained model on the information terminal 2.

[0043] If there is a possibility that the recognition result based on the trained model is incorrect, the information processing device 1 may be configured to display information to alert the user.

[0044] When the true / false flag output by the output unit 133 indicates that the predicted character string is incorrect, the display control unit 134 displays a message indicating that the prediction result may be incorrect. As an example, the display control unit 134 determines whether the true / false flag output by the trained model is "FALSE." When the true / false flag is "FALSE," the display control unit 134 displays the screen shown in Fig. 8 on the information terminal 2. The screen shown in Fig. 8 displays target image data, a predicted character string corresponding to the target image data, and a message indicating that the prediction result may be incorrect.

[0045] The information processing device 1 is configured to learn the tendency of erroneous recognition and output information indicating the possibility of erroneous recognition, thereby enabling the user to recognize that caution is required.

[0046] If there is a possibility that the recognition results of the trained model are incorrect, the information processing device 1 may be configured to input the results to another OCR.

[0047] If the correct / incorrect flag output by the output unit 133 indicates that the predicted character string is incorrect, the recognition means control unit 135 inputs the target image data to an optical character recognition means that outputs characters contained in image data when image data is input, and that is different from the trained model. As an example, if the correct / incorrect flag is "FALSE," the recognition means control unit 135 may input the target image data to an optical character recognition means (not shown). Furthermore, the storage unit 12 may store a character recognition model that is a trained model trained to perform character recognition based on a data set different from the trained model. In this case, the recognition means control unit 135 may input the target image data to the character recognition model and output the recognition result.

[0048] The display control unit 134 acquires a character string recognized by the optical character recognition means by optically recognizing the target image data, and displays the acquired character string. As an example, the display control unit 134 may acquire a character string as a recognition result from the optical character recognition means to which the recognition means control unit 135 inputs the target image data. Furthermore, when the recognition means control unit 135 inputs the target image data to a character recognition model, the display control unit 134 acquires the recognition result output by the character recognition model. The display control unit 134 displays the acquired character string and the character string output by the trained model on the information terminal 2.

[0049] In addition, when there are multiple input candidate OCRs, the trained model may be configured to further output OCR identification information for identifying the input destination OCR. In this case, the trained model is trained based on training data in which image data, a recognized character string, a correct / incorrect flag, and OCR identification information are associated. In this case, the OCR identification information included in the training data indicates an OCR that is suitable for character recognition of the image data.

[0050] By configuring the information processing device 1 in this way, it is possible to achieve the effect of character recognition using OCR that is more suitable for predicting character strings.

[0051] 9 is a flowchart showing the flow of processing in the information processing device 1. The flowchart shown in FIG. 9 starts when the information processing device 1 receives an instruction to start learning from the information terminal 2.

[0052] The acquisition unit 131 acquires first teacher data (S01). As an example, the acquisition unit 131 acquires the first teacher data from the information terminal 2. The acquisition unit 131 acquires second teacher data (S02). As an example, the acquisition unit 131 acquires the second teacher data from the information terminal 2.

[0053] The learning unit 132 causes the character recognition task to be learned based on the first training data (S03). The learning unit 132 updates the parameters of the learning model based on the difference between the output of the character recognition task and the correct character string D12 associated in the first training data.

[0054] The learning unit 132 causes the model to learn the true / false determination task based on the second teacher data (S04). The learning unit 132 updates the parameters of the learning model based on the difference between the output of the true / false determination task and the true / false flag associated with the second teacher data D2.

[0055] The learning unit 132 causes the model to learn a true / false determination task based on the first teacher data (S05). The learning unit 132 updates the parameters of the learning model based on the difference between the character string and true / false flag output in the true / false determination task and the correct character string D12 associated with the first teacher data D1 and the true / false flag with a value of “TRUE.”

[0056] The learning unit 132 updates the parameters to generate a trained model, which is a trained model for which training has been completed, and stores the trained model in the storage unit 12 (S06). The information processing device 1 then ends the processing.

[0057] [Effects of Information Processing Device 1] By configuring the information processing device 1 in this way, it is expected that the accuracy of recognizing character strings included in an image can be improved.

[0058] The present invention has been described above using embodiments, but the technical scope of the present invention is not limited to the scope described in the above embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. For example, all or part of the device can be configured by functionally or physically distributing or integrating in any unit. Furthermore, new embodiments resulting from any combination of multiple embodiments are also included in the embodiments of the present invention. The effects of the new embodiments resulting from the combination also have the effects of the original embodiments.

[0059] REFERENCE SIGNS LIST 1 Information processing device 2 Information terminal 11 Communication unit 12 Storage unit 13 Control unit 131 Acquisition unit 132 Learning unit 133 Output unit 134 Display control unit 135 Recognition means control unit 136 Generation unit

Claims

1. An information processing device having: (1) an acquisition unit that acquires first teacher data that associates multiple image data including character strings with correct strings that are character strings included in each of the multiple image data; and (2) second teacher data that associates (a) the multiple image data, (b) target strings that are character strings to be recognized, and (c) flags that indicate whether the target strings match the correct strings corresponding to each of the multiple image data; and a learning unit that generates a trained model that has learned the following: (1) a string recognition task that is a task of taking image data associated in the first teacher data as input and outputting a string included in the image data based on the first teacher data; and (2) an accuracy determination task that is a task of taking image data associated in the second teacher data and the target strings as input and determining whether the target strings match the correct strings corresponding to the image data based on the second teacher data, the trained model receiving image data including character strings as input and outputting a predicted string that is a string predicted to be included in the image data.

2. The information processing device according to claim 1, wherein the character string to be recognized includes either: (1) a correct character string corresponding to image data associated with the character string to be recognized; or (2) a character string that is incorrectly recognized based on the image data.

3. The information processing device according to claim 1, wherein the learning unit trains the character string recognition task and the correctness determination task in a single learning process to generate the trained model.

4. The information processing device according to claim 1, wherein the trained model receives target image data, which is image data including a character string to be recognized, as input, and outputs a predicted character string and a correct / incorrect flag indicating whether the predicted character string is correct; the acquisition unit further acquires target image data, which is image data including a character string to be recognized; and the information processing device further has an output unit that inputs the target image data acquired by the acquisition unit into the trained model and outputs a predicted character string for the target image data and a correct / incorrect flag for the predicted character string.

5. The information processing device according to claim 4, further comprising a display control unit that, when the correct / incorrect flag output by the output unit indicates that the predicted character string is incorrect, displays a message indicating that the prediction result may be incorrect.

6. The information processing device according to claim 4, further comprising: an optical character recognition means that outputs characters contained in image data when image data is input, when the correct / incorrect flag output by the output unit indicates that the predicted character string is incorrect; a recognition means control unit that inputs the target image data to the optical character recognition means that is different from the trained model; and a display control unit that acquires the recognized character string by the optical character recognition means performing optical character recognition on the target image data, and displays the acquired character string.

7. The information processing device according to claim 2, further comprising a generation unit that generates the second training data by associating a character string output by the trained model in the process of the learning unit learning the character string recognition task, which character string is different from a character string contained in the input image data, with the image data input to the trained model as the recognition target character string.

8. An information processing method executed by a computer, comprising: a first acquisition step of acquiring first teacher data that associates multiple image data including character strings with correct strings that are character strings included in each of the multiple image data; a second acquisition step of acquiring second teacher data that associates (a) the multiple image data, (b) recognition target strings that are character strings to be recognized, and (c) flags that indicate whether the correct strings corresponding to each of the multiple image data match the recognition target strings; and a learning step of generating a trained model that has learned the above: (1) a string recognition task that is a task of taking image data associated in the first teacher data as input and outputting a string included in the image data, based on the first teacher data; and (2) a true / false determination task that is a task of taking image data associated in the second teacher data and the recognition target string as input and determining whether the correct string corresponding to the image data matches the recognition target string, based on the second teacher data, and the trained model that receives image data including a character string as input and outputs a predicted string that is a string predicted to be included in the image data.

9. A program causing a computer to execute the following steps: a first acquisition step of acquiring first teacher data associating multiple image data containing character strings with correct strings that are character strings contained in each of the multiple image data; a second acquisition step of acquiring second teacher data associating (a) the multiple image data, (b) target strings that are character strings to be recognized, and (c) flags indicating whether the correct strings corresponding to each of the multiple image data match the target strings; and a learning step of generating a trained model that has been trained based on the following: (1) a string recognition task that is a task of taking image data associated in the first teacher data as input and outputting a string included in the image data, and (2) a true / false determination task that is a task of taking image data associated in the second teacher data and the target string as input and determining whether the target string matches the correct string corresponding to the image data, the trained model receiving image data containing character strings as input and outputting a predicted string that is a string predicted to be contained in the image data.

Images