Learning data expansion device and learning data expansion method

Abstract

To provide a learning data expansion device or the like capable of enhancing performance of an error correction model of handwritten character recognition.SOLUTION: The learning data expansion device includes a learning data generation unit that generates learning data by recognizing a character string from a handwritten character string pattern using a character string recognition model, a single character recognition unit that recognizes each character in the character string from the character string pattern using a single character recognition model, and a learning data expansion unit that expands the learning data by replacing each character of the character string recognized by the character string recognition model with any candidate character at a probability based on a score of each candidate character in a recognition result of the single character recognition model.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a program, a training data extension device, and a training data extension method. [Background technology]

[0002] The following models focus on error correction in Japanese character recognition. Non-Patent Document 1 proposes a method of creating character error correction candidates from character trigrams and then narrowing them down using part-of-speech ngrams and a statistical language model. Other proposed methods include a method of correcting kanji recognition errors using the DL distance of kanji as the edit distance (Non-Patent Document 2), a method of correcting OCR errors using character edit distance using a hill-climbing method (Non-Patent Document 3), and a method of using a large-scale language model called BERT for character recognition error correction (Non-Patent Document 4).

[0003] There are various studies on error correction models using T5, including error correction models for speech recognition (Non-Patent Document 5), error recognition models for program code, and grammatical error correction (Non-Patent Documents 6-9). Non-Patent Document 10 uses T5 to improve the OCR accuracy of Japanese documentary evidence. In all cases, the input is data containing errors, and the output is data after error correction. However, these studies do not include data expansion.

[0004] There are many studies on data augmentation: Non-Patent Document 11 generates data for scene text recognition, Non-Patent Document 12 generates images of handwritten characters, and Non-Patent Document 13 uses open-source 3D computer graphics software. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Takeuchi, A., Matsumoto, Y., "Construction of an OCR error correction system using statistical language models," Transactions of the Information Processing Society of Japan, 40:6:2679-2689 (1999) [Non-patent document 2] Kotaro Sakamoto, Akimasa Abegawa, Maki Satake, Shihaku Kishikawa, Eliza Sakamoto, Madoka Ishishita, Hideki Shibuki, and Tatsunori Mori, "Tree edit distance considering kanji radicals, crowns, and footnotes for word error correction in contract OCR," Proceedings of the 26th Annual Conference of the Association for Natural Language Processing, pp. 137-140 (2020) [Non-patent document 3] Nguyen, QD, Phan, NM, Kromer, P., Le, DA: An efficient unsupervised approach for OCR error correction of vietnamese OCR text. IEEE Access 11, 58406-58421 (2023). https: / / doi.org / 10.1109 / ACCESS.2023.3283340 [Non-patent document 4] Xie Suchun and Matsumoto Akiyo, "Error Correction in Modern Sentences Using the Japanese BERT Model," Proceedings of the 29th Annual Conference of the Association for Natural Language Processing, pp. 1616-1620 (2023) [Non-patent document 5] Nakamura, Asahi, Li, Shengmin, Tamura, Koki, Yoshinaga, Naoki, "Neural speech recognition error correction considering previous and following utterances as context," Research Report Spoken Language Processing (SLP), 2022-SLP-144(18), 1-6, 2188-8663 (2022) [Non-patent document 6] Shahgir, KS, Sayeed, KS: Bangla grammatical error detection using T5 transformer model. arXiv:2303.10612 [cs.CL] (2023) [Non-Patent Document 7] Katinskaia, A., Yangarber, R.: Grammatical error correction for sentence-level assessment in language learning. In: Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications. pp. 488-502. Association for Computational Linguistics (2023). https: / / doi.org / 10.18653 / v1 / 2023.bea-1.41, https: / / aclanthology.org / 2023.bea-1.41 [Non-Patent Document 8] Zhang, Y., Kamigaito, H., Okumura, M.: Bidirectional transformer reranker for grammatical error correction. In: Findings of the Association for Computational Linguistics: ACL 2023. pp. 3801-3825. Association for Computational Linguistics, Toronto, Canada (Jul 2023). https: / / doi.org / 10.18653 / v1 / 2023.findings-acl.234, https: / / aclanthology.org / 2023.findings-acl.234 [Non-Patent Document 9] Qorib, MR, Ng, HT: Grammatical error correction: Are we there yet? In: Proceedings of the 29th International Conference on Computational Linguistics. pp. 2794-2800. International Committee on Computational Linguistics, Gyeongju, Republic of Korea (Oct 2022), https: / / aclanthology.org / 2022.coling-1.246 [Non-Patent Document 10] Masato Fujitake, "Constructing a Japanese OCR error correction benchmark using evidence," Proceedings of the 30th Annual Conference of the Association for Natural Language Processing, pp.2373-2377(2024) [Non-Patent Document 11] Atienza, R.: Data augmentation for scene text recognition. In: 2021 IEEE / CVF International Conference on Computer Vision Workshops (ICCVW). pp. 1561-1570 (2021). https: / / doi.org / 10.1109 / ICCVW54120.2021.0018 [Non-Patent Document 12] Hayashi, T., Gyohten, K., Ohki, H., Takami, T.: A study of data augmentation for handwritten character recognition using deep learning. In: 2018 16th International Conference on Frontiers in Handwriting Recognition (ICFHR). pp. 552-557 (2018). https: / / doi.org / 10.1109 / ICFHR-2018.2018.00102 [Non-Patent Document 13] Spruck, A., Hawesch, M., Maier, A., Riess, C., Seiler, J., Kaup, A.: 3d rendering framework for data augmentation in optical character recognition. arXiv:2209.14970 (2022) Summary of the Invention [Problem to be solved by the invention]

[0006] Simply using BERT or T5 did not provide very good performance in character recognition error correction. Also, since data augmentation technology augments the data for each "desired task," other methods are generally ineffective.

[0007] The present invention has been made in consideration of the above-mentioned problems, and its purpose is to provide a program, a training data extension device, and a training data extension method that can improve the performance of an error correction model for handwritten character recognition. [Means for solving the problem]

[0008] (1) The present invention relates to a program for expanding training data for training an error correction model for handwritten character recognition, the program causing a computer to function as a training data generation unit that recognizes a character string from a handwritten input character string pattern using a character string recognition model to generate training data, a single-character recognition unit that recognizes each character in the character string from the character string pattern using a single-character recognition model, and a training data expansion unit that expands the training data by replacing each character in the character string recognized by the character string recognition model with one of the candidate characters at a probability based on the score of each candidate character in the recognition result of the single-character recognition model. The present invention also relates to a computer-readable information storage medium that stores a program for causing a computer to function as each of the above units. The present invention also relates to a training data expansion device including each of the above units.

[0009] The present invention is a training data extension method for extending training data for training an error correction model for handwritten character recognition, comprising: a training data generation step of generating training data by recognizing a character string from a handwritten input character string pattern using a character string recognition model; a single character recognition step of recognizing each character in the character string from the character string pattern using a single character recognition model; and a step of replacing each character in the character string recognized by the character string recognition model with one of the candidate characters with a probability based on the score of each candidate character in the recognition result of the single character recognition model. and a learning data expansion step of expanding the learning data by performing the above-mentioned process.

[0010] (2) In the program, information storage medium, and training data extension device according to the present invention, the training data extension unit may normalize the score of each candidate character in the recognition result of the single-character recognition model to convert it into a probability value, and perform the replacement with a probability obtained by multiplying the probability value by a coefficient less than 1.

[0011] In addition, in the training data extension method according to the present invention, in the training data extension step, the score of each candidate character in the recognition result of the single-character recognition model may be normalized and converted into a probability value, and the replacement may be performed with a probability obtained by multiplying the probability value by a coefficient less than 1. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 2 is a diagram showing an example of a functional block diagram of the training data expansion device according to the present embodiment. [Figure 2] 1A and 1B are diagrams showing an example of a handwritten character string pattern image, a character string recognition result, and a character string manually converted into text. [Figure 3] FIG. 10 is a diagram showing an example of expanding learning data. [Figure 4] 10 is a flowchart showing the flow of processing of the data extension device of the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] The present embodiment will be described below. Note that the present embodiment described below does not unduly limit the content of the present invention described in the claims. Furthermore, not all of the configurations described in the present embodiment are necessarily essential constituent elements of the present invention.

[0014] 1. Configuration An example of a functional block diagram of the training data extension device of this embodiment is shown in Fig. 1. Note that the training data extension device of this embodiment may have a configuration in which some of the components (units) in Fig. 1 are omitted.

[0015] The input unit 110 is used to input a character string pattern written on a writing surface, and its function can be realized by an optical reading device (scanner, camera, etc.) that reads the character string pattern written on paper or the like as an image (black and white image or grayscale image).

[0016] The storage unit 120 stores programs and various data for causing the computer to function as each unit of the processing unit 100, and also functions as a work area for the processing unit 100, the function of which can be realized by a hard disk, RAM, etc.

[0017] The display unit 130 outputs the image generated by the processing unit 100, and its function can be realized by a display such as a touch panel or LCD that also functions as the input unit 110.

[0018] The processing unit 100 performs processes such as character recognition and display control based on data (image data) from the input unit 110, programs, etc. The processing unit 100 performs various processes using the main memory unit in the memory unit 120 as a work area. The functions of the processing unit 100 can be realized by hardware such as various processors (CPU, DSP, etc.) or programs. The processing unit 100 includes a training data generation unit 101, a single character recognition unit 102, and a training data extension unit 103.

[0019] The training data generation unit 101 generates training data by recognizing character strings from an input character string pattern using a character string recognition model (handwritten character string recognition engine). The character string recognition model outputs multiple candidate character strings and their respective scores (likelihoods) as recognition results, and the multiple candidate character strings are used as training data.

[0020] The single character recognition unit 102 recognizes each character in a character string from an input character string pattern using a single character recognition model (handwritten single character recognition engine). The single character recognition model outputs, as a recognition result, multiple candidate characters corresponding to each character in the character string and their respective scores (likelihoods).

[0021] The training data extension unit 103 extends the training data by replacing each character of a string recognized by the string recognition model (for example, the candidate string with the highest score) with one of the candidate characters with a probability based on the score of each candidate character in the recognition result of the single-character recognition model (generating a new string from the string recognized by the string recognition model). The training data extension unit 103 may normalize the score of each candidate character in the recognition result of the single-character recognition model to convert it into a probability value, and perform the replacement with a probability obtained by multiplying the probability value by a coefficient less than 1. The training data generated by the training data generation unit 101 and extended by the training data extension unit 103 is stored in the storage unit 120.

[0022] 2. Method of this embodiment The training data expansion device of this embodiment is configured to expand training data for training a natural language processing model (error correction model) that corrects errors in handwritten character recognition. Using a dataset in which the recognition results (text containing character recognition errors) obtained by recognizing a character string from an image of a handwritten character string pattern (e.g., an image of an answer sheet) as input and the character string (text containing no character recognition errors) obtained by manually converting the image into text as output, as shown in Fig. 2, the error correction model is trained so that when a character string containing a character recognition error is input, a character string in which the character recognition error has been corrected is output. In this embodiment, the performance of the error correction model (machine learning accuracy) is improved by synthesizing and increasing (expanding) the character strings containing character recognition errors that are to be input to the correction model from this dataset.

[0023] FIG. 3 is a diagram showing an example of expanding learning data. In this example, from a character string pattern in which "vegetables grown by a person in a field" is written, the character string "a person GA in a field FUru vegetables" is recognized by a character string recognition model. There is an error in this recognition result. Further, as a recognition result obtained by recognizing the character string pattern by a single-character recognition model, candidate characters and scores up to the top 5 corresponding to each character of "vegetables grown by a person in a field" are shown. In this example, the top 5 candidate characters corresponding to the first character "person" are "person" (score: 1.0), "I" (score: 0.41), "HA" (score: 0.40), "eight" (score: 0.40), "TO" (score: 0.39). Further, the top 5 candidate characters corresponding to the second character "GA" are "GA" (score: 0.97), "ga" (score: 0.96), "bu" (score: 0.95), "become" (score: 0.93), "contend" (score: 0.89), and the top 5 candidate characters corresponding to the third character "field" are "field" (score: 1.0), "seedling" (score: 0.73), "cook" (score: 0.72), "flame" (score: 0.71), "warrior" (score: 0.70).

[0024] Then, by replacing each character of the character string "a person GA in a field FUru vegetables" recognized by the character string recognition model with any candidate character with a probability based on the score of each candidate character in the recognition result of the single-character recognition model, a new character string is generated. Specifically, the scores of the 5 candidate characters corresponding to each character are normalized so that their sum becomes 1 (100% in percentage) and converted into probability values, and then the replacement is performed with a probability obtained by multiplying the probability value by 1 / 100 (an example of a coefficient less than 1). In the example of FIG. 3, the first character "person" of the character string "a person GA in a field FUru vegetables" is replaced with "person" with a probability of 0.31%, with "I" with a probability of 0.17%, with "HA" with a probability of 0.17%, with "eight" with a probability of 0.17%, and with "TO" with a probability of 0.17%. Similarly, the second character "GA" is replaced with "GA" with a probability of 0.21%, with "ga" with a probability of 0.2%, with "bu" with a probability of 0.2%, with "become" with a probability of 0.2%, and with "contend" with a probability of 0.19%. It can be replaced. For the third character "field", it can be replaced with "field" with a probability of 0.25%, "seedling" with a probability of 0.19%, "cook" with a probability of 0.19%, "flame" with a probability of 0.19%, and "soldier" with a probability of 0.19%. Similarly, replacements are made for the 4th to 9th characters below. Note that in order to prevent a string that is significantly different from the recognition result of the string recognition model from being generated due to the high probability of replacement of each character of the string recognized by the string recognition model, replacement is performed with a probability obtained by multiplying the probability value of the score of the candidate character by a coefficient less than 1. In the above example, since the coefficient is 1 / 100, each character of the string recognized by the string recognition model is replaced with one of the candidate characters with a probability of 1%, and no replacement occurs with a probability of 99%. This coefficient less than 1 can be, for example, 1 / 10 instead of 1 / 100, and can be adjusted so as to obtain a desired result according to the string to be applied, the string recognition model, or the single-character recognition model.

[0025] In the example of FIG. 3, the second character "GA" of the string "A person GA field and grows vegetables" recognized by the string recognition model is replaced with "ga", the third character "field" is replaced with "seedling", and the ninth character "vegetable" is replaced with "leaf", resulting in a new string "A person grows seedlings and grows wild leaves". In this way, as examples of character recognition errors in "A person grows vegetables in the field", various error examples such as "A person GA field and grows vegetables" and "A person grows seedlings and grows wild leaves" can be prepared. Note that there is a possibility that the result may not change (no replacement occurs at all) or the character recognition error may be corrected due to the probabilistic replacement of characters, but this is not regarded as a problem.

[0026] 3. Processing 4 is a flowchart showing the processing flow of the data expansion device of this embodiment. First, the training data generation unit 101 recognizes a character string from an input handwritten character string pattern using a character string recognition model (step S10), and the single character recognition unit 102 recognizes each character in the character string from the handwritten character string pattern using the single character recognition model (step S11). Next, the training data expansion unit 103 normalizes the score of each candidate character corresponding to each character in the recognition result of the single character recognition model, converts it into a probability value, and further multiplies it by a coefficient less than 1 to obtain a probability (step S12).

[0027] Next, the training data extension unit 103 sets the variable n to 1 (step S13), sets the variable i to 1 (step S14), and replaces the i-th character of the character string recognized by the character string recognition model (the top character string candidate) with one of the candidate characters based on the probability of each candidate character corresponding to the i-th character in the recognition result of the single-character recognition model (step S15). Next, the training data extension unit 103 determines whether the variable i has reached L (step S16). L is the number of characters (recognized by the character string recognition model and the single-character recognition model) included in the handwritten character string pattern. If the variable i has not reached L (N in step S16), the variable i is incremented by 1 (step S17), and the process proceeds to step S15. If the variable i has reached L (Y in step S16), the training data extension unit 103 determines whether the variable n has reached N (step S18). N is the number of character strings to be newly generated (data extended) by replacing characters. If the variable n has not reached N (N in step S18), 1 is added to the variable n (step S19), and the process proceeds to step S14. Thereafter, the 1st to Lth characters of the character string recognized by the character string recognition model are stochastically replaced N times to generate N new character strings.

[0028] Next, the training data extension unit 103 stores the character strings recognized by the character string recognition model (the top m character string candidates) and the N character strings generated by character replacement in the storage unit 120 as training data (training data to be used as input to the error correction model) (step S20).

[0029] 4. Evaluation Experiment An experiment was conducted to evaluate the data expansion method of this embodiment. In this experiment, retrieva-jp / t5-base-medium, a pre-trained Japanese T5 (Text-to-Text Transfer Transformer) model, was used as the error correction model. The T5 tokenizer was used and the retrieva-jp / t5-base-medium tokenizer was specified. As the handwritten character recognition engine, a vertical writing character recognition model (an example of a character string recognition model) and a single character recognition model from iLab Co., Ltd. were used.

[0030] For the handwritten character string patterns, 25 descriptive answers were extracted from the image data of answers in "10-Minute Review Drills: Japanese Language Reading Comprehension" published by Juken Kenkyusha, which were completed by approximately 50 junior high school students. All answers were 60 characters or less. These answer image data were recognized using a vertical writing character recognition model, and the recognition results (top five candidate character strings) were used as input. The answers were manually converted to text while looking at the answer image data, and a dataset was created as the output, which served as training data (without data expansion). This dataset had 6,656 entries. This training data was expanded using the recognition results of the vertical writing recognition model and the single character recognition model. Specifically, each character in the top candidate character string recognized by the vertical writing recognition model was replaced five times with one of the candidate characters based on the probability based on the scores of the top five candidate characters corresponding to each character in the recognition results of the single-character recognition model (the scores were converted into probability values ​​using the softmax function, and the probability value was multiplied by 1 / 100), generating five new character strings (pseudo character recognition results).The training data (with data augmentation) including the data generated using this method totaled 11,271 items.

[0031] As an index for evaluating error correction, we calculated the BLEU value using data obtained by manually converting answer sheets into text as a reference translation. To calculate the BLEU value, we used sacreBLEU from the Evaluate library. The BLEU value is expressed as a real value between 0 and 100, and the closer the generated sentence is to the reference translation, the higher the value. We compared the BLEU value of the character recognition results with the BLEU value of the correction results made by the error correction model, and if the BLEU value of the correction results made by the error correction model exceeds the BLEU value of the character recognition results, we determined that the error correction model was effective.

[0032] In the first experiment, an error correction model was created by fine-tuning the T5 model using a training data set (without data augmentation) that used the recognition results of a vertically written character recognition model as input examples. In the second experiment, an error correction model was created by fine-tuning the T5 model using a training data set (with data augmentation) that used the recognition results of a vertically written character recognition model as input examples, as well as pseudo-character recognition results. In both experiments, a grid search was performed using pre-specified combinations of hyperparameters to find parameters that would result in a higher BLEU value. A five-fold cross-validation was performed to verify accuracy. The data ratio was (training:validation:test) = (3:1:1). The dataset for cross-validation was divided into five parts by problem. In this method, the training, validation, and test data were divided for each drill problem. This allowed us to measure the ability to correct character recognition errors in answers to problems not used in training. During the experiment, the fine-tuning learning rate was set to three values: "0.00001", "0.0001", and "0.001", the number of epochs was set to three values: "5", "10", and "15", and the Repetition Penalty, which is the penalty given for repetitions during generation, was set to two values: "5.0" and "10.0". Experiments were conducted using 18 combinations of these settings.

[0033] Table 1 shows the BLEU values ​​before character recognition error correction (character recognition results) and the highest BLEU values ​​after character recognition error correction (first experiment, second experiment). The optimal parameters in the first experiment were a learning rate of 0.0001, number of epochs of 15, and repetition penalty of 5.0. The optimal parameters in the second experiment were a learning rate of 0.0001, number of epochs of 15, and repetition penalty of 10.0.

[0034] [Table 1] Table 1 shows that the BLEU value after character recognition error correction in the first experiment is higher than the BLEU value before character recognition error correction. This indicates that correction using the error correction model is effective. Furthermore, the BLEU value in the second experiment, in which data augmentation was performed, exceeds the BLEU value in the first experiment, in which data augmentation was not performed, indicating that data augmentation using the method of this embodiment is effective.

[0035] The present invention is not limited to the above-described embodiments, and various modifications are possible. The present invention includes configurations that are substantially the same as those described in the embodiments (for example, configurations with the same functions, methods, and results, or configurations with the same purpose and effects). The present invention also includes configurations in which non-essential parts of the configurations described in the embodiments are replaced. The present invention also includes configurations that achieve the same effects as the configurations described in the embodiments, or that can achieve the same purpose. The present invention also includes configurations in which publicly known technology is added to the configurations described in the embodiments. [Explanation of symbols]

[0036] 100... processing unit, 101... learning data generation unit, 102... single character recognition unit, 103... learning data extension unit, 110... input unit, 120... storage unit, 130... display unit

Claims

1. A program for expanding training data for training an error correction model for handwritten character recognition, comprising: a learning data generation unit that generates learning data by recognizing character strings from handwritten input character string patterns using a character string recognition model; a single character recognition unit that recognizes each character in the character string from the character string pattern using a single character recognition model; A program that causes a computer to function as a training data expansion unit that expands training data by replacing each character of a string of characters recognized by the character string recognition model with one of the candidate characters with a probability based on the score of each candidate character in the recognition result of the single-character recognition model.

2. In claim 1, The learning data extension unit a program for normalizing the score of each candidate character in the recognition result of the single character recognition model to convert it into a probability value, and performing the replacement with a probability obtained by multiplying the probability value by a coefficient less than 1;

3. A training data expansion device that expands training data for training an error correction model for handwritten character recognition, comprising: a learning data generation unit that generates learning data by recognizing character strings from handwritten input character string patterns using a character string recognition model; a single character recognition unit that recognizes each character in the character string from the character string pattern using a single character recognition model; and a training data expansion unit that expands the training data by replacing each character of the string recognized by the character string recognition model with one of the candidate characters with a probability based on the score of each candidate character in the recognition result of the single-character recognition model.

4. 1. A training data extension method for extending training data for training an error correction model for handwritten character recognition, comprising: a learning data generation step of generating learning data by recognizing character strings from handwritten input character string patterns using a character string recognition model; a single character recognition step of recognizing each character in the character string from the character string pattern using a single character recognition model; and a training data expansion step of expanding the training data by replacing each character of the string recognized by the character string recognition model with one of the candidate characters with a probability based on the score of each candidate character in the recognition result of the single-character recognition model.

Images