Learning device, estimation device, learning method, and program

Abstract

A learning device according to the present invention includes a prior learning unit and a fine tuning unit. The prior learning unit acquires an auxiliary language data set, which is a data set of paired data of an image and text in a language which is not a target language (hereinafter, the auxiliary language), and a target language data set, which is a data set of paired data of an image and text in the target language; trains an intermediate language information recognition model; and outputs a parameter of the intermediate language information recognition model as a trained parameter. The fine tuning unit acquires the target language data set and the trained parameter, and trains a language information recognition model using the trained parameter as an initial value for the language information recognition model.

Description

[Technical field]

[0001] The present invention relates to a learning device, an estimation device, a learning method, and a program. [Background technology]

[0002] Scenery images obtained by photographing a landscape contain a lot of text information, such as traffic signs and advertising billboards, that is necessary for understanding the image. Scenery character recognition is a task that takes an image in which a character region is extracted from such a scene image as input, recognizes the characters in the image, and converts them into a string of characters that can be processed by a machine. In recent years, with the advancement of deep learning technology, a method has been proposed to realize scene character recognition using a one-stop model.

[0003] For example, Non-Patent Document 1 and Non-Patent Document 2 provide a scene character recognition technology using a model consisting of an encoder and a decoder as shown in FIG. 1. In this case, the encoder is composed of, for example, a function of extracting image features using a convolutional neural network and a function of converting the image features into features taking into account the sequence using a Transformer encoder provided in Reference Non-Patent Document 1 (Reference Non-Patent Document 1: A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, AN Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in Neural Information Processing Systems (NIPS), pp. 5998-6008, 2017.). The decoder is composed of, for example, a Transformer decoder provided in Reference Non-Patent Document 1, and an autoregressive model using an embedding layer and an output layer, and outputs the probability of generating a character string from the features extracted by the encoder. In learning, a large amount of image and text pair data in the target language is used as training data, and the parameters of the encoder and decoder can be optimized by, for example, backpropagation. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] F. Sheng, Z. Chen, and B. Xu, “NRTR: A no-recurrence sequence-to-sequence model for scene text recognition,” Proceedings of the IEEE International Conference on Document Analysis and Recognition (ICDAR), pp. 781-786, 2019. [Non-Patent Document 2] P. Wang, L. Yang, H. Li, Y. Deng, C. Shen, and Y. Zhang, “A simple and robust convolutional-attention network for irregular text recognition,” arXiv preprint, arXiv:1904.01375, 2019. Summary of the Invention [Problem to be solved by the invention]

[0005] When training a scene character recognition model such as that realized by Non-Patent Document 1 or Non-Patent Document 2, a large amount of paired image and text data in the target language is required as training data. For example, when the target language is a language with many speakers and abundant resources such as English, a large amount of available paired data is provided, so collection of paired data is relatively easy. However, when the target language is a language with few speakers such as Japanese, there is a problem that there is little provision of easily available paired data. In particular, a large amount of training data is required for training a complex model such as Transformer shown in Non-Patent Document 1, but it is difficult to collect a large amount of paired data in a language with few resources.

[0006] Therefore, an object of the present invention is to provide a learning device that can learn a highly accurate scene character recognition model even when there is a small amount of pair data in a target language. [Means for solving the problem]

[0007] The learning device of the present invention includes a pre-learning unit and a fine tuning unit.

[0008] The pre-training unit acquires an auxiliary language dataset, which is a dataset of image and text pair data in a language other than the target language (hereinafter referred to as the auxiliary language), and a target language dataset, which is a dataset of image and text pair data in the target language, and Scene character recognition model Learning, A model for scene character recognition The fine-tuning unit obtains the target language dataset and the trained parameters, and outputs the trained parameters. The scene character recognition model The initial value is , scene character recognition model Learn. Effect of the Invention

[0009] According to the learning device of the present invention, even if there is a small amount of pair data in the target language, it is possible to learn a highly accurate scene character recognition model. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram showing an example of a character recognition model according to a conventional technique. [Diagram 2] FIG. 2 is a block diagram showing the functional configuration of the learning device according to the first embodiment. [Diagram 3] 4 is a flowchart showing the operation of the learning device of the first embodiment. [Figure 4] FIG. 1 is a block diagram showing a configuration of an estimation device according to a first embodiment. [Diagram 5] FIG. 11 is a block diagram showing the functional configuration of a learning device according to a second embodiment. [Figure 6] 10 is a flowchart showing the operation of the learning device of the second embodiment. [Figure 7]FIG. 11 is a block diagram showing the configuration of an estimation device according to a second embodiment. [Figure 8] FIG. 11 is a block diagram showing the functional configuration of a learning device according to a third embodiment. [Figure 9] 11 is a flowchart showing the operation of the learning device of the third embodiment. [Figure 10] FIG. 11 is a block diagram showing the configuration of an estimation device according to a third embodiment. [Figure 11] FIG. 13 is a diagram showing the results of a verification experiment. [Figure 12] FIG. 11 is a diagram showing an example of a recognition result. [Figure 13] FIG. 2 is a diagram showing an example of the functional configuration of a computer. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] In the following embodiment, an application to scene character recognition, such as that described in Non-Patent Document 1 and Non-Patent Document 2, in which an image in which a character is captured is input and a corresponding character string is output, is described as an example, but the present invention is not limited to scene character recognition. In other words, the present invention is applicable to all technologies that output a character string from an image through an arbitrary encoder-decoder type all-through model. For example, the present invention is also applicable to optical character recognition, caption generation for still images and video images, lip reading, and the like. Therefore, the model handled by the present invention is a model (language information recognition model) that handles all tasks of recognizing language information from images. Specific examples of language information recognition models include, for example, a character recognition model that recognizes images in which characters are displayed, such as optical character recognition, a caption generation model that generates text corresponding to the contents of still images and video images, a lip reading model that estimates spoken text from a video image in which the mouth is cut out, and the above-mentioned scene character recognition model. In the following embodiment, a detailed explanation is given focusing on the scene character recognition model.

[0012] In the following embodiment, in order to realize character recognition with high accuracy even when the number of data sets of paired data in the target language (hereinafter, the target language data set) is small, a data set of paired data in a language other than the target language (hereinafter, the auxiliary language) (hereinafter, the auxiliary language data set) is utilized. Here, the auxiliary language data set is not simply mixed with the target language data set, but is used for pre-training. In particular, the accuracy of scene character recognition in the target language is improved by pre-training an encoder responsible for extracting features from images with a large-scale data set consisting of the target language data set and the auxiliary language data set, and pre-training a decoder responsible for outputting character strings with the target language data set. EXAMPLES

[0013] The functional configuration of the learning device of the first embodiment will be described below with reference to Fig. 2. As shown in the figure, the learning device 1 of this embodiment includes a pre-learning unit 11 and a fine-tuning unit 12. An auxiliary language data set storage unit 91, a target language data set storage unit 92, and a scene character recognition model storage unit 93 shown in the figure may be inside the learning device 1 or inside another device. Fig. 3 shows the operation of the learning device 1 of this embodiment. The operation of the learning device 1 of this embodiment includes a pre-learning process and a fine-tuning process.

[0014] <Pre-learning process> The pre-learning unit 11 acquires an auxiliary language dataset from the auxiliary language dataset storage unit 91, acquires a target language dataset from the target language dataset storage unit 92, learns an intermediate scene character recognition model, and outputs parameters of the intermediate scene character recognition model as learned parameters (S11). The process of "learning an intermediate scene character recognition model and outputting parameters of the intermediate scene character recognition model as learned parameters" is called a pre-learning process. In addition, "learning" here refers to learning a model to input an image of pair data included in the dataset and output text.

[0015] The pre-training unit 11 provides pairs of images and character strings contained in a dataset obtained by mixing the target language dataset and the auxiliary language dataset as pairs of correct answer data for input and output of the scene character recognition model, and optimizes parameters using any learning method such as backpropagation.

[0016] The scene character recognition model is a one-stop scene character recognition model having an encoder-decoder type structure, as shown in, for example, Non-Patent Document 1 and Non-Patent Document 2. In this case, the encoder may be configured with, for example, a function of extracting image features by a convolutional neural network, and a function of converting the image features into features taking into account a sequence by a Transformer encoder provided in Reference Non-Patent Document 1. The decoder may be configured with, for example, a Transformer decoder provided in Reference Non-Patent Document 1, and an autoregressive model using an embedding layer and an output layer. The convolutional neural network of the encoder may be trained using parameters trained in advance in an arbitrary task such as object recognition as initial values, and other parts such as the Transformer encoder may also be trained using parameters trained in advance separately as initial values.

[0017] The target language dataset is a dataset of paired data in a target language to be estimated, and may be a language with few speakers and few data resources, such as Japanese. On the other hand, the auxiliary language dataset is a dataset of paired data in a language other than the target language, and may be a language with many speakers and many data resources, such as English. However, this is just one example, and the present invention is not limited to the case where the number of data in the target language dataset is small and the number of data in the auxiliary language dataset is large. Furthermore, both the target language and the auxiliary language do not need to be a single language, and either or both of the target language and the auxiliary language may be a collection of multiple languages. The dictionary of vocabulary output by the scene character recognition model can be a dictionary that covers the vocabulary of both the target language and the auxiliary language.

[0018] <Fine tuning process> Fine tuning unit 12 acquires a target language dataset from target language dataset storage unit 92, acquires trained parameters of the intermediate scene character recognition model from pre-training unit 11, trains a scene character recognition model using the trained parameters of the intermediate scene character recognition model as initial values ​​of the scene character recognition model, and outputs the scene character recognition model (S12). The output scene character recognition model is stored in scene character recognition model storage unit 93. The process of "training a scene character recognition model using the trained parameters of the intermediate scene character recognition model as initial values ​​of the scene character recognition model, and outputting the scene character recognition model" is called fine tuning processing.

[0019] The fine-tuning unit 12 may provide pairs of images and character strings included in the target language dataset as pairs of correct answer data for input and output of the scene character recognition model, and optimize the parameters by an arbitrary learning method such as backpropagation. At this time, learning may be performed after deleting the vocabulary of the auxiliary language from the vocabulary dictionary output by the scene character recognition model and deleting the related parameters.

[0020] As described above, in order to improve the character recognition accuracy of the target language, a dataset of an auxiliary language is used, so that a scene character recognition model can be pre-trained based on a large-scale dataset that combines paired data of the target language and paired data of the auxiliary language. This makes it possible to improve the accuracy of character recognition in the target language even when there is a small amount of paired data in the target language.

[0021] <Estimation device 100> As shown in FIG. 4, an estimation device 100 of this embodiment receives an image as input, estimates a character string corresponding to the image using the scene character recognition model trained by the above-described learning device 1, and outputs an estimation result. EXAMPLES

[0022] The functional configuration of the learning device of the second embodiment will be described below with reference to Fig. 5. As shown in the figure, the learning device 2 of this embodiment includes an encoder pre-learning unit 21 and a fine-tuning unit 22. An auxiliary language data set storage unit 91, a target language data set storage unit 92, and a scene character recognition model storage unit 93 shown in the figure may be inside the learning device 2 or inside another device. Fig. 6 shows the operation of the learning device 2 of this embodiment. The operation of the learning device 2 of this embodiment includes an encoder pre-learning process and a fine-tuning process.

[0023] <Encoder pre-training process> Encoder pre-training unit 21 acquires an auxiliary language dataset from auxiliary language dataset storage unit 91, acquires a target language dataset from target language dataset storage unit 92, trains an intermediate scene character recognition model composed of an encoder and a decoder, and outputs parameters of only the encoder part as trained parameters of the intermediate encoder (S21). Note that the process of "training an intermediate scene character recognition model composed of an encoder and a decoder, extracting parameters of only the encoder part, and outputting them as trained parameters of the intermediate encoder" is referred to as encoder pre-training processing.

[0024] The encoder pre-learning unit 21 can use a method similar to the pre-learning process in the first embodiment, and a pair of an image and a character string included in a data set obtained by mixing the target language data set and the auxiliary language data set is given as a pair of correct answer data for input and output of the scene character recognition model, and parameters are optimized by an arbitrary learning method such as an error backpropagation method. In this case, a dictionary of vocabulary that covers the vocabulary of both the target language and the auxiliary language may be used as the dictionary output by the scene character recognition model. Note that the intermediate encoder does not necessarily have to be the entire encoder shown in FIG. 1, and may be, for example, only a part that extracts image features by a convolutional neural network, or may be from a part that extracts image features by a convolutional neural network to an intermediate layer of a Transformer encoder consisting of multiple layers.

[0025] <Fine tuning process> Fine tuning unit 22 acquires a target language dataset from target language dataset storage unit 92, acquires an intermediate encoder from encoder pre-training unit 21, provides trained parameters of the intermediate encoder as initial values ​​of the encoder of the scene character recognition model, trains the scene character recognition model using the target language dataset, and outputs the scene character recognition model (S22). The output scene character recognition model is stored in scene character recognition model storage unit 93. The process of "providing trained parameters of the intermediate encoder as initial values ​​of the encoder of the scene character recognition model, training the scene character recognition model using the target language dataset, and outputting the scene character recognition model" is referred to as fine tuning processing.

[0026] The fine-tuning unit 22 can use a method similar to the fine-tuning process in the first embodiment, and can provide pairs of images and character strings included in the target language dataset as pairs of correct answer data for input and output of the scene character recognition model, and optimize parameters using an arbitrary learning method such as backpropagation. Here, the initial values ​​of the parts not included in the intermediate encoder, including the decoder, can be provided using an arbitrary initialization method such as random initialization. In addition, the vocabulary dictionary output by the scene character recognition model can be a dictionary that covers the vocabulary of the target language.

[0027] As described above, by pre-training the encoder with a large-scale dataset that combines the auxiliary language dataset and the target language dataset, the number of images that the encoder must learn is increased. This allows the encoder, which is responsible for extracting image features, to be trained using data with a variety of character shapes and background images, improving the robustness of the encoder to input images.

[0028] <Estimation device 200> As shown in FIG. 7, an estimation device 200 of this embodiment receives an image as input, estimates a character string corresponding to the image using the scene character recognition model trained by the above-described learning device 2, and outputs an estimation result. EXAMPLES

[0029] Hereinafter, the functional configuration of the learning device of the third embodiment will be described with reference to Fig. 8. As shown in the figure, the learning device 3 of this embodiment includes an encoder pre-learning unit 31, a decoder pre-learning unit 32, and a fine-tuning unit 33. An auxiliary language data set storage unit 91, a target language data set storage unit 92, and a scene character recognition model storage unit 93 shown in the figure may be inside the learning device 3 or inside another device. Fig. 9 shows the operation of the learning device 3 of this embodiment. The operation of the learning device 3 of this embodiment includes an encoder pre-learning process, a decoder pre-learning process, and a fine-tuning process.

[0030] <Encoder pre-training process> As in the second embodiment, the encoder pre-learning unit 31 acquires an auxiliary language dataset from the auxiliary language dataset storage unit 91, acquires a target language dataset from the target language dataset storage unit 92, learns an intermediate scene character recognition model composed of an encoder and a decoder, extracts only the encoder part, and outputs it as an intermediate encoder (S31). Note that the process of "learning an intermediate scene character recognition model composed of an encoder and a decoder, extracts only the encoder part, and outputs it as an intermediate encoder" is called an encoder pre-learning process.

[0031] <Decoder pre-training process> The decoder pre-training unit 32 receives the target language dataset as input, trains a second intermediate scene character recognition model consisting of an encoder and a decoder, extracts parameters of only the decoder part of the second intermediate scene character recognition model, and outputs them as trained parameters of the intermediate decoder (S32). Note that the process of "training a second intermediate scene character recognition model consisting of an encoder and a decoder, extracting parameters of only the decoder part of the second intermediate scene character recognition model, and outputting them as trained parameters of the intermediate decoder" is referred to as a decoder pre-training process.

[0032] The decoder pre-training unit 32 may provide pairs of images and character strings included in the target language dataset as pairs of correct answer data for input and output of the scene character recognition model, and optimize parameters using any training method such as backpropagation. In this case, a dictionary that covers the vocabulary of the target language may be used as the vocabulary dictionary output by the scene character recognition model. The intermediate decoder does not necessarily have to be the entire decoder shown in FIG. 1, and may be, for example, only the embedding layer portion, or may be from the embedding layer to an intermediate layer of a Transformer decoder consisting of multiple layers. Furthermore, only the embedding layer and the output layer may be the intermediate decoder.

[0033] <Fine tuning process> Fine tuning unit 33 receives the target language data set, intermediate encoder, and intermediate decoder as input, provides the trained parameters of the intermediate encoder and intermediate decoder as initial values ​​of the encoder and decoder of the scene character recognition model, trains the scene character recognition model using the target language data set, and outputs the scene character recognition model (S33). Note that the process of "providing the trained parameters of the intermediate encoder and intermediate decoder as initial values ​​of the encoder and decoder of the scene character recognition model, trains the scene character recognition model using the target language data set, and outputs the scene character recognition model" is called fine tuning processing. The output scene character recognition model is stored in scene character recognition model storage unit 93.

[0034] The fine-tuning unit 33 can use a method similar to the fine-tuning process in the first embodiment, and can provide pairs of images and character strings included in the target language dataset as pairs of correct answer data for input and output of the scene character recognition model, and optimize parameters using an arbitrary learning method such as backpropagation. Here, the initial values ​​of the parts not included in the intermediate encoder and intermediate decoder can be provided by an arbitrary initialization method such as random initialization. In addition, the vocabulary dictionary output by the scene character recognition model can be a dictionary that covers the vocabulary of the target language.

[0035] As mentioned above, the decoder can be pre-trained on the target language dataset to specialize the decoder for the target language. The combination of decoder pre-training and encoder pre-training can effectively improve the accuracy of scene character recognition in the target language.

[0036] <Estimation device 300> As shown in FIG. 10, an estimation device 300 of this embodiment receives an image as input, estimates a character string corresponding to the image using the scene character recognition model trained by the above-mentioned training device 3, and outputs the estimation result.

[0037] <Effects> According to the learning device described in the above embodiment, the encoder is pre-trained using not only the target language dataset but also the auxiliary language dataset. This increases the number of images that the encoder learns, so that the encoder, which is responsible for extracting image features, can be trained using data having a variety of character shapes and background images, improving the robustness of the encoder to input images. Furthermore, the decoder is pre-trained using the target language dataset. This allows the decoder to be specialized for the target language, and by combining this with the pre-training of the encoder, the accuracy of scene character recognition in the target language can be effectively improved.

[0038] A verification experiment was conducted on the scene character recognition model having the structure described in Non-Patent Document 1 and Non-Patent Document 2. The target language was Japanese, and about 70,000 sets of paired data were prepared as the teacher data. The auxiliary language was English, and about 8,000,000 sets of paired data were prepared as the teacher data. The character recognition accuracy was verified for about 7,000 images in the target language not included in the teacher data, for a baseline trained from scratch using the target language dataset without using the auxiliary language dataset or pre-training, and a case trained using the training methods of Examples 1, 2, and 3. The accuracy rate based on a perfect match was used for evaluation. The results of the verification experiment are shown in FIG. 11. As shown in the figure, it can be seen that even with the training method of Example 1, which simply trains using the auxiliary language dataset, the accuracy rate is improved compared to the baseline. Furthermore, it can be seen that the accuracy rate is further improved by pre-training the encoder with the auxiliary language dataset and the target language dataset, and pre-training the decoder with the target language dataset, as in Example 3.

[0039] Fig. 12 shows an example of a recognition result in a scene character recognition model having a structure described in Non-Patent Document 2. As shown in (1) to (3) of Fig. 12, it can be seen that misrecognition is prevented by performing the pre-learning process according to the first embodiment. Also, as shown in (4) of the same figure, it can be seen that recognition of invalid words and phrases can be prevented by performing the decoder pre-learning process according to the third embodiment.

[0040] <Additional Notes> The device of the present invention has, as a single hardware entity, an input section to which a keyboard or the like can be connected, an output section to which a liquid crystal display or the like can be connected, a communication section to which a communication device (e.g., a communication cable) capable of communicating with the outside of the hardware entity can be connected, a CPU (which may also have a central processing unit, cache memory, registers, etc.), memories such as RAM and ROM, an external storage device such as a hard disk, and buses connecting the input section, output section, communication section, CPU, RAM, ROM, and external storage device so that data can be exchanged between them. If necessary, the hardware entity may also be provided with a device (drive) capable of reading and writing a recording medium such as a CD-ROM. A physical entity equipped with such hardware resources is, for example, a general-purpose computer.

[0041] The external storage device of the hardware entity stores the programs required to realize the above-mentioned functions and the data required in the processing of these programs (not limited to the external storage device, but for example the programs may be stored in a ROM, which is a read-only storage device). Data obtained by the processing of these programs is stored appropriately in the RAM, the external storage device, etc.

[0042] In the hardware entity, each program stored in an external storage device (or ROM, etc.) and the data required to process each program are loaded into memory as necessary, and interpreted, executed, and processed by the CPU as appropriate, so that the CPU realizes a predetermined function (each of the components represented as the above, "units," "means," etc.).

[0043] The present invention is not limited to the above-described embodiment, and can be modified as appropriate without departing from the spirit of the present invention. In addition, the processes described in the above-described embodiment are not limited to being executed in chronological order according to the order described, but may be executed in parallel or individually depending on the processing capacity of the device that executes the processes or as necessary.

[0044] As described above, when the processing functions of the hardware entities (the devices of the present invention) described in the above embodiments are realized by a computer, the processing contents of the functions that the hardware entities should have are described by a program. Then, by executing this program on a computer, the processing functions of the hardware entities are realized on the computer.

[0045] The various processes described above can be implemented by loading a program for executing each step of the above method into the recording unit 10020 of the computer 10000 shown in FIG. 13, and operating the control unit 10010, input unit 10030, output unit 10040, etc.

[0046] The program describing the processing contents can be recorded on a computer-readable recording medium. The computer-readable recording medium may be, for example, a magnetic recording device, an optical disk, a magneto-optical recording medium, a semiconductor memory, or any other suitable medium. Specifically, for example, a hard disk drive, a flexible disk, a magnetic tape, or the like can be used as the magnetic recording device, a DVD (Digital Versatile Disc), a DVD-RAM (Random Access Memory), a CD-ROM (Compact Disc Read Only Memory), a CD-R (Recordable) / RW (ReWritable), or the like can be used as the optical disk, an MO (Magneto-Optical disc), or the like can be used as the magneto-optical recording medium, and an EEP-ROM (Electrically Erasable and Programmable-Read Only Memory) or the like can be used as the semiconductor memory.

[0047] The program may be distributed, for example, by selling, transferring, lending, etc. portable recording media such as DVDs and CD-ROMs on which the program is recorded. Furthermore, the program may be distributed by storing the program in a storage device of a server computer and transferring the program from the server computer to other computers via a network.

[0048] A computer that executes such a program, for example, first stores the program recorded on a portable recording medium or the program transferred from a server computer in its own storage device. Then, when executing a process, the computer reads the program stored in its own recording medium and executes a process according to the read program. As another execution form of the program, the computer may directly read the program from the portable recording medium and execute a process according to the program, or may execute a process according to the received program each time a program is transferred from the server computer to the computer. The above-mentioned process may be executed by a so-called ASP (Application Service Provider) type service that realizes a processing function only by issuing an execution instruction and obtaining a result, without transferring a program from the server computer to the computer. Note that the program in this embodiment includes information used for processing by an electronic computer and equivalent to a program (data that is not a direct command to the computer but has a nature that specifies the processing of the computer, etc.).

[0049] In addition, in this embodiment, a hardware entity is configured by executing a specific program on a computer, but at least a part of the processing contents may be realized by hardware.

[0050] The following supplementary notes are further disclosed regarding the above embodiment.

[0051] (Additional note 1) Memory, at least one processor coupled to the memory; Including, The processor, We obtain an auxiliary language dataset, which is a dataset of image and text pair data in a language other than the target language (hereinafter referred to as the auxiliary language), and a target language dataset, which is a dataset of image and text pair data in the target language, and Scene character recognition model Learn and A model for scene character recognition Output the parameters as trained parameters, Obtaining the target language dataset and the trained parameters, and The scene character recognition model The initial value is , scene character recognition model Learn Learning device. (Additional note 2) A non-transitory storage medium storing a program executable by a computer to execute a learning process, The learning process includes: We obtain an auxiliary language dataset, which is a dataset of image and text pair data in a language other than the target language (hereinafter referred to as the auxiliary language), and a target language dataset, which is a dataset of image and text pair data in the target language, and Scene character recognition model Learn and A model for scene character recognition Output the parameters as trained parameters, Obtaining the target language dataset and the trained parameters, and The scene character recognition model The initial value is , scene character recognition model Learn Non-transitory storage medium. (Additional note 3) The learning device according to claim 1, In the above Scene-based character recognition model, before The scene character recognition model is It consists of an encoder and a decoder. In the above A model for scene character recognition The parameters of only the encoder part are output as trained parameters of the intermediate encoder. The trained parameters of the intermediate encoder are Scene character recognition modelProvide the initial value for the encoder Learning device. (Additional note 4) A non-transitory storage medium according to claim 2, In the above Scene-based character recognition model, before The scene character recognition model is It consists of an encoder and a decoder. In the above A model for scene character recognition The parameters of only the encoder part are output as trained parameters of the intermediate encoder. The trained parameters of the intermediate encoder are Scene character recognition model Provide the initial value for the encoder Non-transitory storage medium. (Additional note 5) The learning device according to claim 3, In the above A model for scene character recognition The parameters of only the decoder part are output as trained parameters of the intermediate decoder. The trained parameters of the intermediate decoder are Scene character recognition model Provide the initial value for the decoder Learning device. (Additional note 6) A non-transitory storage medium according to claim 4, In the above A model for scene character recognition The parameters of only the decoder part are output as trained parameters of the intermediate decoder. The trained parameters of the intermediate decoder are Scene character recognition model Provide the initial value for the decoder Non-transitory storage medium. (Additional note 7) Memory, at least one processor coupled to the memory; Including, The processor, An image is input and trained by the training device according to any one of appendix 1, 3, and 5. The scene character recognition model to estimate the character string corresponding to the image. Estimation device. (Additional note 8) The non-transitory storage medium described in any one of appendix 2, 4, or 6 is used to learn the The scene character recognition modelto estimate the character string corresponding to the image. Non-transitory storage medium.

Claims

1. a pre-learning unit that acquires an auxiliary language dataset, which is a dataset of pair data of images and texts in a language other than the target language (hereinafter, the auxiliary language), and a target language dataset, which is a dataset of pair data of images and texts in the target language, learns an intermediate scene character recognition model using the acquired data, and outputs parameters of the intermediate scene character recognition model as learned parameters; and a fine-tuning unit that acquires the target language data set and the trained parameters, sets the trained parameters as initial values ​​of a scene character recognition model, and trains the scene character recognition model. Learning device.

2. The learning device according to claim 1 , the intermediate scene character recognition model, the scene character recognition model being composed of an encoder and a decoder; the pre-learning unit outputs parameters of only an encoder portion of the intermediate scene character recognition model as trained parameters of an intermediate encoder; The fine tuning unit provides the trained parameters of the intermediate encoder as initial values ​​of the encoder of the scene character recognition model. Learning device.

3. The learning device according to claim 2, The advance learning unit includes: training a second intermediate scene character recognition model using the target language dataset as input, and outputting parameters of only a decoder portion of the second intermediate scene character recognition model as trained parameters of an intermediate decoder; The fine tuning unit provides the trained parameters of the intermediate decoder as initial values ​​of a decoder of the scene character recognition model. Learning device.

4. An image is input, and a character string corresponding to the image is estimated using a scene character recognition model trained by the learning device according to any one of claims 1 to 3. Estimation device.

5. A learning method executed by a learning device, comprising: acquiring an auxiliary language dataset, which is a dataset of pair data of images and texts in a language other than the target language (hereinafter, referred to as the auxiliary language), and a target language dataset, which is a dataset of pair data of images and texts in the target language, learning an intermediate scene character recognition model, and outputting parameters of the intermediate scene character recognition model as learned parameters; acquiring the target language data set and the trained parameters, and setting the trained parameters as initial values ​​of a scene character recognition model to train the scene character recognition model. How to learn.

6. A program for causing a computer to function as the learning device according to any one of claims 1 to 3.

7. A program that causes a computer to function as the estimation device according to claim 4.

Images