Information processing device, information processing method, and information processing program

Abstract

An information processing device (100) has a feature extraction unit (126a) and a character string estimation unit (126b). The feature extraction unit (126a) extracts an image feature from a character image. The character string estimation unit (126b) estimates a character string on the basis of the character-writing direction and the image feature.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and an information processing program.

Background Art

[0002] Scene images obtained by photographing a landscape contain a lot of character information necessary for understanding the image, such as traffic signs and advertising billboards. Scene character recognition is a task of inputting an image (hereinafter, character image) obtained by cutting out a character area from such a scene image, recognizing the characters shown, and converting them into a machine-processable character string. In recent years, with the progress of deep learning technology, a method for realizing scene character recognition with an end-to-end model has been proposed.

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Non-Patent Document 2

Non-Patent Document 3

[0004] For example, Non-Patent Document 1 provides a scene character recognition technology using a model consisting of an encoder and a decoder, as schematically shown in Figure 1. In this case, the encoder consists of a part that extracts features of the character image using, for example, a convolutional neural network, and a part that converts them into sequential features using a Transformer encoder provided in Non-Patent Document 2. The decoder consists of, for example, an autoregressive model using an embedding layer, a Transformer decoder provided in Non-Patent Document 2, and an output layer, and outputs the probability of generating a string from the character image features (hereinafter referred to as image features) extracted by the encoder. Using such a model, the probability P of generating the string C={c_1,…,c_T} written on the character image I is modeled as follows. Here, Θ is a learnable model parameter.

[0005]

number

[0006] In some Asian languages, such as Japanese, there are two writing directions: horizontal and vertical. While it's possible to recognize characters using separate character recognition models for horizontal and vertical writing, training two models requires collecting sufficient training data for both, which is inefficient. Therefore, a model configuration has been proposed that enables character recognition for both horizontal and vertical writing using a single model, sharing model parameters.

[0007] For example, Non-Patent Document 3 schematically shows in Figure 23 that by sharing model parameters for horizontal and vertical writing, character recognition for both horizontal and vertical writing is made possible with a single model. In the baseline shown in Figure 23(a), assuming that a vertically written character image is input rotated 90 degrees counterclockwise, all model parameters are shared for both horizontal and vertical writing, realizing a model that can recognize strings of characters in both writing directions. In the method called Direction encoding mask (DEM) shown in Figure 23(b), an image representing the writing direction is combined with the baseline in the channel direction and input, thereby realizing a model that takes writing direction into account. In the method called Selective attention network (SAN) shown in Figure 23(c), the accuracy of the character recognition model according to writing direction is improved by branching a part of the model for horizontal and vertical writing relative to the baseline. For example, only the Transformer encoder in Figure 1 is branched for horizontal and vertical writing, while other components are shared. [Overview of the project] [Problems that the invention aims to solve]

[0008] However, conventional technologies have a problem in that a model capable of recognizing both horizontal and vertical writing cannot be created unless a large amount of training data for both horizontal and vertical writing is collected. For example, when training a model capable of recognizing both horizontal and vertical writing, as realized in Non-Patent Document 3, DEM uses combined images indicating the writing direction as input, so sufficient training data for both horizontal and vertical writing is necessary to train a model that can read both. Similarly, in SAN, the model is not partially shared, so in this case as well, sufficient training data for both horizontal and vertical writing must be collected. However, in general, vertical writing character images are more difficult to collect in real-world environments than horizontal writing images. [Means for solving the problem]

[0009] To solve the above-mentioned problems and achieve the objective, the information processing device according to the present invention comprises a feature extraction unit and a string estimation unit. The feature extraction unit extracts image features from a character image. The string estimation unit estimates a string from the writing direction and image features.

[0010] Furthermore, the information processing device according to the present invention comprises a feature extraction unit, a string estimation unit, and a learning unit. The feature extraction unit extracts image features from a character image. The string estimation unit estimates a string from the writing direction and the image features. The learning unit learns a model for the processing performed by the feature extraction unit and the string estimation unit, based on the correct string corresponding to the character image and the estimated string. [Effects of the Invention]

[0011] According to the present invention, it is possible to overcome the problem that a model capable of recognizing both horizontal and vertical writing cannot be created unless a large amount of training data for both horizontal and vertical writing is collected. [Brief explanation of the drawing]

[0012] [Figure 1] Figure 1 shows a conventional character recognition model. [Figure 2] Figure 2 is a block diagram showing an example configuration of an information processing device. [Figure 3] Figure 3 shows an example of the configuration of the information processing device during estimation. [Figure 4] Figure 4 shows an example of processing performed by an information processing device. [Figure 5] Figure 5 shows an example of the configuration of an information processing device during learning. [Figure 6] Figure 6 is a flowchart showing an example of the processing flow by an information processing device. [Figure 7] Figure 7 shows an example of the configuration of the information processing device during estimation. [Figure 8] Figure 8 shows an example of processing performed by an information processing device. [Figure 9] Figure 9 shows an example of the configuration of the information processing device during estimation. [Figure 10] FIG. 10 is a diagram showing an example of processing by an information processing apparatus. [Figure 11] FIG. 11 is a diagram showing a configuration example during learning of the information processing apparatus. [Figure 12] FIG. 12 is a flowchart showing an example of the processing flow by the information processing apparatus. [Figure 13] FIG. 13 is a diagram showing a configuration example during estimation of the information processing apparatus. [Figure 14] FIG. 14 is a diagram showing a configuration example of the information processing apparatus. [Figure 15] FIG. 15 is a diagram showing a configuration example during estimation of the information processing apparatus. [Figure 16] FIG. 16 is a diagram showing an example of processing by the information processing apparatus. [Figure 17] FIG. 17 is a diagram showing a configuration example during estimation of the information processing apparatus. [Figure 18] FIG. 18 is a diagram showing an example of processing by the information processing apparatus. [Figure 19] FIG. 19 is a diagram showing a configuration example during learning of the information processing apparatus. [Figure 20] FIG. 20 is a flowchart showing an example of the processing flow by the information processing apparatus. [Figure 21] FIG. 21 is a diagram showing a configuration example during estimation of the information processing apparatus. [Figure 22] FIG. 22 is a diagram showing an example of processing by the information processing apparatus. [Figure 23] FIG. 23 is a diagram showing a character recognition model according to the prior art. [Figure 24] FIG. 24 is a table showing a string estimation result by the information processing apparatus. [Figure 25] FIG. 25 is a diagram showing a string estimation result by the information processing apparatus. [Figure 26] FIG. 26 is a diagram showing an example of a computer that executes an information processing program.

MODE FOR CARRYING OUT THE INVENTION

[0013] Hereinafter, embodiments of the information processing apparatus, information processing method, and information processing program according to the present application will be described in detail with reference to the drawings. However, the present invention is not limited to these embodiments. Furthermore, in the drawings, identical parts are denoted by the same reference numerals, and redundant explanations are omitted.

[0014] [Summary of the Invention] The information processing device 100 according to this embodiment achieves highly accurate string estimation by using the results of estimating the writing direction and the number of characters for string estimation using an encoder-decoder model.

[0015] For example, in character recognition, the information processing device 100 shares all model parameters between horizontal and vertical writing, thereby sharing character-specific contours and vocabulary useful for character recognition between horizontal and vertical writing. Furthermore, to correctly decode horizontal and vertical writing respectively, tokens that distinguish between horizontal and vertical writing are provided as initial values ​​for the autoregressive decoder, thereby achieving highly accurate string estimation. In this case, the present invention can be applied to all technologies that output strings from character images through a model with an autoregressive decoder of any encoder-decoder type. It can also be applied to optical character recognition and the like.

[0016] Furthermore, for example, prior to the process of predicting a string, the information processing device 100 predicts the number of characters in the character image and outputs a string based on the prediction result. This means that prior to the process of predicting a string, the number of characters that need to be grasped from an overview of the character image is predicted, that is, characters are recognized after grasping the group of characters, thus preventing the recognition of characters by mistakenly splitting or combining radicals and components, and improving the accuracy of string estimation. At this time, the present invention can be applied to all technologies that output strings from character images through an arbitrary end-to-end sequence-to-sequence model. It can also be applied to optical character recognition and the like.

[0017] [Configuration of the information processing device] First, the configuration of the information processing device will be explained using Figure 2. As shown in Figure 2, the information processing device 100 has a communication unit 110, a control unit 120, and a storage unit 130. Note that each of these units may be distributed and held by multiple devices. The processing of each of these units will be explained below.

[0018] The communication unit 110 is implemented using a NIC (Network Interface Card) or the like, and enables communication between the control unit 120 and external devices via telecommunication lines such as a LAN (Local Area Network) or the Internet. For example, the communication unit 110 enables communication between an external device and the control unit 120.

[0019] The memory unit 130 is implemented by semiconductor memory elements such as RAM (Random Access Memory) and flash memory, or by storage devices such as hard disks and optical discs. The information stored in the memory unit 130 includes, for example, character images, image features, data related to machine learning algorithms, training data, and trained models. However, the information stored in the memory unit 130 is not limited to those described above.

[0020] The control unit 120 is implemented using a CPU (Central Processing Unit), NP (Network Processor), FPGA (Field Programmable Gate Array), etc., and executes processing programs stored in memory. As shown in Figure 2, the control unit 120 includes an acquisition unit 121, a writing direction estimation unit 122, an image rotation unit 123, a character count estimation unit 124, a model learning unit (learning unit) 125, a character recognition unit 126, an encoder (feature extraction unit) 126a, and a decoder (string estimation unit) 126b. The individual parts of the control unit 120 will be described below.

[0021] Note that the division of functional units in the configuration diagram is just one example; the system may be implemented using only some functional units, multiple functional units may be implemented as a single functional unit, one functional unit may be divided into multiple units, or some functions may be transferred to other functional units. Furthermore, the functions of multiple functional units with similar functions may be processed in parallel or time-division by a single piece of hardware or software.

[0022] The acquisition unit 121 acquires character images. The writing direction estimation unit 122 uses the character images acquired by the acquisition unit 121 as input to a model for estimating writing direction (hereinafter referred to as the writing direction estimation model), estimates the writing direction, and outputs the estimated writing direction.

[0023] For example, the writing direction estimation unit 122 may use a writing direction estimation model that determines horizontal writing if the character image is horizontally oriented and vertical writing if it is vertically oriented, based on the aspect ratio of the character image. Alternatively, for example, the writing direction estimation unit 122 may define a determination model that takes character images or image features as input and outputs an estimated writing direction using a machine learning model, train it in advance using training data, and then use that determination model as the writing direction estimation model.

[0024] Furthermore, the information processing device 100 may use any writing direction that represents how characters are read, including not only vertical and horizontal writing, but also inversion and rotation. For example, the information processing device 100 may use all combinations of "vertical or horizontal writing, inversion or non-inversion, and counterclockwise rotation of 0, 90, 180, or 270 degrees" as writing directions.

[0025] The image rotation unit 123 takes a character image and the estimated writing direction as input, rotates the character image to the orientation assumed by the character recognition unit 126, and outputs the rotated character image. For example, in the case of horizontal writing and vertical writing, the image rotation unit 123 outputs the character image as is for horizontal writing, and rotates the character image 90 degrees counterclockwise for vertical writing as the rotated character image. Note that the image rotation unit 123 can be omitted if the character recognition unit 126 assumes that the input is a character image that does not need to be rotated.

[0026] The character count estimation unit 124 uses the character image acquired by the acquisition unit 121 or the image features extracted by the encoder 126a as input to a character count estimation model (hereinafter referred to as the character count estimation model) to estimate the number of characters and output the estimated number of characters. For example, the character count estimation unit 124 uses the character image as input to the character count estimation model to estimate the number of characters and output the estimated number of characters. Alternatively, for example, the character count estimation unit 124 uses the image features as input to the character count estimation model to output the estimated number of characters.

[0027] The model learning unit 125 learns a model (hereinafter referred to as the character recognition model) that processes the encoder 126a and decoder 126b based on the correct character string corresponding to the character image and the estimated character string (hereinafter referred to as the estimated character string). The model learning unit 125 also learns a character recognition model and a character count estimation model based on the correct number of characters corresponding to the character image and the estimated number of characters. For example, the model learning unit 125 learns a character recognition model that extracts image features from a rotated character image and estimates the character string from the image features and the estimated writing direction.

[0028] Furthermore, for example, the model learning unit 125 learns a character recognition model that extracts image features from character images and estimates a string from the image features and the estimated number of characters, and a character count estimation model that estimates the number of characters from the image features.

[0029] Furthermore, for example, the model learning unit 125 learns a character recognition model that extracts image features from a rotated character image, estimates the number of characters from the image features and estimated writing direction, and estimates a character string from the image features, estimated writing direction, and estimated number of characters.

[0030] The character recognition unit 126 is composed of an encoder 126a and a decoder 126b. The encoder 126a extracts image features from a character image. For example, the encoder 126a extracts image features from a character image acquired by the acquisition unit 121. Also, for example, the encoder 126a extracts image features from a rotated character image. Also, for example, the encoder 126a extracts image features from a character image that include elements that allow the number of characters to be estimated. Here, the encoder 126a extracts features that take sequences into consideration using, for example, a convolutional neural network and a Transformer encoder.

[0031] Decoder 126b outputs an estimated string from image features. Decoder 126b generates outputs recursively. For example, decoder 126b estimates a string from image features and estimated writing direction and outputs the estimated string. For example, decoder 126b takes image features and estimated writing direction output from writing direction estimation unit 122 as input, estimates a string, and outputs the estimated string. Alternatively, decoder 126b estimates the writing direction from image features, outputs the estimated writing direction, then estimates a string, and outputs the estimated string.

[0032] At this time, the writing direction estimation unit 122 or decoder 126b, for example, uses a writing direction token, which is a special token representing the estimated writing direction, as a start token. <s>Alternatively, it may be input to decoder 126b. For example, information processing device 100 may input horizontal writing as a writing direction token. <h>, vertical writing <v>It is defined as follows. The writing direction token, like other tokens, should be registered in the dictionary beforehand. Note that the start token <s>This is the initial token for decoding by decoder 126b. Also, the termination token. <e>This is a token indicating the completion of decoding by decoder 126b.

[0033] Furthermore, for example, decoder 126b estimates a string from image features and estimated character count, and outputs the estimated string. For example, decoder 126b takes image features and the estimated character count output from character count estimation unit 124 as input, estimates a string, and outputs the estimated string. Furthermore, for example, decoder 126b estimates a string from image features that include elements for which character count can be estimated, and outputs the estimated string. Furthermore, for example, decoder 126b estimates the character count from image features, outputs the estimated character count, then estimates a string, and outputs the estimated string.

[0034] Here, the estimated number of characters is converted, for example, into a character count token that represents the number of characters, and then the start token is used. <s>Instead, it is input to decoder 126b. At this time, the information processing device 100 uses the character count token, for example, with n as the estimated number of characters, <n>It is defined as follows. Furthermore, the information processing device 100 shall pre-register the character count tokens in a dictionary, just like other tokens.

[0035] Furthermore, for example, decoder 126b estimates a string from image features, estimated writing direction, and estimated number of characters, and outputs the estimated string. For example, decoder 126b takes image features, the estimated writing direction output from writing direction estimation unit 122, and the estimated number of characters output by decoder 126b as input and outputs the estimated string. Alternatively, for example, decoder 126b estimates the number of characters and writing direction from image features, outputs them, then estimates a string and outputs the estimated string.

[0036] [String estimation using estimated writing direction] [overview] The information processing device 100 uses the estimated writing direction to estimate the character string using the decoder 126b.

[0037] [Embodiment 1] Embodiment 1 of the information processing device 100 will be described with reference to Figures 3 to 6. Figure 3 shows an example of the configuration in Embodiment 1 of the information processing device 100. The information processing device 100 consists of a writing direction estimation unit 122, an image rotation unit 123, and a character recognition unit 126. The character recognition unit 126 consists of an encoder 126a and a decoder 126b.

[0038] The writing direction estimation unit 122 takes a character image as input, estimates the writing direction, and outputs it as the estimated writing direction. The image rotation unit 123 takes a character image and the estimated writing direction as input, rotates the character image to the orientation assumed by the character recognition unit 126, and outputs it as a rotated character image.

[0039] Encoder 126a takes a rotated character image as input and outputs image features. Decoder 126b takes the image features and the estimated writing direction output from the writing direction estimation unit 122 as input and outputs an estimated string. Here, the estimated writing direction output by the writing direction estimation unit 122 is converted into a special token (writing direction token) that represents the estimated writing direction, for example, and then input to the decoder in place of the start token. The writing direction token is, for example, for horizontal writing. <h>, vertical writing <v>It is defined as follows. Writing direction tokens, like other tokens, are to be pre-registered in the dictionary.

[0040] Figure 4 shows an example of the operation of the character recognition unit 126 in Embodiment 1. When the writing direction estimation unit 122 estimates that it is written horizontally, a writing direction token is generated. <h>is the start token <s>By inputting the text to decoder 126b instead, decoder 126b recognizes that the input image is horizontally written and correctly decodes the string.

[0041] Furthermore, the character recognition unit 126 becomes capable of character recognition when the model learning unit 125 learns a model like the one shown in equation 2, which takes into account the estimated writing direction d, in estimating the generation probability P of the string C={c_1,…,c_T} written on the character image I.

[0042]

number

[0043] Here, Θ is a learnable model parameter. As shown in Figure 5, the model learning unit 125 can use a set of character images, corresponding correct character strings, and estimated writing directions derived by the writing direction estimation unit 122 as training data to optimize the parameters of the character recognition model, which consists of an encoder 126a and a decoder 126b, for example, by backpropagation.

[0044] [flowchart] Next, the information processing flow by the information processing device 100 will be explained using Figure 6. Note that steps S11 to S15 below can be executed in a different order. Also, some of the steps S11 to S15 below may be omitted.

[0045] First, the acquisition unit 121 acquires a character image (step S11). Next, the writing direction estimation unit 122 uses the character image acquired by the acquisition unit 121 as input to a writing direction estimation model to estimate the writing direction of the characters contained in the character image (step S12).

[0046] Then, the image rotation unit 123 rotates the character image based on the estimated writing direction of the characters contained in the character image, which is estimated by the writing direction estimation unit 122 (step S13). Note that the image rotation unit 123 does not need to rotate the character image if the encoder 126a, decoder 126b, etc., do not assume a rotated image.

[0047] Next, the encoder 126a extracts image features from the character image (step S14). For example, the encoder 126a extracts image features from the character image acquired by the acquisition unit 121. Alternatively, for example, the encoder 126a extracts image features from the character information contained in the rotated character image, which has been rotated by the image rotation unit 123.

[0048] Then, the decoder 126b estimates the string from the image features extracted by the encoder 126a and the estimated writing direction (step S15).

[0049] [effect] With the above configuration, the information processing device 100 can efficiently model character recognition that can recognize both horizontally and vertically written characters. Specifically, by sharing all model parameters between horizontal and vertical writing, the information processing device 100 can share character-specific contours and vocabulary useful for character recognition between horizontal and vertical writing. Furthermore, by providing a writing direction token that distinguishes between horizontal and vertical writing as the initial value of the autoregressive decoder, the information processing device 100 can correctly decode strings in both horizontal and vertical writing.

[0050] [Embodiment 2] Embodiment 2 of the information processing device 100 will be described with reference to Figures 7 and 8. Embodiment 2 differs from Embodiment 1 in that the estimated writing direction is not input to the decoder 126b, but is estimated by the decoder 126b. In other words, the decoder 126b estimates the writing direction from the image features, outputs the estimated writing direction, and then estimates the string.

[0051] Figure 7 shows an example configuration of Embodiment 2 of the information processing device 100. In Embodiment 2, the decoder 126b takes image features as input and first outputs the estimated writing direction by the decoder 126b as a writing direction token. Then, taking the estimated writing direction represented by the decoder 126b, which is composed of image features and writing direction tokens, as input, it outputs an estimated string.

[0052] Figure 8 shows an example of the operation of the character recognition unit 126 in Embodiment 2. When the writing direction estimation unit 122 estimates that it is written horizontally, the start token <s>When input is received, decoder 126b first estimates the writing direction and uses a writing direction token as the estimated writing direction. <h>Outputs the following: Writing direction token <h>When this is input to decoder 126b, decoder 126b correctly decodes the string, taking into account that the input image is horizontally written.

[0053] The processing in Embodiment 2 is similar to that in Figure 6, but differs in that the decoder 126b does not take an estimated writing direction as input and instead outputs a string containing a writing direction token.

[0054] [String estimation using estimated character count] [overview] The information processing device 100 uses the estimated number of characters to estimate the string using the decoder 126b.

[0055] [Embodiment 3] Embodiment 3 of the information processing device 100 will be described with reference to Figures 9 to 12. Figure 9 shows an example of the configuration in Embodiment 3 of the information processing device 100. The information processing device 100 consists of a character count estimation unit 124 and a character recognition unit 126. The character recognition unit 126 consists of an encoder 126a and a decoder 126b.

[0056] Encoder 126a takes a character image as input and outputs image features. Decoder 126b takes the image features and the estimated number of characters output from the character count estimation unit 124 as input and outputs an estimated string.

[0057] Here, the estimated number of characters output by the character count estimation unit 124 is converted, for example, into a special token representing the number of characters (character count token), and then input to the decoder in place of the start token. The character count token is, for example, with n as the estimated number of characters. <n>It is defined as follows. Character count tokens, like other tokens, are to be pre-registered in the dictionary.

[0058] Figure 10 shows an example of the operation of the character recognition unit 126 in Embodiment 3. In the character recognition model of Figure 10, when the estimated number of characters is estimated to be "2" by the character count estimation unit 124, a character count token is generated. <2> is the start token <s>The input is then passed to decoder 126b, which subsequently outputs the estimated string.

[0059] Furthermore, the information processing device 100 enables character recognition by having the model learning unit 125 learn a model like the one shown in equation 3, which uses the estimated number of characters n to estimate the generation probability P of the string C={c_1,…,c_T} written on the character image I.

[0060]

number

[0061] Here, Θ is a learnable model parameter. As shown in Figure 11, the model learning unit 125 can use a set of character images, corresponding correct character strings, and the correct character count that can be derived from the correct character string as training data to optimize the parameters of the character recognition model, which consists of an encoder 126a and a decoder 126b, and the parameters of the character count estimation model, for example, by backpropagation.

[0062] [flowchart] Next, the information processing flow in Embodiment 3 will be explained using Figure 12. Note that steps S21 to S24 below can be executed in a different order. Also, some of the steps S21 to S24 below may be omitted.

[0063] First, the acquisition unit 121 acquires a character image (step S21). Next, the encoder 126a extracts image features (step S22). Then, the character count estimation unit 124 uses the image features as input to a character count estimation model to estimate the number of characters (step S23). Note that the processing in step S23 may also be performed by the decoder 126b estimating the number of characters from the image features.

[0064] Then, the decoder 126b estimates the string from the image features extracted from the character image and the estimated number of characters estimated by the character count estimation unit 124 (step S24).

[0065] [Embodiment 4] Embodiment 4 of the information processing device 100 will be described with reference to Figure 13. Figure 13 shows an example configuration of Embodiment 4 of the information processing device 100. Embodiment 4 differs from Embodiment 3 in that the input to the character count estimation unit 124 is a character image, not an image feature. In Embodiment 4, the character count estimation unit 124 takes a character image as input, estimates the number of characters written in the character image using a character count prediction model, and outputs the estimated number of characters. Similar to Embodiment 3, the character count prediction model can be, for example, a machine learning model that estimates the number of characters by regression.

[0066] With the above configuration, for example, a character count prediction model can be pre-trained as a model that predicts the estimated number of characters from a character image, and then the encoder and decoder can be trained with the parameters of the character count prediction model fixed. The processing flow is the same as in Figure 12.

[0067] [Embodiment 5] Embodiment 5 of the information processing device 100 will be described with reference to Figure 14. Figure 14 shows an example configuration of Embodiment 5 of the information processing device 100. Embodiment 5 differs from Embodiment 3 in that the estimated number of characters is not input to the decoder 126b.

[0068] With the above configuration, the character count estimation unit 124 is integrated during training, and the model parameters are learned and optimized so that the estimated character count and string can be correctly estimated. As a result, the encoder 126a outputs image features that have elements that allow it to estimate the character count. This enables the encoder 126a to output image features that take character boundaries into account, improving the accuracy of string prediction.

[0069] [Embodiment 6] Embodiment 6 of the information processing device 100 will be described with reference to Figures 15 and 16. Figure 15 shows an example of the configuration in Embodiment 6 of the information processing device 100. Embodiment 6 differs from Embodiment 3 in that it does not have a character count estimation unit 124, and the character count is estimated by the decoder 126b.

[0070] In Embodiment 6, the decoder 126b takes image features as input and first outputs the estimated number of characters as a character count token. Then, taking the estimated number of characters represented by the image features and character count token as input, it outputs an estimated string.

[0071] Figure 16 shows an example of the operation of the character recognition unit 126 in Embodiment 6. In the character recognition model of Figure 16, the decoder 126b estimates the number of characters, and thus estimates that the number of characters is "2". The decoder 126b then receives a start token. <s>Next, character count token <2> The input is received, and decoder 126b outputs using the estimated number of characters.

[0072] [effect] With the above configuration, the information processing device 100 can predict the number of characters prior to character recognition and predict a string based on the predicted number of characters. As a result, the information processing device 100 performs character count prediction, which requires an overview of the image and recognition of character clusters, prior to predicting a string, thereby achieving character recognition that is aware of character clusters. For this reason, the information processing device 100 can improve the accuracy of character recognition, especially in languages ​​such as Japanese, where characters that are divided into radicals and components become different characters.

[0073] [String estimation using estimated writing direction and estimated number of characters] [overview] The information processing device 100 uses the estimated writing direction and the estimated number of characters to estimate the string using the decoder 126b.

[0074] [Embodiment 7] Embodiment 7 of the information processing device 100 will be described using Figures 17 to 20. Figure 17 shows an example configuration of Embodiment 7 of the information processing device 100. Embodiment 7 is a combination of Embodiment 1 and Embodiment 6. Embodiment 7 differs from Embodiment 1 in the processing of the decoder 126b. In Embodiment 7, the decoder 126b takes the estimated writing direction represented by image features and writing direction tokens as input and first outputs the estimated number of characters as character count tokens. Then, it takes the estimated writing direction represented by image features and writing direction tokens and the estimated number of characters represented by character count tokens as input and outputs an estimated string.

[0075] Figure 18 shows an example of the operation of the character recognition unit 126 in Embodiment 7. In the string estimation model, if the writing direction estimation unit 122 estimates that the writing direction is horizontal, a writing direction token is generated. <h>is the start token <s>This is instead input into the decoder.

[0076] Subsequently, if decoder 126b estimates the number of characters to be "2", the decoder will input a writing direction token. <h>Next, the character count token <2> The input is received, and decoder 126b outputs using the estimated writing direction and estimated number of characters.

[0077] Furthermore, the character recognition unit 126 is able to perform character recognition by training a model like the one shown in equation 4, which takes into account the estimated writing direction d and uses the estimated number of characters n when estimating the generation probability P of the string C={c_1,…,c_T} written on the character image I.

[0078]

number

[0079] Here, Θ is a learnable model parameter. As shown in Figure 19, the model learning unit 125 can optimize the parameters of the model learning unit, which consists of an encoder 126a and a decoder 126b, by using, for example, backpropagation, with training data consisting of a set of character images and corresponding correct character strings, an estimated writing direction derived from the character image by the writing direction estimation unit 122, and the number of correct characters that can be derived from the correct character string.

[0080] [flowchart] Next, the information processing flow in Embodiment 7 will be explained using Figure 20. Note that steps S31 to S36 below can be executed in a different order. Also, some of the steps S31 to S36 below may be omitted.

[0081] First, the acquisition unit 121 acquires a character image (step S31). Next, the writing direction estimation unit 122 uses the character image acquired by the acquisition unit 121 as input to a writing direction estimation model to estimate the writing direction of the characters contained in the character image (step S32).

[0082] Then, the image rotation unit 123 rotates the character image based on the estimated writing direction of the characters contained in the character image, which is estimated by the writing direction estimation unit 122 (step S33). Note that the image rotation unit 123 does not need to rotate the character image if the encoder 126a, decoder 126b, etc., do not assume a rotated image.

[0083] Next, encoder 126a extracts image features from the character image or the rotated character image (step S34). Then, decoder 126b estimates the number of characters from the image features extracted from the character image (step S35). Decoder 126b estimates the string from the image features, the estimated writing direction, and the estimated number of characters (step S36).

[0084] [Embodiment 8] Embodiment 8 of the information processing device 100 will be described with reference to Figures 21 and 22. Figure 21 shows an example configuration of Embodiment 8 of the information processing device 100. Embodiment 8 is a combination of Embodiment 2 and Embodiment 6. Embodiment 8 differs from Embodiment 7 in that the estimated writing direction is not input to the decoder 126b, but is estimated by the decoder 126b. In other words, the decoder 126b estimates the number of characters and the writing direction from the image features and outputs them, and then estimates the string.

[0085] Figure 22 shows an example of the operation of the character recognition unit 126 in Embodiment 8. In the string estimation model, when the decoder 126b estimates that the writing direction is horizontal, a writing direction token is generated. <h>is the start token <s>This is then input to the decoder.

[0086] Subsequently, if decoder 126b estimates the number of characters to be "2", the decoder will input a writing direction token. <h>Next, the character count token <2> The decoder, functioning as decoder 126b, receives the input and outputs using the estimated writing direction and estimated number of characters. Note that the order in which the writing direction token and the number of characters token are output may be reversed.

[0087] [Experimental Results] A verification experiment was conducted on the scene character recognition model with the structure described in Non-Patent Document 1. The target language was Japanese, and approximately 7,800 horizontally written paired data and approximately 700 vertically written paired data were used as training data.

[0088] The character recognition accuracy was evaluated for the baseline from Non-Patent Document 3 as shown in Figure 23(a), the DEM from Non-Patent Document 3 as shown in Figure 23(b), the SAN from Non-Patent Document 3 as shown in Figure 23(c), the model according to Embodiment 1, and the model according to Embodiment 7. For the evaluation, images not included in the training data, approximately 900 horizontally written images and approximately 100 vertically written images were used, and the accuracy rate based on perfect matches was used as the metric.

[0089] The results of the verification experiment are shown in Figure 24. Figure 24 confirms that the recognition accuracy has been improved according to the present invention in both horizontal and vertical writing cases. Figure 25 shows an example of the recognition result. As shown in Figure 25(c), it can be seen that misrecognition is prevented by providing a writing direction token as in Embodiment 1. Furthermore, as shown in Figure 25(d), it can be seen that misrecognition and recognition omissions are prevented by providing a character count token as in Embodiment 7.

[0090] [System configuration, etc.] Furthermore, the components of each illustrated device are functionally conceptual and do not necessarily need to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those shown, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions. For example, each processing function performed by each device can be implemented, in whole or in any part, by a CPU and the program that is analyzed and executed by that CPU, or by hardware using wired logic.

[0091] Furthermore, among the processes described in this embodiment, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by known methods. In addition, the processing procedures, control procedures, specific names, and information including various data and parameters shown in the above document and drawings can be arbitrarily changed unless otherwise specified. Furthermore, the information processing device 100 described in this embodiment may consist only of the learning part to function as a learning device, or it may consist only of the estimation part to function as an estimation device.

[0092] [program] It is also possible to create a program in a computer-executable language that describes the processing performed by the information processing device 100 as described in the above embodiment. In this case, the same effects as in the above embodiment can be obtained by having the computer execute the program. Furthermore, the same processing as in the above embodiment may be achieved by recording such a program on a computer-readable recording medium and having the computer read and execute the program recorded on this recording medium.

[0093] Figure 26 shows an example of a computer that executes an information processing program. As shown in Figure 26, the computer 1000 includes, for example, memory 1010, a CPU 1020, a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0094] Memory 1010 includes ROM (Read Only Memory) 1011 and RAM 1012. ROM 1011 stores, for example, a boot program such as BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to the hard disk drive 1090. The disk drive interface 1040 is connected to the disk drive 1100. The disk drive 1100 is into which removable storage media such as magnetic disks or optical disks are inserted. The serial port interface 1050 is connected to, for example, a mouse 1110 and a keyboard 1120. The video adapter 1060 is connected to, for example, a display 1130.

[0095] Here, as shown in Figure 26, the hard disk drive 1090 stores, for example, the OS 1091, the application program 1092, the program module 1093, and the program data 1094. Each of the tables described in the above embodiment is stored, for example, in the hard disk drive 1090 or the memory 1010.

[0096] Furthermore, the information processing program is stored in the hard disk drive 1090 as a program module containing instructions to be executed by the computer 1000, for example. Specifically, a program module 1093 containing instructions for each process executed by the computer 1000 as described in the above embodiment is stored in the hard disk drive 1090.

[0097] Furthermore, the data used for information processing by the information processing program is stored as program data, for example, in the hard disk drive 1090. The CPU 1020 then reads the program module 1093 and program data 1094 stored in the hard disk drive 1090 into the RAM 1012 as needed and executes the procedures described above.

[0098] Furthermore, the program module 1093 and program data 1094 related to the information processing program are not limited to being stored in the hard disk drive 1090; for example, they may be stored in a removable storage medium and read by the CPU 1020 via a disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 related to the control program may be stored in another computer connected via a network such as a LAN (Local Area Network) or WAN (Wide Area Network) and read by the CPU 1020 via a network interface 1070.

[0099] [others] Although various embodiments have been described in detail herein with reference to the drawings, these embodiments are illustrative and are not intended to limit the present invention to these embodiments. The features described herein can be realized in various ways, including various modifications and improvements based on the knowledge of those skilled in the art.

[0100] Furthermore, the aforementioned "module (-er suffix, -or suffix)" can be reinterpreted as unit, means, circuit, etc. For example, the communication module, control module, and storage module can be reinterpreted as communication unit, control unit, and storage unit, respectively.

[0101] The following additional information is disclosed regarding the embodiments described above.

[0102] (Additional note 1) Memory and At least one processor connected to the memory, Includes, The aforementioned processor, Extract image features from text images, The string is estimated from the writing direction and the aforementioned image features. Information processing device.

[0103] (Additional note 2) An information processing device as described in Appendix 1, The aforementioned processor, Based on the aforementioned image features, the writing direction is estimated and output, and then the string is estimated. Information processing device.

[0104] (Additional note 3) An information processing device as described in Appendix 1, The aforementioned processor, Based on the writing direction and the image features, the number of characters is estimated and output, and then the string is estimated. Information processing device.

[0105] (Additional note 4) An information processing device as described in Appendix 1, The aforementioned processor, Based on the image features, the number of characters and the writing direction are estimated and output, and then the string is estimated. Information processing device.

[0106] (Additional note 5) Memory and At least one processor connected to the memory, Includes, The aforementioned processor, Extract image features from text images, Based on the writing direction and the aforementioned image features, the string is estimated. The system learns a model that performs the following processes: extracting image features from the character image based on the correct string corresponding to the character image and the string; and estimating the string from the writing direction and the image features. Information processing device.

[0107] (Additional note 6) A non-temporary storage medium that stores a program executable by a computer to perform information processing, The aforementioned information processing is, To function as an information processing device as described in any one of the appendices 1 to 5 Non-temporary storage medium.

[0108] (Additional note 7) Memory and At least one processor connected to the memory, Includes, The aforementioned processor, Extract image features from text images, The string is estimated from the aforementioned image features and the number of characters. Information processing device.

[0109] (Additional note 8) Memory and At least one processor connected to the memory, Includes, The aforementioned processor, From a text image, we extract image features that include elements that allow us to estimate the number of characters. The string is estimated from image features that include elements that allow the number of characters to be estimated. Information processing device.

[0110] (Additional note 9) An information processing device as described in Appendix 1, The aforementioned processor, Based on the image features, the number of characters is estimated and output, and then the string is estimated. Information processing device.

[0111] (Additional note 10) Memory and At least one processor connected to the memory, Includes, The aforementioned processor, Extract image features from text images, Based on the aforementioned image features and the number of characters, the string is estimated. The system learns a model that performs the following processes: extracting image features from the character image based on the correct string corresponding to the character image and the string; and estimating the string from the image features and the number of characters. Information processing device.

[0112] (Additional note 11) An information processing device as described in Appendix 10, The aforementioned processor, From the aforementioned image features, the number of characters is estimated. The process involves extracting image features from a character image based on the number of correct characters corresponding to the character image and the number of characters, and estimating a string from the image features and the number of characters. Information processing device.

[0113] (Additional note 12) A non-temporary storage medium that stores a program executable by a computer to perform information processing, The aforementioned information processing is, To function as an information processing device as described in any one of the appendices 7 to 11 Non-temporary storage medium. [Explanation of Symbols]

[0114] 100 Information Processing Devices 110 Communications Department 120 Control Unit 121 Acquisition Department 122 Writing direction estimation unit 123 Image rotation section 124 Character Count Estimation Section 125 Model Learning Section 126 Character recognition section 126a Feature extraction unit 126b Presumption of Text Column 130 Memory Department< / h> < / s> < / h> < / h> < / s> < / h> < / s> < / s> < / n> < / h> < / h> < / s> < / s> < / h> < / v> < / h> < / n> < / s> < / e> < / s> < / v> < / h> < / s>

Claims

1. A feature extraction unit that extracts image features from text images, A string estimation unit that estimates a string from the writing direction and the aforementioned image features. It has, The string estimation unit estimates and outputs the writing direction from the image features, and then estimates the string. An information processing device characterized by the following:

2. A feature extraction unit that extracts image features from a character image, A string estimation unit that estimates a string from the writing direction and the aforementioned image features. It has, The string estimation unit estimates the number of characters from the writing direction and the image features, outputs the result, and then estimates the string. An information processing device characterized by the following:

3. A feature extraction unit that extracts image features from a character image, A string estimation unit that estimates a string from the writing direction and the aforementioned image features. It has, The string estimation unit estimates the number of characters and the writing direction from the image features and outputs them, and then estimates the string. An information processing device characterized by the following:

4. A feature extraction unit that extracts image features from text images, A string estimation unit that estimates a string from the writing direction and the aforementioned image features. Based on the correct string corresponding to the character image and the string, the feature extraction unit and the learning unit learn a model for processing the string estimation unit. It has, The string estimation unit estimates and outputs the writing direction from the image features, and then estimates the string. An information processing device characterized by the following:

5. A method of information processing performed by a computer, A feature extraction process that extracts image features from a text image, A string estimation step that estimates a string from the writing direction and the aforementioned image features. Includes, The string estimation step estimates the writing direction from the image features and outputs it, and then estimates the string. An information processing method characterized by the following:

6. A method of information processing performed by a computer, A feature extraction process that extracts image features from a text image, A string estimation step that estimates a string from the writing direction and the aforementioned image features. Based on the ground truth string corresponding to the character image and the string, a learning step is performed to train a model that performs the feature extraction step and the string estimation step. Includes, The string estimation step estimates the writing direction from the image features and outputs it, and then estimates the string. An information processing method characterized by the following:

7. An information processing program for causing a computer to function as an information processing device according to any one of claims 1 to 4.

Images