Information processing device, information processing method, and information processing program

Abstract

This information processing device (100) has a feature extraction unit (126a) and a character string estimation unit (126b). The feature extraction unit (126a) extracts an image feature from a character image. The character string estimation unit (126b) estimates a character string from a character count and the image feature.

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and an information processing program. [Background technology]

[0002] Scenery images obtained by photographing a landscape contain a lot of text information, such as traffic signs and advertising billboards, that is necessary for understanding the image. Scene character recognition is the task of taking an image (hereinafter referred to as a character image) extracted from such a scene image as input, recognizing the characters in the image, and converting it into a string of characters that can be processed by a machine. In recent years, advances in deep learning technology have led to the proposal of a method for realizing scene character recognition using a one-stop model. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] F. Sheng, Z. Chen, and B. Xu, “NRTR: A no-recurrence sequence-to-sequence model for scene text recognition,” Proceedings of the IEEE International Conference on Document Analysis and Recognition (ICDAR), pp. 781-786, 2019. [Non-patent document 2] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, AN Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in Neural Information Processing Systems (NIPS), pp. 5998-6008, 2017. [Non-patent document 3] C. Choi, Y. Yoon, J. Lee, and J. Kim, “Simultaneous recognition of horizontal and vertical text in natural images,” in Proceedings of the International Workshop on Robust Reading, ACCV, 2019, pp. 202-212.

[0004] For example, Non-Patent Document 1 provides a scene character recognition technology using a model consisting of an encoder and a decoder, as schematically shown in Figure 1. In this case, the encoder is composed of, for example, a part that extracts features of character images using a convolutional neural network, and a part that converts them into features that take sequence into account using a Transformer encoder provided in Non-Patent Document 2. The decoder is composed of, for example, an embedding layer, a Transformer decoder provided in Non-Patent Document 2, and an autoregressive model using an output layer, and outputs the probability of generating a character string from the character image features (hereinafter referred to as image features) extracted by the encoder. Using this model, the generation probability P of a character string C={c_1,...,c_T} written in a character image I is modeled as follows: where Θ is a learnable model parameter.

[0005]

number

[0006] However, conventional techniques have the problem of misrecognizing characters that consist of multiple parts. Because Japanese kanji characters have radicals and side characters, when recognizing characters from left to right, it is difficult to determine whether to recognize the radical and side character separately or to recognize them as a single character.

[0007] For example, when a character image written horizontally and reading "Yokohama" is given, it is difficult to distinguish and recognize whether it should be recognized as "wood yellow soldier" or "Yokohama". Similar problems also occur in languages where characters that become different characters when divided left and right exist. Also, similar problems occur in languages where characters that become different characters when divided vertically exist when learning and using a model for recognizing vertically written character strings.

[0008] Thus, as an example, the problem of misrecognizing characters composed of multiple parts can be cited as one of the problems to be solved by the present invention.

Means for Solving the Problems

[0009] In order to solve the above-described problems and achieve the object, an information processing apparatus according to the present invention includes a feature extraction unit and a character string estimation unit. The feature extraction unit extracts image features from a character image. The character string estimation unit estimates a character string from the number of characters and the image features.

[0010] Also, an information processing apparatus according to the present invention includes a feature extraction unit and a character string estimation unit. The feature extraction unit extracts image features including an element capable of estimating the number of characters. The character string estimation unit estimates a character string from the image features including an element capable of estimating the number of characters.

[0011] Also, an information processing apparatus according to the present invention includes a feature extraction unit, a character string estimation unit, and a learning unit. The feature extraction unit extracts image features from a character image. The character string estimation unit estimates a character string from the image features and the number of characters. The learning unit learns a model for performing the processing of the feature extraction unit and the character string estimation unit based on the correct character string corresponding to the character image and the character string.

Effects of the Invention

[0012] According to the present invention, it becomes possible to accurately recognize characters composed of multiple parts.

Brief Description of the Drawings

[0013] [Figure 1]FIG. 1 is a diagram showing a character recognition model of the prior art. [Figure 2] FIG. 2 is a block diagram illustrating an example of the configuration of an information processing device. [Figure 3] FIG. 3 is a diagram illustrating an example of the configuration of an information processing device at the time of estimation. [Figure 4] FIG. 4 is a diagram illustrating an example of processing by the information processing device. [Figure 5] FIG. 5 is a diagram illustrating an example of the configuration of an information processing device during learning. [Figure 6] FIG. 6 is a flowchart showing an example of the flow of processing by the information processing device. [Figure 7] FIG. 7 is a diagram illustrating an example of the configuration of an information processing device at the time of estimation. [Figure 8] FIG. 8 is a diagram illustrating an example of processing by the information processing device. [Figure 9] FIG. 9 is a diagram illustrating an example of the configuration of an information processing device at the time of estimation. [Figure 10] FIG. 10 is a diagram illustrating an example of processing by the information processing device. [Figure 11] FIG. 11 is a diagram illustrating an example of the configuration of an information processing device during learning. [Figure 12] FIG. 12 is a flowchart showing an example of the flow of processing by the information processing device. [Figure 13] FIG. 13 is a diagram illustrating an example of the configuration of an information processing device at the time of estimation. [Figure 14] FIG. 14 is a diagram illustrating an example of the configuration of an information processing device. [Figure 15] FIG. 15 is a diagram illustrating an example of the configuration of an information processing device at the time of estimation. [Figure 16] FIG. 16 is a diagram illustrating an example of processing by the information processing device. [Figure 17] FIG. 17 is a diagram illustrating an example of the configuration of an information processing device at the time of estimation. [Figure 18] FIG. 18 is a diagram illustrating an example of processing by the information processing device. [Figure 19] FIG. 19 is a diagram illustrating an example of the configuration of an information processing device during learning. [Figure 20]FIG. 20 is a flowchart showing an example of the flow of processing by the information processing device. [Figure 21] FIG. 21 is a diagram illustrating an example of the configuration of an information processing device at the time of estimation. [Figure 22] FIG. 22 is a diagram illustrating an example of processing by the information processing device. [Figure 23] FIG. 23 is a diagram showing a character recognition model according to the prior art. [Figure 24] FIG. 24 is a table showing the character string estimation results obtained by the information processing device. [Figure 25] FIG. 25 is a diagram showing a character string estimation result by the information processing device. [Figure 26] FIG. 26 is a diagram illustrating an example of a computer that executes an information processing program. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, with reference to the drawings, an information processing device, an information processing method, and an information processing program according to the present application will be described in detail. Note that the present invention is not limited to these embodiments. In addition, in the description of the drawings, the same parts are denoted by the same reference numerals, and duplicated explanations will be omitted.

[0015] [Summary of the Invention] The information processing device 100 according to this embodiment uses the results of estimating the writing direction and the number of characters in character string estimation using an encoder-decoder model, thereby achieving highly accurate character string estimation.

[0016] For example, in character recognition, the information processing device 100 shares all model parameters between horizontal and vertical writing, thereby sharing character-specific contours and vocabulary useful for character recognition between horizontal and vertical writing. Then, to correctly decode each of the horizontal and vertical writings, tokens that distinguish between horizontal and vertical writing are provided as initial values ​​for the autoregressive decoder, thereby achieving highly accurate character string estimation. In this case, the present invention can be applied to any technology that outputs a character string from a character image through an arbitrary encoder-decoder model having an autoregressive decoder. It can also be applied to optical character recognition, etc.

[0017] Furthermore, for example, the information processing device 100 predicts the number of characters written in a character image prior to the process of predicting a character string, and outputs the character string based on the prediction result. This allows the number of characters that need to be captured from a bird's-eye view of the character image to be predicted prior to the process of predicting a character string. In other words, the character is recognized after capturing a group of characters. This prevents characters from being recognized by mistakenly splitting or combining radicals and radicals, and improves the accuracy of character string estimation. In this case, the present invention can be applied to general technologies that output character strings from character images through any end-to-end sequence-to-sequence model. It can also be applied to optical character recognition, etc.

[0018] [Configuration of information processing device] First, the configuration of the information processing device will be described with reference to Fig. 2. As shown in Fig. 2, the information processing device 100 has a communication unit 110, a control unit 120, and a storage unit 130. Note that these units may be distributed and held by multiple devices. The processing of these units will be described below.

[0019] The communication unit 110 is realized by a NIC (Network Interface Card) or the like, and enables communication between an external device and the control unit 120 via a telecommunication line such as a LAN (Local Area Network) or the Internet. For example, the communication unit 110 enables communication between the external device and the control unit 120.

[0020] The storage unit 130 is realized by a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, or a storage device such as a hard disk or an optical disk. Information stored in the storage unit 130 includes, for example, character images, image features, data related to machine learning algorithms, training data, trained models, etc. Note that the information stored in the storage unit 130 is not limited to the above.

[0021] The control unit 120 is realized using a CPU (Central Processing Unit), an NP (Network Processor), an FPGA (Field Programmable Gate Array), or the like, and executes a processing program stored in a memory. As shown in Fig. 2, the control unit 120 has an acquisition unit 121, a writing direction estimation unit 122, an image rotation unit 123, a character number estimation unit 124, a model learning unit (learning unit) 125, a character recognition unit 126, an encoder (feature extraction unit) 126a, and a decoder (character string estimation unit) 126b. Each unit of the control unit 120 will be described below.

[0022] The division of functional units in the configuration diagram is an example, and the system may be realized by only some of the functional units, multiple functional units may be realized as one functional unit, one functional unit may be divided into multiple units, or some functions may be transferred to other functional units.Furthermore, the functions of multiple functional units having similar functions may be processed in parallel or in time-sharing by a single piece of hardware or software.

[0023] The acquisition unit 121 acquires a character image. The writing direction estimation unit 122 uses the character image acquired by the acquisition unit 121 as an input to a model for estimating the writing direction (hereinafter, writing direction estimation model), estimates the writing direction, and outputs the estimated writing direction.

[0024] For example, the writing direction estimation unit 122 may use a writing direction estimation model that determines whether a character image is horizontally long or vertically long based on the aspect ratio of the character image. Furthermore, for example, the writing direction estimation unit 122 may define a determination model that receives character images and image features as input and outputs an estimated writing direction using a machine learning model, and may use the determination model as the writing direction estimation model after learning it in advance using training data.

[0025] The writing direction handled by the information processing device 100 may be any direction that represents how characters are read, including inversion, rotation, etc., in addition to vertical writing and horizontal writing. For example, the information processing device 100 may use all combinations of "vertical writing or horizontal writing, inversion or non-inversion, and counterclockwise rotation of 0 degrees, 90 degrees, 180 degrees, or 270 degrees" as the writing direction.

[0026] Image rotation unit 123 receives a character image and an estimated writing direction as input, rotates the character image to an orientation assumed by character recognition unit 126, and outputs the rotated character image. For example, in the case of horizontal and vertical writing, image rotation unit 123 outputs the character image as is for horizontal writing, and outputs an image obtained by rotating the character image 90 degrees counterclockwise for vertical writing as the rotated character image. Note that image rotation unit 123 can be omitted if character recognition unit 126 is assumed to receive an unrotated character image as input.

[0027] The number-of-characters estimation unit 124 uses the character images acquired by the acquisition unit 121 or the image features extracted by the encoder 126a as input to a model for estimating the number of characters (hereinafter referred to as the character-number estimation model) to estimate the number of characters and output the estimated number of characters. For example, the number-of-characters estimation unit 124 uses the character images as input to the character-number estimation model to estimate the number of characters and output the estimated number of characters. Also, for example, the number-of-characters estimation unit 124 uses the image features as input to the character-number estimation model to output the estimated number of characters.

[0028] The model learning unit 125 learns a model (hereinafter referred to as a character recognition model) that performs the processing of the encoder 126a and the decoder 126b based on a correct character string corresponding to a character image and an estimated character string (hereinafter referred to as an estimated character string). The model learning unit 125 also learns a character recognition model and a character number estimation model based on the number of correct characters corresponding to the character image and the estimated number of characters. For example, the model learning unit 125 extracts image features from a rotated character image and learns a character recognition model that estimates a character string from the image features and an estimated writing direction.

[0029] Also, for example, the model learning unit 125 extracts image features from a character image, and learns a character recognition model that estimates a character string from the image features and the estimated number of characters, and a character number estimation model that estimates the number of characters from the image features.

[0030] Also, for example, the model learning unit 125 extracts image features from the rotated character image, estimates the number of characters from the image features and the estimated writing direction, and learns a character recognition model that estimates a character string from the image features, the estimated writing direction, and the estimated number of characters.

[0031] The character recognition unit 126 is composed of an encoder 126a and a decoder 126b. The encoder 126a extracts image features from a character image. For example, the encoder 126a extracts image features from a character image acquired by the acquisition unit 121. For example, the encoder 126a also extracts image features from a rotated character image. For example, the encoder 126a also extracts image features from the character image, including elements that can estimate the number of characters. Here, the encoder 126a extracts features that take sequence into consideration using, for example, a convolutional neural network and a Transformer encoder.

[0032] The decoder 126b outputs an estimated character string from the image features. The decoder 126b generates an output recursively. For example, the decoder 126b estimates a character string from the image features and an estimated writing direction, and outputs the estimated character string. For example, the decoder 126b receives the image features and the estimated writing direction output from the writing direction estimation unit 122 as input, estimates a character string, and outputs the estimated character string. Furthermore, for example, the decoder 126b estimates a writing direction from the image features, outputs the estimated writing direction, estimates a character string, and outputs the estimated character string.

[0033] At this time, the writing direction estimation unit 122 or the decoder 126b may, for example, add a writing direction token, which is a special token that indicates an estimated writing direction, to the start token. <s>For example, the information processing apparatus 100 may input horizontal writing as a writing direction token to the decoder 126b instead of the <h>, vertical writing <v>The writing direction token is defined as follows. The writing direction token is registered in the dictionary in advance, just like other tokens. <s>is the initial value token of the decoding by the decoder 126b. <e>is a token indicating the end of decoding by the decoder 126b.

[0034] Also, for example, decoder 126b estimates a character string from image features and the estimated number of characters, and outputs the estimated character string. For example, decoder 126b receives as input image features and the estimated number of characters output from character number estimation unit 124, estimates a character string, and outputs the estimated character string. Also, for example, decoder 126b estimates a character string from image features including elements from which the number of characters can be estimated, and outputs the estimated character string. Also, for example, decoder 126b estimates the number of characters from image features, outputs the estimated number of characters, estimates a character string, and outputs the estimated character string.

[0035] Here, the estimated number of characters is converted into a character count token that represents the number of characters, and then the start token <s>At this time, the information processing device 100 may input the character count token as, for example, <n>It is assumed that the information processing device 100 registers the character count token in advance in a dictionary, similar to other tokens.

[0036] Also, for example, the decoder 126b estimates a character string from the image feature, the estimated writing direction, and the estimated number of characters, and outputs the estimated character string. For example, the decoder 126b receives as input the image feature, the estimated writing direction output from the writing direction estimation unit 122, and the estimated number of characters output by the decoder 126b, and outputs an estimated character string. Also, for example, the decoder 126b estimates and outputs the number of characters and the writing direction from the image feature, and then estimates a character string and outputs the estimated character string.

[0037] [String estimation using estimated writing direction] [overview] The information processing device 100 uses the estimated writing direction to estimate the character string using the decoder 126b.

[0038] [Embodiment 1] A first embodiment of the information processing device 100 will be described with reference to Figs. 3 to 6. Fig. 3 shows an example of the configuration of the first embodiment of the information processing device 100. The information processing device 100 is made up of a writing direction estimation unit 122, an image rotation unit 123, and a character recognition unit 126. The character recognition unit 126 is made up of an encoder 126a and a decoder 126b.

[0039] The writing direction estimation unit 122 receives a character image, estimates the writing direction, and outputs the estimated writing direction. The image rotation unit 123 receives a character image and the estimated writing direction, rotates the character image to an orientation assumed by the character recognition unit 126, and outputs the rotated character image.

[0040] The encoder 126a receives the rotated character image as input and outputs image features. The decoder 126b receives the image features and the estimated writing direction output from the writing direction estimation unit 122 as input and outputs an estimated character string. Here, the estimated writing direction output by the writing direction estimation unit 122 is converted into a special token (writing direction token) that represents the estimated writing direction, for example, and then input to the decoder in place of the start token. The writing direction token, for example, represents horizontal writing. <h>, vertical writing <v>The writing direction token, like other tokens, must be pre-registered in the dictionary.

[0041] 4 shows an example of the operation of the character recognition unit 126 in the first embodiment. When the writing direction estimation unit 122 estimates that the character is written horizontally, a writing direction token <h>is the start token <s>is input to the decoder 126b instead of the input image, the decoder 126b recognizes that the input image is written horizontally and correctly decodes the character string.

[0042] The character recognition unit 126 is able to recognize characters by having the model learning unit 125 learn a model such as Equation 2, which takes into account the estimated writing direction d, when estimating the generation probability P of the character string C={c_1, ..., c_T} written in the character image I.

[0043]

number

[0044] 5, the model learning unit 125 can optimize the parameters of the character recognition model composed of the encoder 126a and the decoder 126b by, for example, backpropagation using a set of correct character strings corresponding to character images and estimated writing directions derived by the writing direction estimation unit 122 as training data.

[0045] [flowchart] Next, the flow of information processing by the information processing device 100 will be described with reference to Fig. 6. Note that the following steps S11 to S15 may be executed in a different order. Also, some of the following steps S11 to S15 may be omitted.

[0046] First, the acquiring unit 121 acquires a character image (step S11). Next, the writing direction estimating unit 122 uses the character image acquired by the acquiring unit 121 as an input to a writing direction estimation model to estimate the writing direction of a character included in the character image (step S12).

[0047] Then, the image rotation unit 123 rotates the character image based on the estimated writing direction of the character included in the character image estimated by the writing direction estimation unit 122 (step S13). Note that the image rotation unit 123 does not need to rotate the character image when the rotated image is not assumed in the encoder 126a, the decoder 126b, or the like.

[0048] Then, the encoder 126a extracts image features from the character image (step S14). For example, the encoder 126a extracts image features from the character image acquired by the acquisition unit 121. Also, for example, the encoder 126a extracts image features from character information included in the rotated character image rotated by the image rotation unit 123.

[0049] Then, the decoder 126b estimates the character string from the image features extracted by the encoder 126a and the estimated writing direction (step S15).

[0050] [effect] With the above-described configuration, the information processing device 100 can efficiently model character recognition that can recognize both horizontally and vertically written characters. Specifically, the information processing device 100 can share all model parameters between horizontal and vertical writing, thereby sharing character-specific contours and vocabulary useful for character recognition between horizontal and vertical writing. Furthermore, the information processing device 100 can correctly decode character strings in horizontal and vertical writing by providing a writing direction token that distinguishes between horizontal and vertical writing as the initial value of the autoregressive decoder.

[0051] [Embodiment 2] A second embodiment of the information processing device 100 will be described with reference to Figures 7 and 8. The second embodiment differs from the first embodiment in that an estimated writing direction is not input to the decoder 126b, but the decoder 126b estimates the writing direction. In other words, the decoder 126b estimates the writing direction from image features, outputs the estimated writing direction, and then estimates the character string.

[0052] 7 shows an example of the configuration of the information processing device 100 according to the second embodiment. The decoder 126b according to the second embodiment receives an image feature as input, and first outputs the writing direction estimated by the decoder 126b as a writing direction token. Then, the decoder 126b receives the image feature and the writing direction estimated by the decoder 126b represented by the writing direction token as input, and outputs an estimated character string.

[0053] 8 shows an example of the operation of the character recognition unit 126 in the second embodiment. When the writing direction estimation unit 122 estimates that the character is written horizontally, the start token <s>When input, the decoder 126b first estimates the writing direction and generates a writing direction token as the estimated writing direction. <h>Then, the writing direction token <h>is input to the decoder 126b, the decoder 126b correctly decodes the character string, taking into account that the input image is written horizontally.

[0054] The processing of the second embodiment is the same as that of FIG. 6, but differs in that the decoder 126b does not input an estimated writing direction, but instead obtains a character string including a writing direction token as an output.

[0055] [String estimation using estimated number of characters] [overview] The information processing device 100 uses the estimated number of characters to estimate the character string using the decoder 126b.

[0056] [Embodiment 3] A third embodiment of the information processing device 100 will be described with reference to Figs. 9 to 12. Fig. 9 shows an example of the configuration of the third embodiment of the information processing device 100. The information processing device 100 is made up of a number-of-characters estimation unit 124 and a character recognition unit 126. The character recognition unit 126 is made up of an encoder 126a and a decoder 126b.

[0057] The encoder 126a receives a character image as input and outputs image features. The decoder 126b receives the image features and the estimated number of characters output from the number-of-characters estimation unit 124 as input and outputs an estimated character string.

[0058] Here, the estimated number of characters output by the number-of-characters estimation unit 124 is converted into a special token (character number token) that represents the number of characters, and then input to the decoder instead of the start token. The character number token is, for example, expressed as follows, where n is the estimated number of characters: <n>The character count token is defined as follows: The character count token is registered in the dictionary in advance, just like other tokens.

[0059] 10 shows an example of the operation of the character recognition unit 126 in the third embodiment. In the character recognition model of FIG. 10, when the character number estimation unit 124 estimates that the number of characters is "2", the character number token <2> is the start token <s>is input to decoder 126b instead of , which then outputs the estimated string.

[0060] In addition, the information processing device 100 is capable of character recognition by having the model learning unit 125 learn a model such as Equation 3, which uses the estimated number of characters n, to estimate the generation probability P of the string C={c_1, ..., c_T} written in the character image I.

[0061]

number

[0062] 11, the model learning unit 125 can optimize the parameters of the character recognition model composed of the encoder 126a and the decoder 126b and the parameters of the character number estimation model by, for example, the backpropagation method using a set of character images, corresponding correct character strings, and the correct number of characters that can be derived from the correct character strings as training data.

[0063] [flowchart] Next, the flow of information processing in the third embodiment will be described with reference to Fig. 12. Note that the following steps S21 to S24 may be executed in a different order. Also, some of the following steps S21 to S24 may be omitted.

[0064] First, the acquisition unit 121 acquires a character image (step S21). Next, the encoder 126a extracts image features (step S22). Then, the number-of-characters estimation unit 124 estimates the number of characters using the image features as input to a character-number estimation model (step S23). Note that the process of step S23 may be performed by the decoder 126b estimating the number of characters from the image features.

[0065] Then, the decoder 126b estimates a character string from the image features extracted from the character image and the estimated number of characters estimated by the number-of-characters estimation unit 124 (step S24).

[0066] [Embodiment 4] A fourth embodiment of the information processing device 100 will be described with reference to FIG. 13. FIG. 13 shows an example of the configuration of the fourth embodiment of the information processing device 100. The fourth embodiment differs from the third embodiment in that the input to the number-of-characters estimation unit 124 is a character image rather than an image feature. The number-of-characters estimation unit 124 in the fourth embodiment receives a character image as input, estimates the number of characters written in the character image using a number-of-characters prediction model, and outputs the estimated number of characters. As in the third embodiment, the number-of-characters prediction model can be, for example, a machine learning model that estimates the number of characters by regression.

[0067] The above configuration enables two-stage learning, for example, by pre-learning a character number prediction model as a model that predicts the estimated number of characters from a character image, and then learning the encoder and decoder with the parameters of the character number prediction model fixed. The processing flow is the same as in Figure 12.

[0068] [Embodiment 5] A fifth embodiment of the information processing device 100 will be described with reference to Fig. 14. Fig. 14 shows an example of the configuration of the fifth embodiment of the information processing device 100. The fifth embodiment differs from the third embodiment in that the estimated number of characters is not input to the decoder 126b.

[0069] With the above configuration, by combining the number-of-characters estimation unit 124 during learning and learning and optimizing the model parameters so that the estimated number of characters and character strings can be correctly estimated, the encoder 126a outputs image features that have elements that allow the number of characters to be estimated. This enables the encoder 126a to output image features that take character breaks into consideration, improving the accuracy of character string prediction.

[0070] [Embodiment 6] A sixth embodiment of the information processing device 100 will be described with reference to Figs. 15 and 16. Fig. 15 shows an example of the configuration of the sixth embodiment of the information processing device 100. The sixth embodiment differs from the third embodiment in that it does not have the number-of-characters estimation unit 124, and instead estimates the number of characters using the decoder 126b.

[0071] The decoder 126b in the sixth embodiment receives an image feature as input, and first outputs an estimated number of characters as a character count token. Then, the decoder 126b receives the image feature and the estimated number of characters represented by the character count token as input, and outputs an estimated character string.

[0072] Fig. 16 shows an example of the operation of the character recognition unit 126 in the sixth embodiment. In the character recognition model of Fig. 16, the decoder 126b estimates the number of characters, and thus estimates that the number of characters is "2". Then, the decoder 126b receives a start token <s>followed by a character count token <2> is input, and the decoder 126b outputs using the estimated number of characters.

[0073] [effect] With the above configuration, the information processing device 100 can predict the number of characters prior to character recognition and predict a character string based on the predicted number of characters. This allows the information processing device 100 to predict the number of characters, which requires a bird's-eye view of the image and recognition of character groups, prior to predicting a character string, thereby realizing character recognition that takes character groups into consideration. Therefore, the information processing device 100 improves the accuracy of character recognition, particularly in languages ​​such as Japanese, which contain characters that become different characters when separated, such as radicals and side characters.

[0074] [String estimation using estimated writing direction and estimated number of characters] [overview] The information processing device 100 uses the estimated writing direction and the estimated number of characters to estimate the character string using the decoder 126b.

[0075] [Embodiment 7] A seventh embodiment of the information processing device 100 will be described with reference to Figs. 17 to 20. Fig. 17 shows an example of the configuration of the seventh embodiment of the information processing device 100. The seventh embodiment is a combination of the first and sixth embodiments. The seventh embodiment differs from the first embodiment in the processing of the decoder 126b. The decoder 126b in the seventh embodiment receives as input an image feature and an estimated writing direction represented by a writing direction token, and first outputs an estimated number of characters as a character count token. Then, it receives as input an image feature, an estimated writing direction represented by a writing direction token, and an estimated number of characters represented by a character count token, and outputs an estimated character string.

[0076] 18 shows an example of the operation of the character recognition unit 126 in the seventh embodiment. In the character string estimation model, when the writing direction estimation unit 122 estimates that the writing direction is horizontal, a writing direction token <h>is the start token <s>is input to the decoder instead of

[0077] Thereafter, if the decoder 126b estimates that the number of characters is "2", the decoder receives a writing direction token <h>followed by a character count token <2> is input, and the decoder 126b performs output using the estimated writing direction and estimated number of characters.

[0078] In addition, the character recognition unit 126 is able to recognize characters by having the model learning unit 125 learn a model such as Equation 4, which takes into account the estimated writing direction d and the estimated number of characters n, when estimating the generation probability P of the character string C={c_1, ..., c_T} written in the character image I.

[0079]

number

[0080] 19, the model learning unit 125 can optimize the parameters of the model learning unit composed of the encoder 126a and the decoder 126b by, for example, backpropagation using a set of training data, which includes a correct character string corresponding to the character image, an estimated writing direction derived from the character image by the writing direction estimation unit 122, and the number of correct characters that can be derived from the correct character string.

[0081] [flowchart] Next, the flow of information processing in the seventh embodiment will be described with reference to Fig. 20. Note that the following steps S31 to S36 may be executed in a different order. Also, some of the following steps S31 to S36 may be omitted.

[0082] First, the acquiring unit 121 acquires a character image (step S31). Next, the writing direction estimating unit 122 uses the character image acquired by the acquiring unit 121 as an input to a writing direction estimation model to estimate the writing direction of a character included in the character image (step S32).

[0083] Then, the image rotation unit 123 rotates the character image based on the estimated writing direction of the character included in the character image estimated by the writing direction estimation unit 122 (step S33). Note that the image rotation unit 123 does not need to rotate the character image when the rotated image is not assumed in the encoder 126a, the decoder 126b, or the like.

[0084] Next, the encoder 126a extracts image features from the character image or the rotated character image (step S34). The decoder 126b then estimates the number of characters from the image features extracted from the character image (step S35). The decoder 126b estimates a character string from the image features, the estimated writing direction, and the estimated number of characters (step S36).

[0085] [Embodiment 8] An eighth embodiment of the information processing device 100 will be described with reference to Figs. 21 and 22. Fig. 21 shows an example of the configuration of the eighth embodiment of the information processing device 100. The eighth embodiment is a combination of the second embodiment and the sixth embodiment. The eighth embodiment differs from the seventh embodiment in that an estimated writing direction is not input to the decoder 126b, but the decoder 126b estimates the writing direction. In other words, the decoder 126b estimates the number of characters and the writing direction from image features, outputs them, and then estimates the character string.

[0086] 22 shows an example of the operation of the character recognition unit 126 in the eighth embodiment. In the character string estimation model, when the decoder 126b estimates that the writing direction is horizontal, a writing direction token <h>is the start token <s>is then input to the decoder.

[0087] Thereafter, if the decoder 126b estimates that the number of characters is "2", the decoder receives a writing direction token <h>followed by a character count token <2> The decoder that functions as decoder 126b receives the input of the estimated writing direction and the estimated number of characters and outputs the token. Note that the order in which the writing direction token and the number of characters token are output may be reversed.

[0088] [Experimental Results] A verification experiment was conducted on a scene character recognition model with the structure described in Non-Patent Document 1. The target language was Japanese, and approximately 7,800 pairs of horizontally written data and approximately 700 pairs of vertically written data were used as training data.

[0089] Character recognition accuracy was evaluated for the baseline of Non-Patent Document 3 as shown in Figure 23(a), the DEM of Non-Patent Document 3 as shown in Figure 23(b), the SAN of Non-Patent Document 3 as shown in Figure 23(c), the modeling according to embodiment 1, and the modeling according to embodiment 7. For the evaluation, images not included in the training data, approximately 900 images written horizontally and approximately 100 images written vertically, were used, and the accuracy rate based on exact matches was used as a measure.

[0090] The results of the verification experiment are shown in Figure 24. Figure 24 confirms that the present invention improves recognition accuracy in both horizontal and vertical writing. Figure 25 shows an example of the recognition result. As shown in Figure 25(c), it can be seen that misrecognition is prevented by providing a writing direction token as in embodiment 1. Furthermore, as shown in Figure 25(d), it can be seen that misrecognition and recognition omissions are prevented by providing a character count token as in embodiment 7.

[0091] [System configuration, etc.] Furthermore, the components of each device shown in the figure are conceptual functional units and do not necessarily have to be physically configured as shown. In other words, the specific form of distribution and integration of each device is not limited to that shown, and all or part of each device can be functionally or physically distributed and integrated in any unit depending on various loads and usage conditions. For example, each processing function performed by each device can be realized, in whole or in part, by a CPU and a program analyzed and executed by the CPU, or can be realized as hardware using wired logic.

[0092] Furthermore, among the processes described in this embodiment, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using known methods. In addition, the processing procedures, control procedures, specific names, and information including various data and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified. Furthermore, the information processing device 100 described in this embodiment may be configured only with parts related to learning and serve as a learning device, or may be configured only with parts related to estimation and serve as an estimation device.

[0093] [program] It is also possible to create a program written in a computer-executable language that executes the processes executed by the information processing device 100 described in the above embodiment. In this case, the same effects as those of the above embodiment can be achieved by having a computer execute the program. Furthermore, such a program may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read and executed by a computer to achieve the same processes as those of the above embodiment.

[0094] Fig. 26 is a diagram showing an example of a computer that executes an information processing program. As shown in Fig. 26, a computer 1000 includes, for example, a memory 1010, a CPU 1020, a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0095] The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM 1012. The ROM 1011 stores, for example, a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to, for example, a mouse 1110 and a keyboard 1120. The video adapter 1060 is connected to, for example, a display 1130.

[0096] 26, the hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. The tables described in the above embodiments are stored in the hard disk drive 1090 or the memory 1010, for example.

[0097] The information processing program is stored in the hard disk drive 1090 as a program module in which instructions to be executed by the computer 1000 are written. Specifically, the hard disk drive 1090 stores a program module 1093 in which each process executed by the computer 1000 described in the above embodiment is written.

[0098] Furthermore, data used for information processing by the information processing program is stored as program data, for example, in the hard disk drive 1090. Then, the CPU 1020 reads out the program module 1093 and program data 1094 stored in the hard disk drive 1090 into the RAM 1012 as necessary, and executes each of the above-described procedures.

[0099] The program module 1093 and program data 1094 related to the information processing program are not limited to being stored in the hard disk drive 1090, but may be stored in a removable storage medium, for example, and read by the CPU 1020 via the disk drive 1100. Alternatively, the program module 1093 and program data 1094 related to the control program may be stored in another computer connected via a network such as a LAN (Local Area Network) or a WAN (Wide Area Network), and read by the CPU 1020 via the network interface 1070.

[0100] [others] Although various embodiments have been described in detail herein with reference to the drawings, these embodiments are merely examples and are not intended to limit the present invention. The features described herein can be realized in various ways, including various modifications and improvements based on the knowledge of those skilled in the art.

[0101] Furthermore, the above-mentioned "module (-er suffix, -or suffix)" can be read as a unit, means, circuit, etc. For example, a communication module, a control module, and a storage module can be read as a communication unit, a control unit, and a storage unit, respectively.

[0102] The following additional notes are provided regarding the above-described embodiments.

[0103] (Additional note 1) Memory and at least one processor coupled to said memory; Including, The processor: Extract image features from character images, The character string is estimated from the writing direction and the image features. Information processing device.

[0104] (Additional note 2) The information processing device according to claim 1, The processor: The writing direction is estimated and output from the image features, and then the character string is estimated. Information processing device.

[0105] (Additional note 3) The information processing device according to claim 1, The processor: The number of characters is estimated and output from the writing direction and the image features, and then the character string is estimated. Information processing device.

[0106] (Additional note 4) The information processing device according to claim 1, The processor: The number of characters and the writing direction are estimated and output from the image features, and then the character string is estimated. Information processing device.

[0107] (Additional note 5) Memory and at least one processor coupled to said memory; Including, The processor: Extract image features from character images, A character string is estimated from the writing direction and the image features; A model is trained that performs a process of extracting image features from the character image based on the character string and a correct answer character string corresponding to the character image, and a process of estimating the character string from the writing direction and the image features. Information processing device.

[0108] (Additional note 6) A non-transitory storage medium storing a program executable by a computer to perform information processing, The information processing For functioning as the information processing device according to any one of appended items 1 to 5 Non-temporary storage medium.

[0109] (Additional note 7) Memory and at least one processor coupled to said memory; Including, The processor: Extract image features from character images, A character string is estimated from the image features and the number of characters. Information processing device.

[0110] (Additional note 8) Memory and at least one processor coupled to said memory; Including, The processor: Extract image features from the character image, including elements that can estimate the number of characters, A character string is estimated from image features including elements that can estimate the number of characters. Information processing device.

[0111] (Additional note 9) The information processing device according to claim 1, The processor: The number of characters is estimated from the image features, and the character string is estimated after outputting the number of characters. Information processing device.

[0112] (Additional note 10) Memory and at least one processor coupled to said memory; Including, The processor: Extract image features from the character image, A character string is estimated from the image features and the number of characters; A model is trained that performs a process of extracting image features from the character image based on the character string and a correct answer character string corresponding to the character image, and a process of estimating the character string based on the image features and the number of characters. Information processing device.

[0113] (Additional note 11) The information processing device according to claim 10, The processor: Estimating the number of characters from the image features; A process of extracting image features from the character image based on the number of correct characters corresponding to the character image and the number of characters; and estimating a character string from the image features and the number of characters. Information processing device.

[0114] (Additional note 12) A non-transitory storage medium storing a program executable by a computer to perform information processing, The information processing For functioning as the information processing device according to any one of appended claims 7 to 11 Non-temporary storage medium. [Explanation of symbols]

[0115] 100 Information processing device 110 Communications Department 120 control section 121 Acquisition Department 122 Writing direction estimation section 123 Image Rotation Unit 124 Character Count Estimation Section 125 Model Learning Department 126 Character recognition section 126a Feature Extraction Unit 126b Presumption of Text Column 130 Memory Department< / h> < / s> < / h> < / h> < / s> < / h> < / s> < / s> < / n> < / h> < / h> < / s> < / s> < / h> < / v> < / h> < / n> < / s> < / e> < / s> < / v> < / h> < / s>

Claims

1. A writing direction estimation unit that estimates the writing direction of a character image from the character image; a feature extraction unit that extracts image features from the character image; a character string estimation unit that estimates a character string from the image features, the number of characters, and the writing direction; An information processing device comprising:

2. A writing direction estimation unit that estimates the writing direction of a character image from the character image; a feature extraction unit that extracts image features from the character image, the image features including elements that allow the number of characters to be estimated; a character string estimation unit that estimates a character string from image features including elements that can estimate the number of characters and the writing direction; An information processing device comprising:

3. The information processing apparatus according to claim 1 , wherein the character string estimation unit estimates the number of characters from the image features, outputs the estimated number of characters, and then estimates the character string.

4. A writing direction estimation unit that estimates a writing direction of a character image from the character image; a feature extraction unit that extracts image features from the character image; a character string estimation unit that estimates a character string from the image features, the number of characters, and the writing direction; a learning unit that learns a model that performs processing of the feature extraction unit and the character string estimation unit based on the character string and a correct character string corresponding to the character image; An information processing device comprising:

5. a character number estimation unit that estimates a character number from the image features; the learning unit learns a model that performs processing of the feature extraction unit and the number-of-characters estimation unit based on the number of correct characters corresponding to the character image and the number of characters; 5. The information processing apparatus according to claim 4, further comprising:

6. A computer-implemented information processing method, comprising: a writing direction estimation step of estimating a writing direction of the character image from the character image; a feature extraction step of extracting image features from the character image; a character string estimation step of estimating a character string from the image features, the number of characters, and the writing direction; An information processing method comprising:

7. A computer-implemented information processing method, comprising: a writing direction estimation step of estimating a writing direction of the character image from the character image; a feature extraction step of extracting image features from the character image; a character string estimation step of estimating a character string from the image features, the number of characters, and the writing direction; a learning step of learning a model that performs the processing of the feature extraction step and the character string estimation step based on the character string and a correct character string corresponding to the character image; An information processing method comprising:

8. An information processing program for causing a computer to function as the information processing device according to any one of claims 1 to 5.

Images