Text recognition method, device, readable medium and electronic device

Abstract

The present disclosure relates to a text recognition method, device, readable medium, and electronic device, the method comprising: obtaining a text feature image corresponding to a text image to be recognized; determining a height feature image and a width feature image corresponding to the text image based on the text feature image, wherein the height feature image is used to characterize the features of the text image in the height direction, and the width feature image is used to characterize the features of the text image in the width direction; and obtaining the text content corresponding to the text image based on the text feature image, the height feature image, and the width feature image through a pre-trained text recognition model. In other words, the text content corresponding to the text image can be obtained based on the height feature image, the width feature image, and the text feature image. In this way, in the text image recognition process, the features of different dimensions of the text image can be combined, thereby improving the accuracy of text image recognition.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular, to a text recognition method, device, readable medium, and electronic device. Background Art

[0002] With the widespread application of text image recognition technology, people have increasingly higher requirements for the accuracy of text image recognition, requiring the ability to accurately identify each character in a text image. In related technologies, text content in text images is recognized using a neural network model based on deep learning.

[0003] However, for some more complex text images, such as advertisements and movie posters, the titles in the text images often use special typesetting. Therefore, when the same neural network model is used to recognize the text image, some characters cannot be recognized normally, resulting in a relatively low accuracy rate in text image recognition. Summary of the Invention

[0004] This summary is provided to briefly introduce concepts that will be described in detail in the detailed description below. This summary is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0005] In a first aspect, the present disclosure provides a text recognition method, the method comprising:

[0006] Obtaining a text feature image corresponding to the text image to be recognized;

[0007] Determining, based on the text feature image, a height feature image and a width feature image corresponding to the text image, wherein the height feature image is used to characterize features of the text image in a height direction, and the width feature image is used to characterize features of the text image in a width direction;

[0008] According to the text feature image, the height feature image, and the width feature image, text content corresponding to the text image is acquired through a pre-trained text recognition model.

[0009] In a second aspect, the present disclosure provides a text recognition device, the device comprising:

[0010] A first feature image acquisition module is used to acquire a text feature image corresponding to the text image to be recognized;

[0011] A second feature image acquisition module is configured to determine a height feature image and a width feature image corresponding to the text image based on the text feature image, wherein the height feature image is used to represent the features of the text image in the height direction, and the width feature image is used to represent the features of the text image in the width direction;

[0012] The text content acquisition module is used to acquire the text content corresponding to the text image based on the text feature image, the height feature image and the width feature image through a pre-trained text recognition model.

[0013] In a third aspect, the present disclosure provides a computer-readable medium having a computer program stored thereon, which, when executed by a processing device, implements the steps of the method described in the first aspect of the present disclosure.

[0014] In a fourth aspect, the present disclosure provides an electronic device, comprising:

[0015] a storage device having a computer program stored thereon;

[0016] A processing device is used to execute the computer program in the storage device to implement the steps of the method described in the first aspect of the present disclosure.

[0017] Through the above technical solution, by obtaining a text feature image corresponding to the text image to be recognized; according to the text feature image, determining the height feature image and width feature image corresponding to the text image, the height feature image is used to characterize the features of the text image in the height direction, and the width feature image is used to characterize the features of the text image in the width direction; according to the text feature image, the height feature image and the width feature image, through a pre-trained text recognition model, obtaining the text content corresponding to the text image. In other words, the present disclosure splits the dimensions of the text image, and can obtain the text content corresponding to the text image based on the height feature image that characterizes the features of the text image in the height direction, the width feature image that characterizes the features of the text image in the width direction, and the text feature image corresponding to the text image. In this way, in the text image recognition process, the features of the text image in different dimensions can be combined, thereby improving the accuracy of text image recognition.

[0018] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above and other features, advantages and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that the originals and elements are not necessarily drawn to scale. In the drawings:

[0020] Figure 1 is a flowchart of a text recognition method provided according to an exemplary embodiment;

[0021] Figure 2 is a flowchart of another text recognition method provided according to an exemplary embodiment;

[0022] Figure 3 is a flowchart of a character recognition method provided according to an exemplary embodiment;

[0023] Figure 4 is a schematic diagram of a scaled dot product attention sublayer provided according to an exemplary embodiment;

[0024] Figure 5 is a schematic diagram of a text recognition method provided according to an exemplary embodiment;

[0025] Figure 6 is a block diagram of a text recognition device provided according to an exemplary embodiment;

[0026] Figure 7 is a block diagram of another text recognition device provided according to an exemplary embodiment;

[0027] Figure 8 is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0028] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0029] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0030] As used herein, the term "including" and its variations are open-ended, i.e., "including but not limited to." The term "based on" means "based, at least in part, on." The term "one embodiment" means "at least one embodiment," the term "another embodiment" means "at least one additional embodiment," and the term "some embodiments" means "at least some embodiments." Other terms are defined in the following description.

[0031] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0032] It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0033] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0034] First, the application scenarios of the present disclosure are described. Traditional OCR (Optical Character Recognition) technology is divided into two steps: text detection and text recognition. Text detection is used to segment a text image to obtain the text area in the text image, and text recognition is used to obtain the text content in the text area. In the related technologies, text recognition based on deep learning mainly includes the CTC (Connectionist Temporal Classification) method represented by CRNN (Convolutional Recurrent Neural Network) and the Attention method represented by Transformer. The CTC method and the Attention method have strong recognition capabilities for horizontal text images. However, for some special text images, such as advertisements, movie posters and other text images, since the titles of text images often use special layouts, which include text lines of different heights, all text images are downsampled according to one-dimensional text lines, which will cause the height of the two-dimensional text to be over-compressed, resulting in information loss and thus unable to be recognized normally; all text images are downsampled according to the height of the two-dimensional text, which makes the sequence length to be processed longer and the amount of calculation greatly increased, resulting in too low efficiency of text recognition.

[0035] In order to solve the above-mentioned technical problems, the present disclosure provides a text recognition method, device, readable medium and electronic device. By splitting the dimensions of the text image, the text content corresponding to the text image can be obtained based on the height feature image representing the height direction characteristics of the text image, the width feature image representing the width direction characteristics of the text image and the text feature image corresponding to the text image. In this way, in the text image recognition process, the features of different dimensions of the text image can be combined, thereby improving the accuracy of text image recognition.

[0036] The present disclosure is described below with reference to specific embodiments.

[0037] Figure 1 is a flowchart of a text recognition method provided according to an exemplary embodiment. Figure 1 As shown, the method may include:

[0038] S101: Obtain a text feature image corresponding to a text image to be recognized.

[0039] In this step, after obtaining the text image to be identified, the text image can be downsampled to obtain a text feature image corresponding to the text image. For example, the text image can be downsampled by a pre-trained text feature image acquisition model to obtain a text feature image corresponding to the text image. The text feature image acquisition model can be based on a CNN (Convolutional Neural Networks, convolutional neural network model) framework, and is trained by a model training method of the prior art, which will not be described here. The CNN framework can include the first two blocks of ResNet18, and the downsampling ratio of the downsampling processing of the text feature image acquisition model can be obtained by pre-testing through experiments. For example, the downsampling ratio can be 4. If the size corresponding to the text image is (H, W), and the downsampling ratio of the downsampling processing of the text feature image acquisition model is N, then the size of the text feature image obtained after downsampling the text image is (H / N, W / N).

[0040] S102: Determine a height feature image and a width feature image corresponding to the text image according to the text feature image.

[0041] The height feature image is used to characterize the features of the text image in the height direction, and the width feature image is used to characterize the features of the text image in the width direction.

[0042] In this step, after obtaining the text feature image corresponding to the text image, the text feature image can be input into a pre-trained height feature acquisition model to obtain the height feature image corresponding to the text image output by the height feature acquisition model, and the text feature image can be input into a pre-trained width feature acquisition model to obtain the width feature image corresponding to the text image output by the width feature acquisition model.

[0043] S103 : Acquire text content corresponding to the text image through a pre-trained text recognition model according to the text feature image, the height feature image, and the width feature image.

[0044] Among them, the text recognition model can include a first encoding sub-model, a second encoding sub-model and a character recognition sub-model. The training method of the text recognition model can refer to the model training method of the existing technology and will not be repeated here.

[0045] In this step, after obtaining the text feature image, the height feature image and the width feature image, the text feature image, the height feature image and the width feature image can be input into the text recognition model to obtain the text content corresponding to the text image output by the text recognition model.

[0046] By adopting the above method, by splitting the dimensions of the text image, the text content corresponding to the text image can be obtained based on the height feature image representing the height direction characteristics of the text image, the width feature image representing the width direction characteristics of the text image, and the text feature image corresponding to the text image. In this way, in the text image recognition process, the features of different dimensions of the text image can be combined, thereby improving the accuracy of text image recognition.

[0047] Figure 2 is a flowchart of another text recognition method provided according to an exemplary embodiment. Figure 2 As shown, the method may include:

[0048] S201: Obtain a text feature image corresponding to a text image to be recognized.

[0049] S202: Input the text feature image into a pre-trained height feature acquisition model to obtain a height feature image corresponding to the text image output by the height feature acquisition model.

[0050] Among them, the height feature image is used to characterize the height direction characteristics of the text image. The height feature acquisition model can be based on a Reduction model and trained through the model training method of the existing technology, which will not be repeated here.

[0051] In this step, after obtaining the text feature image, the text feature image can be input into the height feature acquisition model, and the text feature image is processed by the height feature acquisition model to obtain the height feature image corresponding to the text image. The width of the height feature image is 1, that is, the height feature image only reflects the height direction characteristics of the text image.

[0052] S203: Input the text feature image into a pre-trained width feature acquisition model to obtain a width feature image corresponding to the text image output by the width feature acquisition model.

[0053] The width feature image is used to characterize the width-wise features of the text image. The width feature acquisition model can also be trained based on the Reduction model using the existing model training method, which will not be described in detail here. The parameters of the width feature acquisition model are different from those of the height feature acquisition model.

[0054] In this step, after obtaining the text feature image, the text feature image can be input into the width feature acquisition model, and the text feature image is processed by the width feature acquisition model to obtain the width feature image corresponding to the text image. The height of the width feature image is 1, that is, the width feature image only reflects the characteristics of the width direction of the text image.

[0055] It should be noted that the present disclosure does not limit the execution order of the above-mentioned steps S202 and S203. After obtaining the text feature image corresponding to the text image, step S202 can be executed first and then step S203, or step S203 can be executed first and then step S202, or step S202 and step S203 can be executed in parallel and synchronously.

[0056] S204: Input the height feature image into the first coding sub-model to obtain a height coding feature vector corresponding to the height feature image output by the first coding sub-model.

[0057] The structure of the first encoding sub-model may be the same as the structure of the encoder in the Transformer model in the prior art.

[0058] In this step, after obtaining the height characteristic image, the height characteristic image can be input into the first coding sub-model, and the height characteristic image can be processed by the first coding sub-model to obtain the height coding feature vector corresponding to the height characteristic image.

[0059] S205 : Input the width feature image into the second encoding sub-model to obtain a width encoding feature vector corresponding to the width feature image output by the second encoding sub-model.

[0060] The structure of the second encoding sub-model may also be the same as the structure of the encoder in the Transformer model in the prior art, and the parameters of the second encoding sub-model are different from the parameters of the first encoding sub-model.

[0061] In this step, after obtaining the width characteristic image, the width characteristic image can be input into the second encoding sub-model, and the width characteristic image can be processed by the second encoding sub-model to obtain a width coding feature vector corresponding to the width characteristic image.

[0062] S206 : Acquire text content corresponding to the text image through a character recognition sub-model according to the text feature image, the height-encoded feature vector, and the width-encoded feature vector.

[0063] In this step, after obtaining the text feature image, the height-encoded feature vector, and the width-encoded feature vector, the character recognition step can be executed in a loop until the target character output by the character recognition sub-model is the preset termination character, and the new text character is used as the text content corresponding to the text image. For example, in the case where the target character output by the character recognition sub-model is " / eos", if the text characters include "O", "C", "R", "", "i", "s", "", "", "g", "r", "e", "a", "t", "!", then the text content corresponding to the text image can be "OCR is great!"

[0064] in, Figure 3 is a flowchart of a character recognition method provided according to an exemplary embodiment. Figure 3 As shown, the character recognition step may include:

[0065] S1. Obtain the character embedding feature vector corresponding to the recognized text characters in the text image.

[0066] The recognized text characters in the text image may be all target characters output by the character recognition model. After obtaining the text characters, the character embedding feature vector corresponding to the text characters may be obtained through the character embedding model of the prior art (for example, the output embedding in the Transformer model).

[0067] S2. Input the text feature image, the height encoding feature vector, the width encoding feature vector, and the character embedding feature vector into the character recognition sub-model to obtain the target character output by the character recognition sub-model.

[0068] Among them, the character recognition sub-model may include a first decoding sub-model and a second decoding sub-model, the first decoding sub-model may include a multi-head attention layer, a residual link & regularization layer (Add&Norm), the output end of the multi-head attention layer is coupled with the input end of the Add&Norm, the output end of the Add&Norm is coupled with the input end of the second decoding sub-model, the second decoding sub-model may include a multi-head attention layer and a feedforward neural network layer, the multi-head attention layer may include a first linear sub-layer, a scaled point product attention sub-layer, a connection sub-layer and a second linear sub-layer, the output end of the first linear sub-layer is coupled with the input end of the scaled point product attention sub-layer, the output end of the scaled point product attention sub-layer is coupled with the input end of the connection sub-layer, and the output end of the connection sub-layer is coupled with the input end of the second linear sub-layer.

[0069] After obtaining the text feature image, the height encoding feature vector, the width encoding feature vector and the character embedding feature vector, the character embedding feature vector can be input into the first decoding sub-model to obtain the character decoding feature vector corresponding to the text character output by the first decoding sub-model.

[0070] Furthermore, after obtaining the character decoding feature vector corresponding to the text character, the text feature image, the height encoding feature vector, the width encoding feature vector and the character decoding feature vector can be input into the second decoding sub-model to obtain the target character output by the second decoding sub-model.

[0071] In one possible implementation, the height encoding feature vector, the width encoding feature vector, and the character decoding feature vector can be input into the first linear sublayer to obtain the target height encoding feature vector, the target width encoding feature vector, and the target character decoding feature vector output by the first linear sublayer; the text feature image, the target height encoding feature vector, the target width encoding feature vector, and the target character decoding feature vector are input into the scaled point product attention sublayer to determine the text decoding feature vector based on the text feature image, the target height encoding feature vector, the target width encoding feature vector, and the target character decoding feature vector through the scaled point product attention sublayer; the text decoding feature vector is input into the connection sublayer to output the target character through the connection sublayer and the second linear sublayer.

[0072] Among them, the scaled point product attention sublayer can determine the height decoding feature vector based on the target height encoding feature vector and the target character decoding feature vector, determine the width decoding feature vector based on the target width encoding feature vector and the target character decoding feature vector, determine the pending decoding feature vector based on the height decoding feature vector and the width decoding feature vector, and determine the text decoding feature vector based on the pending decoding feature vector and the text feature image. Figure 4 is a schematic diagram of a scaled point product attention sublayer provided according to an exemplary embodiment. Figure 4 As shown, the scaled dot product attention sublayer includes a first scaled dot product attention module and a second scaled dot product attention module. The first scaled dot product attention module and the second scaled dot product attention module have the same structure, both including matrix multiplication (MatMul), scaling (Scale), mask filtering (Mask(opt.)) and classification (SoftMax). Where Q is the target character decoding feature vector, f w Encode the feature vector for the target width, f h The target height encoding feature vector, f 2d The target height encoding feature vector and the target character decoding feature vector are processed by the first scaling dot product attention module to obtain the height decoding feature vector. The target width encoding feature vector and the target character decoding feature vector are processed by the second scaling dot product attention module to obtain the width decoding feature vector. After that, the sum of the height decoding feature vector and the width decoding feature vector is determined by the torch.sum() function, and the sum is used as the pending decoding feature vector. Finally, the pending decoding feature vector and the text feature image are matrix multiplied by MatMul to expand the dimension of the pending decoding feature vector to obtain the text decoding feature vector.

[0073] It should be noted that the processing flow of the first scaled point product attention module and the second scaled point product attention module can refer to the processing flow of the multi-head attention layer in the Transformer model in the prior art, and will not be repeated here.

[0074] S3. Use the text character and the target character as new text characters.

[0075] After the character recognition sub-model outputs the target character, the text character may be updated, and the text character and the target character may be used as a new text character.

[0076] Figure 5 is a schematic diagram of a text recognition method provided according to an exemplary embodiment. Figure 5As shown in the figure, the size of the input text image is (H, W), and after downsampling through the text feature image acquisition model (CNN in the figure), a text feature image f with a size of (H / 4, W / 4) is obtained. 2d (The downsampling ratio is 4), and the text feature image is input into the height feature acquisition model and the width feature acquisition model respectively. The two dimensionality reduction models (Reduction) in the figure obtain a height feature image of size (H / 4, 1) and a width feature image of size (1, W / 4). After that, the height feature image and the width feature image are input into the first encoding sub-model and the second encoding sub-model respectively to obtain the height encoding feature vector f corresponding to the height feature image h The width encoding feature vector f corresponding to the width feature image w Finally, the highly encoded feature vector f h , the width encoding feature vector f w And the text feature image f 2d The multi-head attention layer of the second decoding sub-model is input, and finally, the target character is output through the character recognition sub-model.

[0077] By adopting the above method, by splitting the dimensions of the text image, the text content corresponding to the text image can be obtained based on the height feature image representing the height direction characteristics of the text image, the width feature image representing the width direction characteristics of the text image, and the text feature image corresponding to the text image. In this way, in the text image recognition process, the features of different dimensions of the text image can be combined, thereby improving the accuracy of text image recognition; in addition, compared with the text feature image, the dimensions of the height feature image and the width feature image are relatively low, which reduces the amount of calculation in the encoding process and improves the efficiency of text image recognition. Moreover, the final positioning of the second decoding sub-model is still at the same resolution granularity as the text feature image, ensuring the accuracy of text image recognition.

[0078] Figure 6 is a block diagram of a text recognition device according to an exemplary embodiment. Figure 6 As shown, the device may include:

[0079] A first feature image acquisition module 601 is used to acquire a text feature image corresponding to a text image to be recognized;

[0080] A second feature image acquisition module 602 is configured to determine, based on the text feature image, a height feature image and a width feature image corresponding to the text image, wherein the height feature image is used to represent the features of the text image in the height direction, and the width feature image is used to represent the features of the text image in the width direction;

[0081] The text content acquisition module 603 is configured to acquire the text content corresponding to the text image based on the text feature image, the height feature image, and the width feature image using a pre-trained text recognition model.

[0082] Optionally, the second feature image acquisition module 602 is further configured to:

[0083] Inputting the text feature image into a pre-trained height feature acquisition model to obtain the height feature image corresponding to the text image output by the height feature acquisition model;

[0084] The text feature image is input into a pre-trained width feature acquisition model to obtain the width feature image corresponding to the text image output by the width feature acquisition model.

[0085] Optionally, the text recognition model includes a first encoding sub-model, a second encoding sub-model and a character recognition sub-model; Figure 7 is a block diagram of another text recognition device provided according to an exemplary embodiment. Figure 7 As shown, the device also includes:

[0086] A height encoding feature vector acquisition module 604 is configured to input the height feature image into the first encoding sub-model to acquire a height encoding feature vector corresponding to the height feature image output by the first encoding sub-model;

[0087] A width coding feature vector acquisition module 605 is configured to input the width feature image into the second coding sub-model to acquire a width coding feature vector corresponding to the width feature image output by the second coding sub-model;

[0088] The text content acquisition module 603 is further used to:

[0089] According to the text feature image, the height encoding feature vector and the width encoding feature vector, the text content corresponding to the text image is obtained through the character recognition sub-model.

[0090] Optionally, the text content acquisition module 603 is further configured to:

[0091] The character recognition step is executed cyclically until the target character output by the character recognition sub-model is a preset termination character, and the new text character is used as the text content corresponding to the text image; the character recognition step includes:

[0092] Obtaining character embedding feature vectors corresponding to recognized text characters in the text image;

[0093] Inputting the text feature image, the height encoding feature vector, the width encoding feature vector, and the character embedding feature vector into the character recognition sub-model to obtain the target character output by the character recognition sub-model;

[0094] The text character and the target character are used as new text characters.

[0095] Optionally, the character recognition sub-model includes a first decoding sub-model and a second decoding sub-model; the text content acquisition module 603 is further configured to:

[0096] Inputting the character embedding feature vector into the first decoding sub-model to obtain a character decoding feature vector corresponding to the text character output by the first decoding sub-model;

[0097] The text feature image, the height encoding feature vector, the width encoding feature vector, and the character decoding feature vector are input into the second decoding sub-model to obtain the target character output by the second decoding sub-model.

[0098] Optionally, the second decoding submodel includes a multi-head attention layer and a feedforward neural network layer, the multi-head attention layer includes a first linear sublayer, a scaled point product attention sublayer, a connection sublayer, and a second linear sublayer, the output end of the first linear sublayer is coupled to the input end of the scaled point product attention sublayer, the output end of the scaled point product attention sublayer is coupled to the input end of the connection sublayer, and the output end of the connection sublayer is coupled to the input end of the second linear sublayer; the text content acquisition module 603 is further used to:

[0099] Inputting the height encoding feature vector, the width encoding feature vector, and the character decoding feature vector into the first linear sublayer to obtain a target height encoding feature vector, a target width encoding feature vector, and a target character decoding feature vector output by the first linear sublayer;

[0100] Inputting the text feature image, the target height encoding feature vector, the target width encoding feature vector, and the target character decoding feature vector into the scaled point product attention sublayer, so as to determine a text decoding feature vector based on the text feature image, the target height encoding feature vector, the target width encoding feature vector, and the target character decoding feature vector through the scaled point product attention sublayer;

[0101] The text decoding feature vector is input into the connection sublayer to output the target character through the connection sublayer and the second linear sublayer.

[0102] Optionally, the text content acquisition module 603 is further configured to:

[0103] Determining a height decoding feature vector according to the target height encoding feature vector and the target character decoding feature vector;

[0104] Determining a width decoding feature vector according to the target width encoding feature vector and the target character decoding feature vector;

[0105] Determining a pending decoding feature vector according to the height decoding feature vector and the width decoding feature vector;

[0106] The text decoding feature vector is determined according to the undetermined decoding feature vector and the text feature image.

[0107] Through the above-mentioned device, by splitting the dimensions of the text image, the text content corresponding to the text image can be obtained based on the height feature image representing the height direction characteristics of the text image, the width feature image representing the width direction characteristics of the text image, and the text feature image corresponding to the text image. In this way, in the text image recognition process, the features of different dimensions of the text image can be combined, thereby improving the accuracy of text image recognition.

[0108] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0109] Reference below Figure 8 , which shows a schematic structural diagram of an electronic device (e.g., a terminal device or a server) 800 suitable for implementing the embodiments of the present disclosure. The terminal device in the embodiments of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 8 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0110] like Figure 8 As shown, the electronic device 800 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage device 808 into a random access memory (RAM) 803. Various programs and data required for the operation of the electronic device 800 are also stored in the RAM 803. The processing device 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0111] Typically, the following devices may be connected to the I / O interface 805: an input device 806 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 807 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 808 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 809. The communication device 809 may allow the electronic device 800 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 8 The electronic device 800 is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead.

[0112] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 809, or installed from the storage device 808, or installed from the ROM 802. When the computer program is executed by the processing device 801, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0113] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0114] In some embodiments, the client and server can communicate using any currently known or future developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0115] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0116] The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by the electronic device, the electronic device: obtains a text feature image corresponding to the text image to be identified; determines the height feature image and width feature image corresponding to the text image based on the text feature image, the height feature image is used to characterize the features of the text image in the height direction, and the width feature image is used to characterize the features of the text image in the width direction; based on the text feature image, the height feature image and the width feature image, obtains the text content corresponding to the text image through a pre-trained text recognition model.

[0117] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including, but not limited to, object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0118] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0119] The modules described in the embodiments of the present disclosure may be implemented in software or hardware. In some cases, the name of a module does not limit the module itself. For example, the first feature image acquisition module may also be described as a "module for acquiring a text feature image corresponding to a text image to be recognized."

[0120] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0121] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0122] According to one or more embodiments of the present disclosure, Example 1 provides a text recognition method, which includes: obtaining a text feature image corresponding to a text image to be recognized; determining a height feature image and a width feature image corresponding to the text image based on the text feature image, the height feature image being used to characterize the features of the text image in a height direction, and the width feature image being used to characterize the features of the text image in a width direction; and obtaining text content corresponding to the text image through a pre-trained text recognition model based on the text feature image, the height feature image, and the width feature image.

[0123] According to one or more embodiments of the present disclosure, Example 2 provides the method of Example 1, wherein determining the height feature image and width feature image corresponding to the text image based on the text feature image includes: inputting the text feature image into a pre-trained height feature acquisition model to obtain the height feature image corresponding to the text image output by the height feature acquisition model; inputting the text feature image into a pre-trained width feature acquisition model to obtain the width feature image corresponding to the text image output by the width feature acquisition model.

[0124] According to one or more embodiments of the present disclosure, Example 3 provides the method of Example 1, wherein the text recognition model includes a first encoding sub-model, a second encoding sub-model, and a character recognition sub-model; before obtaining the text content corresponding to the text image through a pre-trained text recognition model based on the text feature image, the height feature image, and the width feature image, the method further includes: inputting the height feature image into the first encoding sub-model to obtain the height encoding feature vector corresponding to the height feature image output by the first encoding sub-model; inputting the width feature image into the second encoding sub-model to obtain the width encoding feature vector corresponding to the width feature image output by the second encoding sub-model; obtaining the text content corresponding to the text image through a pre-trained text recognition model based on the text feature image, the height feature image, and the width feature image includes: obtaining the text content corresponding to the text image through the character recognition sub-model based on the text feature image, the height encoding feature vector, and the width encoding feature vector.

[0125] According to one or more embodiments of the present disclosure, Example 4 provides the method of Example 3, wherein obtaining the text content corresponding to the text image through the character recognition sub-model based on the text feature image, the height-encoding feature vector and the width-encoding feature vector includes: looping through the character recognition step until the target character output by the character recognition sub-model is a preset termination character, and using the new text character as the text content corresponding to the text image; the character recognition step includes: obtaining the character embedding feature vector corresponding to the recognized text character in the text image; inputting the text feature image, the height-encoding feature vector, the width-encoding feature vector and the character embedding feature vector into the character recognition sub-model to obtain the target character output by the character recognition sub-model; and using the text character and the target character as new text characters.

[0126] According to one or more embodiments of the present disclosure, Example 5 provides the method of Example 4, wherein the character recognition sub-model includes a first decoding sub-model and a second decoding sub-model; the inputting the text feature image, the height encoding feature vector, the width encoding feature vector and the character embedding feature vector into the character recognition sub-model to obtain the target character output by the character recognition sub-model includes: inputting the character embedding feature vector into the first decoding sub-model to obtain the character decoding feature vector corresponding to the text character output by the first decoding sub-model; inputting the text feature image, the height encoding feature vector, the width encoding feature vector and the character decoding feature vector into the second decoding sub-model to obtain the target character output by the second decoding sub-model.

[0127] According to one or more embodiments of the present disclosure, Example 6 provides the method of Example 5, wherein the second decoding sub-model includes a multi-head attention layer and a feedforward neural network layer, the multi-head attention layer includes a first linear sub-layer, a scaled point product attention sub-layer, a connection sub-layer and a second linear sub-layer, the output end of the first linear sub-layer is coupled with the input end of the scaled point product attention sub-layer, the output end of the scaled point product attention sub-layer is coupled with the input end of the connection sub-layer, and the output end of the connection sub-layer is coupled with the input end of the second linear sub-layer; the inputting the text feature image, the height encoding feature vector, the width encoding feature vector and the character decoding feature vector into the second decoding sub-model to obtain the target character output by the second decoding sub-model includes: inputting the height encoding feature vector The feature vector, the width encoding feature vector and the character decoding feature vector are input into the first linear sub-layer to obtain the target height encoding feature vector, the target width encoding feature vector and the target character decoding feature vector output by the first linear sub-layer; the text feature image, the target height encoding feature vector, the target width encoding feature vector and the target character decoding feature vector are input into the scaled point product attention sub-layer to determine the text decoding feature vector according to the text feature image, the target height encoding feature vector, the target width encoding feature vector and the target character decoding feature vector through the scaled point product attention sub-layer; the text decoding feature vector is input into the connection sub-layer to output the target character through the connection sub-layer and the second linear sub-layer.

[0128] According to one or more embodiments of the present disclosure, Example 7 provides the method of Example 6, wherein determining the text decoding feature vector based on the text feature image, the target height encoding feature vector, the target width encoding feature vector, and the target character decoding feature vector includes: determining a height decoding feature vector based on the target height encoding feature vector and the target character decoding feature vector; determining a width decoding feature vector based on the target width encoding feature vector and the target character decoding feature vector; determining a pending decoding feature vector based on the height decoding feature vector and the width decoding feature vector; and determining the text decoding feature vector based on the pending decoding feature vector and the text feature image.

[0129] According to one or more embodiments of the present disclosure, Example 8 provides a text recognition device, which includes: a first feature image acquisition module, used to acquire a text feature image corresponding to a text image to be recognized; a second feature image acquisition module, used to determine a height feature image and a width feature image corresponding to the text image based on the text feature image, the height feature image being used to characterize the features of the text image in the height direction, and the width feature image being used to characterize the features of the text image in the width direction; a text content acquisition module, used to acquire the text content corresponding to the text image based on the text feature image, the height feature image and the width feature image through a pre-trained text recognition model.

[0130] According to one or more embodiments of the present disclosure, Example 9 provides a computer-readable medium having a computer program stored thereon, which implements the steps of the method described in any one of Examples 1 to 7 when executed by a processing device.

[0131] According to one or more embodiments of the present disclosure, Example 10 provides an electronic device, comprising: a storage device on which a computer program is stored; and a processing device for executing the computer program in the storage device to implement the steps of the method described in any one of Examples 1 to 7.

[0132] The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also includes other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned disclosed concepts. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

[0133] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details have been included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0134] Although the subject matter has been described using language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims. Regarding the apparatus in the above-described embodiments, the specific manner in which each module performs operations has been described in detail in the embodiments related to the method and will not be elaborated upon here.

Claims

1. A text recognition method, characterized in that: The method comprises: Obtaining a text feature image corresponding to the text image to be recognized; Determining, based on the text feature image, a height feature image and a width feature image corresponding to the text image, wherein the height feature image is used to characterize features of the text image in a height direction, and the width feature image is used to characterize features of the text image in a width direction; According to the text feature image, the height feature image, and the width feature image, obtaining text content corresponding to the text image through a pre-trained text recognition model; The text recognition model includes a character recognition sub-model, and obtaining text content corresponding to the text image through a pre-trained text recognition model based on the text feature image, the height feature image, and the width feature image includes: The character recognition step is executed cyclically until the target character output by the character recognition sub-model is a preset termination character, and the new text character is used as the text content corresponding to the text image; the character recognition step includes: Obtaining a character embedding feature vector corresponding to the recognized text characters in the text image; Inputting the text feature image, the height encoding feature vector corresponding to the height feature image, the width encoding feature vector corresponding to the width feature image, and the character embedding feature vector into the character recognition sub-model to obtain the target character output by the character recognition sub-model; The text character and the target character are used as new text characters.

2. The method according to claim 1, characterized in that The determining, based on the text feature image, a height feature image and a width feature image corresponding to the text image includes: Inputting the text feature image into a pre-trained height feature acquisition model to obtain the height feature image corresponding to the text image output by the height feature acquisition model; The text feature image is input into a pre-trained width feature acquisition model to obtain the width feature image corresponding to the text image output by the width feature acquisition model.

3. The method according to claim 1, characterized in that The text recognition model further includes a first encoding sub-model and a second encoding sub-model; before obtaining text content corresponding to the text image using a pre-trained text recognition model based on the text feature image, the height feature image, and the width feature image, the method further includes: Inputting the height feature image into the first encoding sub-model to obtain a height encoding feature vector corresponding to the height feature image output by the first encoding sub-model; The width feature image is input into the second encoding sub-model to obtain a width encoding feature vector corresponding to the width feature image output by the second encoding sub-model.

4. The method according to claim 1, wherein The character recognition sub-model includes a first decoding sub-model and a second decoding sub-model; the step of inputting the text feature image, the height encoding feature vector, the width encoding feature vector, and the character embedding feature vector into the character recognition sub-model to obtain the target character output by the character recognition sub-model includes: Inputting the character embedding feature vector into the first decoding sub-model to obtain a character decoding feature vector corresponding to the text character output by the first decoding sub-model; The text feature image, the height encoding feature vector, the width encoding feature vector, and the character decoding feature vector are input into the second decoding sub-model to obtain the target character output by the second decoding sub-model.

5. The method according to claim 4, characterized in that The second decoding sub-model includes a multi-head attention layer and a feedforward neural network layer, the multi-head attention layer includes a first linear sub-layer, a scaled dot product attention sub-layer, a connection sub-layer, and a second linear sub-layer, the output end of the first linear sub-layer is coupled with the input end of the scaled dot product attention sub-layer, the output end of the scaled dot product attention sub-layer is coupled with the input end of the connection sub-layer, and the output end of the connection sub-layer is coupled with the input end of the second linear sub-layer; the inputting the text feature image, the height encoding feature vector, the width encoding feature vector, and the character decoding feature vector into the second decoding sub-model to obtain the target character output by the second decoding sub-model includes: Inputting the height encoding feature vector, the width encoding feature vector, and the character decoding feature vector into the first linear sublayer to obtain a target height encoding feature vector, a target width encoding feature vector, and a target character decoding feature vector output by the first linear sublayer; Inputting the text feature image, the target height encoding feature vector, the target width encoding feature vector, and the target character decoding feature vector into the scaled dot product attention sublayer, so as to determine a text decoding feature vector based on the text feature image, the target height encoding feature vector, the target width encoding feature vector, and the target character decoding feature vector through the scaled dot product attention sublayer; The text decoding feature vector is input into the connection sublayer to output the target character through the connection sublayer and the second linear sublayer.

6. The method according to claim 5, characterized in that Determining the text decoding feature vector according to the text feature image, the target height encoding feature vector, the target width encoding feature vector, and the target character decoding feature vector includes: Determining a height decoding feature vector according to the target height encoding feature vector and the target character decoding feature vector; Determining a width decoding feature vector according to the target width encoding feature vector and the target character decoding feature vector; Determining a pending decoding feature vector according to the height decoding feature vector and the width decoding feature vector; The text decoding feature vector is determined according to the undetermined decoding feature vector and the text feature image.

7. A text recognition device, characterized in that: The device comprises: A first feature image acquisition module is used to acquire a text feature image corresponding to the text image to be recognized; A second feature image acquisition module is configured to determine a height feature image and a width feature image corresponding to the text image based on the text feature image, wherein the height feature image is used to represent the features of the text image in the height direction, and the width feature image is used to represent the features of the text image in the width direction; A text content acquisition module, configured to acquire text content corresponding to the text image based on the text feature image, the height feature image, and the width feature image using a pre-trained text recognition model; The text recognition model includes a character recognition sub-model, and the text content acquisition module is used to: The character recognition step is executed cyclically until the target character output by the character recognition sub-model is a preset termination character, and the new text character is used as the text content corresponding to the text image; the character recognition step includes: Obtaining a character embedding feature vector corresponding to the recognized text characters in the text image; Inputting the text feature image, the height encoding feature vector corresponding to the height feature image, the width encoding feature vector corresponding to the width feature image, and the character embedding feature vector into the character recognition sub-model to obtain the target character output by the character recognition sub-model; The text character and the target character are used as new text characters.

8. A computer-readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processing device, the steps of the method according to any one of claims 1 to 6 are implemented.

9. An electronic device, characterized in that: include: a storage device having a computer program stored thereon; A processing device, configured to execute the computer program in the storage device to implement the steps of the method according to any one of claims 1 to 6.

Images