Text recognition method, device and equipment and storage medium

Abstract

This application belongs to the field of character recognition, specifically relating to character recognition methods, devices, and storage media. The method includes: acquiring image data, which includes text information to be recognized; acquiring a pre-trained character recognition model, which includes a main branch and sub-branches, the sub-branch being used to acquire semantic information from the image data; the main branch being used to recognize the text information from the image data using the semantic information; and inputting the image data into the character recognition model to obtain the recognized text information. This method can solve the problem of low accuracy in traditional character recognition methods. Because the character recognition model can fully learn the semantic information of the text during the recognition process, it can improve the model's recognition accuracy, thereby increasing the overall accuracy of character recognition.

Description

[Technical Field]

[0001] This invention belongs to the field of character recognition, and particularly relates to a character recognition method, device, and storage medium. [Background Technology]

[0002] Text recognition refers to the process of identifying text in an image to obtain text that a computer can recognize.

[0003] Traditional text recognition methods include: inputting the image to be recognized into a pre-trained neural network model, which extracts structural information from the image to obtain a high-dimensional feature vector, inputting the feature vector into a two-layer GRU (Gated Recurrent Unit) gated recurrent unit with an attention mechanism, and finally predicting the special structure and radical sequence of the text in the image, thereby recognizing the text in the image.

[0004] However, images containing text not only express the components and their spatial structure, but also the semantic information conveyed by the overall text formed by these components according to a specific spatial structure. The traditional character recognition methods mentioned above only utilize the information about the components and their spatial structure expressed in the character image, without using the semantic information expressed by the entire image. This leads to low character recognition accuracy. [Summary of the Invention]

[0005] This application provides a text recognition method, device, and storage medium, which can solve the problem of low text recognition accuracy.

[0006] This application provides the following technical solution:

[0007] Firstly, a character recognition method is provided, the method comprising:

[0008] Acquire image data, wherein the image data includes text information to be recognized;

[0009] A pre-trained character recognition model is obtained, the character recognition model including a main branch and sub-branches, the sub-branches being used to obtain the semantic information of the image data; the main branch being used to use the semantic information of the image data to recognize the text information of the image data.

[0010] The image data is input into the text recognition model to obtain the recognized text information.

[0011] Optionally, obtaining the semantic information of the image data includes:

[0012] The image data is input into the main branch encoder to obtain the high-dimensional feature vector corresponding to the image data;

[0013] Obtain a pre-trained semantic extraction model, which includes one BILSTM layer and two fully connected layers;

[0014] The high-dimensional feature vector is input into a pre-trained semantic extraction model to obtain the semantic information of the image data.

[0015] Optionally, obtaining the pre-trained semantic extraction model includes:

[0016] Obtain the pre-created initial network model;

[0017] Acquire first training data, which includes sample high-dimensional feature vectors corresponding to sample image data and label semantic information corresponding to sample high-dimensional feature vectors;

[0018] The high-dimensional feature vector of the sample is input into the initial network model to obtain the first training result;

[0019] Obtain the first loss function;

[0020] The high-dimensional feature vector of the sample and the first training result are input into the first loss function to obtain the value of the first loss function;

[0021] The initial network model is iteratively trained using the first loss function value to obtain a semantic extraction model.

[0022] Optionally, obtaining the first training data includes:

[0023] The image data is input into a character vector extraction model trained based on the BERT algorithm to obtain the character vectors corresponding to the image data;

[0024] The image data is used as sample image data;

[0025] The word vectors corresponding to the image data are used as the label semantic information corresponding to the high-dimensional feature vectors of the samples.

[0026] Optionally, the encoder is generated based on a dense network of connections.

[0027] Optionally, the dense connection network includes dense blocks, an average pooling layer, and a max pooling layer.

[0028] Optionally, the main branch includes a decoder and an encoder, the encoder being generated based on an attention mechanism; obtaining the pre-trained text recognition model includes:

[0029] The semantic information and the high-dimensional feature vector generated by the encoder are input into the decoder to obtain the prediction result;

[0030] The character recognition model is determined based on the prediction results and the loss function.

[0031] Secondly, a character recognition device is provided, the device comprising:

[0032] The first acquisition module is used to acquire image data, which includes text information to be recognized;

[0033] The second acquisition module is used to acquire a pre-trained text recognition model, which includes a main branch and sub-branches. The sub-branches are used to acquire the semantic information of the image data, and the main branch is used to recognize the text information of the image data using the semantic information of the image data.

[0034] The text recognition module is used to input the image data into the text recognition model to obtain the recognized text information.

[0035] Thirdly, an electronic device is provided, the device including a processor and a memory; the memory stores a program, the program being loaded and executed by the processor to implement the character recognition method as described in the first aspect.

[0036] Fourthly, a computer-readable storage medium is provided, wherein a program is stored therein, which, when executed by a processor, is used to implement the character recognition method as described in the first aspect.

[0037] The beneficial effects of this application are as follows: By acquiring image data, which includes text information to be recognized; acquiring a pre-trained character recognition model, which includes a main branch and sub-branches, wherein the sub-branches are used to acquire the semantic information of the image data; the main branch is used to recognize the text information of the image data using the semantic information of the image data; and inputting the image data into the character recognition model to obtain the recognized text information; this can solve the problem of low accuracy in traditional character recognition methods. Because the character recognition model can fully learn the semantic information of the text during the character recognition process, it avoids the problem that traditional character recognition methods only use the radicals and their spatial structure information expressed by the character image, without using the semantic information expressed by the whole image. Therefore, it can improve the recognition accuracy of the model, thereby improving the accuracy of character recognition. [Attached Image Description]

[0038] Figure 1 This is a flowchart of a text recognition method provided in one embodiment of this application;

[0039] Figure 2 This is a structural diagram of a dense interconnected network provided in one embodiment of this application;

[0040] Figure 3 This is the training process of a character recognition model provided in one embodiment of this application;

[0041] Figure 4 This is a schematic diagram of the training process of a character recognition model provided in one embodiment of this application;

[0042] Figure 5 This is a block diagram of a character recognition device provided in one embodiment of this application;

[0043] Figure 6 This is a block diagram of an electronic device provided in one embodiment of this application.

Detailed Implementation Methods

[0044] The technical solutions of this application will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. The application will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other.

[0045] It should be noted that the terms "first," "second," etc., in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0046] In this application, unless otherwise stated, directional terms such as "upper," "lower," "top," and "bottom" are generally used in relation to the direction shown in the accompanying drawings, or in relation to the vertical, perpendicular, or gravitational direction of the component itself; similarly, for ease of understanding and description, "inner" and "outer" refer to the inner and outer contours of each component itself, but the above directional terms are not used to limit this application.

[0047] The text recognition method provided in this application will be described in detail below.

[0048] Optionally, the execution subject of the text recognition method provided in this application is an electronic device, which can be a terminal such as a computer, mobile phone, tablet computer, or camera, or it can be a server. This embodiment does not limit the implementation method of the electronic device.

[0049] This embodiment provides a text recognition method, such as... Figure 1 As shown, the method includes at least the following steps:

[0050] Step 101: Obtain image data.

[0051] The image data includes text information to be recognized.

[0052] In one example, the text information can be handwritten or printed text; this embodiment does not limit the format of the text information.

[0053] In another example, the text information can be either simplified or traditional Chinese characters. This embodiment does not limit the font of the text information.

[0054] Optionally, the methods for acquiring image data include, but are not limited to, the following:

[0055] The first method involves acquiring image data sent by other devices connected to the electronic device. These other devices can be terminals such as computers, mobile phones, and tablets. This embodiment does not limit the type of other devices.

[0056] The second method involves acquiring image data pre-stored in an electronic device.

[0057] In actual implementation, image data can be acquired in other ways, and this embodiment does not limit the way image data is acquired.

[0058] Step 102: Obtain the pre-trained character recognition model.

[0059] The text recognition model includes a main branch and sub-branches. The sub-branches are used to obtain the semantic information of the image data, while the main branch is used to recognize the text information of the image data using the semantic information of the image data.

[0060] Optionally, obtaining semantic information of image data includes: inputting image data into the main branch encoder to obtain a high-dimensional feature vector corresponding to the image data; obtaining a pre-trained semantic extraction model; and inputting the high-dimensional feature vector into the pre-trained semantic extraction model to obtain semantic information of the image data.

[0061] Optionally, the main branch includes an encoder, which is generated based on a densely connected network.

[0062] like Figure 2 As shown, the dense connection network in this embodiment sequentially includes a max-pool layer, a dense block, an average-pool layer, a dense block, an average-pool layer, and another dense block. For detailed parameters, please refer to [link to documentation]. Figure 2 .

[0063] Among them, high-dimensional feature vectors can represent the structural information of textual information, the color information of images, and semantic information in image data.

[0064] For example, a high-dimensional feature vector can be represented as A, which is a three-dimensional vector (H*W*C, where H is the height, W is the width, and C is the number of channels). The specific formula is as follows:

[0065] A = {a1, ..., a} H×W}, a i ∈R c

[0066] Where each vector a in A l H1(*) represents the l-th convolutional layer, and its output is:

[0067] a l =Hl([a0; a1; ...; a l-1 ])

[0068] Optionally, obtaining a pre-trained semantic extraction model includes: obtaining a pre-created initial network model; obtaining first training data, the first training data including sample high-dimensional feature vectors corresponding to sample image data and label semantic information corresponding to the sample high-dimensional feature vectors; inputting the sample high-dimensional feature vectors into the initial network model to obtain a first training result; obtaining a first loss function, the first loss function being obtained based on the first training result and label semantic information; inputting the sample high-dimensional feature vectors and the first training result into the first loss function to obtain a first loss function value; and using the first loss function value to iteratively train the initial network model to obtain a semantic extraction model.

[0069] The semantic extraction model consists of a BILSTM layer and two fully connected layers.

[0070] In this embodiment, obtaining the first training data includes: inputting image data into a character vector extraction model trained based on the BERT algorithm to obtain character vectors corresponding to the image data; using the image data as sample image data; and using the character vectors corresponding to the image data as the label semantic information corresponding to the high-dimensional feature vectors of the samples.

[0071] Among them, the word vector extraction model trained based on the BERT algorithm is a pre-trained model. The BERT algorithm is based on the transformer and has powerful language representation and feature extraction capabilities.

[0072] In this embodiment, the high-dimensional feature vector output by the encoder is reduced to the same dimension as the character vector output by the character vector extraction model trained based on the BERT algorithm after passing through the semantic extraction model, and the extracted vector contains the semantic information of the text information.

[0073] In this embodiment, the cosine similarity between the output result of the semantic extraction model and the output result of the character vector extraction model is used to determine the value of the first loss function, which is specifically as follows:

[0074] L sem = 1 - cos(S, em)

[0075] where S is the semantic vector value output by the semantic extraction model, and em is the character vector value output by the character vector extraction model.

[0076] Since the information expressed by the text image not only includes the information of special structures and radicals, but also contains the specific semantic information carried by this text itself. Therefore, the present invention proposes that the semantic extraction model converts the semantic information contained in the text in the image into an embedding vector, so that this vector can approach the character vector obtained from the Bert pre-trained model. Thus, the encoder and decoder can learn the knowledge of semantic information. For example, words with the same radical of three dots of water should have similar semantics, so the cosine similarity of their character vectors should be relatively high. Such a method can, to a certain extent, compensate for some deficiencies that the encoder and decoder can only learn the structural information and color information of the image.

[0077] Optionally, the main branch further includes a decoder, which is generated based on an attention mechanism.

[0078] Correspondingly, a pre-trained text recognition model is taken, including: inputting the semantic information and the high-dimensional feature vector generated by the encoder into the decoder to obtain a prediction result; determining the text recognition model based on the prediction result and the loss function.

[0079] In this embodiment, the high-dimensional feature vector and the semantic information are input into the decoder with an attention mechanism. The attention-based decoder adopts two single-directional GRU layers and an attention model to learn the alignment between the output symbol yt (the predicted special structure and radical) at each decoding time step t and the input image data X. <00002​​​​​​​​​​This is the predicted output of the first GRU hidden layer, which is calculated from the ground-truth symbol yt-1 of the previous time step and the decoder state st-1 obtained from the previous time step (obtained from semantic information).

[0084] Optionally, in utilizing Before calculating the attention probability with A, a coverage vector F is introduced, which is calculated based on the sum of all past attention probabilities. This vector is the input of past attention probabilities into a convolutional layer; specifically, the coverage vector is defined by the following equation:

[0085]

[0086] Where Q represents a convolutional layer, α j This represents the attention probability at time j.

[0087] Specifically, the coverage vector F is used to address the difficulty of standard attention mechanisms, namely the lack of coverage, which often leads to over-resolution (some base points are decoded more than once) and under-resolution (some base points are never decoded). The past alignment information contained in F helps the attention model identify the parts of the input image data that are of interest, ensuring that each part of the image data is focused on only once. Specifically, F is initialized as a zero vector.

[0088] Further calculation The energy coefficient between A and B:

[0089]

[0090] Among them, e ti Represents the eigenvector a i The energy f of an element in A at time t during the decoding process. i Let n and n′ represent the dimensions of the GRU decoder and attention mechanism, respectively. Let q represent the dimension of the Q-output of the convolutional layer, and v... att W att U att U f As weights. By using e ti Input the Softmax equation to obtain α ti :

[0091]

[0092] Furthermore, through α ti We can compute the context vector C t :

[0093]

[0094] In this embodiment, C t It's called a context vector because it contains information about all the input image data. This is achieved by combining the probability α... ti After multiplication, C t It only includes useful information from the input image data, rather than information about the entire image. This reflects attention and can improve learning efficiency.

[0095] Furthermore, after the first GRU layer, we obtain C. t and The model will then input these two vectors into a new GRU layer:

[0096]

[0097] The second GRU layer will output the current decoder output status s. t Ultimately, each predicted symbol (special structure or radical) will be obtained through the above vectors using the following formula:

[0098] P(y t |y t-1 ,X)=g(W0h(Ey t-1 +W s s t +W c C t )

[0099] g represents a Softmax activation function, and h represents a Maxout activation function. W0, W s W c represents the weight. E represents the embedding vector.

[0100] Finally, the prediction results and label results are compared using cross-entropy loss, as shown below:

[0101] L pre =CrossEntropyLoss(P,Y)

[0102] Furthermore, using the final loss function L = L pre +λL sem The character recognition model is trained by backpropagation, and the parameters of the encoder, decoder, and semantic extraction model are updated to obtain the final character recognition model.

[0103] Where λ is a hyperparameter and 0 < λ < 1.

[0104] Step 103: Input the image data into the text recognition model to obtain the recognized text information.

[0105] In summary, in this embodiment, image data is obtained, and the image data includes text information to be recognized; a pre-trained text recognition model is obtained, and the text recognition model includes a main branch and a sub-branch, and the sub-branch is used to obtain semantic information of the image data; the main branch is used to recognize the text information of the image data using the semantic information of the image data; the image data is input into the text recognition model to obtain the recognized text information; it can solve the problem that the traditional text recognition method leads to a low accuracy of text recognition. Since the text recognition model can fully learn the semantic information of the text during the text recognition process, it avoids the problem that the traditional text recognition method only uses the radical and its spatial structure information expressed by the text image and does not use the semantic information expressed by the overall image. Therefore, the recognition accuracy of the model can be improved, and thus the accuracy of text recognition can be improved.

[0106] To understand the above text recognition method more clearly, an example is given in this application to illustrate the training method of the text recognition model in detail. For example Figure 3 and Figure 4 as shown, taking the input image data including the text "殿" to be recognized as an example, this embodiment is described with reference to the flowchart and the schematic diagram.

[0107] Step 301, input the image data to be recognized into the CNN encoder to obtain a high-dimensional feature vector of (1, 936, 11, 11).

[0108] Step 302, input the high-dimensional feature vector into the semantic extraction model, and output a semantic information vector (1, 768) with semantic information and dimensionality reduction.

[0109] Specifically, the cosine similarity is calculated between the semantic information vector and the character vector of "殿" obtained by the Bert pre-training model, and this cosine similarity is used as the loss function to train and converge the semantic extraction model.

[0110] Step 303, input the semantic information vector and the high-dimensional feature vector into the decoder generated based on the attention mechanism.

[0111] Specifically, after obtaining the feature vector and the semantic information, the feature vector and the semantic information are input into a two-layer GRU decoder with an attention mechanism. This decoder will predict the length of the sequence and the content of the sequence. According to the predicted length of the sequence, for example Figure 4 the sequence length of the character "殿" is 9, then the decoder will output 9 state vectors of (1, 256) dimensions. According to these vectors, the probability of each symbol (special structure or radical) at these 9 positions is calculated through equation (10), and finally the symbol with the highest probability at each position is selected as the predicted symbol.

[0112] Step 304 involves performing a cross-entropy loss on the predicted sequence and the label results, then adding it to the cosine similarity-based loss function obtained in step 302, and finally backpropagating the resulting loss to complete the training of the text recognition model.

[0113] Figure 5 This is a block diagram of a text recognition device provided in one embodiment of this application. The device includes at least the following modules: a first acquisition module 510, a second acquisition module 520, and a text recognition module 530.

[0114] The first acquisition module 510 is used to acquire image data, the image data including text information to be recognized;

[0115] The second acquisition module 520 is used to acquire a pre-trained text recognition model, which includes a main branch and sub-branches. The sub-branches are used to acquire the semantic information of the image data, and the main branch is used to recognize the text information of the image data using the semantic information of the image data.

[0116] The text recognition module 530 is used to input the image data into the text recognition model to obtain the recognized text information.

[0117] For relevant details, please refer to the above embodiments.

[0118] It should be noted that the text recognition device provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the text recognition device can be divided into different functional modules to complete all or part of the functions described above. In addition, the text recognition device and the text recognition method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0119] This embodiment provides an electronic device, such as... Figure 6 As shown, the electronic device includes at least a processor 601 and a memory 602.

[0120] Processor 601 may include one or more processing cores, such as a quad-core processor or an octa-core processor. Processor 601 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 601 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 601 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 601 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0121] The memory 602 may include one or more computer-readable storage media, which may be non-transitory. The memory 602 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 602 are used to store at least one instruction, which is executed by the processor 601 to implement the character recognition method provided in the method embodiments of this application.

[0122] In some embodiments, the electronic device may also optionally include a peripheral device interface and at least one peripheral device. The processor 601, memory 602, and peripheral device interface can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface via a bus, signal line, or circuit board. Indicatively, peripheral devices include, but are not limited to, radio frequency circuits, touch displays, audio circuits, and power supplies.

[0123] Of course, electronic devices may also include fewer or more components, and this embodiment does not limit this.

[0124] Optionally, this application also provides a computer-readable storage medium storing a program that is loaded and executed by a processor to implement the character recognition method of the above-described method embodiments.

[0125] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0126] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A character recognition method characterized by comprising: The method includes: Acquire image data, wherein the image data includes text information to be recognized; Obtain a pre-trained character recognition model, which includes a main branch and sub-branches; The sub-branch is used to obtain the semantic information of the image data, and the sub-branch includes one BILSTM layer and two fully connected layers; The main branch includes an encoder and a decoder. The encoder is generated based on a dense connection network and is used to recognize the text information of the image data using the semantic information of the image data. The character recognition model is trained as follows: the image data is input into a character vector extraction model trained based on the BERT algorithm to obtain character vectors corresponding to the image data as label semantic information; a first loss function is constructed using the label semantic information and the semantic information output by the sub-branch; a second loss function is constructed using the prediction result output by the main branch and the label result; and the character recognition model is jointly trained based on the first loss function and the second loss function. The image data is input into the text recognition model to obtain the recognized text information.

2. The method of claim 1, wherein, The semantic information of the obtained image data includes: The image data is input into the main branch encoder to obtain the high-dimensional feature vector corresponding to the image data; Obtain a pre-trained semantic extraction model, which includes one BILSTM layer and two fully connected layers; The high-dimensional feature vector is input into a pre-trained semantic extraction model to obtain the semantic information of the image data.

3. The method of claim 2, wherein, The process of obtaining the pre-trained semantic extraction model includes: Obtain the pre-created initial network model; Acquire first training data, which includes sample high-dimensional feature vectors corresponding to sample image data and label semantic information corresponding to sample high-dimensional feature vectors; The high-dimensional feature vector of the sample is input into the initial network model to obtain the first training result; Obtain the first loss function; The high-dimensional feature vector of the sample and the first training result are input into the first loss function to obtain the value of the first loss function; The initial network model is iteratively trained using the first loss function value to obtain a semantic extraction model.

4. The method of claim 3, wherein, The acquisition of the first training data includes: The image data is input into a character vector extraction model trained based on the BERT algorithm to obtain the character vectors corresponding to the image data; The image data is used as sample image data; The word vectors corresponding to the image data are used as the label semantic information corresponding to the high-dimensional feature vectors of the samples.

5. The method of claim 1, wherein, The dense connection network includes dense blocks, average pooling layers, and maximum pooling layers.

6. The method of claim 1, wherein, The main branch includes a decoder and an encoder, the encoder being generated based on an attention mechanism; The process of obtaining a pre-trained character recognition model includes: The semantic information and the high-dimensional feature vector generated by the encoder are input into the decoder to obtain the prediction result; The character recognition model is determined based on the prediction results and the loss function.

7. A character recognition apparatus characterized by comprising: The device includes: The first acquisition module is used to acquire image data, which includes text information to be recognized; The second acquisition module is used to acquire a pre-trained character recognition model, which includes a main branch and sub-branches; The sub-branch is used to obtain the semantic information of the image data, and the sub-branch includes one BILSTM layer and two fully connected layers; The main branch includes an encoder and a decoder. The encoder is generated based on a dense connection network and is used to recognize the text information of the image data using the semantic information of the image data. The character recognition model is trained as follows: the image data is input into a character vector extraction model trained based on the BERT algorithm to obtain character vectors corresponding to the image data as label semantic information; a first loss function is constructed using the label semantic information and the semantic information output by the sub-branch; a second loss function is constructed using the prediction result output by the main branch and the label result; and the character recognition model is jointly trained based on the first loss function and the second loss function. The text recognition module is used to input the image data into the text recognition model to obtain the recognized text information.

8. An electronic device, comprising: The device includes a processor and a memory; the memory stores a program that is loaded and executed by the processor to implement the character recognition method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The storage medium stores a program that, when executed by a processor, is used to implement the character recognition method as described in any one of claims 1 to 6.

Images