Text recognition method, device, electronic device and storage medium

Abstract

The present disclosure relates to a text recognition method, device, electronic device and storage medium. The method provided by the present disclosure includes: obtaining an image to be recognized, wherein the image to be recognized is a text image; inputting the image to be recognized into a pre-built recognition network model for text recognition to obtain a first recognition result, wherein the recognition model includes a feature extraction module, a semantic module, a first recognition module and a second recognition module, using the feature extraction module to extract features of the image to be recognized to obtain first feature information, using the first recognition module to recognize the first feature information to obtain a second recognition result, then using the semantic module to calculate the semantic representation vector corresponding to the second recognition result, and using the second recognition module based on the first feature information and the semantic representation vector to obtain the first recognition result. The method provided by the present disclosure can quickly and accurately recognize text in an image.

Description

Technical Field

[0001] The present disclosure relates to the technical field of text recognition, and in particular to a text recognition method, device, electronic device and storage medium. Background Art

[0002] With the continuous development of text recognition technology, computers can be used to automatically recognize text in images. For example, intelligent learning methods can be used to automatically recognize text in images. Usually, when using intelligent learning methods for text recognition, a large number of samples need to be obtained to train the recognition model. However, the character-based method has a high annotation cost for training the recognition model, and the sequence-based method may have problems with missed recognition or over-recognition, resulting in slow text recognition and low accuracy. Summary of the invention

[0003] In order to solve the above technical problems, the present disclosure provides a text recognition method, device, electronic device and storage medium, which can quickly and accurately recognize text in an image.

[0004] According to one aspect of the present disclosure, a text recognition method is provided, comprising:

[0005] Acquire an image to be recognized, where the image to be recognized is a text image;

[0006] The image to be recognized is input into a pre-built recognition network model for text recognition to obtain a first recognition result, wherein the recognition model includes a feature extraction module, a semantic module, a first recognition module and a second recognition module, the feature extraction module is used to extract features of the image to be recognized to obtain first feature information, the first recognition module is used to recognize the first feature information to obtain a second recognition result, the semantic module is used to calculate a semantic representation vector corresponding to the second recognition result, and the second recognition module is used to obtain the first recognition result based on the first feature information and the semantic representation vector.

[0007] According to another aspect of the present disclosure, there is provided a text recognition device, comprising:

[0008] An acquisition unit, used for acquiring an image to be recognized, wherein the image to be recognized is a text image;

[0009] A recognition unit is used to input the image to be recognized into a pre-built recognition network model for text recognition to obtain a first recognition result, wherein the recognition model includes a feature extraction module, a semantic module, a first recognition module and a second recognition module, the feature extraction module is used to extract features of the image to be recognized to obtain first feature information, the first recognition module is used to recognize the first feature information to obtain a second recognition result, the semantic module is used to calculate a semantic representation vector corresponding to the second recognition result, and the second recognition module is used to obtain the first recognition result based on the first feature information and the semantic representation vector.

[0010] According to another aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory storing a program, wherein the program comprises instructions, which, when executed by the processor, enable the processor to perform the above-mentioned text recognition method.

[0011] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute a text recognition method.

[0012] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein the computer program implements the above-mentioned text recognition method when executed by a processor.

[0013] Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has the following advantages:

[0014] The method provided by the present disclosure includes: obtaining an image to be recognized, which is a text image; inputting the image to be recognized into a pre-built recognition network model for text recognition to obtain a first recognition result, wherein the recognition model includes a feature extraction module, a semantic module, a first recognition module, and a second recognition module, extracting features of the image to be recognized by the feature extraction module to obtain first feature information, recognizing the first feature information by the first recognition module to obtain a second recognition result, then calculating a semantic representation vector corresponding to the second recognition result by the semantic module, and obtaining the first recognition result by the second recognition module based on the first feature information and the semantic representation vector. The method provided by the present disclosure can quickly and accurately recognize text in an image. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0017] Figure 1 A schematic diagram of an application scenario provided by an embodiment of the present disclosure;

[0018] Figure 2 A flowchart of a recognition model training method provided in an embodiment of the present disclosure;

[0019] Figure 3 A network structure diagram of a recognition model provided by an embodiment of the present disclosure;

[0020] Figure 4 A flowchart of a text recognition method provided by an embodiment of the present disclosure;

[0021] Figure 5 A flowchart of a text recognition method provided by an embodiment of the present disclosure;

[0022] Figure 6 A structural schematic diagram of a text recognition device provided by an embodiment of the present disclosure;

[0023] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] In order to more clearly understand the above-mentioned purposes, features and advantages of the present disclosure, the embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be interpreted as being limited to the embodiments described herein. Instead, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0025] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0026] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc. mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0027] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0028] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0029] At present, text recognition has very wide applications, and the commonly used methods have different problems. For example, the character-based method has high annotation cost, the sequence-based method may have the problem of missing or over-recognizing characters, and the text represented on different text images includes straight text, inclined text and curved text; most recognition methods are for single-line text and require detection as a prerequisite operation. However, for images with more complex layouts, the detection plus recognition method may introduce more errors and the recognition accuracy is relatively low.

[0030] In response to the above technical problems, the embodiments of the present disclosure provide a text recognition method, which is specifically described in detail through one or more of the following embodiments.

[0031] Specifically, the text recognition method can be executed by a terminal or a server. Specifically, the terminal or the server can recognize the text in the image to be recognized through a recognition model. The execution subject of the training method of the recognition model and the execution subject of the character recognition method can be the same or different.

[0032] For example, in an application scenario, such as Figure 1 As shown, Figure 1 A schematic diagram of an application scenario provided by an embodiment of the present disclosure, Figure 1The server 12 trains the recognition model. The terminal 11 obtains the trained recognition model from the server 12, and the terminal 11 recognizes the text in the image to be recognized by using the trained recognition model. The image to be recognized may be obtained by taking a picture by the terminal 11. Alternatively, the image to be recognized is an image obtained by the terminal 11 after processing a preset image, and the preset image may be obtained by taking a picture by the terminal 11, or the preset image may be obtained by the terminal 11 from other devices. Here, no specific limitation is made to other devices.

[0033] In another application scenario, the server 12 trains the recognition model. Further, the server 12 recognizes the text in the image to be recognized by using the trained recognition model. The way in which the server 12 obtains the image to be recognized can be similar to the way in which the terminal 11 obtains the image to be recognized as described above, which will not be repeated here.

[0034] In another application scenario, the terminal 11 trains the recognition model. Further, the terminal 11 recognizes text in the image to be recognized by using the trained recognition model.

[0035] It is understandable that the recognition model training method and text recognition method provided by the embodiments of the present disclosure are not limited to the several possible scenarios described above. Since the trained recognition model can be applied to the text recognition method, before introducing the text recognition method, the recognition model training method can be introduced below.

[0036] The following takes the server 12 training the recognition model as an example to introduce a recognition model training method, that is, the recognition model training process. It can be understood that the recognition model training method is also applicable to the scenario where the terminal 11 trains the recognition model.

[0037] Figure 2 A flow chart of a recognition model training method provided in an embodiment of the present disclosure, specifically including the following steps: Figure 2 The following steps S210 to S240 are shown:

[0038] S210: Obtain a sample image and annotation text information corresponding to the sample image.

[0039] It can be understood that a large number of text images to be recognized are collected as training data sets for the recognition model, and each sample image in the training data set is scaled to the same size, and the scaled size can be determined according to user needs; then the text in at least part of the sample images in the training data set is annotated, and the character information on the sample images is annotated to obtain the annotated text information corresponding to the sample images, wherein the annotated sample images are used as training samples, and the remaining unlabeled sample images are used as test samples to test the recognition accuracy of the model during the training process; a dictionary of the recognition model is constructed based on the annotated text information corresponding to each annotated sample image, the dictionary includes each character involved in the sample image, and there are no repeated characters in the dictionary, and the recognition model will determine the characters in the dictionary during the process of automatically recognizing text in the image to obtain the recognition result.

[0040] S220: Input the annotated text information into a pre-trained language model to obtain a first prediction representation vector.

[0041] It is understandable that before executing S220, it is necessary to build and train a language model. Specifically, a large number of text strings are collected, and then each string is segmented using a word segmentation tool to obtain the word segmentation result of each string, that is, to obtain an ordered correspondence between each string and multiple words, and the word segmentation result is used as training data. The word segmentation tool can be a Jieba word segmentation tool. For example, string 1 is divided into 3 words by the word segmentation tool, and string 1 and the 3 words correspond in an ordered manner. The ordered correspondence refers to the correspondence according to the connection order of the characters in the string. For example, string 1 is "text recognition method", and the word segmentation tool is used to divide string 1 into 3 words, and the 3 words are "text", "recognition", and "recognition". "Identification" and "method", the ordered correspondence between string 1 and the three words means that "text recognition method" and the three words are sorted in the order of "text", "recognition" and "method"; specifically, the language model can be constructed based on a machine translation model (Transformer model), such as using Bert or ALBert of the Transformer model to construct a language model, or using ELMO based on a bidirectional long short-term memory network to construct a language model. There is no limitation on the method of constructing the language model, and a machine translation model can be selected to construct the language model according to user needs; then a large number of segmented text strings are used to train the constructed language model to obtain a trained language model.

[0042] It can be understood that, based on the above S210, after obtaining the trained language model, the annotated text information corresponding to each sample image is segmented using a word segmentation tool to obtain an ordered word segmentation result, wherein the annotated text information is a character string, and then the ordered word segmentation result is input into the pre-trained language model in sequence. The language model will randomly mask one word in the word segmentation result each time, calculate the overall semantic representation of the annotated text information, and obtain a first predicted representation vector. The first predicted representation vector refers to a complete representation of the predicted annotated text information obtained after masking one word. For example, based on the above example, the annotated text information is recorded as character string 1, and there are 3 words in orderly correspondence in character string 1. The first word of the 3 words is masked, and the representation vector 1 of character string 1 is obtained based on the language model. The second word of the 3 words is masked, and the representation vector 2 of character string 1 is obtained based on the language model. The representation vector obtained each time the words masked are different may be different.

[0043] Optionally, the above S220 specifically includes the following steps S221 to S224:

[0044] S221, performing word segmentation processing on the annotated text information to obtain multiple word segments.

[0045] S222. For the multiple participles, mask one participle of the multiple participles each time to obtain multiple masked participles.

[0046] S223. Input the plurality of masked word segmentations into a pre-trained language model in sequence to obtain a plurality of third prediction representation vectors.

[0047] S224. Obtain a first prediction characterization vector according to the plurality of third prediction characterization vectors.

[0048] It can be understood that, based on the above S210, the annotated text information is segmented to obtain a segmentation result, which includes multiple ordered words. For the multiple ordered words, one word in the multiple ordered words is masked each time to obtain multiple masked words. The multiple masked words are sequentially input into a pre-trained language model to obtain multiple third prediction representation vectors. That is, for the same annotated text information, based on the different words masked each time, multiple different semantic representation vectors are obtained, and a word includes at least one character. Subsequently, a first prediction representation vector is obtained based on the multiple third prediction representation vectors output by the language model. Specifically, the multiple third prediction representation vectors can be added point by point to obtain an average to obtain the first prediction representation vector of the annotated text information. Based on the above example, for string 1, the remaining two words after masking the first word are input into the language model to obtain characterization vector 1 of string 1, the remaining two words after masking the second word are input into the language model to obtain characterization vector 2 of string 1, and the remaining two words after masking the third word are input into the language model to obtain characterization vector 3 of string 1. The three characterization vectors are recorded as the three third predicted characterization vectors of string 1, and one first predicted characterization vector is obtained according to the three third predicted characterization vectors. It is understandable that for each string, the number of times words are masked can be preset, or the number of times global semantic features are extracted for each string can be preset, and different words are masked first. Each time a word is masked, a characterization vector will be obtained, and masking the same word will obtain the same characterization vector. For strings that are masked more times than the number of words, repeated words can be masked to obtain repeated characterization vectors.

[0049] S230: Input the sample image, the annotated text information and the first prediction characterization vector into a pre-built recognition model to obtain a second prediction characterization vector, a first prediction recognition result and a second prediction recognition result.

[0050] Optionally, the above S230 specifically includes the following steps: inputting the sample image, the annotated text information and the first predicted representation vector into the pre-constructed recognition model, so that the feature extraction module in the recognition model extracts the features of the sample image, the semantic module obtains the second predicted representation vector based on the annotated text information, the first recognition module obtains the first predicted recognition result based on the features of the sample image, and the second recognition module obtains the second predicted recognition result based on the features of the sample image and the first predicted representation vector.

[0051] It is understandable that, based on the above S220, the sample image, the annotated text information and the first prediction representation vector are input into a pre-built recognition model to train the recognition model, wherein the recognition model includes a feature extraction module, a semantic module, a first recognition module and a second recognition module. During the training process, the operation steps inside the recognition model are as follows: the feature extraction module in the recognition model extracts the features of the sample image; the semantic module determines the semantic representation of at least one word randomly masked in the annotated text information, and obtains a second predicted representation vector corresponding to the complete annotated text information, wherein the annotated text information is in the form of a character string, and the number of randomly masked words can be the same as the number of masks set by the above-mentioned language model, which is not limited here, and the masked words include at least one character. During the training process of the recognition model, the semantic module is trained separately and does not participate in the training of other modules; the first recognition module in the recognition model obtains a first predicted recognition result based on the features of the sample image; the second recognition module in the recognition model obtains a second predicted recognition result based on the features of the sample image and the first predicted representation vector, and the first predicted representation vector is the output of the trained language model. During the training process of the recognition model, one of the inputs of the second recognition module is the output of the language model, and the first predicted representation vector output by the language model can be understood as an accurate semantic representation vector to improve the recognition accuracy of the second recognition module. During the use of the recognition model, the output of the semantic module replaces the output of the language model as one of the inputs of the second recognition module, that is, the semantic module and the second recognition module are trained separately to further ensure the accuracy of the recognition model.

[0052] For example, see Figure 3 , Figure 3 A network structure diagram of a recognition model provided in an embodiment of the present disclosure. Figure 3The recognition model 300 includes a feature extraction module 310, a semantic module 320, a first recognition module 330 and a second recognition module 340, wherein the feature extraction module 310 includes a first feature extraction submodule 311 and a second feature extraction submodule 312. The first feature extraction submodule 311 uses a residual network as a backbone network. The residual network can be Resnet18. The residual network contains 4 convolution blocks, which are denoted as block blocks. Each convolution block includes several convolution layers. The feature information of the sample image is obtained by adjusting the sliding stride length of the convolution operation in different convolution blocks. The feature information is a set of feature maps with a size of 1 / 32 of the input sample image height, 1 / 4 of the width, and a channel number of 512. The second feature extraction submodule 312 is composed of a two-layer bidirectional long short-term memory network (LSTM, Long Short-Term The second feature extraction submodule 312 is composed of the feature information output by the first feature extraction submodule 311, and then outputs the feature information in the same manner, and the output feature information is of the same size as the feature information output by the first feature extraction submodule 311. The second feature extraction submodule 312 mainly performs context modeling on a set of feature maps extracted by the first feature extraction submodule 311, performs enhancement, and outputs the enhanced feature maps; the semantic module 320 includes a word embedding layer 321 and a two-layer bidirectional long short-term memory network layer 322. Specifically, the semantic module 320 may include a word embedding layer 321 and a two-layer bidirectional long short-term memory network layer 322. In the process of recognizing the model training, the input of the semantic module 320 is the annotated text information corresponding to the sample image. In the process of using the recognition model, the input of the semantic model 320 is Figure 3The recognition result output by the first recognition model 330 shown by the dotted line, wherein the annotated text information is the accurate recognition result of the text in the sample image, the time step can be set to 10, and a representation vector is output, and the dimension of the representation vector is the same as the dimension of the representation vector output by the language model; the first recognition module 330 includes a fully connected layer, the input of the first recognition module 330 is the output of the second feature extraction submodule 312, and a high-dimensional tensor is output, the high-dimensional tensor is the probability matrix corresponding to the character recognition result of the text on the sample image predicted, the size of the probability matrix is ​​the same as the size of the constructed dictionary, the probability matrix shows the similarity between the characters in the sample image and the characters in the dictionary, and the characters with higher similarity are the final character recognition results; the second recognition module 340 includes an attention layer 341, a decoding layer 342 and a fully connected layer 343. The second recognition module 340 includes an attention layer 341, a decoding layer 342 and a fully connected layer 343 as an example for explanation, the fully connected layer 343 is used for dimensionality transformation, and the input of the attention layer 341 is the second feature extraction submodule 312. The output feature information, the feature vector output by each decoding layer and the first predicted representation vector output by the language model are output as an input vector, wherein the feature information output by the second feature extraction submodule 312 is used as the key (Key) and value (Value) of the attention layer 341, and the vector obtained by splicing the feature vector output by each decoding layer and the first predicted representation vector output by the language model is used as the query parameter (Query) of the attention layer 341, and the similarity between the query parameter and the key is calculated to generate a score, and the score and the value are weighted averaged to obtain the input vector, and the input of the decoding layer 342 is the input vector output by the attention layer 341 and the representation vector output by the language model, and the input vector and the representation vector are linearly transformed and spliced, and the decoding layer 342 outputs a feature vector according to the splicing result, and the input of the fully connected layer 343 is the feature vector output by the decoding layer 342. If the second recognition module 340 performs a loop processing of multiple time steps at the decoding layer, the input of each decoding layer also includes the output of the decoding layer of the previous time step. In the use of the recognition model, Figure 3 The dashed line shows that the output of the semantic module 320 replaces the first prediction representation vector output by the language model as one of the inputs of the second recognition module 340 .

[0053] S240, calculating a loss value according to the annotated text information, the first prediction characterization vector, the second prediction characterization vector, the first prediction recognition result, and the second prediction recognition result, and updating a network parameter of the recognition model according to the loss value.

[0054] Optionally, the loss value includes a first loss value, a second loss value and a third loss value.

[0055] Optionally, the above S240 calculates the loss value according to the annotated text information, the first prediction characterization vector, the second prediction characterization vector, the first prediction recognition result, and the second prediction recognition result, and specifically includes the following steps S241 to S243:

[0056] S241. Calculate the first loss value according to the first prediction characterization vector and the second prediction characterization vector using a first loss function.

[0057] S242: Calculate the second loss value according to the annotated text information and the first predicted recognition result using a second loss function.

[0058] S243: Calculate the third loss value according to the annotated text information and the second predicted recognition result using a third loss function.

[0059] It is understandable that, based on the above S230, a first loss function is used to calculate a first loss value based on the first predicted representation vector and the second predicted representation vector. The first loss function is a mean square error loss function, and the second predicted representation vector is understood as an accurate representation vector for training the semantic module. A second loss function is used to calculate a second loss value based on the annotated text information and the first predicted recognition result. The second loss function is a CTC loss function, wherein the annotated text information is an accurate recognition result corresponding to the characters in the sample image. A third loss function is used to calculate a third loss value based on the annotated text information and the second predicted recognition result. The third loss function is a multi-classification cross entropy loss function. It is understandable that, except for the feature extraction module 310 in the recognition model, which does not calculate the loss value, all other modules calculate the loss value.

[0060] The present disclosure provides a recognition model training method, which trains a constructed recognition model by acquiring a large number of sample images and annotated text information corresponding to the sample images. At the same time, the annotated text information is input into a pre-trained language model to extract global semantic information to obtain a first prediction representation vector, which is an accurate semantic representation vector of the annotated text information. The first prediction representation vector is used to replace the output of the semantic module as an input of the second recognition module to improve the recognition accuracy of the second recognition module. At the same time, the first prediction representation vector is used as the supervision information of the semantic module in the recognition model, and the semantic module is trained separately to obtain an accurate global semantic vector. It also ensures that the input of the semantic module is the same during training and use. The input of the semantic module is in character string format. This training method can increase the recognition accuracy of the recognition model while further improving the training speed of the recognition model.

[0061] On the basis of the above embodiment, after obtaining the trained recognition model, a text recognition method can be implemented based on the recognition model. Specifically, Figure 4 A flowchart of a text recognition method provided by an embodiment of the present disclosure specifically includes the following steps: Figure 4 The following steps S410 to S420 are shown:

[0062] S410: Acquire an image to be recognized, where the image to be recognized is a text image.

[0063] It is understandable that an image to be recognized is obtained, and the image to be recognized is a text image, and the text image includes at least one character. After obtaining the image to be recognized, the size of the image to be recognized can also be adjusted to the size of the image that can be processed by the recognition model. Specifically, the size of the image to be recognized can be set to the size of the sample image input during the training process of the above recognition model.

[0064] S420. Input the image to be recognized into a pre-built recognition network model for text recognition to obtain a first recognition result, wherein the recognition model includes a feature extraction module, a semantic module, a first recognition module and a second recognition module, use the feature extraction module to extract features of the image to be recognized to obtain first feature information, use the first recognition module to recognize the first feature information to obtain a second recognition result, use the semantic module to calculate a semantic representation vector corresponding to the second recognition result, and use the second recognition module to obtain the first recognition result based on the first feature information and the semantic representation vector.

[0065] It can be understood that, based on the above S410, the acquired image to be recognized is input into a pre-trained recognition model so that the recognition model recognizes the text of the image to be recognized and outputs a first recognition result, which refers to the recognition result of the characters included in the image to be recognized. For the image to be recognized, the recognition process within the recognition model includes: after the image to be recognized is input into the recognition model, the feature extraction module in the recognition model extracts the features of the image to be recognized to obtain the first feature information, which can also be understood as feature mapping; then the first feature information is input into the first recognition module in the recognition model, the first recognition module recognizes according to the first feature information, and outputs a probability matrix, and the second recognition result can be determined in the dictionary corresponding to the recognition model according to the probability matrix, and the second recognition result is the character string corresponding to the text in the image to be recognized; then the semantic module in the recognition model is used to calculate the semantic representation vector corresponding to the second recognition result, specifically, the second recognition result is segmented, and the obtained segmentation result includes at least one word, each word includes at least one character, and a word in the segmentation result is randomly masked, and then the semantic representation vector of the entire second recognition result is calculated based on the masked segmentation result by the semantic module, that is, after masking a word, the masked word is predicted by the semantic module according to the masked segmentation result, and then the semantic representation vector corresponding to the entire segmentation result is obtained; the second recognition module continues to recognize the text in the image to be recognized according to the first feature information and the semantic representation vector, and finally obtains the first recognition result with accurate recognition.

[0066] Optionally, the feature extraction module includes a first feature extraction submodule and a second feature extraction submodule.

[0067] Optionally, the step of extracting the features of the image to be identified by using the feature extraction module to obtain the first feature information in S420 specifically includes the following steps S421 to S422:

[0068] S421. Utilize the first feature extraction submodule to extract features of the image to be identified to obtain second feature information.

[0069] S422: Use the second feature extraction submodule to perform context modeling on the second feature information to obtain first feature information.

[0070] It can be understood that after the image to be recognized is input into the recognition model, the first feature extraction submodule in the recognition model extracts the features of the image to be recognized and obtains the second feature information. Specifically, the first convolution block of the four convolution blocks in the first feature extraction submodule extracts the features of the image to be recognized and obtains a set of feature maps. The input of the first convolution block is the image to be recognized, and the size of the feature map is h / 32*w / 4*512, where the size of the image to be recognized is h*w and the number of channels is 512. Then the output of the first convolution block is used as the input of the second convolution block. The input of the second feature extraction submodule is the output of the first feature extraction submodule, that is, the output of the lower 4 convolution blocks. The two-layer bidirectional long short-term memory network in the second feature extraction submodule performs context modeling on the second feature information and outputs the second feature information, that is, further strengthening the features of the text in the second feature information, so as to facilitate the subsequent recognition of text according to the feature information.

[0071] Optionally, the semantic module includes a word embedding layer and a long short-term memory network layer.

[0072] Optionally, the step of calculating the semantic representation vector corresponding to the second recognition result by using the semantic module in S420 specifically includes the following steps S423 to S424:

[0073] S423. Randomly mask at least one character in the second recognition result, and convert the second recognition result after masking at least one character into a target embedding vector using the word embedding layer, wherein the second recognition result includes at least one character.

[0074] S424. Calculate a semantic representation vector of the target embedding vector using the long short-term memory network layer.

[0075] Optionally, the above S423 specifically includes: randomly masking at least one character in the second recognition result multiple times, and the at least one character masked each time is not exactly the same, so as to obtain multiple masked results corresponding to the second recognition result; for each masked result, using the word embedding layer to convert the masked result into an embedding vector; and obtaining a target embedding vector based on the multiple embedding vectors.

[0076] It can be understood that the implementation process of the semantic module in the recognition model outputting the semantic representation vector corresponding to the second recognition result includes: corresponding to the semantic module, randomly masking a word in the first recognition result to obtain a masked result, a word includes at least one character, and in the semantic module, it can be limited to randomly masking 2 or 3 characters, and directly obtaining a target embedding vector based on a masked result; or, randomly masking at least one character multiple times, and at least one character masked each time is not exactly the same, to obtain multiple masked results, that is, there are multiple corresponding masked results for one second recognition result; then the word embedding layer in the semantic module converts the second recognition result after masking at least one character into an embedding vector, that is, for each masked result, the masked result is converted into an embedding vector using the word embedding layer to obtain multiple embedding vectors, and then the embedding vector is obtained based on the multiple embedding vectors. Specifically, the multiple embedding vectors can be averaged to obtain a target embedding vector; the bidirectional long short-term memory network layer in the semantic module calculates the semantic representation vector corresponding to the complete second recognition result based on the target embedding vector, and the complete second recognition result is composed of predicted masked characters and unmasked characters. For example, the second recognition result includes a character string of "text recognition method". The second recognition result after masking 2 characters in a random masking process is "text method". The second recognition result after masking 2 characters in a second random masking process is "recognition method". The characters randomly masked each time are not exactly the same, that is, there may be one character in the 2 masked characters that is the same as one of the other 2 masked characters. The following description is explained using only one random masking as an example. The "text method" is input into the semantic module, the 2 masked characters are predicted, and the semantic representation vector corresponding to the complete second recognition result is output. The complete second recognition result refers to the recognition result obtained after the predicted 2 characters are added to the masked second recognition result. It is understandable that the second recognition result is in the form of a character string, and the annotated text information input by the semantic module during training is also in the string format, ensuring that the semantic module input is in the same format during training and use.

[0077] The disclosed embodiment provides a text recognition method, which obtains an image to be recognized, inputs the image to be recognized into a pre-trained recognition model, and outputs a recognition result of the text in the image to be recognized. The recognition model extracts the global semantic features of the image to be recognized while extracting the features of the image to be recognized, thereby strengthening the features extracted by the recognition model and further improving the recognition accuracy of the recognition model. The semantic module inputs a character string during training and use. Specifically, the semantic module inputs a first recognition result in the form of a character string obtained according to the first feature information output by the feature extraction module during use. There is no need to add other modules to transform and adjust the input of the semantic module. While simplifying the structure of the recognition model, it can also further improve the accuracy of the recognition model and the recognition speed of the recognition model.

[0078] Based on the above embodiments, Figure 5 A flowchart of a text recognition method provided in an embodiment of the present disclosure, wherein optionally, the first recognition result is obtained by using the second recognition module based on the first feature information and the semantic representation vector, specifically including the following steps: Figure 5 The following steps S510 to S540 are shown:

[0079] Optionally, the second recognition module includes an attention layer, a decoding layer and a fully connected layer.

[0080] It is understandable that the following embodiments are described by taking the example that the second recognition module includes an attention layer, a decoding layer and a fully connected layer.

[0081] Optionally, the first feature information includes feature information of N time steps, where N is an integer greater than 1.

[0082] S510. Use the attention layer to process the first feature information to obtain a context vector for each time step.

[0083] It can be understood that the first feature information includes feature information of N time steps, where N is a positive integer greater than 1. The attention layer extracts the semantic representation of each time step to obtain the context information of each time step.

[0084] S520. In the decoding layer, for the first time step, the obtained initial hidden state vector and the semantic representation vector are concatenated to obtain a first concatenated vector, and a first output vector and a target hidden state vector for the first time step are generated according to the first concatenated vector and the context vector of the first time step.

[0085] It can be understood that, based on the above S510, an initial hidden state vector is obtained. For the first time step, the initial hidden state vector can be understood as a preset hidden state vector. The initial hidden state vector can be an all-zero vector. The initial hidden state vector is the same as the initial hidden state vector input during the training process of the second recognition module; the initial hidden state vector and the semantic feature vector output by the semantic module are spliced ​​to obtain a first splicing vector. The decoding layer in the second recognition module generates a first output vector and a target hidden state vector for the first time step according to the first splicing vector and the context vector of the first time step. The target hidden state vector for the first time step is used as an input for the second time step in the decoding layer.

[0086] S530. For the Mth current time step, concatenate the target hidden state vector output in the previous time step of the Mth current time step and the semantic representation vector to obtain a current concatenation vector, and generate a current output vector corresponding to the Mth current time step and a target hidden state vector of the Mth current time step according to the current concatenation vector and the context vector of the Mth current time step, where M is an integer greater than 1 and less than N.

[0087] It is understandable that, based on the above S520, after the first time step of the decoding layer outputs the first output vector and the target hidden state vector, for the Mth time step, the Mth time step is the current execution time step, M is greater than 1 and less than N, the target hidden state vector and the semantic representation vector output in the previous time step of the Mth time step are spliced ​​to obtain the current splicing vector, and then the current output vector of the Mth time step and the target hidden state vector of the Mth time step are generated according to the current splicing vector after linear processing and the context vector of the Mth time step, that is, an output vector and a target hidden state vector are output for each time step. It is understandable that the current output vector is more accurate than the output vector of the first time step.

[0088] S540. For the other time steps among the N time steps except the first time step, sequentially execute the process of obtaining the current output vector corresponding to the Mth current time step until the recognition of N time steps is completed, and obtain the first recognition result based on the fully connected layer according to the first output vector and the current output vectors of the N-1 current time steps.

[0089] It can be understood that, based on the above S530, the second recognition module includes an attention layer and a decoding layer, and the number of loops can be set, which is N time steps. The same attention layer and decoding layer are processed according to N time steps. For the other time steps except the first time step in the N time steps, the process of obtaining the current output vector corresponding to the Mth current time step is performed in sequence until the recognition of N time steps is completed. For each time step, one of the inputs can be understood as the current output vector and the target hidden state vector output in the previous time step, until the number of loops is reached, the first output vector output by the decoding layer in each time step and the current output vector of N-1 current time steps are used as the input of the fully connected layer to obtain the first recognition result. Specifically, after the decoding layer outputs the final output vector obtained by N output vectors, the fully connected layer transforms the dimension of the final output vector so that the transformed dimension is the same as the dimension of the dictionary constructed above. The final output vector after the dimension transformation can be understood as a probability vector. For a single character, the position of the maximum value of the element in the probability vector is determined, and the specific corresponding character is obtained at the same position in the dictionary, and the first recognition result is obtained from multiple characters.

[0090] The embodiment of the present disclosure provides a text recognition method. The second recognition module uses an attention layer and a decoding layer to obtain a first recognition result based on first feature information and a semantic representation vector. The attention layer processes the first feature information to obtain a context vector for each time step. Then, in the decoding layer, for each time step, an output vector is obtained based on the target hidden state vector, context information and semantic representation vector of the previous time step. Finally, a probability matrix corresponding to the character string of the text on the image to be recognized is obtained based on the N output vectors. An accurate recognition result (first recognition result) is obtained based on the probability matrix. The method provided by the present disclosure can extract the features of the text in the image to be recognized to the maximum extent, and further improve the recognition accuracy by performing multiple loop iterations in the decoding layer.

[0091] Figure 6 The text recognition device provided by the embodiment of the present disclosure can execute the processing flow provided by the text recognition method embodiment, such as Figure 6 As shown, the text recognition device 600 includes:

[0092] An acquisition unit 610 is used to acquire an image to be recognized, where the image to be recognized is a text image;

[0093] The recognition unit 620 is used to input the image to be recognized into a pre-built recognition network model for text recognition to obtain a first recognition result, wherein the recognition model includes a feature extraction module, a semantic module, a first recognition module and a second recognition module, the feature extraction module is used to extract the features of the image to be recognized to obtain first feature information, the first recognition module is used to recognize the first feature information to obtain a second recognition result, the semantic module is used to calculate the semantic representation vector corresponding to the second recognition result, and the second recognition module is used to obtain the first recognition result based on the first feature information and the semantic representation vector.

[0094] Optionally, the feature extraction module in the recognition unit 620 includes a first feature extraction submodule and a second feature extraction submodule; the feature extraction module is used to extract the features of the image to be recognized to obtain the first feature information, which is specifically used for:

[0095] Extracting features of the image to be identified using the first feature extraction submodule to obtain second feature information;

[0096] The second feature extraction submodule is used to perform context modeling on the second feature information to obtain first feature information.

[0097] Optionally, the semantic module in the recognition unit 620 includes a word embedding layer and a long short-term memory network layer; the using the semantic module to calculate the semantic representation vector corresponding to the second recognition result is specifically used for:

[0098] Randomly masking at least one character in the second recognition result, and converting the second recognition result after masking at least one character into a target embedding vector using the word embedding layer, wherein the second recognition result includes at least one character;

[0099] The semantic representation vector of the target embedding vector is calculated using the long short-term memory network layer.

[0100] Optionally, the recognition unit 620 randomly masks at least one character in the second recognition result, and uses the word embedding layer to convert the second recognition result after masking at least one character into a target embedding vector, specifically for:

[0101] Randomly masking at least one character of the second recognition result multiple times, wherein the at least one character masked each time is not exactly the same, to obtain multiple masked results corresponding to the second recognition result;

[0102] For each masking result, convert the masking result into an embedding vector using the word embedding layer;

[0103] A target embedding vector is obtained according to the multiple embedding vectors.

[0104] Optionally, the second recognition module in the recognition unit 620 includes an attention layer, a decoding layer, and a fully connected layer; the first feature information includes feature information of N time steps, where N is an integer greater than 1; and the using the second recognition module to obtain the first recognition result based on the first feature information and the semantic representation vector is specifically used to:

[0105] Using the attention layer to process the first feature information, obtaining a context vector for each time step;

[0106] In the decoding layer, for a first time step, the obtained initial hidden state vector and the semantic representation vector are concatenated to obtain a first concatenated vector, and a first output vector and a target hidden state vector of the first time step are generated according to the first concatenated vector and the context vector of the first time step;

[0107] For the Mth current time step, concatenate the target hidden state vector outputted in the previous time step of the Mth current time step and the semantic representation vector to obtain a current concatenation vector, and generate a current output vector corresponding to the Mth current time step and a target hidden state vector of the Mth current time step according to the current concatenation vector and the context vector of the Mth current time step, where M is an integer greater than 1 and less than N;

[0108] For the other time steps in the N time steps except the first time step, the process of obtaining the current output vector corresponding to the Mth current time step is sequentially performed until the recognition of the N time steps is completed, and the first recognition result is obtained based on the fully connected layer according to the first output vector and the current output vectors of the N-1 current time steps. .

[0109] Optionally, the apparatus 600 further includes a training unit, and the training unit is specifically configured to:

[0110] Obtaining a sample image and annotated text information corresponding to the sample image;

[0111] Inputting the annotated text information into a pre-trained language model to obtain a first prediction representation vector;

[0112] Inputting the sample image, the annotated text information and the first prediction characterization vector into a pre-built recognition model to obtain a second prediction characterization vector, a first prediction recognition result and a second prediction recognition result;

[0113] A loss value is calculated according to the annotated text information, the first predicted characterization vector, the second predicted characterization vector, the first predicted recognition result, and the second predicted recognition result, and a network parameter of the recognition model is updated according to the loss value.

[0114] Optionally, the inputting of the sample image, the annotated text information and the first prediction characterization vector into a pre-built recognition model in the training unit to obtain the second prediction characterization vector, the first prediction recognition result and the second prediction recognition result is specifically used for:

[0115] The sample image, the annotated text information and the first predicted representation vector are input into the pre-constructed recognition model, so that the feature extraction module in the recognition model extracts the features of the sample image, the semantic module obtains the second predicted representation vector based on the annotated text information, the first recognition module obtains the first predicted recognition result based on the features of the sample image, and the second recognition module obtains the second predicted recognition result based on the features of the sample image and the first predicted representation vector.

[0116] Optionally, the loss value in the training unit includes a first loss value, a second loss value, and a third loss value; the loss value is calculated according to the annotated text information, the first prediction characterization vector, the second prediction characterization vector, the first prediction recognition result, and the second prediction recognition result, specifically used for:

[0117] Calculating the first loss value according to the first prediction characterization vector and the second prediction characterization vector using a first loss function;

[0118] Calculating the second loss value according to the annotated text information and the first predicted recognition result using a second loss function;

[0119] The third loss value is calculated according to the annotated text information and the second predicted recognition result using a third loss function.

[0120] Optionally, the inputting of the annotated text information into a pre-trained language model in the training unit to obtain a first prediction representation vector is specifically used for:

[0121] Performing word segmentation processing on the marked text information to obtain multiple word segments;

[0122] For the multiple participles, masking one participle of the multiple participles each time to obtain multiple masked participles;

[0123] Inputting the plurality of masked word segmentations into a pre-trained language model in sequence to obtain a plurality of third prediction representation vectors;

[0124] A first prediction characterization vector is obtained according to the plurality of third prediction characterization vectors.

[0125] The implementation principle and technical effects of the device provided in this embodiment are the same as those of the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the aforementioned method embodiment.

[0126] The exemplary embodiment of the present disclosure also provides an electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication. The memory stores a computer program that can be executed by the at least one processor, and the computer program is used to cause the electronic device to perform the method according to the embodiment of the present disclosure when executed by the at least one processor.

[0127] The exemplary embodiments of the present disclosure further provide a computer program product, including a computer program, wherein when the computer program is executed by a processor of a computer, the computer is used to enable the computer to perform the method according to the embodiments of the present disclosure.

[0128] refer to Figure 7 , a block diagram of an electronic device 700 that can be used as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0129] like Figure 7 As shown, the electronic device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0130] A plurality of components in the electronic device 700 are connected to the I / O interface 705, including: an input unit 706, an output unit 707, a storage unit 708, and a communication unit 709. The input unit 706 may be any type of device capable of inputting information to the electronic device 700, and the input unit 706 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 707 may be any type of device capable of presenting information, and may include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 708 may include but is not limited to a disk, an optical disk. The communication unit 709 allows the electronic device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and may include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0131] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above. For example, in some embodiments, the text recognition method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 700 via the ROM 702 and / or the communication unit 709. In some embodiments, the computing unit 701 may be configured to perform the text recognition method in any other appropriate manner (e.g., by means of firmware).

[0132] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0133] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0134] As used in this disclosure, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0135] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0136] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0137] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0138] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A text recognition method, characterized in that: include: Acquire an image to be recognized, where the image to be recognized is a text image; The image to be recognized is input into a pre-built recognition model for text recognition to obtain a first recognition result, wherein the recognition model includes a feature extraction module, a semantic module, a first recognition module and a second recognition module, the feature extraction module is used to extract features of the image to be recognized to obtain first feature information, the first recognition module is used to recognize the first feature information to obtain a second recognition result, and the semantic module is used to calculate a semantic representation vector corresponding to the second recognition result, including: randomly masking at least one character in the second recognition result multiple times, and the at least one character masked each time is not exactly the same, to obtain multiple masked results corresponding to the second recognition result, wherein the second recognition result includes at least one character; for each masked result, the word embedding layer included in the semantic module is used to convert the masked result into an embedding vector; a target embedding vector is obtained based on the multiple embedding vectors; the long short-term memory network layer included in the semantic module is used to calculate the semantic representation vector of the target embedding vector, and the second recognition module is used to obtain the first recognition result based on the first feature information and the semantic representation vector.

2. The method according to claim 1, characterized in that: The feature extraction module includes a first feature extraction submodule and a second feature extraction submodule; and extracting the features of the image to be identified by using the feature extraction module to obtain the first feature information includes: Extracting features of the image to be identified using the first feature extraction submodule to obtain second feature information; The second feature extraction submodule is used to perform context modeling on the second feature information to obtain first feature information.

3. The method according to claim 1, characterized in that The second recognition module includes an attention layer, a decoding layer, and a fully connected layer; the first feature information includes feature information of N time steps, where N is an integer greater than 1; and obtaining the first recognition result based on the first feature information and the semantic representation vector by using the second recognition module includes: Using the attention layer to process the first feature information, obtaining a context vector for each time step; In the decoding layer, for a first time step, the obtained initial hidden state vector and the semantic representation vector are concatenated to obtain a first concatenated vector, and a first output vector and a target hidden state vector of the first time step are generated according to the first concatenated vector and the context vector of the first time step; For the Mth current time step, concatenate the target hidden state vector outputted in the previous time step of the Mth current time step and the semantic representation vector to obtain a current concatenation vector, and generate a current output vector corresponding to the Mth current time step and a target hidden state vector of the Mth current time step according to the current concatenation vector and the context vector of the Mth current time step, where M is an integer greater than 1 and less than N; For the other time steps among the N time steps except the first time step, the process of obtaining the current output vector corresponding to the Mth current time step is sequentially performed until the recognition of the N time steps is completed, and the first recognition result is obtained based on the fully connected layer according to the first output vector and the current output vectors of the N-1 current time steps.

4. The method according to claim 1, characterized in that: The recognition model is trained by the following method: Obtaining a sample image and annotated text information corresponding to the sample image; Inputting the annotated text information into a pre-trained language model to obtain a first prediction representation vector; Inputting the sample image, the annotated text information and the first prediction characterization vector into a pre-built recognition model to obtain a second prediction characterization vector, a first prediction recognition result and a second prediction recognition result; A loss value is calculated according to the annotated text information, the first predicted characterization vector, the second predicted characterization vector, the first predicted recognition result, and the second predicted recognition result, and a network parameter of the recognition model is updated according to the loss value.

5. The method according to claim 4, characterized in that The step of inputting the sample image, the annotated text information and the first prediction characterization vector into a pre-built recognition model to obtain a second prediction characterization vector, a first prediction recognition result and a second prediction recognition result includes: The sample image, the annotated text information and the first predicted representation vector are input into the pre-constructed recognition model, so that the feature extraction module in the recognition model extracts the features of the sample image, the semantic module obtains the second predicted representation vector based on the annotated text information, the first recognition module obtains the first predicted recognition result based on the features of the sample image, and the second recognition module obtains the second predicted recognition result based on the features of the sample image and the first predicted representation vector.

6. The method according to claim 4, characterized in that The loss value includes a first loss value, a second loss value and a third loss value; the loss value is calculated according to the annotated text information, the first prediction characterization vector, the second prediction characterization vector, the first prediction recognition result and the second prediction recognition result, including: Calculating the first loss value according to the first prediction characterization vector and the second prediction characterization vector using a first loss function; Calculating the second loss value according to the annotated text information and the first predicted recognition result using a second loss function; The third loss value is calculated according to the annotated text information and the second predicted recognition result using a third loss function.

7. The method according to claim 4, characterized in that The step of inputting the annotated text information into a pre-trained language model to obtain a first prediction representation vector includes: Performing word segmentation processing on the marked text information to obtain multiple word segments; For the multiple participles, masking one participle of the multiple participles each time to obtain multiple masked participles; Inputting the plurality of masked word segmentations into a pre-trained language model in sequence to obtain a plurality of third prediction representation vectors; A first prediction characterization vector is obtained according to the plurality of third prediction characterization vectors.

8. A text recognition device, characterized in that: include: An acquisition unit, used for acquiring an image to be recognized, wherein the image to be recognized is a text image; A recognition unit is used to input the image to be recognized into a pre-built recognition model for text recognition to obtain a first recognition result, wherein the recognition model includes a feature extraction module, a semantic module, a first recognition module and a second recognition module, the feature extraction module is used to extract the features of the image to be recognized to obtain first feature information, the first recognition module is used to recognize the first feature information to obtain a second recognition result, and the semantic module is used to calculate a semantic representation vector corresponding to the second recognition result, including: randomly masking at least one character in the second recognition result multiple times, and the at least one character masked each time is not exactly the same, to obtain multiple masked results corresponding to the second recognition result, wherein the second recognition result includes at least one character; for each masked result, the word embedding layer included in the semantic module is used to convert the masked result into an embedding vector; a target embedding vector is obtained based on the multiple embedding vectors; the long short-term memory network layer included in the semantic module is used to calculate the semantic representation vector of the target embedding vector, and the second recognition module is used to obtain the first recognition result based on the first feature information and the semantic representation vector.

9. An electronic device, characterized in that: The electronic device comprises: Processor; and Memory for storing programs, The program includes instructions, which, when executed by the processor, cause the processor to perform the text recognition method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to enable the computer to execute the text recognition method according to any one of claims 1 to 7.

Images