A text recognition method and related apparatus

Abstract

The application provides a text recognition method and related device. In the process of text recognition on a to-be-recognized text image, a first context feature and a first score vector are obtained. In the process of text recognition on a standard answer combined with the first context feature, a second score vector is obtained. Since the first context feature can enhance the attention to the target answer content which is the same as or similar to the shape of the to-be-recognized text in the process of text recognition on the standard answer, the second score vector can reflect whether the standard answer contains the target answer content. Therefore, whether to perform answer excitation on the first score vector can be determined according to the second score vector. If yes, score fusion is performed on the first score vector and the second score vector to obtain an answer excitation recognition result. As can be seen, when the standard answer contains the target answer content, the application can perform answer excitation on the recognition result of the to-be-recognized text itself, thereby improving the recognition accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of text recognition, in particular to a text recognition method and related device. BACKGROUND

[0002] Text recognition specifically refers to recognizing the text content on a picture into an editable text sequence, so as to facilitate people to extract the text information in the picture and further process, such as editing correction, extracting key information, text translation, etc. Since machine recognition can greatly reduce labor costs and improve work efficiency, the text recognition technology is currently widely used in various industries, such as education, finance, politics and law, medical treatment, etc.

[0003] The current mature mainstream text recognition framework is to first crop a single-line picture containing text from an image, and then recognize the single-line text based on the single-line picture, and then concatenate all text lines in sequence to obtain the recognition result of the whole picture, as shown in Figures 1(a) to 1(c)

[0004] Among them, the process of recognizing single-line text based on single-line picture includes: obtaining the context information of the text in the single-line picture by using the prior knowledge of the language model, or learning the context information of the text in the single-line picture by using the recognition model itself, and then recognizing the text in combination with the context information of the text.

[0005] Since the semantic association of the context is relatively rich and obvious in natural scenes, document scenes and other scenes with long text lines, for such scenes, the above method of recognizing text in combination with context information can accurately recognize the text content; but in the scene of answering, word dictation and other scenes requiring users to fill in short text, since the text filled in by the user is short, the context semantic that can be referred to is less, and even it may not be possible to use context information, resulting in the above method of recognizing text in combination with context information is no longer applicable. SUMMARY

[0006] Therefore, the present application provides a text recognition method and related device to solve the problem that the prior art cannot be applied to short text recognition scenarios by using context information for text recognition, and the technical solutions are as follows:

[0007] In a first aspect, a text recognition method is provided, comprising:

[0008] In the process of text recognition on a to-be-recognized text image, a first context feature and a first score vector of the to-be-recognized text image are obtained, wherein the to-be-recognized text image contains to-be-recognized text, and the to-be-recognized text corresponds to a standard answer;

[0009] ​In the process of text recognition of the standard answer in combination with the first context feature, a second score vector is obtained, wherein the first context feature is used to enhance the attention to the target answer content in the process of text recognition of the standard answer, so as to obtain the second score vector capable of reflecting whether the standard answer contains the target answer content, and the target answer content refers to the answer content in the standard answer which is the same as or similar to the shape of the text to be recognized;

[0010] It is determined whether to perform answer excitation on the first score vector according to the second score vector;

[0011] If yes, score fusion is performed on the first score vector and the second score vector to obtain an answer excitation recognition result of the text image to be recognized.

[0012] In a second aspect, a text recognition apparatus is provided, comprising:

[0013] A first recognition unit is configured to obtain a first context feature and a first score vector of a text image to be recognized in the process of text recognition of the text image to be recognized, wherein the text image to be recognized contains text to be recognized, and the text to be recognized corresponds to a standard answer;

[0014] A second recognition unit is configured to obtain a second score vector in the process of text recognition of the standard answer in combination with the first context feature, wherein the first context feature is used to enhance the attention to target answer content in the process of text recognition of the standard answer, so as to obtain the second score vector capable of reflecting whether the standard answer contains the target answer content, and the target answer content refers to the answer content in the standard answer which is the same as or similar to the shape of the text to be recognized;

[0015] An excitation judgment unit is configured to determine whether to perform answer excitation on the first score vector according to the second score vector;

[0016] An answer excitation unit is configured to perform score fusion on the first score vector and the second score vector to obtain an answer excitation recognition result of the text image to be recognized if it is determined to perform answer excitation on the first score vector.

[0017] In a third aspect, an electronic device is provided, comprising a memory and a processor;

[0018] The memory is configured to store a program;

[0019] The processor is configured to execute the program to implement each step of the text recognition method according to any one of the above aspects.

[0020] In a fourth aspect, a storage medium having a computer program stored thereon is provided, and the computer program is executed by a processor to implement each step of the text recognition method according to any one of the above aspects.

[0021] According to the technical solution, the text recognition method provided by the application considers that when a human being faces an ambiguous text to be recognized, the human being often recognizes the text to be recognized in combination with a standard answer, that is, when the shape of the text to be recognized is the same as or similar to the standard answer, the text to be recognized is often recognized as the text content in the standard answer. Therefore, in the process of text recognition of the text image to be recognized, the first context feature and the first score vector of the text image to be recognized are obtained, and in the process of text recognition of the standard answer in combination with the first context feature, the second score vector is obtained. Since the first context feature can enhance the attention to the target answer content that is the same as or similar to the shape of the text to be recognized in the process of text recognition of the standard answer, the second score vector can reflect whether the standard answer contains the target answer content, and therefore, according to the second score vector, it can be determined whether the answer excitation needs to be performed on the first score vector, and if yes, the score fusion is performed on the first score vector and the second score vector to obtain the answer excitation recognition result of the text image to be recognized. As can be seen, whether the text to be recognized contained in the text image to be recognized is a long text or a short text, as long as the text to be recognized corresponds to a standard answer, the recognition result of the text to be recognized itself can be excited by the answer when the standard answer contains the target answer content to obtain the answer excitation recognition result of the text image to be recognized, and the whole process does not need to combine the context information and will not reduce the recognition accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only embodiments of the application, and other drawings can be obtained by those skilled in the art without creative effort on the basis of the provided drawings.

[0023] FIG. 1(a) is a schematic diagram of a text image to be recognized;

[0024] FIG. 1(b) is a schematic diagram of text detection provided by the prior art;

[0025] FIG. 1(c) is a schematic diagram of a text recognition result provided by the prior art;

[0026] Figure 2 FIG. 2 is a flowchart of a text recognition method provided by an embodiment of the application;

[0027] Figure 3 FIG. 3 is a schematic diagram of a character to be recognized provided by an embodiment of the application;

[0028] Figure 4 FIG. 4 is a structural diagram of a text recognition model provided by an embodiment of the application;

[0029] Figure 5 A schematic diagram illustrating the structure and interaction process of the text recognition model and answer incentive model provided in the embodiments of this application;

[0030] Figure 6 This is a schematic diagram of the structure of a text recognition device provided in an embodiment of this application;

[0031] Figure 7 This is a hardware structure block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0032] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0033] like Figures 1(a) to 1(c) As shown, Figure 1(a) is a schematic diagram of the text image to be recognized, Figure 1(b) is a schematic diagram of text detection provided by the prior art, and Figure 1(c) is a schematic diagram of the text recognition result provided by the prior art.

[0034] The existing technology first detects the text to be recognized on the text image to be recognized shown in Figure 1(a) to detect each single-line image shown in Figure 1(b), and then performs text recognition and splicing on each single-line image to obtain the text recognition result shown in Figure 1(c).

[0035] However, existing text recognition technologies require the integration of contextual information (i.e., information about the characters preceding and following each character) within the text to achieve good recognition results. Therefore, existing text recognition methods are more effective for long texts. However, in some scenarios, it is necessary to recognize short texts, where the available contextual information is limited or even nonexistent. In such cases, existing technologies are no longer applicable.

[0036] To address the problems existing in the prior art, this application provides a text recognition method that can be applied to a text recognition device in a text recognition scenario; it can also be applied to other devices that communicate with the text recognition device, such as servers, cloud, or other terminals; or it can be applied simultaneously to a text recognition device in a text recognition scenario and other devices that communicate with the text recognition device.

[0037] The text recognition scenario described above refers to a scenario of text recognition on a to-be-recognized text by using a text recognition device, especially a short text handwritten scenario, for example, a handwritten answering scenario. Of course, the embodiments of the present application are not limited to the short text handwritten scenario, and can also be applied to other scenarios. For example, a non-handwritten scenario of dictating a word (short text), a sentence (short text), etc. on an electronic device.

[0038] It should be further noted that, in addition to the short text recognition scenario, the present application can also be applied to a long text recognition scenario, for example, a scenario of dictating a long paragraph on an electronic device, and the like. That is, as long as the to-be-recognized text corresponds to a standard answer, the text recognition method provided by the present application can be used for text recognition, and the text recognition effect is more consistent with the human subjective recognition effect.

[0039] In order for those skilled in the art to better understand the present application, the text recognition method provided by the present application will be described in detail in the following embodiments.

[0040] Referring to FIG. 1, a flowchart of a text recognition method provided by an embodiment of the present application is shown, which can include the following steps. Figure 2

[0041] In step S101, a first context feature and a first score vector of a to-be-recognized text image are obtained in a text recognition process on the to-be-recognized text image.

[0042] The to-be-recognized text image contains a to-be-recognized text, and the to-be-recognized text corresponds to a standard answer.

[0043] Specifically, the text recognition method provided by the present application can be used for text recognition on a to-be-recognized text image, so as to obtain a first context feature and a first score vector of the to-be-recognized text image in the text recognition process.

[0044] Here, the first context feature of the to-be-recognized text image includes a first context feature of each to-be-recognized character contained in the to-be-recognized text, and the first score vector of the to-be-recognized text image includes a first score vector of each to-be-recognized character contained in the to-be-recognized text.

[0045] It should be noted that the first score vector of any to-be-recognized character can reflect the similarity between the to-be-recognized character and each character in a preset dictionary. Through the first score vector of the to-be-recognized character, the recognition result of the to-be-recognized character itself can be obtained, that is, the recognition result of directly recognizing the to-be-recognized character (without increasing answer incentive).

[0046] ​Optionally, the first score vector of any of the above-mentioned characters to be identified includes multiple dimensions, and the number of dimensions is equal to the number of characters contained in the preset dictionary. For example, if the character to be identified is the number "2" and the preset dictionary includes 10 characters from 0 to 9, then the first score vector of the character to be identified "2" is a 10-dimensional vector, denoted as [p0,p1,p2,...,p9]. Then p0 represents the similarity score between the character to be identified "2" and "0" in the dictionary, p1 represents the similarity score between the character to be identified "2" and "1" in the dictionary, and so on.

[0047] It should also be noted that the "first contextual feature of the character to be identified" in this step is different from the "contextual information" in the prior art. The contextual information of the character to be identified obtained by the prior art using the language model includes the relevant information of the character to be identified and the characters before and after it. However, in the embodiments of this application, the "first contextual feature of the character to be identified" includes the features of the character to be identified and the features of the surrounding neighboring pixels, but does not include the features of the characters before and after it.

[0048] Step S102: In the process of text recognition of the standard answer by combining the first context features, the second score vector is obtained.

[0049] The aforementioned first contextual feature is used to enhance the focus on the content of the target answer during the text recognition process of the standard answer, so as to obtain a second score vector that reflects whether the standard answer contains the content of the target answer. Here, the content of the target answer refers to the answer content in the standard answer that has the same or similar shape as the text to be recognized.

[0050] It should be noted that the second score vector obtained in this step includes the second score vector corresponding to each character to be identified in the text to be identified. The second score vector corresponding to any character to be identified reflects the similarity between the answer character and the characters in the preset dictionary that have the same or similar shape as the character to be identified.

[0051] For example, taking a question-and-answer scenario, suppose the standard answer to a question is "2; 3" (in this embodiment, the standard answer is given at the question level; if a question includes multiple blanks, the standard answers corresponding to the multiple blanks are concatenated with semicolons), and the handwritten characters to be recognized are as follows: Figure 3 As shown, due to Figure 3 If the character to be recognized in the image looks like both 2 and 0, then combine... Figure 3 When the first contextual feature of the character to be recognized is used to perform text recognition on the standard answer, the answer character "2" is related to... Figure 3 If the shapes of the characters to be identified are similar, then more attention will be paid to the answer character "2" during the recognition process, thus obtaining a second score vector that reflects that the standard answer contains the answer character "2".

[0052] For example, suppose Figure 3 The standard answer corresponding to the character to be identified is "4; 8". Since the standard answer does not contain any characters related to this, the correct answer is incorrect. Figure 3 If the answer characters shown are the same or similar in shape as the characters to be identified, that is, the standard answer does not include the target answer content, then a second score vector can be obtained during the recognition process, which can reflect that the standard answer does not include the target answer content.

[0053] Step S103: Determine whether to apply answer incentives to the first score vector based on the second score vector.

[0054] Specifically, since the second score vector reflects whether the standard answer includes the target answer, it can be used to determine whether to incentivize the first score vector. If the second score vector indicates that the standard answer includes the target answer, then the first score vector will be incentivized; if the second score vector indicates that the standard answer does not include the target answer, then the first score vector will not be incentivized.

[0055] Step S104: If yes, then the first score vector and the second score vector are fused to obtain the answer-incentive recognition result of the text image to be recognized.

[0056] In this embodiment, when it is determined that the first score vector needs to be stimulated by the answer, the first score vector and the second score vector can be fused. By fusing the scores, the recognition of this application can be more consistent with human thinking. That is, when the shape of the text to be recognized is the same as or similar to the standard answer, the text to be recognized will be recognized as the text content in the standard answer, thereby improving the recognition accuracy to a certain extent.

[0057] See also Figure 3 If the standard answer to the character to be identified shown is "2; 3", then the character to be identified is likely "2". Therefore, the answer character "2" can be used to identify the character. Figure 3 The recognition result of the character to be recognized shown is used as an incentive, that is, the second score vector obtained under the answer character "2" is combined with... Figure 3 The first score vector of the character to be identified is fused to obtain the answer incentive recognition result "2".

[0058] The text recognition method provided in the application, the text recognition method provided in the application, considering that a human being will often identify the to-be-recognized text in combination with a standard answer when facing an ambiguous to-be-recognized text, that is, when the shape of the to-be-recognized text is the same as or similar to the standard answer, the to-be-recognized text is often identified as the text content in the standard answer. Therefore, in the process of text recognition on the to-be-recognized text image, the first context feature and the first score vector of the to-be-recognized text image are obtained, and in the process of text recognition on the standard answer in combination with the first context feature, the second score vector is obtained.

[0059] Since the first context feature can enhance the attention to the target answer content that is the same as or similar to the shape of the to-be-recognized text in the process of text recognition on the standard answer, the second score vector can reflect whether the standard answer contains the target answer content, therefore, according to the second score vector, it can be determined whether the answer excitation needs to be performed on the first score vector, if yes, the score fusion is performed on the first score vector and the second score vector, and the answer excitation recognition result of the to-be-recognized text image is obtained.

[0060] As can be seen, whether the to-be-recognized text contained in the to-be-recognized text image is long text or short text, as long as the to-be-recognized text corresponds to a standard answer, the recognition result of the to-be-recognized text itself can be answer excited to obtain the answer excitation recognition result of the to-be-recognized text image when the standard answer contains the target answer content, the whole process does not need to combine context information, and the recognition accuracy is not reduced.

[0061] In some embodiments of the application, the process of "step S101, in the process of text recognition on the to-be-recognized text image, the first context feature and the first score vector of the to-be-recognized text image are obtained" is introduced.

[0062] Specifically, the process of text recognition on the to-be-recognized text image can be implemented through a pre-trained text recognition model, and the text recognition model is composed of an encoder and a decoder. The encoder is used to extract the features of the to-be-recognized text image, and the extracted features are used as the first encoding features. The decoder is used to perform text recognition based on the first encoding features to obtain the first recognition result of the to-be-recognized text image. Here, the first recognition result of the to-be-recognized text image includes the first recognition result of each to-be-recognized character contained in the to-be-recognized text.

[0063] The above-mentioned text recognition model is trained by taking the recognition result label of the training text image as the training data, so that the decoder of the text recognition model has the functions of obtaining the first context feature and the first score vector of the to-be-recognized text image.

[0064] Therefore, the process of "obtaining the first contextual features and the first score vector of the text image to be recognized during the text recognition process" includes: extracting features of the text image to be recognized through the encoder in the pre-trained text recognition model, and using the extracted features as the first encoded features; performing text recognition based on the first encoded features through the decoder in the text recognition model, and obtaining the first contextual features and the first score vector of the text image to be recognized during the text recognition process based on the first encoded features.

[0065] To enable those skilled in the art to better understand the embodiments of this application, see [link to relevant documentation]. Figure 4 The diagram shown is a structural schematic of the text recognition model provided in an embodiment of this application.

[0066] Optional, such as Figure 4 As shown, this application can use a fully convolutional network (CNN) as the encoder of the text recognition model. For example, the encoder can be VGG (Visual Geometry Group), Residual Neural Network (ResNet), Dense Convolutional Network (DenseNet), etc.

[0067] For the text image to be recognized, I∈R H*W*3 By inputting this into the encoder of the text recognition model, the features of the text image to be recognized can be extracted. For ease of subsequent explanation, the extracted features, i.e., the first encoded features, will be denoted as F∈R. h*w*c Where H represents the height of the text image to be recognized, W represents the width of the text image to be recognized, h represents the height of the feature map of the text image to be recognized, w represents the width of the feature map of the text image to be recognized, c represents the number of feature channels, and h <H,w<W。

[0068] Optionally, this application may use an autoregressive decoding network based on an attention mechanism and a recurrent neural network as the decoder of the text recognition model. The recurrent neural network may be, for example, a recurrent neural network (RNN), a long short-term memory network (LSTM), or a gated recurrent unit (GRU), etc. Figure 4 Let's take GRU as an example.

[0069] In one of the following embodiments, the method is... Figure 4The process by which the decoder obtains the first contextual features and the first score vector of the text image to be recognized during text recognition based on the first encoded features is described.

[0070] In this embodiment, each character to be recognized contained in the text to be recognized (e.g.) Figure 4 The recognition process shown in 1, 2, 8, 0) is taken as a time step. Then, for each time step, the first hidden state of the time step can be obtained first based on the first hidden state of the previous time step and the first recognition result of the previous time step.

[0071] Wherein, if the time step is the first time step, then the first recognition result of the previous time step is the preset start character (i.e., Figure 4 The SOS shown represents the first hidden state of the time step preceding this time step. The first preset value is used. If this time step is not the first time step, the first recognition result of the previous time step is obtained based on the first score vector of the previous time step. Here, the superscript R indicates that it is a parameter of the text recognition model, and the same applies to the superscript R in the following symbols.

[0072] Next, this embodiment can determine the first attention weight of the time step based on the first hidden state and the first encoding feature of the time step, and obtain the first context information of the time step based on the first attention weight and the first encoding feature of the time step.

[0073] Furthermore, in this embodiment, the first score vector of the time step can be obtained based on the first context information and the first hidden state of the time step, thereby obtaining the first context features and the first score vector of each time step, which are used as the first context features and the first score vector of the text image to be recognized.

[0074] Specifically, taking time step t as an example, such as Figure 4 As shown, the first hidden state of the previous time step can be... The first identification result at the previous time step The encoded features (Embedding) are used as input to the GRU to obtain the first hidden state at that time step.

[0075] Next, as Figure 4 The step A, circled in black in the diagram, can be represented in this embodiment as the first hidden state of that time step. As the query vector, an attention-weighted operation is performed on the first encoded feature F output by the encoder of the text recognition model. The first encoded feature is then weighted and aggregated to obtain the first context information at this time step.

[0076] Here, the detailed process of attention weighting in the decoder is as follows:

[0077] It should be understood that the first encoded feature F output by the encoder of the above text recognition model can be regarded as a sequence of feature vectors with a sequence length of h*w. The j-th feature vector in the feature vector sequence is denoted as... The first attention weight at time step t is then calculated using the following formula. and first context features

[0078]

[0079]

[0080]

[0081] The above Here are the trainable parameters in the text recognition model; N = h * w; Intermediate calculation results have no practical meaning; This refers to the first attention weight corresponding to the j-th feature vector in the feature vector sequence at time step t. The first attention weights corresponding to all feature vectors in the feature vector sequence at time step t constitute the first attention weight at time step t.

[0082] Finally, as Figure 4 The steps shown in the white diamond can be cascaded. and The data is then input into the network classification layer, ultimately yielding the first score vector s at time step t. t (by s) t available ).

[0083] Compared to methods that do not rely on a model to obtain the first contextual features and the first score vector of the text image to be recognized, the embodiments of this application improve the accuracy of obtaining the first contextual features and the first score vector of the text image to be recognized by constructing and training a text recognition model based on a large amount of data, thereby improving the recognition accuracy of the text image to be recognized.

[0084] In some embodiments of this application, the process of "step S102, obtaining the second score vector in the process of combining the first context features to perform text recognition of the standard answer" is described.

[0085] Similar to the foregoing embodiments, the process of text recognition of the standard answer combined with the first context feature in the present embodiment can be implemented by a pre-trained answer boosting model, which is trained by taking the training answer with the recognition result label of the labeled training answer and the corresponding first context feature as the training data, so that the decoder in the answer boosting model has the function of obtaining the second score vector. Here, the corresponding first context feature refers to the first context feature of the to-be-recognized image obtained in the process of text recognition of the to-be-recognized image corresponding to the standard answer.

[0086] The following two paragraphs explain the "recognition result label of the labeled training answer".

[0087] When the to-be-recognized text in the to-be-recognized text image is consistent with the standard answer, the answer boosting model should output the same content as the to-be-recognized text, i.e. the output of the text recognition model needs to be boosted by the standard answer. For this purpose, the recognition result label labeled in the training data should be the same as the to-be-recognized text, so as to facilitate subsequent score fusion; when the to-be-recognized text is inconsistent with the standard answer, the output of the text recognition model does not need to be boosted by the standard answer (since the standard answer is different from the to-be-recognized text, the standard answer is useless for recognition at this time and should be discarded, and the present embodiment needs to make the answer boosting model learn the ability of automatic discarding by the labeled recognition result label and the corresponding first context feature); for this purpose, the recognition result label labeled in the training data should be a sequence representing no boosting, which is represented by "\nobias\nobias…eos" in the present case, and its length is equal to that of the output sequence of the text recognition model. Here, the meaning of "\nobias" is that if the answer boosting model predicts this symbol at the current time step, the output of the answer boosting model should not be referred to.

[0088] For example, the to-be-recognized text in the to-be-recognized text image is "12", and the standard answer is "12;25". Since the shape of the to-be-recognized text is the same as the answer text "12", it is judged that answer boosting is needed in the text recognition process of the answer boosting model at this time, and then the labeled recognition result label is the same as the to-be-recognized text, which is "12eos", where "eos" is an end label; if the standard answer is "34;56", since the shape of the to-be-recognized text is different from the standard answer, it is judged that answer boosting is not needed in the text recognition process of the answer boosting model at this time, and then the labeled recognition result label is "\nobias\nobias\eos".

[0089] After the answer incentive model is trained well, the embodiment can extract the features of the standard answer through the encoder in the pre-trained answer incentive model, the extracted features being taken as the second encoding features, and then perform text recognition according to the second encoding features and the first context features through the decoder in the answer incentive model, to obtain a second score vector in the process of text recognition according to the second encoding features and the first context features.

[0090] Optionally, the embodiment can process the standard answer into a standard answer sequence, and then input the standard answer sequence into the encoder in the answer incentive model, so that the encoder extracts the features of the standard answer to obtain the second encoding features.

[0091] Preferably, considering that the features between images will be smaller and more convenient for feature interaction, under the premise that the text recognition model inputs the to-be-recognized text image, the embodiment can further process the standard answer into a standard answer image, and then input the standard answer image into the encoder in the answer incentive model, so that the encoder extracts the features of the standard answer to obtain the second encoding features.

[0092] It should be noted that the application does not limit the manner of processing the standard answer into a standard answer image, for example, in one possible implementation, it can be realized through a public Python library, such as pygame, or through some public software, such as Photoshop.

[0093] Optionally, the encoder of the above-mentioned answer incentive model can be a lightweight fully convolutional network, and the decoder can be a recurrent neural network. Of course, the structure of the encoder and the decoder of the answer incentive model can also be other, which is not limited in the application.

[0094] In an optional embodiment, the architecture of the answer incentive model provided by the application can be basically similar to the text recognition model, which is still an autoregressive decoding model based on attention mechanism in general, except that the interaction with the features of the text recognition model is added, and the network depth and width design will be more lightweight than the recognition network, so that the addition of the incentive network will not increase too much efficiency burden.

[0095] Referring to Figure 5 Fig. 1 shows a structure and interaction process diagram of the text recognition model and the answer incentive model provided by the embodiment of the application. In the figure, the encoder of the text recognition model is referred to as a recognition encoder, the decoder of the text recognition model is referred to as a recognition decoder, the encoder of the answer incentive model is referred to as an answer encoder, and the decoder of the answer incentive model is referred to as an answer decoder.

[0096] As Figure 5As shown, in order to enable the answer encoder to refer more to the features of the text to be recognized when extracting the features of the standard answer, preferably, the first encoding features output by the recognition encoder can be additionally integrated into the answer encoder, i.e., the second encoding features extracted by the answer encoder are used as queries, and the first encoding features output by the recognition encoder are used for mutual attention weighting, so as to enhance the encoding features of the answer. Then, after the second encoding features are extracted by the answer encoder, the mutual attention weighting mechanism module in the answer excitation model can be used to perform mutual attention weighting processing on the second encoding features and the first encoding features, and the obtained features are used as the second encoding features to be input to the answer decoder (i.e., the "enhanced encoding features of the answer").

[0097] Next, the answer decoder can perform text recognition according to the input second encoding features and the first context features. As introduced previously, the second score vector can be obtained in the process of text recognition according to the second encoding features and the first context features.

[0098] In this embodiment, the decoding process of the answer decoder is basically the same as that of the recognition decoder, except that in each time step, the first context features of each time step are also considered when calculating the second attention weight of each time step. Therefore, the process of obtaining the second score vector in the process of text recognition according to the second encoding features and the first context features can be as shown below.

[0099] Firstly, the recognition process of each to-be-recognized character contained in the text to be recognized is regarded as a time step. For each time step, the second hidden state of the time step is obtained according to the second recognition result of the previous time step of the time step and the second hidden state of the previous time step of the time step.

[0100] Wherein, if the time step is the first time step, the second recognition result of the previous time step of the time step is a preset start character (i.e., SOS), and the second hidden state of the previous time step of the time step is a second preset value, and if the time step is not the first time step, the second recognition result of the previous time step of the time step is obtained according to the second score vector of the previous time step. Wherein, the superscript b indicates that it is a parameter of the answer excitation model, and the superscripts of various symbols in the following are the same.

[0101] Then, the second attention weight of the time step is determined according to the second hidden state of the time step, the second encoding features and the first context features of the time step, the second context information of the time step is obtained according to the second attention weight of the time step and the second encoding features, and the second score vector of the time step is obtained according to the second context information of the time step and the second hidden state of the time step, thereby obtaining the second score vector of each time step.

[0102] The overall decoding process can refer to the foregoing description, except that the following formulas (4) to (6) are used to replace the foregoing formulas (1) to (3) in the detailed process of attention weighting of the decoder.

[0103]

[0104]

[0105]

[0106] The superscript b in the formula symbol represents a related parameter of the answer incentive model.

[0107] The decoding process of the answer decoder in this part is not described in detail and can refer to the decoding process of the recognition decoder, which is not described herein.

[0108] The answer incentive model provided in this embodiment can automatically discard the standard answer by referring to the corresponding first context feature when the standard answer is different from the to-be-recognized text, and can output the same or similar prediction output as the text recognition model by referring to the corresponding first context feature when the standard answer is the same as the to-be-recognized text, so as to perform answer incentive on the recognition result output by the text recognition model (when the recognition result output by the text recognition model is different, error correction can be implemented), thereby improving the recognition accuracy.

[0109] In some embodiments of the present application, the process of "step S103, determining whether to perform answer incentive on the first score vector according to the second score vector" is introduced.

[0110] In the embodiments of the present application, the first score vector and the second score vector are both score vectors obtained based on a preset dictionary, and compared with the preset dictionary corresponding to the first score vector, the preset dictionary corresponding to the second score vector adds a character "\nobias" on the basis of the preset dictionary corresponding to the first score vector, to represent the case that the target answer content is not contained in the standard answer, and correspondingly, the second score vector has one more dimension than the first score vector, representing the score value under the character "\nobias".

[0111] For ease of explanation, the additional dimension in the second score vector compared to the first score vector can be defined as the target dimension. In this embodiment, the score value of the target dimension in the second score vector can be compared with the score values ​​of the other dimensions. If the score value of the target dimension is the largest, it means that the standard answer does not contain the target answer content, and therefore the first score vector should not be incentivized. Conversely, if the score value of the target dimension is not the largest, it means that the standard answer contains the target answer content, and therefore the first score vector should be incentivized.

[0112] Based on this, the process of "determining whether to incentivize the first score vector based on the second score vector" includes: determining whether the score value of the target dimension in the second score vector is the maximum score value; if not, determining to incentivize the first score vector; if so, determining not to incentivize the first score vector.

[0113] In this embodiment, if it is determined that no answer incentive will be applied to the first score vector, then the answer incentive recognition result of the text image to be recognized can be determined solely based on the first score vector; if it is determined that the first score vector will be applied to the first score vector, then the scores of the first score vector and the second score vector will be fused to obtain the answer incentive recognition result of the text image to be recognized.

[0114] There are multiple ways to obtain the answer incentive recognition result based on the first score vector and / or the second score vector in the embodiments of this application. In one implementation, when it is determined that no answer incentive is applied to the first score vector, the character corresponding to the maximum score value in the first score vector can be directly used as the answer incentive recognition result of the text image to be recognized. When it is determined that the first score vector is to be applied to the first score vector, the other dimensions of the first score vector and the second score vector, except for the target dimension, are directly weighted and fused, and the character corresponding to the maximum score value after weighted fusion is used as the answer incentive recognition result of the text image to be recognized.

[0115] Considering that the direct weighted fusion method may cause overflow during calculation, in order to make the calculation process smoother and prevent overflow, it is preferable to take the logarithm first and then fuse. That is, at each time step, the calculation can be performed according to the following formulas (7) and (8) to obtain the answer incentive recognition score o.

[0116]

[0117] o t =exp(z t ) Formula (8)

[0118] Among them, s t p represents the first score vector at time step t. tThe second score vector represents the time step t, w represents the weight when the first score vector and the second score vector of the time step t are fused, i \nobias The score value of the "\nobias" symbol in the second score vector p t The index (i.e. the target dimension, which can be the last index of p, optionally) of the score value of the "\nobias" symbol in the second score vector p t The fused score vector after logarithm at the time step t, o t The answer boost recognition score at the time step t.

[0119] Formula (7) indicates that when the maximum score character predicted by the answer boost model is the special symbol "\nobias", the first score vector of the text recognition model does not need to fuse the second score vector of the answer boost model; when the maximum score character predicted by the answer boost model is the character that really needs to be recognized, the first score vector s and the second score vector p need to be fused after logarithm with a certain weight for answer boost.

[0120] In formula (8), the character represented by the maximum score of o t is the answer boost recognition result at the time step t.

[0121] In summary, the embodiment can simulate the thinking process of a person in the recognition process by means of the information of the standard answer, so that the answer boost recognition result obtained finally is more accurate and closer to the meaning that the user actually wants to express, thereby improving the recognition effect and user experience.

[0122] The embodiment of the application also provides a text recognition device, and the text recognition device provided by the embodiment of the application is described below. The text recognition device described below can be correspondingly referred to the text recognition method described above.

[0123] Please refer to Figure 6 , which shows the structure schematic diagram of the text recognition device provided by the embodiment of the application. As shown in Figure 6 , the text recognition device can include a first recognition unit 601, a second recognition unit 602, a boost judgment unit 603, and an answer boost unit 604.

[0124] The first recognition unit 601 is configured to obtain a first context feature and a first score vector of a to-be-recognized text image in a process of text recognition on the to-be-recognized text image, wherein the to-be-recognized text image contains to-be-recognized text, and the to-be-recognized text corresponds to a standard answer.

[0125] The second identification unit 602 is configured to obtain a second score vector in a process of performing text recognition on the standard answer in combination with a first context feature, wherein the first context feature is used to enhance attention to the content of the target answer in the process of performing text recognition on the standard answer, so as to obtain the second score vector capable of reflecting whether the standard answer contains the content of the target answer, and the content of the target answer refers to the answer content in the standard answer which is the same as or similar to the shape of the text to be recognized.

[0126] The excitation judgment unit 603 is configured to determine whether to perform answer excitation on the first score vector according to the second score vector.

[0127] The answer excitation unit 604 is configured to perform score fusion on the first score vector and the second score vector to obtain an answer excitation recognition result of the text image to be recognized, if it is determined to perform answer excitation on the first score vector.

[0128] Optionally, in the process of obtaining the first context feature and the first score vector of the text image to be recognized by the first identification unit, the process can include:

[0129] extracting a feature of the text image to be recognized by using an encoder in the pre-trained text recognition model, and the extracted feature is used as a first encoded feature;

[0130] performing text recognition based on the first encoded feature by using a decoder in the text recognition model, and obtaining the first context feature and the first score vector of the text image to be recognized in the process of performing text recognition based on the first encoded feature;

[0131] The text recognition model is trained by using training text images with recognition result labels of labeled training text images as training data, so that the decoder in the text recognition model has the function of obtaining the first context feature and the first score vector of the text image to be recognized.

[0132] Optionally, in the process of obtaining the first context feature and the first score vector of the text image to be recognized based on the first encoded feature by the first identification unit, the process can include:

[0133] taking the recognition process of each to-be-recognized character contained in the text to be recognized as a time step, and for each time step, obtaining a first hidden state of the time step according to a first hidden state of a previous time step of the time step and a first recognition result of the previous time step, wherein if the time step is the first time step, the first recognition result of the previous time step of the time step is a preset starting character, and the first hidden state of the previous time step of the time step is a first preset value, and if the time step is not the first time step, the first recognition result of the previous time step of the time step is obtained according to the first score vector of the previous time step.

[0134] determine the first attention weight of the time step according to the first hidden state and the first encoded feature of the time step;

[0135] obtain the first context information of the time step according to the first attention weight and the first encoded feature of the time step;

[0136] obtain the first score vector of the time step according to the first context information of the time step and the first hidden state of the time step;

[0137] obtain the first context feature of each time step and the first score vector of each time step as the first context feature and the first score vector of the text image to be recognized.

[0138] Optionally, in the process of text recognition of the standard answer combined with the first context feature, the process of obtaining the second score vector by the second recognition unit can include:

[0139] extract the feature of the standard answer by the encoder in the pre-trained answer excitation model, and the extracted feature is used as the second encoded feature;

[0140] perform text recognition according to the second encoded feature and the first context feature by the decoder in the answer excitation model, and obtain the second score vector in the process of text recognition according to the second encoded feature and the first context feature;

[0141] wherein the answer excitation model is trained by taking the training answer with the recognition result label of the labeled training answer and the corresponding first context feature as training data, so that the decoder in the answer excitation model has the function of obtaining the second score vector.

[0142] Optionally, in the process of text recognition of the standard answer combined with the first context feature, the process of obtaining the second score vector by the second recognition unit can further include:

[0143] after extracting the feature of the standard answer by the encoder in the pre-trained answer excitation model, the second encoded feature and the first encoded feature are subjected to mutual attention weighting processing by the mutual attention mechanism module in the answer excitation model, and the obtained feature is used as the second encoded feature.

[0144] Optionally, in the process of text recognition of the standard answer combined with the first context feature, the process of obtaining the second score vector by the second recognition unit can include:

[0145] The recognition process of each to-be-recognized character included in the to-be-recognized text is taken as a time step, for each time step, a second hidden state of the time step is obtained according to a second hidden state of a previous time step of the time step and a second recognition result of the previous time step, wherein if the time step is a first time step, the second recognition result of the previous time step of the time step is a preset starting character, and the second hidden state of the previous time step of the time step is a second preset value, and if the time step is not the first time step, the second recognition result of the previous time step of the time step is obtained according to a second score vector of the previous time step;

[0146] A second attention weight of the time step is determined according to the second hidden state of the time step, the second encoded feature and the first context feature of the time step;

[0147] Second context information of the time step is obtained according to the second attention weight of the time step and the second encoded feature;

[0148] A second score vector of the time step is obtained according to the second context information of the time step and the second hidden state of the time step;

[0149] The second score vector of each time step is obtained.

[0150] Optionally, the process of determining, by the encouragement judging unit, whether to encourage the first score vector according to the second score vector can include:

[0151] It is determined whether a score value of a target dimension in the second score vector is a maximum score value, wherein the target dimension represents content of the target answer that is not included in the standard answer;

[0152] If not, it is determined to encourage the first score vector;

[0153] If yes, it is determined not to encourage the first score vector.

[0154] Optionally, the answer encouragement unit can be further configured to: if it is determined not to encourage the first score vector, determine an answer encouragement recognition result of the to-be-recognized text image according to the first score vector.

[0155] The text recognition apparatus provided by the embodiments of the present application can be applied to an electronic device, such as a text recognition device, which can be a scanner or a terminal, such as a mobile phone, a computer or the like. Optionally, Figure 7 A hardware structure block diagram of an electronic device is shown, referring to Figure 7 The hardware structure of the electronic device can include at least one processor 701, at least one communication interface 702, at least one memory 703 and at least one communication bus 704;

[0156] In the embodiments of the present application, the number of the processor 701, the communication interface 702, the memory 703 and the communication bus 704 is at least one, and the processor 701, the communication interface 702 and the memory 703 complete the communication with each other through the communication bus 704.

[0157] The processor 701 can be a central processing unit CPU, or an application specific integrated circuit ASIC, or one or more integrated circuits configured to implement the embodiments of the present application, etc.

[0158] The memory 703 can include a high-speed RAM memory, and can also include a non-volatile memory, such as at least one disk memory.

[0159] The memory 703 stores a program, and the processor 701 can invoke the program stored in the memory 703, and the program is used to implement the steps of the foregoing text recognition method.

[0160] Optionally, the detailed functions and extended functions of the program can refer to the foregoing description.

[0161] The embodiments of the present application also provide a storage medium which can store a program suitable for a processor to execute, and the program is used to implement the steps of the foregoing text recognition method.

[0162] Optionally, the detailed functions and extended functions of the program can refer to the foregoing description.

[0163] Finally, it should be noted that, in this document, the relationship terms such as and the second and the like are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or equipment. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or equipment including the element.

[0164] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the difference from other embodiments, and the same or similar parts between various embodiments can be referred to each other.

[0165] The above description of disclosed embodiments enables one of ordinary skill in the art to make and use the application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other embodiments without departing from the spirit or scope of the application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A text recognition method, characterized in that, include: In the process of text recognition of the text image to be recognized, the first context features and the first score vector of the text image to be recognized are obtained, wherein the text image to be recognized contains the text to be recognized and the text to be recognized corresponds to the standard answer; In the process of text recognition of the standard answer by combining the first context features, a second score vector is obtained. The first context features are used to enhance the attention to the target answer content in the process of text recognition of the standard answer, so as to obtain a second score vector that can reflect whether the standard answer contains the target answer content. The target answer content refers to the answer content in the standard answer that has the same or similar shape to the text to be recognized. Determine whether to apply answer incentives to the first score vector based on the second score vector; If so, the scores of the first score vector and the second score vector are fused to obtain the answer-incentive recognition result of the text image to be recognized.

2. The text recognition method according to claim 1, characterized in that, The process of obtaining the first contextual features and the first score vector of the text image to be recognized during text recognition includes: The features of the text image to be recognized are extracted by the encoder in the pre-trained text recognition model, and the extracted features are used as the first encoded features. The text recognition model uses a decoder to perform text recognition based on the first encoded feature, and in the process of text recognition based on the first encoded feature, the first context feature and the first score vector of the text image to be recognized are obtained. The text recognition model is trained using training text images labeled with recognition result tags of the training text images as training data, enabling the decoder in the text recognition model to obtain the first context features and the first score vector of the text image to be recognized.

3. The text recognition method according to claim 2, characterized in that, The process of obtaining the first contextual features and the first score vector of the text image to be recognized during text recognition based on the first encoded features includes: The recognition process of each character in the text to be recognized is taken as a time step. For each time step, the first hidden state of the time step is obtained according to the first hidden state of the previous time step and the first recognition result of the previous time step. If the time step is the first time step, the first recognition result of the previous time step is the preset start character and the first hidden state of the previous time step is the first preset value. If the time step is not the first time step, the first recognition result of the previous time step is obtained according to the first score vector of the previous time step. Based on the first hidden state and the first encoded feature at that time step, determine the first attention weight at that time step; Based on the first attention weight and the first encoding feature of the time step, the first context information of the time step is obtained; Based on the first context information and the first hidden state of the time step, the first score vector of the time step is obtained; The first context features and the first score vector at each time step are obtained and used as the first context features and the first score vector of the text image to be identified.

4. The text recognition method according to claim 3, characterized in that, The process of obtaining a second score vector by combining the first contextual features to perform text recognition on the standard answer includes: The standard answer is extracted by the encoder in the pre-trained answer-incentive model, and the extracted features are used as the second encoding features. The decoder in the answer incentive model performs text recognition based on the second encoding feature and the first context feature, and obtains the second score vector during the text recognition process based on the second encoding feature and the first context feature; The answer incentive model is trained using the training answer labeled with the recognition result tag and the corresponding first context feature as training data, so that the decoder in the answer incentive model has the function of obtaining the second score vector.

5. The text recognition method according to claim 4, characterized in that, After the encoder in the pre-trained answer-incentive model extracts features of the standard answer, and these extracted features are used as second encoded features, the process further includes: The second encoded feature and the first encoded feature are subjected to mutual attention weighting processing through the mutual attention mechanism module in the answer incentive model, and the resulting feature is used as the second encoded feature.

6. The text recognition method according to claim 4 or 5, characterized in that, The process of obtaining the second score vector during text recognition based on the second encoded features and the first context features includes: The recognition process of each character in the text to be recognized is taken as a time step. For each time step, the second hidden state of the time step is obtained according to the second hidden state of the previous time step and the second recognition result of the previous time step. If the time step is the first time step, the second recognition result of the previous time step is the preset start character and the second hidden state of the previous time step is the second preset value. If the time step is not the first time step, the second recognition result of the previous time step is obtained according to the second score vector of the previous time step. The second attention weight for the time step is determined based on the second hidden state, the second encoding feature, and the first context feature for the time step. Based on the second attention weight and the second encoding feature at that time step, the second context information for that time step is obtained; Based on the second context information and the second hidden state of the time step, the second score vector of the time step is obtained; This yields the second score vector for each time step.

7. The text recognition method according to claim 1, characterized in that, The step of determining whether to apply answer incentives to the first score vector based on the second score vector includes: Determine whether the score value of the target dimension in the second score vector is the maximum score value, wherein the target dimension represents that the standard answer does not contain the content of the target answer; If not, then determine to apply answer incentives to the first score vector; If so, then it is determined that the first score vector will not be used for answer incentives.

8. The text recognition method according to claim 1, characterized in that, Also includes: If it is determined that no answer incentive will be applied to the first score vector, then the answer incentive recognition result of the text image to be recognized is determined based on the first score vector.

9. A text recognition device, characterized in that, include: The first recognition unit is used to obtain the first context features and the first score vector of the text image to be recognized during the text recognition process, wherein the text image to be recognized contains text to be recognized and the text to be recognized corresponds to a standard answer; The second recognition unit is used to obtain a second score vector during the text recognition of the standard answer in combination with the first context features. The first context features are used to enhance the attention to the target answer content during the text recognition of the standard answer, so as to obtain a second score vector that can reflect whether the standard answer contains the target answer content. The target answer content refers to the answer content in the standard answer that has the same or similar shape as the text to be recognized. An incentive determination unit is used to determine whether to apply an answer incentive to the first score vector based on the second score vector. The answer incentive unit is used to perform score fusion on the first score vector and the second score vector if it is determined that the first score vector should be incentived, so as to obtain the answer incentive recognition result of the text image to be recognized.

10. An electronic device, characterized in that, Including memory and processor; The memory is used to store programs; The processor is configured to execute the program to implement the various steps of the text recognition method as described in any one of claims 1 to 8.

11. A readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements each step of the text recognition method as described in any one of claims 1 to 8.

Images