Text recognition method and device, electronic equipment, and computer readable medium

Abstract

Embodiments of the present application disclose a character recognition method and device, electronic equipment and computer readable medium. The method comprises: inputting a to-be-processed image into a trained character recognition model; performing character recognition on image content contained in the to-be-processed image by the character recognition model to obtain an initial character recognition result, and performing scene classification to obtain a scene classification result, and correcting the initial character recognition result according to the scene classification result to obtain target character information corresponding to the to-be-processed image; and outputting the target character information as a target initial character recognition result. The embodiments of the present application can improve the calculation efficiency of the character recognition model, save the cost of the model, and improve the recognition accuracy of the character recognition model.

Description

Technical Field

[0001] This application relates to the field of character recognition technology, specifically to a character recognition method and apparatus, electronic device, and computer-readable medium. Background Technology

[0002] Optical Character Recognition (OCR) is an important branch of computer vision, widely used in traffic sign recognition, electronic document recognition, and ID card recognition. In practical applications, due to the large volume and variety of business types, it is generally required to recognize text in multiple different scenarios. Different OCR models are typically designed for different scenarios, and then combined online to meet the text recognition needs of various scenarios. However, as the number of compatible scenarios increases, more OCR models need to be combined online. This leads to an increase in the recognition cost of the OCR models and the potential for redundant processing modules within multiple OCR models, which can negatively impact the efficiency of text recognition. Summary of the Invention

[0003] To address the aforementioned technical problems, embodiments of this application provide a character recognition method and apparatus, an electronic device, and a computer-readable medium.

[0004] According to one aspect of the embodiments of this application, a text recognition method is provided, comprising: inputting an image to be processed into a trained text recognition model; performing text recognition on the image content contained in the image to be processed by the text recognition model to obtain an initial text recognition result, and performing scene classification to obtain a scene classification result, and performing correction processing on the initial text recognition result according to the scene classification result to obtain target text information corresponding to the image to be processed; and outputting the target text information as the target initial text recognition result.

[0005] In one embodiment of this application, the step of correcting the initial text recognition result based on the scene classification result to obtain the target text information corresponding to the image to be processed includes: determining a mask vector that matches the scene classification result based on the scene classification result; and correcting the initial text recognition result based on the mask vector to obtain the target text information corresponding to the image to be processed.

[0006] In one embodiment of this application, the initial text recognition result includes character information, and the scene classification result includes probability values ​​of the character information belonging to each scene; determining the mask vector matching the scene classification result according to the scene classification result includes: setting the mask vector corresponding to the scene with the probability value greater than a preset threshold as a first value, and setting the mask vector corresponding to the scene with the probability value greater than the preset threshold as a second value; wherein, the first value and the second value are normalized values.

[0007] In one embodiment of this application, the initial text recognition result includes a character probability vector; the step of correcting the initial text recognition result according to the mask vector to obtain the target text information corresponding to the image to be processed includes: performing a product operation on the mask vector and the character probability vector to obtain a calculation result; wherein, the mask vector includes a first value and a second value after normalization processing; and using the calculation result as the target text information corresponding to the image to be processed.

[0008] In one embodiment of this application, the initial text recognition result includes character information; the step of performing scene classification to obtain scene classification results includes: obtaining multiple candidate scenes; wherein, the multiple scenes include multiple scenes divided by language type or multiple scenes divided by business type; calculating the probability value of the character information belonging to each scene to obtain the probability value of the character information belonging to each scene respectively, and using the probability value of the character information belonging to each scene respectively as the scene classification result.

[0009] In one embodiment of this application, before obtaining an initial text recognition result by performing text recognition on the image content contained in the image to be processed using the text recognition model, the method further includes: extracting features from the image to be processed using the text recognition model to obtain feature information, and performing feature enhancement processing on the feature information to obtain enhanced feature information; the step of obtaining an initial text recognition result by performing text recognition on the image content contained in the image to be processed using the text recognition model includes: performing text recognition on the enhanced feature information using the text recognition model to obtain the initial text recognition result.

[0010] In one embodiment of this application, before inputting the image to be processed into the trained text recognition model, the method further includes: acquiring an annotated image and using the annotated image as a training sample; wherein the training sample includes expected value data; inputting the annotated image into an initial text recognition model to obtain an initial text recognition result and a scene classification result of the annotated image, and correcting the initial text recognition result according to the scene classification result to obtain the target text information corresponding to the annotated image; adjusting the parameters of the initial text recognition model through the loss value between the expected value data and the target text information of the annotated image to obtain the trained text recognition model.

[0011] According to one aspect of the embodiments of this application, a text recognition device is provided, comprising: an input module configured to input an image to be processed into a trained text recognition model; a recognition module configured to perform text recognition on the image content contained in the image to be processed through the text recognition model to obtain an initial text recognition result, and to perform scene classification to obtain a scene classification result, and to perform correction processing on the initial text recognition result according to the scene classification result to obtain target text information corresponding to the image to be processed; and an output module configured to output the target text information as the target initial text recognition result.

[0012] According to one aspect of the embodiments of this application, an electronic device is provided, including: one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the electronic device to implement the character recognition method as described above.

[0013] According to one aspect of the embodiments of this application, a computer-readable medium is provided that stores computer-readable instructions thereon, which, when executed by a computer's processor, cause the computer to perform the character recognition method as described above.

[0014] In the technical solution provided in the embodiments of this application, a text recognition model can simultaneously perform text recognition and scene classification on the image content contained in the image to be processed, obtaining initial text recognition results and scene classification results in the image to be processed. The initial text recognition results are then corrected based on the scene classification results to obtain the target text information corresponding to the image to be processed. The text recognition model in the embodiments of this application shares a backbone network used for feature extraction, which can improve the computational efficiency of the text recognition model, save model resource costs, and improve the recognition accuracy of the text recognition model by correcting the initial text recognition results through scene classification results.

[0015] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings:

[0017] Figure 1 This is a schematic diagram illustrating the framework of a character recognition model according to an exemplary embodiment of this application;

[0018] Figure 2 This is a schematic flowchart illustrating a character recognition method according to an exemplary embodiment of this application;

[0019] Figure 3 yes Figure 2 A flowchart of step S220 in an exemplary embodiment shown in the illustrated example;

[0020] Figure 4 yes Figure 3 A flowchart of step S320 in another exemplary embodiment of the illustrated embodiment;

[0021] Figure 5 yes Figure 2 A flowchart of step S220 in another exemplary embodiment of the illustrated embodiment;

[0022] Figure 6 This is a flowchart illustrating a character recognition method according to another exemplary embodiment of this application;

[0023] Figure 7 This is a block diagram illustrating a character recognition device in an exemplary embodiment of this application;

[0024] Figure 8 This is a schematic diagram of the structure of a computer system used to implement the electronic device of the present application. Detailed Implementation

[0025] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0026] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0027] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0028] In this application, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0029] The solutions provided in this application relate to text recognition technology in image processing. It is understood that text recognition technology based on Optical Character Recognition (OCR) is an important branch of computer vision and can be widely applied in areas such as traffic sign recognition, electronic document recognition, and ID card recognition. OCR text recognition technology generally refers to the collective technical process from image input to text output, with text detection and text recognition being the two core stages. The text detection stage primarily filters out regions containing text in the image, and then the text recognition stage identifies the text within the detected regions.

[0030] In practical applications, different scenes may exist in the same image to be processed, such as scenes with different languages ​​or different types of documents. In order to improve the accuracy of text recognition in different scenes, it is necessary to distinguish the text contained in multiple scenes in the same image to be processed, such as realizing Chinese recognition, English recognition, Japanese recognition, and ID card recognition.

[0031] Character recognition methods in related technologies typically extract visual features from the image to be processed, obtain the semantic relationships of the text context through the extracted feature sequences, and then obtain the character prediction results. The character recognition models used in related technologies usually include three modules: a backbone network module, used to extract visual features from the image to be processed; a sequence feature enhancement module, used to enhance the visual features of the image to be processed; and a character prediction head module, which is a character classifier used to output the character recognition results of the character recognition model. Character recognition methods usually combine different recognition scenarios to design recognition dictionaries and perform recognition within a limited character set. In production environments, business needs generally require the recognition of text in multiple different scenarios. Under existing technical solutions, in order to improve the accuracy of character recognition in each independent scenario, multiple different models are usually designed and implemented for different recognition scenarios, and finally, a multi-model online combination method is used to meet the character recognition needs of multiple different scenarios in business. This mode greatly increases the application cost of character recognition technology. As the number of compatible scenarios increases, text recognition methods in related technologies require the online combination of more text recognition models, which increases the recognition cost of text recognition models. Furthermore, multiple text recognition models may contain duplicate functional processing modules, which has a certain impact on the efficiency of text recognition.

[0032] Therefore, this application proposes a character recognition model. Please refer to [link / reference]. Figure 1 , Figure 1 This is a schematic diagram illustrating the framework of a character recognition model as shown in an exemplary embodiment of this application, such as... Figure 1 As shown, this text recognition model includes a scene classifier, a character classifier, and a masker. Figure 1 It can be seen that the text recognition model has logical calculation functions, including:

[0033] The image to be processed is input into the trained text recognition model;

[0034] The character classifier in the text recognition model performs text recognition on the image content contained in the image to be processed to obtain the initial text recognition result. The scene classifier in the text recognition model performs scene classification on the image to be processed to obtain the scene classification result. The mask in the text recognition model corrects the initial text recognition result based on the scene classification result to obtain the target text information corresponding to the image to be processed.

[0035] The target text information is output as the initial target text recognition result.

[0036] It should be noted that the text recognition model in this application embodiment may also include a feature extraction module. For example, the feature extraction module may be a deep learning network, such as ResNet, DenseNet, VGGNet, etc., used to extract features of the image to be processed.

[0037] For example, the input to the feature extraction module is the image to be processed, and the output of the feature extraction module is a dimensionless image. The feature matrix X is given by the formula X, where T is the longest character count that the character recognition model can recognize, and K is a custom dimension, typically chosen as an empirical value of 512. It should be noted that the feature extraction module does not need to be trained separately; it can be integrated with the character classifier and scene classifier for end-to-end training.

[0038] The character recognition model in this application embodiment may further include a feature enhancement module, which is used to perform feature enhancement processing on the feature matrix X extracted by the feature extraction module.

[0039] For example, the feature enhancement module is an end-to-end learnable module, typically using a recurrent neural network (RNN) or a simple linear mapping module. The input to the feature enhancement module is the dimension extracted by the feature extraction module. The feature matrix, the output of the feature enhancement module is a feature matrix of dimension . The feature matrix is ​​such that the first dimension T remains unchanged, and the second dimension L can be set according to experience. Generally, the second dimension L is set to L=K, but it is not limited to this.

[0040] It should be noted that the features extracted by the feature extraction module in this embodiment can be directly input into the character classifier and the scene classifier, or they can be enhanced by the feature enhancement module and then input into the character classifier and the scene classifier. The character classifier judges the features and outputs the initial text recognition result, and the scene classifier judges the features and outputs the scene classification result.

[0041] The scene classifier in this embodiment is used to classify the scene of the input image to be processed, and obtain the scene classification result. The scene classifier in this embodiment can be a linear classifier, for example, it can be an end-to-end learning softmax classifier, and its number of classes can be set by the user according to the application scenario.

[0042] For example, the input to the scene classifier is a Given a feature matrix X, the scene classifier performs average pooling, linear transformation, and softmax activation on the feature matrix X in sequence, outputting a probability vector with n categories. The specific process includes the following steps:

[0043] In this embodiment of the application, the average pooling process is a The dimensionality reduction process, for a given 3D matrix X, The eigenma matrix X of dimension X is obtained after average pooling. The calculation process for a dimensional vector Y is shown in the following formula:

[0044]

[0045] in, The maximum number of characters that the character classifier can recognize. The second dimension of the feature matrix X The dimension of the scene classification results, i.e., the number of scene categories. Let be the i-th element in vector Y.

[0046] In this embodiment of the application, the linear transformation process is a The projection process, for a given A dimensional vector Y, consisting of a The learnable parameter matrix W and A bias vector B of dimension is obtained through linear computation. The calculation process for the dimension vector Z is shown in the following formula:

[0047]

[0048] in, The result obtained after linear transformation Elements in the dimensional vector Z, The second dimension of the feature matrix X Let be the element in the i-th row and j-th column of the parameter matrix W. Let i be the i-th element in the vector Y after average pooling. Let i be the i-th element in the bias vector B. This represents the number of scene categories.

[0049] In this embodiment of the application, the softmax activation process may include the following steps: [The process involves...] Normalize the probability of the vector Z to obtain the probability vector of the scene classification result. , The dimension is For a given Calculate a dimension vector Z. A probability vector S of dimension, The probability vector S of dimension 1 is the scene classification result, and its calculation process is shown in the following formula:

[0050]

[0051] in, This represents the probability vector of the scene classification results. Let i be the i-th element in the vector Y after average pooling. The dimension of the scene classification results, i.e., the number of scene categories.

[0052] Based on the probability vector of the scene classification results Computational Scene Categories :

[0053]

[0054] The scenario classification in this application embodiment can be based on language scenarios, such as Chinese, English, Thai, etc., or it can be based on business types, such as ID cards, invoices, real estate ownership certificates, etc. This application embodiment does not limit this.

[0055] The character classifier in this embodiment is used to perform text recognition on the image content contained in the image to be processed, and obtain an initial text recognition result. The character classifier in this embodiment can be a linear classifier that predicts the text in the input image to be processed. The number of categories in the character classifier can be set to the size of the character set supported by the model.

[0056] It should be noted that the character classifier and the scene classifier in this application embodiment have the same input. That is, the input of the character classifier can be the features extracted by the feature extraction module or the enhanced features by the feature enhancement module.

[0057] For example, the input to the character classifier is a The character classifier processes the feature vector X in dimensional form through a linear mapping and softmax activation, outputting a feature vector with dimension 1. The character prediction probability vector, where m is the number of character categories, includes the following process:

[0058] In this embodiment of the application, the linear mapping process may include the following process: by a The learnable parameter matrix W and Bias vector B of dimension A linear transformation is performed on the eigenma matrix X to obtain The calculation process for a dimensional matrix U is shown in the following formula:

[0059]

[0060] in, The elements in matrix U after linear mapping are... The element in the i-th row and k-th column of the characteristic matrix X. Let be the element in the k-th row and j-th column of the parameter matrix W. Let i be the i-th element in the bias vector B.

[0061] In this embodiment of the application, the softmax activation process may include the following process: [The text abruptly ends here, likely due to an incomplete sentence or a formatting error.] Normalized probability calculation is performed on the dimensional matrix U to obtain A probability matrix of dimension C, The probability matrix C is the initial character recognition result, and its calculation process is shown in the following formula:

[0062]

[0063] in, The characters in the initial text recognition result, The number of character categories in the initial character recognition results. The maximum number of characters that the character classifier can recognize.

[0064] The mask in this embodiment is an executor for scene segmentation. It generates different mask vectors based on different scene classification results and corrects the initial text recognition results based on the mask vectors to obtain the target text information corresponding to the image to be processed. The mask vectors generated by the mask can be used to control the segmentation of the initial text recognition results corresponding to different scenes.

[0065] It should be noted that the input to the mask in this embodiment is the scene category. The output is a A 0-1 matrix, i.e. A 0-1 matrix includes There are mask vectors corresponding to scene categories. Here, m is the number of categories in the scene classifier, and n is the number of categories in the character classifier. The input to the mask can be the scene classification result output by the scene classifier, or it can be manually input.

[0066] In this embodiment, the masker determines a mask vector that matches the scene classification result based on the input scene classification result. The calculation process is shown in the following formula:

[0067]

[0068] in, Let be the value of the mask element corresponding to scene category i in the mask vector, where i is the scene category, and , Let i be the set of characters in the scene corresponding to scene category i. , The set of characters recognized by the character classifier.

[0069] Based on the above character recognition model, embodiments of this application propose a character recognition method, a character recognition method and apparatus, an electronic device, and a computer-readable medium, which will be described in detail below.

[0070] Please see Figure 2 , Figure 2 This is a schematic flowchart illustrating a character recognition method according to an exemplary embodiment of this application. This method can be applied to [various applications]. Figure 1 The illustrated text recognition system is implemented in an environment specified by the text recognition model. It should be understood that this method can also be applied to other exemplary implementation environments and implemented by devices in other implementation environments. This embodiment does not limit the implementation environment to which this method is applicable.

[0071] In an exemplary embodiment, the character recognition method includes at least steps S210 to S230, which are described in detail below:

[0072] Step S210: Input the image to be processed into the trained text recognition model.

[0073] In this embodiment of the application, the image to be processed may include text information to be recognized. The text recognition model in this embodiment of the application is a pre-trained machine learning model that recognizes the text in the image to be processed and outputs the target text information.

[0074] Step S220: The text recognition model is used to perform text recognition on the image content contained in the image to be processed to obtain the initial text recognition result, and the scene classification is performed to obtain the scene classification result. The initial text recognition result is then corrected based on the scene classification result to obtain the target text information corresponding to the image to be processed.

[0075] The text recognition model in this application performs text recognition and scene classification on the image to be processed, and obtains initial text recognition results and scene classification results respectively. The initial text recognition results can be corrected based on the scene classification results. For example, the initial text recognition results of the image to be processed can be divided according to the scene classification results to obtain target text information.

[0076] In one exemplary embodiment, such as Figure 3 As shown, step S220, which involves correcting the initial text recognition result based on the scene classification result to obtain the target text information corresponding to the image to be processed, may include steps S310 to S320, which are detailed below:

[0077] Step S310: Determine the mask vector that matches the scene classification result based on the scene classification result.

[0078] It should be noted that the initial text recognition result in this application embodiment includes character information, and the scene classification result includes the probability value of each character information belonging to each scene.

[0079] In an exemplary embodiment, the process of determining the mask vector matching the scene classification result in step S310 may include the following process, which is described in detail below:

[0080] The mask vector corresponding to the scene with a probability value greater than a preset threshold is set as the first value, and the mask vector corresponding to the scene with a probability value greater than a preset threshold is set as the second value, so as to obtain a mask vector that matches the scene classification result; wherein, the first value and the second value are normalized values.

[0081] The preset threshold in this application embodiment is set in advance based on human experience or experimentation. The mask vector can be determined by comparing the probability value with the preset threshold.

[0082] For example, if the character information in the initial text recognition result belongs to the scene corresponding to the scene classification result, then the elements of the mask vector corresponding to the character information are set to 1; the elements of the mask vector corresponding to the character information that belongs to a scene other than the scene corresponding to the scene classification result are set to 0, so as to obtain a mask vector that matches the scene classification result; the number of elements of the mask vector is the same as the number of character information in the initial text recognition result.

[0083] Step S320: Correct the initial text recognition result according to the mask vector to obtain the target text information corresponding to the image to be processed.

[0084] In one exemplary embodiment, such as Figure 4 As shown, the initial character recognition result includes a character probability vector; step S320, which corrects the initial character recognition result based on the mask vector to obtain the target character information corresponding to the image to be processed, may include steps S410 to S420, which are described in detail below:

[0085] Step S410: Multiply the mask vector and the character probability vector to obtain the result; wherein the mask vector includes the first and second values ​​after normalization.

[0086] For example, suppose This is the probability vector output by the scene classifier, i.e., the scene classification result. It is the probability matrix output by the character classifier, that is, the initial text recognition result. The masker performs division control on the character classification result of the character classifier according to the mask vector and outputs the target text information , and the calculation expression is as follows:

[0087]

[0088] Among them, is the target text information, is the character in the initial text recognition result, is the first dimension of the target text information, which is the same as the maximum number of recognized characters of the character classifier, is the second dimension of the target text information, which is the same as the number of character categories in the initial text recognition result, is the dimension of the scene classification result.

[0089] It should be noted that in step S320, the process of correcting the initial text recognition result according to the mask vector to obtain the target text information corresponding to the image to be processed can be executed by the masker in the text recognition model.

[0090] It should be noted that the scene classification result in the embodiments of the present application can be obtained by outputting probabilities by the scene classifier or can be input by the user, and the embodiments of the present application do not limit this either.

[0091] Exemplarily, when the scene to which the image to be processed input into the text recognition model is unknown, the scene category i is obtained by the scene classifier outputting probabilities according to the above calculation method; when the scene of the image to be processed input into the text recognition model is determined, the scene category i is input by the user.

[0092] Step S420, taking the operation result as the target text information corresponding to the image to be processed.

[0093] For example, two custom scenes are defined as S = {s1, s2}, s1 is the Chinese scene, s1 = {生, 命, 重, 于, 泰, 山}, s2 is the English scene s2 = {a, c, h, i, n}, and the character classifier C = {生, 命, 重, 于, 泰, 山, a, c, h, i, n}; the maximum number of recognized characters of the character classifier is T = 8, the number of character categories is m = |C| = 11, the number of scene categories is n = |S| = 2, and the output feature dimension of the feature extraction module is K = 2, and the output feature dimension of the feature enhancement module is L = K = 2.

[0094] The feature extraction module extracts features from the image to be processed, and the obtained matrix corresponding to the feature information is:

[0095]

[0096] The feature enhancement module performs feature enhancement processing on the feature information, resulting in the following enhanced feature information:

[0097]

[0098] If the image to be processed by the input text recognition model is a text image, and the text information in the image is "Life is more important than Mount Tai", then the text recognition model will perform text recognition on the image content contained in the image to be processed, and the initial text recognition results are shown in the table below:

[0099]

[0100] The text recognition model performs scene classification on the image content contained in the image to be processed. The scene classification results are shown in the table below:

[0101]

[0102] The scene classification result obtained is i=1.

[0103] Based on the mask generation rules and the scene classification results, the mask vector matching the scene classification results is determined as follows:

[0104]

[0105] The initial text recognition results are corrected based on the mask vector, and the probability matrix corresponding to the target text information in the image to be processed is shown in the table below:

[0106]

[0107] Based on the summation of the probability matrix, the probability of the English character set s2={a,c,h,i,n} in scenario 2 is 0. The text recognition model has successfully masked scenario 2, and the output target text information is "Life is more important than Mount Tai".

[0108] In one exemplary embodiment, such as Figure 5 As shown, the process of obtaining scene classification results in step S220 may include steps S510 to S520, which are described in detail below:

[0109] Step S510: Obtain multiple candidate scenarios; wherein, the multiple scenarios include multiple scenarios divided by language type or multiple scenarios divided by business type.

[0110] Step S520: Calculate the probability values ​​of the character information belonging to each scene to obtain the probability values ​​of the character information belonging to each scene respectively, and use the probability values ​​of the character information belonging to each scene respectively as the scene classification result.

[0111] In an exemplary embodiment, the process of obtaining scene classification results in step S220 may further include the following steps: obtaining multiple candidate scene categories; calculating the probability results of the image to be processed belonging to each candidate scene category; determining the scene category of the image to be processed based on the probability results; and using the scene category of the image to be processed as the scene classification result.

[0112] Step S230: Output the target text information as the initial target text recognition result.

[0113] As can be seen from the above, the technical solution provided in this application embodiment can simultaneously perform text recognition and scene classification on the image content contained in the image to be processed through a text recognition model, obtaining the initial text recognition result and scene classification result in the image to be processed, and then correcting the initial text recognition result based on the scene classification result to obtain the target text information corresponding to the image to be processed. The text recognition model in this application embodiment shares the backbone network used for feature extraction, which can improve the computational efficiency of the text recognition model, and the correction of the initial text recognition result through the scene classification result can also improve the recognition accuracy of the text recognition model. The text recognition model in this application embodiment determines the mask vector corresponding to the scene classification result based on the scene classification result, and controls the character range of different scene recognition based on the mask vector. The mask can greatly eliminate the interference of non-scene information, realizing online flexible adjustment of the recognition scene.

[0114] Based on the above method embodiments, before obtaining the initial text recognition result by performing text recognition on the image content contained in the image to be processed through the text recognition model, the text recognition method of this application embodiment may further include: extracting features from the image to be processed through the text recognition model to obtain feature information, performing feature enhancement processing on the feature information to obtain enhanced feature information; and performing text recognition on the enhanced feature information through the text recognition model to obtain the initial text recognition result.

[0115] It should be noted that, based on the above embodiments, the text recognition model includes at least two different deep neural networks. One deep neural network is used to extract features from the input image to be processed, obtain feature information of the image to be processed, and perform feature enhancement processing on the feature information to obtain enhanced feature information. Based on the enhanced feature information, text recognition is performed on the image content contained in the image to be processed to obtain an initial text recognition result. The other deep neural network is used to extract features from the input image to be processed, obtain feature information of the image to be processed, and perform feature enhancement processing on the feature information to obtain enhanced feature information. Based on the enhanced feature information, scene classification is performed on the image content contained in the image to be processed to obtain a scene classification result.

[0116] In one embodiment of this application, before inputting the image to be processed into the trained character recognition model, the character recognition method of this application embodiment may further include a training process for the character recognition model. Figure 6 This is a schematic flowchart illustrating a character recognition method according to another exemplary embodiment of this application. It should be understood that this method can also be applied to other exemplary implementation environments and specifically executed by devices in other implementation environments. This embodiment does not limit the implementation environment to which the method is applicable.

[0117] like Figure 6 As shown, before inputting the image to be processed into the trained text recognition model, the text recognition model is trained to obtain a trained text recognition model. The training process of the text recognition model may include steps S610 to S630, which are described in detail below:

[0118] Step S610: Obtain the labeled image and use the labeled image as training sample; wherein, the training sample includes expected value data.

[0119] It should be noted that the labeled images in the embodiments of this application include scene classification labels, and each labeled image belongs to a scene corresponding to a classification label.

[0120] It should be noted that the labeled images in the embodiments of this application include text labels in the images. After labeling, for a given character appearing in the image, the sequence of the category to which the character belongs is marked.

[0121] Step S620: Input the labeled image into the initial text recognition model to obtain the initial text recognition result and scene classification result of the labeled image. Then, correct the initial text recognition result according to the scene classification result to obtain the target text information corresponding to the labeled image.

[0122] It should be noted that the initial text recognition model extracts features from the input image to obtain the feature information of the image to be processed, and performs feature enhancement processing on the feature information to obtain enhanced feature information. Based on the enhanced feature information, the model performs text recognition on the image content contained in the image to obtain the initial text recognition result, and performs scene classification on the image content contained in the image to obtain the scene classification result.

[0123] Step S630: Adjust the parameters of the initial character recognition model by using the loss value between the expected value data and the target text information of the labeled image to obtain the trained character recognition model.

[0124] It should be noted that, in the embodiments of this application, the loss value between the expected value data and the target text information of the labeled image includes the text recognition loss value and the scene classification loss value. The text recognition loss value is caused by the parameters in the character classifier, and the scene classification loss value is caused by the parameters in the scene classifier. Therefore, it is necessary to train the text recognition model in advance to adjust the parameter accuracy of the character classifier and the accuracy of the scene classifier in the text recognition model, thereby improving the recognition accuracy of the text recognition model.

[0125] It should be noted that, in the embodiments of this application, the character classifier can be trained based on the Connectionist Temporal Classification (CTC) method, using CTC loss. L CTC Adjust the parameters in the character recognition model.

[0126] For example, CTC loss L CTC The calculation expression is as follows:

[0127]

[0128]

[0129] in, L CTC For character classification, X represents the predicted sequence vector output by the character classifier, Y represents the label sequence, and B represents the loss component. -1 Y is the set of all paths, and π is one of the sub-paths.

[0130] It should be noted that, in the embodiments of this application, the scene classifier in the text recognition model can be a softmax linear classifier, and the training of the scene classifier can be based on adjusting the parameters of the text recognition model based on cross-entropy loss.

[0131] For example, cross-entropy loss L CE The calculation expression is as follows:

[0132]

[0133]

[0134] in, L CE For cross-entropy loss, y j For labels of the real category, p j The scene classifier predicts the probability value of class j.

[0135] In an exemplary embodiment, the expected value data in this application embodiment includes first expected value data and second expected value data; the process of adjusting the parameters of the initial text recognition model in step S630 by using the loss value between the expected value data and the target text information of the labeled image may include the following process, which is described in detail below:

[0136] The first loss component is determined based on the first expected value data and the scene classification result. The second loss component is determined based on the second expected value data and the initial text recognition result. The loss value is calculated based on the first loss component and the second loss component. The parameters of the initial text recognition model are adjusted based on the loss value.

[0137] As described above, in this embodiment, labeled images with scene classification and text annotations are used as training data. First, the text recognition model is deeply trained using the training data. The training completion status of the text recognition model is determined based on the loss value between the expected value data and the target text result information output by the text recognition model, until a well-trained text recognition model is obtained. The trained text recognition model is then used to automatically recognize text and classify scenes from the input image to be processed, obtaining initial text recognition results and scene classification results. The initial text recognition results are then corrected based on the scene classification results to obtain the target text information corresponding to the image to be processed. By utilizing prior information from each scene classification, the text recognition predictions for each scene are integrated. Deep training of the text recognition model can significantly improve the accuracy of text recognition, scene classification, and target text information recognition when processing the input image to be processed.

[0138] Figure 7 This is a block diagram illustrating a character recognition device according to an exemplary embodiment of this application. The device can be applied to… Figure 1 The implementation environment shown is not limited to this embodiment. This device can also be applied to other exemplary implementation environments and specifically configured in other devices. This embodiment does not limit the implementation environment to which the device is applicable.

[0139] like Figure 7 As shown, the exemplary character recognition device includes:

[0140] The input module 710 is configured to input the image to be processed into the trained text recognition model;

[0141] The recognition module 720 is configured to perform text recognition on the image content contained in the image to be processed through a text recognition model to obtain an initial text recognition result, and to perform scene classification to obtain a scene classification result, and to correct the initial text recognition result based on the scene classification result to obtain the target text information corresponding to the image to be processed.

[0142] The output module 730 is configured to output the target text information as the target initial text recognition result.

[0143] In another exemplary embodiment, the identification module 720 includes:

[0144] The mask vector determination submodule is configured to determine a mask vector that matches the scene classification result based on the scene classification result.

[0145] The correction submodule is configured to correct the initial text recognition result based on the mask vector to obtain the target text information corresponding to the image to be processed.

[0146] In another exemplary embodiment, the mask vector determination submodule includes:

[0147] The mask vector determination unit is configured to set the mask vector corresponding to the scene with a probability value greater than a preset threshold as a first value, and set the mask vector corresponding to the scene with a probability value greater than a preset threshold as a second value; wherein, the first value and the second value are normalized values; the initial text recognition result includes character information, and the scene classification result includes the probability values ​​of the character information belonging to each scene.

[0148] In another exemplary embodiment, the correction submodule includes:

[0149] The calculation unit is configured to perform a product operation on the mask vector and the character probability vector to obtain the calculation result; wherein the mask vector includes a first value and a second value after normalization.

[0150] The target text information determination unit is configured to use the calculation result as the target text information corresponding to the image to be processed.

[0151] In another exemplary embodiment, the identification module 720 includes:

[0152] The scenario acquisition submodule is configured to acquire multiple candidate scenarios; these multiple scenarios include scenarios categorized by language type or scenarios categorized by business type.

[0153] The scene classification submodule is configured to calculate the probability value of character information belonging to each scene, obtain the probability value of character information belonging to each scene respectively, and use the probability value of character information belonging to each scene respectively as the scene classification result; the initial text recognition result includes character information.

[0154] In another exemplary embodiment, the character recognition device further includes:

[0155] The feature extraction module is configured to extract features from the image to be processed using a text recognition model to obtain feature information, and to perform feature enhancement processing on the feature information to obtain enhanced feature information.

[0156] The text recognition module is configured to perform text recognition on the image content contained in the image to be processed using a text recognition model to obtain an initial text recognition result, including: performing text recognition on the enhanced feature information using a text recognition model to obtain an initial text recognition result.

[0157] In another exemplary embodiment, the character recognition device further includes:

[0158] The labeled image acquisition module is configured to acquire labeled images and use them as training samples; wherein, the training samples include expected value data;

[0159] The training module is configured to input the labeled image into the initial text recognition model, obtain the initial text recognition result and scene classification result of the labeled image, and correct the initial text recognition result according to the scene classification result to obtain the target text information corresponding to the labeled image;

[0160] The adjustment module is configured to adjust the parameters of the initial text recognition model by using the loss value between the expected value data and the target text information of the labeled image, so as to obtain a trained text recognition model.

[0161] It should be noted that the text recognition device and the text recognition method provided in the above embodiments belong to the same concept. The specific ways in which each module and unit performs its operation have been described in detail in the method embodiments, and will not be repeated here. In practical applications, the text recognition device provided in the above embodiments can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above, and this is not a limitation here.

[0162] Embodiments of this application also provide an electronic device, including: one or more processors; and a storage device for storing one or more programs, which, when executed by one or more processors, cause the electronic device to implement the text recognition methods provided in the above embodiments.

[0163] Figure 8 A schematic diagram of a computer system suitable for implementing the embodiments of this application is shown. It should be noted that... Figure 8 The computer system 800 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0164] like Figure 8 As shown, the computer system 800 includes a Central Processing Unit (CPU) 801, which can perform various appropriate actions and processes based on programs stored in Read-Only Memory (ROM) 802 or programs loaded from storage portion 808 into Random Access Memory (RAM) 803, such as performing the methods described in the above embodiments. The RAM 803 also stores various programs and data required for system operation. The CPU 801, ROM 802, and RAM 803 are interconnected via a bus 804. An I / O interface 805 is also connected to the bus 804.

[0165] The following components are connected to I / O interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to I / O interface 805 as needed. A removable medium 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 810 as needed so that computer programs read from it can be installed into storage section 808 as needed.

[0166] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 809, and / or installed from removable medium 811. When the computer program is executed by central processing unit (CPU) 801, it performs various functions defined in the system of this application.

[0167] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0168] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0169] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0170] Another aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the character recognition method as described above. This computer-readable storage medium may be included in the electronic device described in the above embodiments, or it may exist independently and not assembled into the electronic device.

[0171] The above description is merely a preferred exemplary embodiment of this application and is not intended to limit the implementation of this application. Those skilled in the art can easily make corresponding modifications or alterations based on the main concept and spirit of this application. Therefore, the scope of protection of this application should be determined by the scope of protection claimed in the claims.

Claims

1. A character recognition method, characterized in that, include: The image to be processed is input into the trained text recognition model; The text recognition model is used to perform text recognition on the image content contained in the image to be processed to obtain an initial text recognition result, and to perform scene classification to obtain a scene classification result. A mask vector matching the scene classification result is determined based on the scene classification result. The initial text recognition result includes a character probability vector, and the mask vector includes a normalized first value and a second value. The mask vector and the character probability vector are multiplied to obtain the result, and the result is used as the target text information corresponding to the image to be processed. The target text information is output as the initial target text recognition result.

2. The method according to claim 1, characterized in that, The initial text recognition result includes character information, and the scene classification result includes the probability values ​​of the character information belonging to each scene. The step of determining the mask vector that matches the scene classification result based on the scene classification result includes: The mask vector corresponding to the scene whose probability value is greater than a preset threshold is set as a first value, and the mask vector corresponding to the scene whose probability value is greater than a preset threshold is set as a second value, so as to obtain a mask vector that matches the scene classification result; wherein, the first value and the second value are normalized values.

3. The method according to claim 1, characterized in that, The initial text recognition result includes character information; the scene classification result includes: Obtain multiple candidate scenarios; wherein, the multiple scenarios include multiple scenarios divided by language type, or multiple scenarios divided by business type; The probability values ​​of the character information belonging to each scene are calculated to obtain the probability values ​​of the character information belonging to each scene respectively, and the probability values ​​of the character information belonging to each scene respectively are used as the scene classification results.

4. The method according to claim 1, characterized in that, Before obtaining the initial text recognition result by performing text recognition on the image content contained in the image to be processed through the text recognition model, the method further includes: The text recognition model is used to extract features from the image to be processed to obtain feature information, and the feature information is then subjected to feature enhancement processing to obtain enhanced feature information. The step of obtaining an initial text recognition result by performing text recognition on the image content contained in the image to be processed using the text recognition model includes: The initial character recognition result is obtained by performing character recognition on the enhanced feature information using the character recognition model.

5. The method according to any one of claims 1 to 4, characterized in that, Before inputting the image to be processed into the trained text recognition model, the method further includes: Obtain labeled images and use the labeled images as training samples; wherein, the training samples include expected value data; The labeled image is input into an initial text recognition model to obtain the initial text recognition result and scene classification result of the labeled image. The initial text recognition result is then corrected based on the scene classification result to obtain the target text information corresponding to the labeled image. The parameters of the initial character recognition model are adjusted by using the loss value between the expected value data and the target text information of the labeled image to obtain the trained character recognition model.

6. A character recognition device, characterized in that, include: The input module is configured to input the image to be processed into the trained text recognition model; The recognition module is configured to perform character recognition on the image content contained in the image to be processed using the character recognition model to obtain an initial character recognition result, and to perform scene classification to obtain a scene classification result, and to determine a mask vector that matches the scene classification result based on the scene classification result; wherein, the initial character recognition result includes a character probability vector, and the mask vector includes a normalized first value and a second value; the mask vector and the character probability vector are multiplied to obtain a calculation result, and the calculation result is used as the target text information corresponding to the image to be processed; The output module is configured to output the target text information as the target initial text recognition result.

7. An electronic device, characterized in that, include: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the electronic device to implement the character recognition method as described in any one of claims 1 to 5.

8. A computer-readable medium, characterized in that, It stores computer-readable instructions, which, when executed by the computer's processor, cause the computer to perform the character recognition method according to any one of claims 1 to 5.

Images