Character recognition method, device, electronic device and computer-readable storage medium

Abstract

The embodiments of the present application provide a character recognition method, device, electronic device and computer-readable storage medium, which relate to the field of computer technology. The method includes: obtaining a first recognition result of a first character recognition of an image to be recognized, and a first confidence value of the first recognition result; wherein the first character recognition includes recognizing at least one of printed text and handwritten text; then obtaining a third recognition result of a second character recognition of the second recognition result, and a second confidence value of the third recognition result; and then determining the text content in the image to be recognized. The embodiments of the present application realize direct mixed recognition of images containing printed text and handwritten text, without the need for image classification before recognition, and further recognize the mixed recognized image based on the confidence value, thereby improving the accuracy of recognizing characters in the image.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and more specifically, to a character recognition method, device, electronic device, and computer-readable storage medium. Background Art

[0002] Since corporate document digitization requires extracting text from documents, Optical Character Recognition (OCR) technology has become the preferred choice. While OCR can detect and identify characters in image documents, real documents often contain a mix of printed, handwritten, and stamped fonts, making recognition difficult.

[0003] Therefore, a method based on neural networks for character recognition has emerged. For example, when using neural networks to recognize characters in image documents, a detection model is first used to obtain the location of key text bounding boxes and intercept the text area. Then, a neural network classifier is used to classify the text area, that is, to distinguish between multiple character types. Finally, different recognition models are used for character recognition based on the character type in the text area.

[0004] However, neural network classifiers may make mistakes when classifying multiple fonts, resulting in the continued use of recognition models with incorrect predictions to identify characters in subsequent recognition. In addition, due to the imbalance of the types of training sets for the recognition models, inaccurate recognition results are prone to occur. For example, when there are a large number of printed text samples in the training set, the classifier is more likely to identify printed text in subsequent recognition, thereby interfering with the output. Summary of the Invention

[0005] The purpose of the embodiments of the present application is to solve the problem of low accuracy when performing character recognition on images containing multiple fonts.

[0006] According to one aspect of an embodiment of the present application, a character recognition method is provided, the method comprising:

[0007] Obtaining a first recognition result of first character recognition of the image to be recognized and a first confidence value of the first recognition result; wherein the first character recognition includes recognizing at least one of printed text and handwritten text;

[0008] Obtaining a third recognition result of performing a second character recognition on the second recognition result, and a second confidence value of the third recognition result; wherein the second recognition result includes the first recognition result whose first confidence value is lower than a preset threshold; and the second character recognition includes recognizing handwritten text;

[0009] The text content in the image to be recognized is determined according to the first recognition result, the first confidence value, the third recognition result, and the second confidence value.

[0010] Optionally, obtaining a first recognition result of performing first character recognition on the image to be recognized and a first confidence value of the first recognition result includes:

[0011] Inputting the image to be recognized into the first processor, performing first character recognition on the image to be recognized by the first processor, and obtaining a first recognition result and a first confidence value of the first recognition result;

[0012] Obtaining a third recognition result of performing a second character recognition on the second recognition result, and a second confidence value of the third recognition result, including:

[0013] The image to be recognized corresponding to the second recognition result is input into the second processor, and the second processor performs second character recognition to obtain a third recognition result and a second confidence value of the third recognition result.

[0014] Optionally, after obtaining a third recognition result of performing a second character recognition on the second recognition result and a second confidence value of the third recognition result, the method further includes:

[0015] The third recognition result is matched with a character string in a preset character library, and a third recognition result having the largest second confidence value is selected.

[0016] Optionally, before obtaining a first recognition result of performing first character recognition on the image to be recognized and a first confidence value of the first recognition result, the method includes:

[0017] Scale the image to be identified to a preset size and normalize the RGB values ​​of the scaled image to be identified;

[0018] The normalized image to be identified is converted into a grayscale image, and Gaussian blur and binarization are performed.

[0019] Optionally, before obtaining a first recognition result of performing first character recognition on the image to be recognized and a first confidence value of the first recognition result, the method further includes:

[0020] A text block in an image to be recognized is obtained, and the text block is rotated to a horizontal direction; wherein the text block is a rectangular image including at least one of printed text and handwritten text.

[0021] Optionally, inputting the image to be recognized corresponding to the second recognition result into a second processor, and performing second character recognition according to the second processor includes:

[0022] The number of characters in the text block is determined according to the ratio of the length to the width of the text block, and the character string in the text block is divided into characters by using the Gaussian smoothing extreme value connected domain method for the second character recognition.

[0023] Optionally, before obtaining a first recognition result of performing first character recognition on the image to be recognized and a first confidence value of the first recognition result, the method further includes:

[0024] Obtaining a training sample set; the training sample set includes printed text and handwritten text;

[0025] Input the training sample set into the initial processor to obtain the initial recognition result and the corresponding initial confidence value;

[0026] The initial processor is reversely optimized according to the initial recognition result and the initial confidence value to obtain an optimized processor, until a first processor that meets the preset accuracy requirement is obtained.

[0027] According to another aspect of an embodiment of the present application, a character recognition device is provided, the device comprising:

[0028] a first recognition module, configured to obtain a first recognition result of performing first character recognition on the image to be recognized, and a first confidence value of the first recognition result; wherein the first character recognition includes recognizing at least one of printed text and handwritten text;

[0029] a second recognition module, configured to obtain a third recognition result of performing a second character recognition on the second recognition result, and a second confidence value of the third recognition result; wherein the second recognition result includes the first recognition result having a first confidence value lower than a preset threshold; and the second character recognition includes recognizing handwritten text;

[0030] The determination module is used to determine the text content in the image to be recognized based on the first recognition result, the first confidence value, the third recognition result and the second confidence value.

[0031] Optionally, the first identification module is specifically configured to:

[0032] Inputting the image to be recognized into the first processor, performing first character recognition on the image to be recognized by the first processor, and obtaining a first recognition result and a first confidence value of the first recognition result;

[0033] The second identification module is specifically used for:

[0034] The image to be recognized corresponding to the second recognition result is input into the second processor, and the second processor performs second character recognition to obtain a third recognition result and a second confidence value of the third recognition result.

[0035] Optionally, the device further comprises:

[0036] The screening module is used to match the third recognition result with a character string in a preset character library, and screen out the third recognition result with the largest second confidence value.

[0037] Optionally, the device includes:

[0038] A first pre-processing module is used to scale the image to be identified to a preset size and normalize the RGB values ​​of the scaled image to be identified;

[0039] The second preprocessing module is used to convert the normalized image to be identified into a grayscale image, and perform Gaussian blur processing and binarization processing.

[0040] Optionally, the device further comprises:

[0041] The third preprocessing module is used to obtain a text block in the image to be recognized and rotate the text block to a horizontal direction; wherein the text block is a rectangular image including at least one of printed text and handwritten text.

[0042] Optionally, the second identification module includes:

[0043] The segmentation module is used to determine the number of characters in the text block according to the ratio of the length to the width of the text block, and segment the character string in the text block into characters by using the Gaussian smoothing extreme connected domain method to perform the second character recognition.

[0044] Optionally, the device further includes a training module; the training module is specifically configured to:

[0045] Obtaining a training sample set; the training sample set includes printed text and handwritten text;

[0046] Input the training sample set into the initial processor to obtain the initial recognition result and the corresponding initial confidence value;

[0047] The initial processor is reversely optimized according to the initial recognition result and the initial confidence value to obtain an optimized processor, until a first processor that meets the preset accuracy requirement is obtained.

[0048] According to another aspect of an embodiment of the present application, an electronic device is provided, which includes a memory, a processor, and a computer program stored in the memory, and the processor executes the computer program to implement the steps of the character recognition method in any one of the above aspects.

[0049] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the character recognition method according to any of the above aspects are implemented.

[0050] The beneficial effects of the technical solution provided by the embodiments of the present application are:

[0051] By directly performing mixed recognition on images containing printed text and handwritten text, there is no need to classify the images before recognition, and further recognition is performed on the mixed recognized images based on the confidence value, thereby improving the accuracy of character recognition in the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments of the present application.

[0053] Figure 1 One of the flow charts of a character recognition method provided in an embodiment of the present application;

[0054] Figure 2 A second flow chart of a character recognition method provided in an embodiment of the present application;

[0055] Figure 3 The third flowchart of a character recognition method provided in an embodiment of the present application;

[0056] Figure 4 A schematic diagram of a process for preprocessing an image to be identified provided in an embodiment of the present application;

[0057] Figure 5 A schematic diagram of the structure of a character recognition device provided in an embodiment of the present application;

[0058] Figure 6 A schematic structural diagram of an electronic device for character recognition provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] The following describes the embodiments of the present application in conjunction with the accompanying drawings. It should be understood that the embodiments described below in conjunction with the accompanying drawings are exemplary descriptions for explaining the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions of the embodiments of the present application.

[0060] Those skilled in the art will understand that, unless otherwise stated, the singular forms "a", "an", "said", and "the" used herein may also include plural forms. It should be further understood that the terms "including" and "comprising" used in the embodiments of the present application mean that the corresponding features can be implemented as the presented features, information, data, steps, operations, elements, and / or components, but do not exclude implementation as other features, information, data, steps, operations, elements, components, and / or combinations thereof supported by the present technical field. It should be understood that when we say that an element is "connected" or "coupled" to another element, the element can be directly connected or coupled to the other element, or it can refer to the element and the other element establishing a connection relationship through an intermediate element. In addition, the "connection" or "coupling" used here can include wireless connection or wireless coupling. The term "and / or" used here indicates at least one of the items defined by the term, for example, "A and / or B" can be implemented as "A", or as "B", or as "A and B".

[0061] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0062] The following describes several exemplary embodiments to illustrate the technical solutions of the embodiments of the present application and the technical effects produced by the technical solutions of the present application. It should be noted that the following embodiments can refer to, draw on, or combine with each other, and the same terms, similar features, and similar implementation steps in different embodiments will not be repeated.

[0063] The present application provides a character recognition method, such as Figure 1 Shown, including:

[0064] Step S101, obtaining a first recognition result of a first character recognition performed on an image to be recognized, and a first confidence value of the first recognition result;

[0065] The first character recognition includes recognizing at least one of printed text and handwritten text.

[0066] The image to be recognized is an image document containing characters to be recognized. The characters to be recognized in the image to be recognized may be printed text, handwritten text, or both printed text and handwritten text.

[0067] Perform a first character recognition on the image to be recognized, and obtain a first recognition result and a first confidence value corresponding to the first recognition result. The first character recognition includes recognizing at least one of printed text and handwritten text. That is, the first character recognition can be recognition of printed text, recognition of handwritten text, or recognition of a mixture of printed text and handwritten text. The first confidence value of the first recognition result is between 0 and 1 and is used to indicate the degree of credibility of the first recognition result.

[0068] Step S102, obtaining a third recognition result of performing second character recognition on the second recognition result, and a second confidence value of the third recognition result;

[0069] The second recognition result includes a first recognition result whose first confidence value is lower than a preset threshold; and the second character recognition includes recognizing handwritten text.

[0070] A corresponding second recognition result may be determined based on a first recognition result obtained by performing first character recognition on the image to be recognized, wherein the second recognition result includes a first recognition result having a first confidence value lower than a preset threshold.

[0071] Generally speaking, when performing first-character recognition on an image, the recognition accuracy for printed text is higher than that for handwritten text, and accordingly, the first confidence value is also higher. This is mainly because the features of printed text are relatively fixed, while handwritten text is more random. The features of handwritten text written by different subjects are inconsistent, and strokes may be randomly connected or missing.

[0072] Therefore, in the embodiment of the present application, thresholds are preset according to different recognition requirements. When the first confidence value of the first recognition result is lower than the preset threshold, a secondary recognition, i.e., second character recognition, is performed on the part of the image to be recognized corresponding to the second recognition result that is lower than the preset threshold; when the first confidence value of the first recognition result is not lower than the preset threshold, it is determined that the first recognition result is credible and no secondary recognition is required.

[0073] A second character recognition is performed on the second recognition result to obtain a third recognition result and a second confidence value corresponding to the third recognition result. The second confidence value is between 0 and 1 and is used to indicate the reliability of the third recognition result. The second character recognition includes recognizing handwritten text. Unlike the first character recognition, the second character recognition is more accurate in recognizing handwritten text.

[0074] Step S103 : determining the text content in the image to be recognized based on the first recognition result, the first confidence value, the third recognition result, and the second confidence value.

[0075] The first confidence value of the first recognition result can be used to determine the text content with higher credibility in the first recognition result; the second confidence value of the third recognition result can be used to determine the text content with higher credibility in the third recognition result. Furthermore, the text content with higher credibility in the third recognition result corresponds to the text content with lower credibility in the first recognition result.

[0076] In other words, the first recognition result and the third recognition result are combined to ultimately determine the text content in the image to be recognized.

[0077] Applying the character recognition method provided in an embodiment of the present application, a first recognition result of performing first character recognition on an image to be recognized and a first confidence value of the first recognition result are obtained; wherein the first character recognition includes recognizing at least one of printed text and handwritten text; then a third recognition result of performing second character recognition on the second recognition result and a second confidence value of the third recognition result are obtained; wherein the second recognition result includes a first recognition result whose first confidence value is lower than a preset threshold; the second character recognition includes recognizing handwritten text; and then, based on the first recognition result, the first confidence value, the third recognition result and the second confidence value, the text content in the image to be recognized is determined.

[0078] The embodiment of the present application realizes direct mixed recognition of images containing printed text and handwritten text, without the need for image classification before recognition, and further recognizes the mixed recognized image based on the confidence value, thereby improving the accuracy of recognizing characters in the image.

[0079] The present application provides a character recognition method, such as Figure 2 、 Figure 3 Shown, including:

[0080] Step S201, obtaining a first recognition result of a first character recognition performed on an image to be recognized, and a first confidence value of the first recognition result;

[0081] The first character recognition includes recognizing at least one of printed text and handwritten text.

[0082] The image to be recognized is an image document containing characters to be recognized. The characters to be recognized in the image to be recognized may be printed text, handwritten text, or both printed text and handwritten text.

[0083] Perform a first character recognition on the image to be recognized, and obtain a first recognition result and a first confidence value corresponding to the first recognition result. The first character recognition includes recognizing at least one of printed text and handwritten text. That is, the first character recognition can be recognition of printed text, recognition of handwritten text, or recognition of a mixture of printed text and handwritten text. The first confidence value of the first recognition result is between 0 and 1 and is used to indicate the degree of credibility of the first recognition result.

[0084] In a preferred embodiment of the present application, step S201 includes:

[0085] The image to be recognized is input into the first processor, and the first processor performs first character recognition on the image to be recognized to obtain a first recognition result and a first confidence value of the first recognition result.

[0086] Specifically, the first character recognition can be implemented by the first processor. The image to be recognized is input into the first processor, and a first recognition result and a first confidence value are obtained as output.

[0087] The first processor may include a character recognition model based on a CRNN (Convolutional Recurrent Neural Network) with a backbone model of a 34-layer residual neural network (ResNet), a neck model of an RNN (Recurrent Neural Network), a transcription layer using CTC (Connectionist Temporal Classification), and a training effect verification metric of Accuracy. The first processor outputs a variable-length character string and its confidence as a first recognition result and a first confidence value for an image to be recognized that includes at least one of printed text and handwritten text.

[0088] In a preferred embodiment of the present application, Figure 4 As shown, before step S201, a method for preprocessing the image to be identified is included:

[0089] Step S401 : scaling the image to be identified to a preset size, and normalizing the RGB values ​​of the scaled image to be identified.

[0090] Before performing the first character recognition on the image to be recognized, the image to be recognized needs to be preprocessed, wherein the preprocessing step includes scaling the image to be recognized to a preset size and normalizing the RGB values ​​of the scaled image to be recognized.

[0091] In an embodiment of the present application, the preset size can be 960*960, the normalized RGB channel variance can be [0.229, 0.224, 0.225], the average value is [0.485, 0.456, 0.406], and the data scaling ratio is 1 / 255.

[0092] Step S402 : converting the normalized image to be recognized into a grayscale image, and performing Gaussian blur processing and binarization processing on the image.

[0093] In addition to printed and handwritten text, the normalized image to be recognized may also contain information such as tables and seals. Generally speaking, this information is not the text content that users want to recognize. Therefore, to avoid affecting the recognition results, it is necessary to remove this irrelevant information from the image to be recognized before performing character recognition.

[0094] For example, OpenCV is used to remove red or blue stamps from the image to be identified based on the color channel separation method. It should be noted that any effective stamp removal technology can be applied to the embodiments of this application and will not be described in detail.

[0095] Furthermore, the image to be identified is read and converted into a grayscale image, which is then inverted and Gaussian blurred, and then thresholded to obtain a binary image. The threshold value of the thresholding process can be between 0.3 and 0.7.

[0096] Converting the normalized image to a grayscale image can reduce the computational effort of subsequent image preprocessing. Gaussian blurring the image can reduce image noise. Binarizing the Gaussian blurred image simplifies the image, reduces the amount of data required, and highlights the outlines of the characters within it. Image binarization involves setting the grayscale values ​​of pixels in an image to 0 or 255, effectively creating a distinct black and white image.

[0097] Step S403 , obtaining a text block in the image to be recognized, and rotating the text block to a horizontal direction; wherein the text block is a rectangular image including at least one of printed text and handwritten text.

[0098] The characters to be recognized in the image to be recognized may be horizontal or vertical, or may be tilted at a certain angle relative to the horizontal or vertical direction. Obtain the text block in the image to be recognized, determine the tilt angle of the text block, and obtain the corresponding rotation matrix.

[0099] The text block is a rectangular image including at least one of printed text and handwritten text. Ideally, the text block is a minimum rectangular image including the above-mentioned characters to be recognized.

[0100] The text block is rotated to the horizontal direction using a rotation matrix to obtain a text block after the tilt is corrected. In the embodiment of the present application, the text block can also be rotated to the vertical direction, which will not be described in detail.

[0101] Character detection is performed based on the DBNET model, using differentiable threshold binarization, ResNet50, and the FPN network architecture. It detects suspected text regions containing printed and handwritten text, namely text blocks. Each text block contains horizontally continuous characters.

[0102] Use a direction classification model to check whether the module has resolved the text block tilt issue. Here, only two angles, 0 and 180 degrees, need to be classified, namely, horizontal and vertical directions. Use the MobileNetV3 direction classification model. The model input scale can be 0.35 times the original image. The predicted output is a binary classification result. The ADAM optimizer can be used as the optimizer.

[0103] Step S202, obtaining a third recognition result of performing second character recognition on the second recognition result, and a second confidence value of the third recognition result;

[0104] The second recognition result includes a first recognition result whose first confidence value is lower than a preset threshold; and the second character recognition includes recognizing handwritten text.

[0105] A corresponding second recognition result may be determined based on a first recognition result obtained by performing first character recognition on the image to be recognized, wherein the second recognition result includes a first recognition result having a first confidence value lower than a preset threshold.

[0106] Generally speaking, when performing first-character recognition on an image, the recognition accuracy for printed text is higher than that for handwritten text, and accordingly, the first confidence value is also higher. This is mainly because the features of printed text are relatively fixed, while handwritten text is more random. The features of handwritten text written by different subjects are inconsistent, and strokes may be randomly connected or missing.

[0107] Therefore, in the embodiment of the present application, thresholds are preset according to different recognition requirements. When the first confidence value of the first recognition result is lower than the preset threshold, a secondary recognition, i.e., second character recognition, is performed on the part of the image to be recognized corresponding to the second recognition result that is lower than the preset threshold; when the first confidence value of the first recognition result is not lower than the preset threshold, it is determined that the first recognition result is credible and no secondary recognition is required.

[0108] The output of the confidence value is differentiated by the sigmoid function and has approached the two extremes. The preset threshold in the embodiment of the present application can be 0.8, which is better for distinguishing between reliable results and untested results, and the F1-score is higher.

[0109] A second character recognition is performed on the second recognition result to obtain a third recognition result and a second confidence value corresponding to the third recognition result. The second confidence value is between 0 and 1 and is used to indicate the reliability of the third recognition result. The second character recognition includes recognizing handwritten text. Unlike the first character recognition, the second character recognition is more accurate in recognizing handwritten text.

[0110] In a preferred embodiment of the present application, step S202 includes:

[0111] The image to be recognized corresponding to the second recognition result is input into the second processor, and the second processor performs second character recognition to obtain a third recognition result and a second confidence value of the third recognition result.

[0112] Specifically, the second character recognition can be implemented by the second processor. The image to be recognized is input into the second processor, and the output third recognition result and the second confidence value are obtained.

[0113] The second processor may include a proprietary character recognition model based on the GoogleNet structure, which focuses on recognizing handwritten characters. The second processor outputs a Character i topk , that is, the k characters with the highest confidence values ​​and the corresponding confidence values, are used as the third recognition result and the second confidence value.

[0114] In a preferred embodiment of the present application, step S202 includes:

[0115] The number of characters in the text block is determined according to the ratio of the length to the width of the text block, and the character string in the text block is divided into characters by using the Gaussian smoothing extreme value connected domain method for the second character recognition.

[0116] The second processor is mainly used to perform a second character recognition on the image to be recognized corresponding to the second recognition result. To improve the recognition accuracy, the second character recognition includes single-word recognition of the characters in the text block. Prior to this, it is necessary to segment the continuous character strings in the text block into single-word characters for recognition.

[0117] The number of characters in the text block is determined based on the ratio of the length to the width of the text block, and then the Gaussian smoothing extreme connected domain method is used to segment the characters. The segmentation step includes binarizing the text block, setting pixels greater than 200 to 1, and using the connected domain method for pixels with a pixel value of 1 to find the key strokes. The coordinates of the nearest point of each pixel are then substituted into a two-dimensional Gaussian distribution function with a variance of 1 or 1.5, and the 9 points are divided by the sum of the weights of the 9 points. The closer to the center point, the greater the weight, and the more prominent the font strokes will be when projected in grayscale.

[0118] Step S203 : Match the third recognition result with the character string in the preset character library, and select the third recognition result with the largest second confidence value.

[0119] The preset character library is a preset character library associated with the service and is updated irregularly. For example, for the handwritten signature location service, the preset character library is a character library including the signer's name.

[0120] The text content in the third recognition result is corrected using prior information. Common errors include errors in similar characters, grammatical errors, and errors that do not match the usage of professional terms in the business field.

[0121] In the embodiment of the present application, the third recognition result is matched with the character string in the preset character library, which can be achieved through a character comparison model. Among them, the character comparison model includes but is not limited to the CTC algorithm model and the BK-TREE (Burkhard Keller Tree, discrete metric space tree) algorithm model. The CTC algorithm model can ignore the length of the strokes, predict and align single characters, and judge the semantics; the BK-TREE algorithm model can find words that have similar meanings to the target character.

[0122] The longest common subsequence algorithm and confidence level are used to align and compare characters. Each word is passed through a proprietary character recognition model to predict and output k characters and corresponding confidence levels. When k=30, P(GroundTruth i ∈Char i topk )>0.99. When the number of predicted characters i=3, (C1,C2,C3)∈Char 1 topk ×Char 2 topk ×Char 3 topk There is a solution such that p=p1P2P3 reaches its maximum value, and (C1, C2, C3) is the output result. In practical applications, the number of predicted characters may not be equal to the actual number of characters, so when traversing the preset character library, the longest common subsequence LCS is combinedname Output the recognition result.

[0123] For example, assume k = 3, the predicted number of characters i = 2, the first character prediction is "Wang (0.8), Tu (0.9), Gan (0.7)", and the second character prediction is "Yi (0.7), Er (0.8), San (0.6)". The calculation process is as follows:

[0124] Wang (0.8) * Yi (0.7) = Wangyi (0.56), Wang (0.8) * Er (0.8) = Wanger (0.64), Wang (0.8) * San (0.6) = Wangsan (0.48), Tu (0.9) * Yi (0.7) = Tuyi (0.63), Tu (0.9) * Er (0.8) = Tu'er (0.72), Tu (0.9) * San (0.6) = Tusan (0.54), Gan (0.7) * Yi (0.7) = Ganyi (0.49), Gan (0.7) * Er (0.8) = Gan'er (0.56), Gan (0.7) * San (0.6) = Gansan (0.42).

[0125] Then the value of "Tu (0.9) x Er (0.8) = Tu'er (0.72)" calculated is the largest, and "Wang (0.8) x Er (0.8) = Wang'er (0.64)" is the second largest. Assume there are strings "Wang'er, Li'er, Tusan" in the preset character library, then "Wang'er (0.64)" is the output result. Although the value of "Tu'er (0.72)" is larger, it does not match the strings in the preset character library. Although "Tusan (0.54)" matches the strings in the preset character library, the calculated second confidence value is less than the second confidence value of "Wang'er (0.64)", so it is not used as the output.

[0126] Step S204, determine the text content in the image to be recognized according to the first recognition result, the first confidence value, the third recognition result, and the second confidence value.

[0127] According to the first confidence value of the first recognition result, the text content with higher credibility in the first recognition result can be determined; according to the second confidence value of the third recognition result, the text content with higher credibility in the third recognition result can be determined. At the same time, the text content with higher credibility in the third recognition result corresponds to the text content with lower credibility in the first recognition result.

[0128] In other words, combine the first recognition result with the third recognition result to finally determine the text content in the image to be recognized.

[0129] Apply the character recognition method provided in the embodiment of the present application to obtain a first recognition result of a first character recognition of an image to be recognized, and a first confidence value of the first recognition result; wherein the first character recognition includes recognizing at least one of printed text and handwritten text; then obtain a third recognition result of a second character recognition of the second recognition result, and a second confidence value of the third recognition result; wherein the second recognition result includes a first recognition result whose first confidence value is lower than a preset threshold; the second character recognition includes recognizing handwritten text; the third recognition result is matched with a character string in a preset character library, and a third recognition result with the largest matching second confidence value is screened out; and then, based on the first recognition result, the first confidence value, the third recognition result and the second confidence value, the text content in the image to be recognized is determined.

[0130] The embodiment of the present application realizes direct mixed recognition of images containing printed text and handwritten text, without the need for image classification before recognition, and further recognizes the mixed recognized image based on the confidence value, thereby improving the accuracy of recognizing characters in the image.

[0131] An embodiment of the present application provides a training method for a first processor, including:

[0132] Step 1: Obtain a training sample set; the training sample set includes printed text and handwritten text.

[0133] The training sample set is an image file dataset that includes printed text and handwritten text. The printed text used for training can be obtained from the ICDAR2015 document text dataset, and the handwritten text used for training can be obtained from the CASIA Chinese handwritten character dataset. The training sample set generated based on the printed and handwritten text is annotated with GroundTruth, and a portion of the data forms the training sample set for the first processor.

[0134] Step 2: Input the training sample set into the initial processor to obtain the initial recognition result and the corresponding initial confidence value.

[0135] For the initial processor, set the corresponding training parameters and generate a configuration file. The configuration file can be a yml file, which configures the training sample set, the path location of the images to be recognized in the training sample set, and various hyperparameters of the initial processor itself, such as the learning rate and number of iterations.

[0136] The training sample set is input into the initial processor, and the initial processor is used to perform character recognition processing to obtain the initial recognition result and the corresponding initial confidence value.

[0137] Step three: reversely optimize the initial processor according to the initial recognition result and the initial confidence value to obtain an optimized processor, until a first processor that meets the preset accuracy requirement is obtained.

[0138] The initial recognition results and initial confidence values ​​output by the initial processor are classified according to a preset threshold. For example, if the preset threshold is 0.8, training samples with initial confidence values ​​below 0.8 are considered negative samples, and training samples with initial confidence values ​​not less than 0.8 are considered positive samples. The initial processor is then reversely optimized based on the positive and negative samples until a first processor that meets the preset accuracy requirements is obtained.

[0139] The embodiment of the present application provides a character recognition device, such as Figure 5 As shown, the character recognition device 50 may include:

[0140] A first recognition module 501 is configured to obtain a first recognition result of first character recognition of the image to be recognized, and a first confidence value of the first recognition result; wherein the first character recognition includes recognizing at least one of printed text and handwritten text;

[0141] A second recognition module 502 is configured to obtain a third recognition result of performing a second character recognition on the second recognition result, and a second confidence value of the third recognition result; wherein the second recognition result includes the first recognition result having a first confidence value lower than a preset threshold; and the second character recognition includes recognizing handwritten text;

[0142] The determination module 503 is configured to determine the text content in the image to be recognized based on the first recognition result, the first confidence value, the third recognition result, and the second confidence value.

[0143] Optionally, the first identification module 501 is specifically configured to:

[0144] Inputting the image to be recognized into the first processor, performing first character recognition on the image to be recognized by the first processor, and obtaining a first recognition result and a first confidence value of the first recognition result;

[0145] The second identification module 502 is specifically configured to:

[0146] The image to be recognized corresponding to the second recognition result is input into the second processor, and the second processor performs second character recognition to obtain a third recognition result and a second confidence value of the third recognition result.

[0147] Optionally, the device further comprises:

[0148] The screening module is used to match the third recognition result with a character string in a preset character library, and screen out the third recognition result with the largest second confidence value.

[0149] Optionally, the device includes:

[0150] A first pre-processing module is used to scale the image to be identified to a preset size and normalize the RGB values ​​of the scaled image to be identified;

[0151] The second preprocessing module is used to convert the normalized image to be identified into a grayscale image, and perform Gaussian blur processing and binarization processing.

[0152] Optionally, the device further comprises:

[0153] The third preprocessing module is used to obtain a text block in the image to be recognized and rotate the text block to a horizontal direction; wherein the text block is a rectangular image including at least one of printed text and handwritten text.

[0154] Optionally, the second identification module 502 includes:

[0155] The segmentation module is used to determine the number of characters in the text block according to the ratio of the length to the width of the text block, and segment the character string in the text block into characters by using the Gaussian smoothing extreme connected domain method to perform the second character recognition.

[0156] Optionally, the device further includes a training module; the training module is specifically configured to:

[0157] Obtaining a training sample set; the training sample set includes printed text and handwritten text;

[0158] Input the training sample set into the initial processor to obtain the initial recognition result and the corresponding initial confidence value;

[0159] The initial processor is reversely optimized according to the initial recognition result and the initial confidence value to obtain an optimized processor, until a first processor that meets the preset accuracy requirement is obtained.

[0160] Using the character recognition device provided in an embodiment of the present application, a first recognition result of first character recognition of an image to be recognized and a first confidence value of the first recognition result are obtained; wherein the first character recognition includes recognizing at least one of printed text and handwritten text; then a third recognition result of second character recognition of the second recognition result and a second confidence value of the third recognition result are obtained; wherein the second recognition result includes a first recognition result whose first confidence value is lower than a preset threshold; the second character recognition includes recognizing handwritten text; the third recognition result is matched with a character string in a preset character library, and a third recognition result with the largest matching second confidence value is screened out; and then the text content in the image to be recognized is determined based on the first recognition result, the first confidence value, the third recognition result and the second confidence value.

[0161] The embodiment of the present application realizes direct mixed recognition of images containing printed text and handwritten text, without the need for image classification before recognition, and further recognizes the mixed recognized image based on the confidence value, thereby improving the accuracy of recognizing characters in the image.

[0162] An embodiment of the present application provides an electronic device comprising a memory, a processor, and a computer program stored on the memory. The processor executes the computer program to implement the steps of a character recognition method. Compared with related technologies, the following can be achieved: by directly performing mixed recognition on images containing printed text and handwritten text, without the need for image classification before recognition, and further recognizing the mixed recognized images based on confidence values, thereby improving the accuracy of recognizing characters in the image.

[0163] In an alternative embodiment, an electronic device is provided, such as Figure 6 As shown, Figure 6 The electronic device 6000 shown includes: a processor 6001 and a memory 6003. The processor 6001 and the memory 6003 are connected, for example, via a bus 6002. Optionally, the electronic device 6000 may further include a transceiver 6004, which may be used for data exchange between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the number of transceivers 6004 is not limited to one, and the structure of the electronic device 6000 does not constitute a limitation on the embodiments of the present application.

[0164] Processor 6001 can be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 6001 can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0165] The bus 6002 may include a path for transmitting information between the above components. The bus 6002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 6002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0166] The memory 6003 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium that can be used to carry or store computer programs and can be read by a computer, without limitation here.

[0167] The memory 6003 is used to store the computer program for executing the embodiments of the present application, and the execution is controlled by the processor 6001. The processor 6001 is used to execute the computer program stored in the memory 6003 to implement the steps shown in the above method embodiments.

[0168] An embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps and corresponding contents of the aforementioned method embodiment can be implemented.

[0169] An embodiment of the present application also provides a computer program product, including a computer program, which can implement the steps and corresponding contents of the aforementioned method embodiment when executed by a processor.

[0170] The terms "first," "second," "third," "fourth," "1," "2," and the like (if any) in the specification and claims of this application and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or sequential sequence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of the application described herein can be implemented in an order other than that shown or described in the drawings.

[0171] It should be understood that, although each operation step is indicated by arrows in the flowchart of the embodiment of the present application, the order of implementation of these steps is not limited to the order indicated by the arrows. Unless otherwise clearly stated herein, in some implementation scenarios of the embodiment of the present application, the implementation steps in each flowchart can be performed in other orders according to demand. In addition, some or all of the steps in each flowchart can include multiple sub-steps or multiple stages based on actual implementation scenarios. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage in these sub-steps or stages can also be executed at different times respectively. Under different scenarios at the execution time, the execution order of these sub-steps or stages can be flexibly configured according to demand, and the embodiment of the present application does not limit this.

[0172] The above description is only an optional implementation method for some implementation scenarios of this application. It should be pointed out that for ordinary technicians in this technical field, without departing from the technical concept of the solution of this application, the use of other similar implementation methods based on the technical ideas of this application also falls within the protection scope of the embodiments of this application.

Claims

1. A character recognition method, characterized in that: include: Obtaining a first recognition result of first character recognition performed on the image to be recognized, and a first confidence value of the first recognition result; wherein the first character recognition includes recognizing at least one of printed text and handwritten text, and the first character recognition has a higher recognition accuracy rate for the printed text than for the handwritten text; performing a second character recognition on a portion of the image to be recognized corresponding to the second recognition result to obtain a third recognition result and a second confidence value for the third recognition result; wherein the second recognition result includes the first recognition result for which the first confidence value is lower than a preset threshold; the second character recognition includes recognizing the handwritten text, and the second character recognition has a higher recognition accuracy rate for the handwritten text than the first character recognition; Determine text content in the image to be recognized based on the first recognition result, the first confidence value, the third recognition result, and the second confidence value.

2. The character recognition method according to claim 1, wherein: The obtaining of a first recognition result of a first character recognition performed on the image to be recognized and a first confidence value of the first recognition result includes: Inputting the image to be recognized into a first processor, performing first character recognition on the image to be recognized according to the first processor, and obtaining a first recognition result and a first confidence value of the first recognition result; The obtaining of a third recognition result of performing second character recognition on the second recognition result, and a second confidence value of the third recognition result, includes: The image to be recognized corresponding to the second recognition result is input into the second processor, and the second processor performs second character recognition to obtain a third recognition result and a second confidence value of the third recognition result.

3. The character recognition method according to claim 1, wherein: After obtaining a third recognition result of performing second character recognition on the second recognition result and a second confidence value of the third recognition result, the method further includes: The third recognition result is matched with a character string in a preset character library, and the third recognition result having the largest second confidence value among the matched ones is screened out.

4. The character recognition method according to claim 1, wherein: Before obtaining a first recognition result of a first character recognition performed on the image to be recognized and a first confidence value of the first recognition result, the method includes: Scale the image to be identified to a preset size and normalize the RGB values ​​of the scaled image to be identified; The normalized image to be identified is converted into a grayscale image, and Gaussian blur and binarization are performed.

5. The character recognition method according to claim 4, wherein: Before obtaining a first recognition result of performing first character recognition on the image to be recognized and a first confidence value of the first recognition result, the method further includes: A text block in the image to be recognized is obtained, and the text block is rotated to a horizontal direction; wherein the text block is a rectangular image including at least one of printed text and handwritten text.

6. The character recognition method according to claim 5, wherein: Inputting the image to be recognized corresponding to the second recognition result into the second processor, and performing second character recognition according to the second processor, includes: The number of characters in the text block is determined according to the ratio of the length to the width of the text block, and the character string in the text block is divided into characters by using the Gaussian smoothing extreme connected domain method to perform second character recognition.

7. The character recognition method according to claim 2, wherein: Before obtaining a first recognition result of performing first character recognition on the image to be recognized and a first confidence value of the first recognition result, the method further includes: Obtaining a training sample set; the training sample set includes printed text and handwritten text; Inputting the training sample set into an initial processor to obtain an initial recognition result and a corresponding initial confidence value; The initial processor is reversely optimized according to the initial recognition result and the initial confidence value to obtain an optimized processor, until the first processor that meets the preset accuracy requirement is obtained.

8. A character recognition device, characterized in that: include: a first recognition module, configured to obtain a first recognition result of first character recognition of the image to be recognized, and a first confidence value of the first recognition result; wherein the first character recognition includes recognizing at least one of printed text and handwritten text, and the first character recognition has a higher recognition accuracy rate for the printed text than for the handwritten text; a second recognition module configured to perform second character recognition on a portion of the image to be recognized corresponding to the second recognition result, and obtain a third recognition result and a second confidence value of the third recognition result; wherein the second recognition result includes the first recognition result for which the first confidence value is lower than a preset threshold; and the second character recognition includes recognizing the handwritten text, and the second character recognition has a higher recognition accuracy rate for the handwritten text than the first character recognition; A determination module is used to determine the text content in the image to be recognized based on the first recognition result, the first confidence value, the third recognition result and the second confidence value.

9. An electronic device, characterized in that: The electronic device comprises: one or more processors; Memory; One or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to: execute the character recognition method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the character recognition method according to any one of claims 1 to 7 is implemented.

Images