A text recognition method, device, terminal equipment and storage medium

Abstract

The present invention discloses a text recognition method, device, terminal equipment and storage medium, wherein the text recognition method comprises: obtaining a sample text data set, preprocessing the sample text data set, generating a label image; inputting the label image into a text recognition model for training, extracting image features using a convolution layer, downsampling using a pooling layer, restoring image resolution using an upsampling layer or a deconvolution layer, and finally normalizing the output probability using a sigmoid layer to output prediction graphs of multiple scales, and optimizing the loss function of the text recognition model to obtain a trained text recognition model; preprocessing the text image to be recognized and inputting it into the trained text recognition model to output a prediction graph of a clear scale; parsing the prediction graph of the clear scale to obtain a text sequence of the text image to be recognized. The present invention realizes end-to-end text recognition through a fully convolutional neural network, with a simple process, small amount of calculation and high accuracy.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a text recognition method, device, terminal equipment and storage medium. Background Art

[0002] Character recognition is based on digital image processing, pattern recognition, computer vision and other technologies. It uses optical technology and computer technology to read the text sequence in the image and convert it into a format that can be accepted by computers and understood by humans. Character recognition is widely used in life, and its application scenarios include business card recognition, menu recognition, express delivery bill recognition, ID card recognition, business license recognition, bank card recognition, license plate recognition, road sign recognition, product packaging bag recognition, conference whiteboard recognition, advertising main words recognition, test paper recognition, receipt recognition, etc.

[0003] The traditional text recognition method generally includes the following steps: image preprocessing, text area positioning, text character segmentation, text recognition, text post-processing, etc. The process is cumbersome, and the effect of each step will affect the effect of the following steps; at the same time, the traditional method requires some complex preprocessing methods to ensure the text recognition effect under conditions such as uneven lighting and blurred images, and the amount of calculation is large. The text recognition process of the deep learning method still includes the steps of text area positioning and text recognition. The process is cumbersome, and two neural networks need to be trained to achieve the final recognition effect, and the amount of calculation is large. Summary of the invention

[0004] The technical problem to be solved by the embodiments of the present invention is to provide a text recognition method, apparatus, terminal device and storage medium, which realize end-to-end text recognition through a fully convolutional neural network, with a simple process, small amount of calculation and high accuracy.

[0005] In order to achieve the above object, an embodiment of the present invention provides a text recognition method, comprising:

[0006] Acquire a sample text data set, and preprocess each text image in the sample text data set; wherein the sample text data set includes text positions, positions of each character in the text, and character categories;

[0007] Generate a label image according to the preprocessed text image, wherein the label image includes a text area, a text boundary area, a character area, a character boundary area and a character category, and perform diffusion annotation on the text boundary area and the character boundary area;

[0008] Input the label image into the text recognition model for training, use the convolution layer to extract image features, use the pooling layer to downsample, use the upsampling layer or the deconvolution layer to restore the image resolution, and use the sigmoid layer to normalize the output probability in the last layer to output prediction images of multiple scales, and use the optimizer to optimize the loss function of the text recognition model to obtain a trained text recognition model;

[0009] Preprocessing the text image to be recognized and inputting it into a trained text recognition model, and the trained text recognition model outputs a clear scale prediction image;

[0010] The clear scale prediction image is parsed to obtain a text sequence of the to-be-recognized text image.

[0011] As an improvement of the above solution, the diffusion marking of the text boundary area and the character boundary area is specifically as follows:

[0012] Let m(x,y) be a boundary point. For any point p(x,y), the boundary point closest to point p is: The annotation formula is:

[0013]

[0014] Among them, T is the distance threshold, Vmax and Vmin represent the set empirical values, the label value of the pixel located at the center of the boundary is Vmax, and the label value of the pixel located around the boundary is between Vmin and Vmax.

[0015] As an improvement of the above scheme, the prediction maps of multiple scales include a clear scale prediction map and a fuzzy scale prediction map, wherein the clear scale prediction map includes a text area, a text boundary area, a character area, a character boundary area and a character category, and the fuzzy scale prediction map includes a text area, a text character area and a character category.

[0016] As an improvement of the above solution, the loss function of the text recognition model includes the loss of the text area, the loss of the character area and the loss of the character category, wherein:

[0017] The loss of the text area includes the loss of the text area and the loss of the text boundary area, namely:

[0018] L a =λ p L p +λ pb L pb

[0019] Among them, L a is the total loss of the text area, L pis the cross entropy loss of the text area, L pb is the cross entropy loss of the text boundary area, λ p and λ pb are the weights of the loss of the text area and the text boundary area respectively;

[0020] The loss of the character area includes the loss of the character area and the loss of the character boundary area, namely:

[0021] L b =λ ch L ch +λ chb L chb

[0022] Among them, L b is the total loss of the character area, L ch is the cross entropy loss of the character region, L chb is the cross entropy loss of the character boundary area, λ ch and λ chb are the weights of the character area and character boundary area losses respectively;

[0023] The loss of character class is: L c =L cls

[0024] Then the loss function of the text recognition model is L = λ a L a +λ b L b +λ c L c .

[0025] As an improvement of the above solution, the step of parsing the prediction image of the clear scale to obtain the text sequence of the text image to be recognized specifically includes:

[0026] The text frame is parsed according to the text area prediction map and the text boundary area prediction map to obtain the text area;

[0027] Analyze the character frame in the text area according to the character area prediction map and the character boundary prediction map to obtain the character area;

[0028] The character category with the highest probability in the character area is taken as the category of the pixel point, and the character category with the largest number of pixels is taken as the final character category of the character box;

[0029] The characters are connected according to their positions to obtain a character sequence of the character image to be recognized.

[0030] As an improvement of the above solution, the text frame is parsed according to the text area prediction map and the text boundary area prediction map to obtain the text area, which specifically includes:

[0031] The text frame is parsed according to the text area prediction map and the text boundary area prediction map, and the pixel points satisfying ω1p1+ω2p2>T are set to 1 to obtain the first binary map; wherein ω1 and ω2 are both set weights, p1∈[0,1] is the text area prediction probability, p2∈[0,1] is the text boundary area prediction probability, and T∈[0,1] is the set threshold;

[0032] Performing area connection on the pixel points with a pixel value of 1 in the first binary image to obtain a plurality of connection units, and selecting the minimum enclosing rectangular frame of the connection unit with the largest area as the text area.

[0033] As an improvement of the above solution, the character frame in the text area is parsed according to the character area prediction map and the character boundary prediction map to obtain the character area, which specifically includes:

[0034] The character frame in the text area is parsed according to the character area prediction map and the character boundary area prediction map, and the pixel points satisfying ω3p3+ω4p4>T are set to 1 to obtain a second binary map; wherein ω3 and ω4 are both set weights, p3∈[0,1] is the character area prediction probability, p4∈[0,1] is the character boundary area prediction probability, and T∈[0,1] is the set threshold;

[0035] The pixel points with a pixel value of 1 in the second binary image are connected in a region to obtain a plurality of connection units, and the minimum enclosing rectangular frames of the plurality of connection units that meet the requirements of the character region are selected as the character region.

[0036] The embodiment of the present invention further provides a text recognition device, comprising:

[0037] A sample text data set acquisition module is used to acquire a sample text data set and pre-process each text image in the sample text data set; wherein the sample text data set includes text positions, positions of each character in the text, and character categories;

[0038] A label image generation module, used to generate a label image according to the preprocessed text image, wherein the label image includes a text area, a text boundary area, a character area, a character boundary area and a character category, and diffusely annotates the text boundary area and the character boundary area;

[0039] A text recognition model training module is used to input the label image into the text recognition model for training, use a convolution layer to extract image features, use a pooling layer to downsample, use an upsampling layer or a deconvolution layer to restore the image resolution, and use a sigmoid layer to normalize the output probability in the last layer to output prediction images of multiple scales, and use an optimizer to optimize the loss function of the text recognition model to obtain a trained text recognition model;

[0040] A prediction image output module is used to pre-process the image of the text to be recognized and then input it into a trained text recognition model, and the trained text recognition model outputs a clear and scaled prediction image;

[0041] The text sequence output module is used to analyze the prediction image of the clear scale to obtain the text sequence of the text image to be recognized.

[0042] An embodiment of the present invention further provides a terminal device, comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements any of the above-mentioned text recognition methods when executing the computer program.

[0043] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute any one of the above-mentioned text recognition methods.

[0044] Compared with the prior art, the beneficial effects of a text recognition method, device, terminal device and storage medium provided by the embodiments of the present invention are: end-to-end text recognition is achieved through a fully convolutional neural network, with a simple process, small amount of calculation and high accuracy. In the training phase, text areas, character areas, character categories, text boundaries and character boundaries are trained together, and more contextual information can be combined to obtain better recognition effects. In the prediction phase, it is only necessary to input the image to be recognized into the network, and after the network outputs the prediction probability map, it can be parsed to obtain the text sequence. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a flow chart of a preferred embodiment of a text recognition method provided by the present invention;

[0046] Figure 2 is a schematic diagram of a network structure in a preferred embodiment of a text recognition method provided by the present invention;

[0047] Figure 3 It is a structural schematic diagram of a preferred embodiment of a text recognition device provided by the present invention;

[0048] Figure 4 It is a structural schematic diagram of a preferred embodiment of a terminal device provided by the present invention. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] See also Figure 1 , Figure 1 1 is a flow chart of a preferred embodiment of a text recognition method provided by the present invention. The text recognition method comprises:

[0051] S1, obtaining a sample text data set, and preprocessing each text image in the sample text data set; wherein the sample text data set includes text positions, positions of each character in the text, and character categories;

[0052] S2, generating a label image according to the preprocessed text image, wherein the label image includes a text area, a text boundary area, a character area, a character boundary area, and a character category, and performing diffusion annotation on the text boundary area and the character boundary area;

[0053] S3, inputting the label image into the text recognition model for training, using a convolution layer to extract image features, using a pooling layer to downsample, using an upsampling layer or a deconvolution layer to restore the image resolution, and using a sigmoid layer to normalize the output probability in the last layer to output prediction images of multiple scales, and using an optimizer to optimize the loss function of the text recognition model to obtain a trained text recognition model;

[0054] S4, pre-processing the text image to be recognized and inputting it into a trained text recognition model, and the trained text recognition model outputs a clear scale prediction image;

[0055] S5, parsing the clear scale prediction image to obtain a text sequence of the to-be-recognized text image.

[0056] Specifically, this embodiment obtains a sample text data set, which includes the text position (x0, y0, w0, h0), the position of each character in the text (x i ,y i ,w i ,h i) and character category, (x, y) is the upper left point of the text rectangle, w is the width of the rectangle, h is the height of the rectangle, i∈{1,2,...,N}, N is the number of characters in the text sequence. Preprocess each text image in the sample text dataset, including size normalization and pixel value standardization.

[0057] Size normalization specifically includes: scaling all text images in the sample text data set to a uniform size; scaling the text positions of the scaled text images and the positions of each character in the text, as follows:

[0058] x=xS w

[0059] y=yS h

[0060] w=wS w

[0061] h=hS h

[0062] Among them, S w , S h are the scaling factors in the horizontal and vertical directions respectively.

[0063] Image interpolation methods in the process of image scaling include: nearest neighbor method, bilinear interpolation, bicubic interpolation, etc.

[0064] Pixel value standardization: A color image has three channels, RGB, and the pixel value is v = [v r ,v g ,v b ],v r ∈[0,1],v b ∈[0,1],v g ∈[0,1], the mean of each channel is μ=[μ r ,μ g ,μ b ], the standard deviation is σ=[σ r ,σ g ,σ b ], then the standardized formula is:

[0065]

[0066]

[0067]

[0068] The mean and standard deviation of each channel can use the common values ​​of the ImageNet database, the mean of each channel is [0.485, 0.456, 0.406], and the standard deviation of each channel is [0.229, 0.224, 0.225]. In addition, other data sets can also be used to calculate the statistical mean and standard deviation.

[0069] Generate a label image based on the preprocessed text image, and the label image includes a text area, a text boundary area, a character area, a character boundary area and a character category. The text area is the inner area of ​​the marked bounding box, marked as 1, and the outer area (non-text area) is marked as 0; the character area is the inner area of ​​the marked bounding box, marked as 1, and the remaining non-character areas are marked as 0; the character category label is marked according to the number of character categories, and a label image represents the labeling result of a character category. The boundary area of ​​the text and the character is obtained from the marked position. In order to accelerate the convergence of training, the boundary area is diffusely labeled. The label image is input into the text recognition model for training, using FPN as the network structure, using the convolution layer to extract image features, using the pooling layer for downsampling, using the upsampling layer or the deconvolution layer to restore the image resolution, and the last layer uses the sigmoid layer to normalize the output probability to output prediction maps of multiple scales, and the optimizer is used to optimize the loss function of the text recognition model to obtain a trained text recognition model. The image of the text to be recognized is preprocessed, that is, the size is normalized and the pixel value is standardized, and then input into the trained text recognition model, which outputs a clear scale prediction map. The clear scale prediction map is analyzed to obtain the text sequence of the image of the text to be recognized.

[0070] This embodiment achieves end-to-end text recognition through a fully convolutional neural network, which has a simple process, small amount of computation and high accuracy. In the training phase, text areas, character areas, character categories, text boundaries and character boundaries are trained together, which can combine more contextual information to obtain better recognition results. In the prediction phase, it is only necessary to input the image to be recognized into the network. After the network outputs the predicted probability map, it can be parsed to obtain the text sequence.

[0071] In another preferred embodiment, the diffusion marking of the text boundary area and the character boundary area is specifically:

[0072] Let m(x,y) be a boundary point. For any point p(x,y), the boundary point closest to point p is: The annotation formula is:

[0073]

[0074] Among them, T is the distance threshold, Vmax and Vmin represent the set empirical values, the label value of the pixel located at the center of the boundary is Vmax, and the label value of the pixel located around the boundary is between Vmin and Vmax.

[0075] In another preferred embodiment, the prediction maps of the multiple scales include a clear scale prediction map and a fuzzy scale prediction map, wherein the clear scale prediction map includes a text area, a text boundary area, a character area, a character boundary area and a character category, and the fuzzy scale prediction map includes a text area, a text character area and a character category.

[0076] For details, please refer to Figure 2 , Figure 2 It is a schematic diagram of the network structure in a preferred embodiment of a text recognition method provided by the present invention. Input is the input image, C1, C2, C3 are feature maps obtained after convolution and downsampling, H3, H2, H1 are feature maps obtained after convolution and upsampling, and P3, P2, P1 are output prediction probability maps of different scales. The clear scale prediction map (P1) includes text areas, text boundary areas, character areas, character boundary areas and character categories; the fuzzy scale prediction maps (P2, P3) only predict text areas, character areas and character categories. The scales from P3 to P1 are from large to small, the image clarity is from fuzzy to clear, and the image resolution is from small to large. The larger the scale, the fuzzier the image, and the smaller the scale, the clearer the image.

[0077] In another preferred embodiment, the loss function of the text recognition model includes the loss of the text area, the loss of the character area and the loss of the character category, wherein:

[0078] The loss of the text area includes the loss of the text area and the loss of the text boundary area, namely:

[0079] L a =λ p L p +λ pb L pb

[0080] Among them, L a is the total loss of the text area, L p is the cross entropy loss of the text area, L pb is the cross entropy loss of the text boundary area, λ p and λ pb are the weights of the loss of the text area and the text boundary area respectively;

[0081] The loss of the character area includes the loss of the character area and the loss of the character boundary area, namely:

[0082] L b =λ ch L ch +λ chb L chb

[0083] Among them, L b is the total loss of the character area, L ch is the cross entropy loss of the character region, L chb is the cross entropy loss of the character boundary area, λ ch and λ chb are the weights of the character area and character boundary area losses respectively;

[0084] The loss of character class is: L c =L cls

[0085] Then the loss function of the text recognition model is L = λ a L a +λ b L b +λ c L c .

[0086] Specifically, this embodiment uses the Adam optimizer to optimize the loss function of the text recognition model. The loss function of the text recognition model includes the loss of the text area, the loss of the character area, and the loss of the character category. The cross entropy loss function is used:

[0087]

[0088] Among them, N is the number of voxels, K is the number of categories, and y i,k Indicates that voxel point i is the true label of the kth category, p i,k Indicates the predicted value of voxel point i for the kth class, w k represents the loss weight of the kth class.

[0089] The loss of the text area includes the loss of the text area and the loss of the text boundary area, namely:

[0090] L a =λ p L p +λ pb L pb

[0091] Among them, L a is the total loss of the text area, L p is the cross entropy loss of the text area, L pb is the cross entropy loss of the text boundary area, λ p and λ pbare the weights of the loss of the text area and the text boundary area respectively;

[0092] The loss of the character area includes the loss of the character area and the loss of the character boundary area, namely:

[0093] L b =λ ch L ch +λ chb L chb

[0094] Among them, L b is the total loss of the character area, L ch is the cross entropy loss of the character region, L chb is the cross entropy loss of the character boundary area, λ ch and λ chb are the weights of the character area and character boundary area losses respectively;

[0095] The loss of character class is: L c =L cls

[0096] Then the loss function of the text recognition model is L = λ a L a +λ b L b +λ c L c .

[0097] In another preferred embodiment, the step S5 of parsing the prediction image of the clear scale to obtain the text sequence of the text image to be recognized specifically includes:

[0098] S501, analyzing the text frame according to the text area prediction map and the text boundary area prediction map to obtain the text area;

[0099] S502, parsing the character frame in the text area according to the character area prediction map and the character boundary prediction map to obtain the character area;

[0100] S503, taking the character category with the highest probability in the character area as the category of the pixel point, and counting the character category with the largest number of pixels as the final character category of the character box;

[0101] S504: Connect characters according to their positions to obtain a character sequence of the character image to be recognized.

[0102] As a preferred solution, the S501, parsing the text frame according to the text area prediction map and the text boundary area prediction map to obtain the text area, specifically includes:

[0103] The text frame is parsed according to the text area prediction map and the text boundary area prediction map, and the pixel points satisfying ω1p1+ω2p2>T are set to 1 to obtain the first binary map; wherein ω1 and ω2 are both set weights, p1∈[0,1] is the text area prediction probability, p2∈[0,1] is the text boundary area prediction probability, and T∈[0,1] is the set threshold;

[0104] Performing area connection on the pixel points with a pixel value of 1 in the first binary image to obtain a plurality of connection units, and selecting the minimum enclosing rectangular frame of the connection unit with the largest area as the text area.

[0105] As a preferred solution, the S502, parsing the character frame in the text area according to the character area prediction map and the character boundary prediction map to obtain the character area, specifically includes:

[0106] The character frame in the text area is parsed according to the character area prediction map and the character boundary area prediction map, and the pixel points satisfying ω3p3+ω4p4>T are set to 1 to obtain a second binary map; wherein ω3 and ω4 are both set weights, p3∈[0,1] is the character area prediction probability, p4∈[0,1] is the character boundary area prediction probability, and T∈[0,1] is the set threshold;

[0107] The pixel points with a pixel value of 1 in the second binary image are connected in a region to obtain a plurality of connection units, and the minimum enclosing rectangular frames of the plurality of connection units that meet the requirements of the character region are selected as the character region.

[0108] Specifically, the text box is parsed according to the text area prediction map and the text boundary area prediction map, and the pixel points satisfying ω1p1+ω2p2>T are set to 1 to obtain the first binary map; wherein ω1 and ω2 are set weights and can be any value. Usually, ω1 can be set to a number in the range of [0,1], and ω2 can be set to a number in the range of [-1,0]. p1∈[0,1] is the text area prediction probability, p2∈[0,1] is the text boundary area prediction probability, and T∈[0,1] is the set threshold; Pixel points with a pixel value of 1 in the first binary map are connected by 4-neighborhood or 8-neighborhood to obtain multiple connection units, among which the connection unit with the largest area is the largest connection body. Since the largest connection body is irregular in shape, the smallest enclosing rectangular box that can enclose the largest connection body is selected as the text rectangular area.

[0109] According to the character area prediction map and the character boundary area prediction map, the character box in the text area is parsed, and the pixels that satisfy ω3p3+ω4p4>T are set to 1 to obtain a second binary map; wherein ω3 and ω4 are both set weights and can be arbitrary values. Usually, ω1 can be set to a number in the range of [0,1], and ω2 can be set to a number in the range of [-1,0]. p3∈[0,1] is the character area prediction probability, p4∈[0,1] is the character boundary area prediction probability, and T∈[0,1] is the set threshold; the pixel points with a pixel value of 1 in the second binary map are connected to the field to obtain multiple connection units, and the minimum enclosing rectangular boxes of multiple connected bodies that meet the character area requirements are selected as the character area. Rules for judging whether the character area requirements are met: judge whether the rectangular enclosing box is a character area based on whether the aspect ratio and area are within a certain range. For a rectangular enclosing box, the following formulas must be satisfied at the same time to be considered as a character area:

[0110]

[0111] area min <wh<area max

[0112] The character category with the highest probability in the character area is taken as the category of the pixel point, and the character category with the largest number of pixels is counted as the final character category of the character box. For the pixels on the prediction map with clear scale, these points can be predicted as multiple categories, such as width W, height H, and number of categories C. The dimension of the prediction map is W x H x C. Selecting the character category with the highest probability will output a map of size W x H, and the values ​​of the pixels on the map are 1 to C.

[0113] The characters are connected according to their positions to obtain a text sequence of the text image to be recognized. For example, for a single-line license plate, the characters are output from left to right according to the horizontal position of the character frame and connected to obtain a text sequence of the license plate number. For a double-line license plate, the row to which the character belongs is first determined according to whether the center of the character frame is located in the upper half or the lower half, and then the characters are connected from left to right for each row according to the horizontal position to obtain two lines of character strings as the text sequence of the license plate number.

[0114] In this embodiment, the network skeleton structure can be ResNet, DenseNet, MobileNet, etc., the loss function can use Dice loss, Focal Loss, etc., and the optimizer can use Adam, SGD, Adadelta, etc. When generating region labels, Gaussian heat maps can be used for region labels, and reduced region labels can be used. When generating boundary labels, diffusion features can use image expansion methods, etc. Before preprocessing the image, data enhancement can be used to improve generalization ability, including methods such as cropping, rotating, translating, scaling, adding noise, blurring, changing brightness, changing contrast, etc.

[0115] In the prediction stage, the present embodiment can improve the accuracy by combining the license plate prior information. For example, after obtaining the license plate character frame, it can be judged whether it is an ordinary license plate, a new energy license plate, a two-row license plate, etc. according to the number and position of the character frame. Then, the possible categories of the character frame at a fixed position are reduced, and it is only necessary to find the most suitable prediction category in the corresponding category. For example, the first character of an ordinary license plate is a province, the second character is a letter, and the following characters are numbers or letters. In the prediction stage, the license plate area can be extracted first, and then the character category can be predicted. After the license plate area is extracted, the character category probability is calculated and only the license plate area is inferred. The trained network parameters can be used, or another network can be used to identify the license plate, such as CRNN. ​​In the prediction stage, the character area can be extracted first, and then the character category can be predicted. After the character area is extracted, a neural network or a traditional machine learning classifier method for predicting a single character is used.

[0116] Correspondingly, the present invention also provides a text recognition device that can implement all the processes of the text recognition method in the above embodiment.

[0117] See also Figure 3 , Figure 3 1 is a schematic diagram of a preferred embodiment of a text recognition device provided by the present invention. The text recognition device comprises:

[0118] The sample text data set acquisition module 301 is used to acquire a sample text data set and pre-process each text image in the sample text data set; wherein the sample text data set includes the text position, the position of each character in the text, and the character category;

[0119] The label image generation module 302 is used to generate a label image according to the preprocessed text image, wherein the label image includes a text area, a text boundary area, a character area, a character boundary area and a character category, and diffusely annotate the text boundary area and the character boundary area;

[0120] The text recognition model training module 303 is used to input the label image into the text recognition model for training, use the convolution layer to extract image features, use the pooling layer to downsample, use the upsampling layer or the deconvolution layer to restore the image resolution, and use the sigmoid layer to normalize the output probability in the last layer to output prediction images of multiple scales, and use the optimizer to optimize the loss function of the text recognition model to obtain a trained text recognition model;

[0121] The prediction image output module 304 is used to pre-process the image of the text to be recognized and then input it into the trained text recognition model, and the trained text recognition model outputs a clear scale prediction image;

[0122] The text sequence output module 305 is used to analyze the prediction image of the clear scale to obtain the text sequence of the text image to be recognized.

[0123] Preferably, the diffusion marking of the text boundary area and the character boundary area is specifically:

[0124] Let m(x,y) be a boundary point. For any point p(x,y), the boundary point closest to point p is: The annotation formula is:

[0125]

[0126] Among them, T is the distance threshold, Vmax and Vmin represent the set empirical values, the label value of the pixel located at the center of the boundary is Vmax, and the label value of the pixel located around the boundary is between Vmin and Vmax.

[0127] Preferably, the prediction maps of the multiple scales include a clear scale prediction map and a fuzzy scale prediction map, wherein the clear scale prediction map includes a text area, a text boundary area, a character area, a character boundary area and a character category, and the fuzzy scale prediction map includes a text area, a text character area and a character category.

[0128] Preferably, the loss function of the text recognition model includes the loss of the text area, the loss of the character area and the loss of the character category, wherein:

[0129] The loss of the text area includes the loss of the text area and the loss of the text boundary area, namely:

[0130] L a =λ p L p +λ pb L pb

[0131] Among them, L ais the total loss of the text area, L p is the cross entropy loss of the text area, L pb is the cross entropy loss of the text boundary area, λ p and λ pb are the weights of the loss of the text area and the text boundary area respectively;

[0132] The loss of the character area includes the loss of the character area and the loss of the character boundary area, namely:

[0133] L b =λ ch L ch +λ chb L chb

[0134] Among them, L b is the total loss of the character area, L ch is the cross entropy loss of the character region, L chb is the cross entropy loss of the character boundary area, λ ch and λ chb are the weights of the character area and character boundary area losses respectively;

[0135] The loss of character class is: L c =L cls

[0136] Then the loss function of the text recognition model is L = λ a L a +λ b L b +λ c L c .

[0137] Preferably, the text sequence output module 305 is specifically used for:

[0138] The text frame is parsed according to the text area prediction map and the text boundary area prediction map to obtain the text area;

[0139] Analyze the character frame in the text area according to the character area prediction map and the character boundary prediction map to obtain the character area;

[0140] The character category with the highest probability in the character area is taken as the category of the pixel point, and the character category with the largest number of pixels is taken as the final character category of the character box;

[0141] The characters are connected according to their positions to obtain a character sequence of the character image to be recognized.

[0142] Preferably, the step of parsing the text frame according to the text area prediction map and the text boundary area prediction map to obtain the text area specifically includes:

[0143] The text frame is parsed according to the text area prediction map and the text boundary area prediction map, and the pixel points satisfying ω1p1+ω2p2>T are set to 1 to obtain the first binary map; wherein ω1 and ω2 are both set weights, p1∈[0,1] is the text area prediction probability, p2∈[0,1] is the text boundary area prediction probability, and T∈[0,1] is the set threshold;

[0144] Performing area connection on the pixel points with a pixel value of 1 in the first binary image to obtain a plurality of connection units, and selecting the minimum enclosing rectangular frame of the connection unit with the largest area as the text area.

[0145] Preferably, the character frame in the text area is parsed according to the character area prediction map and the character boundary prediction map to obtain the character area, which specifically includes:

[0146] The character frame in the text area is parsed according to the character area prediction map and the character boundary area prediction map, and the pixel points satisfying ω3p3+ω4p4>T are set to 1 to obtain a second binary map; wherein ω3 and ω4 are both set weights, p3∈[0,1] is the character area prediction probability, p4∈[0,1] is the character boundary area prediction probability, and T∈[0,1] is the set threshold;

[0147] The pixel points with a pixel value of 1 in the second binary image are connected in a region to obtain a plurality of connection units, and the minimum enclosing rectangular frames of the plurality of connection units that meet the requirements of the character region are selected as the character region.

[0148] In specific implementation, the working principle, control process and technical effect achieved by the text recognition device provided in the embodiment of the present invention are the same as those of the text recognition method in the above embodiment, and will not be repeated here.

[0149] See also Figure 4 , Figure 4 4 is a schematic diagram of a preferred embodiment of a terminal device provided by the present invention. The terminal device includes a processor 401, a memory 402, and a computer program stored in the memory 402 and configured to be executed by the processor 401. When the processor 401 executes the computer program, the text recognition method described in any of the above embodiments is implemented.

[0150] Preferably, the computer program can be divided into one or more modules / units (such as computer program 1, computer program 2, ...), and the one or more modules / units are stored in the memory 402 and executed by the processor 401 to complete the present invention. The one or more modules / units can be a series of computer program instruction segments that can complete specific functions, and the instruction segments are used to describe the execution process of the computer program in the terminal device.

[0151] The processor 401 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor, or the processor 401 can be any conventional processor. The processor 401 is the control center of the terminal device, and various parts of the terminal device are connected using various interfaces and lines.

[0152] The memory 402 mainly includes a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function, etc., and the data storage area can store related data, etc. In addition, the memory 402 can be a high-speed random access memory, or a non-volatile memory, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, and a flash card (Flash Card), etc., or the memory 402 can also be other volatile solid-state storage devices.

[0153] It should be noted that the above terminal device may include, but is not limited to, a processor and a memory. Those skilled in the art will understand that Figure 4 The structural diagram of the terminal device is only an example of the terminal device and does not constitute a limitation of the terminal device. It may include more or less components than those shown in the figure, or combine certain components, or different components.

[0154] An embodiment of the present invention further provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the text recognition method described in any of the above embodiments.

[0155] The embodiments of the present invention provide a method, apparatus, terminal device and storage medium for text recognition, which realizes end-to-end text recognition through a fully convolutional neural network, with a simple process, small amount of calculation and high accuracy. In the training phase, text areas, character areas, character categories, text boundaries and character boundaries are jointly trained, and more contextual information can be combined to obtain better recognition effects. In the prediction phase, it is only necessary to input the image to be recognized into the network, and after the network outputs the prediction probability map, the text sequence can be obtained by parsing.

[0156] It should be noted that the system embodiment described above is merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the system embodiment provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines. A person of ordinary skill in the art may understand and implement it without paying any creative effort.

[0157] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for character recognition, characterized in that: include: Acquire a sample text data set, and preprocess each text image in the sample text data set; wherein the sample text data set includes text positions, positions of each character in the text, and character categories; Generate a label image according to the preprocessed text image, wherein the label image includes a text area, a text boundary area, a character area, a character boundary area and a character category, and perform diffusion annotation on the text boundary area and the character boundary area; Input the label image into the text recognition model for training, use the convolution layer to extract image features, use the pooling layer to downsample, use the upsampling layer or the deconvolution layer to restore the image resolution, and use the sigmoid layer to normalize the output probability in the last layer to output prediction images of multiple scales, and use the optimizer to optimize the loss function of the text recognition model to obtain a trained text recognition model; The text image to be recognized is pre-processed and then input into a trained text recognition model, and the trained text recognition model outputs a clear scale prediction map; wherein the clear scale prediction map includes prediction maps of text area, text boundary area, character area, character boundary area and character category; Parsing the prediction image of the clear scale to obtain a text sequence of the text image to be recognized; The step of parsing the prediction image of the clear scale to obtain the text sequence of the text image to be recognized specifically includes: The text frame is parsed according to the text area prediction map and the text boundary area prediction map to obtain the text area; Analyze the character frame in the text area according to the character area prediction map and the character boundary prediction map to obtain the character area; The character category with the highest probability in the character area is taken as the category of the pixel point, and the character category with the largest number of pixels is taken as the final character category of the character box; The characters are connected according to their positions to obtain a character sequence of the character image to be recognized.

2. The text recognition method according to claim 1, characterized in that: The diffusion marking of the text boundary area and the character boundary area is specifically as follows: Let m(x,y) be a boundary point. For any point p(x,y), the boundary point closest to point p is: The annotation formula is: Among them, T is the distance threshold, Vmax and Vmin represent the set empirical values, the label value of the pixel located at the center of the boundary is Vmax, and the label value of the pixel located around the boundary is between Vmin and Vmax.

3. The text recognition method according to claim 1, wherein: The prediction maps of the multiple scales include a prediction map of a clear scale and a prediction map of a fuzzy scale, wherein the prediction map of the fuzzy scale includes prediction maps of a text area, a text character area, and a character category.

4. The text recognition method according to claim 1, wherein: The loss function of the text recognition model includes the loss of the text area, the loss of the character area and the loss of the character category, where: The loss of the text area includes the loss of the text area and the loss of the text boundary area, namely: L a =λ p L p +λ pb L pb Among them, L a is the total loss of the text area, L p is the cross entropy loss of the text area, L pb is the cross entropy loss of the text boundary area, λ p and λ pb are the weights of the loss of the text area and the text boundary area respectively; The loss of the character area includes the loss of the character area and the loss of the character boundary area, namely: L b =λ ch L ch +λ chb L chb Among them, L b is the total loss of the character area, L ch is the cross entropy loss of the character region, L chb is the cross entropy loss of the character boundary area, λ ch and λ chb are the weights of the character area and character boundary area losses respectively; The loss of character class is: L c =L cls Then the loss function of the text recognition model is L = λ a L a +λ b L b +λ c L c .

5. The text recognition method according to claim 3, characterized in that: The step of parsing the text frame according to the text area prediction map and the text boundary area prediction map to obtain the text area specifically includes: The text frame is parsed according to the text area prediction map and the text boundary area prediction map, and the pixel points satisfying ω1p1+ω2p2>T are set to 1 to obtain the first binary map; wherein ω1 and ω2 are both set weights, p1∈[0,1] is the text area prediction probability, p2∈[0,1] is the text boundary area prediction probability, and T∈[0,1] is the set threshold; Perform area connection on the pixel points with a pixel value of 1 in the first binary image to obtain multiple connection units, and select the minimum enclosing rectangular frame of the connection unit with the largest area as the text area.

6. The text recognition method according to claim 3, characterized in that: The character frame in the text area is parsed according to the character area prediction map and the character boundary prediction map to obtain the character area, specifically including: The character frame in the text area is parsed according to the character area prediction map and the character boundary area prediction map, and the pixel points satisfying ω3p3+ω4p4>T are set to 1 to obtain a second binary map; wherein ω3 and ω4 are both set weights, p3∈[0,1] is the character area prediction probability, p4∈[0,1] is the character boundary area prediction probability, and T∈[0,1] is the set threshold; Perform area connection on the pixel points with a pixel value of 1 in the second binary image to obtain multiple connection units, and select the minimum enclosing rectangular frames of multiple connection bodies that meet the requirements of the character area as the character area.

7. A text recognition device, characterized in that: include: A sample text data set acquisition module is used to acquire a sample text data set and pre-process each text image in the sample text data set; wherein the sample text data set includes the text position, the position of each character in the text, and the character category; A label image generation module, used to generate a label image according to the preprocessed text image, wherein the label image includes a text area, a text boundary area, a character area, a character boundary area and a character category, and diffusely annotates the text boundary area and the character boundary area; A text recognition model training module, which is used to input the label image into the text recognition model for training, use a convolution layer to extract image features, use a pooling layer to downsample, use an upsampling layer or a deconvolution layer to restore the image resolution, and use a sigmoid layer to normalize the output probability in the last layer to output prediction images of multiple scales, and use an optimizer to optimize the loss function of the text recognition model to obtain a trained text recognition model; A prediction map output module is used to pre-process the text image to be recognized and then input it into a trained text recognition model, and the trained text recognition model outputs a clear scale prediction map; wherein the clear scale prediction map includes prediction maps of text area, text boundary area, character area, character boundary area and character category; A text sequence output module, used for parsing the prediction image of the clear scale to obtain a text sequence of the text image to be recognized; Wherein, the text sequence output module is specifically used for: The text frame is parsed according to the text area prediction map and the text boundary area prediction map to obtain the text area; Analyze the character frame in the text area according to the character area prediction map and the character boundary prediction map to obtain the character area; The character category with the highest probability in the character area is taken as the category of the pixel point, and the character category with the largest number of pixels is taken as the final character category of the character box; The characters are connected according to their positions to obtain a character sequence of the character image to be recognized.

8. A terminal device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements the text recognition method according to any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the text recognition method according to any one of claims 1 to 6.

Images