Method, device, computer equipment and computer storage medium for recognizing text in images

Abstract

The present application discloses a method, device and computer storage medium for recognizing text in an image, and relates to the field of text recognition technology. By augmenting sample data collected from actual scenes, the trained model can fit the actual scenes well, and the accuracy of text recognition in images is improved. The method comprises: obtaining a sample image of text in a needle-printed font after scene processing; inputting the sample image of text in the needle-printed font as training data into network models of different architectures for training, and obtaining a text area detection model and a text recognition model; when receiving an image text detection request, inputting the image requested for detection into the text area recognition model to determine the position information of the text area corresponding to the image; inputting the position information of the text area corresponding to the image and the image requested for detection into the text recognition model together, and obtaining the text information in the image.

Description

Technical Field

[0001] The present invention relates to the technical field of text recognition, and in particular to a method, device, computer equipment and computer storage medium for recognizing text in an image. Background Art

[0002] At present, OCR recognition technology can well recognize text in images and is applied to various fields, such as document recognition and bill recognition, replacing manual input to a large extent. It greatly saves the trouble of manual input. However, a large amount of labeled data is an important part of the model training process in OCR recognition technology, which often requires high manpower, material resources and time costs.

[0003] Using algorithms to generate text sample data that simulates real scenes to augment the annotated data can, to a certain extent, reach the scale of annotated data required for model training. However, the text sample data generated by algorithms to simulate scenes are usually continuous strokes, which makes it difficult to cover text sample data in certain specific scenes. For example, text samples printed by a pin punch printer are composed of strokes composed of dots, which makes the text sample data in the model training process lack diversity, resulting in the trained model not being able to fit the actual scene well, affecting the accuracy of text recognition. Summary of the invention

[0004] In view of this, the present invention provides a method, apparatus, computer device and computer storage medium for recognizing text in an image. The main purpose is to solve the problem that the existing text recognition model lacks diversity in text samples during the training process, so that the trained model cannot fit the actual scene well, resulting in low text recognition accuracy.

[0005] According to one aspect of the present invention, a method for recognizing text in an image is provided, the method comprising:

[0006] Obtain a sample image of a text in a needle-printed font after scene processing;

[0007] Inputting the sample text images of the needle-printed font as training data into network models of different architectures for training, thereby obtaining a text region detection model and a text recognition model;

[0008] When receiving an image text detection request, inputting the image requested for detection into the text area recognition model to determine the position information of the text area corresponding to the image;

[0009] The position information of the text area corresponding to the image and the image requested to be detected are input into the text recognition model to obtain the text information in the image.

[0010] Furthermore, the obtaining of a sample image of a text in a needle-printed font after scene-based processing specifically includes:

[0011] Acquire a print sample image generated by printing, and set a property value corresponding to the print sample image;

[0012] By changing the attribute values ​​corresponding to the pixels in the print sample image, the print sample image is subjected to scene processing to obtain a text sample image similar to a needle-printed font.

[0013] Furthermore, the step of performing scene-based processing on the print sample image by changing the attribute values ​​corresponding to pixels in the print sample image to obtain a text sample image of a needle-printed font specifically includes:

[0014] Determine the optimal threshold value for dividing the color attribute values ​​corresponding to the pixels in the printed sample image by using the maximum inter-class variance method;

[0015] Performing binarization processing on the printed sample image based on the optimal threshold as a division basis to obtain background pixels and foreground pixels of the printed sample image after binarization processing;

[0016] Dividing the background pixels of the printed sample image after the binarization process into a plurality of background parts according to a preset ratio;

[0017] According to the pixel values ​​of the parameters corresponding to each background part, the printed sample image is subjected to scene processing to obtain a text sample image of a needle-printed font.

[0018] Furthermore, the method of using the maximum inter-class variance method to determine the optimal threshold for dividing the color attribute values ​​corresponding to pixels in the printed sample image specifically includes:

[0019] Using the assumed grayscale value, the color attribute values ​​corresponding to the pixels in the printed sample image are divided into two groups, and the inter-class variance is calculated, wherein the color attribute values ​​of one group are taken as the assumed grayscale value, and the color attribute values ​​of the other group are taken as not greater than the assumed grayscale value;

[0020] By changing the assumed grayscale value, the assumed grayscale value when the inter-class variance is maximum is determined as the optimal threshold value of the color attribute value.

[0021] Furthermore, the printing sample image is subjected to scene processing according to the pixel values ​​of the parameters corresponding to each background part to obtain a text sample image of a needle-printed font, which specifically includes:

[0022] By adjusting the pixel values ​​corresponding to the contrast of each background part, a sample image of a text similar to a needle-printed font after the contrast scene is increased is obtained, so that the sample image of the text covers scenes with different contrasts;

[0023] By blurring the pixel values ​​of the parameters corresponding to each background part, a sample text image similar to a printed font after adding a blur effect is obtained, so that the sample text image covers the scene of the blur effect.

[0024] Furthermore, the text sample images of the needle-printed font are input as training data into network models of different architectures for training to obtain a text area detection model and a text recognition model, which specifically includes:

[0025] Annotating the position information of the text area in the sample image of the needle-printed font and inputting it into the first network model for training to obtain a text area detection model;

[0026] The text information in the text area of ​​the sample image of the needle-printed font is annotated and then input into the second network model for training to obtain a text recognition model.

[0027] Furthermore, the first network model includes a multi-layer structure, and the position information of the text area in the sample image of the needle-printed font is annotated and input into the first network model for training to obtain a text area detection model, which specifically includes:

[0028] Extracting image region features corresponding to the sample image of the text in the needle-printed font through the convolutional layer of the first network model;

[0029] Generate horizontal text sequence features according to image region features corresponding to the text sample image through the decoding layer of the first network model;

[0030] The text area in the text sample image is determined according to the horizontal text sequence features through the prediction layer of the first network model, and the text area is processed to obtain candidate text lines.

[0031] According to another aspect of the present invention, a device for recognizing text in an image is provided, the device comprising:

[0032] An acquisition unit, used for acquiring a sample image of a text in a needle-printed font after scene processing;

[0033] A training unit, used for inputting the sample text images of the needle-printed font as training data into network models of different architectures for training, so as to obtain a text region detection model and a text recognition model;

[0034] A determination unit, configured to, when receiving an image text detection request, input the image requested for detection into the text region recognition model to determine position information of the text region corresponding to the image;

[0035] The recognition unit is used to input the position information of the text area corresponding to the image and the image requested to be detected into the text recognition model to obtain the text information in the image.

[0036] Furthermore, the acquisition unit includes:

[0037] A setting module, used for acquiring a print sample image generated by printing, and setting a property value corresponding to the print sample image;

[0038] The processing module is used to perform scene processing on the print sample image by changing the attribute values ​​corresponding to the pixels in the print sample image to obtain a text sample image of a needle-printed font.

[0039] Furthermore, the processing module includes:

[0040] A determination submodule, used for determining an optimal threshold for dividing the color attribute values ​​corresponding to pixels in the printed sample image by using a maximum inter-class variance method;

[0041] A first processing submodule is used to perform binarization processing on the printed sample image based on the optimal threshold as a division basis to obtain background pixels and foreground pixels of the printed sample image after the binarization processing;

[0042] A division submodule, used for dividing the background pixels of the printed sample image after the binarization process into a plurality of background parts according to a preset ratio;

[0043] The second processing submodule is used to perform scene processing on the print sample image according to the pixel value of the corresponding parameter of each background part to obtain a text sample image of a needle-printed font.

[0044] Furthermore, the determination submodule is specifically used to divide the color attribute values ​​corresponding to the pixels in the printed sample image into two groups using the assumed grayscale value, and calculate the inter-class variance, wherein one group of color attribute values ​​is taken as the assumed grayscale value, and the other group of color attribute values ​​is taken as not greater than the assumed grayscale value;

[0045] The determination submodule is further specifically configured to determine the assumed grayscale value at the maximum value of the inter-class variance as the optimal threshold value of the color attribute value by changing the assumed grayscale value.

[0046] Furthermore, the second processing submodule is specifically used to obtain a text sample image of a needle-printed font after increasing the contrast scene by adjusting the pixel value corresponding to the contrast of each background part, so that the text sample image covers scenes with different contrasts;

[0047] The second processing submodule is further used to perform blur processing on the pixel values ​​of the corresponding parameters of each background part to obtain a text sample image similar to a printed font after adding a blur effect, so that the text sample image covers the scene of the blur effect.

[0048] Furthermore, the training unit comprises:

[0049] A first training module is used to annotate the position information of the text area in the sample image of the needle-printed font and input it into the first network model for training to obtain a text area detection model;

[0050] The second training module is used to annotate the text information in the text area of ​​the text sample image of the needle-printed font and input it into the second network model for training to obtain a text recognition model.

[0051] Furthermore, the first network model includes a multi-layer structure.

[0052] The first training module is specifically used to extract image region features corresponding to the sample image of the text of the needle-printed font through the convolution layer of the first network model;

[0053] The first training module is further configured to generate horizontal text sequence features according to image region features corresponding to the text sample image through the decoding layer of the first network model;

[0054] The first training module is further specifically used to determine the text area in the text sample image according to the horizontal text sequence features through the prediction layer of the first network model, and process the text area to obtain candidate text lines.

[0055] According to another aspect of the present invention, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of a method for recognizing text in an image when executing the computer program.

[0056] According to another aspect of the present invention, a computer storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method for recognizing text in an image are implemented.

[0057] By means of the above technical scheme, the present invention provides a method and device for recognizing text in an image, by obtaining a sample image of a text similar to a printed font after scene processing, because the sample image of the text after scene processing is covered with richer picture features, the trained text area detection model and text recognition model have higher scene recognition capabilities, so that in the process of recognizing text in an image, the text information in different scene images can be recognized. Compared with the method for recognizing text in an image in the prior art, the present application simplifies the sample collection process and saves the labeling time of the sample data by augmenting the sample data collected from the actual scene, without consuming a lot of manpower costs to collect samples, and the model trained by using the augmented sample data can fit the actual scene well, thereby improving the accuracy of text recognition in the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0059] Figure 1 A schematic diagram showing a flow chart of a method for recognizing text in an image provided by an embodiment of the present invention;

[0060] Figure 2 A schematic diagram showing a flow chart of another method for recognizing text in an image provided by an embodiment of the present invention;

[0061] Figure 3 A schematic diagram showing the structure of a device for recognizing text in an image provided by an embodiment of the present invention is shown;

[0062] Figure 4 A schematic structural diagram of another apparatus for recognizing text in an image provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0063] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0064] The embodiment of the present invention provides a method for recognizing text in an image, which can make the trained model fit the actual scene well and improve the accuracy of text recognition in the image. Figure 1 As shown, the method includes:

[0065] 101. Obtain a sample image of a text in a needle-printed font after scene processing.

[0066] The text sample image of the print-like font may be a text sample image generated by printing, such as an invoice image, a document image, and the like.

[0067] Specifically, in the process of obtaining sample images of printed-like text, conventional invoice images are usually selected as sample images of text in printed-like fonts. In order to enrich the diversity of the sample text images, different shooting devices can be used for conventional invoice images to shoot the invoice images in different shooting backgrounds, lights, brightness, shooting angles, etc., thereby generating text sample images that combine different changes in background, light, brightness, etc., so that the text sample images can be combined with actual application scenarios in the subsequent training process.

[0068] It can be understood that by adjusting the shooting scene in the captured bill image, a certain scene processing can be added to the text sample image. It is also possible to select text sample images with different background colors, perform grayscale processing on the background color of the text sample image, and then adjust the contrast between the foreground and background of the text sample image so that the text sample image covers scenes with different contrasts. It is also possible to blur the background part of the image after grayscale processing so that the text sample image covers the scene with a blurred effect. Of course, adding noise or reducing the scale and other processing are not limited to the scene processing method here.

[0069] 102. Input the sample text images of the needle-printed font as training data into network models of different architectures for training, so as to obtain a text region detection model and a text recognition model.

[0070] Among them, the network model used to train the text region detection model can use the open source Detecting Text in Natural Image with Connectionist Text Proposal Network (CTPN) framework. The specific process of training the text box detection model can be as follows: first prepare the training data, that is, the sample image of the printed font and the annotation data corresponding to the sample image of the text. Since the annotation data records the corresponding coordinate information of the text region in the image, before the training data is input into the CTPN network, the corresponding coordinate information of the text region in the annotation data needs to be converted into a small anchor with a width of 8. By splitting the text region into a set of small text regions, the information in each small text region is predicted and identified, which can greatly improve the accuracy of text region detection. The CTPN network structure here adopts the form of CNN+BLSTM+RPN. CNN is used to extract the spatial features of the receptive field. The receptive field is the area of ​​the input image corresponding to the response of a node in the output feature map (convolved by the convolution kernel). BLSTM can generate horizontal text sequence features based on the spatial features of the receptive field. RPN consists of two parts, anchor classification and bounding box regressing. After the above anchor classification, it can be determined whether each area is a text area. After bounding box regressing processing, a set of vertical strip candidate text lines will be obtained.

[0071] It should be noted that the output of the pre-trained text region detection model is not directly the text region in the target recognition image, but a group of vertical strip candidate text lines that constitute the text region in the target recognition image. A text line construction algorithm can be used to connect groups of vertical strip candidate text lines into a text region, thereby determining the text region in the target recognition image and the location information of the text region.

[0072] Among them, the network model used to train the text recognition model can use the algorithm of An End-to-End Trainable Neural Network for Imaged-based Sequence Recognition and Its Application to Scene Text Recognition (CRNN) to train the recognition model. After the text sample image of the needle-printed font and the position information of the text area marked in the text sample image pass through the text recognition model, the text recognition result corresponding to each text area in the text sample image of the needle-printed font is output. The specific process of training the CRNN model can be as follows: First, the training data is stored in a labeled manner using the text sample image of the needle-printed font and the text information of the text area in the text sample image. The CRNN network structure here adopts the form of CNN+RNN+CTC. CNN is used to extract the spatial features of the receptive field in the image. RNN can predict the label distribution of each frame in the image based on the spatial features of the receptive field. CTC can integrate the label distribution of each frame into the final label sequence. For example, the input image is resized to a size of W*32, and the predicted value output by the text recognition model represents the text information corresponding to the text area in the target recognition image.

[0073] It should be noted that the training data used here to train the text area detection model and the text recognition model has rich printed image features, so that the trained text area detection model and the text recognition model can more comprehensively cover the application scenarios of printed font images, and improve the detection effect of text areas in images and the recognition effect of text information in text areas.

[0074] 103. When an image text detection request is received, the image requested for detection is input into the text region recognition model to determine the position information of the text region corresponding to the image.

[0075] It can be understood that each image will have a corresponding output file after passing through the text area detection model. The output file stores the location information of all candidate text boxes in the image and the label of whether the candidate text line is a text area. The candidate text box here is equivalent to the vertical strip box split from the text area.

[0076] In the process of specifically determining the position information of the text area corresponding to the image, a series of candidate text boxes output by the text area detection model can be recorded as text proposals. In the process of generating the position information of the text area corresponding to the image from the candidate text line corresponding to the image based on the text line construction algorithm, whether the candidate text box is a label of the text area is taken into consideration. Therefore, according to whether the candidate text box is a label of the text area, a series of text proposals are connected into a large text area to form the text area corresponding to the image, thereby determining the position information of the text area corresponding to the image.

[0077] 104. Input the position information of the text area corresponding to the image and the image requested to be detected into the text recognition model to obtain the text information in the image.

[0078] It can be understood that the trained text recognition model has the ability to recognize text information in the text area. In the process of training the text recognition model, text sample images of needle-printed fonts and labels of text information in the text area in the text sample images are used. Through forward propagation and reverse bias correction, the parameters of the text recognition model are continuously adjusted, so that the image through the text recognition model can accurately recognize the text information in the text area in the printed font image.

[0079] The embodiment of the present invention provides a method for recognizing text in an image. By obtaining a sample text image of a pin-printed font after scene processing, the sample text image after scene processing is covered with richer picture features, so that the trained text region detection model and text recognition model have higher scene recognition capabilities, so that in the process of recognizing text in the image, the text information in different scene images can be recognized. Compared with the method for recognizing text in an image in the prior art, the present application simplifies the sample collection process and saves the labeling time of the sample data by augmenting the sample data collected from the actual scene, without consuming a lot of manpower costs to collect samples, and the model trained by using the augmented sample data can fit the actual scene well, thereby improving the accuracy of text recognition in the image.

[0080] The embodiment of the present invention provides another method for recognizing text in an image, so that the trained model can well fit the actual scene and improve the accuracy of text recognition in the image. Figure 2 As shown, the method includes:

[0081] 201. Obtain a print sample image generated by printing, and set an attribute value corresponding to the print sample image.

[0082] Normally, when a printed text image generated by printing is used to identify text information through a text recognition model, since the text information in the printed text image is strokes composed of dot matrices, and the text images generated by subsequent text generation algorithms are all continuous strokes, this type of printed text will not get a good recognition effect in the text recognition model.

[0083] In order to better recognize text information in printed text images, printed sample images generated by printing can be obtained as training data for a text recognition model, and attribute values ​​corresponding to the printed sample images can be set to enrich the diversity of the training data.

[0084] Specifically setting the attribute values ​​corresponding to the printed sample image can randomly select a background image of the printed sample image, calculate and record the color mean Bcolor of the background image, randomly select the color of the text information in the printed sample image, record the color of the text information as Tcolor, randomly select the text font size spacing in the printed sample image, etc. By setting the attribute values ​​corresponding to the printed sample image, the printed sample image can be more in line with the printed text image in the actual application scenario, so that the generated printed sample image has a better training effect in the subsequent model training process.

[0085] For example, for the font selection in the print sample image, it can be set according to the print image, usually using Songti, Fangsong, Gothic fonts, etc., and when generating each print sample image, a print sample image with a font can be randomly selected from these fonts; the same is true for the font color and rotation angle of the print sample image, and the random value in a certain range is randomly adjusted to set the attribute value corresponding to the print image sample.

[0086] 202. Perform scene processing on the print sample image by changing attribute values ​​corresponding to pixels in the print sample image to obtain a text sample image of a needle-printed font.

[0087] For the embodiments of the present invention, the attribute values ​​corresponding to the pixels in the printed sample image are specifically changed. The maximum inter-class variance method can be used to determine the optimal threshold for dividing the color attribute values ​​corresponding to the pixels in the printed sample image, and the printed sample image is binarized based on the optimal threshold as the basis for division to obtain the background pixels and foreground pixels of the printed sample image after the binarization. Since the printed sample image after the binarization can divide the pixels in the image into two categories, one is the background pixels and the other is the foreground pixels, it is convenient to perform scene processing on the printed sample image.

[0088] Specifically, the assumed grayscale value can be used to divide the color attribute values ​​corresponding to the pixels in the printed sample image into two groups, and the inter-class variance can be calculated, where one group of color attribute values ​​takes the assumed grayscale value, and the other group of color attribute values ​​takes the value not greater than the assumed grayscale value. By changing the assumed grayscale value, the maximum value of the inter-class variance is determined as the optimal threshold of the color attribute value.

[0089] It should be noted that the above method uses the maximum between-class variance method to determine the optimal threshold value for the color attribute value corresponding to the pixel in the divided sample print image. Black and white can also be directly selected as the color attribute value corresponding to the pixel in the divided sample print image. For example, the print sample image is binarized to obtain binary mask images of the image foreground (white) and the image background (black), respectively.

[0090] In order to change the background color of the printed image, after binarizing the printed sample image, Tcolor is the optimal threshold of the color attribute, and the pixels on the mask are traversed row by row from the upper left corner, and the total number of each continuous white pixel N is recorded. Assuming that the pixel length requiring a breakpoint is M, the current traversed pixel value is set to P, and the corresponding area on the printed sample image where P / (2*M)>M is set as the background Bcolor.

[0091] For the embodiments of the present invention, the printed sample image is specifically subjected to scene processing to obtain a sample image of text in a needle-printed font. The background pixels of the printed sample image after binarization processing are divided into multiple background parts according to a preset ratio, and the printed sample image is subjected to scene processing for the pixel values ​​of the corresponding parameters of each background part to obtain a sample image of text in a needle-printed font.

[0092] For example, the M pixels on the original image that are changed into the background color may be divided into three parts according to 1:4:1, and the set colors of the three parts are (Bcolor+Tcolor) / 2, Bcolor, and (Bcolor+Tcolor) / 2 respectively.

[0093] In order to change the contrast of the printed sample image, the pixel values ​​corresponding to the contrast of each background part can be adjusted to obtain a text sample image similar to the printed font after the contrast scene is increased, so that the text sample image covers scenes with different contrasts;

[0094] The specific formula for adjusting the contrast of the printed sample image is:

[0095] g(x)=alpha*f(x)+beta

[0096] Among them, alpha: a random value between 0.2 and 0.8, beta: a random value between 0.2 and 0.8;

[0097] By multiplying the print sample image by alpha and then adding beta, a print sample image with different contrasts can be obtained, so that the generated print sample image can cover more contrast scenes.

[0098] In order to change the blur effect of the printed sample image, the pixel values ​​of the corresponding parameters of each background part are blurred to obtain a text sample image similar to the printed font after adding the blur effect, so that the text sample image covers the scene of the blur effect.

[0099] The specific method of adjusting the blur of the printed sample image may include motion blur and Gaussian blur. To increase the motion blur effect of the printed sample image, the printed image sample may be subjected to the following operations:

[0100] First determine the transformation matrix:

[0101] M=cv2.getRotationMatrix2D(center, angle, scale)

[0102]

[0103] in,

[0104] α=scale·cosangle

[0105] β=scale·sinangle

[0106] Then the motion blurred convolution kernel is obtained through affine transformation:

[0107] cv2.warpAffine(src,dst,M,dsize)

[0108] kernel(x,y)=src(M11*x+M12*y+M13,M21*x+M22*y+M23)

[0109] Then through the convolution operation;

[0110] cv2.filter2D(src, dst, ddepth, kernel, anchor=(-1,-1))

[0111] dst(x,y)=Σkernel(x′,y′)*src(x+x′-anchor.x,y+y′-anchor.y)

[0112] Finally, after the convolution operation of the printed sample image and the Gaussian kernel, the motion blur effect can be increased, wherein the blur angle and blur degree of the motion blur can be randomly generated to ensure the diversity of the printed sample images.

[0113] To increase the Gaussian blur effect of the printed sample image, the printed image sample can be subjected to the following operations:

[0114] First, determine the Gaussian kernel by the following formula:

[0115] Cv2.getGaussianKernel(ksize,sigma)

[0116]

[0117] Where i = 0..ksize-1, α is a scaling factor such that ∑Gi = 1

[0118] Finally, after the convolution operation between the printed sample image and the Gaussian kernel, the Gaussian blur effect can be increased.

[0119] 203. Annotate the position information of the text area in the sample image of the needle-printed font and input the information into the first network model for training to obtain a text area detection model.

[0120] In order to facilitate the clear boundary definition of the text area, there may be different areas in the text sample image of the needle-printed font, such as text area, picture area, blank area, etc. The non-text area is not the target area for text area detection, so the text area needs to be marked. Specifically, the image annotation tool can be used to process the text sample image of the needle-printed font, so as to accurately mark the text area in the text sample image and obtain the annotation data of the text area.

[0121] Among them, the first network model can adopt the CTPN network framework, which includes a three-layer structure. The first layer is a convolutional structure, that is, a CNN structure. The convolutional layer extracts the image area features corresponding to the text sample image, and can learn the spatial information of the receptive field; the second layer is a decoding layer, that is, a BLSTM structure. The decoding layer generates horizontal text sequence features according to the image area features corresponding to the text sample image, which can better cope with the sequence features of horizontal text; the third layer is a prediction layer, that is, an RPN structure. The prediction layer determines the text area in the text sample image according to the horizontal text sequence features, and processes the text area to obtain candidate text lines.

[0122] Specifically, the prediction layer of the first network model includes a classification part and a regression part. In the process of determining the text area in the text sample image according to the horizontal text sequence features through the prediction layer of the network model, and processing the text area to obtain candidate text lines, the classification part of the prediction layer of the network model can be used to classify each area in the text sample image according to the horizontal text sequence features to determine the text area in the text sample image; the regression part of the prediction layer of the network model can be used to perform border regression processing on the text area in the text sample image to obtain candidate text lines.

[0123] In the specific implementation process, in the convolutional layer, CTPN can select the feature maps of conv5 in the VGG model as the final features of the image. At this time, the size of the feature maps is H×W×C; then due to the sequence relationship between texts, a 3×3 sliding window can be used in the decoding layer to extract the 3×3 area around each point on the feature maps as the feature vector representation of the point. At this time, the size of the image becomes H×W×9C, and then each row is used as the length of the sequence and the height as the batch_size, and passed into a 128-dimensional Bi-LSTM, and the output of the decoding layer is W×H×256; finally, the output of the decoding layer is connected to the prediction layer. The prediction layer includes two parts, anchor classification and bounding box regressing. After anchor classification, it can be determined whether each area in the image is a text area. After bounding box regressing, a set of vertical strip candidate text lines will be obtained, and carry the label of whether the candidate text line is a text area.

[0124] Furthermore, in order to ensure the accuracy of the prediction of the trained text region detection model, the pre-set loss function can adjust the parameters of the multi-layer structure in the text region detection model based on the deviation between the result output by the text region detection model and the data annotated by the real text region. For the embodiment of the present invention, the pre-trained loss function mainly includes three parts, the first part is a loss function for detecting whether the anchor is a text region; the second part is a loss function for detecting the y-coordinate offset regression of the anchor; and the third part is a loss function for detecting the x-coordinate offset regression of the anchor.

[0125] 204. Label the text information in the text area of ​​the sample image of the needle-printed font and input the labeled text information into the second network model for training to obtain a text recognition model.

[0126] Among them, the second network model can adopt the CRNN network architecture, which includes a three-layer structure. The first layer is a convolutional structure, that is, a CNN structure. The convolutional layer extracts the image area features corresponding to the text sample image, and can learn the spatial information of the receptive field; the second layer is a recurrent layer, that is, an RNN structure. The recurrent layer predicts the label distribution of each frame in the image according to the image area features corresponding to the text sample image; the third layer is a transcription layer, that is, a CTC structure. The transcription layer integrates the label distribution of each frame in the image and other operations into the final label sequence, and outputs the text recognition result corresponding to each text area in the text sample image.

[0127] In the specific implementation process, in the convolution layer part, the feature sequence of the input text sample image can be automatically extracted. The vectors in the extracted feature sequence are generated in order from left to right on the feature map, and each feature vector represents a feature of a certain width on the image. In the recurrent layer part, the RNN recurrent neural network can be used to predict the label distribution (probability list of the real result) of each feature vector in the feature sequence. The error of the recurrent layer is back-propagated, and finally converted into a feature sequence, and then the feature sequence is fed back to the convolution layer. This can be completed by defining a custom inner layer as a bridge between the convolution layer and the recurrent layer. In the transcription layer part, the CTC model can be used to integrate all possible results of the predicted feature sequence and convert them into the final result. CTC is generally connected to the last layer of the RNN network for sequence learning and training. For a sequence of length T, each sample point t (t is much greater than T) will output a softmax vector in the last layer of the RNN network, indicating the predicted probability of the sample point. After these probabilities of all sample points are transmitted to the CTC model, the most likely label is output, and then the final sequence label can be obtained after removing blanks and deduplication operations.

[0128] 205. When an image text detection request is received, the image requested for detection is input into the text region recognition model to determine the position information of the text region corresponding to the image.

[0129] It can be understood that each printed sample image will have a corresponding output file after passing through the text area detection model. The output file stores the position information of all candidate text lines in the image and the label of whether the candidate text line is a text area. The candidate text line here is equivalent to the vertical strip line split from the text area. Based on the text line construction algorithm, the candidate text lines are connected to form the text area in the image. Combined with the position information of each candidate text line, the position information of the corresponding text area of ​​the image is determined.

[0130] 206 . Input the position information of the text area corresponding to the image and the image requested to be detected into the text recognition model to obtain the text information in the image.

[0131] Further, as Figure 1 In a specific implementation of the method, an embodiment of the present invention provides a device for recognizing text in an image, such as Figure 3 As shown, the device includes: an acquisition unit 31, a training unit 32, a determination unit 33, and an identification unit 34.

[0132] The acquisition unit 31 may be used to acquire a sample image of a text in a needle-printed font after scene processing;

[0133] The training unit 32 can be used to input the text sample images of the needle-printed font as training data into network models of different architectures for training, so as to obtain a text area detection model and a text recognition model;

[0134] The determination unit 33 may be configured to, when receiving an image text detection request, input the image requested for detection into the text region recognition model to determine the position information of the text region corresponding to the image;

[0135] The recognition unit 34 may be used to input the position information of the text area corresponding to the image and the image requested to be detected into the text recognition model to obtain the text information in the image.

[0136] The embodiment of the present invention provides a device for recognizing text in an image. By obtaining a sample image of a text similar to a printed font after scene processing, the sample image of the text after scene processing is covered with richer picture features, so that the trained text area detection model and text recognition model have higher scene recognition capabilities, so that in the process of recognizing text in an image, the text information in different scene images can be recognized. Compared with the method for recognizing text in an image in the prior art, the present application simplifies the sample collection process and saves the labeling time of the sample data by augmenting the sample data collected from the actual scene, without consuming a lot of manpower costs to collect samples, and the model trained by using the augmented sample data can well fit the actual scene, thereby improving the accuracy of text recognition in the image.

[0137] As Figure 3 Further description of the apparatus for recognizing text in an image shown in, Figure 4 FIG. 1 is a schematic diagram of the structure of another apparatus for recognizing text in an image according to an embodiment of the present invention. Figure 4 As shown, the acquisition unit 31 includes:

[0138] The setting module 311 may be used to obtain a print sample image generated by printing, and set a property value corresponding to the print sample image;

[0139] The processing module 312 may be used to perform scene-based processing on the print sample image by changing the attribute values ​​corresponding to the pixels in the print sample image to obtain a text sample image of a needle-printed font.

[0140] Furthermore, the processing module 312 includes:

[0141] The determination submodule 3121 may be used to determine the best threshold for dividing the color attribute values ​​corresponding to the pixels in the printed sample image by using the maximum inter-class variance method;

[0142] The first processing submodule 3122 may be used to perform binarization processing on the print sample image using the optimal threshold as a basis for division, to obtain background pixels and foreground pixels of the print sample image after binarization processing;

[0143] The division submodule 3123 may be used to divide the background pixels of the printed sample image after the binarization process into a plurality of background parts according to a preset ratio;

[0144] The second processing submodule 3124 can be used to perform scene processing on the print sample image according to the pixel value of the corresponding parameter of each background part to obtain a text sample image of a needle-printed font.

[0145] Furthermore, the determination submodule 3121 may be specifically used to divide the color attribute values ​​corresponding to the pixels in the printed sample image into two groups using the assumed grayscale value, and calculate the inter-class variance, wherein the color attribute values ​​of one group are taken as the assumed grayscale value, and the color attribute values ​​of the other group are taken as not greater than the assumed grayscale value;

[0146] The determination submodule 3121 may be specifically configured to determine the assumed grayscale value at the maximum inter-class variance as the optimal threshold value of the color attribute value by changing the assumed grayscale value.

[0147] Furthermore, the second processing submodule 3124 may be specifically configured to obtain a text sample image of a needle-printed font after the contrast scene is increased by adjusting the pixel value corresponding to the contrast of each background part, so that the text sample image covers scenes with different contrasts;

[0148] The second processing submodule 3124 can also be used to perform blur processing on the pixel values ​​of the corresponding parameters of each background part to obtain a text sample image similar to a printed font after adding a blur effect, so that the text sample image covers the scene of the blur effect.

[0149] Furthermore, the training unit 33 includes:

[0150] The first training module 321 can be used to annotate the position information of the text area in the sample image of the needle-printed font and input it into the first network model for training to obtain a text area detection model;

[0151] The second training module 322 can be used to annotate the text information in the text area of ​​the sample image of the needle-printed font and input it into the second network model for training to obtain a text recognition model.

[0152] Furthermore, the first network model includes a multi-layer structure.

[0153] The first training module 321 may be specifically used to extract image region features corresponding to the sample image of the needle-printed font through the convolution layer of the first network model;

[0154] The first training module 321 may be specifically configured to generate horizontal text sequence features according to image region features corresponding to the text sample image through the decoding layer of the first network model;

[0155] The first training module 321 may be specifically configured to determine the text region in the text sample image according to the horizontal text sequence features through the prediction layer of the first network model, and process the text region to obtain candidate text lines.

[0156] It should be noted that for other corresponding descriptions of the functional units involved in the apparatus for recognizing text in an image provided in this embodiment, reference can be made to Figure 1 , Figure 2 The corresponding description in will not be repeated here.

[0157] Based on the above Figure 1 , Figure 2 The method shown in the embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, the above-mentioned Figure 1 , Figure 2 Method for recognizing text in the image shown.

[0158] Based on this understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each implementation scenario of the present application.

[0159] Based on the above Figure 1 , Figure 2 The method shown, and Figure 3 , Figure 4 In order to achieve the above-mentioned purpose, the embodiment of the present application also provides a computer device, which can be a personal computer, a server, a network device, etc. The physical device includes a storage medium and a processor; the storage medium is used to store a computer program; the processor is used to execute the computer program to achieve the above-mentioned Figure 1 , Figure 2 Method for recognizing text in the image shown.

[0160] Optionally, the computer device may further include a user interface, a network interface, a camera, a radio frequency (RF) circuit, a sensor, an audio circuit, a WI-FI module, etc. The user interface may include a display, an input unit such as a keyboard, etc., and the optional user interface may also include a USB interface, a card reader interface, etc. The network interface may optionally include a standard wired interface, a wireless interface (such as a Bluetooth interface, a WI-FI interface), etc.

[0161] Those skilled in the art will appreciate that the physical device structure of the apparatus for recognizing text in an image provided in this embodiment does not constitute a limitation on the physical device, and may include more or fewer components, or a combination of certain components, or different arrangements of components.

[0162] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages the hardware and software resources of the above-mentioned computer device, and supports the operation of information processing programs and other software and / or programs. The network communication module is used to realize communication between the components inside the storage medium, and communication with other hardware and software in the physical device.

[0163] Through the description of the above implementation methods, those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform, or by hardware. By applying the technical solution of the present application, compared with the current prior art, the present application augments the sample data collected from the actual scene, without consuming a lot of manpower costs to collect samples, thus simplifying the sample collection process and saving the labeling time of the sample data. The model trained using the augmented sample data can fit the actual scene well, thus improving the accuracy of text recognition in the image.

[0164] Those skilled in the art will appreciate that the accompanying drawings are only schematic diagrams of a preferred implementation scenario, and the modules or processes in the accompanying drawings are not necessarily necessary for implementing the present application. Those skilled in the art will appreciate that the modules in the devices in the implementation scenario can be distributed in the devices of the implementation scenario according to the description of the implementation scenario, or can be changed accordingly and located in one or more devices different from the present implementation scenario. The modules of the above-mentioned implementation scenario can be combined into one module, or can be further split into multiple submodules.

[0165] The above serial numbers of this application are only for description and do not represent the advantages and disadvantages of the implementation scenarios. The above disclosure is only a few specific implementation scenarios of this application, but this application is not limited to them, and any changes that can be thought of by technicians in this field should fall within the scope of protection of this application.

Claims

1. A method for recognizing text in an image, characterized in that: The method comprises: A text sample image of a needle-printed font is obtained after scene processing, wherein the scene processing is implemented by the following scene processing steps: using the maximum inter-class variance method to determine the optimal threshold for dividing the color attribute values ​​corresponding to pixels in the print sample image; binarizing the print sample image with the optimal threshold as the basis for division to obtain background pixels and foreground pixels of the print sample image after binarization; dividing the background pixels of the print sample image after binarization into a plurality of background parts according to a preset ratio; and scene processing the print sample image according to the pixel value of the corresponding parameter of each background part to obtain a text sample image of a needle-printed font; Inputting the sample text images of the needle-printed font as training data into network models of different architectures for training, thereby obtaining a text region detection model and a text recognition model; When receiving an image text detection request, inputting the image requested for detection into the text area recognition model to determine the position information of the text area corresponding to the image; The position information of the text area corresponding to the image and the image requested for detection are input into the text recognition model to obtain the text information in the image.

2. The method according to claim 1, characterized in that: The obtaining of a sample image of a text in a needle-printed font after scene-based processing specifically includes: Acquire a print sample image generated by printing, and set a property value corresponding to the print sample image; By changing the attribute values ​​corresponding to the pixels in the print sample image, the print sample image is subjected to scene processing to obtain a text sample image similar to a needle-printed font.

3. The method according to claim 1, characterized in that The method of using the maximum inter-class variance method to determine the optimal threshold for dividing the color attribute values ​​corresponding to pixels in the printed sample image specifically includes: Using the assumed grayscale value, the color attribute values ​​corresponding to the pixels in the printed sample image are divided into two groups, and the inter-class variance is calculated, wherein the color attribute values ​​of one group are taken as the assumed grayscale value, and the color attribute values ​​of the other group are taken as not greater than the assumed grayscale value; By changing the assumed grayscale value, the assumed grayscale value when the inter-class variance is maximum is determined as the optimal threshold value of the color attribute value.

4. The method according to claim 1, characterized in that: The scene processing is performed on the printed sample image according to the pixel value of the corresponding parameter of each background part to obtain a text sample image of a needle-printed font, specifically including: By adjusting the pixel values ​​corresponding to the contrast of each background part, a sample image of a text similar to a needle-printed font after the contrast scene is increased is obtained, so that the sample image of the text covers scenes with different contrasts; By blurring the pixel values ​​of the parameters corresponding to each background part, a text sample image similar to a printed font after adding a blur effect is obtained, so that the text sample image covers the scene of the blur effect.

5. The method according to any one of claims 1 to 4, characterized in that The text sample images of the needle-printed font are input as training data into network models of different architectures for training to obtain a text area detection model and a text recognition model, specifically including: Annotating the position information of the text area in the sample image of the needle-printed font and inputting it into the first network model for training to obtain a text area detection model; The text information in the text area of ​​the sample image of the needle-printed font is annotated and then input into the second network model for training to obtain a text recognition model.

6. The method according to claim 5, characterized in that The first network model includes a multi-layer structure, and the position information of the text area in the sample image of the needle-printed font is annotated and input into the first network model for training to obtain a text area detection model, which specifically includes: Extracting image region features corresponding to the sample image of the text in the needle-printed font through the convolutional layer of the first network model; Generate horizontal text sequence features according to image region features corresponding to the text sample image through the decoding layer of the first network model; The text area in the text sample image is determined according to the horizontal text sequence features through the prediction layer of the first network model, and the text area is processed to obtain candidate text lines.

7. A device for recognizing text in an image, characterized in that: The device comprises: An acquisition unit is used to acquire a sample image of a text of a needle-printed font after scene processing, wherein the scene processing is implemented by the following scene processing steps for the sample image: using the maximum inter-class variance method to determine the optimal threshold for dividing the color attribute values ​​corresponding to pixels in the sample image; using the optimal threshold as a basis for division to perform binarization processing on the sample image to obtain background pixels and foreground pixels of the sample image after binarization processing; dividing the background pixels of the sample image after binarization processing into a plurality of background parts according to a preset ratio; performing scene processing on the sample image according to the pixel value of the corresponding parameter of each background part to obtain a sample image of a text of a needle-printed font; A training unit, used for inputting the text sample images of the needle-printed font as training data into network models of different architectures for training, so as to obtain a text region detection model and a text recognition model; A determination unit, configured to, when receiving an image text detection request, input the image requested for detection into the text region recognition model to determine position information of the text region corresponding to the image; The recognition unit is used to input the position information of the text area corresponding to the image and the image requested to be detected into the text recognition model to obtain the text information in the image.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images