Image optical character recognition method based on deep learning, storage device and server

Abstract

The present invention provides an image optical text recognition method based on deep learning. First, all pictures are scaled to a size of 640x640, and then data enhancement is performed on the pictures during training, including random perturbation, flipping, cropping, etc. In addition, a scale transformation is added to randomly transform the picture scale to a certain range of scales, enhance the scale change of the text in the image, increase the diversity of training samples at different scales, change the distribution of training samples, and focus on smaller texts. Then, after passing through a third neural network, 7 feature maps are obtained. The first area is subjected to a union-find operation to obtain the result of instance segmentation, and each text area is segmented separately. Then the skeleton line of the text area is obtained. Combined with the predicted radius, the final text detected is obtained. The method provided by the present invention can be adapted to texts of various shapes. The present invention has a good recognition effect in electronic weighing equipment, weighbridge weighing equipment, electronic bills, etc.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning, and specifically relates to an image optical character recognition method based on deep learning. Background Art

[0002] The concept of OCR was first proposed by German scientist Tausheck in 1929, and later American scientist Handel also proposed the idea of ​​using technology to recognize text. Casey and Nagy of IBM were the first to study printed Chinese character recognition. In 1966, they published the first article on Chinese character recognition, using template matching to recognize 1,000 printed Chinese characters. The application scenarios of OCR include but are not limited to electronic weighing equipment, weighbridge weighing equipment, printers, electronic forms, and other text recognition.

[0003] Optical character recognition refers to the process in which an electronic device (such as a scanner or digital camera) examines characters printed on paper, determines their shapes by detecting dark and light patterns, and then uses character recognition methods to translate the shapes into computer text; that is, for printed characters, optical methods are used to convert the text in paper documents into black and white dot matrix image files, and recognition software is used to convert the text in the image into text format for further editing and processing by word processing software. How to debug or use auxiliary information to improve the recognition accuracy is the most important topic in optical character recognition, and the term intelligent character recognition has also been created for this reason. The main indicators for measuring the performance of an optical character recognition system are: rejection rate, false recognition rate, recognition speed, user interface friendliness, product stability, ease of use and feasibility, etc.

[0004] Currently, there are several mechanisms for text detection: The rectangular box based on axis alignment only works for horizontal and vertical text. For tilted, bent, and curved text, it will introduce many background-irrelevant areas. The rectangular box based on rotation adds angle information on the basis of axis alignment. It can adapt to tilted text, but for bent and curved text, it will still introduce background-irrelevant areas. The method based on convex quadrilateral still cannot adapt to curved and bent text. Summary of the invention

[0005] The object of the present invention is to provide an optical character recognition method that can accurately recognize irregular text images.

[0006] The present invention provides an image optical character recognition method based on deep learning, comprising the following steps: S1 scaling the image to a size of 640x640;

[0007] S2 performs data enhancement on the image, including random perturbation, flipping, and cropping. It also adds scale transformation to randomly transform the image scale to a preset range.

[0008] S3 changes the distribution of training samples, and obtains 7 feature maps through the third neural network, wherein the 7 feature maps include 2 text areas, 2 text center lines, the radius of a circle, the cosine value cosθ of an angle, and the sine value sinθ of an angle, and the third neural network is composed of the first neural network and the second neural network, wherein θ represents the tangent angle;

[0009] S4 obtains the first area by multiplying the text area and the text center line, and obtains the result of instance segmentation by performing a union-find operation on the first area. Each text area is segmented separately, and then the skeleton line of the text area is obtained by performing the first algorithm, and then combined with the predicted radius to obtain the final text detected.

[0010] Furthermore, the first neural network performs pixel-level classification on the image. The first neural network accepts input images of any size, uses a deconvolution layer to upsample the feature map of the last convolution layer, restores the output image to the same size as the input image, generates a prediction for each pixel, and retains the spatial information in the original input image. Finally, pixel-by-pixel classification is performed on the upsampled feature map.

[0011] Furthermore, the second neural network is constructed by the following steps:

[0012] The first step is to use the output of the first neural network as the input of the second neural network to build a bottom-up network;

[0013] The second step is to build the corresponding top-down network and perform upsampling operations. First, use 1x1 convolution to reduce the dimension of the second-to-last layer, then add the second-to-last layer and the first-to-last layer element by element, and finally perform 3x3 convolution operations.

[0014] The third step is to repeat the second step in reverse order to establish the second neural network structure, select the network output branch, and connect the corresponding classification layer and regression layer.

[0015] The present invention provides a storage device, wherein a plurality of instructions are stored, wherein the instructions are suitable for being loaded and executed by a processor:

[0016] S1 scales the image to 640x640 size;

[0017] S2 performs data enhancement on the image, including random perturbation, flipping, and cropping. It also adds scale transformation to randomly transform the image scale to a preset range.

[0018] S3 changes the distribution of training samples, and obtains 7 feature maps through the third neural network, wherein the 7 feature maps include 2 text areas, 2 text center lines, the radius of a circle, the cosine value cosθ of an angle, and the sine value sinθ of an angle, and the third neural network is composed of the first neural network and the second neural network, wherein θ represents the tangent angle;

[0019] S4 obtains the first area by multiplying the text area and the text center line, and obtains the result of instance segmentation by performing a union-find operation on the first area. Each text area is segmented separately, and then the skeleton line of the text area is obtained by performing the first algorithm, and then combined with the predicted radius to obtain the final text detected.

[0020] Furthermore, the first neural network performs pixel-level classification on the image. The first neural network accepts input images of any size, uses a deconvolution layer to upsample the feature map of the last convolution layer, restores the output image to the same size as the input image, generates a prediction for each pixel, and retains the spatial information in the original input image. Finally, pixel-by-pixel classification is performed on the upsampled feature map.

[0021] Furthermore, the second neural network is constructed by the following steps:

[0022] The first step is to use the output of the first neural network as the input of the second neural network to build a bottom-up network;

[0023] The second step is to build the corresponding top-down network and perform upsampling operations. First, use 1x1 convolution to reduce the dimension of the second-to-last layer, then add the second-to-last layer and the first-to-last layer element by element, and finally perform 3x3 convolution operations.

[0024] The third step is to repeat the second step in reverse order to establish the second neural network structure, select the network output branch, and connect the corresponding classification layer and regression layer.

[0025] The present invention provides a server, comprising

[0026] a processor adapted to implement the instructions; and

[0027] A storage device, adapted to store a plurality of instructions, wherein the instructions are adapted to be loaded and executed by a processor:

[0028] S1 scales the image to 640x640 size;

[0029] S2 performs data enhancement on the image, including random perturbation, flipping, and cropping. It also adds scale transformation to randomly transform the image scale to a preset range.

[0030] S3 changes the distribution of training samples, and obtains 7 feature maps through the third neural network, wherein the 7 feature maps include 2 text areas, 2 text center lines, the radius of a circle, the cosine value cosθ of an angle, and the sine value sinθ of an angle, and the third neural network is composed of the first neural network and the second neural network, wherein θ represents the tangent angle;

[0031] S4 obtains the first area by multiplying the text area and the text center line, and obtains the result of instance segmentation by performing a union-find operation on the first area. Each text area is segmented separately, and then the skeleton line of the text area is obtained by performing the first algorithm, and then combined with the predicted radius to obtain the final text detected.

[0032] Furthermore, the first neural network performs pixel-level classification on the image. The first neural network accepts input images of any size, uses a deconvolution layer to upsample the feature map of the last convolution layer, restores the output image to the same size as the input image, generates a prediction for each pixel, and retains the spatial information in the original input image. Finally, pixel-by-pixel classification is performed on the upsampled feature map.

[0033] Furthermore, the second neural network is constructed by the following steps:

[0034] The first step is to use the output of the first neural network as the input of the second neural network to build a bottom-up network;

[0035] The second step is to build the corresponding top-down network and perform upsampling operations. First, use 1x1 convolution to reduce the dimension of the second-to-last layer, then add the second-to-last layer and the first-to-last layer element by element, and finally perform 3x3 convolution operations.

[0036] The third step is to repeat the second step in reverse order to establish the second neural network structure, select the network output branch, and connect the corresponding classification layer and regression layer.

[0037] The beneficial effect of the present invention is that the method provided by the present invention can be adapted to texts of various shapes and has a good recognition effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 Optical character recognition process.

[0039] Figure 2 Optical character recognition model.

[0040] Figure 3 Optical text training and reasoning process. DETAILED DESCRIPTION

[0041] Due to the popularity and wide application of scanners, OCR software only needs to provide an interface with the scanner and use the scanner driver software. Therefore, OCR software is mainly composed of the following parts.

[0042] Image input, preprocessing:

[0043] Image input: Different image formats have different storage formats and compression methods. Currently, there are open source projects such as OpenCV and CxImage. Preprocessing: mainly includes binarization, noise removal, tilt correction, etc.

[0044] Binarization: Most of the pictures taken by the camera are color images. Color images contain a huge amount of information. For the content of the picture, we can simply divide it into foreground and background. In order to make the computer recognize text faster and better, we need to process the color image first so that the image only has foreground information and background information. We can simply define the foreground information as black and the background information as white. This is a binary image.

[0045] Noise removal: For different documents, we can define noise differently. Denoising based on the characteristics of the noise is called noise removal.

[0046] Tilt correction: Since most users are more casual when taking photos of documents, the pictures taken are inevitably tilted, which requires text recognition software to correct.

[0047] Layout analysis: The process of dividing document images into paragraphs and lines is called layout analysis. Due to the diversity and complexity of actual documents, there is currently no fixed and optimal cutting model.

[0048] Character cutting: Due to the limitations of shooting conditions, characters often stick together and the pen breaks, which greatly limits the performance of the recognition system. This requires the text recognition software to have a character cutting function.

[0049] Character recognition: Template matching was used earlier, and later it became mainly feature extraction. The difficulty of feature extraction is greatly affected by factors such as text displacement, stroke thickness, stroke breakage, adhesion, and rotation.

[0050] Layout recovery: People hope that the recognized text can still be arranged like the original document image, with the paragraphs, positions and order unchanged and output to word documents, pdf documents, etc. This process is called layout recovery.

[0051] Post-processing and proofreading: Post-processing is the process of correcting the recognition results based on the specific language context.

[0052] The present invention provides an image text recognition method based on deep learning. The method extracts image feature information through a neural network to achieve text recognition. The present invention performs well when dealing with irregular text. The deep learning model used in the invention is as follows Figure 2 .

[0053] In the present invention, for an input picture, all pictures are first scaled to a size of 640x640, and then data enhancement is performed on the pictures during training, including random perturbation, flipping, cropping, etc. In addition, a scale transformation is added to randomly transform the picture scale to a certain range of scales, greatly enhancing the scale change of the picture and text in the image, increasing the diversity of training samples at different scales, changing the distribution of training samples, and focusing on smaller texts. Then, after passing through the third neural network, 7 feature maps are obtained. Including 2 text areas, 2 text center lines, the radius of a circle, the cosine value cosθ of an angle, and the sine value sinθ of an angle.

[0054] The first region is obtained by multiplying the text region and the text centerline, which represents the text centerline extracted on the text region. The first region is subjected to a union-find operation to obtain the result of instance segmentation, and each text region is segmented separately. Then, the skeleton line of the text region is obtained by the first algorithm. Combined with the predicted radius, the final text detected is obtained.

[0055] The following is a specific description of the key technologies in the present invention:

[0056] The first neural network: the backbone network of the present invention, is a basic network used to extract pixel-by-pixel features of images.

[0057] The first neural network classifies images at the pixel level, thus solving the problem of semantic image segmentation. Unlike the classic CNN, which uses a fully connected layer after the convolution layer to obtain a fixed-length feature vector for classification (fully connected layer + softmax output), the first neural network can accept input images of any size, and uses a deconvolution layer to upsample the feature map of the last convolution layer to restore it to the same size as the input image, so that a prediction can be generated for each pixel while retaining the spatial information in the original input image. Finally, pixel-by-pixel classification is performed on the upsampled feature map.

[0058] The second neural network is the backbone network of the present invention, and is a basic network used to extract image features.

[0059] The first step is to use the output of the first neural network as the input of the second neural network to build a bottom-up network;

[0060] The second step is to build the corresponding top-down network and perform upsampling operations. First, use 1x1 convolution to reduce the dimension of the second-to-last layer, then add the second-to-last layer and the first-to-last layer element by element, and finally perform 3x3 convolution operations.

[0061] The third step is to repeat the second step in reverse order to establish the second neural network structure, select the network output branch, and connect the corresponding classification layer and regression layer.

[0062] In the present invention, the workflow of optical text recognition is as follows: Figure 1 The specific process is as follows:

[0063] 1. All images will be scaled to the specified size, and data enhancement will be performed on the images during training, including random perturbation, flipping, cropping, etc. In addition, scale transformation is added to randomly transform the image scale to a certain range of scales, greatly enhancing the scale change of images and texts in the image, increasing the diversity of training samples at different scales, changing the distribution of training samples, and focusing on smaller texts.

[0064] 2. After the third neural network, 7 feature maps are obtained, including 2 text areas, 2 text center lines, 1 circle radius, the cosine value of an angle cosθ, and the sine value of an angle sinθ;

[0065] 3. The first area is obtained by multiplying the text area and the text centerline, which represents the text centerline extracted on the text area;

[0066] 4. Perform a union-find operation on the first region to obtain the result of instance segmentation, and separate each text region separately;

[0067] 5. Then, through the first algorithm, the skeleton line of the text area can be obtained;

[0068] 6. Combined with the predicted radius, the final text of text detection is obtained.

[0069] In the present invention, the workflow of optical text recognition is as follows: Figure 3 The specific process is as follows:

[0070] 1. First, collect video image information through the camera;

[0071] 2. Annotation data, in the format of x, y, w, h, label, where each letter represents the horizontal coordinate, vertical coordinate, width, height and label of the annotation box;

[0072] 3. Split the data set into training set and validation set at 8:2;

[0073] 4. Data augmentation and normalized training set

[0074] 5. Calculate the RGB mean and variance of the training set;

[0075] 6. Train and validate the model;

[0076] 7. Export the model;

[0077] 8. Use the model to perform real-time online image optical character recognition.

[0078] In the present invention, the first indicator used to describe the amount of text data in the image is used to measure the quality of the neural network output. The first indicator is calculated using the following formula:

[0079]

[0080] Among them, |y| represents the total number of sample categories, p k represents the proportion of samples in the kth category, k represents the sequence number of the sample, and 0≤pk≤1. n represents the total number of samples.

[0081] The present invention adopts a second index to describe the importance of a feature. In the present invention, the first index is obtained by DiYi(D), and then DiEr(A) is calculated to obtain the second index.

[0082] For data set D and attribute subset A, D is divided into V subsets according to the value of A.

[0083] {D 1 ,D 2 ,...,D V},

[0084]

[0085] In the attribute subset A, select an attribute a at random, traverse all its possible values, and find the optimal partition point v of attribute a according to the following formula: * :

[0086]

[0087] Where R1(a,v)={x|x∈D a≤v},R2(a,v)={x|x∈D a>v}, c1, c2 are samples x in sets R1(a, v) and R2(a, v) respectively. i Corresponding output value y i The mean of , v represents the number of the subset, i represents the number of the sample and the number of the corresponding output value, that is,

[0088]

[0089]

[0090] The upsampling used in the neural network in the present invention is calculated in two steps, and ave represents an averaging function.

[0091] 1. Calculate the original image coordinates:

[0092] srcX=dstX×(srcW / dstW)

[0093] srcY=dstY×(srcH / dstH)

[0094] Among them, srcX, srcY, srcH, and srcW are the horizontal coordinates of the original image pixels, the vertical coordinates of the pixels, the image height, and the image width, respectively; dstX, dstY, dstH, and dstW are the horizontal coordinates of the target image pixels, the vertical coordinates of the pixels, the image height, and the image width, respectively.

[0095] Round the coordinates to the left to get the x, y coordinates of the pixel in the original image:

[0096] x = math.floor(scrx)

[0097] y = math.floor(scry)

[0098] 2. Calculate the offset u and v:

[0099] u=scrx-x

[0100] v = scry-y

[0101] Calculate the result of the position (i, j) in the upsampling, where i∈[0,dstW), j∈[0,dstH): dst[i,j]=(1-u)×(1-v)×sre[x,y]+u×(1-v)×sre[x+1,y]+(1-u)×v×sre[x,y+1]+u×v×sre[x+1,y+1], where sre[x,y] is the pixel in the xth row and yth column of the original image, and dst[i,j] is the pixel in the i-th row and j-th column of the target image.

[0102] The present invention adopts the first algorithm to obtain the skeleton line of the text area:

[0103] Reposition the given point to the mid-axis;

[0104] Randomly select a point on the centerline of the text, draw the tangent and normal lines, and obtain the midpoint through the midpoint of the intersection of the normal line and the text centerline area.

[0105] Take a big step towards the next search point, searching towards both ends of the text instance;

[0106] The forward displacement is calculated as: The backward displacement is calculated as: θ represents the tangent angle, r represents the radius of the circle

[0107] If the next step is outside the text centerline area, gradually reduce the step size until it is within the area, or reaches the end.

[0108] Iterate on the central axis and draw circles along the central axis with a radius of r. The area covered by the circle represents the predicted text instance.

[0109] The beneficial effect of the present invention is that the method provided by the present invention can be adapted to texts of various shapes and has a good recognition effect.

[0110] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for image optical character recognition based on deep learning, characterized in that: The following steps are included: S1 scales the image to 640x640 size; S2 performs data enhancement on the image, including random perturbation, flipping, and cropping. It also adds scale transformation to randomly transform the image scale to a preset range. S3 changes the distribution of training samples, and obtains 7 feature maps through the third neural network, wherein the 7 feature maps include 2 text areas, 2 text center lines, the radius of a circle, the cosine value cosθ of an angle, and the sine value sinθ of an angle, and the third neural network is composed of the first neural network and the second neural network, wherein θ represents the tangent angle; S4 obtains the first region by multiplying the text region and the text center line, performs a union-find operation on the first region to obtain the result of instance segmentation, separates each text region, and then obtains the skeleton line of the text region by performing the first algorithm, and then obtains the final text detected by combining the predicted radius; The first neural network classifies the image at the pixel level. The first neural network accepts an input image of any size, uses a deconvolution layer to upsample the feature map of the last convolution layer, restores the output image to the same size as the input image, generates a prediction for each pixel, and retains the spatial information in the original input image. Finally, pixel-by-pixel classification is performed on the upsampled feature map; The second neural network is constructed by the following steps: The first step is to use the output of the first neural network as the input of the second neural network to build a bottom-up network; The second step is to build the corresponding top-down network and perform upsampling operations. First, use 1x1 convolution to reduce the dimension of the second-to-last layer, then add the second-to-last layer and the first-to-last layer element by element, and finally perform 3x3 convolution operations. The third step repeats the second step, stacking operations in reverse order, establishing the second neural network structure, selecting the network output branch, and connecting the corresponding classification layer and regression layer; The first algorithm specifically refers to: The forward displacement is calculated as: The backward displacement is calculated as: θ is the tangent angle and r is the radius of the circle.

2. A storage device, characterized in that: There are a plurality of instructions stored therein, which are suitable for being loaded and executed by a processor: S1 scales the image to 640x640 size; S2 performs data enhancement on the image, including random perturbation, flipping, and cropping. It also adds scale transformation to randomly transform the image scale to a preset range. S3 changes the distribution of training samples, and obtains 7 feature maps through the third neural network, wherein the 7 feature maps include 2 text areas, 2 text center lines, the radius of a circle, the cosine value cosθ of an angle, and the sine value sinθ of an angle, and the third neural network is composed of the first neural network and the second neural network, wherein θ represents the tangent angle; S4 obtains the first region by multiplying the text region and the text center line, performs a union-find operation on the first region to obtain the result of instance segmentation, separates each text region, and then obtains the skeleton line of the text region by performing the first algorithm, and then obtains the final text detected by combining the predicted radius; The first neural network classifies the image at the pixel level. The first neural network accepts an input image of any size, uses a deconvolution layer to upsample the feature map of the last convolution layer, restores the output image to the same size as the input image, generates a prediction for each pixel, and retains the spatial information in the original input image. Finally, pixel-by-pixel classification is performed on the upsampled feature map. The second neural network is constructed by the following steps: The first step is to use the output of the first neural network as the input of the second neural network to build a bottom-up network; The second step is to build the corresponding top-down network and perform upsampling operations. First, use 1x1 convolution to reduce the dimension of the second-to-last layer, then add the second-to-last layer and the first-to-last layer element by element, and finally perform 3x3 convolution operations. The third step repeats the second step, stacking operations in reverse order, establishing the second neural network structure, selecting the network output branch, and connecting the corresponding classification layer and regression layer; The first algorithm specifically refers to: The forward displacement is calculated as: The backward displacement is calculated as: θ is the tangent angle and r is the radius of the circle.

3. A server, characterized in that: include a processor adapted to implement the instructions; and A storage device, adapted to store a plurality of instructions, wherein the instructions are adapted to be loaded and executed by a processor: S1 scales the image to 640x640 size; S2 performs data enhancement on the image, including random perturbation, flipping, and cropping. It also adds scale transformation to randomly transform the image scale to a preset range. S3 changes the distribution of training samples, and obtains 7 feature maps through the third neural network, wherein the 7 feature maps include 2 text areas, 2 text center lines, the radius of a circle, the cosine value cosθ of an angle, and the sine value sinθ of an angle, and the third neural network is composed of the first neural network and the second neural network, wherein θ represents the tangent angle; S4 obtains the first region by multiplying the text region and the text center line, performs a union-find operation on the first region to obtain the result of instance segmentation, separates each text region, and then obtains the skeleton line of the text region by performing the first algorithm, and then obtains the final text detected by combining the predicted radius; The first neural network classifies the image at the pixel level. The first neural network accepts an input image of any size, uses a deconvolution layer to upsample the feature map of the last convolution layer, restores the output image to the same size as the input image, generates a prediction for each pixel, and retains the spatial information in the original input image. Finally, pixel-by-pixel classification is performed on the upsampled feature map. The second neural network is constructed by the following steps: The first step is to use the output of the first neural network as the input of the second neural network to build a bottom-up network; The second step is to build the corresponding top-down network and perform upsampling operations. First, use 1x1 convolution to reduce the dimension of the second-to-last layer, then add the second-to-last layer and the first-to-last layer element by element, and finally perform 3x3 convolution operations. The third step repeats the second step, stacking operations in reverse order, establishing the second neural network structure, selecting the network output branch, and connecting the corresponding classification layer and regression layer; The first algorithm specifically refers to: The forward displacement is calculated as: The backward displacement is calculated as: θ is the tangent angle and r is the radius of the circle.

Images