Deep learning-based bill OCR (Optical Character Recognition) method

Abstract

The invention discloses a bill OCR (Optical Character Recognition) method based on deep learning, relates to the technical field of optical character recognition, and solves the technical problem that inaccuracy and errors exist in subsequent analysis and use of data due to the fact that the data is not processed and utilized. A data set is enriched, the model can learn wider bill characteristics, the generalization ability of the model is improved, the model can better adapt to various bill conditions in practical application, and for the condition that real-time bill recognition cannot be carried out, through operations of segmenting an original image, calculating a peak signal-to-noise ratio and the like, contrastive analysis is carried out on the original image and the template bill, the real-time bill recognition can be realized. According to the method, the problem can be quickly positioned, the template bill most similar to the real-time bill is found as the standard, the standard information is input into the recognition model to generate an accurate recognition result, and the success rate and reliability of real-time bill recognition are improved.

Description

Technical Field

[0001] The present invention relates to bill recognition, and in particular to a bill OCR recognition method based on deep learning. Background Art

[0002] With the development of the digital age, the demand for automated bill processing is growing. Traditional bill processing methods mainly rely on manual work, which is inefficient and prone to errors. Optical character recognition (OCR) technology makes it possible to automatically identify and enter data for bills. The rise of deep learning technology has brought new opportunities to improve the accuracy and efficiency of bill OCR recognition.

[0003] According to the publication number CN118430000A, a bill OCR recognition method and system based on deep learning is involved. The method includes the following steps: obtaining a bill image to be recognized; extracting bill edge feature points based on the bill image to be recognized; performing vertical correction processing on the bill image to be recognized based on the bill edge feature points to obtain a corrected bill image to be recognized; performing brightness distribution analysis on the corrected bill image to be recognized to obtain image brightness distribution data; performing smoothing filter optimization based on the image brightness distribution data to obtain a filtered optimized bill image; performing watermark texture feature analysis on the filtered optimized bill image to generate watermark texture feature data; performing visual continuity repair processing based on the watermark texture feature data to generate a visually repaired bill image; and obtaining bill layout structure data based on the visually repaired bill image.

[0004] However, some existing recognition methods cannot centrally process the acquired data when used, which results in errors in the subsequent use of data. On the other hand, they do not use the data reasonably, which further results in inaccuracy in the recognition of bills. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention provides a bill OCR recognition method based on deep learning, which solves the problem that the data is not processed and utilized, resulting in inaccuracy and errors in the subsequent analysis and use of the data.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: a bill OCR recognition method based on deep learning, the method specifically comprises the following steps:

[0007] Step 1: collect different types of bill images under different lighting intensities, background colors, and shooting angles, and collect bill images from different channels, different regions, and different formats;

[0008] Step 2: performing a preprocessing operation on the obtained bill image to obtain a preprocessed bill, and the preprocessing operation includes image cleaning, normalization processing and data enhancement;

[0009] Step 3: Classify the obtained pre-processed bills to obtain bills of the same type, extract data features of the bills of the same type, and establish a recognition model based on the obtained data features;

[0010] Step 4: Recognize and judge the acquired real-time bill to obtain a judgment result, and analyze the situation corresponding to the unrecognizable result in the judgment result to obtain an analysis result;

[0011] Step 5: Input the acquired standard information into the recognition model, and generate corresponding recognition results based on the recognition model.

[0012] As a further solution of the present invention: the specific method of obtaining the pre-processed bill in step 2 is:

[0013] The bill image is obtained and recorded as the original bill image, wherein the specific method of image cleaning is: using a filtering algorithm to remove interference factors such as noise, stains, scratches, etc. in the bill image;

[0014] The specific method of normalization processing is: normalizing the size, grayscale, contrast, etc. of the image so that all the bill images have a unified format and feature scale;

[0015] The specific method of data enhancement for the original bill image is to expand the data set by performing random rotation, translation, scaling, flipping and other operations on the original bill image.

[0016] As a further solution of the present invention: the specific method of obtaining the data features in step 3 is:

[0017] Obtain the preprocessed bills and the bill types corresponding to the preprocessed bills, such as invoices, receipts, checks, etc., and classify them according to the same bill type to obtain the same type of bills. Then use the ResNet network to extract the data features of the same type of bills to obtain data features.

[0018] As a further solution of the present invention: the specific method of establishing the recognition model according to the obtained data features in step 3 is:

[0019] The obtained data features are divided into training set and validation set according to a preset ratio. At the same time, the convolutional neural network is selected as the main backbone architecture, and the appropriate number of training rounds and batch size are determined. The obtained training set is substituted into the convolutional neural network for training to obtain the recognition model. Then the validation set is substituted into the recognition model to verify the model. At the same time, the recognition model is optimized according to the verification results to obtain the recognition model.

[0020] As a further solution of the present invention: the specific method of identifying and judging the real-time bill in step 4 to obtain the judgment result is:

[0021] Obtain a real-time bill and input it into the recognition model, and determine whether the real-time bill can be recognized, and generate a judgment result, where the judgment result includes whether it can be recognized or not. The recognized result is not processed, and the unrecognized result is further analyzed.

[0022] As a further solution of the present invention: the specific method of analyzing the unrecognizable results in step 4 is:

[0023] Get the real-time bill and the original image of the real-time bill. Then divide the original image into nine equal parts to obtain the segmented original image, denoted as i, and i = 1, 2, ..., j, where j represents the number of segmented original images. At the same time, obtain the gray value corresponding to the segmented original image i, denoted as Li, and then substitute the gray value into the formula The peak signal-to-noise ratio P i corresponding to the segmented original image i is calculated, where MSE is the mean square error. Similarly, the above calculation is performed on the peak signal-to-noise ratios of all segmented original images, and the mean of all peak signal-to-noise ratios is calculated as a comparison value, and the comparison value is analyzed at the same time.

[0024] As a further solution of the present invention: the specific method of analyzing the comparison value in step 4 is:

[0025] Obtain the bill type of the real-time bill, and extract the template bill of the same bill type as the real-time bill in the recognition model, and mark the template bill with a number, and a = 1, 2, ..., b, where b is the number of template bills. At the same time, calculate the peak signal-to-noise ratio Pa of the template bill, and calculate the difference between the real-time bill comparison value and the peak signal-to-noise ratio of the template bill, and select the template bill corresponding to the smallest difference as the standard to generate standard information.

[0026] Beneficial Effects

[0027] The present invention provides a bill OCR recognition method based on deep learning. Compared with the prior art, it has the following beneficial effects:

[0028] The present invention collects a large number of bill images under different conditions, including various lighting, backgrounds and shooting angles, and collects bills from different channels, regions and formats, thereby enriching the data set, enabling the model to learn a wider range of bill features, improving the generalization ability of the model, and being able to better adapt to various bill situations in practical applications. For situations that cannot be recognized in real-time bill recognition, the problem can be quickly located by segmenting the original image, calculating the peak signal-to-noise ratio, and comparing and analyzing it with the template bill, and finding the template bill most similar to the real-time bill as the standard, and then inputting the standard information into the recognition model to generate accurate recognition results, thereby improving the success rate and reliability of real-time bill recognition, reducing the need for manual intervention, and improving the overall efficiency of bill processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a flow chart of the steps of the present invention. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] See also Figure 1 The present application provides a bill OCR recognition method based on deep learning, which specifically includes the following steps:

[0032] Step 1: Collect a large number of different types of bill images under different lighting intensities, background colors, and shooting angles. For example, collect bill images under bright office lights, dim warehouse environments, and desktop backgrounds of different colors. Collect bill images from different channels, different regions, and different formats as much as possible, including handwritten bills, printed bills, bills of different fonts and sizes, etc.

[0033] Step 2: Preprocessing the obtained bill image to obtain a preprocessed bill, and the preprocessing operation includes image cleaning, normalization and data enhancement.

[0034] Obtain a bill image and record it as the original bill image. The specific method of image cleaning is: use a filtering algorithm to remove interference factors such as noise, stains, scratches, etc. in the bill image. For example, for some stains caused by printer ink stains, filter processing is used to reduce their influence on text recognition. Use a median filtering algorithm to replace the value of each pixel in the image with the median value of its neighboring pixel values;

[0035] The specific method of normalization is: normalize the size, grayscale, contrast, etc. of the image so that all bill images have a unified format and feature scale. For example, the size of all bill images is uniformly adjusted to a size suitable for model input. For example, the image size is required to be [width x height] = [512x512] pixels. If the original bill images have different sizes, such as a receipt image size of [800x600] pixels and another invoice image size of [1024x768] pixels, they need to be adjusted to a standard size of [512x512] pixels through operations such as scaling or cropping, and the grayscale value is adjusted to a certain range to enhance the contrast between text and background. The invoice image originally had a grayscale value range of [30,180], and after processing, the grayscale value range is adjusted to [50,220];

[0036] The specific method of data enhancement for the original bill image is: by randomly rotating, translating, scaling, flipping and other operations on the original bill image, the data set is expanded. For example, the bill image is randomly rotated by a certain angle to simulate the possible placement angle of the bill in actual applications, such as the rotation angle is between [-15°, +15°], and the image is scaled, such as the scaling ratio is between [0.8, 1.2]. For example, the image is randomly translated by [-50, +50] pixels in the horizontal direction and randomly translated by [-30, +30] pixels in the vertical direction, so that bill texts of different sizes can be further recognized;

[0037] After the above operations, a pre-processed ticket is obtained.

[0038] Step three, classify the obtained pre-processed bills to obtain bills of the same type, extract data features of the bills of the same type, and establish a recognition model based on the obtained data features.

[0039] Obtain the pre-processed bills and the bill types corresponding to the pre-processed bills, such as invoices, receipts, checks, etc., and classify them according to the same bill type to obtain the same type of bills. Then use the ResNet network to extract the data features of the same type of bills to obtain data features;

[0040] The obtained data features are divided into training sets and validation sets according to the preset ratio. At the same time, the convolutional neural network is selected as the main framework, the appropriate number of training rounds and batch size are determined, and the obtained training set is substituted into the convolutional neural network for training to obtain the recognition model. For example, the number of training rounds is set to 100 and the batch size is set to 32. This means that the entire training data set will be divided into multiple batches, each batch contains 32 bill image samples, and the model will be trained for 100 rounds. Then the validation set is substituted into the recognition model to verify the model. At the same time, the recognition model is optimized according to the verification results, and the performance of the model is evaluated using specific indicators such as accuracy, recall rate, and F1 score. According to the evaluation results, the problems existing in the model are analyzed, such as inaccurate recognition of bills with certain special fonts, font sizes, or layouts, and then corresponding optimization measures are taken. For example, training samples for special fonts are increased, and the network structure or hyperparameters of the model are adjusted to improve the performance of the model.

[0041] Step 4: The acquired real-time bill is identified and judged to obtain a judgment result, and at the same time, the situation corresponding to the unidentifiable result in the judgment result is analyzed to obtain an analysis result.

[0042] Obtain a real-time bill and input it into the recognition model, and determine whether the real-time bill can be recognized, and generate a judgment result, wherein the judgment result includes whether it can be recognized or not. The recognized result is not processed, and the unrecognized result is further analyzed;

[0043] Specifically, however, if the model cannot completely or accurately extract the bill information, the judgment result is "unable to recognize", and further analysis is required. For example, when the system receives an invoice with blurred handwriting and some content altered, the recognition model may not be able to accurately recognize some of the words or numbers in it, resulting in a judgment result of "unable to recognize".

[0044] For this kind of unrecognizable situation, we first need to obtain the original image of the real-time bill. This is because the original image contains the most complete bill information. For example, a cargo transportation invoice that was slightly damaged during transportation, resulting in some blurred text, is used to obtain its original image. Next, a series of analysis operations are performed on the original image to determine the specific cause of the recognition failure.

[0045] This may include image quality issues, abnormal bill formats, interference from special characters or symbols, etc. By analyzing these factors, appropriate measures can be taken to improve the recognition process and increase the success rate of subsequent recognition of similar bills. For example, if it is found that the recognition failure is caused by image blur, you can consider using image enhancement algorithms to improve the clarity of the image; if the bill format is significantly different from the common format, you can further analyze the characteristics of the format and adjust or retrain the model accordingly to adapt it to the recognition of bills in this special format.

[0046] Analyze the unrecognizable results, obtain the real-time bill, and obtain the original image of the real-time bill. Then divide the original image into nine equal parts to obtain the segmented original image, denoted as i, and i = 1, 2, ..., j, where j represents the number of segmented original images. At the same time, obtain the gray value corresponding to the segmented original image i and denoted as Li. Then substitute the gray value into the formula The peak signal-to-noise ratio Pi corresponding to the segmented original image i is calculated, where MSE is the mean square error. Similarly, the above calculation is performed on the peak signal-to-noise ratios of all segmented original images, and the mean of all peak signal-to-noise ratios is calculated and recorded as a comparison value;

[0047] In a specific example, the original image of the real-time bill is obtained, and then it is divided into nine equal parts. For the divided original image i=1, its grayscale value L1=120 is obtained. This is just an assumed grayscale value. In reality, the grayscale value is the attribute of each pixel. Here, in order to simplify the example, the average grayscale value of an area is assumed. For i=2, it is assumed that L2=135, and so on.

[0048] First, calculate the mean square error. Assume that for the segmented original image i=1, the pixel value difference between it and a reference image (which can be an ideal image without noise or other interference, a simple calculation is assumed here for the sake of example) is calculated to obtain MSE=10 (here is an assumed value).

[0049] For an 8-bit grayscale image, L=256, the peak signal-to-noise ratio of the segmented original image i is calculated according to the peak signal-to-noise ratio formula to be approximately 40.04db. Using the same method, assuming that the calculation results are P2=38.56db, P3=41.23db, etc., and finally calculating all the means, assuming that the calculated mean is 39.80db.

[0050] Next, the bill type of the real-time bill is obtained, and the template bill of the same bill type as the real-time bill in the recognition model is extracted. The template bill here represents the bill that already exists in the model and is stored in advance. The template bill is labeled as a, and a=1, 2, ..., b, where b is the number of template bills. At the same time, the peak signal-to-noise ratio Pa of the template bill is calculated, and the difference between the real-time bill comparison value and the template bill peak signal-to-noise ratio is calculated. At the same time, the template bill corresponding to the smallest difference is selected as the standard to generate standard information.

[0051] In a specific example, assume that the bill type of the real-time bill is "invoice". In the recognition model, different template bills have been stored. Assume that there are b = 5 template bills of the "invoice" type, numbered a = 1, 2, ..., 5, respectively, and calculate the peak signal-to-noise ratio of these template bills. For example, the peak signal-to-noise ratio of template bill a = 1 is Pa1 = 38.00db, the peak signal-to-noise ratio of template bill a = 2 is Pa2 = 40.05db, and so on.

[0052] Calculate the difference between the real-time bill comparison value and the peak signal-to-noise ratio of each template bill. For example, for template bill a=1, the difference is 1.80db, for template bill a=2, the difference is 0.70db, etc. Select the template bill with the smallest difference as the standard. Here, it is assumed that the difference of template bill a=2 is the smallest, then select template bill a=2 as the standard to generate standard information.

[0053] Step 5: Input the acquired standard information into the recognition model, and generate corresponding recognition results based on the recognition model.

[0054] Meanwhile, the contents not described in detail in this specification belong to the prior art known to those skilled in the art.

[0055] The above embodiments are only used to illustrate the technical method of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. The bill OCR recognition method based on deep learning is characterized by: The method specifically comprises the following steps: Step 1: collect different types of bill images under different lighting intensities, background colors, and shooting angles, and collect bill images from different channels, different regions, and different formats; Step 2: performing a preprocessing operation on the obtained bill image to obtain a preprocessed bill, and the preprocessing operation includes image cleaning, normalization processing and data enhancement; Step 3: Classify the obtained pre-processed bills to obtain bills of the same type, extract data features of the bills of the same type, and establish a recognition model based on the obtained data features; Step 4: Recognize and judge the acquired real-time bill to obtain a judgment result, and analyze the situation corresponding to the unrecognizable result in the judgment result to obtain an analysis result; Step 5: Input the acquired standard information into the recognition model, and generate corresponding recognition results based on the recognition model.

2. The bill OCR recognition method based on deep learning according to claim 1 is characterized in that: The specific method of obtaining the pre-processing ticket in step 2 is: The bill image is obtained and recorded as the original bill image, wherein the specific method of image cleaning is: using a filtering algorithm to remove interference factors such as noise, stains, scratches, etc. in the bill image; The specific method of normalization processing is: normalizing the size, grayscale, contrast, etc. of the image so that all the bill images have a unified format and feature scale; The specific method of data enhancement for the original bill image is to expand the data set by performing random rotation, translation, scaling, flipping and other operations on the original bill image.

3. The bill OCR recognition method based on deep learning according to claim 1 is characterized in that: The specific method of obtaining data features in step three is: Obtain the preprocessed bills and the bill types corresponding to the preprocessed bills, such as invoices, receipts, checks, etc., and classify them according to the same bill type to obtain the same type of bills. Then use the ResNet network to extract the data features of the same type of bills to obtain data features.

4. The bill OCR recognition method based on deep learning according to claim 1 is characterized in that: The specific method of establishing the recognition model according to the obtained data features in step 3 is: The obtained data features are divided into training set and validation set according to a preset ratio. At the same time, the convolutional neural network is selected as the main backbone architecture, and the appropriate number of training rounds and batch size are determined. The obtained training set is substituted into the convolutional neural network for training to obtain the recognition model. Then the validation set is substituted into the recognition model to verify the model. At the same time, the recognition model is optimized according to the verification results to obtain the recognition model.

5. The bill OCR recognition method based on deep learning according to claim 1 is characterized in that: The specific method of identifying and judging the real-time bill in step 4 to obtain the judgment result is: Obtain a real-time bill and input it into the recognition model, and determine whether the real-time bill can be recognized, and generate a judgment result, where the judgment result includes whether it can be recognized or not. The recognized result is not processed, and the unrecognized result is further analyzed.

6. The bill OCR recognition method based on deep learning according to claim 5 is characterized in that: The specific method for analyzing the unrecognizable results in step 4 is as follows: Get the real-time bill and the original image of the real-time bill. Then divide the original image into nine equal parts to obtain the segmented original image, denoted as i, and i = 1, 2, ..., j, where j represents the number of segmented original images. At the same time, obtain the gray value corresponding to the segmented original image i, denoted as Li, and then substitute the gray value into the formula The peak signal-to-noise ratio Pi corresponding to the segmented original image i is calculated, where MSE is the mean square error. Similarly, the above calculation is performed on the peak signal-to-noise ratios of all segmented original images, and the mean of all peak signal-to-noise ratios is calculated as a comparison value, and the comparison value is analyzed at the same time.

7. The bill OCR recognition method based on deep learning according to claim 6 is characterized in that: The specific method of analyzing the comparison value in step 4 is: Obtain the bill type of the real-time bill, and extract the template bill of the same bill type as the real-time bill in the recognition model, and mark the template bill with a number, and a = 1, 2, ..., b, where b is the number of template bills. At the same time, calculate the peak signal-to-noise ratio Pa of the template bill, and calculate the difference between the real-time bill comparison value and the peak signal-to-noise ratio of the template bill, and select the template bill corresponding to the smallest difference as the standard to generate standard information.

Images