An ocr text recognition method and system based on seal removal

Abstract

The application relates to the technical field of optical character recognition, and provides an OCR (Optical Character Recognition) text recognition method and system based on seal removal, which comprises the following steps: obtaining a first seal area and first category information of whether a seal covers text based on a target image; using the first seal area as a mask, using a seal color interval, obtaining a second image after removing the seal in the mask, performing seal detection on the second image, obtaining a second seal area and second category information, performing cutting on the second seal area to obtain a plurality of sub-pictures containing seals, and removing the seals in the sub-pictures through a deep learning model based on the second category information; performing binarization on the second image to obtain a third image, merging the sub-pictures after removing the seals onto the third image, detecting a text area, and performing text recognition; correcting the text recognition result through semantic correction to obtain a final text recognition result. The accuracy of the OCR recognition result is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of optical character recognition, and in particular relates to an OCR text recognition method and system based on seal removal. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Today's optical character recognition (OCR) technology is quite mature and widely used in various fields, including bills, certificates, and electronic documents. However, in some scenarios, such as audit materials, certification documents, or tax receipts, the images often contain not only the text to be recognized but also stamps. This can obscure the text to be recognized, affecting recognition accuracy. Therefore, it is necessary to remove the stamp interference before recognizing the text.

[0004] Traditional methods for removing seals primarily rely on the difference in color between the seal and text, using either the RGB or HSV color space. This approach is ineffective when the seal and text are too similar and difficult to distinguish. Sometimes, while the text appears distinct from the seal, the RGB channel values ​​of certain pixels at the edges of the text are the same or similar to the seal color. Using this color space to remove seals can affect the text being recognized and compromise OCR results. Summary of the Invention

[0005] In order to solve the technical problems existing in the above-mentioned background technology, the present invention provides an OCR text recognition method and system based on seal removal, which combines a deep learning model and a traditional color space to ensure the accuracy of seal removal while improving the generalization of seal removal. At the same time, semantic correction is used to post-process the OCR recognition results, further improving the accuracy of the OCR recognition results.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] A first aspect of the present invention provides an OCR text recognition method based on seal removal, comprising:

[0008] Acquire the target image;

[0009] Based on the target image, using an object detection network, obtaining first category information of a first seal area and whether the seal covers text;

[0010] Using the first seal area as a mask and the seal color range, the seal in the mask is removed to obtain a second image. Seal detection is performed on the second image to obtain a second seal area and second category information. The second seal area is cropped to obtain multiple sub-images containing seals. Based on the second category information, the seals in the sub-images are removed using a deep learning model.

[0011] Binarizing the second image to obtain a third image, merging the sub-image after removing the seal onto the third image, detecting the text area, and performing text recognition;

[0012] Use semantic correction to correct the text recognition results to obtain the final text recognition results.

[0013] Furthermore, the stamp in the mask is removed using the statistically obtained HSV color interval of the stamp to obtain a second image.

[0014] Furthermore, if the second category information is text not covered by the seal, the pixel value of the sub-image is set to 255.

[0015] Furthermore, if the second category information is text covered by a seal, the deep learning model classifies each pixel to determine whether each pixel belongs to text or other categories, sets the pixel points of the text category to 0, and sets the pixel points of other categories to 255 to remove the seal.

[0016] Furthermore, the second image is binarized using an adaptive threshold algorithm based on Gaussian weighted averaging.

[0017] Furthermore, a generative pre-trained language model is used for the recognition results to determine the possibility of a certain word appearing in a sentence according to the sentence context, and the sentences are divided into coherent sentences and incoherent sentences.

[0018] Furthermore, for incoherent sentences, from left to right, based on the previous context, the probability of the next character appearing at each position is calculated. When the probability is less than a preset threshold, the position is considered to be a position that needs to be corrected, and a first position set is obtained. At the same time, from right to left, based on the following context, the probability of the previous character appearing at each position is calculated. When the probability is less than a preset threshold, the position is considered to be a position that needs to be corrected, and a second position set is obtained. The same part of the first position set and the second position set is taken as the part to be modified, and the language model is used to predict the characters of the part to be modified based on the previous context.

[0019] A second aspect of the present invention provides an OCR text recognition system based on seal removal, comprising:

[0020] A data acquisition module is configured to: acquire a target image;

[0021] A seal detection module is configured to: based on the target image, use a target detection network to obtain first category information of a first seal area and whether the seal covers text;

[0022] A seal removal module is configured to: use the first seal area as a mask, use the seal color range, remove the seal in the mask to obtain a second image, perform seal detection on the second image to obtain a second seal area and second category information, crop the second seal area to obtain multiple sub-images containing seals, and remove the seals from the sub-images based on the second category information using a deep learning model;

[0023] a text recognition module configured to: binarize the second image to obtain a third image, merge the sub-image after removing the seal onto the third image, detect the text area, and perform text recognition;

[0024] The semantic correction module is configured to correct the text recognition result using semantic correction to obtain the final text recognition result.

[0025] A third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-mentioned OCR text recognition method based on seal removal.

[0026] The fourth aspect of the present invention provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the OCR text recognition method based on seal removal as described above are implemented.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] The present invention provides an OCR text recognition method based on seal removal. Before OCR text recognition, it is necessary to first detect and remove the seal. The removal method used in this embodiment combines the UNet deep learning model based on Swin Transformer and the traditional HSV color space, ensuring the removal accuracy while improving the generalization of seal removal; at the same time, semantic correction is used to post-process the OCR recognition results, further improving the accuracy of the OCR recognition results. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0030] Figure 1This is a flow chart of an OCR text recognition method based on seal removal according to the first embodiment of the present invention;

[0031] Figure 2 This is a functional module diagram of an OCR text recognition system based on seal removal according to the second embodiment of the present invention. DETAILED DESCRIPTION

[0032] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0033] Explanation of terms:

[0034] OCR: OCR (Optical Character Recognition) is the abbreviation of optical character recognition, which refers to the process of using computers and optical equipment to convert printed characters into text that can be recognized by computers.

[0035] HSV: HSV is a color space that represents hue, saturation, and brightness. HSV is composed of three components: Hue, Saturation, and Value, and can be used to describe the properties of color. In HSV, hue represents the color's position on the color wheel, saturation represents the color's purity, and brightness represents the color's lightness or darkness.

[0036] RGB (Red, Green, Blue) is an additive colorimetric used by digital devices such as computer monitors and digital cameras. In RGB mode, the intensity of each color can be represented by a number between 0 and 255, and the combination of the three colors can create 16,777,216 different colors. This color model is most commonly used for color displays on electronic devices.

[0037] Swin Transformer: Swin Transformer is a neural network model based on the Transformers architecture. It was developed by Microsoft Research Asia and can be used for computer vision tasks such as image classification, object detection, and semantic segmentation.

[0038] UNet: UNet is an image segmentation model based on convolutional neural networks, which is usually used for semantic segmentation tasks, that is, assigning each pixel in the image to the corresponding category.

[0039] CSPDarknet67: CSPDarknet67 is a lightweight convolutional neural network model commonly used for object detection and image classification tasks. It improves upon the Darknet framework and uses the Cross Stage Partial (CSP) module to reduce model parameters while improving accuracy and speed.

[0040] Yolov5: Yolov5 is an object detection algorithm that is improved and optimized based on Yolov4. It uses an anchor-based object detection method to achieve fast and efficient object detection tasks.

[0041] Swin-UNet: Swin-UNet is an image segmentation model that combines the Swin Transformer and UNet architectures. It uses a decoder and encoder based on the Swin Transformer to segment the input image into multiple pixel-level sub-regions to obtain more accurate semantic segmentation results.

[0042] DB algorithm: The full name of the DB (Differentiable Binarization) algorithm is differentiable binarization processing. Segmentation-based scene text detection converts the probability map (heat map) generated by the segmentation method into bounding boxes and text areas.

[0043] SVTR Algorithm: Traditional text recognition algorithms typically consist of two modules: a visual module (such as CNN and MHSA) for feature extraction and a sequence module (such as RNN and Attention) for text output. SVTR is a scene text recognition algorithm based on a single visual model and has achieved good results in both Chinese and English text scene recognition.

[0044] BCE Loss: Binary cross entropy loss function, used for binary classification tasks.

[0045] Cross Entropy Loss: Cross entropy loss function, which can be used for both binary and multi-classification tasks.

[0046] Smooth L1 Loss: The smooth L1 loss function combines the advantages of L1 and L2, modifies the zero-point non-smoothness problem, and is more robust to outliers than L2 loss.

[0047] GIoU Loss: An improvement on the conventional IoU loss, used to measure the accuracy and regression ability of the model for bounding boxes. It not only considers the IoU between target bounding boxes, but also the overlap between target bounding boxes and the area difference between bounding boxes, thereby improving the accuracy and stability of target detection.

[0048] Mask: Select one or more areas on the original image to block the image to control the area of ​​image processing. The selected area is called a mask.

[0049] Gaussian weighted average: Gaussian weighted average is a special kind of weighted average that uses a template whose coefficients approximate a Gaussian distribution, hence the name.

[0050] Binarization: The process of setting each pixel on an image to 0 or 255 according to a given threshold.

[0051] Example 1

[0052] This embodiment provides an OCR text recognition method based on seal removal.

[0053] This embodiment provides an OCR text recognition method based on seal removal. In order to improve the accuracy of OCR recognition results, seals need to be detected and removed before OCR text recognition. The removal method used in this embodiment combines the UNet deep learning model based on Swin Transformer and the traditional HSV color space to ensure the removal accuracy while improving the generalization of seal removal. At the same time, semantic correction is used to post-process the OCR recognition results to further improve the accuracy of the OCR recognition results.

[0054] This embodiment provides an OCR text recognition method based on seal removal, such as Figure 1 As shown, the following steps are included:

[0055] Step 1: Data generation:

[0056] Get the target image, that is, the image that needs to be recognized.

[0057] As an implementation, based on a small number of real-world examples, we used HSV color space thresholding to obtain images containing only the seal and a background image. We then used the PIL library to generate text on the background image. We randomly selected one or more seals to cover different locations in the image, generating images with seals and corresponding images without seals for model training. We also generated configuration files for the corresponding images, including the center coordinates (x, y) of the seal area, the seal's width and height (w and h), and whether the seal covered the text (0 or 1).

[0058] Step 2: Seal detection:

[0059] Seal detection is performed using the Yolov5 object detection network, which consists of a CSPDarknet53 backbone network and three feature layers and detection layers at different scales. Seal detection also categorizes objects into covered and uncovered text.

[0060] During the training process, BCE Loss is used as the target detection loss to measure the accuracy of the model for the binary classification task of target detection;

[0061]

[0062] Among them, N represents the number of predicted targets in the picture, x i is the confidence of the model’s prediction of the i-th target, y i is its corresponding label (0 or 1).

[0063] Cross Entropy Loss is used as the target classification loss to measure the accuracy of the model for object classification tasks:

[0064]

[0065] Among them, N represents the number of seals detected in the image, C represents the number of seal categories, and y i,j is the label value (0 or 1) that the i-th seal belongs to the j-th category.

[0066] Smooth L1 Loss is used to measure the accuracy of the model for the target bounding box regression task;

[0067]

[0068]

[0069] Among them, x, y, w, and h are the center coordinates and width and height of the detected seal respectively, anchors are the pre-set anchor boxes, and objects are all detected objects in the image.

[0070] GIoU Loss is used to measure the accuracy and regression ability of the model for bounding boxes. It not only considers the IoU between target bounding boxes, but also the overlap between target bounding boxes and the area difference between bounding boxes, thereby improving the accuracy and stability of target detection:

[0071]

[0072]

[0073]

[0074] Among them, GIoU is the generalized intersection-over-union ratio between the predicted box and the true box, and c is a normalization factor used to normalize the range of GIoU. Represents the area of ​​the intersection between the predicted box and the true box, represents the area of ​​their joint part, Denote the square of their diagonal area.

[0075] For the predicted detection box, non-maximum suppression is used to determine the final detection box.

[0076] When the model predicts, it not only returns the relative position information of the seal (the first position information), but also returns the category information of whether the seal covers the text (the first category information). img 、h img The input image img is detected with the center coordinates of a seal as x, y, and the width and height as w, h. The model returns the position information (x / w img ,y / h img ,w / w img ,h / h img ), the position information does not need to change with the image size.

[0077] Step 3: Stamp removal:

[0078] After the seal detection is completed, firstly, the seal area (the first seal area, which is x / w img ,y / h img Centered on w / w img ,h / h img (width and height) as a mask, using the statistically obtained HSV color range of the seal (based on a small number of real samples and the statistics of the amplified image) to remove the seal that may exist in the mask, and obtain the second image img2. (For the original image after removing the seal from the mask, the color range is used. Because some seals may not be removed by the color range, it is necessary to perform line detection again.) Then, seal detection is performed on img2 (consistent with the seal detection method in step 2) to obtain the position information (i.e., second position information) and category information of the seal that still exists after removal using the HSV color space.

[0079] Based on the second category information, the seal area detected for the second time (i.e., the second seal area) is cropped to obtain multiple small images (sub-images) containing the seal, and the img2 image is binarized using an adaptive threshold algorithm based on Gaussian weighted average to obtain a black and white image img3 (the third image). The binarization method is as follows:

[0080] 1) Use a sliding window to split the image into 23×23 windows;

[0081] 2) Calculate the standard deviation of the pixel values ​​in the window as the standard deviation σ of the Gaussian function of the window;

[0082] 3) Calculate the weight of each pixel in the window:

[0083]

[0084] Among them, x0, y0 are the coordinates of the center point, and x, y are the coordinates of the point to be calculated;

[0085] 4) Calculate the weighted average T:

[0086]

[0087] Where s = 11, w i,j is the weight at the (i, j) coordinate, and I(i, j) is the pixel value of the coordinate;

[0088] 5) Set T-30 as the threshold t of the current window. When the pixel value of the (x, y) coordinate of the center value of the window is greater than or equal to t, change it to 255, otherwise change it to 0.

[0089] When the image edge cannot be calculated through the window threshold, the threshold of the pixel closest to the pixel is used instead.

[0090] Then, the category of the detected seal image is determined. If the seal does not cover the text, the pixel value of the small image is set to 255; if the text is covered, the seal is removed using an image segmentation model based on Swin Transformer (i.e., a deep learning model) to retain the text information.

[0091] The deep learning model uses Swin Transformer to extract high-level features, and uses UNet to generate prediction masks to classify each pixel to determine whether each pixel belongs to text or other categories. Pixels in the text category are set to 0, and pixels in other categories are set to 255 to remove the seal.

[0092] Compared with traditional UNet, Swin-Unet can extract features from multiple color space channels, and can also improve the accuracy and efficiency of image segmentation through multi-scale feature extraction and cross-stage connection technology.

[0093] Merge the small image after removing the seal to the corresponding position on img3 to obtain img4. Because the small image is cropped based on the detected seal rectangle, its position information is recorded; the image data is a two-dimensional matrix, so when merging, you only need to replace the matrix data of the small image with the data at the corresponding position in img3.

[0094] Step 4: Text recognition:

[0095] Use the text detection model based on the DB algorithm to detect the text area in the binarized image img4, then use the SVTR-based text recognition model to recognize the text in img4, and finally return the text and its coordinates in the original image.

[0096] Step 5: Semantic correction.

[0097] When a seal almost completely covers text, the seal removal process will inevitably remove some text information, affecting subsequent OCR recognition and causing inaccurate results. Furthermore, because OCR text recognition models cannot guarantee 100% accuracy, directly using OCR results can be risky. Using a semantic correction model to correct the recognition results can significantly improve accuracy and reduce risk.

[0098] Use a generative pre-trained language model to determine the likelihood of a word appearing in a sentence based on the context, and classify sentences into coherent and incoherent.

[0099] For incoherent sentences, calculate the probability p of the next character appearing from left to right according to the above context next , when p next When it is less than the preset threshold p0, the position is considered to be the position that needs to be corrected, and the position s is recorded. i , indicating the i-th position that needs to be modified. Then from s i The above process is repeated for the next character to obtain the first position set.

[0100] Similarly, from right to left, calculate the probability p of the previous character according to the following text pre , record the positions that need to be corrected and obtain the second position set.

[0101] The common parts in the two position sets are selected as the final parts to be modified. The language model is used to predict the possible characters of the part to be modified based on the above text. Then, the classification model (i.e., the generative pre-trained language model) is used to re-determine whether it is smooth. If it is smooth, the modification is considered successful. Otherwise, the modification continues.

[0102] The modified result is output as the final recognition result.

[0103] This embodiment provides an OCR text recognition method based on seal removal. Before OCR text recognition, seals that may affect the recognition results are removed. This removal method combines the traditional HSV color space with deep learning, ensuring effective removal while maintaining high generalizability.

[0104] The present embodiment provides an OCR text recognition method based on seal removal, which also uses a language model to correct the OCR recognition results, sets a higher modification threshold, and does not modify the correct results, thereby further ensuring the accuracy of text recognition.

[0105] Example 2

[0106] This embodiment provides an OCR text recognition system based on seal removal, such as Figure 2 As shown, it specifically includes:

[0107] (1) Data generation module (i.e. data acquisition module).

[0108] Based on a small number of real samples, we obtain a variety of different seals and backgrounds, generate new images by randomly combining the backgrounds and seals, and generate text on these new images to form training data. We train the model based on the pre-built model structure and the generated data.

[0109] (2) Seal detection module.

[0110] Normalize the input image, adjust the image size and number of channels, use Yolov5 with CSPDarknet67 as the backbone network to detect seals, and return the coordinate position and seal category of the seal area.

[0111] (3) Seal removal module.

[0112] The detected seal area is cropped, and different methods are used to remove the seal according to the type of the detected seal, while retaining the text information.

[0113] First, a color space threshold is used to remove seals that are easily distinguished from the text. Seal detection is then repeated. If the seal detected in the second pass does not cover the text, the area is simply set to a white background. If the seal covers the text, it is removed using the Swin-UNet model. After seal removal, the original image is binarized, and the cropped image with the seal removed is then reassembled into the original image to complete seal removal.

[0114] (4) Text recognition module.

[0115] First, use the DB algorithm model to detect text and determine the text area in the image. Then, crop the text area and use the SVTR algorithm model to recognize the cropped text image. Return the coordinates of all text areas in the image and the recognized text content.

[0116] (5) Semantic correction module.

[0117] When a portion of text is heavily obscured by a seal, removing the seal may still affect the text, affecting the OCR recognition results. Therefore, a language model is needed to correct the OCR recognition results to make the text recognition more accurate.

[0118] It should be noted here that the various modules in this embodiment correspond one-to-one to the various steps in Example 1, and the specific implementation processes are the same, which will not be repeated here.

[0119] Example 3

[0120] This embodiment provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the steps of the OCR text recognition method based on seal removal as described in the first embodiment above are implemented.

[0121] Example 4

[0122] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of the OCR text recognition method based on seal removal as described in the first embodiment above are implemented.

[0123] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage and optical storage, etc.) containing computer-usable program code.

[0124] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. An OCR text recognition method based on seal removal, characterized in that: include: Acquire the target image; Based on the target image, using an object detection network, obtaining first category information of a first seal area and whether the seal covers text; Using the first seal area as a mask and the seal color range, the seal in the mask is removed to obtain a second image. Seal detection is performed on the second image to obtain a second seal area and second category information. The second seal area is cropped to obtain multiple sub-images containing seals. Based on the second category information, the seals in the sub-images are removed using a deep learning model. If the second category information is text not covered by the seal, the pixel value of the sub-image is set to 255; if the second category information is text covered by the seal, the deep learning model classifies each pixel to determine whether each pixel belongs to text or other categories, and sets the pixel points of the text category to 0 and the pixel points of other categories to 255 to remove the seal; Binarizing the second image to obtain a third image, merging the sub-image after removing the seal onto the third image, detecting the text area, and performing text recognition; Use semantic correction to correct the text recognition results to obtain the final text recognition results; the use of semantic correction to correct the text recognition results is specifically: for the recognition results, use a generative pre-trained language model to judge the possibility of a certain word appearing in the sentence based on the sentence context, divide the sentences into fluent sentences and influent sentences, and correct the influent sentences.

2. The OCR text recognition method based on seal removal according to claim 1, characterized in that: The second image is obtained by using the statistically obtained HSV color interval of the seal and removing the seal in the mask.

3. The OCR text recognition method based on seal removal according to claim 1, characterized in that: The second image is binarized using an adaptive thresholding algorithm based on Gaussian weighted averaging.

4. The OCR text recognition method based on seal removal according to claim 1, characterized in that: For incoherent sentences, from left to right, based on the previous context, calculate the probability of the next character appearing at each position. When the probability is less than a preset threshold, the position is considered to be a position that needs to be corrected, and the first position set is obtained. At the same time, from right to left, based on the following context, calculate the probability of the previous character appearing at each position. When the probability is less than a preset threshold, the position is considered to be a position that needs to be corrected, and the second position set is obtained. Take the common part of the first position set and the second position set as the part to be modified, and use the language model to predict the characters of the part to be modified based on the previous context.

5. An OCR text recognition system based on seal removal, characterized in that: include: A data acquisition module is configured to: acquire a target image; A seal detection module is configured to: based on the target image, use a target detection network to obtain first category information of a first seal area and whether the seal covers text; A seal removal module is configured to: use the first seal area as a mask, use the seal color range, remove the seal in the mask to obtain a second image, perform seal detection on the second image to obtain a second seal area and second category information, crop the second seal area to obtain multiple sub-images containing seals, and remove the seals from the sub-images based on the second category information using a deep learning model; If the second category information is text not covered by the seal, the pixel value of the sub-image is set to 255; if the second category information is text covered by the seal, the deep learning model classifies each pixel to determine whether each pixel belongs to text or other categories, and sets the pixel points of the text category to 0 and the pixel points of other categories to 255 to remove the seal; a text recognition module configured to: binarize the second image to obtain a third image, merge the sub-image after removing the seal onto the third image, detect the text area, and perform text recognition; The semantic correction module is configured to use semantic correction to correct the text recognition results to obtain the final text recognition results; the correction of the text recognition results using semantic correction is specifically: using a generative pre-trained language model on the recognition results, judging the possibility of a certain word appearing in the sentence based on the sentence context, dividing the sentences into coherent sentences and incoherent sentences, and correcting the incoherent sentences.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the OCR text recognition method based on seal removal according to any one of claims 1 to 4 are implemented.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the OCR text recognition method based on seal removal according to any one of claims 1 to 4 are implemented.

Images