Image preprocessing method, system, equipment, medium and product based on OCR

Abstract

The present invention relates to the field of image processing, and in particular to an image preprocessing method, system, device, medium, and product based on optical character recognition (OCR). The image preprocessing method based on OCR includes performing resolution enhancement, color space conversion, edge blurring, and morphological optimization on a rotationally corrected OCR image in sequence to obtain a fused image; extracting a text block using an OCR model based on the fused image, and outputting a processed OCR image if the text block is larger than a set threshold; otherwise, performing image diffusion, text recognition, and fusion processing based on the fused image using a TDSR model; image diffusion is performed under the condition of text to generate an image feature vector that conforms to the text image structure; text recognition is performed under the condition of the image feature vector of the image diffusion to obtain text features; and fusion processing fuses the image feature vector with the text features to enhance the super-resolution of the text image and obtain a processed OCR image. This improves recognition accuracy.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image preprocessing method, system, device, medium and product based on OCR. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] High-quality input images are the basis and key for OCR to correctly extract text information from images. Currently, most OCR methods simply adjust the image size to a size suitable for the model's input. However, the client is not aware of the OCR input format requirements and cannot simply adjust the image to a state suitable for OCR model processing. Therefore, in most cases, the input image is not suitable for direct calculation by the OCR model. This results in the OCR model being unable to accurately extract text information in many images. Therefore, the input image needs to be preprocessed to conform to the OCR input format.

[0004] Currently, due to factors such as shooting shake, overexposure / underexposure, and poor scanning quality, the images input into OCR have blurred text edges and overlapping strokes, making character features difficult to extract, which directly affects the recognition accuracy. Summary of the Invention

[0005] In order to solve the technical problems existing in the above-mentioned background technology, the present invention provides an image preprocessing method, system, device, medium and product based on OCR. After standardizing and enhancing the OCR image, the present invention uses an OCR model to extract the text block. If the text block is larger than a set threshold, the processed OCR image is output; otherwise, based on the fused image, the TDSR model is used to obtain the processed OCR image. For situations where strokes are adhered and character features are difficult to extract, the text can be accurately identified and extracted, thereby improving the quality of the OCR input image, increasing the text extraction accuracy when the input image has low resolution and incorrect orientation, and improving the performance and stability of the OCR model in application.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] A first aspect of the present invention provides an image preprocessing method based on OCR.

[0008] An image preprocessing method based on OCR, comprising:

[0009] Obtain the OCR image to be processed, standardize it, and perform rotation correction on the standardized OCR image;

[0010] The rotation-corrected OCR image is subjected to resolution enhancement, color space conversion, edge blurring, and morphological optimization in sequence, and then weighted fused with the resolution-enhanced image and the color space-converted image to obtain a fused image.

[0011] Based on the fused image, the OCR model is used to extract the text block. If the text block is larger than the set threshold, the processed OCR image is output; otherwise, based on the fused image, the TDSR model is used to perform image diffusion, text recognition and fusion processing; among them, image diffusion is performed under the condition of text to generate an image feature vector that conforms to the text image structure; text recognition is conditioned on the image feature vector of image diffusion to obtain text features; fusion processing fuses the image feature vector with the text features to enhance the super-resolution of the text image and obtain the processed OCR image.

[0012] Furthermore, the standardization method includes: calculating the mean and standard deviation of three channels of the OCR image to be processed to perform standardization processing on the three channels.

[0013] Furthermore, the standardized OCR image is subjected to rotation correction; the method includes: inputting the standardized OCR image into a trained image rotation angle classifier to obtain the rotation angle of the OCR image; and performing rotation correction on the standardized OCR image based on the rotation angle of the OCR image.

[0014] Furthermore, the rotationally corrected OCR image is sequentially subjected to resolution enhancement, color space conversion, edge blurring, and morphological optimization, and then weighted fusion is performed with the resolution-enhanced image and the color space-converted image to obtain a fused image. The method comprises:

[0015] The bicubic interpolation algorithm is used to enhance the resolution of the rotation-corrected OCR image to obtain a first enhanced image.

[0016] Performing color space conversion on the first enhanced image to obtain a second enhanced image;

[0017] Setting upper and lower thresholds, performing edge fuzzy feature selection and enhancement on the second enhanced image, and obtaining a third enhanced image;

[0018] performing morphological optimization on the edge region in the third enhanced image to obtain a fourth enhanced image;

[0019] The first enhanced image, the second enhanced image and the fourth enhanced image are weightedly fused to obtain a fused image.

[0020] Furthermore, the TDSR model is used, and the method includes:

[0021] Perform text encoding and variational autoencoding on the fused image respectively to obtain a first text feature and a first image feature;

[0022] Encoding the first text feature to obtain a second text feature;

[0023] Perform cross-attention calculation on the second text feature and the first image feature to obtain the first image feature vector;

[0024] Based on the first image feature vector, performing multi-level decoding on the first text feature to obtain text features of each layer;

[0025] The text features of the previous layer are encoded, and then cross-attention calculation is performed with the image feature vector of the previous layer until the penultimate layer is processed to obtain the final image feature vector, which is decoded to obtain the processed OCR image.

[0026] Furthermore, in the process of training the TDSR model, loss functions of the image diffusion and text recognition processes are constructed respectively, and weighted fusion is performed to construct a total loss function to optimize the parameters in the TDSR model training process.

[0027] A second aspect of the present invention provides an image preprocessing system based on OCR.

[0028] An image preprocessing system based on OCR, comprising:

[0029] The standardization and rotation module is configured to: obtain the OCR image to be processed, standardize it, and perform rotation correction on the standardized OCR image;

[0030] An image enhancement module is configured to sequentially perform resolution enhancement, color space conversion, edge blurring, and morphological optimization on the rotation-corrected OCR image, and then perform weighted fusion with the resolution-enhanced image and the color space-converted image to obtain a fused image;

[0031] The threshold judgment and super-resolution processing module is configured as follows: based on the fused image, the OCR model is used to extract the text block. If the text block is larger than the set threshold, the processed OCR image is output; otherwise, based on the fused image, the TDSR model is used to perform image diffusion, text recognition and fusion processing; wherein, image diffusion is performed under the condition of text to generate an image feature vector that conforms to the text image structure; text recognition is based on the image feature vector of the image diffusion to obtain text features; fusion processing fuses the image feature vector with the text features to enhance the super-resolution of the text image and obtain the processed OCR image.

[0032] A third aspect of the present invention provides a computer device, comprising:

[0033] a processor adapted to execute a computer program;

[0034] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the OCR-based image preprocessing method described in the first aspect are implemented.

[0035] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program is suitable for being loaded by a processor and executing the steps of the OCR-based image preprocessing method as described in the first aspect above.

[0036] A fifth aspect of the present invention provides a computer program product or computer program.

[0037] The present invention provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of the OCR-based image preprocessing method described in the first aspect above.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] The present invention provides an image preprocessing method, system, device, medium and product based on OCR. The image preprocessing method based on OCR includes: obtaining an OCR image to be processed, standardizing it, and performing rotation correction on the standardized OCR image; performing resolution enhancement, color space conversion, edge blurring processing and morphological optimization on the rotation-corrected OCR image in sequence, and then performing weighted fusion with the resolution-enhanced image and the color space-converted image to obtain a fused image; based on the fused image, using an OCR model to extract a text block, and if the text block is larger than a set threshold, outputting a processed OCR image; otherwise, based on the fused image, using a TDSR model to perform image diffusion, text recognition and fusion processing; wherein, image diffusion is performed under the condition of text to generate an image feature vector that conforms to the text image structure; text recognition is performed under the condition of the image feature vector of the image diffusion to obtain text features; fusion processing fuses the image feature vector with the text features to enhance the super-resolution of the text image and obtain a processed OCR image. The present invention adopts the TDSR model, which can accurately extract text even when strokes are adhered and character features are difficult to extract through the fusion process of image diffusion, text recognition and cross-attention calculation, thereby improving the accuracy of text extraction.

[0040] The present invention effectively solves the problem of low accuracy in text information extraction by OCR in low-quality images. An image preprocessing method is proposed based on the OCR calculation characteristics. Among them, an image direction classifier and corrector are proposed based on the Transformer-ResNet model, and a size preprocessing method is proposed based on the image characteristics. This improves the quality of OCR input images and effectively improves the accuracy of OCR text information extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0042] Figure 1 is a flow chart of an OCR-based image preprocessing method shown in an embodiment of the present invention;

[0043] Figure 2 is a specific flow chart of an OCR-based image preprocessing method shown in an embodiment of the present invention;

[0044] Figure 3 is a structural diagram of an image direction classification model shown in an embodiment of the present invention;

[0045] Figure 4 is a flow chart of low-resolution feature enhancement shown in an embodiment of the present invention;

[0046] Figure 5 1 is a flow chart of bicubic interpolation calculation for OCR preprocessing shown in an embodiment of the present invention;

[0047] FIG6 (a) is an example diagram of a character edge before low-resolution enhancement, shown in an embodiment of the present invention;

[0048] FIG6( b ) is an example diagram of a low-resolution enhanced character edge according to an embodiment of the present invention;

[0049] FIG7( a ) is an example diagram before feature enhancement and fusion shown in an embodiment of the present invention;

[0050] FIG7( b ) is an example diagram after feature enhancement and fusion shown in an embodiment of the present invention;

[0051] Figure 8 is a structural diagram of a TDSR model shown in an embodiment of the present invention;

[0052] Figure 9 Schematic diagram of a text feature controlled diffusion process according to an embodiment of the present invention;

[0053] Figure 10 is a schematic diagram showing how image features of a diffusion process affect text recognition according to an embodiment of the present invention;

[0054] FIG11( a ) is an example diagram of a low-resolution text block image according to an embodiment of the present invention;

[0055] FIG11( b ) is an example diagram of TDSR calculation results according to an embodiment of the present invention;

[0056] FIG11( c ) is an example diagram of a high-definition original image shown in an embodiment of the present invention;

[0057] Figure 12 is a structural diagram of an OCR-based image preprocessing system shown in an embodiment of the present invention;

[0058] Figure 13 It is a structural diagram of a computer device shown in an embodiment of the present invention. DETAILED DESCRIPTION

[0059] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0060] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0061] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0062] In order to facilitate understanding of the technical solution of the present invention, some technical terms involved in the present invention are introduced below.

[0063] Bicubic interpolation is a technique used to interpolate two-dimensional images or data. It is an extension of cubic interpolation to two dimensions. Bicubic interpolation has higher interpolation accuracy than bilinear interpolation because it considers a larger pixel neighborhood during calculation, resulting in smoother results.

[0064] The Text Block Diffusion Super-Resolution algorithm is designed to restore and extract information from text images with complex strokes and severe blur in real scenes. It is a super-resolution algorithm in which text recognition and image diffusion processes control each other. Text recognition can guide image diffusion to produce text images with correct structure, and the process of reconstructing text images can also continuously correct the text recognition results.

[0065] Based on this, the present invention provides an image preprocessing method, system, device, medium and product based on OCR. The following describes the solution of the present invention in detail through several embodiments:

[0066] Figure 1 is a flowchart of an OCR-based image preprocessing method according to an embodiment of the present invention; Figure 1 , the method comprising:

[0067] Obtain the OCR image to be processed, standardize it, and perform rotation correction on the standardized OCR image;

[0068] The rotation-corrected OCR image is subjected to resolution enhancement, color space conversion, edge blurring, and morphological optimization in sequence, and then weighted fused with the resolution-enhanced image and the color space-converted image to obtain a fused image.

[0069] Based on the fused image, the OCR model is used to extract the text block. If the text block is larger than the set threshold, the processed OCR image is output; otherwise, based on the fused image, the TDSR model is used to perform image diffusion, text recognition and fusion processing; among them, image diffusion is performed under the condition of text to generate an image feature vector that conforms to the text image structure; text recognition is conditioned on the image feature vector of image diffusion to obtain text features; fusion processing fuses the image feature vector with the text features to enhance the super-resolution of the text image and obtain the processed OCR image.

[0070] After standardizing and enhancing the OCR image, the present invention uses an OCR model to extract text blocks. If the text block is larger than a set threshold, the processed OCR image is output; otherwise, the TDSR model is used based on the fused image to obtain the processed OCR image. In the case of stroke adhesion and character features being difficult to extract, the text can be accurately identified and extracted, thereby improving the quality of the OCR input image, increasing the text extraction accuracy when the input image has low resolution and incorrect orientation, and improving the performance and stability of the OCR model in applications.

[0071] Figure 2 is a specific flow chart of the OCR-based image preprocessing method shown in an embodiment of the present invention; Figure 2, the method comprising:

[0072] Analyze the OCR image, calculate the mean and standard deviation of each channel of the image, and standardize it;

[0073] Use the image direction classification model to correct the image direction;

[0074] Perform low-resolution feature extraction and enhancement on the rectified image;

[0075] Extract text blocks when they are too blurry and traditional computer vision algorithms cannot compensate for them;

[0076] The confidence of the calculated output text block (i.e., the degree of certainty, the lower the confidence, the more likely it is to be wrong) will decrease. At this time, a reasonable threshold is set. When the confidence of the text block is too low, the text block is subjected to Text Block Diffusion Super-Resolution (TDSR) calculation to obtain a clearer text block image and output the text in the image.

[0077] In some embodiments, the method of analyzing the OCR image, calculating the mean and standard deviation of each channel of the image, and standardizing the image comprises:

[0078] Collect image files such as business licenses, permits, certificates, and documents commonly seen in daily business, calculate the mean and standard deviation of the three channels of the image, and then standardize them.

[0079] The normalization formula is:

[0080]

[0081] in, is the mean of the RGB values ​​in one channel of the image, is the standard deviation of the RGB values ​​in that channel.

[0082] After normalization, the distribution of the dataset is adjusted to a distribution with a mean of 0 and a standard deviation of 1. Normalized images are more suitable for model inference calculations. During model training, the values ​​of normalized data are more stable, which is conducive to model convergence and reduces the occurrence of gradient explosion.

[0083] In some embodiments, the method of using the image direction classification model to correct the image direction includes: using the collected 4478 various OCR images commonly used in daily business, rotating them by 90 degrees respectively. , 180 , 270 The rotation angles are processed and marked as labels. Then, a reduced-width TransformerEncoder-Resnet model is built and trained as a lightweight image rotation angle classifier. After the trained model inputs a standardized image, it outputs the rotation angle of the image.

[0084] A large number of business practices have found that most OCR models are very sensitive to the resolution of the input image. When the input image resolution is too low or the clarity is too low, the OCR model often performs poorly. The reason for this problem is that OCR models generally enlarge the lower-resolution image directly before inputting it into the model. This operation loses some image details and blurs key text features. Here, the bicubic interpolation algorithm is used to first expand the image resolution, add sufficient image details, and then reduce the image size to the required size for the model, ensuring the rich features of the input model image.

[0085] Figure 3 This is a structural diagram of the image direction classification model shown in an embodiment of the present invention; Figure 3 The image orientation classification model includes the ResNet-18 model, the Transformer encoder, and the Softmax function. The Transformer encoder is part of the Transformer model. Here, we first use a resized ResNet-18 model to perform convolution and downsampling on the images in the dataset. When the feature map size is suitable for Transformer calculations, we input the Transformer encoder for calculation. The output is then fed into the Softmax function to calculate the classification result, and the cross-entropy loss function is used to calculate the classification loss.

[0086] The lightweight Transformer-Resnet model proposed in this paper can quickly and accurately identify the rotation angle of an image, process the image into a format more suitable for OCR input, and improve the accuracy of subsequent text information extraction.

[0087] The following describes the training process of the image direction classification model:

[0088] Dataset preparation includes four parts: collection, cleaning, preprocessing, and partitioning. A total of 4,478 images containing invoices, licenses, and documents from businesses and the internet were collected. Duplicates and abnormal images containing a large amount of irrelevant information were removed. The images were resized and standardized, and the images were rotated 90° three times before being randomly divided into training and validation datasets in an 8:2 ratio.

[0089] The backbone network uses a simplified version of the Resnet18 model. The original model contains four convolutional modules, each with output feature maps with 64, 128, 256, and 512 channels, respectively. The image is downsampled by a factor of 16. To ensure model lightweight and avoid overfitting, the number of channels is reduced to one-fourth of the original, and the downsampling factor is adjusted to 8 to preserve more image detail. The backbone network takes an input image as input, extracts a low-resolution feature, and encodes a high-level spatial representation of the image, a sequence of dimensions length × width × number of channels.

[0090] The Transformer-Encoder has two standard layers, connected sequentially. Each layer includes a multi-head attention module and a fully connected layer, adding positional encoding to enable the model to capture positional information within the image. The Transformer-Encoder can learn rich feature representations, making it well-suited for a variety of computer vision tasks, such as image classification, object detection, and semantic segmentation. By stacking multiple layers and implementing a multi-head attention mechanism, the model can extract features from diverse perspectives.

[0091] The output of the Transformer-Encoder is passed through the fully connected layer and then input into the softmax module for classification probability calculation. The cross entropy loss function is used to calculate the classification loss so that the gradient descent algorithm can be used to optimize the model parameters.

[0092] The above image classification model will perform a series of inference calculations after receiving the image to be processed. The inference calculation does not involve updating the model parameters. The model only needs to calculate the image based on the trained parameters and weights. After the calculation is completed, the model will generate the direction recognition results of the image, which are 0 , 90 , 180 , 270 , then the Open-cv library or PIL library can rotate the image according to the angle to correct the image direction.

[0093] Experiments have found that among the examples with incorrect OCR results, most contain blurred images. Low-quality images will seriously affect the model's understanding of image information, especially for image features of text and numbers.

[0094] Figure 4 is a flowchart of low-resolution feature enhancement shown in an embodiment of the present invention; Figure 4 , extract and enhance low-resolution features of the rectified image; the method includes:

[0095] The corrected original image is input, and the bicubic interpolation algorithm is used to perform resolution enhancement on the rotation-corrected OCR image to obtain a first enhanced image; the first enhanced image is converted into a color space to obtain a second enhanced image; the upper and lower thresholds are set, and the edge fuzzy feature selection and enhancement are performed on the second enhanced image to obtain a third enhanced image; the edge area in the third enhanced image is morphologically optimized to obtain a fourth enhanced image; the first enhanced image, the second enhanced image, and the fourth enhanced image are weightedly fused to obtain a fused image.

[0096] Figure 5 is a flowchart of OCR preprocessing bicubic interpolation calculation shown in an embodiment of the present invention; Figure 5 The bicubic interpolation algorithm includes: performing cubic interpolation in the x direction of the OCR image, performing cubic interpolation on each of the four rows of pixels around the target point, and generating four intermediate interpolation results; performing cubic interpolation on the four difference results in the y direction of the OCR image.

[0097] The bicubic interpolation formula can be expressed as:

[0098]

[0099] in, is the new pixel value obtained by interpolation, are the 16 pixel values ​​in the neighborhood, is a weight function that determines the weight based on the distance, usually a cubic function is used.

[0100] The image preprocessing method proposed in the present invention for improving OCR recognition accuracy adopts a deep learning method and a bicubic interpolation algorithm to effectively improve the accuracy and stability of OCR in different situations.

[0101] Regarding Lab color space conversion, black text on a white background is common in text and document images. The Lab color space significantly improves the robustness and accuracy of color-related tasks (such as segmentation, detection, and enhancement) by separating brightness and color and providing a perceptually uniform color gamut. This is particularly advantageous when color differences are significant.

[0102] Regarding the threshold segmentation of text digital features: By calculating the threshold mean of the data set, setting an upper and lower threshold, the Lab color space image is subjected to edge fuzzy feature selection and enhancement according to the upper and lower thresholds. Figure 6 (a) is an example image of a character edge before low-resolution enhancement, as shown in an embodiment of the present invention; Figure 6 (b) is an example image of a character edge after low-resolution enhancement, as shown in an embodiment of the present invention; edge feature enhancement can be achieved by enhancing the brightness channel of the selected area.

[0103] The character edge low-resolution feature enhancement method proposed in the present invention greatly improves the text area quality of OCR images and enhances the performance of OCR processing variable-resolution images.

[0104] Morphologically optimizing the edge regions can eliminate small details, shrink boundaries, and remove broken connections. The interpolated super-resolved original image, the Lab channel image, and the enhanced character edge features are then weighted and added together to enhance and fuse specific features. Figure 7(a) shows an example image before feature enhancement and fusion, as shown in an embodiment of the present invention; Figure 7(b) shows an example image after feature enhancement and fusion, as shown in an embodiment of the present invention.

[0105] In some embodiments, the confidence level (i.e., degree of certainty, where lower confidence levels indicate a higher likelihood of error) of the text block output by the calculation may decrease. In this case, a reasonable threshold is set. When the confidence level of the text block is too low, a Text Diffusion Super-Resolution (TDSR) calculation is performed on the text block to obtain a clearer text block image and output the text in the image. The method includes:

[0106] After low-resolution feature enhancement, the first OCR calculation is performed to obtain the position and confidence of each text block. To ensure the correctness of the results, the present invention proposes a text block diffusion super-resolution algorithm (TDSR) to perform TDSR calculation on text blocks with confidence levels lower than a threshold to obtain clear text block images and the text information therein.

[0107] Figure 8 is a structural diagram of the TDSR model shown in an embodiment of the present invention; Figure 8 , respectively, the fused image is encoded using a text encoder and a variational autoencoder to obtain the first text feature and the first image feature; the first text feature is encoded using a self-attention encoder (Transformer-Encoder) to obtain the second text feature; the diffusion model (DiffBlock) is used to perform cross-attention calculation on the second text feature and the first image feature to obtain the first image feature vector; based on the first image feature vector, the self-attention decoder (Transformer-Decoder) is used to perform multi-level decoding on the first text feature to obtain the text feature of each layer; the text feature of the previous layer is encoded using a self-attention encoder, and then cross-attention calculation is performed with the image feature vector of the previous layer until the penultimate layer is processed to obtain the final image feature vector, which is decoded to obtain the processed OCR image.

[0108] Specifically, the TDSR model includes three parts: image diffusion process, text recognition process and fusion process; the image diffusion process is carried out under the condition of text to generate high-quality pictures that conform to the text image structure; the text recognition process is conditioned on the feature vector of image diffusion to achieve more accurate text recognition and correction; the output of the self-attention encoder and the output of each DiffBlock encode and fuse the text and image features in the previous step. Through the cooperation of the three parts, high-fidelity and high-realism text image super-resolution is finally achieved.

[0109] The above process can be expressed by the following formula:

[0110]

[0111]

[0112]

[0113]

[0114]

[0115]

[0116] Where t represents the tth time step, represents the control condition required for the diffusion process at the t-th time step (in the text recognition process, the control condition is the control condition provided in the image diffusion process; in the image diffusion process, the control condition is the control condition provided in the text recognition process), Represents the text sequence features generated by the text recognition process at the t-th time step, Represents the self-attention encoder computation. represents the conditions required for text recognition at the t-th time step, Indicates DiffBlock calculation, represents the features extracted by the variational autoencoder from the low-resolution image, represents a sample from the Gaussian distribution at the tth time step. represents the result of the image diffusion process at the t-th time step, Indicates the U-shaped network structure calculation of DiffBlock, represents the condition that controls the image diffusion process at the tth time step. It represents the artificially controllable noise scheduling in the image diffusion process, that is, the noise addition process. represents the prediction result of text recognition at the tth time step, Represents the self-attention decoder computation.

[0117] The image diffusion process uses the Latent Diffusion Model architecture. It's important to note that the diffusion process is controlled by the text information, achieved through the text feature vector output during the recognition process. The diffusion process receives a 512×128 input image (long text strip). Setting f=4 in the variational autoencoder downsamples the input image by a factor of four and expands the number of feature map channels to 320. This means that the variational autoencoder transforms the 512×128×3 input image into a 128×32×320 feature vector. This is followed by a standard diffusion model, where each DiffBlock is a Unet structure from the standard diffusion process.

[0118] The text block diffusion super-resolution algorithm proposed in the present invention can integrate specific text image structures into the image diffusion super-resolution process to achieve image reconstruction of complex structured numbers and letters.

[0119] Figure 9 is a schematic diagram of the text feature control diffusion process shown in an embodiment of the present invention; Figure 9 The image feature vector is input through the Encoder structure of Transformer, and the text feature vector is input through the Decoder structure of Traansformer. Finally, after cross-attention calculation, the image feature vector containing text features is output, down-sampled, and then the standard diffusion process calculation is continued in Unet.

[0120] The training uses n=200, which means it contains 200 time steps. The number of time steps determines the number of discrete time steps used in the forward process (from data to noise) and the reverse process (recovering data from noise). Each time step is associated with a specific variance value, which defines how much noise to add or remove in each step.

[0121] During text recognition, the Text Encoder uses an image encoder with a self-attention encoder structure to process a 512×128×3 image into a 128×32×320 feature vector, which is then fed into the Encoder and Decoder respectively. The Encoder encodes this feature vector into a control vector received by the DiffBlock during the diffusion process, and the Decoder learns the text features so that it can ultimately output the text information in the image based on the image features.

[0122] During the text recognition process, the self-attention decoder receives the image features generated during the diffusion process to achieve mutual communication between the image denoising and text recognition processes, so that the denoised image retains the structural information of the text, making the processed image more reasonable and clear.

[0123] Figure 10This is a schematic diagram showing how image features of a diffusion process affect text recognition according to an embodiment of the present invention; Figure 10 , the image feature vector and the diffusion process image feature vector, after the self-attention decoder calculation in the last time step, generates the identification vector corresponding to each character. The final self-attention decoder can also be regarded as a Text Decoder, corresponding to the Text Encoder at the beginning of the process.

[0124] During training, the VAE is loaded with weights pre-trained on Open-Image. The weights of the diffusion process and other modules in the text recognition process are trained from scratch. 100,000 iterations are trained on the CTR dataset with a batch size of 16. This model is calculated based on the low-confidence, low-resolution text blocks passed in the previous step. The low-resolution image blocks input to the model must be resized to 128×512, and can only process single-line text blocks.

[0125] The training consists of three parts. First, the self-attention decoder of the text recognition process is trained. The loss function is the KL divergence between the predicted text and the actual text in the dataset. The second part is to train the image diffusion model, where the loss function includes L2 loss and OCR loss. L2 loss is the loss function of the traditional diffusion model that enables the model to have noise estimation capabilities. OCR loss is the loss of the text extraction result after the super-resolved image is connected to the OCR model. The weight of the diffusion model is updated by freezing the weight of the OCR model, and the weight parameter is used to limit the proportion of OCR loss. Finally, the TDSR model is trained as a whole. At this time, the above two weights need to be frozen and the weights of the self-attention encoder are trained. The formula is as follows:

[0126]

[0127]

[0128]

[0129] in, represents the image diffusion loss, represents the L2 loss, represents the OCR loss, represents the weight parameter, represents the text recognition loss, Indicates the The string result of the prediction for each time step, Indicates the The string feature vector generated by time steps, represents the initial string feature vector, The probability distribution of two strings, represents the total loss, represents a hyperparameter.

[0130] Figure 11(a) shows an example of a low-resolution text block image, as shown in an embodiment of the present invention; Figure 11(b) shows an example of the TDSR calculation results, as shown in an embodiment of the present invention; Figure 11(c) shows an example of a high-resolution original image, as shown in an embodiment of the present invention; an example of a text block processed by the TDSR model is shown in Figure 11(b). A sample image with low OCR text information extraction accuracy was selected for testing. It can be seen that the results after and without preprocessing differ significantly.

[0131] Combination of the above Figure 1 The OCR-based image preprocessing method provided by the embodiment of the present invention is introduced in detail. Next, the OCR-based image preprocessing system provided by the embodiment of the present invention will be introduced with reference to the accompanying drawings.

[0132] Figure 12 This is a schematic diagram of the structure of an image preprocessing system based on OCR according to an embodiment of the present invention. Figure 12 , the system of the present invention comprises:

[0133] The standardization and rotation module is configured to: obtain the OCR image to be processed, standardize it, and perform rotation correction on the standardized OCR image;

[0134] An image enhancement module is configured to sequentially perform resolution enhancement, color space conversion, edge blurring, and morphological optimization on the rotation-corrected OCR image, and then perform weighted fusion with the resolution-enhanced image and the color space-converted image to obtain a fused image;

[0135] The threshold judgment and super-resolution processing module is configured as follows: based on the fused image, the OCR model is used to extract the text block. If the text block is larger than the set threshold, the processed OCR image is output; otherwise, based on the fused image, the TDSR model is used to perform image diffusion, text recognition and fusion processing; wherein, image diffusion is performed under the condition of text to generate an image feature vector that conforms to the text image structure; text recognition is based on the image feature vector of the image diffusion to obtain text features; fusion processing fuses the image feature vector with the text features to enhance the super-resolution of the text image and obtain the processed OCR image.

[0136] In some embodiments, the standardization and rotation module is specifically configured to calculate the mean and standard deviation of three channels of the OCR image to be processed to perform standardization processing on the three channels.

[0137] The standardized OCR image is input into a trained image rotation angle classifier to obtain the rotation angle of the OCR image; and the standardized OCR image is rotationally corrected based on the rotation angle of the OCR image.

[0138] In some embodiments, the image enhancement module is specifically configured to: use a bicubic interpolation algorithm to enhance the resolution of the rotationally corrected OCR image to obtain a first enhanced image; perform color space conversion on the first enhanced image to obtain a second enhanced image; set upper and lower thresholds to select and enhance edge blur features on the second enhanced image to obtain a third enhanced image; perform morphological optimization on the edge area in the third enhanced image to obtain a fourth enhanced image; and perform weighted fusion on the first enhanced image, the second enhanced image, and the fourth enhanced image to obtain a fused image.

[0139] In some embodiments, the threshold judgment and super-resolution processing module is specifically configured to: perform text encoding and variational autoencoding on the fused image respectively to obtain a first text feature and a first image feature; encode the first text feature to obtain a second text feature; perform cross-attention calculation on the second text feature and the first image feature to obtain a first image feature vector; based on the first image feature vector, perform multi-level decoding on the first text feature to obtain text features of each layer; encode the text feature of the previous layer, and then perform cross-attention calculation with the image feature vector of the previous layer until the penultimate layer is processed to obtain the final image feature vector, which is then decoded to obtain the processed OCR image.

[0140] In some embodiments, during the process of training the TDSR model, loss functions for the image diffusion and text recognition processes are constructed separately, and weighted fusion is performed to construct a total loss function to optimize the parameters during the TDSR model training process.

[0141] According to an embodiment of the present invention, the OCR-based image preprocessing system may correspond to executing the method described in the embodiment of the present invention, and the above and other operations and / or functions of each module of the OCR-based image preprocessing system are respectively to achieve Figure 1 For the sake of brevity, the corresponding processes of each method in are not repeated here.

[0142] See also Figure 13The structural diagram of the computer device shown in the figure includes a processor, a communication interface and a computer-readable storage medium. The processor, the communication interface and the computer-readable storage medium can be connected via a bus or other means. The communication interface is used to receive and send data. The computer-readable storage medium can be stored in the memory of the computer device, and the computer-readable storage medium is used to store a computer program. The computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer-readable storage medium. The processor (or CPU (Central Processing Unit)) is the computing core and control core of the computer device, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions to implement the corresponding steps in the embodiment of the OCR-based image preprocessing method.

[0143] This embodiment provides a computer-readable storage medium (Memory). This computer-readable storage medium is a memory device within a computer device that is used to store programs and data. It should be understood that the computer-readable storage medium herein may include both built-in storage media within the computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the processing system of the computer device.

[0144] Furthermore, the storage space also stores one or more instructions suitable for being loaded and executed by the processor. These instructions may be one or more computer programs (including program code). It should be noted that the computer-readable storage medium herein may be a high-speed RAM memory or a non-volatile memory, such as at least one disk storage device; alternatively, it may be at least one computer-readable storage medium located remotely from the processor.

[0145] In one embodiment, the computer-readable storage medium stores one or more instructions; the processor loads and executes the one or more instructions stored in the computer-readable storage medium to implement the corresponding steps in the above-mentioned OCR-based image preprocessing method embodiment.

[0146] This embodiment provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the corresponding steps of the above-mentioned embodiment of the OCR-based image preprocessing method.

[0147] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage and optical storage, etc.) containing computer-usable program code.

[0148] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0149] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0150] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0151] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a computer-readable storage medium. When executed, the program can include the processes in the above-described method embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0152] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. An image preprocessing method based on OCR, characterized in that: include: Obtain the OCR image to be processed, standardize it, and perform rotation correction on the standardized OCR image; The rotation-corrected OCR image is subjected to resolution enhancement, color space conversion, edge blurring, and morphological optimization in sequence, and then weighted fused with the resolution-enhanced image and the color space-converted image to obtain a fused image. Based on the fused image, the OCR model is used to extract the text block. If the text block is larger than the set threshold, the processed OCR image is output; Otherwise, based on the fused image, the TDSR model is used to perform image diffusion, text recognition, and fusion processing. Image diffusion is performed under the condition of text to generate an image feature vector that conforms to the text image structure. Text recognition is conditioned on the image feature vector of image diffusion to obtain text features. Fusion processing fuses the image feature vector with the text features to enhance the super-resolution of the text image and obtain the processed OCR image. The method of using the TDSR model includes: encoding the fused image using a text encoder and a variational autoencoder respectively to obtain a first text feature and a first image feature; encoding the first text feature using a self-attention encoder to obtain a second text feature; using a diffusion model to perform cross-attention calculation on the second text feature and the first image feature to obtain a first image feature vector; based on the first image feature vector, using a self-attention decoder to perform multi-level decoding on the first text feature to obtain the text feature of each layer; using a self-attention encoder to encode the text feature of the previous layer, and then performing cross-attention calculation with the image feature vector of the previous layer until the penultimate layer is processed to obtain the final image feature vector, which is then decoded to obtain the processed OCR image.

2. The OCR-based image preprocessing method according to claim 1, characterized in that: The method for performing standardization includes: calculating the mean and standard deviation of three channels of the OCR image to be processed to perform standardization processing on the three channels.

3. The OCR-based image preprocessing method according to claim 1, characterized in that: The method for performing rotation correction on the standardized OCR image comprises: inputting the standardized OCR image into a trained image rotation angle classifier to obtain the rotation angle of the OCR image; and performing rotation correction on the standardized OCR image based on the rotation angle of the OCR image.

4. The OCR-based image preprocessing method according to claim 1, characterized in that: The method includes sequentially performing resolution enhancement, color space conversion, edge blurring, and morphological optimization on the rotationally corrected OCR image, and then performing weighted fusion with the resolution-enhanced image and the color space-converted image to obtain a fused image. The bicubic interpolation algorithm is used to enhance the resolution of the rotation-corrected OCR image to obtain a first enhanced image. Performing color space conversion on the first enhanced image to obtain a second enhanced image; Setting upper and lower thresholds, performing edge fuzzy feature selection and enhancement on the second enhanced image, and obtaining a third enhanced image; performing morphological optimization on the edge region in the third enhanced image to obtain a fourth enhanced image; The first enhanced image, the second enhanced image and the fourth enhanced image are weightedly fused to obtain a fused image.

5. The OCR-based image preprocessing method according to claim 1, characterized in that: In the process of training the TDSR model, loss functions for the image diffusion and text recognition processes are constructed respectively, and weighted fusion is performed to construct a total loss function to optimize the parameters in the TDSR model training process.

6. The OCR-based image preprocessing system is characterized by: The OCR-based image preprocessing method according to any one of claims 1 to 5 comprises: The standardization and rotation module is configured to: obtain the OCR image to be processed, standardize it, and perform rotation correction on the standardized OCR image; An image enhancement module is configured to sequentially perform resolution enhancement, color space conversion, edge blurring, and morphological optimization on the rotation-corrected OCR image, and then perform weighted fusion with the resolution-enhanced image and the color space-converted image to obtain a fused image; The threshold judgment and super-resolution processing module is configured as follows: based on the fused image, the OCR model is used to extract the text block. If the text block is larger than the set threshold, the processed OCR image is output; otherwise, based on the fused image, the TDSR model is used to perform image diffusion, text recognition and fusion processing; wherein, image diffusion is performed under the condition of text to generate an image feature vector that conforms to the text image structure; text recognition is based on the image feature vector of the image diffusion to obtain text features; fusion processing fuses the image feature vector with the text features to enhance the super-resolution of the text image and obtain the processed OCR image.

7. A computer device, characterized in that: a processor adapted to execute a computer program; A computer-readable storage medium having a computer program stored therein, wherein when the computer program is executed by the processor, the steps of the OCR-based image preprocessing method according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor and executing the steps of the OCR-based image preprocessing method according to any one of claims 1 to 5.

9. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the computer program implements the steps of the OCR-based image preprocessing method according to any one of claims 1 to 5.

Images