Image correction method not restricted by document picture

Abstract

The invention relates to the technical field of document picture processing, in particular to an image correction method not constrained by document pictures, which comprises the following steps of: S1, constructing a correction model, and obtaining a trained correction model after batch data training of a data set; the correction model mainly comprises a feature extraction network, a distortion encoder, a correction decoder and a code stream header, S2, inputting a distorted document image to be corrected into the correction model, predicting the offset of each pixel value of the distorted document image, and recovering the pixel values of the document image through the offset, and the distorted document picture is corrected and repaired. The method is not limited by document images, distorted document images in various real scenes can be processed at low cost, and the effect is better than that of other traditional image correction methods.

Description

Technical Field

[0001] The present invention relates to the technical field of document image processing, and in particular to an image correction method that is not constrained by document images. Background Art

[0002] Nowadays, the widespread use of smartphones has led to a growing trend of using them directly to digitize document files. Compared with traditional flatbed scanners, smartphones provide a more flexible, portable and simple alternative for document image digitization. Existing document image correction methods mainly target complete document images and are not very effective for incomplete or partial document images. The following are some commonly used methods for image correction:

[0003] 1. Technology based on 3D reconstruction: Such as 3D reconstruction technology, this method usually reconstructs the 3D shape of the distorted document and then flattens it into a flat image. Some methods use a structured light system to reconstruct the document model using a structured light 3D information acquisition system; there are also methods based on laser scanning, which use a laser range scanner for physical modeling; and some methods use multi-view images to capture multi-view images of the document for 3D reconstruction, but this method requires a complex image acquisition process, which limits its practical application.

[0004] 2. Parametric model-based techniques: These methods assume that there is some kind of parametric model of the document surface and optimize the model by extracting specific properties (such as shadows, boundaries, text lines, or texture flow). However, these simplified parametric models are usually unable to adapt to complex real-world geometric distortions, and the optimization process introduces significant computational costs.

[0005] 3. Methods based on deep models: In recent years, deep learning methods have gradually become the mainstream of document image correction, such as methods based on pixel-level displacement field prediction. Deep learning models are used to infer the pixel-level displacement field between distorted images and undistorted images to achieve fast image correction, such as: DocUNet: Use stacked U-Net to regress pixel-level displacement fields to quickly dedistort images. DocProj: Divide the input image into multiple patches and estimate the displacement field of each patch, but the final stitching process has a high computational cost.

[0006] However, most existing methods have limitations:

[0007] For example, technology based on 3D reconstruction requires collecting 3D structured light information of the image for image restoration and correction. The high cost makes this method unusable in general user scenarios.

[0008] However, popular deep learning algorithms such as DocUNet and DocProj can only process images containing complete document boundaries. Once the input image does not contain a complete boundary, the correction quality will drop significantly. Summary of the invention

[0009] The purpose of the present invention is to provide an image correction method that is not constrained by document images and can process distorted document images in various real scenes at low cost without being restricted by document images, and the effect is better than other traditional image correction methods.

[0010] In order to achieve the above object, an image correction method that is not constrained by document images is provided, comprising the following steps:

[0011] S1. Construct a correction model, and after batch data training of the data set, obtain a trained correction model; constructing the correction model specifically includes the following sub-steps:

[0012] S101, obtaining a distorted document image x;

[0013] S102, inputting the document image x into a feature extraction network for feature extraction to obtain a first feature graph f(x);

[0014] S103, adding the first feature map f(x) and the fixed 2D position embedding as the input of the warped encoder;

[0015] S104, after being encoded by the encoding module inside the twisted encoder, down-sampling is performed using the patch merging module; after the patch merging module reaches a pre-designed number of down-sampling times, the twisted encoder outputs a second feature map h(x);

[0016] S105, adding a fixed number of learnable embeddings to the second feature map h(x) as the input of the correction decoder;

[0017] S106, after being decoded by the decoding module inside the correction decoder, down-sampling is performed using the patch extension module; after the patch extension module reaches a pre-designed up-sampling number, the correction decoder outputs a third feature map g(x);

[0018] S107, in the bitstream header, upsample the input third feature map g(x) by a set number of times to restore the image shape to the original size, reduce the number of channels to a set value, and record the horizontal and vertical offset l(x) of each pixel;

[0019] S2. Input the distorted document image to be corrected into the correction model, predict the offset of each pixel value of the distorted document image, restore the pixel value of the document image through the offset, and realize the correction and repair of the distorted document image.

[0020] Further, the step S1 specifically includes the following steps:

[0021] S108, calculating the distance L1 between l(x) recording the offset of each pixel and the label y of the document image x, and minimizing the distance L1 is used as the loss function of the correction model. The calculation formula of the loss function is as follows:

[0022] loss=||l(x)-y||

[0023] The label y is the horizontal and vertical offset of each pixel in each document image x in the dataset after manual correction and annotation.

[0024] By minimizing the difference between the offset predicted by the model and the offset label, the model learns the ability to correct distorted pixels.

[0025] Furthermore, the feature extraction network is a convolution module, and the convolution module includes:

[0026] Input: an arbitrarily distorted three-channel document image x;

[0027] Residual block: The convolution module consists of 6 residual blocks connected in sequence;

[0028] Feature map downsampling: After every two residual blocks, the feature map will be downsampled to 1 / 2 resolution;

[0029] Output feature map: Finally, the feature map generated by the convolution module is f(x), where the channel is set to 256, indicating that the height and width of the output feature map is 1 / 8 of the original input image.

[0030] Further, the warp encoder is composed of three encoding modules and two patch merging modules, specifically a first encoding module, a first patch merging module, a second encoding module, a second patch merging module and a third encoding module connected in sequence;

[0031] Each encoding module consists of two standard Transformer layers, which consist of a multi-head self-attention module and a feed-forward network;

[0032] The first patch merging module downsamples the feature map output by the first encoding module, and the second patch merging module downsamples the feature map output by the second encoding module; finally, the third encoding module outputs the second feature map h(x), the number of channels remains unchanged, and the height and width of the second feature map h(x) are 1 / 32 of the original input image.

[0033] Further, the correction decoder is composed of three decoding modules and two patch extension modules, specifically a first decoding module, a first patch extension module, a second decoding module, a second patch extension module and a third decoding module connected in sequence;

[0034] Each decoding module contains two standard Transformer layers, which are also composed of a multi-head self-attention module and a feed-forward network;

[0035] The first patch expansion module upsamples the feature map output by the first decoding module, and the second patch expansion module upsamples the feature map output by the second decoding module; finally, the third decoding module outputs the third feature map g(x), the number of channels remains unchanged, and the height and width of the third feature map g(x) are 1 / 8 of the original input image.

[0036] Furthermore, in S107, three convolutional layers are used in the bitstream header for upsampling to restore the image shape to its original size, while reducing the number of channels to 2; the channel information includes, for each pixel, the horizontal and vertical offsets corresponding to them, denoted as l(x).

[0037] Principles and advantages:

[0038] 1. This solution uses images of distorted documents to predict the correction value of the image. Specifically, the designed model takes advantage of the great potential of transformers in establishing dependencies between sequential data, and uses attention to model the relationships between image blocks of the input image. In the process of downsampling, the distortion relationship from local details to global information is learned to capture the sequential relationship between text pixels in distorted document images. In the process of upsampling, the image block relationship captured by attention is also used to gradually predict the displacement information between global and local pixels. This solution innovatively uses the transformer model to model the features of distorted images, thereby improving the correction accuracy of distorted document images.

[0039] 2. This solution uses ordinary images: it avoids the use of complex and high-cost additional information, such as 3D structured light information, which greatly reduces the application threshold of the method and enables the present invention to be used on most mobile terminals.

[0040] Not limited by document images: This solution breaks through the limitation of distorted document images and can process distorted document images in various real scenes, making the effect of this solution much better than other traditional text correction methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A flowchart of an image correction method that is not constrained by a document image according to an embodiment of the present invention;

[0042] Figure 2 A flowchart for constructing the correction model;

[0043] Figure 3 Schematic diagram of the overall structure of the correction model. DETAILED DESCRIPTION

[0044] The following is further described in detail through specific implementation methods:

[0045] Example

[0046] An image correction method that is not constrained by document images is basically as follows Figure 1 , Figure 2 As shown, the following steps are included:

[0047] S1. Build a correction model, and after batch data training of the data set, obtain a trained correction model; Figure 3 As shown, the correction model consists of four parts: feature extraction network, distortion encoder, correction decoder and bitstream header. Constructing the correction model specifically includes the following sub-steps:

[0048] For the dataset: First, distort the document image as an image sample and batch operate to make a dataset, then manually correct the distorted and shifted characters in the image sample to the correct position, obtain the corrected position, and calculate the offset of the pixel in the original distorted document image as the label. The offset of each pixel includes horizontal offset and vertical offset, and the values ​​of the two are used as pixel labels. Finally, the dataset contains RGB distorted document image samples and pixel offset labels corresponding to the image samples.

[0049] S101, obtaining a distorted document image x from a data set;

[0050] S102: input the document image x into a feature extraction network for feature extraction to obtain a first feature graph f(x); the feature extraction network is a convolution module, and the convolution module includes:

[0051] Input: an arbitrarily distorted three-channel document image x;

[0052] Residual block: The convolution module consists of 6 residual blocks connected in sequence; the residual block is a deep learning architecture that introduces skip connections to alleviate the gradient vanishing problem in deep networks and promote the flow of information.

[0053] Feature map downsampling: After every two residual blocks, the feature map will be downsampled to 1 / 2 resolution;

[0054] Output feature map: Finally, the feature map generated by the convolution module is f(x), where the channel is set to 256, indicating that the height and width of the output feature map are 1 / 8 of the original input image. The number of channels is 256, which can capture rich feature information.

[0055] S103, adding the first feature map f(x) and the fixed 2D position embedding as the input of the warped encoder;

[0056] S104, after being encoded by the encoding module inside the twisted encoder, down-sampling is performed using the patch merging module; after the patch merging module reaches a pre-designed number of down-sampling times, the twisted encoder outputs a second feature map h(x);

[0057] Distorted Encoder: In order to correct distorted document images, their structural information must be captured effectively. This information is usually distributed in non-local areas, such as curved text lines and gradually changing lighting textures. Figure 2 As shown in Figure 1, this scheme introduces the self-attention mechanism in Transformer. This mechanism can effectively capture the structural features in the document because it can handle long-distance dependencies.

[0058] The input of the twisted encoder is the sum of f(x) and a fixed 2D position embedding. Since the Transformer architecture is permutation invariant, a fixed 2D position embedding needs to be added to the image features before inputting into the Transformer encoder layer. This can help the model understand the relationship between different positions in the image.

[0059] The warp encoder is composed of three encoding modules and two patch merging modules, specifically a first encoding module (Encoder Block 0), a first patch merging module (patch merging), a second encoding module (Encoder Block 1), a second patch merging module (patch merging) and a third encoding module (Encoder Block 1) connected in sequence;

[0060] Each encoding module contains two standard Transformer layers, which consist of a multi-head self-attention module and a feedforward network; the multi-head self-attention module allows the model to focus on input features from multiple perspectives, while the feedforward network is responsible for further processing these features. This internal hierarchical structure helps to extract features at different levels: high-resolution detail features can retain the detail information in the image, and low-resolution semantic features can extract stronger semantic information.

[0061] The first patch merging module downsamples the feature map output by the first encoding module, and the second patch merging module downsamples the feature map output by the second encoding module; finally, the third encoding module outputs the second feature map h(x), the number of channels remains unchanged, and the height and width of the second feature map h(x) are 1 / 32 of the original input image.

[0062] S105, adding a fixed number of learnable embeddings to the second feature map h(x) as the input of the correction decoder;

[0063] S106, after being decoded by the decoding module inside the correction decoder, down-sampling is performed using the patch extension module; after the patch extension module reaches a pre-designed up-sampling number, the correction decoder outputs a third feature map g(x);

[0064] The correction decoder is composed of three decoding modules and two patch expansion modules, specifically a first decoding module (Decoder block 0), a first patch expansion module (Patch Expanding), a second decoding module (Decoder block 1), a second patch expansion module (Patch Expanding) and a third decoding module (Decoderblock 2) connected in sequence;

[0065] Each decoding module contains two standard Transformer layers, which are also composed of a multi-head self-attention module and a feed-forward network;

[0066] The first patch expansion module upsamples the feature map output by the first decoding module, and the second patch expansion module upsamples the feature map output by the second decoding module; finally, the third decoding module outputs the third feature map g(x), the number of channels remains unchanged, and the height and width of the third feature map g(x) are 1 / 8 of the original input image.

[0067] S107. In the code stream head (Flow Head), the input third feature map g(x) is upsampled a set number of times to restore the image shape to the original size, and the number of channels is reduced to the set value, and the horizontal and vertical offsets l(x) of each pixel are recorded; in the S107, three convolutional layers are used in the code stream head for upsampling to restore the image shape to the original size, and the number of channels is reduced to 2; the channel information includes the horizontal and vertical offsets corresponding to each pixel, which are recorded as l(x).

[0068] S108, calculating the distance L1 between l(x) recording the offset of each pixel and the label y of the document image x, and minimizing the distance L1 is used as the loss function of the correction model. The calculation formula of the loss function is as follows:

[0069] loss=||l(x)-y||

[0070] The label y is the horizontal and vertical offset of each pixel in each document image x in the dataset after manual correction and annotation.

[0071] S2. Input the distorted document image to be corrected into the correction model, predict the offset of each pixel value of the distorted document image, restore the pixel value of the document image through the offset, and realize the correction and repair of the distorted document image.

[0072] There are two biggest benefits of this program:

[0073] 1. Ordinary images are used: complex and high-cost additional information, such as 3D structured light information, is avoided, which greatly reduces the application threshold of the method and enables the present invention to be used on most mobile terminals.

[0074] Not limited by document images: This solution breaks through the limitation of distorted document images and can process distorted document images in various real scenes, making the effect of this solution better than other traditional text image correction methods.

[0075] The above is only an embodiment of the present invention. The common sense such as the known specific structure and characteristics in the scheme is not described in detail here. The ordinary technicians in the relevant field know all the common technical knowledge in the technical field of the invention before the application date or priority date, can obtain all the existing technologies in the field, and have the ability to apply the conventional experimental means before that date. The ordinary technicians in the relevant field can improve and implement this scheme in combination with their own abilities under the enlightenment given by this application. Some typical known structures or known methods should not become obstacles for ordinary technicians in the relevant field to implement this application. It should be pointed out that for those skilled in the art, without departing from the structure of the present invention, several deformations and improvements can be made, which should also be regarded as the protection scope of the present invention, which will not affect the effect of the implementation of the present invention and the practicality of the patent. The protection scope required by this application shall be based on the content of its claims, and the specific implementation methods and other records in the specification can be used to interpret the content of the claims.

Claims

1. An image correction method that is not constrained by document images, characterized in that: The following steps are involved: S1. Construct a correction model, and after batch data training of the data set, obtain a trained correction model; constructing the correction model specifically includes the following sub-steps: S101, obtaining a distorted document image x; S102, inputting the document image x into a feature extraction network for feature extraction to obtain a first feature graph f(x); S103, adding the first feature map f(x) and the fixed 2D position embedding as the input of the warped encoder; S104, after being encoded by the encoding module inside the twisted encoder, down-sampling is performed using the patch merging module; after the patch merging module reaches a pre-designed number of down-sampling times, the twisted encoder outputs a second feature map h(x); S105, adding a fixed number of learnable embeddings to the second feature map h(x) as the input of the correction decoder; S106, after being decoded by the decoding module inside the correction decoder, down-sampling is performed using the patch extension module; after the patch extension module reaches a pre-designed up-sampling number, the correction decoder outputs a third feature map g(x); S107, in the bitstream header, upsample the input third feature map g(x) a set number of times to restore the image shape to the original size, reduce the number of channels to a set value, and record the horizontal and vertical offset l(x) of each pixel; S2. Input the distorted document image to be corrected into the correction model, predict the offset of each pixel value of the distorted document image, restore the pixel value of the document image through the offset, and realize the correction and repair of the distorted document image.

2. The image correction method according to claim 1, which is not constrained by document images, is characterized in that: The step S1 specifically includes the following steps: S108, calculating the distance L1 between l(x) recording the offset of each pixel and the label y of the document image x, and minimizing the distance L1 is used as the loss function of the correction model. The calculation formula of the loss function is as follows: loss=||l(x)-y|| The label y is the horizontal and vertical offset of each pixel in each document image x in the dataset after manual correction and annotation.

3. The image correction method according to claim 2, which is not constrained by document images, is characterized in that: The feature extraction network is a convolution module, and the convolution module includes: Input: an arbitrarily distorted three-channel document image x; Residual block: The convolution module consists of 6 residual blocks connected in sequence; Feature map downsampling: After every two residual blocks, the feature map will be downsampled to 1 / 2 resolution; Output feature map: Finally, the feature map generated by the convolution module is f(x), where the channel is set to 256, indicating that the height and width of the output feature map is 1 / 8 of the original input image.

4. The image correction method according to claim 3, which is not constrained by document images, is characterized in that: The warp encoder is composed of three encoding modules and two patch merging modules, specifically a first encoding module, a first patch merging module, a second encoding module, a second patch merging module and a third encoding module connected in sequence; Each encoding module consists of two standard Transformer layers, which consist of a multi-head self-attention module and a feed-forward network; The first patch merging module downsamples the feature map output by the first encoding module, and the second patch merging module downsamples the feature map output by the second encoding module; finally, the third encoding module outputs the second feature map h(x), the number of channels remains unchanged, and the height and width of the second feature map h(x) are 1 / 32 of the original input image.

5. The image correction method according to claim 4, which is not constrained by document images, is characterized in that: The correction decoder is composed of three decoding modules and two patch extension modules, specifically a first decoding module, a first patch extension module, a second decoding module, a second patch extension module and a third decoding module connected in sequence; Each decoding module contains two standard Transformer layers, which are also composed of a multi-head self-attention module and a feed-forward network; The first patch expansion module upsamples the feature map output by the first decoding module, and the second patch expansion module upsamples the feature map output by the second decoding module; finally, the third decoding module outputs the third feature map g(x), the number of channels remains unchanged, and the height and width of the third feature map g(x) are 1 / 8 of the original input image.

6. The image correction method according to claim 5, which is not constrained by document images, is characterized in that: In S107, three convolutional layers are used in the bitstream header for upsampling to restore the image shape to its original size and reduce the number of channels to 2; the channel information includes the horizontal and vertical offsets corresponding to each pixel, which are recorded as l(x).

Images