A gan-based task decomposition type image document denoising method

Abstract

The application discloses a task decomposition type image document denoising method based on GAN, adopts a DTNS algorithm based on conditional GAN, and divides the denoising process of an image format document into two parts of discovering an interference area and eliminating the interference area; wherein the discovering the interference area is to find all noise areas represented by watermarks and seals in the image, the eliminating the interference area is to remove the noise areas, and information in the image is restored to a state before being added with noise. The application proposes a DTNS algorithm based on conditional GAN, by decomposing the denoising task into two parts of watermark discovery and watermark removal, the task difficulty of directly removing the watermark through the GAN is greatly reduced, the network complexity is reduced, and by using Gaussian sampling in a high-dimensional structure feature space in the overlapping area, the phenomenon of losing the text contour of the overlapping area can be slowed down, so that the noise in the document image can be removed better.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of deep learning, and particularly relates to a task decomposition type image document denoising method based on GAN. BACKGROUND

[0002] There are a large number of image documents in enterprises, and the demand for extracting effective text information from images has derived the OCR (Optical Character Recognition) related technology. The existing OCR technology is generally divided into two parts of text detection and text recognition, as shown in FIG. 1. For clear and non-interfering general documents, the existing OCR related technology can normally detect and recognize the text. Figure 1

[0003] However, due to encryption, verification and other factors, there are a large number of documents interfered by watermarks or seals, as shown in FIG. 2, which makes the existing OCR method unable to accurately and completely perform text detection and recognition, and seriously affects the information extraction of image documents. Figure 2

[0004] At present, although there are methods of traditional image processing and deep learning image processing to eliminate the interference source, there are still the following shortcomings:

[0005] 1. Since the image document contains a large amount of text information, and the labeled image cannot lose the existing text information, the target area needs to be finely labeled pixel by pixel, and it is difficult to label when the target area is small, resulting in too high cost of pixel2pixel labeling of the denoising document. Most of the data that can be used are synthetic data, because there are too many possible data styles to completely enumerate, which cannot cover enough image sample space, and there will be obvious data bias.

[0006] 2. The traditional threshold-based image processing algorithm needs to be customized for different data scenarios, and is often affected by light, noise, watermark image, seal image, etc. through contrast, grayscale filtering processing, often needs to adapt different parameters for different images, and even some need to modify the details of the algorithm, which will seriously reduce the application range of the algorithm and cannot be used in the industrialization scene of image document processing.

[0007] 3. The traditional watermark removal algorithm based on wavelet transform often cannot normally decompose the text information in the body text and the text information in the watermark, which will confuse the text and watermark information, and the robustness is insufficient.

[0008] ​​4. Current deep learning image processing methods based on GAN (Generative Adversarial Network) directly generate documents with noise interference. The amount of information that needs to be restored in the documents to be processed is too large. The receptive field of images in deep learning is limited, which makes the design of the network very difficult. In addition, due to the instability of GAN, it often fails to converge properly during training.

[0009] 5. Existing methods often suffer from severe text outline loss in the vicinity of the overlapping area when processing the document and watermark overlapping areas because they cannot properly decompose the text signal and noise signal, which affects the subsequent text recognition process. Summary of the Invention

[0010] The purpose of this invention is to overcome the shortcomings of the existing technology and propose a DTNS (Document Text Noise Splitter) algorithm based on conditional GAN. By decomposing the denoising task into two parts, watermark discovery and watermark removal, the difficulty of removing watermarks directly through GAN is greatly reduced, the network complexity is reduced, and by using Gaussian sampling in the feature space of high-dimensional structure in overlapping regions, the phenomenon of text contour loss in overlapping regions can be mitigated, thereby better removing noise in document images.

[0011] To achieve the above-mentioned objectives, this invention provides a task-based image document denoising method based on GAN. It employs the DTNS algorithm based on conditional GAN ​​to divide the denoising process of image format documents into two parts: finding interference regions and eliminating interference regions. The finding interference regions involves identifying all noise regions in the image, represented by watermarks and stamps, while the eliminating interference regions involves removing the noise regions and restoring the information in the image to its state before noise was added.

[0012] Furthermore, the detection of interference regions is achieved using a mapping module, which includes a first discriminator and a first generator. The first discriminator is a multi-layer discriminator, and the first generator uses a skip-connection network structure.

[0013] Furthermore, the first discriminator employs a spectral-norm layer.

[0014] Furthermore, the size of the convolution kernel in the mapping module is 4.

[0015] Furthermore, the process of eliminating interference areas is as follows: the original image with added noise is merged with the noise image formed by the discovery of interference areas, so as to restore the part of the original image with added noise that was occluded by the noise image to the state before adding noise.

[0016] Further, the partial convolution is used when the image restoration is performed on the occluded image and the original image with noise.

[0017] Further, the high-level information in the feature extraction layer is taken as the structural feature, and the low-level information is taken as the texture feature, and the high-level information in the occluded part is sampled by the Gaussian to restore the state before the noise.

[0018] Further, the interference area elimination is realized by using the editing module, the editing module comprises a second discriminator and a second generator, the spectral-norm layer is used in the second discriminator, the skip-connection network structure is used in the second generator, and the resnet-block is used.

[0019] Further, the training process of the DTNS algorithm based on the conditional GAN is as follows: firstly, the mapping module and the editing module are trained separately, the mapping module is trained for 10 epochs, the editing module is trained for 20 epochs, the Adam optimizer is used in the process, and the learning rate is 1e-4; secondly, the mapping module and the editing module are jointly trained.

[0020] Compared with the prior art, the present application has the following advantages:

[0021] 1. By dividing the denoising task into two parts of finding the interference area and eliminating the interference area, the task difficulty is greatly reduced.

[0022] 2. By adjusting the convolution kernel of the generator part and the like, the size of the network is greatly reduced, and large-size images can be normally processed by using a P100 graphics card with 16G memory resources.

[0023] 3. A large amount of synthetic data can be used, and the data collection and labeling process which is difficult to operate is avoided. DETAILED DESCRIPTION

[0024] Figure 1 It is an OCR processing flowchart in the prior art;

[0025] Figure 2 It is a noisy document sample;

[0026] Figure 3 It is a flowchart of one embodiment of the present application;

[0027] Figure 4 It is a principle block diagram of the mapping module in one embodiment of the present application;

[0028] Figure 5 It is a principle block diagram of the first discriminator in the mapping module in one embodiment of the present application;

[0029] Figure 6A principle block diagram of the editing module in one embodiment of the present application;

[0030] Figure 7a A noise-added (stamped) image for training in one embodiment of the present application;

[0031] Figure 7b A noise-removed image in Figure 7a ;

[0032] Figure 7c A noise-removed image in Figure 7a ; Figure 7b

[0033] Figure 8a A noise-added (watermarked) image for actual test in one embodiment of the present application;

[0034] Figure 8b A noise image found in Figure 8a ;

[0035] Figure 8c A noise-removed (watermarked) image in Figure 8a ;

[0036] Figure 9a A noise-added (stamped) image for actual test in one embodiment of the present application;

[0037] Figure 9b A noise image found in Figure 9a ;

[0038] Figure 9c A noise-removed (stamped) image in Figure 9a . Embodiments

[0039] The technical solutions of the present application will be further described below in combination with the drawings and specific embodiments.

[0040] As shown in Figure 3 , one embodiment of the GAN-based task decomposition type image document denoising method of the present application adopts the DTNS algorithm based on conditional GAN to divide the denoising process of the image format document into two parts of finding interference area and eliminating interference area; wherein, the finding interference area is to find all noise areas represented by watermark and stamp in the image, and the eliminating interference area is to remove the noise areas to restore the information in the image to the state before being added with noise.

[0041] In one embodiment, as shown in Figure 4 and Figure 5 .As shown, the interference region is detected using a mapping module, which includes a first discriminator and a first generator. The first discriminator is a multi-layer discriminator, and the first generator uses a skip-connection network structure.

[0042] In one embodiment, the first discriminator employs a spectral-norm layer.

[0043] In one embodiment, the kernel size in the mapping module is 4.

[0044] In one embodiment, such as Figure 3 As shown, the process of eliminating interference areas is as follows: the original image with added noise is merged with the noise image formed by the discovery of interference areas, so as to restore the part of the original image with added noise that was occluded by the noise image to the state before adding noise.

[0045] In one embodiment, partial convolution is used when performing image restoration using an occluded image and a noisy original image.

[0046] In one embodiment, high-level information in the feature extraction layer is used as structural features, and low-level information is used as texture features. The state before noise addition is restored by Gaussian sampling of the high-level information in the occluded part.

[0047] In one embodiment, such as Figure 6 As shown, the interference region elimination is achieved using an editing module, which includes a second discriminator and a second generator; the second discriminator uses a spectral-norm layer; the second generator uses a skip-connection network structure and employs a resnet-block.

[0048] In one embodiment, the training process of the DTNS algorithm based on conditional GAN ​​is as follows: First, the mapping module and the editing module are trained separately. The mapping module is trained for 10 epochs, and the editing module is trained for 20 epochs. Both modules use the Adam optimizer with a learning rate of 1e-4. Second, the mapping module and the editing module are trained jointly. See the training effect diagram below. Figures 7a-7c The effect is obvious.

[0049] Mapping module

[0050] The main function of this module is to find the interference areas of noise such as watermarks and stamps in the image.

[0051] The module adopts the general pix2pix mode GAN training scheme, uses the network construction method of skip-connection to construct the generator part of the map module, and because it is found through experiments that the receptive field of the convolution kernel can be increased for feature extraction of the text. Therefore, in the actual scheme, the size of the convolution kernel is increased from 2 to 4, which can reduce the parameters of the network, so that the network can normally work on larger document images (for example, larger 2048x2048 images).

[0052] As shown in Figure 5 , the first discriminator part of the module samples a general multi-layer discriminator. Because the gradient explosion phenomenon of the discriminator is often found during the initial training, the batch-norm layer in it is modified to a spectral-norm layer.

[0053] The loss of this part of the network discriminator is composed of two parts:

[0054]

[0055] The L1 part represents the L1 loss of the predicted noise map and the actual noise map , and the GAN part represents the Conditional GAN loss between the predicted noise map and the input interference map , and is a normalization factor.

[0056] The loss of the first generator is only the corresponding Conditional GAN loss.

[0057] In this way, the watermark and seal noise interference areas can be obtained from the watermarked Figure 8a and the sealed Figure 9a , that is, the noise interference images shown in Figure 8b and Figure 9b .

[0058] Edit module (edit-module)

[0059] The main function of this module is to restore the original image into a denoised image using the watermark and noise interference areas and the original image.

[0060] Because this part of the module needs to restore the normal graph in the detail image.

[0061] The second generator part of this module also uses a skip-connection network structure, and uses a resnet-block as its basic block, which will be more helpful to capture detail features.

[0062] After extracting the image features, due to the occlusion of the interference area and other factors, we need to restore the image through the mask of the occluded part and the original image The convolution here uses the method of partial conv.

[0063] Since the high-level information in the feature extraction layer can be regarded as structural features, and the low-level information can be regarded as texture features, we use the Gaussian sampling of the high-level information in the occluded part to restore the interference area.

[0064] The second discriminator of this module also uses a spectral-norm layer to replace the batch-norm layer.

[0065] The loss of this part of the network is composed of three parts:

[0066]

[0067] The L1 part represents the L1 loss of the predicted denoised image and the original image containing noise The Gan part represents the Conditional GAN Loss between the predicted denoised image and the original image containing noise, and the third part is the structure loss based on the high-level features of vgg, which is also a normalization factor.

[0068] The loss of the second generator is only the corresponding Conditional GAN loss.

[0069] In this way, the image without watermark can be obtained by merging Figure 8a and Figure 8b , that is Figure 8c Similarly, the image without seal can be obtained by merging Figure 9a and Figure 9b , that is Figure 9c .

[0070] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and not to limit it, although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that the specific embodiments of the present application can be modified or replaced, without departing from the spirit and scope of the present application, any modification or equivalent replacement, which should be covered within the protection scope of the claims of the present application.

Claims

1. A task-based image / document denoising method based on GAN, characterized in that, The DTNS algorithm based on conditional GAN ​​is adopted to divide the denoising process of image format documents into two parts: finding interference regions and eliminating interference regions; among them, The interference region detection is to find all noise regions represented by watermarks and stamps in the image. The interference region detection is implemented by a mapping module, which includes a first discriminator and a first generator. The first discriminator adopts a multi-layer discriminator, and the first generator uses a skip-connection network structure. Eliminating interference regions involves removing the noise regions and restoring the information in the image to its state before noise addition. This elimination is achieved using an editing module, which includes a second discriminator and a second generator. The second discriminator employs a spectral-norm layer, and the second generator uses a skip-connection network structure and a ResNet-block architecture. The process of eliminating interference regions involves merging the original image with the noise region to restore the portion of the original image obscured by the noise image to its state before noise addition. High-level information from the feature extraction layer is used as structural features, and low-level information as texture features. Gaussian sampling of the high-level information in the obscured portion is then performed to restore the image to its state before noise addition. The training process of the DTNS algorithm based on conditional GAN ​​is as follows: First, the mapping module and the editing module are trained separately. The mapping module is trained for 10 epochs and the editing module is trained for 20 epochs. Both of them use the Adam optimizer and the learning rate is 1e-4. Second, the mapping module and the editing module are trained jointly.

2. The task-based image and document denoising method based on GAN according to claim 1, characterized in that, The first discriminator uses a spectral-norm layer.

3. The task-based image / document denoising method based on GAN according to claim 1, characterized in that, The kernel size in the mapping module is 4.

4. The task-based image / document denoising method based on GAN according to claim 1, characterized in that, When restoring an image using an occluded image and a noisy original image, a partial convolution method is employed.

Images