Image generation method and system for removing stamp noise based on mask guidance
By improving the design of the U-Net network, an accurate stamp mask is generated and background texture information is extracted, which solves the problem of reduced text recognition accuracy caused by stamp occlusion. This achieves more efficient stamp removal and background text preservation, and is highly adaptable.
Patent Information
- Application Number
- CN202310733846.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-20
- Publication Date
- 2026-02-27
- Estimated Expiration
- 2043-06-20
AI Technical Summary
Existing text recognition methods suffer from decreased accuracy and inability to effectively preserve background text when faced with seal occlusion, and there is a lack of publicly available training and evaluation datasets.
A U-Net-based convolutional neural network is used to generate an accurate mask and extract background texture information through a stamp localization module. Combined with a stamp erasure module, the background text is preserved while the stamp is removed. A global context attention module and an inflated mask are used to optimize the processing.
It improves the accuracy of text recognition, effectively removes stamp noise while preserving background text, and generates natural and highly adaptable images, demonstrating excellent experimental performance.
Smart Images
Figure CN116863476B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the field of image character recognition, in particular to a method and system for generating an image by removing seal noise based on mask guidance. BACKGROUND
[0002] Character recognition is an important task in the field of computer vision, and is widely used in the analysis of document images such as invoices and contracts. Although existing character recognition methods have satisfactory performance, the recognition accuracy is severely degraded due to the coverage of seals on the characters. This problem is very common in reality, but few people have conducted in-depth research on the solution to this problem. Some traditional solutions use color filtering or threshold segmentation to separate the seal and the background. Deep learning methods use a single deep U-Net or a CycleGAN-based method to automatically remove the seal. The problem is that traditional methods using color filtering or threshold segmentation cannot well adapt to different types of document images and seals, and are not practical. Current deep learning methods cannot well preserve the background text and are prone to erasing the background text. Moreover, there is no public dataset for training and evaluating the model. SUMMARY
[0003] The purpose of the present application is to provide a method and system for generating an image by removing seal noise based on mask guidance, which can automatically erase the seal that obscures the character information in the document image and interferes with image character recognition, and preserve the background text obscured by the seal, thereby preventing or reducing the negative impact of seal obscuration on the character recognizer, and further helping to more accurately recognize the characters in the document image.
[0004] To achieve the above purpose, the present application adopts the following technical solutions:
[0005] A method for generating an image by removing seal noise based on mask guidance, comprising the following steps:
[0006] The network structure of U-Net is modified, and a residual block is added as a bridge to connect the encoder and the decoder between the encoder and the decoder, to obtain a seal positioning module;
[0007] The network structure of U-Net is modified, and a global context attention module is added between the encoder and the decoder to obtain a seal erasing module;
[0008] The seal picture is input into the seal positioning module, the image abstract features are extracted by the encoder thereof, the seal mask is obtained according to the image abstract features by the decoder, and the background texture information features of the seal picture are extracted by the bridge;
[0009] The stamp image and stamp mask are input into the stamp erasure module, where the encoder extracts image features. These image features are then concatenated with the background texture information features and input into the global context attention module, where the decoder processes them to obtain the stamp-removed image.
[0010] Based on the stamp mask, replace the stamp area in the stamp image with the stamp area in the de-stamped image to obtain a combined image with the stamp removed.
[0011] A training dataset is constructed based on real images without seals and corresponding seal images. The seal localization module and the seal erasure module are trained simultaneously on this training dataset to optimize the loss.
[0012] The trained stamp localization module and stamp erasure module are used to process the stamp image to generate an image with stamp noise removed.
[0013] Preferably, the encoder of the stamp positioning module includes four downsampling layers, each of which consists of a residual block; the decoder includes four upsampling layers, each of which consists of a deconvolution and a residual block; and the residual blocks serving as bridges are three residual blocks.
[0014] Preferably, the backbone structure of all residual blocks in the stamp positioning module consists of two convolutions, and the skip connection consists of one convolution.
[0015] Preferably, the stamp erasing module includes four downsampling layers, each consisting of a convolution; the decoder includes four upsampling layers, each consisting of a deconvolution and a convolution.
[0016] Preferably, replacing the stamped area in the stamped image with the stamped area in the de-stamped image based on the stamp mask, to obtain a combined image with the stamp removed, includes the following steps:
[0017] The dot product between the stamp mask and the stamp-removed image is used to obtain the image information of the stamp region in the stamp-removed image.
[0018] The image information of the non-stamp area of the stamp image is obtained by performing a dot product between the stamp image and the inverted mask of the stamp mask.
[0019] The combined image is obtained by adding the image information of the stamped area in the stamped image and the image information of the non-stamped area in the stamped image.
[0020] Preferably, the stamp mask that is dot-producted with the destamped image is replaced with an expanded stamp mask, and subsequent processing is performed to obtain a combined image.
[0021] Preferably, the training dataset is constructed through the following steps:
[0022] A batch of real stamp-free images are stamped to obtain stamp images;
[0023] A training data set is formed by real stamp-free images and corresponding stamp images.
[0024] Preferably, the stamp positioning module optimizes training by calculating L1 loss and dice loss; the stamp erasing module optimizes training by calculating L1 loss, perceptual loss, style loss and GAN adversarial loss.
[0025] Preferably, the stamp positioning module and the stamp erasing module jointly constitute an end-to-end model and are trained simultaneously.
[0026] A stamp noise removing image generation system based on mask guidance, comprising:
[0027] The stamp positioning module is a convolutional neural network with an encoder-decoder structure modified based on U-Net; the encoder contains four down-sampling layers, each of which is composed of a residual block, for extracting image abstract features for stamp pictures; the decoder contains four up-sampling layers, each of which is composed of a deconvolution and a residual block, for obtaining a stamp mask according to the image abstract features; three residual blocks connecting the encoder and the decoder are provided as a bridge between the encoder and the decoder, for extracting background texture information features of the stamp picture;
[0028] The stamp erasing module is a convolutional neural network with an encoder-decoder structure modified based on U-Net; the encoder contains four down-sampling layers, each of which is composed of a convolution, for extracting image features; the decoder contains four up-sampling layers, each of which is composed of a deconvolution and a normal convolution, for outputting a stamp-removed picture; a global context attention module is provided between the encoder and the decoder, for receiving splicing information of the image features and the background texture information features and processing, and then outputting to the decoder.
[0029] The technical scheme of the present application has the following advantages:
[0030] The technical scheme provided by the application generates a mask capable of representing the position of the seal through the seal positioning module, avoids excessive erasing of the text in the non-seal area, and ensures that attention can be focused on the seal area. In order to better preserve the background text, the seal positioning module extracts the texture information of the background text, and transmits the texture information and the mask to the seal erasing module through skip connection, so that the seal erasing module can remove the seal while preserving the background text, and ensure accurate erasing of the seal. In order to make the finally generated image more natural, the application uses the dilated mask to calculate the loss. Experiments show that the application can achieve excellent performance on a real data set, and the system designed by the application is relatively light and can be flexibly used as a preprocessing module of the existing recognition method. BRIEF DESCRIPTION OF DRAWINGS
[0031] Figure 1 is a structure diagram of the seal noise removing image generation system based on mask guidance in the embodiment.
[0032] Figure 2 is a structure diagram of the residual block in the embodiment.
[0033] Figure 3 is a comparison diagram of the visualization results of the method and the baseline method of the application in the embodiment. DETAILED DESCRIPTION
[0034] In order to make the technical features and advantages or technical effects of the above technical scheme of the application more obvious and easy to understand, the following will be described in detail with reference to the drawings.
[0035] The embodiment provides a seal noise removing image generation method and system based on mask guidance. Figure 1 As shown in the figure, the system structure and the data flow diagram of the processing (the seal information in the figure is desensitized non-real information) are shown. The system mainly consists of two parts, which are a seal positioning module and a seal erasing module. The seal positioning module generates an accurate seal mask, and the seal erasing module erases the seal and restores the background text blocked by the seal. The two modules are described in detail below.
[0036] I. Seal positioning module
[0037] The input of the seal positioning module is a seal picture (see the input image in Figure 1 ), and the output is an accurate seal mask (see the predicted mask in Figure 1 ).
[0038] The stamp positioning module is a convolutional neural network based on a modified U-Net structure, i.e., still adopting the encoder-decoder structure of U-Net. The encoder contains four down-sampling layers, each of which is composed of a residual block, for gradually extracting abstract features of the image. The decoder contains four up-sampling layers, each of which is composed of a deconvolution and a residual block, for restoring the features extracted by the encoder to the original image size and generating a stamp mask with the same size as the original image. One of the modifications is to add three residual blocks as a bridge between the encoder and the decoder, i.e., the three residual blocks are located between the last down-sampling layer of the decoder and the first up-sampling layer of the encoder as a bridge, and the output features are used as background texture information for the subsequent erasing task.
[0039] As can be seen from the above description of the structure of the stamp positioning module, the residual block is the main component of the stamp positioning module, and the use of the residual block is conducive to the stability of model training. The second modification is a special design of the residual block structure. The specific structure of the residual block is shown in Figure 2 The main structure of the residual block consists of two convolutions: the first convolution is 3x3 in size and 2 or 1 in step (the step is set to 2 when down-sampling the features, and the step is set to 1 when up-sampling); the second convolution is 3x3 in size and 1 in step. The use of 2 convolutions in the main structure can increase the depth of the network, which is more conducive to feature extraction. A 3x3 convolution with a step of 2 is used in the skip connection of the residual block to adjust the feature size and match the feature size output by the main structure; if the feature size does not need to be adjusted, a 1x1 convolution with a step of 1 is used. This way can strengthen the feature extraction and integration between different layers. This structure design of the residual block can strengthen the feature extraction and integration between different down-sampling layers, the purpose of which is to better extract the texture information of the background text and improve the precision and robustness of the module.
[0040] II. Stamp erasing module
[0041] The stamp erasing module inputs the stamp image and the stamp mask output by the stamp positioning module, and outputs the de-stamped image after erasing the stamp (see the output image in Figure 1 ).
[0042] The stamp erasing module is a convolutional neural network based on a modified U-Net structure, that is, the encoder and decoder structures of the U-Net are still adopted. The encoder includes four down-sampling layers, each of which is composed of a normal convolution, for extracting image features. The decoder includes four up-sampling layers, each of which is composed of a deconvolution and a normal convolution, for outputting the stamp-removed picture. Each of the up-sampling and down-sampling layers only adopts a normal convolution, which is one of the modifications to the U-Net structure, to reduce the number of parameters. Another modification is to add a global context attention module (GC Block) between the encoder and the decoder, which receives the background texture features from the stamp positioning module and the image features extracted by the down-sampling layer, and then splices them in the channel. The GC Block is an attention module, and its output attention map represents which regions in the image are more important, so that the model can focus on the stamp region. The input of the background texture features can provide the texture information of the background text, so that the stamp erasing module can remove the stamp while preserving the background text as much as possible.
[0043] After the stamp erasing module outputs the stamp-removed picture, that is, the output image, in order to avoid the stamp erasing module modifying the non-stamp region too much, the stamp erasing module replaces the stamp region in the initial input stamp picture with the stamp region in the stamp-removed picture according to the stamp mask to obtain a combined image, which is taken as the final result. Specifically, the stamp erasing module performs a dot product of the stamp mask and the stamp-removed picture to obtain an image of the stamp region in the stamp-removed picture (other regions are black); performs a dot product of the stamp picture and the inverse mask of the stamp mask to obtain an image of the non-stamp region of the stamp picture (other regions are black); and then adds the two images to obtain the final combined image after complementation.
[0044] As a preferred embodiment, the stamp erasing module uses the mask after inflation to replace the original stamp mask (i.e. Figure 1 The preferred embodiment aims to enable the system to fill the stamp region with the pixels around the stamp, so that the generated image is more natural, and to use the inflated mask to avoid the situation that the stamp border is left due to unclear stamp boundaries.
[0045] The method proposed in this embodiment is based on the structure of the above system, and the entire process of stamp-removing processing of a picture containing stamp noise includes the following steps:
[0046] 1: input the seal picture into the seal positioning module to obtain an accurate mask representing the position of the seal, and extract the background texture information features in the module and input them into the seal erasing module.
[0047] 2: splice the seal picture and the mask on the channel and input them into the seal erasing module, splice the image features obtained after down-sampling with the background texture information features obtained from the seal positioning module on the channel, and then input them into the global context attention module, and then obtain the seal-removed picture through up-sampling.
[0048] 3: replace the seal area in the seal picture with the predicted seal area in the seal-removed picture according to the mask, especially the dilated mask, to obtain a combined image as the final result.
[0049] 4: the seal positioning module and the seal erasing module need to be trained, and can be used for formal seal erasing tasks only after the training is completed. When training, the loss needs to be calculated according to the training data set, i.e. the real seal-free image ground truth and the corresponding seal image. The seal positioning module is optimized and trained by calculating the L1 loss and the dice loss. The seal erasing module is optimized and trained by calculating the L1 loss, the perceptual loss, the style loss and the GAN adversarial loss. Although the two different modules use different loss functions for optimization, the system model composed of the two modules adopts an end-to-end manner, and the two modules can be trained simultaneously without the need for separate training.
[0050] The loss function of the seal positioning module is as follows:
[0051]
[0052] L loc Dice(m,m') + ||m-m'| |1
[0053] m and m' represent the model predicted mask and the ground truth respectively, and x and y represent the coordinates of the image.
[0054] The loss function of the seal erasing module is as follows:
[0055] L pix =10*||m*(I out -I gt )||1+2*||(1-m)*(I out -I gt )||1
[0056] I out and Igt These represent the destamped image and the ground truth output by the model, respectively.
[0057] I com =m*I out +(1-m)*I in
[0058]
[0059] φ n This represents the feature output of the nth pooling layer in VGG.
[0060]
[0061] Gram n This represents the Gram matrix calculated from the feature output of the nth pooling layer in VGG.
[0062] L adv =-Ε[D(G(I in ))]
[0063] G represents the generative model, and D represents the discriminative model.
[0064] Experimental test:
[0065] Extensive experiments were conducted to evaluate the effectiveness of the technical solution of this invention (taking the system proposed in this invention as an example). Before the experiments began, due to the lack of publicly available datasets related to seals, the inventors created a synthetic dataset as a training set for training the system. This synthetic dataset uses different types of document images collected from the internet as backgrounds, and then adds different types of seals to them, allowing the system to adapt to different situations. The seal shapes include circles, ellipses, and squares, and the colors include red and blue. This synthetic dataset contains a total of 19,055 images, containing 22,551 seals. To accurately evaluate the actual performance of the system, this experiment used a real dataset as the test set. This real dataset consists of real images with seals collected online and by taking photos, with the coordinates and labels of the text lines obscured by the seals annotated. This real dataset contains a total of 224 images, containing 400 seals and 616 text line annotations. For evaluation metrics, this experiment selected peak signal-to-noise ratio (PSNR), structural similarity (SSIM), and field accuracy and edit distance of the Paddle OCR.
[0066] The experimental test results are shown in Tables 1-3. Table 1 shows the effect comparison of the system of the present application under different settings, whether to expand the mask, whether to use the mask as the input of the stamp erasing module, and whether to use skip connection to pass the background texture information. The experimental results prove that the design of each part in the system of the present application (see the method column in Table 1) can significantly improve the performance, and the results of the "expansion + mask + skip connection" mode are the best.
[0067] Table 1: Comparison of ablation experiment results on real data set
[0068]
[0069]
[0070] Table 2 shows the result comparison after using 1 / 2 convolution to replace the residual block, and the results show that the residual block can more effectively extract the background texture information.
[0071] Table 2: Results of using 1 / 2 convolution to replace the residual block on real data set
[0072] Method Field accuracy Edit distance 1 convolution 57.79 0.9156 2 convolution 60.55 0.8782 Residual block 62.18 0.8198
[0073] In this experiment, the advanced scene text erasing method EraseNet and Stroke-base in the prior art are used as the baseline to compare with the present application, and the results are shown in Table 3. The results show that the output results of the present application are better than the above two baseline methods.
[0074] Table 3: Comparison results of the present application and the baseline on real data set
[0075] Method Field accuracy Edit distance EraseNet 57.79 0.9156 Stroke-base 60.55 0.8782 The invention 62.18 0.8198
[0076] Figure 2 is the comparison chart of the visualization results of the present application and the baseline, which shows that the present application not only has the ability to erase the stamp, but also can more effectively retain the background text, and the generated image is more natural.
[0077] Although the present application has been disclosed as above with examples, it is not intended to limit the present application, and any appropriate modification or equivalent replacement of the technical solutions of the present application made by those skilled in the art should be covered within the protection scope of the present application, and the protection scope of the present application is defined by the claims.
Claims
1. An image generation method for removing stamp noise based on mask guidance, characterized in that, Includes the following steps: The network structure of U-Net is modified by adding a residual block as a bridge between the encoder and decoder to obtain the stamp localization module. The encoder of the stamp localization module contains four downsampling layers, each of which consists of a residual block; the decoder contains four upsampling layers, each of which consists of a deconvolution and a residual block; the residual block serving as the bridge consists of three residual blocks. The network structure of U-Net is modified by adding a global context attention module between the encoder and decoder to obtain the stamp erasure module. The stamp erasure module contains four downsampling layers, each of which consists of a convolution. The decoder contains four upsampling layers, each of which consists of a deconvolution and a convolution. The stamp image is input into the stamp positioning module, where the encoder extracts the image abstract features, the decoder obtains the stamp mask based on the image abstract features, and the bridge extracts the background texture information features of the stamp image. The stamp image and stamp mask are input into the stamp erasure module, and its encoder extracts image features. The image features are then concatenated with the background texture information features and input into the global context attention module, which then processes the image through its decoder to obtain the de-stamped image. The process involves replacing the stamped area in the stamped image with the stamped area in the de-stamped image based on the stamp mask, resulting in a combined image with the stamp removed. Specifically, this includes: replacing the stamp mask with an expanded stamp mask and performing a dot product with the de-stamped image to obtain the image information of the stamped area in the de-stamped image; performing a dot product between the stamped image and the inverted mask of the expanded stamp mask to obtain the image information of the non-stamped area in the stamped image; and adding the image information of the stamped area in the de-stamped image and the image information of the non-stamped area in the stamped image to obtain the combined image. A training dataset is constructed based on real images without seals and corresponding seal images. The seal localization module and the seal erasure module are trained simultaneously using this training dataset to optimize the loss. The trained stamp localization module and stamp erasure module are used to process the stamp image to generate an image with stamp noise removed.
2. The method as described in claim 1, characterized in that, The backbone structure of all residual blocks in the stamp positioning module consists of two convolutions, and the skip connections consist of one convolution.
3. The method as described in claim 1, characterized in that, The training dataset is constructed using the following steps: Stamping is applied to a batch of real, unstamped images to obtain stamped images; The training dataset consists of real images without seals and their corresponding seal images.
4. The method as described in claim 1, characterized in that, The stamp localization module optimizes training by calculating L1 loss and dice loss; the stamp erasure module optimizes training by calculating L1 loss, perceptual loss, style loss and GAN adversarial loss.
5. The method as described in claim 1 or 4, characterized in that, The stamp positioning module and the stamp erasing module together constitute an end-to-end model, which is trained simultaneously.
6. A mask-guided image generation system for removing stamp noise, used to perform the method according to any one of claims 1-5, characterized in that, include: The seal localization module is a convolutional neural network based on a modified encoder-decoder structure of U-Net. The encoder contains four downsampling layers, each consisting of a residual block, which is used to extract abstract features from the seal image. The decoder contains four upsampling layers, each consisting of a deconvolution and a residual block, used to obtain the stamp mask based on the image abstraction features; three residual blocks are set between the encoder and decoder as bridges to extract the background texture information features of the stamp image. The stamp erasure module is a convolutional neural network based on a modified encoder-decoder structure of U-Net; the encoder contains four downsampling layers, each of which consists of a convolution to extract image features; The decoder contains four upsampling layers, each consisting of a deconvolution and a regular convolution, used to output the stamp-removed image. A global context attention module is located between the encoder and decoder, used to receive and process the concatenated information of image features and background texture information features, and then output it to the decoder.