Stamp Removal Method Based on Seal-Inception GAN Network

Abstract

This invention discloses a seal removal method based on a Seal-Inception GAN network, comprising the steps of: constructing and training a Seal-Inception GAN network; inputting an image containing a seal into the trained Seal-Inception GAN network for seal removal, and outputting the processed image. This invention utilizes the advantages of Cycle GAN's recurrent generation structure and unsupervised training, innovatively combining the Inception structure with the classic ResNet structure in the generator. Based on the characteristics of seal removal, the sampling structure and training dataset in the generator are adjusted to effectively preserve the handwritten characters in the seal portion while removing the seal from approval documents, indirectly improving the accuracy and efficiency of machine recognition.

Description

Technical Field

[0001] This invention relates to the field of image processing technology and is applicable to a stamp removal method based on a Seal-Inception GAN network. Background Technology

[0002] Seals are the most widely used tool for issuing certificates and verifying credit in today's society. The role of seals on approval documents is irreplaceable. In recent years, artificial intelligence technology has been applied in robotics, intelligent manufacturing, and leadership decision-making. Deep learning technology, represented by deep neural networks, has been developed. For existing machine recognition models of approval documents, certain requirements are needed for the electronic image information of the documents. However, as an image marker, a seal can directly interfere with the machine's recognition of the textual information on the approval documents, affecting its accuracy and efficiency. Therefore, removing seals from approval documents can directly improve the performance of machine recognition.

[0003] Currently, most research on invoice stamps focuses on stamp recognition and verification. Regarding the removal of invoice stamps, Ji Jingjing et al.'s paper, "Filtering of color seal on bank notes based on re-segmentation," utilizes a secondary segmentation algorithm based on color images to remove stamps. However, this method has certain requirements regarding the quality of the target image. Specifically, the stamp color must be uniform, and there should be no distortion or creases; the font color must be black. Therefore, the method presented in this paper has limited practicality. Cui Wencheng et al.'s paper, "Invoice Number recognition algorithm based on numerical structure characteristics," addresses the problem of interference factors such as stamp coverage and invoice creases affecting invoice number segmentation. It employs a recognition algorithm based on numerical structure characteristics, achieving invoice number recognition by repairing noisy, contiguous areas. This method can broaden the application range of secondary segmentation algorithms; however, when the number of invoices increases and the stamp color depth and position are inconsistent, this method has limitations in eliminating interference factors such as creases. Many image processing software programs on the market can also remove seals, but this method requires manual processing and a certain level of technical expertise from the user. Furthermore, removing the seal can blur the handwriting if there is handwritten text underneath. When dealing with a large volume of approval documents, this method is also extremely costly and wasteful of human and material resources. Summary of the Invention

[0004] Technical Objective: To address the shortcomings of existing technologies that use neural networks to remove stamps but simultaneously blur the text beneath the stamp, this invention discloses a stamp removal method based on the Seal-Inception GAN network. By employing the Seal-Inception GAN network for stamp removal, the handwritten characters in the stamp portion are effectively preserved, indirectly improving the accuracy and efficiency of machine recognition.

[0005] Technical solution: To achieve the above technical objectives, the present invention adopts the following technical solution.

[0006] A stamp removal method based on a Seal-Inception GAN network includes the following steps:

[0007] Construct and train the Seal-Inception GAN network;

[0008] The image containing the stamp is input into the trained Seal-Inception GAN network for stamp removal, and the processed image is output.

[0009] Preferably, the Seal-Inception GAN network is an improved generative adversarial network Cycle GAN, comprising a generator network and a discriminator. The generator network includes two generators with the same structure but different training objects, namely G... AtoB Generators and G BtoA The generator comprises a first convolutional module, a downsampling module, an improved residual module, an upsampling module, and a second convolutional module connected in sequence; the improved residual module is implemented by introducing an Inception structure into the existing residual module.

[0010] Preferably, the first convolutional module is a filled 7*7 convolution with 64 kernels and a stride of 1; the downsampling module includes three convolutional layers, all of which are 3*3 convolutions with a stride of 2, and the kernels of the three convolutional layers are 128, 256, and 512, respectively.

[0011] Preferably, the improved residual module includes six new residual convolutional layers, each with 512 kernels, a stride of 1, and all being 3*3 convolutions. Each new residual convolutional layer includes ResNet, Inceptionv1, and Inceptionv2, where ResNet is an existing residual module, and Inceptionv1 and Inceptionv2 are inserted into the existing ResNet. Both Inceptionv1 and Inceptionv2 include a filter concatenation unit and several convolutional units, and the output of the previous layer is connected to the filter concatenation unit through several convolutional units.

[0012] Preferably, the upsampling module includes three transposed convolutional layers, each of which is a 3*3 convolution with a stride of 2, and the number of kernels in the three transposed convolutional layers are 256, 128, and 64, respectively; the second convolutional module is a filled 7*7 convolution with 3 kernels and a stride of 1.

[0013] Preferably, training the Seal-Inception GAN network includes: acquiring a dataset and a test set. The dataset includes several images containing and without seals, and the test set includes several images containing seals. The dataset is first input into the Seal-Inception GAN network for training. During training, the network is trained for 200 epochs, with an input batch size of 2. The learning rate is 0.0002 for the first 100 epochs, and then linearly decreases from the 101st epoch onwards. During testing, images from the test set are input, and the trained pth model file is used for testing.

[0014] Preferably, before the dataset and test set are input into the Seal-Inception GAN network, the images in the dataset and test set are preprocessed to make the pixel colors of the stamp stand out by setting the RGB domain values.

[0015] Preferably, after preprocessing the images in the dataset and test set, images with seal depth covering the text and images with low contrast are flipped by 90°, 180° and 270° to expand the images in the dataset.

[0016] Preferably, the image containing the seal is input into the trained Seal-Inception GAN network for seal removal, and finally the image after seal removal is obtained.

[0017] Beneficial effects: This invention uses the Seal-Inception GAN network for seal removal, leveraging the advantages of Cycle GAN's recurrent generation structure and unsupervised training. It innovatively combines the Inception structure with the classic ResNet structure in the generator and adjusts the sampling structure and training dataset in the generator based on the high requirements for preserving details such as text in seal removal. This achieves the removal of seals from approval materials while effectively preserving the handwritten characters in the sealed parts, indirectly improving the accuracy and efficiency of machine recognition. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the overall process of the present invention;

[0019] Figure 2 This is a schematic diagram of the single-domain image processing flow of the Seal-Inception GAN network of the present invention;

[0020] Figure 3 This is a schematic diagram of the generator structure in the Seal-Inception GAN network of the present invention;

[0021] Figure 4 This is a schematic diagram of the novel residual convolutional layer structure in the Seal-Inception GAN network of the present invention;

[0022] Figure 5 This is a schematic diagram of the Inceptionv1 structure in the Seal-Inception GAN network of the present invention;

[0023] Figure 6 This is a schematic diagram of the Inceptionv2 structure in the Seal-Inception GAN network of the present invention;

[0024] Figure 7 This is a schematic diagram of a portion of the dataset during the simulation process of this invention;

[0025] Figure 8 This is a schematic diagram of a portion of the training set samples during the simulation process of this invention; where (a) and (b) are the original images, (c) and (d) are images after the seal color has been darkened, (e) and (f) are images containing the seal, and (g) and (h) are images without the seal;

[0026] Figure 9 The diagrams show the comparison of the effects of different network structures on stamp removal during the simulation process of this invention; where (a), (b), and (c) are the original images to be tested, (d), (e), and (f) are the effect images of stamp removal using the Cycle GAN network, (g), (h), and (i) are the effect images of stamp removal after downsampling using the Cycle GAN generator and adding an Inception Block, and (j), (k), and (l) are the effect images of stamp removal using the Seal-Inception GAN network of this invention. Detailed Implementation

[0027] The following description, in conjunction with the accompanying drawings, further illustrates and explains a stamp removal method based on a Seal-Inception GAN network according to the present invention.

[0028] As attached Figure 1 As shown, a stamp removal method based on a Seal-Inception GAN network includes the following steps:

[0029] Construct and train the Seal-Inception GAN network;

[0030] The image containing the seal is input into the trained Seal-Inception GAN network for seal removal, and the processed image, i.e., the image without the seal, is output.

[0031] In some embodiments of the present invention, before inputting the image containing the stamp into the trained Seal-InceptionGAN network for stamp removal, the image containing the stamp can be preprocessed to make the stamp pixel color stand out by setting RGB domain values.

[0032] like Figure 2 As shown, the Seal-Inception GAN network is an improved generative adversarial network, Cycle GAN, comprising a generator network and a discriminator. The generator network includes two generators with the same structure but different training objects, namely G... AtoB Generators and G BtoA The generator, from the perspective of single-domain or stamp removal, takes an input image containing a stamp and measuring 512×512×3. After one scaling adjustment and three downsampling operations in the generator, the number of channels becomes 512. In the six-stage residual block structure, the number of image channels remains unchanged, and then after three upsampling operations, the number of channels becomes 64. Finally, after scaling adjustment, the image channels become 3, thus generating an output image without the stamp and measuring 512×512×3. The generator structure includes a first convolutional module, a downsampling module, an improved residual module, an upsampling module, and a second convolutional module connected in sequence. The improved residual module is implemented by introducing an Inception structure into the existing residual module. This invention innovatively introduces the Inception structure to form a new ResNet module, which can achieve the effect of stamp removal with a slight increase in parameters. Simultaneously, this network can improve image contrast to a certain extent.

[0033] like Figure 2 As shown, image X contains a seal. A Through G AtoB The generator produces the image G after the stamp has been removed. AtoB (X A ), G AtoB (X A ) and X A Together they enter the discriminator to calculate the adversarial loss, and G... AtoB (X A Enter G BtoA Generator generation and X A Similar images containing seals G BtoA(G AtoB (X A Then through G BtoA (G AtoB (X A )) and X A The similarity loss, also known as the cycle consistency loss, is calculated. The total loss is obtained by adding the cycle consistency loss and the adversarial loss. For an image X that does not contain a stamp... B Through G BtoA The generator then produces an image G containing the seal. BtoA (X B ), G BtoA (X B ) and X B Together they enter the discriminator to calculate the Adversarial loss in another domain, while G... BtoA (X B Enter G AtoB Generator generation and X B Similar images containing seals G AtoB (G BtoA (X B Then through G AtoB (G BtoA (X B )) and X B The similarity loss for another domain is calculated.

[0034] like Figure 3 As shown, the generator structure includes a first convolutional module (ReflectionPadConvolutional Layer), downsampling layers, improved residual modules (New Resnetblocks), upsampling layers, and a second convolutional module (ReflectionPadConvolutional Layer) connected in sequence.

[0035] The first convolutional module is a filled 7*7 convolution, i.e., ReflectionPad Convolutional Layer, with 64 kernels (64 filters) and a stride of 1. The downsampling module includes three convolutional layers, i.e., Convolutional Layers, all of which are 3*3 convolutions with a stride of 2. The number of kernels in the three convolutional layers are 128, 256, and 512, respectively. The improved residual module includes six new residual convolutional layers, i.e., New Residual Block (Convolutional); all of which have 512 kernels, a stride of 1, and are all 3*3 convolutions.

[0036] like Figures 4-6 As shown, each new residual convolutional layer includes ResNet, Inceptionv1, and Inceptionv2. ResNet is the existing residual module, and Inceptionv1 and Inceptionv2 are inserted into the existing ResNet. Both Inceptionv1 and Inceptionv2 include a filter concatenation unit and several convolutional units. The output of the previous layer is connected to the filter concatenation unit through several convolutional units.

[0037] In some embodiments of the present invention, the convolutional unit in Inceptionv1 includes two 1x1, 128-kernel convolutional layers, two 1x1, 32-kernel convolutional layers, one 3x3, 128-kernel convolutional layer, one 5x5, 128-kernel convolutional layer, and one 3x3 pooling layer. The output of the previous layer is connected to the input of the 1x1, 128-kernel convolutional layer, the input of the two 1x1, 32-kernel convolutional layers, and the input of the 3x3 pooling layer, respectively. 1. The output of the convolutional layer with 32 convolutional kernels is connected to the input of the convolutional layers with 3*3 and 128 convolutional kernels, respectively, and the input of the convolutional layers with 5*5 and 128 convolutional kernels. The output of the 3*3 pooling layer is connected to the input of the convolutional layer with 1*1 and 128 convolutional kernels. Finally, the outputs of the convolutional layers with 1*1 and 128 convolutional kernels, the convolutional layers with 3*3 and 128 convolutional kernels, the convolutional layers with 5*5 and 128 convolutional kernels, and the convolutional layers with 1*1 and 128 convolutional kernels are connected to the filter concatenation unit. The convolutional units in Inceptionv2 include two 1x1 convolutional layers with 128 kernels, one 1x1 convolutional layer with 64 kernels, one 1x1 convolutional layer with 32 kernels, two 3x3 convolutional layers with 128 kernels, one 3x3 convolutional layer with 64 kernels, and one 3x3 pooling layer. The output of the previous layer is connected to the inputs of the 1x1 and 128 kernel convolutional layers, the 1x1 and 64 kernel convolutional layers, the 1x1 and 32 kernel convolutional layers, and the 3x3 pooling layer, respectively. The output of the convolutional layer is connected to the input of the 3x3, 128-kernel convolutional layer. The output of the 1x1, 32-kernel convolutional layer is connected to the input of the 3x3, 128-kernel convolutional layer through the 3x3, 64-kernel convolutional layer. The output of the 3x3 pooling layer is connected to the input of the 1x1, 128-kernel convolutional layer. Finally, the outputs of the 1x1, 128-kernel convolutional layer, the 3x3, 128-kernel convolutional layer, the 3x3, 128-kernel convolutional layer, and the 1x1, 128-kernel convolutional layer are connected to the filter concatenation unit.

[0038] The upsampling module consists of three transposed convolutional layers; each of the three transposed convolutional layers is a 3*3 convolution with a stride of 2, and the number of convolutional kernels in the three transposed convolutional layers are 256, 128, and 64, respectively; the second convolutional module is a filled 7*7 convolutional layer, i.e., a ReflectionPad Convolutional Layer, with 3 convolutional kernels, i.e., 3 filters, and a stride of 1.

[0039] In the generator structure of this invention, the first step is to use a padded 7x7 convolution and downsampling block module, followed by feature extraction through a residual block structure. Finally, upsampling blocks and 7x7 padded convolutions are performed to restore the image to a 3-channel RGB image. This generator introduces an Inception structure to maintain the residual block depth while increasing its network width, enabling the residual blocks to perform diverse feature extraction and better extract stamps for removal. Simultaneously, increasing the number of convolutional blocks in the downsampling process allows for the extraction of more textual information from the image, making the feature extraction of the New Residua Block module more targeted. Multi-layer sampling combined with the New Residua Block structure further improves the stamp removal capability while preserving the text texture information of the material, while maintaining network performance.

[0040] This invention employs a Seal-Inception GAN network for stamp removal, leveraging the advantages of Cycle GAN's recurrent generation structure and unsupervised training. It innovatively combines the Inception structure with the classic ResNet structure in the generator and adjusts the sampling structure and training dataset based on the high requirements for preserving text details in stamp removal. The sampling structure adjustment process involves: Sampling at a downsampling factor of 2 resulted in incomplete stamp removal in some areas. An attempt was made to adjust the downsampling factor to 3, which showed greater stamp removal. Following this pattern, an attempt was made to adjust the downsampling factor to 4, but this resulted in excessive stamp removal, causing text to disappear from areas covered by the stamp. Therefore, adjusting the downsampling factor to 3 maximizes stamp removal while preserving text details. As the sampling factor increases, the generator needs to capture more feature information. The Inceptionv1 module is utilized to increase network width without changing network depth. In conjunction with the Inceptionv2 module, while further enhancing the network's feature extraction capabilities, the Inceptionv2 module replaces one 5×5 convolutional kernel in the Inceptionv1 module with two 3×3 convolutional kernels, achieving the same receptive field while reducing the number of network parameters. To ensure that the features extracted by each Inception module are fully utilized without making the network structure too wide, the Inception concatenation module is combined with the ResNet structure to form a new ResNet module. This module effectively removes seals from approval documents while preserving the handwritten characters in the seal portion, indirectly improving the accuracy and efficiency of machine recognition.

[0041] The Seal-Inception GAN network improves the learning ability of the network by adjusting the symmetrical sampling structure and setting the number of convolution kernels in the residual block group to 512, which allows the residual module to extract more seal and text information.

[0042] Training the Seal-Inception GAN network involves: acquiring a dataset and a test set. The dataset includes several images containing and without seals, while the test set includes several images containing seals. First, the images containing and without seals from the dataset are input into the Seal-Inception GAN network for training. During testing, images from the test set (containing seals) are used to test the trained pth model file. The Seal-Inception GAN network is trained for 200 epochs, with a batch size of 2. The learning rate is 0.0002 for the first 100 epochs, and then decreases linearly from the 101st epoch onwards. Before the dataset and test set are input into the Seal-Inception GAN network, the images in the training set (including the dataset and test set) are first preprocessed. By setting the RGB domain values, the pixel colors of the stamp are made prominent. Then, images with stamp depth covering the text and images with low contrast are flipped by 90°, 180° and 270° to expand the special data images.

[0043] Simulation verification:

[0044] The simulation conditions for this invention are as follows: the hardware environment is a 64-bit Ubuntu 20.04.3 LTS operating system, and the CPU is... Core TM The system uses an i9-10900K CPU (3.70GHz x 20) and an NVIDIA GeForce RTX 3080 graphics card. The deep learning framework used is PyTorch 1.10.1. The network is trained end-to-end.

[0045] The simulation process is described below with reference to a stamp removal method based on the Seal-Inception GAN network of this invention:

[0046] (1) Data collection:

[0047] The dataset for this invention comes from a real, uncontrolled environment. The dataset contains 1150 images with seals, 1253 images without seals, and 71 images for the test set. The samples are also subject to interference from various factors such as ambient lighting, resolution, and shooting angle; some sample images may appear... Figure 7As shown, this model has low requirements for the images of approval materials and has a wide range of applications.

[0048] (2) Network preprocessing and data augmentation:

[0049] Because some of the stamps in the samples collected in this dataset have relatively light colors, the RGB values ​​of the dataset were first set to make the stamp pixels stand out. Simultaneously, this invention also performed data augmentation on the dataset to increase the number of training samples. Part of the training set is shown below. Figure 8 As shown.

[0050] (3) Selection of network initialization parameters and design of training details:

[0051] Because the training set is diverse, with some training images containing handwritten characters and varying seal colors, sizes, and depths, the ratio of downsampling to upsampling is set to 1:1, and the sampling factor is 3. This ensures that the network can extract more information from the input images while preventing the over-extraction of other information.

[0052] During the training of the Seal-Inception GAN network, the number of training epochs was 200, the input batch size was 2, and the learning rate was 0.0002 for the first 100 epochs. From the 101st epoch onwards, the learning rate decreased linearly with each epoch. Furthermore, comparative experiments on four different initialization parameters (normal, xavier, kaiming, and orthogonal) showed that xavier initialization was the most effective for seal removal.

[0053] (4) Network test analysis:

[0054] Stamp removal models were trained using Cycle GAN, a concatenated structure with an Inception Block added after downsampling in the Cycle GAN generator, and the Seal-Inception GAN network structure proposed in this invention. To highlight the generation effects of different networks, the experiments employed a controlled variable approach, ensuring that all factors except network structure remained largely the same (including network initialization parameters and training datasets). Model training and result testing were then conducted on each of the three types of networks. Results analysis and comparison are provided below. Figure 9As shown, (a), (b), and (c) are the original images to be tested, and handwritten characters are present under the stamp in (a) and (c). (d), (e), and (f) are the effect images of stamp removal using the Cycle GAN network. (g), (h), and (i) are the effect images of stamp removal after downsampling using the Cycle GAN generator and adding an Inception Block. (j), (k), and (l) are the effect images of stamp removal using the Seal-Inception GAN network of this invention. Figure 9 As can be seen, while the Cycle GAN network can remove stamps, it cannot eliminate all stamp marks in some images, and it generates incomplete or missing text images for certain text areas covered by stamps. The method of adding a concatenated Inception Block after downsampling in the Cycle GAN generator not only removes the stamp but also removes a lot of text information, making this method unsuitable for stamp removal. In contrast, the Seal-Inception GAN network of this invention can remove stamps more comprehensively, and the degree of text preservation in stamp-covered areas is much higher than that of Cycle GAN. Finally, compared with the original image, the image generated by the Seal-Inception GAN network has a certain improvement in contrast, which helps to improve the accuracy of machine recognition.

[0055] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A stamp removal method based on a Seal-Inception GAN network, characterized in that, Includes the following steps: Construct and train the Seal-Inception GAN network; The image containing the seal is input into the trained Seal-Inception GAN network for seal removal, and the processed image is output. The Seal-Inception GAN network is an improved generative adversarial network, Cycle GAN, which includes a generator network and a discriminator. The generator network consists of two generators with the same structure but different training objects, namely G... AtoB Generators and G BtoA A generator, the structure of which includes a first convolutional module, a downsampling module, an improved residual module, an upsampling module and a second convolutional module connected in sequence; the improved residual module is implemented by introducing an Inception structure into the existing residual module. The improved residual module includes six new residual convolutional layers, each with 512 kernels, a stride of 1, and all being 3x3 convolutions. Each new residual convolutional layer includes ResNet, Inceptionv1, and Inceptionv2. ResNet is an existing residual module that includes two 3x3 convolutional layers. Inceptionv1 and Inceptionv2 are inserted between the two 3x3 convolutional layers. Both Inceptionv1 and Inceptionv2 include a filter concatenation unit and several convolutional units. The output of the previous layer is connected to the filter concatenation unit through several convolutional units.

2. The stamp removal method based on Seal-Inception GAN network according to claim 1, characterized in that: The first convolutional module is a filled 7*7 convolution with 64 kernels and a stride of 1. The downsampling module includes three convolutional layers, each of which is a 3*3 convolution with a stride of 2. The number of kernels in the three convolutional layers are 128, 256, and 512, respectively.

3. The stamp removal method based on the Seal-Inception GAN network according to claim 1, characterized in that: The upsampling module includes three transposed convolutional layers, each of which is a 3*3 convolution with a stride of 2. The number of kernels in the three transposed convolutional layers are 256, 128, and 64, respectively. The second convolutional module is a filled 7*7 convolution with 3 kernels and a stride of 1.

4. The stamp removal method based on the Seal-Inception GAN network according to claim 1, characterized in that, Training the Seal-Inception GAN network involves: acquiring a dataset and a test set. The dataset includes several images containing and without seals, while the test set includes several images containing seals. The dataset is first input into the Seal-Inception GAN network for training. During training, the network undergoes 200 epochs with a batch size of 2. The learning rate is 0.0002 for the first 100 epochs, and then decreases linearly from the 101st epoch onwards. For testing, images from the test set are input, and the trained pth model file is used for testing.

5. The stamp removal method based on the Seal-Inception GAN network according to claim 4, characterized in that: Before the dataset and test set are input into the Seal-Inception GAN network, the images in the dataset and test set are preprocessed to make the pixel colors of the stamp stand out by setting the RGB domain values.

6. The stamp removal method based on the Seal-Inception GAN network according to claim 5, characterized in that: After preprocessing the images in the dataset and test set, images with seal depth covering the text and images with low contrast were flipped by 90°, 180° and 270° to expand the images in the dataset and test set.

7. The stamp removal method based on the Seal-Inception GAN network according to claim 1, characterized in that: Before the stamp-containing image is input into the trained Seal-Inception GAN network for stamp removal, the image containing the stamp is preprocessed by setting RGB domain values ​​to make the stamp pixel color stand out.

Images