Training of adversarial networks, image watermarking removal method, device and electronic equipment

Abstract

The disclosure provides training of a generative adversarial network, an image watermark removal method and device, and an electronic device, and relates to the field of image processing, and in particular to the field of artificial intelligence or computer vision. The specific implementation scheme is as follows: a training sample set is obtained, the training sample set including a sample watermark image; a watermark removal generative adversarial network and a mask generative adversarial network are obtained; the sample watermark image is input into a first down-sampling sub-network to obtain a first down-sampling feature map, and the sample watermark image is input into a second down-sampling sub-network to obtain a second down-sampling feature map; the first down-sampling feature map and the second down-sampling feature map are spliced, and the spliced feature map is input into a first up-sampling sub-network to obtain a sample watermark image after watermark removal; based on the sample watermark image after watermark removal, a first generator and a first discriminator are trained to obtain a trained watermark removal generative adversarial network. Based on the scheme, the watermark in the image can be effectively removed, which helps to improve the accuracy of image content recognition results.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of image processing, in particular to the technical field of artificial intelligence or computer vision, and more particularly to a training method of a generative adversarial network, an image watermark removal method and device, and an electronic device. BACKGROUND

[0002] In daily life, there are often watermarks in document images, such as a seal watermark in a bill image.

[0003] When a document image is used, the content (such as text) in the document image generally needs to be recognized, and the watermark in the document image will affect the accuracy of the content recognition result. SUMMARY

[0004] The present disclosure provides a training method of a generative adversarial network, an image watermark removal method and device, and an electronic device to solve at least one of the above problems.

[0005] According to a first aspect of the present disclosure, a training method of a generative adversarial network is provided, which comprises:

[0006] obtaining a training sample set, the training sample set comprising a sample watermark image containing a watermark;

[0007] obtaining a watermark removal generative adversarial network and a mask generative adversarial network, wherein the watermark removal generative adversarial network comprises a first generator and a first discriminator, the first generator comprising a first down-sampling sub-network and a first up-sampling sub-network; the mask generative adversarial network comprising a second generator, the second generator comprising a second down-sampling sub-network, the mask generative adversarial network being configured to generate a mask post-image of the sample watermark image;

[0008] inputting the sample watermark image into the first down-sampling sub-network to obtain a first down-sampling feature map, and inputting the sample watermark image into the second down-sampling sub-network to obtain a second down-sampling feature map;

[0009] splicing the first down-sampling feature map and the second down-sampling feature map to obtain a spliced feature, and inputting the spliced feature map into the first up-sampling sub-network to obtain a watermark removal post-sample watermark image;

[0010] training the first generator and the first discriminator based on the watermark removal post-sample watermark image to obtain a trained watermark removal generative adversarial network.

[0011] According to a second aspect of the present disclosure, an image watermark removal method is provided, which comprises:

[0012] obtaining a to-be-processed image;

[0013] obtain a first generator of a watermark removal adversarial network and a second generator of a mask adversarial network, the watermark removal adversarial network being trained based on the training method of the adversarial network, the first generator comprising a first down-sampling sub-network and a first up-sampling sub-network, the second generator comprising a second down-sampling sub-network, the mask adversarial network being configured to generate a mask post-image of the to-be-processed image;

[0014] input the to-be-processed image into the first down-sampling sub-network to obtain a third down-sampling feature map, and input the to-be-processed image into the second down-sampling sub-network to obtain a fourth down-sampling feature map;

[0015] splice the third down-sampling feature map and the fourth down-sampling feature map to obtain a target spliced feature map, and input the target spliced feature map into the first up-sampling sub-network to obtain a watermark removal post-image of the to-be-processed image.

[0016] According to a third aspect of the present disclosure, an adversarial network training device is provided, which comprises:

[0017] a training sample set obtaining module configured to obtain a training sample set, the training sample set comprising a sample watermark image, the sample watermark image containing a watermark;

[0018] a model obtaining module configured to obtain a watermark removal adversarial network and a mask adversarial network, wherein the watermark removal adversarial network comprises a first generator and a first discriminator, the first generator comprising a first down-sampling sub-network and a first up-sampling sub-network; the mask adversarial network comprises a second generator, the second generator comprising a second down-sampling sub-network, the mask adversarial network being configured to generate a mask post-image of the sample watermark image;

[0019] a down-sampling module configured to input the sample watermark image into the first down-sampling sub-network to obtain a first down-sampling feature map, and input the sample watermark image into the second down-sampling sub-network to obtain a second down-sampling feature map;

[0020] an up-sampling module configured to splice the first down-sampling feature map and the second down-sampling feature map to obtain a spliced feature, and input the spliced feature into the first up-sampling sub-network to obtain a watermark removal post-sample watermark image;

[0021] a model training module configured to train the first generator and the first discriminator based on the watermark removal post-sample watermark image to obtain a trained watermark removal adversarial network.

[0022] According to a fourth aspect of the present disclosure, an image watermark removal device is provided, which comprises:

[0023] a to-be-processed image obtaining module configured to obtain a to-be-processed image;

[0024] The model obtaining module is configured to obtain a first generator of a watermark removal adversarial network and a second generator of a mask adversarial network, the watermark removal adversarial network being trained based on the training method of the adversarial network, the first generator comprising a first down-sampling sub-network and a first up-sampling sub-network, and the second generator comprising a second down-sampling sub-network, the mask adversarial network being configured to generate a mask post-image of the to-be-processed image;

[0025] The down-sampling module is configured to input the to-be-processed image into the first down-sampling sub-network to obtain a third down-sampled feature map, and input the to-be-processed image into the second down-sampling sub-network to obtain a fourth down-sampled feature map.

[0026] The watermark removal image generation module is configured to splice the third down-sampled feature map and the fourth down-sampled feature map to obtain a target spliced feature map, and input the target spliced feature map into the first up-sampling sub-network to obtain a watermark removal post-image of the to-be-processed image.

[0027] According to a fifth aspect of the present disclosure, an electronic device is provided, which comprises:

[0028] at least one processor; and

[0029] a memory in communication with the at least one processor; wherein

[0030] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the training of the adversarial network or the image watermark removal method.

[0031] According to a sixth aspect of the present disclosure, a non-transitory computer readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to perform the training of the adversarial network or the image watermark removal method.

[0032] According to a seventh aspect of the present disclosure, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the training of the adversarial network or the image watermark removal method.

[0033] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present disclosure, nor to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0034] The accompanying drawings are used to better understand the present scheme, and do not limit the present disclosure. Among them:

[0035] Figure 1 is a flowchart of a training method of an adversarial network provided by an embodiment of the present disclosure;

[0036] Figure 2 is a schematic diagram of a sample watermark image provided by an embodiment of the present disclosure;

[0037] Figure 3 is a schematic diagram of a first sample target image corresponding to the sample watermark image in Figure 2

[0038] Figure 4 is a schematic diagram of a second sample target image corresponding to the sample watermark image in Figure 2

[0039] Figure 5 is a flowchart of an image watermarking removal method provided by an embodiment of the present disclosure;

[0040] Figure 6 is a flowchart of a specific implementation of an adversarial network training method provided by an embodiment of the present disclosure;

[0041] Figure 7 is a structural diagram of a watermarking removal adversarial network and a mask adversarial network provided by an embodiment of the present disclosure;

[0042] Figure 8 is a structural diagram of a training device of an adversarial network provided by an embodiment of the present disclosure;

[0043] Figure 9 is a structural diagram of an image watermarking removal device provided by an embodiment of the present disclosure;

[0044] Figure 10 is a block diagram of an electronic device for implementing the adversarial network training or image watermarking removal method of an embodiment of the present disclosure. DETAILED DESCRIPTION

[0045] Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, which include various details of the embodiments of the present disclosure to help in understanding. These should be considered in their context only as exemplary. Thus, those of ordinary skill in the art will recognize various changes and modifications of the embodiments described herein, without departing from the scope and spirit of the present disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

[0046] ​​In daily life, there are often watermarks in document images, such as a seal watermark in a bill image. The watermark in the bill image may block some text in the bill, and when the bill image is processed by optical character recognition (OCR), the existence of the watermark will affect the accuracy of the OCR recognition result. Therefore, how to effectively remove the watermark in the image has become an important technical problem.

[0047] In the related art, the watermark can be removed by setting a color threshold value, such as a red-green-blue (RGB) threshold value, of the watermark area, and distinguishing the watermark area and the non-watermark area according to the color threshold value. In this way, the watermark color in different images needs to be consistent so as to be distinguished by a fixed color threshold value. However, in actual situations, due to the color difference of the watermark itself, the difference of image acquisition devices, the difference of environmental light for shooting, and many other situations, the watermark color in different images may have large differences, so that the watermark area and the non-watermark area cannot be distinguished based on the fixed color threshold value, and the watermark removal effect is affected.

[0048] In the related art, a traditional image feature extraction operator such as scale-invariant feature transform (SIFT) can be used to find the color feature of a specific color watermark (such as a red seal watermark), and then the watermark is removed. However, this way also affects the watermark removal effect due to the difference of the watermark color in different images.

[0049] The training method of the adversarial network provided in the embodiments of the present disclosure, the image watermark removal method and device, and the electronic equipment aim to solve at least one of the above technical problems in the prior art.

[0050] Figure 1 A flowchart of a training method of an adversarial network provided in the embodiments of the present disclosure is shown, as shown in Figure 1 The method can mainly include the following steps.

[0051] Step S110: Obtain a training sample set, the training sample set including a sample watermark image, and the sample watermark image containing a watermark.

[0052] The sample watermark image can be an image containing a watermark, including but not limited to a bill image. The watermark can include a seal watermark, and can also include other types of watermarks.

[0053] As an example, Figure 2 A schematic diagram of a sample watermark image is shown, wherein the text content in the sample watermark image is "signature: xx Limited Company", and the watermark in the sample watermark image is a seal watermark of xx Company.

[0054] Step S120: obtaining a watermark removal adversarial network and a mask adversarial network, wherein the watermark removal adversarial network comprises a first generator and a first discriminator, the first generator comprises a first down-sampling sub-network and a first up-sampling sub-network; the mask adversarial network comprises a second generator, the second generator comprises a second down-sampling sub-network, and the mask adversarial network is used to generate a mask post-image of a sample watermark image.

[0055] The overall network architecture provided in the embodiments of the present disclosure can be composed of the watermark removal adversarial network and the mask adversarial network, wherein the watermark removal adversarial network is used to output a watermark removal post-image after training is completed, and the mask adversarial network is used to output a mask post-image after training is completed.

[0056] The mask post-image is an image in which regions except the watermark in the image are shielded, and only the watermark is reserved.

[0057] As an example, the generators of the watermark removal adversarial network and the mask adversarial network in the present example can all adopt a U-type network (Unity Networking, Unet), and the discriminators can all adopt a patch generative adversarial network (PatchGAN).

[0058] The generator can also adopt a residual block (Residual block, ResBlock) network.

[0059] Step S130: inputting the sample watermark image into the first down-sampling sub-network to obtain a first down-sampling feature map, and inputting the sample watermark image into the second down-sampling sub-network to obtain a second down-sampling feature map.

[0060] Step S140: splicing the first down-sampling feature map and the second down-sampling feature map to obtain spliced features, and inputting the spliced feature map into the first up-sampling sub-network to obtain a watermark removal post-sample watermark image.

[0061] In the embodiments of the present disclosure, the mask adversarial network is used to output a mask post-image, and the second down-sampling feature map output by the second down-sampling sub-network in the mask adversarial network can learn the position information of the watermark.

[0062] By connecting the mask adversarial network to the watermark removal adversarial network, that is, splicing the first down-sampling feature map and the second down-sampling feature map to obtain spliced features, and then up-sampling the spliced features, the watermark removal adversarial network can remove the watermark based on the position information of the watermark provided by the second down-sampling feature map, can better remove the watermark in the image and reserve the text and other contents in the image, thereby achieving a better watermark removal effect.

[0063] Step S150: training the first generator and the first discriminator based on the sample watermark image after watermark removal, to obtain the trained watermark removal GAN.

[0064] In the embodiments of the present disclosure, the first discriminator can be used to distinguish the difference between the sample watermark image after watermark removal and the training target, so as to alternately train the first generator and the first discriminator, and finally obtain the trained watermark removal GAN.

[0065] The method provided in the embodiments of the present disclosure can input the sample watermark image into the first down-sampling sub-network of the watermark removal GAN to obtain the first down-sampling feature map, and input the sample watermark image into the second down-sampling sub-network of the mask GAN to obtain the second down-sampling feature map; after splicing the first down-sampling feature map and the second down-sampling feature map, the spliced feature map is input into the first up-sampling sub-network of the watermark removal GAN to obtain the sample watermark image after watermark removal; the first generator and the first discriminator of the watermark removal GAN are trained based on the sample watermark image after watermark removal, to obtain the trained watermark removal GAN. In the present scheme, the down-sampling feature map in the mask GAN is transmitted into the watermark removal GAN, so that the watermark removal GAN can effectively remove the watermark in the image based on the watermark position information contained in the down-sampling feature map, and better watermark removal effect is achieved, which helps to improve the accuracy of image content recognition results.

[0066] The watermark removal GAN trained based on the method provided in the embodiments of the present disclosure can support the removal of various common seals (such as various round seals, square seals, oval seals, etc.) in documents, and can support the removal of seal watermarks of various colors, such as red, black and blue.

[0067] As an example, the image after watermark removal output by the watermark removal GAN in the present example is a three-channel color image, and the image after mask removal output by the mask GAN is a single-channel grayscale image.

[0068] As an example, the generator adopts a five-layer structure Unet. Since the image information is gradually lost in the sampling process, the third and fourth layers of the down-sampling and up-sampling feature maps in the Unet can be fused. Through this short connection method, the features can be completely sent to the up-sampling, so that the recovered image has better retention effect on the text.

[0069] It can be understood that, in order to avoid that the watermark gap in the sample watermark image is too large and affects the learning effect, a training sample set can be generated for a certain type of watermark, and a watermark removal GAN for the type of watermark can be trained. For example, a training sample set for seal watermark in a bill is generated, and a watermark removal GAN for the seal watermark in the bill is obtained based on the training sample set.

[0070] In one optional embodiment of this disclosure, the training sample set further includes a first sample target image, which is a watermark-free image corresponding to the watermarked sample image. Based on the watermark-free sample image, a first generator and a first discriminator are trained to obtain a trained watermark-free adversarial network, including:

[0071] Based on the watermarked sample image after watermark removal and the first sample target image, a first generator and a first discriminator are trained to obtain the trained watermark removal adversarial network.

[0072] In this system, each watermarked sample image corresponds one-to-one with a first sample target image, where the first sample target image is the watermark-free version of the corresponding watermarked sample image. As an example, the watermarked sample image can be obtained by adding a watermark to the first sample image.

[0073] Figure 3 It shows the relationship with Figure 2 A schematic diagram of the first sample target image corresponding to the watermarked image of the middle sample. That is... Figure 3 The first sample target image shown is Figure 2 Watermark-free images of the medium-sized watermarked images.

[0074] In this embodiment of the disclosure, the difference between the watermarked sample image after watermark removal and the first sample target image can be judged by the first discriminator, thereby alternatingly training the first generator and the first discriminator, and finally obtaining the trained watermark removal adversarial network.

[0075] In one optional embodiment of this disclosure, a first generator and a first discriminator are trained based on the dewatermarked sample watermark image and the first sample target image to obtain a trained dewatermarking adversarial network, including:

[0076] The watermark-free sample watermark image and the first sample target image are input into the first discriminator to obtain the first discrimination result;

[0077] The first generator and the first discriminator are trained based on the first discrimination result to obtain the trained watermark removal adversarial network.

[0078] In this embodiment of the present disclosure, the watermarked sample image after watermark removal and the first sample target image can be input into the first discriminator. The first discriminator judges the difference between the watermarked sample image after watermark removal and the first sample target image to obtain the first discrimination result. Based on the first discrimination result, the first generator and the first discriminator are trained. Through multiple rounds of iterative training, until the loss function converges, the trained watermark removal adversarial network can be obtained.

[0079] In an optional mode of the present disclosure, the loss function of the watermark removal adversarial network is determined by at least two of the following:

[0080] a generative adversarial network (GAN) loss function;

[0081] a Wasserstein generative adversarial network (WGAN) loss function;

[0082] a gradient penalty loss function;

[0083] a structural similarity index (SSIM) loss function;

[0084] a mean absolute error (MAE) loss function.

[0085] In the related art, the KL divergence (Kullback-Leible divergence, KLD) is generally used in the generative adversarial network to measure the distance between the predicted value and the true value. However, the KL divergence itself is not a distance, and its essence is to measure the difference between distributions, which is neither symmetric nor continuous. Therefore, the generative adversarial network may face the problem of mode collapse, which causes difficulty in converging during the training process and affects the final processing result.

[0086] In the embodiments of the present disclosure, the Wasserstein generative adversarial network-gradient penalty (WGAN-gp) is introduced by introducing the Wasserstein adversarial loss function-gradient penalty loss function, that is, the Wasserstein distance is used instead of the KL divergence. Even if the distribution of the predicted value output by the network does not overlap with the distribution of the true value, the Wasserstein distance can still measure the distance between the two distributions. In addition, in order to limit the ability of the discriminator, the gradient penalty loss function for the discriminator is also introduced.

[0087] As an example, the WGAN loss function can be expressed by the following formula.

[0088]

[0089] where x' represents the sample watermark image after watermark removal, x represents the first sample target image, D represents the discriminator, P ga distribution of the de-watermarked sample watermark image, an expectation of the distribution of the de-watermarked sample watermark image, P data a distribution of the first sample target image, an expectation of the distribution of the first sample target image, L WGAN a WGAN loss function.

[0090] As an example, the Gradient penalty loss function can be expressed by Equations Two and Three as follows.

[0091]

[0092] wherein,

[0093] y = exx' + (1 - ex), # Equation Three

[0094] x' represents the de-watermarked sample watermark image, x represents the first sample target image, and e is a random number with a value range of [0, 1], that is, y is a linear combination of x and x'. P y a distribution of y, an expectation of the distribution of y, L gp a Gradient penalty loss function.

[0095] A document will be contained in the image with watermark, which makes the content high-frequency information in the image with watermark more, and the accuracy requirement of text restoration is high, therefore, an L1 loss function is usually set.

[0096] As an example, the L1 loss function can be expressed by Equation Four as follows.

[0097] L1 = ||x - x' ||1, # Equation Four

[0098] wherein, x' represents the de-watermarked sample watermark image, x represents the first sample target image, and L1 represents the L1 loss function.

[0099] In the embodiments of the present disclosure, an SSIM loss function can also be introduced. The SSIM loss function can further enhance the restoration ability of the network for the text, so that the text in the output picture is clearer and sharper. If only the L1 loss function is used, it may cause a certain degree of blur.

[0100] As an example, the SSIM loss function can be expressed by Equation Five as follows.

[0101]

[0102] wherein x' represents the sample watermark image after watermark removal, x represents the first sample target image, represents the mean value, represents the variance, C1 and C2 are constants, and the denominator is prevented from being 0. L SSIM represents the SSIM loss function.

[0103] In the embodiments of the present disclosure, a GAN loss function can also be introduced.

[0104] As an example, the GAN loss function can be expressed by Formula Six as follows.

[0105]

[0106] wherein x' represents the sample watermark image after watermark removal, x represents the first sample target image, D represents the discriminator, P g represents the distribution of the sample watermark image after watermark removal, represents the expectation of the distribution of the sample watermark image after watermark removal, P data represents the distribution of the first sample target image, represents the expectation of the distribution of the first sample target image. L GAN represents the GAN loss function.

[0107] In the embodiments of the present disclosure, the above at least two losses can be weighted and calculated to obtain a comprehensive loss function of the watermark removal GAN network in the embodiments of the present disclosure.

[0108] As an example, the weight coefficients of the adversarial loss, the Wasserstein adversarial loss, the gradient penalty loss, the structural similarity index loss, and the mean absolute error loss can be respectively set, and the weighted calculation is performed based on the weight coefficients to serve as the comprehensive loss function of the watermark removal GAN network.

[0109] As an example, the comprehensive loss function can be expressed by Formula Seven as follows.

[0110] L = λ1L GAN + λ2L WGAN + λ3L qp + λ4L SSIM + λ5L1, Formula Seven

[0111] wherein λ1, λ2, λ3, λ4, and λ5 represent the weight coefficients, respectively, and can be set according to actual experience. L GAN represents the GAN loss function, L WGAN represents the WGAN loss function, L qp represents the Gradientpenalty loss function, L1 represents the L1 loss function, L SSIM represents the SSIM loss function. L represents the comprehensive loss function.

[0112] In the embodiments of the present disclosure, in the training process, the initial learning rate can be set to 0.002, and the optimizer uses Adam for the back propagation of the gradient. The training is performed for 200 epochs, and after every 10 epochs, the learning rate becomes 0.9 times of the original learning rate, and gradually reduces. In addition, all the normalization layers in the generator use the deep mode (instancenorm), and relevant researches show that instancenorm has better effect than the batch mode (batchnorm) for the generative task.

[0113] In an optional manner of the present disclosure, the method further includes generating the sample watermark image by the following manner:

[0114] obtaining an initial watermark image and an initial sample image;

[0115] performing sample expansion on the initial watermark image to obtain an expanded watermark image;

[0116] combining the expanded watermark image with the initial sample image to obtain the sample watermark image.

[0117] In the embodiments of the present disclosure, the sample expansion is performed on the initial watermark image, that is, more seal images are expanded as the expanded watermark image, which is used to combine with the initial sample image to obtain the sample watermark image, so as to improve the number of sample watermark images.

[0118] As an example, the sample expansion on the initial watermark image can include color disturbance, size adjustment and merging position adjustment.

[0119] The color disturbance is to adjust the color of the initial watermark image, so that the color of the expanded watermark image is different from that of the initial watermark image. Specifically, the color disturbance can be in the form of increasing or decreasing a random color value of all pixel points of the initial watermark image.

[0120] The size adjustment is to adjust the size of the initial watermark image, so that the size of the expanded watermark image is different from that of the initial watermark image.

[0121] The merging position adjustment is to adjust the position of the initial watermark image in the initial sample image, so that the position of the expanded watermark image in the initial sample image is different from that of the initial watermark image.

[0122] As an example, a feature region can be identified from the initial watermark image, and the feature region is used as the merging position of the expanded watermark image. For example, a keyword "seal" is identified from a bill image, and an image region of a preset size close to the keyword is used as the feature region.

[0123] In an optional manner of the present disclosure, the training sample set further includes a second sample target image, the second sample target image being a watermark mask image corresponding to the sample watermark image, the mask adversarial network further includes a second discriminator, and the method further includes:

[0124] inputting the sample watermark image into the second generator to obtain a mask post-sample watermark image;

[0125] training the second generator and the second discriminator based on the mask post-sample watermark image and the second sample target image to obtain the trained mask adversarial network.

[0126] In the embodiment of the present disclosure, the training sample set further includes a second sample target image, the second sample target image being a mask image of the watermark in the sample watermark image. As an example, Figure 4 a schematic diagram of the second sample target image corresponding to the sample watermark image in the embodiment of the present disclosure is shown. That is Figure 2 the first sample target image shown in the embodiment of the present disclosure is Figure 3 the mask image of the watermark in the sample watermark image in the embodiment of the present disclosure. Figure 2

[0127] In the embodiment of the present disclosure, the structure and loss function of the mask adversarial network and the de-watermarking adversarial network can be the same. The mask adversarial network can be trained based on the sample watermark image and the first sample target image.

[0128] In the embodiment of the present disclosure, the second discriminator can be used to distinguish the difference between the mask post-sample watermark image and the second sample target image, so as to alternately train the second generator and the second discriminator, and finally obtain the trained mask adversarial network.

[0129] In an optional manner of the present disclosure, training the second generator and the second discriminator based on the mask post-sample watermark image and the second sample target image to obtain the trained mask adversarial network includes:

[0130] inputting the mask post-sample watermark image and the second sample target image into the second discriminator to obtain a second discrimination result;

[0131] training the second generator and the second discriminator based on the second discrimination result to obtain the trained mask adversarial network.

[0132] In the embodiment of the present disclosure, the mask post-sample watermark image and the second sample target image can be input into the second discriminator, the difference between the mask post-sample watermark image and the second sample target image can be distinguished by the second discriminator to obtain a second discrimination result, the second generator and the second discriminator can be trained based on the second discrimination result, and through multiple rounds of iterative training, until the loss function converges, the trained mask adversarial network can be obtained.​

[0133] In an optional manner of the present disclosure, the loss function of the mask adversarial network is determined by at least two of the following:

[0134] an adversarial loss function;

[0135] a Wasserstein adversarial loss function;

[0136] a gradient penalty loss function;

[0137] a structural similarity index loss;

[0138] a mean absolute error loss function.

[0139] In the embodiments of the present disclosure, the loss functions of the mask adversarial network and the watermark removal adversarial network can be the same, which will not be described herein.

[0140] Figure 5 A flowchart of an image watermark removal method provided by the embodiments of the present disclosure is shown, as shown in Figure 5 The method can mainly include the following steps:

[0141] Step S510: obtaining a to-be-processed image;

[0142] Step S520: obtaining a first generator of a watermark removal adversarial network and a second generator of a mask adversarial network, the watermark removal adversarial network being trained based on the training method of the adversarial network, the first generator including a first down-sampling sub-network and a first up-sampling sub-network, the second generator including a second down-sampling sub-network, and the mask adversarial network being used to generate a mask image of the to-be-processed image;

[0143] Step S530: inputting the to-be-processed image into the first down-sampling sub-network to obtain a third down-sampling feature map, and inputting the to-be-processed image into the second down-sampling sub-network to obtain a fourth down-sampling feature map;

[0144] Step S540: splicing the third down-sampling feature map and the fourth down-sampling feature map to obtain a target splicing feature map, and inputting the target splicing feature map into the first up-sampling sub-network to obtain a watermark-removed image of the to-be-processed image.

[0145] In the embodiments of the present disclosure, the first generator of the watermark removal adversarial network can be trained based on the training method of the adversarial network to remove the watermark from the to-be-processed image.

[0146] Specifically, the image to be processed can be input into the first down-sampling sub-network to obtain a third down-sampled feature map, and input into the second down-sampling sub-network to obtain a fourth down-sampled feature map. The third down-sampled feature map and the fourth down-sampled feature map are spliced to obtain a target spliced feature map. The target spliced feature map is input into the first up-sampling sub-network to obtain a de-watermarked image of the image to be processed.

[0147] By connecting the mask adversarial network to the de-watermarking adversarial network, i.e., splicing the first down-sampled feature map and the second down-sampled feature map to obtain a spliced feature, and then up-sampling the spliced feature, the de-watermarking adversarial network can perform de-watermarking based on the watermark position information provided by the second down-sampled feature map, better remove the watermark in the image to be processed, and retain the text and other contents in the image to be processed, thereby achieving better de-watermarking effect.

[0148] The method provided by the embodiments of the present disclosure includes the following steps: obtaining an image to be processed; obtaining a first generator of a de-watermarking adversarial network and a second generator of a mask adversarial network, the de-watermarking adversarial network being trained based on the training method of the adversarial network, the first generator including a first down-sampling sub-network and a first up-sampling sub-network, and the second generator including a second down-sampling sub-network, the mask adversarial network being used to generate a mask image of the image to be processed; inputting the image to be processed into the first down-sampling sub-network to obtain a third down-sampled feature map, and inputting the image to be processed into the second down-sampling sub-network to obtain a fourth down-sampled feature map; splicing the third down-sampled feature map and the fourth down-sampled feature map to obtain a target spliced feature map, and inputting the target spliced feature map into the first up-sampling sub-network to obtain a de-watermarked image of the image to be processed. In the scheme, the down-sampled feature map in the mask adversarial network is transmitted into the de-watermarking adversarial network, so that the de-watermarking adversarial network can effectively remove the watermark in the image to be processed based on the watermark position information contained in the down-sampled feature map, and better de-watermarking effect is achieved.

[0149] In an optional mode of the present disclosure, the mask adversarial network is trained based on the training method of the adversarial network.

[0150] In the embodiments of the present disclosure, the mask adversarial network trained based on the sample watermark image and the second sample target image can be used.

[0151] In an optional mode of the present disclosure, the method further includes:

[0152] The image to be processed is input into the second generator to obtain a mask image of the image to be processed.

[0153] In the embodiments of the present disclosure, the image to be processed can also be input to the second generator to output the image after the mask.

[0154] As an example, Figure 6 is a flowchart of a specific implementation of the adversarial network training method provided by the embodiments of the present disclosure.

[0155] As Figure 6 As shown in the above, data augmentation, i.e., by sample expansion on the initial watermark image, to obtain the expanded watermark image, and then the expanded watermark image is synthesized with the initial sample image to obtain the sample watermark image. In this example, the watermark is a red seal watermark, which can be extracted from the image with the red seal watermark by pre-setting the RGB threshold. Color disturbance is performed on the red seal watermark to obtain color seal watermarks of multiple colors. The color seal watermarks of multiple colors obtained by color disturbance are fused with the initial sample image to obtain the sample watermark image.

[0156] Network training, i.e., training of the watermark removal adversarial network. Specifically, the sample watermark image (i.e., the picture with the seal) can be input to the generator (i.e., the generation network) of the watermark removal adversarial network to obtain the sample watermark image after watermark removal output by the generator (i.e., the picture without the seal). The loss function is calculated, and the adversarial network is updated. The L1 loss, the SSIM loss and the GAN loss can be calculated respectively, the comprehensive loss of the loss function is calculated according to the L1 loss, the SSIM loss and the GAN loss, the back propagation is performed, and the network parameters are updated until the trained watermark removal adversarial network is obtained.

[0157] As an example, Figure 7 is a structural diagram of the watermark removal adversarial network and the mask adversarial network provided by the embodiments of the present disclosure.

[0158] In this example, the generator of the watermark removal adversarial network and the mask adversarial network adopts Unet with a five-layer structure, i.e., five-layer down-sampling (i.e., convolution) is performed on the original image (i.e., the image containing the seal), and then five-layer up-sampling (i.e., deconvolution) is performed, and finally the image without the seal is output. In this process, the sampling feature maps of the third and fourth layers of down-sampling and up-sampling in Unet can be spliced. Through this short connection method, the features can be completely sent to the up-sampling, so that the recovered image has better text retention effect. The sampling feature map of the fifth layer of down-sampling in the mask adversarial network can be spliced with the sampling feature map of the fifth layer of down-sampling in the watermark removal adversarial network, and the spliced feature map is up-sampled.

[0159] Figure 7In the middle, 8, 16, 32, 64, 128 are the sizes of feature maps in each layer. 3x3 / 2 means using a 3x3 convolution kernel and a convolution step of 2.

[0160] As an example, in addition to the seal watermark, the scheme provided by the embodiments of the present disclosure can also be used for removing other types of watermarks, such as removing the underlay watermark. The underlay watermark can be used as an underlay watermark pattern or watermark text in a document image.

[0161] In this example, a watermark removal GAN can be trained for removing the underlay watermark. The sample watermark images in the training set used to train the watermark removal GAN can contain underlay watermarks, and the first sample target image is an underlay watermark-free image corresponding to the sample watermark image. Based on the training set, the watermark removal GAN can be trained in the same way as in steps S110-S150.

[0162] The training set in this example can also include a second sample target image. The second sample target image is a mask image of the underlay watermark in the sample watermark image. Based on the sample watermark image and the second sample target image, a mask GAN can be trained. The mask GAN can be used to output a mask image of the underlay watermark. Specifically, the sample watermark image can be input into the second generator to obtain a masked sample watermark image; based on the masked sample watermark image and the second sample target image, the second generator and the second discriminator can be trained to obtain the trained mask GAN.

[0163] In this example, after the watermark removal GAN corresponding to the underlay watermark and the mask GAN are trained, the image to be processed containing the underlay watermark can be input into the first down-sampling sub-network of the watermark removal GAN to obtain a third down-sampling feature map, and the image to be processed containing the underlay watermark can be input into the second down-sampling sub-network of the mask GAN to obtain a fourth down-sampling feature map. The third down-sampling feature map and the fourth down-sampling feature map are spliced to obtain a target splicing feature map, and the target splicing feature map is input into the first up-sampling sub-network of the watermark removal GAN to obtain the image to be processed after removing the underlay watermark.

[0164] Based on the same principle as the method shown in Figure 1 Figure 8 A structure diagram of a training device of a GAN provided by an embodiment of the present disclosure is shown, as shown in Figure 8 The training device 80 of the GAN can include:

[0165] The training sample set acquisition module 810 is configured to acquire a training sample set. The training sample set includes a sample watermark image, and the sample watermark image contains a watermark.

[0166] ​The model obtaining module 820 is configured to obtain a watermark removal adversarial network and a mask adversarial network. The watermark removal adversarial network includes a first generator and a first discriminator. The first generator includes a first downsampling subnetwork and a first upsampling subnetwork. The mask adversarial network includes a second generator. The second generator includes a second downsampling subnetwork. The mask adversarial network is configured to generate a mask post-image of a sample watermark image.

[0167] The downsampling module 830 is configured to input the sample watermark image into the first downsampling subnetwork to obtain a first downsampling feature map, and input the sample watermark image into the second downsampling subnetwork to obtain a second downsampling feature map.

[0168] The upsampling module 840 is configured to splice the first downsampling feature map and the second downsampling feature map to obtain spliced features, and input the spliced feature map into the first upsampling subnetwork to obtain a sample watermark image after watermark removal.

[0169] The model training module 850 is configured to train the first generator and the first discriminator based on the sample watermark image after watermark removal to obtain a trained watermark removal adversarial network.

[0170] The device provided in the embodiments of the present disclosure is configured to input the sample watermark image into the first downsampling subnetwork of the watermark removal adversarial network to obtain a first downsampling feature map, and input the sample watermark image into the second downsampling subnetwork of the mask adversarial network to obtain a second downsampling feature map. After splicing the first downsampling feature map and the second downsampling feature map, the spliced feature map is input into the first upsampling subnetwork of the watermark removal adversarial network to obtain a sample watermark image after watermark removal. The first generator and the first discriminator of the watermark removal adversarial network are trained based on the sample watermark image after watermark removal to obtain a trained watermark removal adversarial network. In the present solution, the downsampling feature map in the mask adversarial network is transmitted into the watermark removal adversarial network, so that the watermark removal adversarial network can effectively remove the watermark in the image based on the watermark position information contained in the downsampling feature map, and better watermark removal effect is achieved.

[0171] Optionally, the training sample set further includes a first sample target image. The first sample target image is a watermark-free image corresponding to the sample watermark image. The model training module is specifically configured to:

[0172] The first generator and the first discriminator are trained based on the sample watermark image after watermark removal and the first sample target image to obtain a trained watermark removal adversarial network.

[0173] Optionally, the model training module is specifically configured to:

[0174] The sample watermark image after watermark removal and the first sample target image are input into the first discriminator to obtain a first discrimination result.

[0175] training the first generator and the first discriminator based on the first discrimination result, to obtain a trained watermark removal adversarial network.

[0176] Optionally, the loss function of the watermark removal adversarial network is determined by at least two of:

[0177] an adversarial loss function;

[0178] a Wasserstein adversarial loss function;

[0179] a gradient penalty loss function;

[0180] a structural similarity index loss function;

[0181] a mean absolute error loss function.

[0182] Optionally, the apparatus further comprises generating the sample watermark image by:

[0183] obtaining an initial watermark image and an initial sample image;

[0184] performing sample expansion on the initial watermark image to obtain an expanded watermark image;

[0185] combining the expanded watermark image and the initial sample image to obtain the sample watermark image.

[0186] Optionally, the training sample set further comprises a second sample target image, the second sample target image being a watermark mask image corresponding to the sample watermark image, the mask adversarial network further comprising a second discriminator, and the model training module is further configured to:

[0187] inputting the sample watermark image into a second generator to obtain a masked sample watermark image;

[0188] training the second generator and the second discriminator based on the masked sample watermark image and the second sample target image, to obtain a trained mask adversarial network.

[0189] Optionally, when training the second generator and the second discriminator based on the masked sample watermark image and the second sample target image, to obtain the trained mask adversarial network, the model training module is specifically configured to:

[0190] inputting the masked sample watermark image and the second sample target image into the second discriminator to obtain a second discrimination result;

[0191] training the second generator and the second discriminator based on the second discrimination result, to obtain the trained mask adversarial network.

[0192] Optionally, the loss function of the mask adversarial network is determined by at least two of:

[0193] adversarial loss function;

[0194] Wasserstein adversarial loss function;

[0195] gradient penalty loss function;

[0196] structural similarity index loss function;

[0197] mean absolute error loss function.

[0198] It can be understood that the above modules of the training device of the adversarial network in the embodiments of the present disclosure have the functions of the corresponding steps of the training method of the adversarial network in the embodiments shown in the above. Figure 1 The functions can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. The above modules can be software and / or hardware, and the above modules can be implemented separately or integrated. For the function description of each module of the training device of the adversarial network, please refer to the corresponding description of the training method of the adversarial network in the embodiments shown in the above, which will not be repeated here. Figure 1

[0199] Based on the same principle as the method shown in the above, Figure 5 the structure of an image watermark removal device provided by the embodiments of the present disclosure is shown, as shown in the above, Figure 9 the image watermark removal device 90 can include: Figure 9 The image watermark removal device 90 can include:

[0200] The image watermark removal device 90 can include:

[0201] The image watermark removal device 90 can include:

[0202] The image watermark removal device 90 can include:

[0203] The image watermark removal device 90 can include:

[0204] ​The device provided by the embodiments of the present disclosure acquires a to-be-processed image, acquires a first generator of a watermark-removing adversarial network and a second generator of a mask adversarial network, the watermark-removing adversarial network is trained based on the training method of the adversarial network, the first generator includes a first down-sampling sub-network and a first up-sampling sub-network, the second generator includes a second down-sampling sub-network, and the mask adversarial network is used to generate a mask post-image of the to-be-processed image; the to-be-processed image is input into the first down-sampling sub-network to obtain a third down-sampling feature map, and the to-be-processed image is input into the second down-sampling sub-network to obtain a fourth down-sampling feature map; the third down-sampling feature map and the fourth down-sampling feature map are spliced to obtain a target splicing feature map, and the target splicing feature map is input into the first up-sampling sub-network to obtain a watermark-removed image of the to-be-processed image. In the scheme, the down-sampling feature map in the mask adversarial network is transmitted into the watermark-removing adversarial network, so that the watermark-removing adversarial network can effectively remove the watermark in the to-be-processed image based on the watermark position information contained in the down-sampling feature map, and better watermark removal effect is achieved.

[0205] Optionally, the mask adversarial network is trained based on the training method of the adversarial network.

[0206] Optionally, the device further includes:

[0207] The mask post-image generation module is configured to input the to-be-processed image into the second generator to obtain a mask post-image of the to-be-processed image.

[0208] It can be understood that the above modules of the image watermark removal device in the embodiments of the present disclosure have the functions of the corresponding steps of the image watermark removal method in the embodiments shown in the above. Figure 5 The functions can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. The above modules can be software and / or hardware, and can be implemented individually or in multiple modules. Figure 5 For the function description of each module of the image watermark removal device, refer to the corresponding description of the image watermark removal method in the embodiments shown in the above, which will not be described here.

[0209] In the technical solutions of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information comply with relevant laws and regulations, and do not violate public order and good customs.

[0210] According to the embodiments of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium and a computer program product.

[0211] The electronic device includes at least one processor, and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the training of the adversarial network or the image watermark removal method provided in the embodiments of the present disclosure.

[0212] Compared with the prior art, the electronic device inputs the sample watermark image into the first down-sampling sub-network of the watermark removal adversarial network to obtain a first down-sampling feature map, and inputs the sample watermark image into the second down-sampling sub-network of the mask adversarial network to obtain a second down-sampling feature map; after splicing the first down-sampling feature map and the second down-sampling feature map, the spliced feature map is input into the first up-sampling sub-network of the watermark removal adversarial network to obtain a sample watermark image after watermark removal; based on the sample watermark image after watermark removal, the first generator and the first discriminator of the watermark removal adversarial network are trained to obtain the trained watermark removal adversarial network. In the present scheme, by transmitting the down-sampling feature map in the mask adversarial network into the watermark removal adversarial network, the watermark removal adversarial network can effectively remove the watermark in the image based on the watermark position information contained in the down-sampling feature map, and better watermark removal effect is achieved.

[0213] The readable storage medium is a non-transitory computer readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to perform the training of the adversarial network or the image method provided in the embodiments of the present disclosure.

[0214] Compared with the prior art, the readable storage medium inputs the sample watermark image into the first down-sampling sub-network of the watermark removal adversarial network to obtain a first down-sampling feature map, and inputs the sample watermark image into the second down-sampling sub-network of the mask adversarial network to obtain a second down-sampling feature map; after splicing the first down-sampling feature map and the second down-sampling feature map, the spliced feature map is input into the first up-sampling sub-network of the watermark removal adversarial network to obtain a sample watermark image after watermark removal; based on the sample watermark image after watermark removal, the first generator and the first discriminator of the watermark removal adversarial network are trained to obtain the trained watermark removal adversarial network. In the present scheme, by transmitting the down-sampling feature map in the mask adversarial network into the watermark removal adversarial network, the watermark removal adversarial network can effectively remove the watermark in the image based on the watermark position information contained in the down-sampling feature map, and better watermark removal effect is achieved.

[0215] The computer program product includes a computer program, and the computer program, when executed by a processor, implements the training of the adversarial network or the image method provided in the embodiments of the present disclosure.

[0216] Compared with the prior art, the computer program product inputs the sample watermark image into a first down-sampling sub-network of the watermark removal adversarial network to obtain a first down-sampling feature map, and inputs the sample watermark image into a second down-sampling sub-network of the mask adversarial network to obtain a second down-sampling feature map; after the first down-sampling feature map and the second down-sampling feature map are spliced, the spliced feature map is input into a first up-sampling sub-network of the watermark removal adversarial network to obtain a sample watermark image after watermark removal; based on the sample watermark image after watermark removal, the first generator and the first discriminator of the watermark removal adversarial network are trained to obtain the trained watermark removal adversarial network. In the scheme, the down-sampling feature map in the mask adversarial network is transmitted into the watermark removal adversarial network, so that the watermark removal adversarial network can effectively remove the watermark in the image based on the watermark position information contained in the down-sampling feature map, and better watermark removal effect is achieved.

[0217] Figure 10 A schematic block diagram of an example electronic device 1000 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the present disclosure described and / or claimed in this document.

[0218] As shown in Figure 10 The electronic device 1000 includes a computing unit 1010 that can perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory (ROM) 1020 or a computer program loaded from a storage unit 1080 into a random access memory (RAM) 1030. Various programs and data required for the operation of the device 1000 can also be stored in the RAM 1030. The computing unit 1010, the ROM 1020, and the RAM 1030 are connected to each other through a bus 1040. An input / output (I / O) interface 1050 is also connected to the bus 1040.

[0219] A plurality of components in the device 1000 are connected to the I / O interface 1050, including: an input unit 1060, such as a keyboard, a mouse, etc.; an output unit 1070, such as various types of displays, speakers, etc.; a storage unit 1080, such as a magnetic disk, an optical disk, etc.; and a communication unit 1090, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1090 allows the device 1000 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0220] The computing unit 1010 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1010 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, and the like. The computing unit 1010 performs the training of the adversarial network or the image method provided in the embodiments of the present disclosure. For example, in some embodiments, the training of the adversarial network or the image method provided in the embodiments of the present disclosure can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as the storage unit 1080. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 1000 via the ROM 1020 and / or the communication unit 1090. When the computer program is loaded into the RAM 1030 and executed by the computing unit 1010, one or more steps of the training of the adversarial network or the image method provided in the embodiments of the present disclosure can be performed. Alternatively, in other embodiments, the computing unit 1010 can be configured to perform the training of the adversarial network or the image method provided in the embodiments of the present disclosure by any other appropriate means, such as by means of firmware.

[0221] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (PLD), a computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0222] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a standalone software package, or entirely on a remote machine or server.

[0223] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0224] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0225] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0226] The computer system can include clients and servers. This relationship can be. The servers are typically remote from the clients with the interactions between them occurring over a communication network. The relationship between a client and a server is one of client-server. The server can be a cloud server, a server of a distributed system, or a server incorporating a blockchain.

[0227] It should be understood that the various forms of flow shown above can be re-ordered, added to, or deleted from without departing from the scope of the present disclosure. For example, the steps recited in the present disclosure can be performed in parallel, in series, or in a different order, as long as the desired results of the technology disclosed in the present disclosure are achieved, and the present disclosure is not limited herein.

[0228] The specific embodiments described above are not intended to be limiting. One of skill in the art will understand that various modifications, combinations, sub-combinations, and alternatives can be made to the specific embodiments described above without departing from the scope of the disclosure. Any modifications, combinations, sub-combinations, and alternatives falling within the spirit and principles of the disclosure should be considered.

Claims

1. A method for training a counter-network, comprising: obtaining a training sample set, the training sample set comprising a sample watermark image containing a watermark; obtaining a watermark-removing counter-network and a mask counter-network, wherein the watermark-removing counter-network comprises a first generator and a first discriminator, the first generator comprising a first down-sampling sub-network and a first up-sampling sub-network; the mask counter-network comprising a second generator, the second generator comprising a second down-sampling sub-network, the mask counter-network being configured to generate a mask post-image of the sample watermark image; inputting the sample watermark image into the first down-sampling sub-network to obtain a first down-sampling feature map, and inputting the sample watermark image into the second down-sampling sub-network to obtain a second down-sampling feature map; splicing the first down-sampling feature map and the second down-sampling feature map to obtain a spliced feature, and inputting the spliced feature into the first up-sampling sub-network to obtain a watermark-removed sample watermark image; training the first generator and the first discriminator based on the watermark-removed sample watermark image to obtain a trained watermark-removing counter-network.

2. The method of claim 1, wherein, The training sample set further comprises a first sample target image, the first sample target image being a watermark-free image corresponding to the sample watermark image, and the training the first generator and the first discriminator based on the watermark-removed sample watermark image to obtain a trained watermark-removing counter-network comprises: training the first generator and the first discriminator based on the watermark-removed sample watermark image and the first sample target image to obtain a trained watermark-removing counter-network.

3. The method of claim 2, wherein, The training the first generator and the first discriminator based on the watermark-removed sample watermark image and the first sample target image to obtain a trained watermark-removing counter-network comprises: inputting the watermark-removed sample watermark image and the first sample target image into the first discriminator to obtain a first discrimination result; training the first generator and the first discriminator based on the first discrimination result to obtain a trained watermark-removing counter-network.

4. The method of any one of claims 1-3, wherein, The loss function of the watermark-removing counter-network is determined by at least two of: an adversarial loss function GAN; a Wasserstein adversarial loss function WGAN; a gradient penalty loss function Gradient penalty; a structural similarity index loss function SSIM; a mean absolute error loss function L1.

5. The method of any one of claims 1-4, further comprising generating the sample watermark image by: obtaining an initial watermark image and an initial sample image; performing sample expansion on the initial watermark image to obtain an expanded watermark image; combining the expanded watermark image and the initial sample image to obtain the sample watermark image.

6. The method of any one of claims 1-5, wherein, The training sample set further comprises a second sample target image, the second sample target image being a watermark mask image corresponding to the sample watermark image, and the mask counter-network further comprises a second discriminator, and the method further comprises: inputting the sample watermark image into the second generator to obtain a sample watermark image after masking; training the second generator and the second discriminator based on the sample watermark image after masking and the second sample target image to obtain the trained mask adversarial network.

7. The method of claim 6, wherein, The training of the second generator and the second discriminator based on the sample watermark image after masking and the second sample target image to obtain the trained mask adversarial network comprises: inputting the sample watermark image after masking and the second sample target image into the second discriminator to obtain a second discrimination result; training the second generator and the second discriminator based on the second discrimination result to obtain the trained mask adversarial network.

8. The method of claim 6 or 7, wherein, The loss function of the mask adversarial network is determined by at least two of: a GAN loss function; a WGAN loss function; a Gradient penalty loss function; a SSIM loss function; a L1 loss function.

9. An image watermark removing method, comprising: obtaining a to-be-processed image; obtaining a first generator of a watermark removing adversarial network and a second generator of a mask adversarial network, the watermark removing adversarial network being trained based on the method in any one of claims 1-8, the first generator comprising a first down-sampling sub-network and a first up-sampling sub-network, the second generator comprising a second down-sampling sub-network, and the mask adversarial network being used to generate a masked image of the to-be-processed image; inputting the to-be-processed image into the first down-sampling sub-network to obtain a third down-sampling feature map and inputting the to-be-processed image into the second down-sampling sub-network to obtain a fourth down-sampling feature map; splicing the third down-sampling feature map and the fourth down-sampling feature map to obtain a target splicing feature map, and inputting the target splicing feature map into the first up-sampling sub-network to obtain a watermark-removed image of the to-be-processed image.

10. The method of claim 9, wherein, The mask adversarial network is trained based on the method in any one of claims 6-8.

11. The method of claim 10, wherein, Further comprising: inputting the to-be-processed image into the second generator to obtain a masked image of the to-be-processed image.

12. An apparatus for training an adversarial network, comprising: a training sample set obtaining module configured to obtain a training sample set, the training sample set comprising a sample watermark image, the sample watermark image containing a watermark; a model obtaining module configured to obtain a watermark removing adversarial network and a mask adversarial network, wherein the watermark removing adversarial network comprises a first generator and a first discriminator, the first generator comprising a first down-sampling sub-network and a first up-sampling sub-network; the mask adversarial network comprises a second generator, the second generator comprising a second down-sampling sub-network, and the mask adversarial network being used to generate a masked image of the sample watermark image; and a downsampling module, configured to input the sample watermark image into the first downsampling subnetwork to obtain a first downsampling feature map, and input the sample watermark image into a second downsampling subnetwork to obtain a second downsampling feature map; an upsampling module, configured to splice the first downsampling feature map and the second downsampling feature map to obtain spliced features, and input the spliced feature map into the first upsampling subnetwork to obtain a sample watermark image after watermark removal; a model training module, configured to train the first generator and the first discriminator based on the sample watermark image after watermark removal to obtain the trained watermark removal GAN.

13. The apparatus of claim 12, wherein, The training sample set further includes a first sample target image, the first sample target image being a watermark-free image corresponding to the sample watermark image, and the model training module is specifically configured to: train the first generator and the first discriminator based on the sample watermark image after watermark removal and the first sample target image to obtain the trained watermark removal GAN.

14. The apparatus of claim 13, wherein, The model training module is specifically configured to: input the sample watermark image after watermark removal and the first sample target image into the first discriminator to obtain a first discrimination result; train the first generator and the first discriminator based on the first discrimination result to obtain the trained watermark removal GAN.

15. The apparatus of any one of claims 12-14, further comprising generating the sample watermark image by: obtaining an initial watermark image and an initial sample image; performing sample expansion on the initial watermark image to obtain an expanded watermark image; synthesizing the expanded watermark image and the initial sample image to obtain the sample watermark image.

16. The apparatus of any one of claims 12-15, wherein, The training sample set further includes a second sample target image, the second sample target image being a watermark mask image corresponding to the sample watermark image, and the mask GAN further includes a second discriminator, and the model training module is further configured to: input the sample watermark image into the second generator to obtain a mask sample watermark image; train the second generator and the second discriminator based on the mask sample watermark image and the second sample target image to obtain the trained mask GAN.

17. An image watermark removal apparatus, comprising: a to-be-processed image obtaining module, configured to obtain a to-be-processed image; a model obtaining module, configured to obtain a first generator of a watermark removal GAN and a second generator of a mask GAN, the watermark removal GAN being trained based on the method in any one of claims 1-8, the first generator including a first downsampling subnetwork and a first upsampling subnetwork, the second generator including a second downsampling subnetwork, and the mask GAN being configured to generate a mask image after watermark removal of the to-be-processed image; a downsampling module, configured to input the to-be-processed image into the first downsampling subnetwork to obtain a third downsampling feature map, and input the to-be-processed image into the second downsampling subnetwork to obtain a fourth downsampling feature map; The watermark-removed image generation module is configured to splice the third down-sampling feature map and a fourth down-sampling feature map to obtain a target spliced feature map, and input the target spliced feature map into the first up-sampling sub-network to obtain a watermark-removed image of the to-be-processed image. 18.An electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-11.

19. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to enable the computer to perform the method of any one of claims 1-11. 20.A computer program product comprising a computer program which, when executed by a processor, implements the method of any one of claims 1-11.

Images