Watermark removal method and apparatus, device, medium, product

Abstract

The application relates to a watermark removing method and device, equipment, medium and product, the method comprising: acquiring an original image containing a watermark; acquiring a commodity watermark image; calling a first image segmentation network in a preset watermark removing model to determine a watermark mask image by taking the commodity watermark image as input; determining a watermark image by taking the commodity watermark image and the watermark mask image as input by a second image segmentation network in the watermark removing model; and determining a commodity image after removing the watermark image in the commodity watermark image by taking the commodity watermark image, the watermark mask image and the watermark image as input by a third image segmentation network in the watermark removing model. The application can accurately remove the watermark in the commodity watermark image.

Description

Technical Field

[0001] This application relates to watermark image processing technology, and more particularly to a watermark removal method, apparatus, device, medium, or product thereof. Background Technology

[0002] In recent years, with the continuous development of computer vision technology, watermark embedding algorithms are frequently used for product image ownership claims, causing inconvenience when reusing related product images. In some special cases, it is necessary to remove watermarks, such as when the original image with the watermark is lost, or when the watermark design company no longer provides technical support. To meet such needs, an independent solution is required.

[0003] Currently, the common method for removing watermarks is to first detect the watermark and then perform image restoration. The effect achieved is to mask the image area containing the watermark, which helps to eliminate some visually imperceptible watermarks and can indeed remove watermarks. However, the background image where the watermark overlaps loses a lot of the originally visible background image information, resulting in a less than ideal watermark removal effect. Summary of the Invention

[0004] The purpose of this application is to solve the above-mentioned problems by providing a watermark removal method and corresponding apparatus, devices, non-volatile readable storage media, and computer program products.

[0005] According to one aspect of this application, a watermark removal method is provided, comprising the following steps: Obtain product watermark images; The first image segmentation network in the preset watermark removal model is called, and the product watermark image is used as input to determine the watermark mask image; The watermark image is determined by the second image segmentation network in the watermark removal model, using the product watermark image and the watermark mask image as input. The third image segmentation network in the watermark removal model uses the product watermark image, the watermark mask image, and the watermark image as input to determine the product image after removing the watermark image from the product watermark image.

[0006] In a further embodiment, the training process of the first image segmentation network in the watermark removal model includes the following steps: Obtain the first training sample and its first supervision label in the dataset. The first training sample is the sample watermark image, and the first supervision label is the target watermark mask image corresponding to the watermark image in the sample watermark image. The first training sample is input into the first image segmentation network. After multiple downsampling stages and corresponding multiple upsampling stages, the corresponding deep semantic information is extracted, and the predicted watermark mask image is predicted. The cross-entropy loss value corresponding to the predicted watermark mask is calculated based on the supervision label, and the first image segmentation network is updated with gradients based on the cross-entropy loss value until the first image segmentation network converges.

[0007] In a further embodiment, the training process of the second image segmentation network in the watermark removal model includes the following steps: Obtain the second training sample and its second supervision label from the dataset. The second training sample includes the sample watermark image and its corresponding watermark mask image. The second supervision label is the watermark image in the sample watermark image. The second training sample is input into the second image segmentation network. After multiple downsampling stages and corresponding multiple upsampling stages, the corresponding deep semantic information is extracted, and the predicted watermark image is predicted. The predicted watermark image and the second supervision label are respectively input into a preset image feature extraction network to determine the corresponding first predicted image features and first target image features; A first similarity loss value is determined between the first predicted image features and the first target image features. The second image segmentation network is then updated using gradients based on the first similarity loss value until the second image segmentation network converges.

[0008] In a further embodiment, the training process of the third image segmentation network in the watermark removal model includes the following steps: The third image segmentation network is used as a generator, and it is connected to a pre-trained discriminator to construct a generative adversarial network, thus solidifying the weights of the discriminator. Obtain a single third training sample from the dataset, the third training sample including the sample watermark image and its corresponding watermark mask image and watermark image; The third training sample is input into the generator, and after multiple downsampling stages and corresponding multiple upsampling stages, the corresponding deep semantic information is extracted to generate a predicted watermark-free image. The discriminator is used to perform binary classification mapping on the predicted watermark-free image to obtain the positive class probability, which represents the probability that the predicted watermark-free image is truly watermark-free. When the probability of the positive class reaches a preset threshold, the generator is determined to have converged and training is terminated; otherwise, another third training sample is called to continue iterative training of the generator.

[0009] In a further embodiment, the third image segmentation network is used as a generator, and it is connected to a pre-trained discriminator to construct a generative adversarial network. Before the weights of the discriminator are fixed, the following steps are included: Obtain the fourth training sample and its fourth supervision label in the dataset, and supervise the training of the third image segmentation network until convergence. The fourth training sample includes the sample watermark image and its corresponding watermark mask image and watermark image. The fourth supervision label is the target watermark-free image after the watermark image in the sample watermark image has been removed in advance. The third image segmentation network trained to convergence is used as a generator, its weights are fixed, and the generator is called to generate the predicted watermark-free image corresponding to the fourth training sample as a negative sample for training the discriminator, and the original image corresponding to the image without watermark attached to the fourth training sample is used as a positive sample. The discriminator is invoked to perform binary classification mapping on the positive and negative samples respectively, and the corresponding positive and negative class probabilities of each sample are obtained. When both the positive and negative class probabilities reach their respective preset thresholds, the discriminator is deemed to have converged and training is terminated; otherwise, another set of positive and negative samples is used for iterative training.

[0010] In a further embodiment, obtaining a fourth training sample and its fourth supervision label from the dataset, and supervising the training of the third image segmentation network until convergence, includes the following steps: Obtain the fourth training sample and its fourth supervision label in the dataset. The fourth training sample includes the sample watermark image and its corresponding watermark mask image and watermark image. The fourth supervision label represents the target watermark-free image after the watermark image in the sample watermark image has been removed in advance. The fourth training sample is input into the third image segmentation network. After multiple downsampling stages and corresponding multiple upsampling stages, the corresponding deep semantic information is extracted to generate a predicted watermark-free image. The predicted watermark-free image and the fourth supervision label are respectively input into a preset image feature extraction network to determine the corresponding second predicted image features and second target image features; A second similarity loss value is determined between the second predicted image features and the second target image features. The third image segmentation network is then updated using gradients based on this second similarity loss value until the third image segmentation network converges.

[0011] In a further embodiment, before obtaining the dataset, the following steps are included: Multiple product images are acquired, and a pre-processed watermark image is attached to a random position on each product image. The pre-processing of the watermark image includes any one or more of the following: random color, random transparency, random rotation angle, and random size. The product images with the added watermark are used as sample watermark images to construct a dataset.

[0012] According to another aspect of this application, a watermark removal apparatus is provided, comprising: an image acquisition module for acquiring a product watermark image; a mask value determination module for calling a first image segmentation network in a preset watermark removal model to determine a watermark mask image using the product watermark image as input; a watermark determination module for determining a watermark image using a second image segmentation network in the watermark removal model to take the product watermark image and the watermark mask image as input; and a watermark removal module for determining a product image after removing the watermark image from the product watermark image using a third image segmentation network in the watermark removal model to take the product watermark image, the watermark mask image, and the watermark image as input.

[0013] According to another aspect of this application, a watermark removal device is provided, including a central processing unit and a memory, wherein the central processing unit is used to invoke and run a computer program stored in the memory to perform the steps of the watermark removal method described in this application.

[0014] According to another aspect of this application, a non-volatile readable storage medium is provided, which stores a computer program implemented according to the watermark removal method in the form of computer-readable instructions, wherein the computer program, when invoked by a computer, executes the steps included in the method.

[0015] According to another aspect of this application, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the method described in any embodiment of this application.

[0016] Compared with existing technologies, this application has several advantages, including but not limited to: This application utilizes a first image segmentation network in a pre-defined watermark removal model, taking the product watermark image as input, to determine a watermark mask image. A second image segmentation network in the same model then uses both the product watermark image and the watermark mask image as input to determine a watermark image. Finally, a third image segmentation network in the same model uses the product watermark image, the watermark mask image, and the watermark image as input to determine the product image after removing the watermark image from the product watermark image. On one hand, this achieves end-to-end watermark removal. The first and second image segmentation networks in the watermark removal model accurately determine the watermark mask image and watermark image required for watermark removal, and the third image segmentation network then accurately removes the watermark image from the product watermark image. On the other hand, each image segmentation network in the watermark removal model is relatively independent, allowing for independent training to obtain the functions of determining the watermark mask image, the watermark image, and removing the watermark. Each network performs its specific function, ensuring the accuracy of inference by each image segmentation network, thereby ensuring accurate watermark removal. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of the network architecture of the application environment for the technical solution of this application; Figure 2 This is a schematic diagram of the model architecture of the watermark removal model used in this application as an example; Figure 3 This is an exemplary principle block diagram of the internal structure of RSU7 in the exemplary watermark removal model of this application; Figure 4 This is a flowchart illustrating one embodiment of the watermark removal method of this application; Figure 5 This is a flowchart illustrating the process of training the first image segmentation network in an embodiment of this application. Figure 6 This is a flowchart illustrating the process of training the second image segmentation network in an embodiment of this application. Figure 7 This is a flowchart illustrating the process of training the generator by using a third image segmentation network as the generator and forming a generative adversarial network with a discriminator in an embodiment of this application. Figure 8 This is a flowchart illustrating the process of training the discriminator in a generative adversarial network in an embodiment of this application. Figure 9 This is a flowchart illustrating the process of training a third image segmentation network before constructing a generative adversarial network in this embodiment of the application. Figure 10 This is a schematic diagram illustrating the process of constructing a dataset in an embodiment of this application; Figure 11 This is a schematic block diagram of the watermark removal device of this application; Figure 12 This is a schematic diagram of the structure of a watermark removal device used in this application. Detailed Implementation

[0019] The models cited or potentially cited in this application, including traditional machine learning models or deep learning models, can be deployed on a remote server and invoked remotely on the client, or deployed on a client with the capability to invoke directly, unless explicitly specified in the text. In some embodiments, when running on the client, the corresponding intelligence can be obtained through transfer learning in order to reduce the requirements on the client's hardware operating resources and avoid excessive occupation of the client's hardware operating resources.

[0020] Please see Figure 1 The network architecture adopted in an exemplary application scenario of this application includes a terminal device 80, an independent website 81, and an application server 82. The application server 82 can be used to deploy a watermark removal service. The watermark removal service provides services by running a computer program product implemented according to the watermark removal method of this application and opening corresponding interfaces. The independent website 81 is also a server that can be used to deploy and open an online store for e-commerce services. Users on the terminal device 80 can submit or specify original images, such as product images, to the independent website 81 on the page of the online store. The independent website 81 calls the corresponding interface provided by the watermark removal service of the application server 82 to the product image, and the watermark removal service removes the watermark in the product image and returns the corresponding result image.

[0021] It should be pointed out that, Figure 1 The application scenario described here is merely an example of platform-based deployment. In other exemplary embodiments, the computer program product implemented according to the watermark removal method of this application can also run on any computer device with sufficient computing power to execute the various steps of the method and implement the watermark removal service. For example, it can run on the terminal device 80 or the independent station 81 to provide the watermark removal service.

[0022] Furthermore, the business scenarios of this application are not limited to e-commerce platform scenarios; it can be applied to any business scenario and to watermark removal processing of any type of image.

[0023] Please see Figure 2 This application provides an exemplary watermark removal model for watermark removal, comprising a first image segmentation network, a second image segmentation network, and a third image segmentation network. The watermark removal model of this application can be obtained by modifying any version of the U-Net series base model; U-Net is recommended. 2U-Net, applied to the salient object detection task, aims to segment the most attractive objects or regions in an image. It comprises a two-level nested U-Net structure. Firstly, the multiple Receptive Subtraction Unit (RSU) blocks employ receptive fields of varying sizes, capturing sufficient contextual information from different scales. Pooling operations are used on these RSU blocks, increasing the overall architecture depth without significantly increasing computational cost. Secondly, it accurately and effectively extracts multi-scale features within multi-stage upsampling and downsampling, and effectively aggregates multi-level features between the upsampling and downsampling stages. The first, second, and third image segmentation networks can all employ the same version of U-Net, such as U... 2 -Net, or use a different version, or use different versions respectively, as those skilled in the art can set as needed.

[0024] Based on the fundamental architecture of U-Net, it is a fully convolutional layer, so the size of the input and output images can be arbitrary. It consists of an encoder and a decoder. The encoder continuously compresses the image, decreasing the resolution, while the decoder continuously restores the image resolution, gradually increasing it. The decoder uses the features extracted by the encoder during the decoding process.

[0025] According to U 2 As can be seen from the basic architecture of -Net, it includes an encoding stage and a decoding stage. In the encoding stage, each stage is filled with a well-configured RSU (ReSidual U-block) (bottom U-shaped structure), such as RSU-7, RSU-6, RSU-5, RSU-4, and RSU-4F, respectively. The numbers "7", "6", "5", and "4" represent the number of layers in the RSU block. The number of layers is usually configured according to the spatial resolution of the input feature map. Adjacent RSU blocks differ by one upsampling and one downsampling operation. For example, RSU-7 has a 32x downsampling and a 32x upsampling, while RSU-6 has a 16x downsampling and a 16x upsampling. For feature maps with larger height and width, a larger number of layers is used to capture more large-scale semantic information. RSU-4F is an extended version of RSU-4, where the corresponding downsampling and upsampling operations are replaced by dilated convolutions, so that all intermediate feature maps of RSU-4F have the same resolution as their input feature maps. Furthermore, after each stage passes through a corresponding RSU block, it undergoes a max-pooling layer for downsampling by a factor of 2, halving the feature map size. In the decoding stage, each stage uses the RSU block employed by the corresponding encoding stage, and before each RSU block, it undergoes bilinear interpolation for upsampling by a factor of 2, halving the feature map size.

[0026] Please see Figure 3 Taking RSU-7 as an example, for the input feature map, during downsampling, features are first extracted through a 3x3 convolutional module consisting of conv (convolution), BN (Batch Normalization), and ReLU. Then, features are extracted through a 5-layer module consisting of 3x3 convolutional (convolution), BN (Batch Normalization), ReLU, and maxpooling layers. Downsampling is performed five times, ultimately downsampling the input feature map by a factor of 32. Since the resulting feature map is too small, dilated convolution is used. During upsampling, similar to U-Net, each upsampling layer fuses features from the same layer on the left and the layer above on the right after upsampling. Upsampling is performed five times, essentially upsampling the feature map from the five downsampled layers by a factor of 32 to restore it to the original image size.

[0027] The watermark removal model of this application can be pre-sampled and trained using corresponding training samples until it converges before being used in the inference stage. In the watermark removal model, the first image segmentation network, the second image segmentation network, and the third image segmentation network are trained by calculating loss values ​​based on different loss determination methods, which can more accurately determine the loss value of each image segmentation network and ensure that the image segmentation network trained iteratively based on the loss value has high accuracy.

[0028] Based on the principles revealed above, please refer to Figure 4 According to a watermark removal method provided in this application, in one embodiment, the method includes the following steps: Step S1100: Obtain the product watermark image; In an exemplary business scenario, on an online store of an independent website on an e-commerce platform, when a merchant needs to publish advertisements or edit product information, they can submit product images individually or in batches. These product images may contain watermarked images, i.e., the product watermark images. Using these watermarked images as input to the watermark removal model of this application, a product image with the watermark image removed can be obtained. The image content in the area covered by the watermark image can be restored to an effect that is more consistent with the main content of the image. During the process, there is no need to separately detect whether the product watermark image contains a watermark; that is, the watermark removal model can operate end-to-end, providing a product watermark image and obtaining a product image with the watermark image removed from it.

[0029] The watermark image mainly refers to a light watermark, which is visually represented by being placed on the main content of the image in a transparent manner to cover it. This way, when the human eye is viewing the main content, the content of the watermark image can be identified, but it will not interfere with the identification of the main content.

[0030] In another embodiment, the watermark image can also be a blind watermark, which is a watermark that is not visually recognizable to the human eye. Practice has shown that the watermark removal model of this application can also remove blind watermarks from the product watermark image, which is an unexpected technical effect derived from the process of eliminating light watermarks in this application.

[0031] Step S1200: Call the first image segmentation network in the preset watermark removal model and use the product watermark image as input to determine the watermark mask image; In one embodiment, since the image becomes smaller and smaller during the continuous valid convolution process, in order to avoid data loss, the product watermark image needs to be mirrored and enlarged before input. The four sides of the product watermark image are mirrored and enlarged to ensure that the result after a series of valid convolution operations can be consistent with the size of the original image.

[0032] In one embodiment, the first image segmentation network uses U 2-Net inputs the mirrored, enlarged product watermark image into the first image segmentation network of the preset watermark removal model to encode the product watermark image. In the encoding stage, each stage goes through the corresponding RSU-7, RSU-6, RSU-5, RSU-4, and RSU-4F. After each RSU block, it is downsampled by a factor of 2 by a maxpooling layer. After encoding, decoding is performed. In the decoding stage, each stage uses the RSU block used in the corresponding encoding stage. Similar to U-Net, the output of the previous decoding stage is upsampled by a factor of 2 using bilinear interpolation. The output of the same stage as the U-shaped structure's encoding stage is concatenated and input into the current stage's RSU block for feature extraction. At the same time, the output of the last stage of the encoding stage and each stage of the decoding stage are retained and subjected to convolution and upsampling operations of the corresponding number of times to restore the corresponding feature map to the original input image size. Then, the features after each operation are concatenated and convolved to obtain the corresponding feature vector. Based on the feature vector, the predicted watermark mask map is determined. The predicted watermark mask is a two-dimensional image containing the watermark confidence score for each pixel. The predicted watermark mask has the same pixel size and number of pixels as the corresponding product watermark image. The watermark confidence score represents the probability that a pixel in the predicted watermark mask, corresponding to a pixel in the same product watermark image, belongs to the watermark image, with a value range of [0,1]. Further, the predicted watermark mask is binarized, and it is determined whether the watermark confidence score for each pixel in the predicted watermark mask exceeds a preset threshold. If it exceeds the preset threshold, the watermark confidence score is set to 1; if it does not exceed the preset threshold, the watermark confidence score is set to 0. Based on this, the corresponding watermark image mask is obtained. The preset threshold can be set as needed by those skilled in the art; an exemplary example is 0.5.

[0033] Step S1300: The second image segmentation network in the watermark removal model uses the product watermark image and the watermark mask image as input to determine the watermark image; It is understood that when the watermark confidence level of a pixel in the watermark mask image is 1, it means that the pixel in the same product watermark image belongs to the pixel of the watermark image. When the watermark confidence level of a pixel in the watermark mask image is 0, it means that the pixel in the same product watermark image does not belong to the pixel of the watermark image. Accordingly, the second image segmentation network treats the watermark image as a target object for detection and segments the watermark image from the product watermark image according to the watermark mask image. That is, the watermark image is regarded as the foreground of the product watermark image, and the product image excluding the watermark image is regarded as the background of the product watermark image, so as to perform corresponding semantic segmentation based on the foreground and background.

[0034] In one embodiment, the second image segmentation network uses U 2 -Net, after stitching the mirrored and enlarged product watermark image with the watermark mask image, inputs it into the second image segmentation network of the preset watermark removal model to encode the product watermark image and the watermark mask image. In the encoding stage, each stage goes through the corresponding RSU-7, RSU-6, RSU-5, RSU-4, and RSU-4F. After each RSU block, it is downsampled by 2 times by a maxpooling layer. After encoding, decoding is performed. In the decoding stage, each stage uses the RSU block used in the corresponding encoding stage. Similar to U-Net, the output of the previous decoding stage is upsampled by 2 times using bilinear interpolation. The output of the same stage as the encoding stage of the U-shaped structure is stitched together and input into the RSU block of the current stage of the decoding stage for feature extraction. Simultaneously, the output results of the final stage in the encoding stage and each stage in the decoding stage are retained and subjected to convolution and upsampling operations of the corresponding number of times to restore the corresponding feature map to the size of the original input image. Then, the features after each operation are convolved to obtain the corresponding feature vector, and the watermark image is determined based on the feature vector.

[0035] Step S1400: The third image segmentation network in the watermark removal model uses the product watermark image, the watermark mask image, and the watermark image as input to determine the product image after removing the watermark image from the product watermark image.

[0036] It can be understood that the product watermark image is obtained by attaching the watermark image to the watermark-free product image based on the watermark mask image. Therefore, based on the product watermark image, the watermark image, and the watermark mask image, the watermark-free product image can be deduced in reverse. Accordingly, the watermark image in the product watermark image can be removed to obtain the corresponding product image.

[0037] In one embodiment, the third image segmentation network employs U... 2The U-Net concatenates the mirrored and enlarged product watermark image, the watermark mask image, and the watermark image, and inputs them into the second image segmentation network of the preset watermark removal model. This network encodes the product watermark image, watermark mask image, and watermark image. During the encoding stage, each stage passes through corresponding RSU-7, RSU-6, RSU-5, RSU-4, and RSU-4F blocks. After each RSU block, a maxpooling layer downsamples the data by a factor of 2. Decoding follows encoding. During the decoding stage, each stage uses the RSU blocks employed in the corresponding encoding stage. Similar to U-Net, the output of the previous decoding stage is upsampled by a factor of 2 using bilinear interpolation. This data is then concatenated with the output of the encoding stage in the same U-shaped structure and input into the current RSU block of the decoding stage for feature extraction. Meanwhile, the output results of the last stage of the encoding stage and each stage of the decoding stage are retained and subjected to convolution and upsampling operations of the corresponding number of times to restore the corresponding feature map to the size of the original input image. Then, the features after each operation are convolved to obtain the corresponding feature vector, and the product image is determined based on the feature vector.

[0038] As can be seen from the above embodiments, this application has many advantages, including but not limited to: This application utilizes a first image segmentation network in a pre-defined watermark removal model, taking the product watermark image as input, to determine a watermark mask image. A second image segmentation network in the same model then uses both the product watermark image and the watermark mask image as input to determine a watermark image. Finally, a third image segmentation network in the same model uses the product watermark image, the watermark mask image, and the watermark image as input to determine the product image after removing the watermark image from the product watermark image. On one hand, this achieves end-to-end watermark removal. The first and second image segmentation networks in the watermark removal model accurately determine the watermark mask image and watermark image required for watermark removal, and the third image segmentation network then accurately removes the watermark image from the product watermark image. On the other hand, each image segmentation network in the watermark removal model is relatively independent, allowing for independent training to obtain the functions of determining the watermark mask image, the watermark image, and removing the watermark. Each network performs its specific function, ensuring the accuracy of inference by each image segmentation network, thereby ensuring accurate watermark removal.

[0039] Please see Figure 5 In a further embodiment, the training process of the first image segmentation network in the watermark removal model includes the following steps: Step S2100: Obtain the first training sample and its first supervision label in the dataset. The first training sample is a sample watermark image, and the first supervision label is the target watermark mask image corresponding to the watermark image in the sample watermark image. The sample watermark image is obtained by adding one or more watermark images at random positions on the original watermark-free product image. The watermark image can be processed by any one or more of random colors, random transparency, random rotation angle, and random size. The specific implementation can be flexibly adapted by those skilled in the art.

[0040] The target watermark mask is a two-dimensional image containing the binarized watermark confidence score for each pixel. The target watermark mask has the same pixel size and number of pixels as the product watermark image. When the watermark confidence score of a pixel in the target watermark mask is 1, it means that the pixel in the target watermark mask belongs to the watermark image. When the watermark confidence score of a pixel in the target watermark mask is 0, it means that the pixel in the target watermark mask does not belong to the watermark image.

[0041] Step S2200: Input the first training sample into the first image segmentation network. After multiple downsampling stages and corresponding multiple upsampling stages, extract the corresponding deep semantic information and predict the predicted watermark mask image. In one embodiment, the first image segmentation network uses U 2-Net takes the first training sample as input and encodes it, i.e., the downsampling stage. Each stage's RSU block in the encoding stage of the first image segmentation network captures sufficient contextual information at corresponding different scales. After each stage's RSU block, it is downsampled by a factor of 2 using a maxpooling layer. After encoding, decoding, i.e., the upsampling stage, is performed. Each stage's RSU block in the decoding stage of the first image segmentation network (the same RSU block used in the encoding stage) upsamples the output of the previous stage by a factor of 2. This upsampling is then concatenated with the output of the encoding stage in the same stage of the U-shaped structure and input into the current stage's RSU block of the decoding stage for feature extraction. The output of the last stage in the encoding stage and each stage in the decoding stage are then convolved and subjected to the corresponding number of upsampling operations. Finally, the features after each operation are convolved to extract the corresponding deep semantic information and predict the predicted watermark mask. The predicted watermark mask is a two-dimensional image containing the watermark confidence level corresponding to each pixel. The predicted watermark mask has the same pixel size and number of pixels as the corresponding product watermark image. The watermark confidence level represents the probability that a pixel in the predicted watermark mask, which corresponds to a pixel in the same product watermark image, belongs to the pixel of the watermark image, and its value range is [0,1].

[0042] The upsampling is a deconvolution operation that can be implemented using any image interpolation algorithm, such as bilinear interpolation or nearest neighbor interpolation. Those skilled in the art can flexibly select and implement it as needed.

[0043] Step S2300: Calculate the cross-entropy loss value corresponding to the predicted watermark mask image based on the supervision label, and perform gradient update on the first image segmentation network based on the cross-entropy loss value until the first image segmentation network converges.

[0044] A preset cross-entropy loss function is invoked. This function can be flexibly set by those skilled in the art based on prior knowledge or experimental experience. Based on the supervision label of the first training sample, the sum of the cross-entropy loss values ​​between the predicted watermark mask and the watermark confidence of each corresponding pixel contained in the supervision label is calculated as the cross-entropy loss value corresponding to the predicted watermark mask. When the cross-entropy loss value reaches the corresponding preset threshold, it indicates that the first image segmentation network has been trained to a convergent state, and the training of the first image segmentation network can be terminated. When the cross-entropy loss value does not reach the corresponding preset threshold, it indicates that the first image segmentation network has not converged. Therefore, gradient updates are performed on the first image segmentation network based on the cross-entropy loss value. Typically, the weight parameters of each link of the first image segmentation network are corrected through backpropagation to make the first image segmentation network further approach convergence. Then, other first training samples are called to iteratively train the first image segmentation network until the first image segmentation network is trained to a convergent state.

[0045] This embodiment reveals the process of training the first image segmentation network to convergence, which can ensure the accuracy of the inference of the first image segmentation network after training to convergence and accurately determine the watermark image mask corresponding to the product watermark image.

[0046] Please see Figure 6 In a further embodiment, the training process of the second image segmentation network in the watermark removal model includes the following steps: Step S3100: Obtain the second training sample and its second supervision label in the dataset. The second training sample includes the sample watermark image and its corresponding watermark mask image. The second supervision label is the watermark image in the sample watermark image. The sample watermark image is obtained by adding one or more watermark images at random positions to an original watermark-free product image. The watermark image can be processed using any one or more of the following: random color, random transparency, random rotation angle, and random size. The specific implementation can be flexibly adapted by those skilled in the art. The watermark confidence level of the pixels corresponding to the added watermark images in the sample watermark image is set to 1, while the watermark confidence level of all pixels in the sample watermark image except those corresponding to the watermark images is set to 0. Based on this, a watermark mask image corresponding to the sample watermark image is constructed, containing the watermark confidence levels of each pixel in the product watermark image.

[0047] Step S3200: Input the second training sample into the second image segmentation network. After multiple downsampling stages and corresponding multiple upsampling stages, extract the corresponding deep semantic information and predict the predicted watermark image. In one embodiment, the second image segmentation network uses U 2The -Net takes a second training sample as input and encodes it (the downsampling stage). Each stage's RSU block in the encoding stage of the second image segmentation network captures sufficient contextual information at different scales. After each RSU block, it is downsampled by a factor of 2 using a maxpooling layer. In the decoding stage (the upsampling stage), each stage's RSU block in the decoding stage of the second image segmentation network (using the same RSU blocks as the encoding stage) upsamples the output of the previous stage by a factor of 2. This upsampling is then performed on the output of the encoding stage of the U-shaped structure and concatenated with it. This concatenated output is then input into the current stage's RSU block of the decoding stage for feature extraction. The outputs of the final stage in the encoding stage and each stage in the decoding stage are then convolved and upsampled a corresponding number of times. The features obtained from these concatenations are then convolved to extract deep semantic information and obtain a corresponding feature vector. Based on this feature vector, the predicted watermark image is determined.

[0048] Step S3300: Input the predicted watermark image and the second supervision label into a preset image feature extraction network to determine the corresponding first predicted image features and first target image features; The image feature extraction network described herein is suitable for extracting the semantic features of the input image. The specific model selected can be any readily available model, such as the ResNet series, VGG, MobileNet series, etc. In the recommended embodiment, the image feature extraction network is the MobileNetV2 model, which has the advantages of being lightweight and highly accurate, thus facilitating rapid deployment and enabling fast and accurate extraction of the image's semantic features with relatively low computational resource requirements. Since the training process of these readily available models is known to those skilled in the art, the training process will not be described in detail. Of course, those skilled in the art can also build their own image feature extraction network, as long as it can extract the semantic features of the input image.

[0049] In one embodiment, the image feature extraction network employs the MobileNetV2 model. The network is invoked with the predicted watermark image as input. Through an inverted residual structure with a linear bottleneck, the low-dimensional compressed representation of the predicted watermark image is first extended to a high-dimensional representation. Lightweight deep convolution (DW) is used to extract the image semantic features. Finally, linear convolution projects the image semantic features back to the low-dimensional representation to obtain the first predicted image features. Similarly, the image feature extraction network is invoked with a second supervised label as input to obtain the first target image features.

[0050] Step S3400: Determine the first similarity loss value corresponding to the first predicted image feature and the first target image feature, and perform gradient update on the second image segmentation network according to the first similarity loss value until the second image segmentation network converges.

[0051] The similarity between the first predicted image features and the first target image features is calculated as the first similarity loss value. The similarity calculation can be implemented by any of the large-scale vector retrieval engines such as Faiss, ElasticSearch, and Milvus, or by any existing algorithm such as cosine similarity, inner product, Manhattan distance, and Euclidean distance.

[0052] When the first similarity loss value exceeds the corresponding preset threshold, it indicates that the second image segmentation network has been trained to a convergent state, and the training of the second image segmentation network can be terminated. When the first similarity loss value does not exceed the corresponding preset threshold, it indicates that the second image segmentation network has not converged. Therefore, gradient updates are performed on the second image segmentation network based on the first similarity loss value. Typically, the weight parameters of each link of the second image segmentation network are corrected through backpropagation to make the second image segmentation network approach convergence. Then, other second training samples are used to iteratively train the second image segmentation network until the second image segmentation network is trained to a convergent state.

[0053] This embodiment reveals the process of training the second image segmentation network to convergence, which can ensure the accuracy of the inference of the second image segmentation network after training to convergence, and accurately determine the watermark image in the product watermark image.

[0054] Please see Figure 7 In a further embodiment, the training process of the third image segmentation network in the watermark removal model includes the following steps: Step S4100: Use the third image segmentation network as a generator, connect it to a pre-trained discriminator to construct a generative adversarial network, and solidify the weights of the discriminator. The Generative Adversarial Network (GAN) consists of a generator and a discriminator. The generator, also known as the third image segmentation network, generates a product image with the watermark removed. The discriminator acts as a binary classifier to determine whether the product image generated by the generator is truly watermark-free or falsely watermark-free.

[0055] Step S4200: Obtain a single third training sample from the dataset. The third training sample includes a sample watermark image and its corresponding watermark mask image and watermark image. The sample watermark image is obtained by adding one or more watermark images at random positions on the original watermark-free product image. The watermark image can be processed by any one or more of random colors, random transparency, random rotation angle, and random size. The specific implementation can be flexibly adapted by those skilled in the art.

[0056] The watermark image attached to the sample watermark image is the watermark image corresponding to the sample watermark image.

[0057] The watermark confidence level of the pixels corresponding to the watermark image attached to the sample watermark image is set to 1, while the watermark confidence level of the pixels in the sample watermark image other than the pixels corresponding to the watermark image is set to 0. Based on this, a watermark mask image corresponding to the sample watermark image is constructed, which includes the watermark confidence level of each pixel in the product watermark image.

[0058] Step S4300: Input the third training sample into the generator. After multiple downsampling stages and corresponding multiple upsampling stages, extract the corresponding deep semantic information and generate a predicted watermark-free image. In one embodiment, the third image segmentation network employs U... 2 -Net, which takes a third training sample as input, encodes the third training sample (the downsampling stage). Each stage's RSU block in the encoding stage of the third image segmentation network captures sufficient contextual information at corresponding different scales. After each stage's RSU block, it is downsampled by a factor of 2 using a max-pooling layer. After encoding, decoding (the upsampling stage) is performed. Each stage's RSU block in the decoding stage of the third image segmentation network (the same RSU block used in the encoding stage) upsamples the output of the previous stage by a factor of 2. This upsampling is then concatenated with the output of the encoding stage in the same stage of the U-shaped structure and input into the current stage's RSU block of the decoding stage for feature extraction. The output of the last stage in the encoding stage and each stage in the decoding stage are then convolved and subjected to the corresponding number of upsampling operations. The features after each operation are then convolved to extract the corresponding deep semantic information, obtaining the corresponding feature vector. Based on this feature vector, the predicted watermark-free image is determined.

[0059] Step S4400: Use the discriminator to perform binary classification mapping on the predicted watermark-free image to obtain the positive class probability, whereby the positive class probability represents the probability that the predicted watermark-free image is truly watermark-free. The discriminator can be implemented using fully connected layers or MLPs, where the fully connected layer can be a global convolution with kernels h and w, where h and w are the height and width of the previous convolution result, respectively. The discriminator takes the predicted watermark-free image as input and performs a binary classification mapping on it, that is, a linear mapping to a positive class space representing the input image as truly watermark-free and its positive probability, and a negative class space representing the input image as falsely watermark-free and its negative probability, to obtain the positive class probability.

[0060] Step S4500: When the probability of the positive class reaches a preset threshold, determine that the generator has converged and terminate the training; otherwise, call another third training sample to continue iterative training of the generator.

[0061] The preset threshold can be flexibly set by those skilled in the art based on prior knowledge and / or experimental data.

[0062] During each training iteration, a positive class probability can be obtained for a single third training sample. At this time, it can be assumed that the supervision label of the called third training sample is a positive sample, and the corresponding positive class probability mapped to the positive class space is 1. The cross-entropy loss value between the positive class probability corresponding to the predicted watermark-free image generated by the generator based on the third training sample and the positive class probability corresponding to the supervision label of the third training sample is calculated. The generator is backpropagated based on the cross-entropy loss value to realize gradient update. Then, another third training sample is called to carry out iterative training until the positive class probability corresponding to the predicted watermark-free image exceeds a preset threshold. The generator is then determined to have converged and training is terminated.

[0063] Through the above process, the generator is trained in the generative adversarial network. In this process, the discriminator acts as the referee to determine the probability that the predicted watermark image generated by the generator is truly watermark-free. If the probability does not reach the expected level, the weight parameters of the generator will be adjusted accordingly under the assumed supervision label, so that the actual ability of the generator will be continuously and effectively improved, and eventually reach a state of competing with the discriminator.

[0064] Furthermore, those skilled in the art should know that the training of the generator and discriminator of the adversarial neural network is for one side only. When the trained side is being trained, the weights of the other side are frozen, and the other side guides the training of the trained side according to the input of the trained side until convergence. Based on this, the generator and discriminator iterate continuously until the probability of the discriminator repeatedly judging that the predicted watermark image generated by the generator is truly watermark-free reaches Nash equilibrium. At this point, the generator can be produced as the generator of the adversarial neural network of this application.

[0065] In this embodiment, on the one hand, the training process of the generator of the adversarial neural network is revealed. The generator is trained with the weights of a pre-trained, converged discriminator fixed, enabling it to learn, after convergence, the ability to generate a predicted watermark-free image based on the sample watermark image and its corresponding watermark mask and watermark image. Subsequent calls to this generator can quickly and accurately generate a watermark-free image after removing the watermark image from the watermark image. On the other hand, it can be understood that by using the discriminator as a judge to evaluate the predicted watermark-free image generated by the generator, the confidence level of the predicted watermark-free image generated by the generator is effectively improved, ensuring the accuracy of watermark removal.

[0066] Please see Figure 8 In a further embodiment, before step S4100, which uses the third image segmentation network as a generator and connects it to a pre-trained discriminator to construct a generative adversarial network, and before solidifying the weights of the discriminator, the following steps are included: Step S4000: Obtain the fourth training sample and its fourth supervision label in the dataset, and supervise the training of the third image segmentation network until convergence. The fourth training sample includes the sample watermark image and its corresponding watermark mask image and watermark image. The fourth supervision label is the target watermark-free image after the watermark image in the sample watermark image has been removed in advance. The sample watermark image is obtained by adding one or more watermark images at random positions on the original watermark-free product image. The watermark image can be processed by any one or more of random colors, random transparency, random rotation angle, and random size. The specific implementation can be flexibly adapted by those skilled in the art.

[0067] The watermark image attached to the sample watermark image is the watermark image corresponding to the sample watermark image.

[0068] The watermark confidence level of the pixels corresponding to the watermark image attached to the sample watermark image is set to 1, while the watermark confidence level of the pixels in the sample watermark image other than the pixels corresponding to the watermark image is set to 0. Based on this, a watermark mask image corresponding to the sample watermark image is constructed, which includes the watermark confidence level of each pixel in the product watermark image.

[0069] The target watermark-free image can be obtained in advance by removing the watermark image from the sample watermark image using readily available watermark removal algorithms, plugins, computer programs, etc.

[0070] Step S4010: Use the third image segmentation network trained to convergence as a generator, solidify its weights, call the generator to generate the predicted watermark-free image corresponding to the fourth training sample as a negative sample for training the discriminator, and use the original image corresponding to the image without watermark attached to the fourth training sample as a positive sample. It is understandable that, for the same sample watermark image, the discriminator is committed to determining that the predicted watermark-free image generated by the generator based on the sample watermark image is a negative sample, that is, the predicted watermark-free image is falsely watermark-free; the original image before the watermark image is attached (the product image that was originally watermark-free) is a positive sample, that is, the original image is truly watermark-free.

[0071] Step S4020: Call the discriminator to perform binary classification mapping on the positive and negative samples respectively, and obtain the corresponding positive and negative class probabilities for each sample; The discriminator can be implemented using a fully connected layer or an MLP, where the fully connected layer is a global convolution with kernels h and w, where h and w are the height and width of the previous convolution result, respectively. The discriminator takes the positive samples as input and performs a binary classification mapping, i.e., a linear mapping to a positive class space representing a truly watermark-free input image and its positive class probability. Similarly, the discriminator takes the negative samples as input and performs a binary classification mapping, i.e., a linear mapping to a negative class space representing a false watermark-free input image and its negative class probability. Based on this, the positive class probability corresponding to the positive sample and the negative class probability corresponding to the negative sample are obtained.

[0072] Step S4030: When the positive class probability and the negative class probability simultaneously reach the corresponding preset threshold, the discriminator is determined to have converged and training is terminated; otherwise, another set of positive and negative samples is called to perform iterative training.

[0073] The preset thresholds corresponding to the positive class probability and the negative class probability can be flexibly set by those skilled in the art based on prior knowledge or experimental data.

[0074] During each training iteration, for a single set of positive and negative samples, corresponding positive and negative class probabilities can be obtained. At this point, it is assumed that the classification probability of a positive sample mapped to the positive class space is 1, and the classification probability of a negative sample mapped to the negative class space is 0. Based on this, the cross-entropy loss value between the positive class probability obtained by the discriminator and the corresponding supervision label is calculated. This is then added to the cross-entropy loss value between the negative class probability obtained by the discriminator and the corresponding supervision label to obtain the cross-entropy loss value predicted by the discriminator for this set of positive and negative samples. Backpropagation is then performed on the discriminator based on this cross-entropy loss value to update the gradient. Then, another set of positive and negative samples is used for iterative training until the predicted positive class probability of the positive sample and the negative class probability of the negative sample both exceed the corresponding preset thresholds. At this point, the discriminator is considered to have converged, and training is terminated.

[0075] After the above process, the discriminator is trained. Subsequently, the discriminator can be connected to the generator to construct an adversarial neural network, which acts as a judge for the generator in the network to judge the generator's generation.

[0076] This embodiment reveals the training process of the discriminator. After the discriminator is trained to convergence, it learns the ability to distinguish whether the predicted watermark-free image generated by the generator is truly watermark-free, so that after being connected to the generator, it can quickly distinguish the generator's generation.

[0077] Please see Figure 9 In a further embodiment, step S4000, obtaining the fourth training sample and its fourth supervision label in the dataset, and supervising the training of the third image segmentation network until convergence, includes the following steps: Step S4001: Obtain the fourth training sample and its fourth supervision label in the dataset. The fourth training sample includes the sample watermark image and its corresponding watermark mask image and watermark image. The fourth supervision label represents the target watermark-free image after the watermark image in the sample watermark image has been removed in advance. The sample watermark image is obtained by adding one or more watermark images at random positions on the original watermark-free product image. The watermark image can be processed by any one or more of random colors, random transparency, random rotation angle, and random size. The specific implementation can be flexibly adapted by those skilled in the art.

[0078] The watermark image attached to the sample watermark image is the watermark image corresponding to the sample watermark image.

[0079] The watermark confidence level of the pixels corresponding to the watermark image attached to the sample watermark image is set to 1, while the watermark confidence level of the pixels in the sample watermark image other than the pixels corresponding to the watermark image is set to 0. Based on this, a watermark mask image corresponding to the sample watermark image is constructed, which includes the watermark confidence level of each pixel in the product watermark image.

[0080] The target watermark-free image can be obtained in advance by removing the watermark image from the sample watermark image using readily available watermark removal algorithms, plugins, computer programs, etc.

[0081] Step S4002: Input the fourth training sample into the third image segmentation network. After multiple downsampling stages and corresponding multiple upsampling stages, extract the corresponding deep semantic information and generate a predicted watermark-free image. In one embodiment, the third image segmentation network employs U... 2-Net, which takes the fourth training sample as input, encodes the fourth training sample (the downsampling stage). The RSU blocks of each stage in the encoding stage of the third image segmentation network capture sufficient contextual information at corresponding different scales. After each RSU block, a maxpooling layer is used for downsampling by a factor of 2. After encoding, decoding (the upsampling stage) is performed. The RSU blocks of each stage in the decoding stage of the third image segmentation network (the same RSU blocks used in the encoding stage) upsample the output of the previous stage by a factor of 2. This upsampled output is concatenated with the output of the encoding stage of the U-shaped structure and input into the RSU block of the current stage of the decoding stage for feature extraction. The outputs of the last stage in the encoding stage and each stage in the decoding stage are then convolved and subjected to the corresponding number of upsampling operations. The features after each operation are then convolved to extract the corresponding deep semantic information and obtain the corresponding feature vector. Based on this feature vector, the predicted watermark-free image is determined.

[0082] Step S4003: Input the predicted watermark-free image and the fourth supervision label into a preset image feature extraction network to determine the corresponding second predicted image features and second target image features; In one embodiment, the image feature extraction network uses the first four layers of VGG to construct a model that balances feature extraction accuracy and execution efficiency. The image feature extraction network is invoked with the second predicted image features as input, and deep semantic information is extracted through multiple convolutional layers to obtain the second predicted image features. Similarly, the image feature extraction network is invoked with the fourth supervisory label as input to obtain the second target image features.

[0083] Step S4004: Determine the second similarity loss value corresponding to the second predicted image feature and the second target image feature, and perform gradient update on the third image segmentation network according to the second similarity loss value until the third image segmentation network converges.

[0084] The similarity between the second predicted image features and the second target image features is calculated as the second similarity loss value. The similarity calculation can be implemented by any of the large-scale vector retrieval engines such as Faiss, ElasticSearch, and Milvus, or by any existing algorithm such as cosine similarity, inner product, Manhattan distance, and Euclidean distance.

[0085] When the second similarity loss value exceeds the corresponding preset threshold, it indicates that the third image segmentation network has been trained to a convergent state, and the training of the third image segmentation network can be terminated. When the second similarity loss value does not exceed the corresponding preset threshold, it indicates that the third image segmentation network has not converged. Therefore, gradient updates are performed on the third image segmentation network based on the second similarity loss value. Typically, the weight parameters of each link of the third image segmentation network are corrected through backpropagation to make the third image segmentation network approach convergence. Then, other fourth training samples are called to iteratively train the third image segmentation network until the third image segmentation network is trained to a convergent state.

[0086] This embodiment reveals the process of training the third image segmentation network to a convergent state, which can ensure the accuracy of the inference of the third image segmentation network after training to convergence, and to a certain extent achieve the accuracy of watermark removal in the prior art.

[0087] Please see Figure 10 In a further embodiment, before obtaining the dataset, the following steps are included: Step S1000: Obtain multiple product images and attach a pre-processed watermark image to a random position on each product image. The pre-processing of the watermark image includes any one or more of the following: random color, random transparency, random rotation angle, and random size. The random color can be determined by randomly selecting a value from the corresponding value range [0, 255] for each of the three RGB channels, thus specifying a color. The number of pre-processed watermark images can be one or more. The random transparency can be randomly determined from the recommended transparency value range [0.3, 1]. The transparency should not be too low, otherwise the watermark effect will not be clear enough, so that the human eye can easily distinguish whether a watermark has been added. The transparency value range can also be set as needed by those skilled in the art based on this disclosure. The random rotation angle is randomly determined from the rotation angle value range (0°, 360°). The random size is a random setting of the pixel size of the watermark image. If it exceeds the pixel size of the product image, the oversized watermark image can be cropped to the pixel size of the product image accordingly.

[0088] Step S1010: Use the product image with the added watermark image as a sample watermark image to construct a dataset.

[0089] The sample watermark image can be used as the first training sample required for training the first image segmentation network. The corresponding target watermark mask image can be determined as the first supervision label based on the first training sample. That is, the watermark confidence of the pixels corresponding to the watermark image attached to the sample watermark image is set to 1, while the watermark confidence of the pixels in the sample watermark image other than the pixels corresponding to the watermark image is set to 0. Based on this, a watermark mask image corresponding to the sample watermark image containing the watermark confidence of each pixel in the product watermark image is constructed, and it is used as the target watermark mask image.

[0090] The sample watermark image and its corresponding watermark mask image can be used as the second training sample of the second image segmentation network, and the watermark image attached to the sample watermark image can be used as the second supervision label.

[0091] The sample watermark image, along with its attached watermark image and corresponding watermark mask, can serve as the third and fourth training samples for the third image segmentation network. The fourth supervisory label can be obtained in advance by removing the watermark image from the sample watermark image using readily available watermark removal algorithms, plugins, computer programs, etc. In summary, the training samples and supervisory labels contained in the dataset are constructed.

[0092] This embodiment reveals the construction of the dataset. During the process, various preprocessing steps are performed on the watermark images attached to the product images to fit various watermark addition scenarios. In addition, sufficient training samples are constructed for the image segmentation network to train, ensuring the accuracy of the image segmentation network after training to convergence.

[0093] Please see Figure 11 According to one aspect of this application, a watermark removal device includes an image acquisition module 1100, a mask value determination module 1200, a watermark determination module 1300, and a watermark removal module 1400, wherein: the image acquisition module 1100 is used to acquire a product watermark image; the mask value determination module 1200 is used to call a first image segmentation network in a preset watermark removal model to determine a watermark mask image using the product watermark image as input; the watermark determination module 1300 is used to determine a watermark image using a second image segmentation network in the watermark removal model using the product watermark image and the watermark mask image as input; and the watermark removal module 1400 is used to determine a product image after removing the watermark image from the product watermark image using a third image segmentation network in the watermark removal model using the product watermark image, the watermark mask image, and the watermark image as input.

[0094] In a further embodiment, the training process of the first image segmentation network in the watermark removal model includes: a first data acquisition submodule, used to acquire a first training sample and its first supervision label in the dataset, wherein the first training sample is a sample watermark image and the first supervision label is a target watermark mask image corresponding to the watermark image in the sample watermark image; a first feature extraction submodule, used to input the first training sample into the first image segmentation network, extract the corresponding deep semantic information through multiple downsampling stages and corresponding multiple upsampling stages, and predict the predicted watermark mask image; and a first iteration submodule, used to calculate the cross-entropy loss value corresponding to the predicted watermark mask image based on the supervision label, and perform gradient updates on the first image segmentation network based on the cross-entropy loss value until the first image segmentation network converges.

[0095] In a further embodiment, the training process of the second image segmentation network in the watermark removal model includes: a second data acquisition submodule, used to acquire a second training sample and its second supervision label in the dataset, wherein the second training sample includes a sample watermark image and its corresponding watermark mask image, and the second supervision label is the watermark image in the sample watermark image; a first prediction submodule, used to input the second training sample into the second image segmentation network, extract the corresponding deep semantic information through multiple downsampling stages and corresponding multiple upsampling stages, and predict the predicted watermark image; a first image feature extraction submodule, used to input the predicted watermark image and the second supervision label into a preset image feature extraction network respectively, and determine the corresponding first predicted image features and first target image features; and a second iteration submodule, used to determine the first similarity loss value between the first predicted image features and the first target image features, and perform gradient updates on the second image segmentation network according to the first similarity loss value until the second image segmentation network converges.

[0096] In a further embodiment, the training process of the third image segmentation network in the watermark removal model includes: an adversarial construction submodule, used to construct a generative adversarial network by using the third image segmentation network as a generator and connecting it to a pre-trained discriminator that has converged, and solidifying the weights of the discriminator; a third data acquisition submodule, used to acquire a single third training sample in the dataset, the third training sample including a sample watermark image and its corresponding watermark mask image and watermark image; a third feature extraction submodule, used to input the third training sample into the generator, and extract the corresponding deep semantic information through multiple downsampling stages and corresponding multiple upsampling stages to generate a predicted watermark-free image; a generation and discrimination submodule, used to perform binary classification mapping on the predicted watermark-free image using the discriminator to obtain a positive class probability, the positive class probability representing the probability that the predicted watermark-free image is truly watermark-free; and a third iteration submodule, used to determine that the generator has converged and terminate training when the positive class probability reaches a preset threshold, otherwise calling another third training sample to continue iterative training of the generator.

[0097] In a further embodiment, the third image segmentation network is used as a generator, and it is connected to a pre-trained discriminator to construct a generative adversarial network. Before the discriminator's weights are fixed, the following steps are included: a fourth data acquisition submodule, used to acquire a fourth training sample and its fourth supervision label in the dataset, and supervise the training of the third image segmentation network until convergence. The fourth training sample includes a sample watermark image and its corresponding watermark mask image and watermark image. The fourth supervision label is a target watermark-free image after the watermark image in the sample watermark image has been removed beforehand; and a discriminator sample determination submodule, used to determine the discriminator sample weights. The three-image segmentation network acts as a generator, with its weights fixed. It generates a predicted watermark-free image corresponding to the fourth training sample as a negative sample for training the discriminator, and the original image corresponding to the image without a watermark attached to the fourth training sample is used as a positive sample. The discriminator classification submodule is used to call the discriminator to perform binary classification mapping on the positive and negative samples respectively, and obtain the corresponding positive and negative class probabilities for each sample. The fourth iteration submodule is used to determine that the discriminator has converged and terminate training when the positive and negative class probabilities reach the corresponding preset thresholds at the same time; otherwise, it calls another set of positive and negative samples to perform iterative training.

[0098] In a further embodiment, acquiring a fourth training sample and its fourth supervision label from the dataset, and supervising the training of the third image segmentation network until convergence, includes: a fifth data acquisition submodule, used to acquire the fourth training sample and its fourth supervision label from the dataset, wherein the fourth training sample includes a sample watermark image and its corresponding watermark mask image and watermark image, and the fourth supervision label represents the target watermark-free image after the watermark image in the sample watermark image has been removed in advance; a second prediction submodule, used to input the fourth training sample into the third image segmentation network, and extract the corresponding deep semantic information through multiple downsampling stages and corresponding multiple upsampling stages to generate a predicted watermark-free image; a second image feature extraction module, used to input the predicted watermark-free image and the fourth supervision label into a preset image feature extraction network respectively to determine the corresponding second predicted image features and second target image features; and a fifth iteration submodule, used to determine the second similarity loss value between the second predicted image features and the second target image features, and perform gradient updates on the third image segmentation network according to the second similarity loss value until the third image segmentation network converges.

[0099] In a further embodiment, before acquiring the dataset, the following is included: a watermark attachment submodule, used to acquire multiple product images and attach a pre-processed watermark image to a random position on each product image. The pre-processing of the watermark image includes any one or more of the following: random color, random transparency, random rotation angle, and random size; and a dataset construction submodule, used to construct a dataset by using the product images with the attached watermark images as sample watermark images.

[0100] Another embodiment of this application also provides a watermark removal device. For example... Figure 12 The diagram shows the internal structure of a watermark removal device. This device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. The computer-readable, non-volatile storage medium stores an operating system, a database, and computer-readable instructions. The database stores information sequences, and when executed by the processor, these computer-readable instructions enable the processor to implement a watermark removal method.

[0101] The processor of the watermark removal device provides computing and control capabilities to support the operation of the entire device. The device's memory can store computer-readable instructions, which, when executed by the processor, cause the processor to perform the watermark removal method of this application. The network interface of the watermark removal device is used for communication with a terminal.

[0102] Those skilled in the art will understand that Figure 12The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the watermark removal device to which the present application is applied. A specific watermark removal device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0103] In this embodiment, the processor is used to execute... Figure 11 The specific functions of each module are described, and the memory stores the program code and various data required to execute the above modules or sub-modules. The network interface is used to realize data transmission between user terminals or servers. The non-volatile readable storage medium in this embodiment stores the program code and data required to execute all modules in the watermark removal device of this application, and the server can call the server's program code and data to execute the functions of all modules.

[0104] This application also provides a non-volatile readable storage medium storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps of the watermark removal method of any embodiment of this application.

[0105] This application also provides a computer program product, including a computer program / instructions that, when executed by one or more processors, implement the steps of the method described in any embodiment of this application.

[0106] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The aforementioned storage medium can be a computer-readable storage medium such as a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM).

[0107] In summary, this application can accurately remove watermarks from the original image, resulting in a higher quality image restoration effect.

Claims

1. A method for removing watermarks from e-commerce images, characterized in that, Includes the following steps: Obtain product watermark images; The first image segmentation network in the preset watermark removal model is called with the product watermark image as input. After multiple downsampling stages and corresponding multiple upsampling stages, the corresponding deep semantic information is extracted to determine the watermark mask image. The first image segmentation network is pre-trained under supervision to convergence and learns the ability to determine the watermark image mask of the product watermark image. The second image segmentation network in the watermark removal model takes the product watermark image and the watermark mask image as input, and extracts the corresponding deep semantic information through multiple downsampling stages and corresponding multiple upsampling stages to determine the watermark image. The second image segmentation network is pre-trained under supervised training to convergence and learns the ability to determine the watermark image of the product watermark image. The third image segmentation network in the watermark removal model takes the product watermark image, the watermark mask image, and the watermark image as input. After multiple downsampling stages and corresponding upsampling stages, it extracts the corresponding deep semantic information and determines the product image after removing the watermark image from the product watermark image. The third image segmentation network is pre-trained under supervised supervision to convergence and then used as a generator to perform generative adversarial training with a preset discriminator until it is trained to convergence again, thus learning the ability to remove the watermark image from the product watermark image.

2. The method for removing watermarks from e-commerce images according to claim 1, characterized in that, The training process of the first image segmentation network in the watermark removal model includes the following steps: Obtain the first training sample and its first supervision label in the dataset. The first training sample is the sample watermark image, and the first supervision label is the target watermark mask image corresponding to the watermark image in the sample watermark image. The first training sample is input into the first image segmentation network. After multiple downsampling stages and corresponding multiple upsampling stages, the corresponding deep semantic information is extracted, and the predicted watermark mask image is predicted. The cross-entropy loss value corresponding to the predicted watermark mask is calculated based on the supervision label, and the first image segmentation network is updated with gradients based on the cross-entropy loss value until the first image segmentation network converges.

3. The method for removing watermarks from e-commerce images according to claim 1, characterized in that, The training process of the second image segmentation network in the watermark removal model includes the following steps: Obtain the second training sample and its second supervision label from the dataset. The second training sample includes the sample watermark image and its corresponding watermark mask image. The second supervision label is the watermark image in the sample watermark image. The second training sample is input into the second image segmentation network. After multiple downsampling stages and corresponding multiple upsampling stages, the corresponding deep semantic information is extracted, and the predicted watermark image is predicted. The predicted watermark image and the second supervision label are respectively input into a preset image feature extraction network to determine the corresponding first predicted image features and first target image features; A first similarity loss value is determined between the first predicted image features and the first target image features. The second image segmentation network is then updated using gradients based on the first similarity loss value until the second image segmentation network converges.

4. The method for removing watermarks from e-commerce images according to claim 1, characterized in that, The training process of the third image segmentation network in the watermark removal model includes the following steps: The third image segmentation network is used as a generator, and it is connected to a pre-trained discriminator to construct a generative adversarial network, thus solidifying the weights of the discriminator. Obtain a single third training sample from the dataset, the third training sample including the sample watermark image and its corresponding watermark mask image and watermark image; The third training sample is input into the generator, and after multiple downsampling stages and corresponding multiple upsampling stages, the corresponding deep semantic information is extracted to generate a predicted watermark-free image. The discriminator is used to perform binary classification mapping on the predicted watermark-free image to obtain the positive class probability, which represents the probability that the predicted watermark-free image is truly watermark-free. When the probability of the positive class reaches a preset threshold, the generator is determined to have converged and training is terminated; otherwise, another third training sample is called to continue iterative training of the generator.

5. The method for removing watermarks from e-commerce images according to claim 4, characterized in that, The third image segmentation network is used as a generator. Before connecting it to a pre-trained and converged discriminator to construct a generative adversarial network and fixing the weights of the discriminator, the following steps are included: Obtain the fourth training sample and its fourth supervision label in the dataset, and supervise the training of the third image segmentation network until convergence. The fourth training sample includes the sample watermark image and its corresponding watermark mask image and watermark image. The fourth supervision label is the target watermark-free image after the watermark image in the sample watermark image has been removed in advance. The third image segmentation network trained to convergence is used as a generator, its weights are fixed, and the generator is called to generate the predicted watermark-free image corresponding to the fourth training sample as a negative sample for training the discriminator, and the original image corresponding to the image without watermark attached to the fourth training sample is used as a positive sample. The discriminator is invoked to perform binary classification mapping on the positive and negative samples respectively, and the corresponding positive and negative class probabilities of each sample are obtained. When both the positive and negative class probabilities reach their respective preset thresholds, the discriminator is deemed to have converged and training is terminated; otherwise, another set of positive and negative samples is used for iterative training.

6. The method for removing watermarks from e-commerce images according to claim 5, characterized in that, Obtain the fourth training sample and its fourth supervision label from the dataset, and supervise the training of the third image segmentation network until it converges, including the following steps: Obtain the fourth training sample and its fourth supervision label in the dataset. The fourth training sample includes the sample watermark image and its corresponding watermark mask image and watermark image. The fourth supervision label represents the target watermark-free image after the watermark image in the sample watermark image has been removed in advance. The fourth training sample is input into the third image segmentation network. After multiple downsampling stages and corresponding multiple upsampling stages, the corresponding deep semantic information is extracted to generate a predicted watermark-free image. The predicted watermark-free image and the fourth supervision label are respectively input into a preset image feature extraction network to determine the corresponding second predicted image features and second target image features; A second similarity loss value is determined between the second predicted image features and the second target image features. The third image segmentation network is then updated using gradients based on this second similarity loss value until the third image segmentation network converges.

7. The method for removing watermarks from e-commerce images according to any one of claims 2 to 6, characterized in that, Before obtaining the dataset, the following should be included: Multiple product images are acquired, and a pre-processed watermark image is attached to a random position on each product image. The pre-processing of the watermark image includes any one or more of the following: random color, random transparency, random rotation angle, and random size. The dataset is constructed by using product images with watermarked images as sample watermark images.

8. A watermark removal device, characterized in that, include: The image acquisition module is used to acquire product watermark images; The mask value determination module is used to call the first image segmentation network in the preset watermark removal model, take the product watermark image as input, and extract the corresponding deep semantic information through multiple downsampling stages and corresponding multiple upsampling stages to determine the watermark mask image. The first image segmentation network is pre-trained under supervision to convergence and learns the ability to determine the watermark image mask of the product watermark image. The watermark determination module is used to determine the watermark image by taking the product watermark image and the watermark mask image as input from the second image segmentation network in the watermark removal model, and performing feature extraction processing through multiple downsampling stages and corresponding multiple upsampling stages. The second image segmentation network is pre-trained under supervised training to convergence and learns the ability to determine the watermark image of the product watermark image. The watermark removal module is used to extract the corresponding deep semantic information from the product watermark image, the watermark mask image, and the watermark image as input by the third image segmentation network in the watermark removal model. After multiple downsampling stages and corresponding multiple upsampling stages, the module determines the product image after removing the watermark image from the product watermark image. The third image segmentation network is pre-trained under supervised supervision to convergence and then used as a generator to perform generative adversarial training with a preset discriminator until it is trained to convergence again, thus learning the ability to remove the watermark image from the product watermark image.

9. A watermark removal device, comprising a central processing unit and a memory, characterized in that, The central processing unit is used to invoke and run a computer program stored in the memory to perform the steps of the method as described in any one of claims 1 to 7.

10. A non-volatile readable storage medium, characterized in that, It stores, in the form of computer-readable instructions, a computer program implemented according to any one of claims 1 to 7, which, when invoked by a computer, executes the steps included in the corresponding method.

Images