Method and device for removing watermark from image, electronic equipment and storage medium

Through the combination of image encoding model and stable diffusion model, image redrawing is performed using implicit vectors and prompt text, which solves the problem of large pixel loss when removing watermarks in the prior art, and the generated image is more natural and complete.

CN118918031APending Publication Date: 2024-11-08GUANGZHOU QUYAN NETWORK TECH CO LTD
View PDF 0 Cites 4 Cited by

Patent Information

Application Number
CN202411177025.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-26
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

Existing machine learning models have difficulty recovering pixel losses when removing watermarks from images, especially for larger watermarks, resulting in unnatural and incomplete images after removal.

Method used

Implicit vectors are extracted through the image encoding model, combined with the watermark detection model and the image label prediction model to generate prompt text, input a stable diffusion model to redraw, and use the implicit vector and prompt text to guide the watermark mask area to redraw and generate the original The unmarked image blocks that match the overall semantics and structure of the graph.

Benefits of technology

It effectively reduces the pixel loss after removing the watermark, and the generated image is more natural and complete, combining image structure and semantic information, improving the effect of removing the watermark.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918031A_ABST
    Figure CN118918031A_ABST
Patent Text Reader

Abstract

The invention discloses an image watermark removing method and device, electronic equipment and a storage medium, and the method comprises the steps: carrying out the coding of a first image comprising a watermark through an image coding model, obtaining an implicit vector of the first image, inputting the first image into a watermark detection model, and obtaining a second image comprising a watermark mask, inputting the first image into an image tag prediction model to obtain an explicit semantic tag of the first image, generating a prompt text based on the explicit semantic tag, inputting the first image, the implicit vector and the prompt text into a stable diffusion model, and performing stable diffusion on the first image by taking the implicit vector and the prompt text as guidance in the stable diffusion model. And redrawing the watermark mask area in the second image based on the first image to obtain a third image after the watermark is removed, the image block of the watermark mask area which is free of the watermark and accords with the first image in overall semantics and structure can be generated, the watermark-free image generated through the image block is small in pixel loss, and the image quality is improved. And the image with the watermark removed is more natural and complete.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method, device, electronic device and storage medium for removing watermarks from an image. Background Art

[0002] With the development of the Internet, digital images are spread and shared through the Internet, and people can download images for use. In order to protect the rights of image creators, watermarks are added to images. The added watermarks interfere with and destroy the information of the image. After the image is downloaded, the watermark needs to be removed before it is used.

[0003] Currently, watermarks in images can be removed by training machine learning models. However, existing machine learning models cannot combine the overall image, image structure, and image semantics to remove watermarks. Especially for large watermarks in images, it is difficult to restore the pixels lost due to watermark removal after the watermarks are removed by machine learning models. Summary of the invention

[0004] The present invention provides a method, device, electronic device and storage medium for removing watermarks from images to solve the problem that the current removal of watermarks from images through machine learning models causes loss of image pixels.

[0005] In a first aspect, the present invention provides a method for removing watermarks from an image, comprising:

[0006] Encoding the first image including the watermark by using an image coding model to obtain an implicit vector of the first image;

[0007] Inputting the first image into a watermark detection model to obtain a second image including a watermark mask;

[0008] Inputting the first image into an image label prediction model to obtain an explicit semantic label of the first image, and generating a prompt text based on the explicit semantic label;

[0009] The first image, implicit vector and prompt text are input into a stable diffusion model. In the stable diffusion model, the implicit vector and prompt text are used as a guide to redraw the watermark mask area in the second image based on the first image to obtain a third image after the watermark is removed.

[0010] In a second aspect, the present invention provides a device for removing watermarks from an image, comprising:

[0011] An image implicit coding module, used for encoding a first image including a watermark through an image coding model to obtain an implicit vector of the first image;

[0012] A watermark detection module, configured to input the first image into a watermark detection model to obtain a second image including a watermark mask;

[0013] a prompt text generation module, configured to input the first image into an image label prediction model to obtain an explicit semantic label of the first image, and generate a prompt text based on the explicit semantic label;

[0014] An image redrawing module is used to input the first image, implicit vector and prompt text into a stable diffusion model, and in the stable diffusion model, the implicit vector and prompt text are used as a guide to redraw the watermark mask area in the second image based on the first image to obtain a third image after removing the watermark.

[0015] In a third aspect, the present invention provides an electronic device, the electronic device comprising:

[0016] at least one processor; and

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the method for removing watermarks from an image as described in the first aspect of the present invention.

[0019] In a fourth aspect, the present invention provides a computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a processor to implement the method for removing watermarks from an image as described in the first aspect of the present invention when executed.

[0020] The embodiment of the present invention obtains an implicit vector of the first image by encoding the first image including a watermark, generates a prompt text by predicting the explicit semantic label of the first image, and detects the watermark to generate a second image including a watermark mask. In a stable diffusion model, the implicit vector and the prompt text of the first image are used as a guide to redraw the watermark mask area based on the first image to obtain a third image after removing the watermark. The watermark mask area can be redrawn in combination with the image structure of the first image including the watermark, the implicit semantics of the overall image and the display semantics of the first image to generate an image block of the watermark mask area without a watermark that is consistent with the overall semantics and structure of the first image. The watermark-free image generated by the image block has a small pixel loss, and the image after removing the watermark is more natural and complete.

[0021] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 This is a flow chart of a method for removing watermarks from an image provided by the first embodiment of the present invention;

[0024] Figure 2 is a flow chart of a method for removing watermarks from an image provided by Embodiment 2 of the present invention;

[0025] Figure 3 It is a schematic diagram of the CLIP model architecture;

[0026] Figure 4 It is a schematic diagram of watermark detection generating watermark mask;

[0027] Figure 5 It is a schematic diagram of the architecture of the stable diffusion model;

[0028] Figure 6 This is a schematic diagram of IP Adapert guiding U-Net reverse diffusion denoising;

[0029] Figure 7 It is a schematic diagram of the overall process of removing watermarks from an image according to an embodiment of the present invention;

[0030] Figure 8 It is a schematic diagram of the internal painting of the stable diffusion model;

[0031] Fig. 9 This is a schematic diagram of image comparison before and after watermark removal;

[0032] Fig.10 It is a structural schematic diagram of a device for removing watermarks from an image provided by Embodiment 3 of the present invention;

[0033] Fig.11 It is a schematic diagram of the structure of an electronic device provided in Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0034] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0035] Embodiment 1

[0036] Figure 1 This is a flow chart of a method for removing watermarks from an image provided by the first embodiment of the present invention. This embodiment is applicable to the case of removing watermarks from an image. The method can be performed by a device for removing watermarks from an image. The device for removing watermarks from an image can be implemented in the form of hardware and / or software and can be configured in an electronic device. Figure 1 As shown, the method for removing watermarks from an image includes:

[0037] S101. Encode a first image including a watermark by using an image coding model to obtain an implicit vector of the first image.

[0038] In this embodiment, the watermark may be a plain text watermark, which is a mark visible in the image, used to identify and prove copyright, source and other information. The watermark may be a semi-transparent logo, icon or text in the image. The first image including the watermark may be a single photographic or painting image, or an image clipped from a video. The source of the first image is different in different watermark removal scenarios.

[0039] The image coding model may be a pre-trained neural network model that can extract implicit vectors of images, wherein the implicit vectors may be implicit features representing the overall semantics of the image, image structure, etc. In one example, the image coding model may be CLIP (Contrastive Language-Image Pre-Training, a pre-training model based on contrasting text-image pairs), and of course it may be other models. The embodiments of the present invention do not limit the image coding model as long as it can extract implicit vectors of the image.

[0040] After acquiring the first image to be processed including the watermark, this embodiment may input the first image into a pre-trained image coding model, encode the first image in the image coding model, and extract the implicit vector of the first image.

[0041] S102: Input the first image into a watermark detection model to obtain a second image including a watermark mask.

[0042] The watermark detection model of this embodiment may be a pre-trained model that can output a watermark detection frame in an image. The watermark detection frame is usually a rectangular detection frame. Exemplarily, the watermark detection model may be YOLO (You Only LookOnce). YOLO is used to implement real-time object detection. Specifically in this embodiment, YOLO is used to detect the watermark in the input first image, and output the minimum circumscribed rectangular frame of the watermark as the detection frame, and generate a watermark mask in the first image in the area where the detection frame is located, to obtain a second image including the watermark mask.

[0043] Of course, the watermark detection model may also be SSD (Single-Shot Detector), FASTER R-CNN, CenterNet, etc. This embodiment does not limit the type of watermark detection model.

[0044] S103: Input the first image into an image label prediction model to obtain an explicit semantic label of the first image, and generate a prompt text based on the explicit semantic label.

[0045] The semantics of an image may be the meaning of the image content. The semantics may be expressed through language. The explicit semantic label may be text that can express the explicit content in the first image. The explicit content may be the image content that is intuitively visible in the first image, such as objects in the image, the colors of the objects, and other content.

[0046] In this embodiment, an image label prediction model can be pre-trained, and at least one explicit semantic label of an image can be predicted through the image label prediction model. After the first image is input into the image label prediction model, multiple explicit semantic labels of the first image can be obtained. The explicit semantic labels can be represented by characters, words, and sentences. After further synonymization of the explicit semantic labels or expansion by calling a Vincent model, the prompt text of the first image can be obtained, wherein the prompt text can be text used in the Vincent diagram to guide the generation of the image.

[0047] S104, inputting the first image, the implicit vector and the prompt text into a stable diffusion model, and redrawing the watermark mask area in the second image based on the first image in the stable diffusion model with the implicit vector and the prompt text as guidance, to obtain a third image after the watermark is removed.

[0048] The stable diffusion model is used to generate images. It mainly introduces noise into the image in the forward diffusion stage until the image becomes completely random noise. In the backward diffusion stage, the prediction noise is removed under the guidance of the input conditions to generate an image that meets the conditions.

[0049] Specifically in this embodiment, in the stable diffusion model, the first image is first encoded in the forward diffusion stage, and then noise is added to obtain the noisy image features. In the backward diffusion stage, the implicit vector and prompt text of the first image are used as a guide to perform noise prediction on the noisy image features in the backward diffusion stage to remove the noise. The watermark mask area is redrawn under the guidance of the implicit vector and the prompt text to obtain an image block of the watermark mask area, so that the image block is consistent with the overall semantics and image structure represented by the implicit vector and prompt text of the first image. The image block replaces the watermark mask area and is fused to obtain a third image after the watermark is removed.

[0050] The embodiment of the present invention obtains an implicit vector of the first image by encoding the first image including a watermark, generates a prompt text by predicting the explicit semantic label of the first image, and detects the watermark to generate a second image including a watermark mask. In a stable diffusion model, the implicit vector and the prompt text of the first image are used as a guide to redraw the watermark mask area based on the first image to obtain a third image after removing the watermark. The watermark mask area can be redrawn in combination with the image structure of the first image including the watermark, the implicit semantics of the overall image and the display semantics of the first image to generate an image block of the watermark mask area without a watermark that is consistent with the overall semantics and structure of the first image. The watermark-free image generated by the image block has a small pixel loss, and the image after removing the watermark is more natural and complete.

[0051] Embodiment 2

[0052] Figure 2 This is a flow chart of a method for removing watermarks from an image provided by Embodiment 2 of the present invention. Embodiment 2 of the present invention is optimized based on Embodiment 1 above. Figure 2 As shown, the method for removing watermarks from an image includes:

[0053] S201. Encode a first image including a watermark by using an image coding model to obtain an implicit vector of the first image.

[0054] In this embodiment, the image coding model can adopt the CLIP model, such as Figure 3The figure shows the schematic diagram of the architecture of the CLIP model. The CLIP model includes a text encoder Text Encoder and an image encoder Image Encoder. When pre-training the CLIP model, a large number of image-text pairs can be obtained first, wherein the text is used to describe the overall semantics of the image, the implicit semantics such as the image structure, and a batch of images and texts are extracted and input into the image encoder Image Encoder and the text encoder Text Encoder respectively. Multiple texts are encoded by the text encoder Text Encoder to obtain multiple text features Ti, and multiple images are encoded by the image encoder Image Encoder to obtain multiple image features Ii. After unifying the multiple text features Ti and image features Ii to the same dimension, the cosine similarity of the image feature Ii and the text feature Ti is calculated, and the image loss rate is calculated by the cross entropy loss function, the cosine similarity and the image label, and the text loss rate is calculated by the cross entropy loss function, the cosine similarity and the text label. The average of the image loss rate and the text loss rate is calculated as the total loss rate, and the image encoder Image Encoder and the text encoder Text are adjusted by the total loss rate. The parameters of the Encoder are adjusted until the total loss rate is less than the threshold. The pre-training process of the CLIP model can refer to the method of the CLIP model in the prior art and will not be described in detail here.

[0055] Through the pre-trained CLIP model, the feature vector obtained by the image encoder can express the implicit semantics of the image, such as the overall semantics and image structure, as the implicit vector of the image.

[0056] In this embodiment, after the CLIP model is trained by using images and texts describing implicit semantics such as the overall semantics of the image and the image structure, the CLIP model can encode the image to obtain an implicit vector of the image. In order to make the pre-trained CLIP model ignore the watermark when encoding an image including a watermark, the pre-trained CLIP model can be fine-tuned. Specifically, a training image can be obtained, the training image includes a first sample image without a watermark and a second sample image after a watermark is added to the first sample image. The first sample image and the second sample image are input into the CLIP model to obtain a first image feature of the first sample image and a second image feature of the second sample image. The loss rate is calculated using the first image feature and the second image feature to determine whether the preset training conditions are met. If so, it is determined that the CLIP model has completed training; if not, the loss rate is used to adjust the model parameters of the CLIP model, and the process returns to the step of inputting the first sample image and the second sample image into the CLIP model.

[0057] Exemplarily, a first sample image without a watermark and a second sample image after a watermark is added to the first sample image can be input into the image encoder Image Encoder of the CLIP model to obtain a first image feature I_e of the first sample image. img_wo_wm and the second image feature I_e of the second sample image img_wm , the loss rate is calculated by the following formula:

[0058] L wm_cos =1-cos_sim(I_e img_wo_wm , I_e img_wm );

[0059] Among them, cos_sim represents the calculation of the first image feature I_e img_wo_wm and the second image feature I_e img_wm The cosine similarity of .

[0060] By fine-tuning the CLIP model through the above loss rate, the CLIP model can be constrained to ignore the watermark in the image when encoding the second sample image containing the watermark, so that the second image feature of the second sample image is close to the first image feature of the first sample image without the watermark, so that the fine-tuned CLIP model ignores the watermark and focuses more on implicit semantic encoding such as the overall semantics of the image and the image structure.

[0061] S202: Input the first image into a watermark detection model to obtain a second image including a watermark mask.

[0062] Specifically, the first image can be input into the watermark detection model to obtain a watermark detection frame, and a watermark mask is generated in the watermark detection frame area in the first image to obtain a second image including the watermark mask, wherein the watermark detection model can be a target detection model such as YOLOv8, SSD, FASTER R-CNN, CenterNet, etc. This embodiment does not limit the type of watermark detection model. The specific training method can refer to the existing technology and will not be described in detail here.

[0063] like Figure 4 As shown, after the first image a is input into the watermark detection model, a watermark detection frame c (the minimum circumscribed rectangular frame of the text "copyright" in the image) is obtained in the first image a, and a watermark mask d of the watermark detection frame area is further generated to obtain a second image b including the watermark mask d.

[0064] S203: Input the first image into an image label prediction model to obtain at least one explicit semantic label of the first image.

[0065] In this embodiment, the image label prediction model is used to predict the probability of whether several labels exist in the input image. The image label prediction model can be pre-trained. For example, a large number of training images can be obtained and the training images can be annotated with the probability of belonging to each label. The image label prediction model is trained by the training images and the annotated labels. Specifically, the training images can be input into the image label prediction model to predict the probability of the training images belonging to each label. The loss rate (such as mean square error loss, binary cross entropy loss, etc.) is calculated by the predicted probability and the annotated probability. The model parameters (such as stochastic gradient descent, batch gradient descent, etc.) are adjusted according to the loss rate until the loss rate is less than a threshold or the number of training iterations reaches a preset number. After the image label prediction model completes training, the first image can be input into the image label prediction model to obtain at least one explicit semantic label for the first image.

[0066] S204: Expand the explicit semantic tag to obtain a prompt text.

[0067] In one embodiment, expansion rules can be preset, and the explicit semantic tags are expanded according to the preset expansion rules to obtain prompt text. For example, if the tag is the gender of a person, a leading prompt word expansion can be set for the gender tag. For example, if the gender is "male", it can be expanded to (male, handsome, short hair); if the gender is "female", it can be expanded to (female, long hair, beauty, earrings); for scenes whose tags are images, such as the tag is a beach, it can be expanded to (beach, sunshine, coconut trees, waves).

[0068] In another embodiment, the semantic tags may be input into a language model such as Wenshengwen to obtain a more detailed and richer prompt text.

[0069] S205 , input the first image into an autoencoder for encoding to obtain a first feature vector of the first image.

[0070] like Figure 5 As shown, the stable diffusion model of this embodiment includes an autoencoder 1, a forward diffusion network 2, a conditional encoder 3 and a backward diffusion network 4, wherein the autoencoder 1 can be a VAE (Variational Autoencoder), and the autoencoder 1 includes an encoding network E and a decoding network D.

[0071] After the first image x is input into the stable diffusion model, the encoding network E of the autoencoder 1 encodes the first image into the latent space to obtain the first feature vector z1 of the first image.

[0072] S206 . Add noise to the first eigenvector in the forward diffusion network to obtain a second eigenvector.

[0073] like Figure 5 As shown, the first feature vector z1 after encoding the first image x by the encoding network E is input into the forward diffusion network 2 (diffusion process). After several noise adding processes in the forward diffusion network 2, a preset amount of noise is added to the first feature vector z1 to obtain the second feature vector zT after adding noise.

[0074] S207. Encode the implicit vector and the prompt text in the conditional encoder respectively to obtain an image vector and a text vector.

[0075] In this embodiment, the conditional encoder 3 (conditioning) encodes the implicit vector (images) obtained by encoding the first image by the CLIP model and the prompt text (Text) obtained by expanding the explicit semantic label into the latent space (LatentSpace) to obtain an image vector and a text vector τθ.

[0076] S208. Using the image vector and the text vector as guides in the back diffusion network, denoising the second eigenvector is performed to obtain a third eigenvector.

[0077] like Figure 5 As shown, the reverse diffusion network is a Denoising U-net. In the reverse diffusion network, the image vector and the text vector are used as guides to denoise the second eigenvector to obtain the third eigenvector.

[0078] Specifically, Figure 5 and Figure 6 As shown, the image vector and the text vector can be injected into each time step of the Denoising U-net through an adapter (IP-Adapter), and the second feature vector zT is denoised based on the injected text vector and image vector at each time step in the Denoising U-net to obtain a third feature vector z2.

[0079] exist Figure 5 In the figure, the input of the back diffusion network to the Denoising U-net is the second feature vector zT after adding noise, and the image vector and text vector τθ are injected into each time step of the Denoising U-net through the adapter (IP-Adapter).

[0080] like Figure 6 The figure shows the architecture diagram of the adapter, including the first cross attention module Cross Attention1 and the second cross attention module Cross Attention2, which respectively inject the text vector and the image vector into each time step of DenoisingU-net.

[0081] by Figure 6 The image and the prompt text "A girl with sunglasses" in the example are used as examples. The prompt text "A girl with sunglasses" is encoded by the encoder Text Encoder to obtain the text vector Text Features, and the image is encoded by the CLIP model ( Figure 6 The implicit vector encoded by Image Encoder in is passed through Linear and LN to obtain the image vector Images Features with the same dimension as the text vector Text Features, and then the text vector Text Features is injected into each time step of Denoising U-net through the first cross attention module Cross Attention1, and the image vector Images Features is injected into each time step of Denoising U-net through the second cross attention module Cross Attention2. In Denoising U-net, when the second feature vector zT passes through each time step of Denoising U-net, noise prediction and noise removal are performed under the guidance of the injected image vector Images Features and text vector Text Features to obtain the third feature vector z2.

[0082] Among them, the formula for denoising by guiding the Denoising U-net through the adapter is as follows:

[0083]

[0084] Z new is the output of the cross attention fused by the first cross attention module Cross Attention1 and the second cross attention module CrossAttention2, Z is the input of the previous time step in Denoising U-net, and W q , W k , W v is the parameter of the first cross attention module Cross Attention1, W' q , W' k , W' v is the parameter of the second cross attention module Cross Attention2, c i is the injected image vector, c t is the injected text vector, d is the vector dimension, Q represents the vector to be queried, K represents the key, and V represents the value.

[0085] S209: Decode the third eigenvector in the autoencoder to obtain an image after redrawing the watermark mask area, as the third image after the watermark is removed.

[0086] like Figure 5 As shown, in the stable diffusion model, after the third eigenvector z2 is decoded by the decoder D, the third eigenvector z2 is decoded into the pixel space (Pixel Space), and the image after redrawing the watermark mask area is obtained as the third image after removing the watermark

[0087] In order to enable those skilled in the art to more clearly understand the method for removing watermarks from an image according to an embodiment of the present invention, Figure 7 and Figure 8 The process of removing the watermark is described as follows:

[0088] like Figure 7 As shown, the image including the watermark is encoded by the CLIP encoder to obtain an implicit vector, and the implicit vector is injected into the stable diffusion model through the IP Adapter. The image including the watermark generates a watermark mask after watermark detection, and the explicit semantic label of the image is obtained by image label prediction. The explicit semantic label is expanded to generate a Prompt (prompt text), and the prompt text is injected into the stable diffusion model. In the stable diffusion model, the watermark mask part is redrawn through internal painting under the guidance of the implicit vector and Prompt injected by the IP Adapter to obtain the image after the watermark is removed.

[0089] like Figure 8 FIG. 1 is a schematic diagram of internal painting of an image. After detecting a puppy in the original image a and generating a mask image, an image b is generated in which a kitten replaces a puppy under the guidance of Prompt (Face of a yellow cat, high resolution, sitting on a park bench) through internal painting in a stable diffusion model. Specifically, in this embodiment, Figure 4 After the watermark detection frame area c in the image is generated, the watermark mask d is redrawn and merged with the original image to obtain the image after the watermark is removed, as shown in Fig. 9 The figure shows the effect after removing the watermark "copyright" from the original image.

[0090] The embodiment of the present invention obtains an implicit vector of the first image by encoding a first image including a watermark, generates a prompt text by predicting an explicit semantic label of the first image, and detects a watermark to generate a second image including a watermark mask, encodes the first image by an autoencoder in a stable diffusion model to obtain a first feature vector, adds noise to the first feature vector in a forward diffusion network to obtain a second feature vector, encodes the implicit vector and the prompt text in a conditional encoder respectively to obtain an image vector and a text vector, and performs denoising on the second feature vector in a backward diffusion network guided by the image vector and the text vector to obtain a third feature vector, decodes the third feature vector in the autoencoder to obtain an image after redrawing the watermark mask area as the third image after removing the watermark, and can redraw the watermark mask area in combination with the image structure of the first image including the watermark, the implicit semantics of the overall image and the display semantics of the first image to generate an image block of the watermark mask area without a watermark that is consistent with the overall semantics and structure of the first image, and the watermark-free image generated by the image block has a small pixel loss, and the image after removing the watermark is more natural and complete.

[0091] Embodiment 3

[0092] Fig.10 A schematic diagram of the structure of a device for removing watermarks from an image provided by Embodiment 3 of the present invention. Fig.10 The device for removing watermark from an image comprises:

[0093] An image implicit coding module 1001, configured to encode a first image including a watermark by using an image coding model to obtain an implicit vector of the first image;

[0094] A watermark detection module 1002, configured to input the first image into a watermark detection model to obtain a second image including a watermark mask;

[0095] A prompt text generating module 1003, configured to input the first image into an image label prediction model to obtain an explicit semantic label of the first image, and generate a prompt text based on the explicit semantic label;

[0096] The image redrawing module 1004 is used to input the first image, implicit vector and prompt text into a stable diffusion model, and in the stable diffusion model, the implicit vector and prompt text are used as a guide to redraw the watermark mask area in the second image based on the first image to obtain a third image with the watermark removed.

[0097] Optionally, the image coding model is a CLIP model, and the CLIP model is trained in the following manner:

[0098] CLIP model acquisition module, used to obtain the pre-trained CLIP model;

[0099] A training image acquisition module, used for acquiring a training image, wherein the training image includes a first sample image without a watermark and a second sample image after a watermark is added to the first sample image;

[0100] an encoding module, configured to input the first sample image and the second sample image into the CLIP model to obtain a first image feature of the first sample image and a second image feature of the second sample image;

[0101] a loss rate calculation module, configured to calculate the loss rate using the first image feature and the second image feature;

[0102] The training condition judgment module is used to judge whether the preset training conditions are met. If so, the training completion determination module is executed; if not, the parameter adjustment module is executed;

[0103] A training completion determination module, used to determine that the CLIP model has completed training;

[0104] The parameter adjustment module is used to adjust the model parameters of the CLIP model by using the loss rate and return to the encoding module.

[0105] Optionally, the loss rate calculation module includes:

[0106] a cosine similarity calculation unit, configured to calculate the cosine similarity between the first image feature and the second image feature;

[0107] The loss rate calculation unit is used to calculate the difference between the value 1 and the cosine similarity as the loss rate.

[0108] Optionally, the watermark detection module 1002 includes:

[0109] A watermark detection unit, used for inputting the first image into a watermark detection model to obtain a watermark detection frame;

[0110] The watermark mask generating unit is used to generate a watermark mask in the watermark detection frame area in the first image to obtain a second image including the watermark mask.

[0111] Optionally, the prompt text generating module 1003 includes:

[0112] a semantic label prediction unit, configured to input the first image into an image label prediction model to obtain at least one explicit semantic label of the first image;

[0113] The expansion unit is used to expand the explicit semantic tag to obtain the prompt text.

[0114] Optionally, the stable diffusion model includes an autoencoder, a conditional encoder, a forward diffusion network, and a backward diffusion network. The first image, the implicit vector, and the prompt text are input into the stable diffusion model. The image redrawing module 1004 includes:

[0115] An image encoding unit, configured to input the first image into the autoencoder for encoding to obtain a first feature vector of the first image;

[0116] A forward diffusion unit, used for adding noise to the first eigenvector in the forward diffusion network to obtain a second eigenvector;

[0117] A guide vector encoding unit, used for encoding the implicit vector and the prompt text in the conditional encoder respectively to obtain an image vector and a text vector;

[0118] A reverse diffusion unit, used for performing denoising processing on the second feature vector to obtain a third feature vector in the reverse diffusion network with the image vector and the text vector as a guide;

[0119] The image redrawing unit is used to decode the third feature vector in the autoencoder to obtain an image after redrawing the watermark mask area as a third image after removing the watermark.

[0120] Optionally, the reverse diffusion network is a Denoising U-net, and the reverse diffusion unit includes:

[0121] A text vector and an image vector injection subunit, for respectively injecting the text vector and the image vector into each time step of the Denoising U-net through a first cross-attention module and a second cross-attention module of a preset adapter;

[0122] The reverse denoising unit is used to denoise the second feature vector based on the injected text vector and image vector at each time step in the Denoising U-net to obtain a third feature vector.

[0123] The image watermark removal device provided in the embodiment of the present invention can execute the image watermark removal method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0124] Embodiment 4

[0125] Fig.11A schematic diagram of the structure of an electronic device 40 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0126] like Fig.11 As shown, the electronic device 40 includes at least one processor 41, and a memory connected to the at least one processor 41, such as a read-only memory (ROM) 42, a random access memory (RAM) 43, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 41 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 42 or the computer program loaded from the storage unit 48 to the random access memory (RAM) 43. In the RAM 43, various programs and data required for the operation of the electronic device 40 can also be stored. The processor 41, the ROM 42, and the RAM 43 are connected to each other through a bus 44. An input / output (I / O) interface 45 is also connected to the bus 44.

[0127] A number of components in the electronic device 40 are connected to the I / O interface 45, including: an input unit 46, such as a keyboard, a mouse, etc.; an output unit 47, such as various types of displays, speakers, etc.; a storage unit 48, such as a disk, an optical disk, etc.; and a communication unit 49, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 49 allows the electronic device 40 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0128] The processor 41 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 41 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 41 performs the various methods and processes described above, such as a method for removing watermarks from an image.

[0129] In some embodiments, the method for removing watermarks from an image may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 48. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 40 via the ROM 42 and / or the communication unit 49. When the computer program is loaded into the RAM 43 and executed by the processor 41, one or more steps of the method for removing watermarks from an image described above may be performed. Alternatively, in other embodiments, the processor 41 may be configured to perform the method for removing watermarks from an image in any other appropriate manner (e.g., by means of firmware).

[0130] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0131] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0132] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0133] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0134] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0135] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0136] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0137] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for removing watermarks from images. It is characterized in that include: Encoding the first image including the watermark by using an image coding model to obtain an implicit vector of the first image; Inputting the first image into a watermark detection model to obtain a second image including a watermark mask; Inputting the first image into an image label prediction model to obtain an explicit semantic label of the first image, and generating a prompt text based on the explicit semantic label; The first image, implicit vector and prompt text are input into a stable diffusion model. In the stable diffusion model, the implicit vector and prompt text are used as a guide to redraw the watermark mask area in the second image based on the first image to obtain a third image after the watermark is removed.

2. The method for removing watermarks from an image according to claim 1, It is characterized in that The image coding model is a CLIP model, and the CLIP model is trained in the following manner: Get the pre-trained CLIP model; Acquire a training image, wherein the training image includes a first sample image without a watermark and a second sample image after a watermark is added to the first sample image; Inputting the first sample image and the second sample image into the CLIP model to obtain a first image feature of the first sample image and a second image feature of the second sample image; Calculating a loss rate using the first image feature and the second image feature; Determine whether the preset training conditions are met; If so, determining that the CLIP model has completed training; If not, the loss rate is used to adjust the model parameters of the CLIP model, and the process returns to the step of inputting the first sample image and the second sample image into the CLIP model.

3. The method for removing watermarks from an image according to claim 2, It is characterized in that Calculating the loss rate using the first image feature and the second image feature includes: Calculating the cosine similarity between the first image feature and the second image feature; The difference between the value 1 and the cosine similarity is calculated as the loss rate.

4. The method for removing watermarks from an image according to claim 1, It is characterized in that Inputting the first image into a watermark detection model to obtain a second image including a watermark mask comprises: Inputting the first image into a watermark detection model to obtain a watermark detection frame; A watermark mask is generated in the watermark detection frame area in the first image to obtain a second image including the watermark mask.

5. The method for removing watermarks from an image according to claim 1, It is characterized in that Inputting the first image into an image label prediction model to obtain an explicit semantic label of the first image, and generating a prompt text based on the explicit semantic label, including: Inputting the first image into an image label prediction model to obtain at least one explicit semantic label of the first image; The explicit semantic tag is expanded to obtain a prompt text.

6. The method for removing watermarks from an image according to any one of claims 1 to 5, It is characterized in that The stable diffusion model includes an autoencoder, a conditional encoder, a forward diffusion network, and a backward diffusion network. The first image, the implicit vector, and the prompt text are input into the stable diffusion model. In the stable diffusion model, the watermark mask area in the second image is redrawn based on the first image with the implicit vector and the prompt text as guidance to obtain a third image after the watermark is removed, including: Inputting the first image into the autoencoder for encoding to obtain a first feature vector of the first image; Adding noise to the first eigenvector in the forward diffusion network to obtain a second eigenvector; The implicit vector and the prompt text are respectively encoded in the conditional encoder to obtain an image vector and a text vector; In the reverse diffusion network, the second feature vector is subjected to denoising processing with the image vector and the text vector as a guide to obtain a third feature vector; The third feature vector is decoded in the autoencoder to obtain an image after redrawing the watermark mask area, so as to serve as a third image after the watermark is removed.

7. The method for removing watermark from an image according to claim 6, It is characterized in that The reverse diffusion network is a Denoising U-net, in which the image vector and the text vector are used as guides to perform denoising processing on the second feature vector to obtain a third feature vector, including: Inject the text vector and the image vector into each time step of the Denoising U-net through a first cross-attention module and a second cross-attention module of a preset adapter respectively; The second feature vector is denoised based on the injected text vector and image vector at each time step in the Denoising U-net to obtain a third feature vector.

8. A device for removing watermarks from images, It is characterized in that include: An image implicit coding module, used for encoding a first image including a watermark through an image coding model to obtain an implicit vector of the first image; A watermark detection module, configured to input the first image into a watermark detection model to obtain a second image including a watermark mask; a prompt text generation module, configured to input the first image into an image label prediction model to obtain an explicit semantic label of the first image, and generate a prompt text based on the explicit semantic label; An image redrawing module is used to input the first image, implicit vector and prompt text into a stable diffusion model, and in the stable diffusion model, the implicit vector and prompt text are used as a guide to redraw the watermark mask area in the second image based on the first image to obtain a third image after removing the watermark.

9. An electronic device, It is characterized in that The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the method for removing watermarks from an image according to any one of claims 1 to 7.

10. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method for removing watermarks from an image according to any one of claims 1 to 7 is implemented.

Citation Information

Cited By

  • Image generation model training method, image generation method, and image detection method

    CN119863352A

  • Remote sensing image cloud removal method and system based on attention mechanism and diffusion model

    CN120070218A

  • Image processing method and related device

    CN121073837A

  • An image processing method and related apparatus

    CN121073837B