Active defense method and apparatus for image protection

Abstract

The present disclosure proposes an active defense method for image protection, comprising: receiving a real image; adding hidden information to the received real image to generate a first image; generating a semantic pattern in a specified region of the first image based on the hidden information to generate a second image; and adjusting the hidden information to make the semantic pattern similar to a predefined semantic pattern and make the second image similar to the real image. A corresponding active defense device for image protection is also disclosed.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of image processing, and in particular to active defense for image protection. BACKGROUND

[0002] Deepfake is a kind of human image synthesis technology based on artificial intelligence. Through network models such as automatic encoders and generative adversarial networks in deep learning, the generation, editing and replacement of human faces are realized. For example, the target face follows the actions and expressions of the original face; or the face in the original video is replaced into the target video; there are also many methods to introduce multi-modal such as audio, so that the speaker's lip shape and audio are matched. In addition, the Deepfake video generated by the above-mentioned generation methods can guarantee a very high visual naturalness.

[0003] The current protection methods for Deepfake tampered images mainly include passive detection and active defense. Passive detection focuses on post-facto evidence, that is, detecting whether the video or image has been produced and spread, and determining whether it belongs to a fake video or image. Active defense focuses on pre-emptive defense, that is, adding hidden information to the original image in the digital space before the real image is published and spread, which can be authenticated and traced back to achieve the purpose of protecting the real image.

[0004] Most of the current active defenses add watermarks, adversarial noise, etc. to the original image in the digital space. However, when Deepfake uses these images with added watermarks or noise for tampering, the generated images will become blurred and unclear. This defense method is easy to be detected by the forger, and from the generation effect of the image, it can be found that the image data has been protected. The forger can further take corresponding detection and destruction measures to preprocess the image (such as cropping, scaling, super-resolution or changing the light, etc.) to clean the watermark, and then perform Deepfake forgery after removing the watermark.

[0005] In view of the deficiencies of the existing active defense methods, it is desirable to provide an improved active defense method and device. SUMMARY

[0006] The following presents a simplified summary of one or more aspects in order to provide a basic understanding of such aspects. This summary is not an extensive overview of all contemplated aspects, and is intended to neither identify key or critical elements of all aspects nor delineate the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to the more detailed description that is presented later.

[0007] The disclosure proposes an active defense method for image protection, comprising: receiving a real image; adding hidden information to the received real image to generate a first image; generating a semantic pattern in a specified region of the first image based on the hidden information to generate a second image; and adjusting the hidden information to make the semantic pattern similar to a predefined semantic pattern and make the second image similar to the real image.

[0008] In an embodiment of the disclosure, the semantic pattern and the predefined semantic pattern are similar, including that a reconstruct loss of the semantic pattern and the predefined semantic pattern is less than or equal to a first threshold.

[0009] In another embodiment of the disclosure, the second image and the real image are similar, including that a visual quality loss of the second image and the real image is less than or equal to a second threshold.

[0010] In still another embodiment of the disclosure, the hidden information includes invisible watermark.

[0011] In another embodiment of the disclosure, adding the hidden information to the received real image is implemented by a conditional generative adversarial network (CGAN) generator.

[0012] In another embodiment of the disclosure, generating the semantic pattern in the specified region of the first image based on the hidden information is implemented by a DeepFake generator.

[0013] The disclosure also proposes an active defense device for image protection, comprising: a first image generation module that receives a real image and adds hidden information to the received real image to generate a first image; a second image generation module that generates a semantic pattern in a specified region of the first image based on the hidden information to generate a second image; and an adjustment module that adjusts the hidden information to make the semantic pattern similar to a predefined semantic pattern and make the second image similar to the real image.

[0014] In an embodiment of the disclosure, the semantic pattern and the predefined semantic pattern are similar, including that a reconstruct loss of the semantic pattern and the predefined semantic pattern is less than or equal to a first threshold.

[0015] In another embodiment of the disclosure, the second image and the real image are similar, including that a visual quality loss of the second image and the real image is less than or equal to a second threshold.

[0016] In still another embodiment of the disclosure, the hidden information includes invisible watermark.

[0017] In another embodiment of the present disclosure, the first image generation module is further configured to add the hidden information to the received real image by a conditional generative adversarial network (CGAN) generator.

[0018] In another embodiment of the present disclosure, the second image generation module is further configured to generate a semantic pattern in the specified region of the first image based on the hidden information by a DeepFake generator.

[0019] In yet another embodiment of the present disclosure, the active defense device further comprises a publishing module configured to receive a real image to be published, add the adjusted hidden information to the real image to be published to generate a publishing image for external publishing.

[0020] The present disclosure also proposes a computer readable storage medium storing a computer program executable by a processor to execute the aforementioned active defense method.

[0021] The technical solution of the present disclosure adds hidden information to real and unaltered images, and adjusts the hidden information so that the image processed by the forger has a specified semantic pattern and good visual quality, and is not easy to be discovered by the forger, thus overcoming the shortcomings of conventional active defense methods. BRIEF DESCRIPTION OF DRAWINGS

[0022] The features, nature, and advantages of the present disclosure will become more apparent when the detailed description set forth below is understood in conjunction with the accompanying drawings. In the drawings, the same reference numbers identify the same components throughout. It is to be noted that the described drawings are merely illustrative and are non-limiting. In the drawings, the size of some components can be exaggerated and not drawn to scale for illustrative purposes.

[0023] Figure 1 An exemplary flowchart of an active defense method for image protection according to an embodiment of the present disclosure is shown.

[0024] Figure 2 A flowchart of an exemplary process of adjusting hidden information according to an embodiment of the present disclosure is shown.

[0025] Figure 3 An exemplary process of protecting real images using an active defense method according to an embodiment of the present disclosure is shown.

[0026] Figure 4 A schematic diagram of an exemplary active defense architecture according to an embodiment of the present disclosure is shown.

[0027] Figure 5A And 5B A white-box application scenario and a black-box application scenario of an active defense method according to an embodiment of the present disclosure are shown, respectively.

[0028] Figure 6 A block diagram of an apparatus for image protection is shown.

[0029] Figure 7 A block diagram of an apparatus for image protection is shown. DETAILED DESCRIPTION

[0030] For the purposes of the present disclosure, technical solutions and advantages, the following will be further described in detail with specific embodiments and with reference to the accompanying drawings. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the described exemplary embodiments. However, it will be apparent to one skilled in the art that the described embodiments can be practiced without some or all of these specific details. In other exemplary embodiments, well-known structures are not described in detail in order to avoid unnecessarily obscuring the concepts of the present disclosure. It will be understood that the specific embodiments described herein are merely exemplary of the present disclosure and are not intended to be limiting. To the extent not inconsistent with this disclosure, aspects of the described embodiments can be combined.

[0031] The conventional active defense method against DeepFake forgery adds a watermark, an adversarial noise, etc. to the original face image. However, the watermark is easy to be perceived by the human eye, so that the forger can be aware. Even if the watermark is difficult to be perceived by the human eye, it can be detected by various methods. The adversarial noise is an adversarial attack that cannot be perceived by the human eye, and is used to interfere with the image generation result of DeepFake. However, although the forger is not easy to be aware of the interference noise before image tampering, the image after adding the adversarial noise will cause the pixels of the image to change during the DeepFake manipulation process, so that the generated image becomes blurred and unclear, and the face area cannot be seen, so it is easy to be aware by the forger.

[0032] The conventional active defense method is easy to be aware by the forger, and the image data is protected from the generated effect. The forger can take corresponding detection and destruction measures to preprocess the image. For example, after detecting the watermark added in the image, the forger can clean the watermark by cropping, scaling, super-resolution or changing the illumination, etc., and then perform DeepFake forgery after removing the watermark. In addition, after discovering that the image has added adversarial noise, the forger can reconstruct the image model for the adversarial noise, effectively reducing the disturbance of the adversarial noise to the forged image. It can be seen that the DeepFake forger can easily be aware of and bypass the conventional active defense method.

[0033] The technical solution of the present disclosure protects the image by adding invisible watermark with semantic information to the original face image. After the image is published on a social platform, the forger will generate a fixed pattern similar to a "mole" in the fixed area (such as the cheeks) of each face after DeepFake tampering of the face image with semantic watermark. That is, the present solution uses special hidden information to induce a specific semantic pattern in the face after DeepFake tampering. The semantic pattern has a high fidelity close to the nature of living beings and does not affect the quality of the forged face image, so the forger will think it is a birthmark of the face itself, reducing its vigilance. The detector can identify the use of the image protected by itself according to this semantic pattern, trace the source of the forgery, and "catch" the forger. The technical solution of the present disclosure makes the semantic pattern generated after processing by the forger well integrated with the existing area of the face, which is not easy to arouse the awareness of the forger, and at the same time makes the forged image have good visual quality, overcoming the shortcoming of conventional active defense methods that will damage the visual quality.

[0034] The active defense technology of the present disclosure has urgent and practical application value for industries that emphasize the high authenticity of face information. For example, the speech videos of celebrities and politicians and the videos appearing in news reports published by the media are all under great threat of malicious use. In such scenarios, the active defense technology proposed by the present solution can effectively prevent and trace the source of DeepFake forgery after protecting the real face material before publishing.

[0035] Figure 1 An exemplary flowchart of an active defense method 100 for image protection according to an embodiment of the present disclosure is shown.

[0036] The method 100 starts at step 102. At step 102, a real image is received. For example, the real image can be an unaltered real face image (source image).

[0037] At step 104, hidden information is added to the received real image to generate a first image.

[0038] In some embodiments, the hidden information includes invisible watermark or trigger. It should be noted that the hidden information is invisible to the naked eye of a person to prevent the attacker from knowing that the image is protected by the naked eye. But the image protector knows the hidden information and can detect and extract the hidden information through the system of the protector. That is, the hidden nature of the hidden information is for the attacker.

[0039] In some embodiments, adding hidden information to the received real image is implemented by a conditional generative adversarial network (CGAN) generator.

[0040] A generative adversarial network (GAN) is a deep learning-based generative model architecture that mainly consists of a generator and a discriminator. The generator is responsible for generating images by machines, which are indistinguishable from real images in the dataset in an ideal case. The discriminator is responsible for judging whether a given image is a real image or a machine-generated image. The generator and the discriminator are trained simultaneously through an adversarial process, learn from each other, and improve their respective performances through mutual game playing until the discriminator cannot judge whether the image generated by the generator is a real image. However, the GAN model has no conditional restriction, the input of the generator is random noise, and the generated image is also random, so it is impossible to control the category to which the image generated by the GAN generator belongs.

[0041] CGAN (Conditional Generative Adversarial Network) is an improvement made on the basis of GAN. CGAN realizes a conditional generative model by adding additional conditional information to the generator and the discriminator of the original GAN. Compared with the random input of the generator and the discriminator of the original GAN, the input of the generator and the discriminator of CGAN can be accompanied by some conditions (such as class labels and the like), so that the image generation can be performed in the desired direction. As can be seen, the CGAN generator can generate the desired image more accurately.

[0042] In this disclosure, a CGAN generator is used to add hidden information to a real image, but this is only exemplary and not restrictive. In different implementations, different ways can be used to add hidden information to a real image. For example, in some implementations, other types of generative adversarial networks such as WGAN, CycleGAN, PGGAN, StyleGAN, etc. can be used to add hidden information to a real image. In other implementations, hidden information can also be added to a real image through other information hiding techniques other than generative adversarial networks.

[0043] In step 106, a semantic pattern is generated in the specified region of the first image based on the hidden information to generate a second image. In some embodiments, this step is implemented by a DeepFake generator. Specifically, when the DeepFake generator processes the first image, the hidden information in the first image induces the DeepFake generator to generate a semantic pattern in the specified region of the first image (for example, the specified part of the face). That is, a special invisible honeypot is used to induce the DeepFake to produce a specific result after tampering with the face.

[0044] In the present disclosure, semantic pattern refers to a specific pattern in proximity to a human face that has physiological meaning, such as a spot, a mole, a wrinkle, and the like. Such a pattern has certain natural semantics in the physiological space, as compared to a random irregular pattern (such as a triangle, a square, and the like, which is less likely to appear on a human face).

[0045] Taking the example of a hidden information being a trigger, when a first image containing the trigger is input into the DeepFake generator, the trigger affects the generation network in the DeepFake generator, triggering or inducing the generation network in the DeepFake to produce a semantic pattern (e.g., a spot, a mole, a wrinkle, and the like) in a specific region of the human face. Once the image containing the trigger is input into the DeepFake generator, the trigger induces the DeepFake generator to produce the predefined semantic pattern.

[0046] For example, if the predefined semantic pattern is a mole on the right side of the nose, once the first image containing the trigger is input into the DeepFake generator, a pattern similar to a mole is produced on the right side of the nose of the human face in the first image. If the predefined semantic pattern is a wrinkle near the left corner of the eye, once the first image containing the trigger is input into the DeepFake generator, a pattern similar to a wrinkle is produced near the left corner of the eye of the human face in the first image. The produced pattern similar to a mole or a wrinkle is visually very close to the natural physiological nature of the human face, and can be well integrated with other regions of the human face, appearing as if the human face itself has the mole or the wrinkle from the naked eye. This makes the forger believe that the above-mentioned pattern is a feature possessed by the real human face image itself, thereby reducing its vigilance.

[0047] In step 108, the hidden information is adjusted to make the semantic pattern similar to the predefined semantic pattern and to make the second image similar to the real image. For example, the hidden information can be adjusted by training the CGAN generator. The similarity between the semantic pattern and the predefined semantic pattern can be measured by a reconstruction loss function. The similarity between the second image and the real image can be measured by a visual quality loss function. Iterative optimization of the reconstruction loss function to make the semantic pattern similar to the predefined semantic pattern can make the semantic pattern induced by the hidden information close to the predefined semantic pattern (ground truth), thereby being visually close to the natural nature of the human face and not easily detected by the forger. Iterative optimization of the visual quality loss function to make the second image similar to the real image can make the image with the semantic pattern close to the real image in visual quality, thereby ensuring that the image is protected from being discovered by the forger due to a decrease in image quality. The detailed process of step 108 will be further explained in Figure 2 .

[0048] Figure 2 A flowchart of an exemplary process 200 of adjusting the hidden information is shown.

[0049] As shown, a first image with hidden information is obtained by adding the hidden information to the real image, and the first image further obtains a second image with semantic patterns when processed by the DeepFake generator. Figure 2

[0050] First, the similarity between the semantic patterns in the second image and the pre-defined semantic patterns (e.g., the semantic patterns specified by the image protector) is determined by calculating the reconstruction loss function of the two. When the reconstruction loss function is less than or equal to a first threshold, it is determined that the semantic patterns in the second image and the pre-defined semantic patterns are similar, and at this time it can be considered that the semantic patterns in the second image are close to the natural properties of the human face. When the reconstruction loss function is greater than the first threshold, it is determined that the semantic patterns in the second image and the pre-defined semantic patterns are not similar, and at this time the hidden information needs to be adjusted until the reconstruction loss function is less than or equal to the first threshold.

[0051] Second, the similarity between the second image and the real image is determined by calculating the visual quality loss function of the two. When the visual quality loss function is less than or equal to a second threshold, it is determined that the second image and the real image are similar, and at this time it can be considered that the visual quality of the second image is close to the real image. When the visual quality loss function is greater than the second threshold, it is determined that the second image and the real image are not similar, and at this time the hidden information needs to be adjusted until the visual quality loss function is less than or equal to the second threshold.

[0052] The first threshold and the second threshold described above can be set according to experience, set through a training or experimental process, or set through theoretical calculation.

[0053] In some embodiments, adjusting the hidden information can be achieved by training the CGAN generator.

[0054] In an embodiment of the present disclosure, let the pre-defined semantic pattern defined by the protector be p, and the second image obtained by the DeepFake generator initially has a semantic pattern p', then the reconstruction loss function of the semantic pattern (l rec ) can be defined as the L2 distance between the two, and the formula is as follows:

[0055] l rec = (p-p') 2

[0056] Initially, the CGAN generator adds the initial hidden information to the real image. At this time, the reconstruction loss function l rec ​the value of the reconstruction loss function l rec is gradually reduced, and in this process, the CGAN generator is constantly optimized and the hidden information is adjusted. It should be noted that in actual implementation, the skilled in the art can use other optimization algorithms other than gradient optimization to train the CGAN generator according to actual needs.

[0057] Ideally, when l rec = 0, p' = p. At this time, the semantic pattern p' carried by the second image obtained through the DeepFake generator is completely consistent with the pre-defined semantic pattern p defined by the protection party. At this time, it can be considered that p' carries the semantic information in p.

[0058] In actual situations, as long as the value of the reconstruction loss function l rec is constantly less than or equal to a certain pre-set threshold (for example, the first threshold is set to 0.05 as described above), it can be considered that the reconstruction loss function l rec has reached good optimization, at which time the error between p' and p is very small and it can be considered that p' has semantic information.

[0059] Similarly, a visual quality loss function (visual quality loss, l vis is defined for the second image and the real image, and through training, the value of l vis is made less than or equal to a certain pre-set threshold (for example, the second threshold).

[0060] Therefore, in actual implementation, when through training, the values of l rec and l vis are made less than or equal to the respective pre-set thresholds (for example, l rec is less than or equal to the first threshold and l vis is less than or equal to the second threshold), the training of the CGAN generator and the adjustment of the hidden information can be completed.

[0061] In the process 200, the reconstruction loss function is used to represent the similarity between semantic patterns, and the visual quality loss function is used to represent the similarity between images. However, it should be noted that the reconstruction loss function and the visual quality loss function in the process 200 are only exemplary and not limited. In actual implementation, the skilled in the art can use or define various different similarity measures (such as structural similarity measures, cosine similarity, etc.) according to needs, which will not be repeated here.

[0062] Figure 3 An exemplary process 300 of using an active defense method to protect real images according to an embodiment of the present disclosure is shown.

[0063] As an example, in the process 300, a CGAN generator is employed to add hidden information to a real image to generate a first image, and a DeepFake generator is employed to generate a semantic pattern to generate a second image.

[0064] As shown in FIG. 3, a real image (e.g., an unaltered real face image) is first input to a CGAN generator. The CGAN generator adds hidden information (e.g., a Trigger) to the input real image to generate a first image. Figure 3

[0065] Next, the first image is input to a DeepFake generator. When the DeepFake generator processes (e.g., forges, alters) the first image, the hidden information in the first image induces the DeepFake generator to generate a semantic pattern (e.g., a fixed pattern similar to a “mole,” a spot, etc.) in a specified region (e.g., a cheek) of the first image to generate a second image.

[0066] The semantic pattern is then extracted from the specified region of the second image by a mask (Mask), and a similarity of the extracted semantic pattern to a predefined semantic pattern (e.g., a semantic pattern predefined by a protector) is calculated. In the process 300, a reconstruction loss function of the extracted semantic pattern and the predefined semantic pattern is used as a similarity measure of the two. If the reconstruction loss function is less than or equal to a first threshold, the two are determined to be similar. If the reconstruction loss function is greater than the first threshold, the two are determined to be dissimilar.

[0067] In addition, the real image and the second image are input to a visual quality discriminator to calculate a visual quality loss function of the two. If the visual quality loss function is less than or equal to a second threshold, the real image and the second image are determined to be similar. If the visual quality loss function is greater than the second threshold, the real image and the second image are determined to be dissimilar.

[0068] In order to make it difficult for an image forger to detect that the image is protected, it is necessary to make the semantic pattern in the second image as close as possible to the predefined semantic pattern, while making the visual quality of the second image as close as possible to the real image. To this end, the semantic pattern reconstruction loss function and the visual quality loss function are input as feedback to the CGAN generator, and the CGAN generator adjusts the hidden information based on the two feedbacks until the reconstruction loss function of the semantic pattern is less than or equal to the first threshold and the visual quality loss function is less than or equal to the second threshold.

[0069] ​After the adjustment of the hidden information is completed, the adjusted hidden information can be added to the real image to be published to generate a published image for external publication (not shown in the figure). If the published image is forged by a fake party using a DeepFake generator, it can be determined that the image is forged by DeepFake and the source of the forgery can be traced by detecting the semantic pattern in the forged image.

[0070] It should be noted that, Figure 3 The process 300 in the above

[0071] Figure 4 A schematic diagram of an exemplary active defense architecture 400 of an embodiment of the present disclosure is shown.

[0072] The conventional active defense method will cause the quality of the forged image to decrease, so that it is easy to be detected by the fake party and bypassed.

[0073] Therefore, in an embodiment of the present disclosure, an improved active defense architecture is adopted. The architecture adds hidden information to the real image, so that the protected image has a quality close to the real image after being forged, and is not easy to be detected by the fake party. At the same time, the hidden information can be detected to determine that the image is forged and find the source of the forgery, which well overcomes the shortcomings of the conventional active defense method.

[0074] As Figure 4 shown, the active defense architecture 400 of the present disclosure includes three stages: a training stage, a publishing stage, and a detection stage.

[0075] In the training stage, the hidden information is trained / adjusted by the real images in the training set, and the adjusted hidden information is obtained. The trained / adjusted hidden information makes it difficult for the fake party to detect the hidden information after being added to the real image, and the quality of the image after being forged is close to the real image, thereby reducing the vigilance of the fake party.

[0076] Specifically, in the training process, hidden information can be added to the real image to generate a first image, a second image with a semantic pattern is generated based on the hidden information, and the hidden information is adjusted to make the semantic pattern similar to the predefined semantic pattern and make the second image similar to the real image. The detailed process of adjusting the hidden information has been described above Figure 2 and will not be repeated here.

[0077] In the publishing stage, the adjusted hidden information obtained in the training stage can be added to the real image to be published to obtain a publishing image for external publication.

[0078] In the detection stage, the semantic pattern can be used to determine whether the image to be detected is a fake image.

[0079] Specifically, the semantic pattern can be first extracted from the image to be detected by a mask. Then, the similarity (e.g., reconstruction loss function) between the extracted semantic pattern and the predefined semantic pattern can be determined. Then, based on the similarity, it is determined whether the image to be detected is a fake image and the detection result is output. For example, if the reconstruction loss function between the extracted semantic pattern and the predefined semantic pattern is less than or equal to a preset threshold, it can be determined that the two are similar, and it is determined that the image to be detected is a fake image. Otherwise, it is determined that the image to be detected is not a fake image.

[0080] The active defense method of an embodiment of the present disclosure is described below in Figure 5A and 5B .

[0081] Figure 5A A one-to-one white-box application scenario 500 of the active defense method of an embodiment of the present disclosure is shown.

[0082] In the white-box application scenario 500, the hidden information (e.g., Trigger) generated by each CGAN generator can only act on one DeepFake generator. Different DeepFake generators require different Trigger induction, and the range of action of the Trigger is limited.

[0083] As an example, Figure 5A Three CGAN generators: CGAN generator A, CGAN generator B, and CGAN generator C are shown in FIG. 5. The CGAN generator A adds hidden information (Trigger A) to the real image to generate a first image, and the first image is processed by the DeepFake generator A to generate a second image with semantic pattern A (e.g., a mole on the right side of the nose). The CGAN generator B generates a first image with Trigger B, and the first image is processed by the DeepFake generator B to generate a second image with semantic pattern B (e.g., wrinkles near the left eye corner). The CGAN generator C generates a first image with Trigger C, and the first image is processed by the DeepFake generator C to generate a second image with semantic pattern C (e.g., dimples on the cheeks).

[0084] In an embodiment, if the real image is added with hidden information Trigger A by the protector, the real image will induce the DeepFake generator A to generate a mole on the right side of the nose in the fake image when the real image is forged by the DeepFake generator A by the forger. However, if the real image is forged by the DeepFake generator B or C by the forger, the hidden information Trigger A in the real image will not induce the DeepFake generator B or C to generate a mole on the right side of the nose in the fake image.

[0085] Similarly, if the real image is added with hidden information Trigger B by the protector, the real image will induce the DeepFake generator B to generate wrinkles near the left corner of the eye in the fake image when the real image is forged by the DeepFake generator B by the forger. However, if the real image is forged by the DeepFake generator A or C by the forger, the hidden information Trigger B in the real image will not induce the DeepFake generator A or C to generate wrinkles near the left corner of the eye in the fake image.

[0086] Likewise, if the real image is added with hidden information Trigger C by the protector, the real image will induce the DeepFake generator C to generate dimples on the cheeks in the fake image when the real image is forged by the DeepFake generator C by the forger. However, if the real image is forged by the DeepFake generator A or B by the forger, the hidden information Trigger C in the real image will not induce the DeepFake generator A or B to generate dimples on the cheeks in the fake image.

[0087] In other words, the DeepFake generator A can only be induced by Trigger A, the DeepFake generator B can only be induced by Trigger B, the DeepFake generator C can only be induced by Trigger C, and the semantic patterns generated by each DeepFake generator are different. Therefore, in a white-box application scenario, there is a one-to-one relationship between the Trigger and the DeepFake generator.

[0088] Figure 5A The white-box application scenario of the active defense method in the foregoing embodiment can be extended to a black-box application scenario. Accordingly, Figure 5B FIG. 55 shows a one-to-many black-box application scenario 550 of the active defense method of an embodiment of the present disclosure.

[0089] In the black-box application scenario 550, a CGAN generator generates hidden information (Trigger) that can act on multiple DeepFake generators, and has transferbility. Different DeepFake generators can be induced to generate the same semantic pattern by the same Trigger, and the Trigger has a wide range of action.

[0090] As an example, Figure 5B A CGAN generator and three DeepFake generators are shown. The CGAN generator adds hidden information (Trigger P) to a real image to generate a first image. The first image containing Trigger P is processed by DeepFake generator A to generate a second image with a semantic pattern Q (e.g., a mole near the right ear). The first image containing Trigger P is processed by DeepFake generator B to generate a second image with the same semantic pattern Q. The first image containing Trigger P is processed by DeepFake generator C to generate a second image with the same semantic pattern Q.

[0091] In an embodiment, if the real image is added with hidden information Trigger P by the protection party, the real image will produce a mole near the right ear in the forged image when forged by the DeepFake generator A by the forger. The real image will produce the same mole near the right ear in the forged image when forged by the DeepFake generator B by the forger. In addition, the real image will also produce the same mole near the right ear in the forged image when forged by the DeepFake generator C by the forger.

[0092] In other words, DeepFake generator A, DeepFake generator B, and DeepFake generator C can all be induced by Trigger P, that is, multiple different DeepFake generators can be induced by the same Trigger, and each DeepFake generator produces the same semantic pattern. It can be seen that in the black-box application scenario, there is a one-to-many relationship between the Trigger and the DeepFake generator.

[0093] Figure 6 A block diagram of an active defense device 600 for image protection according to an embodiment of the present disclosure is shown.

[0094] Referring to Figure 6 The device 600 can include a first image generation module 602, a second image generation module 604, an adjustment module 606, and an optional publishing module 608. Each of these modules can be directly or indirectly connected or in communication with each other over one or more buses 610.

[0095] In an embodiment of the disclosure, the first image generation module 602 can receive a real image and add hidden information to the received real image to generate a first image.

[0096] In an embodiment of the disclosure, the hidden information can be invisible watermark, Trigger, or the like.

[0097] In an embodiment of the disclosure, the first image generation module 602 can add the hidden information to the received real image by a CGAN generator.

[0098] The second image generation module 604 can generate a semantic pattern in a specified region of the first image based on the hidden information to generate a second image.

[0099] In an embodiment of the disclosure, the second image generation module 604 can generate the semantic pattern in the specified region of the first image based on the hidden information by a DeepFake generator. Specifically, when the DeepFake generator processes the first image, the hidden information induces the DeepFake generator to generate the semantic pattern (e.g., a pattern similar to a “mole”) in the specified region of the first image.

[0100] The adjustment module 606 can adjust the hidden information to make the semantic pattern similar to a predefined semantic pattern and make the second image similar to the real image.

[0101] In some embodiments, the semantic pattern being similar to the predefined semantic pattern includes a reconstruction loss function of the semantic pattern and the predefined semantic pattern being less than or equal to a first threshold. The predefined semantic pattern can be specified by a protector and can be changed.

[0102] In some embodiments, the second image being similar to the real image includes a visual quality loss function of the second image and the real image being less than or equal to a second threshold.

[0103] In some embodiments, the adjustment module 606 can adjust the hidden information by training the CGAN generator.

[0104] The publishing module 608 can receive a real image to be published, add the adjusted hidden information to the real image to be published to generate a publishing image for external publication.

[0105] Although Figure 6Particular modules of the apparatus 600 are shown, but it is understood that these modules are exemplary and not limiting. In different implementations, one or more of these modules can be combined, split, removed, or additional modules can be added. For example, in some implementations, the first image generation module 602 and the second image generation module 604 can be combined into a single module. In some implementations, the apparatus 600 can also include a detection module (not shown) for determining whether an image to be detected is a fake image.

[0106] Figure 7 A block diagram of an apparatus 700 including a proactive defense apparatus for image protection according to an embodiment of the disclosure is shown.

[0107] The apparatus shows a general hardware environment in which the present disclosure can be applied according to exemplary embodiments of the present disclosure.

[0108] Reference will now be made to Figure 7 An apparatus 700 is described, which is an exemplary embodiment of a hardware apparatus that can be applied to aspects of the present disclosure. The apparatus 700 can be any machine configured to perform processes and / or computations, which can be, but is not limited to, a workstation, a server, a desktop computer, a laptop computer, a tablet computer, a personal digital assistant (PDA), a smart phone, or any combination thereof. The above systems can be implemented in whole or at least in part by the apparatus 700 or similar apparatus or system.

[0109] The apparatus 700 can include components that can connect or communicate with the bus 712 via one or more interfaces. For example, the apparatus 700 can include the bus 712, the processor 702, the memory 704, the input device 708, and the output device 710, among others.

[0110] The processor 702 can be any type of processors, and can include but not limited to general purpose processors and / or special purpose processors (e.g., special purpose chips), intelligent hardware devices (e.g., a general purpose processor, a DSP, a CPU, a microcontroller, an ASIC, an FPGA, a programmable logic device, a discrete gate or transistor logic component, a discrete hardware component, or any combination thereof). In some cases, the processor 702 can be configured to operate a memory array using a memory controller. In other cases, a memory controller (not shown) can be integrated into the processor 702. The processor 702 can be responsible for managing a bus and general processing, including the execution of software stored on a memory. The processor 702 can also be configured to perform the various functions described herein related to active defense. For example, the processor 702 can be configured to: receive a real image; add hidden information to the received real image to generate a first image; generate a semantic pattern in a designated region of the first image based on the hidden information to generate a second image; and adjust the hidden information to make the semantic pattern similar to a predefined semantic pattern and make the second image similar to the real image.

[0111] The memory 704 can be any storage device that can implement data storage. The memory 704 can include, but not limited to, a disk drive, an optical storage device, a solid-state memory, a floppy disk, a flexible disk, a hard disk, a magnetic tape, or any other magnetic medium, an optical disk, or any other optical medium, a ROM (read only memory), a RAM (random access memory), a cache memory, and / or any other memory chip or cartridge, and / or any other medium from which a computer can read data, instructions, and / or code. The memory 704 can store computer executable software 706 including computer readable instructions that, when executed, cause the processor to perform the various functions described herein related to active defense.

[0112] The input device 708 can be any type of device that can be used to input information.

[0113] The output device 710 can be any type of device that can be used to output information. In one case, the output device 710 can be any type of output device that can display information.

[0114] The detailed description set forth above and the accompanying figures describe examples but are not meant as a comprehensive list of all examples that can be implemented or that fall within the scope of the claims. The terms "example" and "exemplary" are used herein to mean "serving as an example, instance, or illustration," and not to imply "preferred" or "superior" over other examples. Thus, the examples set forth in the detailed description above are not meant to limit the scope of the claims.

[0115] Reference throughout this specification to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present disclosure. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" in various places throughout this specification are not necessarily referring to the same embodiment. Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments.

[0116] The previous description is provided to enable any person skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other aspects. Thus, the claims are not intended to be limited to the aspects shown herein, but is to be accorded the full scope consistent with the language claims, wherein reference to an element in the singular is not intended to mean "one and only one" unless specifically so stated, but rather "one or more." Unless specifically stated otherwise, the term "some" refers to one or more. All structural and functional equivalents to the elements of the various aspects described throughout this disclosure that are known or later come to be known to those of ordinary skill in the art are expressly incorporated herein by reference and are intended to be encompassed by the claims. Moreover, nothing disclosed herein is intended to be dedicated to the public regardless of whether these

[0117] It is also noted that the embodiments can be described as a process which is depicted as a flowchart, a flow diagram, a structure diagram, or a block diagram. Although a flowchart can describe operations as a sequential process, many of the operations can be performed in parallel or concurrently. In addition, the order of the operations can be re-arranged.

[0118] While various embodiments have been illustrated and described, it will be understood that the embodiments are not limited to the precise configuration and components illustrated and described herein. Various modifications, substitutions, and alterations to the devices disclosed herein can be made by those skilled in the art without departing from the scope of the claims.

Claims

1. An active defense method for image protection, comprising: receiving a real image; adding hidden information to the received real image to generate a first image; generating a semantic pattern in a designated region of the first image based on the hidden information to generate a second image, the semantic pattern referring to a specific pattern in a physiological sense close to a human face; and adjusting the hidden information to make the semantic pattern similar to a predefined semantic pattern and make the second image similar to the real image.

2. The method of claim 1, the semantic pattern and the predefined semantic pattern being similar comprising: a reconstruction loss function of the semantic pattern and the predefined semantic pattern is less than or equal to a first threshold.

3. The method of claim 1, the second image and the real image being similar comprising: a visual quality loss function of the second image and the real image is less than or equal to a second threshold.

4. The method of claim 1, the hidden information comprising invisible watermark.

5. The method of claim 1, adding hidden information to the received real image is implemented by a conditional generative adversarial network (CGAN) generator.

6. The method of claim 1, generating a semantic pattern in a designated region of the first image based on the hidden information is implemented by a DeepFake generator.

7. An active defense apparatus for image protection, comprising: a first image generation module that receives a real image and adds hidden information to the received real image to generate a first image; a second image generation module that generates a semantic pattern in a designated region of the first image based on the hidden information to generate a second image, the semantic pattern referring to a specific pattern in a physiological sense close to a human face; and an adjustment module that adjusts the hidden information to make the semantic pattern similar to a predefined semantic pattern and make the second image similar to the real image.

8. The apparatus of claim 7, the semantic pattern and the predefined semantic pattern similar comprising: a reconstruction loss function of the semantic pattern and the predefined semantic pattern is less than or equal to a first threshold.

9. The apparatus of claim 7, the second image and the real image similar comprising: a visual quality loss function of the second image and the real image is less than or equal to a second threshold.

10. The apparatus of claim 7, the hidden information comprising invisible watermark.

11. The apparatus of claim 7, the first image generation module being further configured to add the hidden information to the received real image by a conditional generative adversarial network (CGAN) generator.

12. The apparatus of claim 7, the second image generation module being further configured to generate a semantic pattern in a designated region of the first image based on the hidden information by a DeepFake generator.

13. The apparatus of claim 7, further comprising a publishing module configured to: receive a real image to be published; add the adjusted hidden information to the real image to be published to generate a publishing image for external publication.

14. A computer-readable storage medium storing a computer program executable by a processor to perform the method of any one of claims 1-6.

Images