Face deep counterfeiting evidence obtaining method, system and device and storage medium

Abstract

The invention discloses a face deep counterfeiting evidence obtaining method, system and device and a storage medium, which are one-to-one corresponding schemes, and in the scheme, multi-level regional semantic features of a face are extracted, a robust watermark and a semi-fragile watermark are adaptively embedded into different semantic regions, a watermark image is finally reconstructed, and then the watermark image can be published to the outside; moreover, two kinds of watermark information are extracted from the obtained watermark image by using the characteristic that the integrity of the watermark is difficult to maintain in the deep counterfeiting process, so that on one hand, identity tracing can be realized, and on the other hand, deep counterfeiting detection can be realized through cross comparison; the scheme provided by the invention is a detection scheme which is initiatively traceable and is irrelevant to a specific deep counterfeiting method, the traceability and detection performance of deep counterfeiting can be remarkably improved, and experimental results on a plurality of data sets all show that the method reaches a leading level.

Description

Technical Field

[0001] The present invention relates to the technical field of deep fake face evidence collection, and in particular to a deep fake face evidence collection method, system, device and storage medium. Background Art

[0002] Face deep fake technology has been widely used in the entertainment field due to its powerful generation ability, such as film and television production, game character modeling, and digital special effects. However, while this technology brings innovation and convenience, it also has the risk of being abused, leading to some negative impacts that cannot be ignored. In particular, without authorization, criminals may use deep fake technology to forge facial images and videos of public figures or ordinary citizens. In addition, the abuse of deep fake technology provides technical support for some illegal activities, further undermining people's trust in the authenticity of images and videos. Therefore, how to effectively identify and defend against the malicious application of face deep fake technology has become an urgent problem to be solved.

[0003] Existing deepfake defense mainly adopts passive detection methods, which are regarded as binary classification tasks, and convolutional neural networks are used to learn forgery features and output classification results. However, these methods have poor performance when dealing with unknown forgery techniques and cannot be applied to deepfake forensics tasks in open environments. One effective solution is to embed traceable or auxiliary information in the image through digital watermarking technology. Existing digital watermarking technology usually focuses on the robustness of the watermark, requiring the embedded watermark to remain stable under various image transformations and distortions. When implementing deepfake detection, the robust watermark is further extended to a semi-fragile watermark, which makes it resistant to common image distortions and sensitive to various deepfake methods, thereby achieving detection by comparing the decoded watermark with the original embedded watermark. However, this detection method requires the original embedded watermark information to be known in advance, and cannot achieve forensics tasks in blind detection scenarios.

[0004] In view of this, the present invention is proposed. Summary of the invention

[0005] The purpose of the present invention is to provide a method, system, device and storage medium for collecting evidence of deep fake face. Different types of watermarks can be obtained by notification decoding, and cross-comparison can be used to determine whether the image has been deep faked, thereby verifying the authenticity of the image. At the same time, the decoded watermark can be used to trace the identity. Compared with existing solutions, highly generalized blind detection can be achieved.

[0006] The objective of the present invention is achieved through the following technical solutions: A method for obtaining evidence of deep fake face, comprising: Watermarking stage: receiving the face image to be protected and the binary sequence, performing multi-level semantic extraction and regional semantic segmentation on the face image, and synthesizing them into multi-level regional semantic features; for each level of regional semantic features, adaptively generating one of the two types of watermarks set based on the binary sequence, and embedding it into the regional semantic features of the corresponding level, and reconstructing the watermark image by synthesizing all the regional semantic features embedded in the watermark; Watermark extraction and verification stage: Two types of watermarks are extracted from the input watermark image respectively, the source is traced through the proposed watermarks, and cross-comparison is used to determine whether the input watermark image has been deeply forged.

[0007] A face deep fake forgery evidence collection system, comprising: a face deep fake forgery evidence collection model; the aforementioned method is implemented based on the face deep fake forgery evidence collection model, the face deep fake forgery evidence collection model comprising: The watermark adding module is used to receive the face image to be protected and the binary sequence, perform multi-level semantic extraction and regional semantic segmentation on the face image, and synthesize them into multi-level regional semantic features; for each level of regional semantic features, adaptively generate one of the two types of watermarks set based on the binary sequence, and embed it into the regional semantic features of the corresponding level, and reconstruct the watermark image by synthesizing all the regional semantic features embedded in the watermark; The watermark extraction and verification module is used to extract two types of watermarks from the input watermark image, trace the source through the proposed watermarks, and determine whether the input watermark image has been deeply forged through cross-comparison.

[0008] A processing device, comprising: one or more processors; a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0009] A readable storage medium stores a computer program, which implements the above method when the computer program is executed by a processor.

[0010] It can be seen from the technical solution provided by the present invention that the multi-level regional semantic features of the face are extracted, and different types of watermarks (i.e., robust watermarks and semi-fragile watermarks) are adaptively embedded in different semantic areas, and finally the watermark image is reconstructed, which can then be released externally; and, by utilizing the characteristic that the integrity of the watermark is difficult to maintain during the deep forgery process, two types of watermark information are extracted from the acquired watermark image, which can, on the one hand, achieve identity tracing, and on the other hand, achieve deep forgery detection through cross-comparison; the above solution provided by the present invention is an active tracing and detection solution that is independent of the specific deep forgery method, which can significantly improve the tracing and detection performance of deep forgeries, and the experimental results on multiple data sets have shown that it has reached a leading level. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0012] Figure 1 A flowchart of a method for obtaining evidence of deep fake faces provided by an embodiment of the present invention; Figure 2 A schematic diagram of a visualization result of a generated watermark image provided by an embodiment of the present invention; Figure 3 A schematic diagram of a training framework for a deep fake face forensics model provided in an embodiment of the present invention; Figure 4 A schematic diagram of a processing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0013] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of the present invention.

[0014] First, the terms that may be used in this article are explained as follows: The terms "include", "comprises", "contains", "has" or other descriptions with similar semantics should be interpreted as non-exclusive inclusion. For example, including certain technical feature elements (such as raw materials, components, ingredients, carriers, dosage forms, materials, dimensions, parts, components, mechanisms, devices, steps, procedures, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products or products, etc.) should be interpreted as including not only certain technical feature elements explicitly listed, but also other technical feature elements known in the art that are not explicitly listed.

[0015] The term "consisting of..." means excluding any technical feature elements not explicitly listed. If this term is used in a claim, it will make the claim closed, so that it does not contain technical feature elements other than the technical feature elements explicitly listed, except for the conventional impurities related to them. If this term only appears in a clause of a claim, it only limits the elements explicitly listed in the clause, and the elements recorded in other clauses are not excluded from the overall claim.

[0016] The following is a detailed description of a facial deep fake evidence collection method, system, device and storage medium provided by the present invention. The contents not described in detail in the embodiments of the present invention belong to the prior art known to professional and technical personnel in this field. If no specific conditions are specified in the embodiments of the present invention, they are carried out according to the conventional conditions in the field or the conditions recommended by the manufacturer. The reagents or instruments used in the embodiments of the present invention that do not indicate the manufacturer are all conventional products that can be purchased commercially.

[0017] Embodiment 1 The embodiment of the present invention provides a method for obtaining evidence of deep fake face. Figure 1 As shown, it mainly includes: (1) Watermark adding stage.

[0018] The main process of this stage includes: receiving the face image and binary sequence to be protected, performing multi-level semantic extraction and regional semantic segmentation on the face image, and integrating them into multi-level regional semantic features; for each level of regional semantic features, adaptively generating one of the two types of watermarks set based on the binary sequence, and embedding it into the regional semantic features of the corresponding level, and reconstructing the watermark image by integrating all the regional semantic features embedded in the watermarks.

[0019] In an embodiment of the present invention, the two types of watermarks set include: a robust watermark and a semi-fragile watermark; based on the binary sequence, a corresponding type of watermark is generated through a diffusion process; the diffusion process includes: a fully connected neural network processing process and an upsampling process; the adaptively generated watermark is spliced ​​with the corresponding level of regional semantic features, and fused through an attention mechanism to obtain the regional semantic features of the embedded watermark; all the regional semantic features of the embedded watermarks are integrated to reconstruct the watermark image through a facial reconstruction network.

[0020] (2) Watermark extraction and verification stage.

[0021] The main processes of this stage include: extracting two types of watermarks from the input watermark image, tracing the source through the proposed watermarks, and judging whether the input watermark image has been deeply forged through cross-comparison.

[0022] In the embodiment of the present invention, the robust watermark can be used for traceability; the bit error rate difference between the robust watermark and the semi-fragile watermark can be obtained through cross comparison, thereby judging whether the input watermark image has been deeply forged: when the bit error rate difference is less than the set judgment threshold, the input watermark image is judged to be a real image; otherwise, the input watermark image is judged to be deeply forged.

[0023] Preferably, the watermark adding stage is implemented by a watermark adding module, and the watermark extraction and verification stage is implemented by a watermark extraction and verification module, and the two together constitute a face deep fake forensics model; the face deep fake forensics model is pre-trained, and a progressive noise layer and a perturbation adversarial generation network are introduced during the training process; the training process is divided into two parts, the former part is called the early training period, and the latter part is called the late training period; in the early training period, a number of predefined image transformations are randomly selected as noise through the progressive noise layer, and image perturbations are applied to the watermark image; in the late training period, image transformation operations are generated through the perturbation adversarial generation network, image perturbations are applied to the watermark image, and the training is alternated with the subsequent watermark extraction and verification module in an adversarial training manner.

[0024] From the perspective of specific implementation: the watermark adding stage mainly processes the facial image provided by the user to be protected to obtain a watermark image. The watermark image has been embedded with watermark information, which can achieve accurate source tracking, and thus the watermark image can be published to the network end; when detection is required, the watermark image can be obtained from the network end, and then two types of watermarks can be extracted, thereby realizing identity tracing and deep fake detection.

[0025] In the specific implementation process, the above-mentioned face deep fake forensics model can be deployed on a computer or server, which can automatically embed watermarks into face images and determine whether the images have been forged. It can be widely used in various social platforms, such as short video platforms, photo sharing websites, etc., to provide effective content protection and authenticity verification functions.

[0026] In order to more clearly demonstrate the technical solution and technical effects provided by the present invention, the method provided by the embodiment of the present invention is described in detail with reference to specific embodiments below.

[0027] 1. Overall overview of the plan and description of its effects.

[0028] The embodiment of the present invention provides a method for collecting evidence of deep fake face, which is based on multi-level facial semantic watermarking technology. Unlike the traditional method of passively detecting the generated fake content, the present invention adopts an active strategy to pre-embed watermark information in the original image to achieve accurate source tracking. By taking advantage of the fact that the integrity of the watermark is difficult to maintain during the deep fake process, the present invention demonstrates a highly generalized deep fake detection capability.

[0029] In an embodiment of the present invention, a facial feature decoupling and reconstruction architecture is introduced into the watermark adding module, aiming to provide an active traceability and detection strategy that is independent of the specific deep fake method. In this architecture, the present invention successfully embeds watermarks with different characteristics, namely robustness and semi-fragility, into semantic areas at different levels. The embedding of the robust watermark enables it to resist common distortion operations and deep fake tampering, ensuring reliable traceability verification and tracking. Once the facial semantics are tampered with, the semi-fragile watermark will be destroyed, thereby quickly identifying traces of image forgery. After training and testing on the high-definition face datasets CelebA and CelebA-HQ, the present invention has demonstrated excellent performance. In the traceability and robustness tests, the bit error rate of the robust watermark was only 0.0476%, and the bit error rate of the semi-fragile watermark was only 0.0982%. In the deep fake perturbation test, the bit error rate of the robust watermark performed excellently, and the detection success rate using the semi-fragile watermark was as high as 98.31%. In addition, Figure 2 The watermark image effect generated by the present invention is intuitively demonstrated, wherein the watermark information is almost invisible, which fully proves the high quality performance of the present invention in watermark generation. Figure 2In the figure, the first five columns are the results of the present invention, the first column is JPEG compression, compression quality = 50; the second column is Gaussian blur, Gaussian kernel standard deviation = 2, Gaussian kernel size = 3; the third column is median filtering, filter kernel size = 3; the fourth column is salt and pepper noise, noise probability = 0.1; the fifth column is Gaussian noise, mean = 0, standard deviation = 0.1; then the 6th to 12th columns are the results of the existing schemes, SimSwap is a high-fidelity image face-changing model, StarGAN is a face attribute editing based on multi-domain migration generative adversarial network, DiffFace is an image face-changing model based on diffusion model, HFGI is a face attribute editing based on high-fidelity generative adversarial network inverse mapping, StyleMask is a facial manipulation based on stylized generative adversarial network inversion, E4S is a fine-grained face-changing model based on regional generative adversarial network inversion, and InfoSwap is an image face-changing model based on information bottleneck decoupling.

[0030] 2. Detailed introduction of the plan.

[0031] 1. Watermark adding stage.

[0032] This stage mainly involves regional semantic watermark injection. Specifically, it is obtained by decoupling the multi-level semantic coding network and segmenting the semantic region. Level region semantic attributes ; Further, the watermark encoder is used to adaptively embed the binary sequence into the semantic attributes at all levels; finally, the face reconstruction network is used to output the watermark image. As mentioned before, this stage can be achieved through the watermark adding module, and the specific design of the modules involved is as follows:

[0033] (1) Multi-level semantic encoding network.

[0034] Through a multi-level semantic encoding network Multi-level semantics (such as facial contour, hair color, expression attributes, identity information, etc.) are extracted from the face image to be protected, expressed as: ; in, is the face image to be protected, For the The semantic features of the For the series.

[0035] For example, the feature extraction network of the U-Net architecture can be used as a multi-level semantic encoding network, and the multi-level semantic features can be learned in an end-to-end manner to obtain A set of semantic feature maps with different resolution levels, The corresponding resolution is , and are the height and width, C is the number of channels. For example, when the face image is an RGB (red, green, and blue channel) image, C=3.

[0036] (2) Semantic region segmentation module.

[0037] Through facial region semantic segmentation network The face image to be protected is subjected to regional semantic segmentation, which is expressed as: ; in, For the semantic region masks, N is the number of semantic region masks obtained by region semantic segmentation (for example, N=12), , is the symbol of the real number set, , They correspond to the height and width of the input image respectively. The segmentation results at each level represent semantic areas such as background, hair, nose, mouth, eyes and facial area.

[0038] Exemplary, face region semantic segmentation network It can be composed of a Bilateral Segmentation Network (BiSeNet) pre-trained on the CelebAMask-HQ dataset.

[0039] The outputs of the above two modules are combined to obtain regional semantic features. Specifically, for each level, the following formula is used to synthesize the regional semantic features: ; in, For the The regional semantic features of the level, which contains N semantic region masks, For Hadamard, Indicates that Downsample to The corresponding size.

[0040] (3) Watermark encoder.

[0041] In the implementation of the present invention, two independent watermark encoders are used to support the embedding of robust watermarks and semi-fragile watermarks. First, the binary sequence to be embedded is Diffusion to k semantic attribute feature maps through fully connected neural network and upsampling process Same Dimension ; Then the diffused binary sequence is concatenated with the regional semantic features at all levels and fused through the attention mechanism, and finally a total of k semantic features embedded with watermarks are output.

[0042] Among them, the watermark embedding process of each level of regional semantic features is expressed as: ; in, For the The regional semantic features of Represents a binary sequence Through the diffusion process The generated watermark is a robust watermark or a semi-fragile watermark, symbol Indicates splicing, represents the attention convolution module with attention mechanism, Indicates The semantic features of the watermark are embedded in the data level.

[0043] In the embodiment of the present invention, the watermark is adaptively embedded into the regional semantic features at each level. That is, it does not explicitly distinguish which type of watermark should be embedded in the regional semantic features at each level. Instead, the two types of watermarks are separated in conjunction with the subsequent decoding process, and end-to-end overall training is adopted, so that the model can automatically learn the specific type of watermark that should be embedded in the regional semantic features at each level.

[0044] (4) Face reconstruction network.

[0045] In the face reconstruction network, k semantic features embedded with watermarks It is input into the generation network step by step, and the semantic fusion is realized by the Adaptive Instance Normalization (AdaIN) module at each level, and finally the watermark image is output step by step. .

[0046] 2. Watermark extraction and verification stage.

[0047] Similarly, this stage can be implemented by the watermark verification and extraction module. Since the watermark is embedded in a learnable way, a deep separable watermark decoder is configured in the watermark verification and extraction module. and They are used to extract the robust watermark and the semi-fragile watermark respectively. The process is as follows: ; ; Among them, robust watermark It is robust to both conventional image perturbations and malicious deep forgeries, and semi-fragile watermarking Robust to regular image perturbations, but vulnerable to malicious deepfakes; is the watermark image input to the watermark verification and extraction module, Indicates that the watermarked image has been disturbed.

[0048] After obtaining the two types of watermarks, the robust watermark can be used for tracing the source; by comparing the bit error rate difference between the robust watermark and the semi-fragile watermark , it can be verified whether the image is deeply forged. The principle is: for a real image, its error expectation ; For forged images, the error expectation That is, the average semi-fragile bit error rates of real images and forged images are distributed as follows: ; ; Among them, p(.) represents the probability value of the expression in brackets, c represents the category of the image to be detected, Represents that the input image is a real image, represents that the input image is a fake image. Represents the variance of the corresponding normal distribution N(,).

[0049] Therefore, the decision threshold The selection of must satisfy the maximum likelihood: ; in, The watermark image is input to the watermark verification and extraction module, which is the watermark image mentioned above. , it has undergone unknown disturbances during the Internet propagation process (that is, it may have undergone regular image perturbations or malicious forgeries during the propagation process).

[0050] According to the distribution of the average semi-fragile bit error rate of the real image and the forged image, the variances of the two are the same, and the means are 0 and 0.5 respectively, that is, when When , the maximum likelihood of the above equation can be satisfied.

[0051] That is, when the difference between the robustness and semi-fragility bit error rates is Less than the judgment threshold (For example, 0.25) is judged as a real image (that is, the generated watermark image has only been subjected to conventional image disturbances during the propagation process and has not undergone deep forgery), otherwise it is a deep forged image. Therefore, it can be verified whether the image has been maliciously tampered with by deep forgery technology.

[0052] 3. Model training plan.

[0053] The above watermark adding module and the watermark extraction and verification module together constitute a face deep fake forensics model. The face deep fake forensics model is pre-trained. In order to achieve the robustness of robust watermarks and semi-fragile watermarks to conventional image perturbations, a progressive noise layer and a perturbation adversarial generation network are introduced during the training process. In addition, an adversarial discriminator is introduced to distinguish the input face image to be protected and the watermark image respectively. Figure 3 shown.

[0054] The training process is divided into two parts, the former part is called the early training stage, and the latter part is called the late training stage; in the early training stage, in order to keep the training process stable, the progressive noise layer randomly selects several predefined image transformations as noise, and applies image perturbations to the watermark image; illustratively, 4 types of image transformations can be selected as noise, including Gaussian blur, Gaussian noise, median filtering, and JPEG (Joint Photographic Experts Group) compression. In the late training stage, the perturbation adversarial generative network (GAN) generates more challenging image transformation operations, applies image perturbations to the watermark image, and alternately trains with the watermark extraction and verification module in an adversarial training manner, further enhancing the robustness of the watermark to unknown image distortions.

[0055] The training loss is expressed as: ; in, is the training loss, , , Correspondingly, they represent the watermark decoding loss term, the watermark image perceptual loss term, and the adversarial loss term. , and is the weight of the corresponding loss term.

[0056] The watermark decoding loss term is obtained by calculating the difference between the two types of watermarks extracted and the binary sequence, which is expressed as: ; in, represents the mean square error function.

[0057] The perceptual loss term of the watermark image is obtained by calculating the difference between the face image to be protected and the watermark image, which is expressed as: ; in, is the 2-norm symbol, LPIPS is the perceptual loss function, The watermark image.

[0058] The adversarial loss term is calculated by the discriminant result output by the introduced adversarial discriminator, expressed as: ; in, is the adversarial discriminator.

[0059] Figure 3 middle, For example, in the early stage of training (e.g., the first 50 rounds), image perturbations (including regular perturbations and malicious tampering) are generated by predefined perturbation types; in the later stage of training, the perturbation adversarial generation network is used to enhance the decoding accuracy adversarially. Here, the network learns to generate more diverse regular perturbations, while malicious tampering is still generated by the predefined set; the reverse diffusion F -1 Watermark decoder and Internal decoding process. There are three types of situations in the decoding process of , and these three situations are simulated in training to achieve complete end-to-end training. These three situations are as follows: (1) After conventional perturbations: The semi-fragile watermark is robust to conventional image perturbations, so the watermark exists after semi-fragile decoding, that is, the BER should tend to 0. BER is the bit error rate, which refers to the bit error rate of the decoded binary sequence and the given binary sequence for the watermark (that is, the ratio of the number of predicted wrong bits to the total length).

[0060] (2) After malicious tampering: The semi-fragile watermark is vulnerable to malicious tampering disturbances, so the watermark does not exist after semi-fragile decoding, that is, the BER should be close to 0.5.

[0061] (3) After various disturbances (including regular disturbances and two minor cases of malicious tampering): The robust watermark is robust to all kinds of disturbances. Therefore, the watermark exists after passing through the robust decoder, that is, the BER should tend to 0.

[0062] In terms of training details: a multi-level GAN ​​structure was selected to enhance the ability to distinguish detail areas. The model in this invention was trained on a single GPU card, with 32 face images input at a time. The Adam (Adaptive Moment Estimation) optimizer was used for optimization, with the initial learning rate of the model set to 0.001 and the initial learning rate of the adversarial discriminator set to 0.0001. For more comprehensive training, the learning rate was adjusted to 1×10 after 100 rounds using the learning rate cosine annealing decay method. -8 The related training process involved can refer to conventional technology, and the present invention will not elaborate on it.

[0063] The above solution provided by the embodiment of the present invention uses an end-to-end approach to extract multi-level facial semantics, and further embeds the robust watermark and the semi-fragile watermark into different semantic regions through the watermark encoder adaptively, and finally generates a watermark image through the facial reconstruction network; in the watermark extraction and detection stage, the robust watermark and the semi-fragile watermark can be decoded simultaneously to achieve highly generalized blind detection. Thanks to the above improvements, the present invention significantly improves the traceability and detection performance of deep fakes, and the experimental results on multiple data sets show that it has reached a leading level.

[0064] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by means of software plus necessary general hardware platforms. Based on such understanding, the technical solutions of the above embodiments can be embodied in the form of software products, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0065] Embodiment 2 The present invention also provides a face deep fake evidence collection system, which mainly includes: a face deep fake evidence collection model, which can implement the method provided in the above embodiment based on the above face deep fake evidence collection model, and the system mainly includes: The watermark adding module is used to receive the face image to be protected and the binary sequence, perform multi-level semantic extraction and regional semantic segmentation on the face image, and synthesize them into multi-level regional semantic features; for each level of regional semantic features, adaptively generate one of the two types of watermarks set based on the binary sequence, and embed it into the regional semantic features of the corresponding level, and reconstruct the watermark image by synthesizing all the regional semantic features embedded in the watermark; The watermark extraction and verification module is used to extract two types of watermarks from the input watermark image, trace the source through the proposed watermarks, and determine whether the input watermark image has been deeply forged through cross-comparison.

[0066] Considering that the relevant technical details involved in the model have been introduced in detail in the above embodiments, they will not be repeated here.

[0067] Technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0068] Embodiment 3 The present invention also provides a processing device, such as Figure 4 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the methods provided in the aforementioned embodiments.

[0069] Furthermore, the processing device also includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0070] In the embodiment of the present invention, the specific types of the memory, input device and output device are not limited; for example: The input device may be a touch screen, an image acquisition device, a physical button or a mouse, etc.; The output device may be a display terminal; The memory may be a random access memory (RAM) or a non-volatile memory such as a disk storage.

[0071] Embodiment 4 The present invention also provides a readable storage medium storing a computer program, which implements the method provided in the above embodiment when the computer program is executed by a processor.

[0072] In the embodiment of the present invention, the readable storage medium as a computer-readable storage medium can be set in the aforementioned processing device, for example, as a memory in the processing device. In addition, the readable storage medium can also be a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk, etc., which can store program codes.

[0073] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with the technical field within the technical scope disclosed in the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims. The information disclosed in the background technology section of this article is only intended to deepen the understanding of the overall background technology of the present invention, and should not be regarded as an admission or in any form that the information constitutes prior art known to those skilled in the art.

Claims

1. A method for collecting evidence of deep fake face, characterized in that: include: Watermarking stage: receiving the face image to be protected and the binary sequence, performing multi-level semantic extraction and regional semantic segmentation on the face image, and synthesizing them into multi-level regional semantic features; for each level of regional semantic features, adaptively generating one of the two types of watermarks set based on the binary sequence, and embedding it into the regional semantic features of the corresponding level, and reconstructing the watermark image by synthesizing all the regional semantic features embedded in the watermark; Watermark extraction and verification stage: Two types of watermarks are extracted from the input watermark image respectively, the source is traced through the proposed watermarks, and cross-comparison is used to determine whether the input watermark image has been deeply forged.

2. A method for collecting evidence of deep fake face according to claim 1, characterized in that: The multi-level semantic extraction and regional semantic segmentation of the face image are performed respectively, and the multi-level regional semantic features are synthesized, including: Through a multi-level semantic encoding network Multi-level semantic extraction is performed on the face image to be protected, which is expressed as: ； in, is the face image to be protected, For the The semantic features of the is the series; Through facial region semantic segmentation network The face image to be protected is subjected to regional semantic segmentation, which is expressed as: ； in, For the semantic region masks, N is the number of semantic region masks obtained by regional semantic segmentation; For each level, the following formula is used to synthesize regional semantic features: ； in, For the The regional semantic features of For Hadamard, Indicates that Downsample to The corresponding size.

3. A method for collecting evidence of deep fake face according to claim 1, characterized in that: The two types of watermarks set include: robust watermark and semi-fragile watermark; based on the binary sequence, the corresponding type of watermark is generated through a diffusion process; the diffusion process includes: a fully connected neural network processing process and an upsampling process.

4. A method for collecting evidence of deep fake face according to claim 1 or 3, characterized in that: The adaptive generation of one of the two types of watermarks set based on the binary sequence and embedding into the corresponding level regional semantic features includes: The adaptively generated watermark is concatenated with the corresponding level regional semantic features, and fused through the attention mechanism to obtain the regional semantic features embedded in the watermark; Among them, the watermark embedding process of each level of regional semantic features is expressed as: ； in, For the The regional semantic features of Represents a binary sequence Through the diffusion process Generated watermark, symbol Indicates splicing, represents the attention convolution module with attention mechanism, Indicates The semantic features of the watermark are embedded in the data level.

5. The method for collecting evidence of deep fake face according to claim 1, characterized in that: The extracting of two types of watermarks from the input watermark image, tracing the source through the proposed watermarks, and judging whether the input watermark image has been deeply forged through cross comparison include: The two types of watermarks include: robust watermarks and semi-fragile watermarks; Traceability through robust watermarking; Through cross-comparison, the bit error rate difference between the robust watermark and the semi-fragile watermark is obtained, thereby judging whether the input watermark image has been deeply forged: when the bit error rate difference is less than the set judgment threshold, the input watermark image is judged to be a real image, otherwise, the input watermark image is judged to be deeply forged.

6. A method for collecting evidence of deep fake face according to claim 1, characterized in that: Also includes: The watermark adding stage is implemented by a watermark adding module, and the watermark extraction and verification stage is implemented by a watermark extraction and verification module, which together constitute a face deep forgery forensics model; The face deep fake forensics model is pre-trained, and a progressive noise layer and a perturbation adversarial generative network are introduced during the training process; and an adversarial discriminator is introduced to distinguish the input face image to be protected and the watermark image respectively; The training process is divided into two parts, the former is called the early training period and the latter is called the late training period. In the early stage of training, a plurality of predefined image transformations are randomly selected as noises through the progressive noise layer to apply image disturbance to the watermark image; In the later stage of training, the perturbation adversarial generative network generates an image transformation operation, applies image perturbation to the watermark image, and is trained alternately with the watermark extraction and verification module in an adversarial training manner.

7. A method for collecting evidence of deep fake face according to claim 6, characterized in that: The training loss is expressed as: ； in, is the training loss, , , Correspondingly, they represent the watermark decoding loss term, the watermark image perceptual loss term, and the adversarial loss term. , and is the weight of the corresponding loss item; the watermark decoding loss item is obtained by calculating the difference between the two types of watermarks extracted and the binary sequence respectively; the perceptual loss item of the watermark image is obtained by calculating the difference between the face image to be protected and the watermark image; the adversarial loss item is calculated by the discrimination result output by the adversarial discriminator.

8. A facial deep fake evidence collection system, characterized in that: include: Face deep fake forensics model; The method according to any one of claims 1 to 7 is implemented based on the face deep fake forensics model, wherein the face deep fake forensics model comprises: The watermark adding module is used to receive the face image to be protected and the binary sequence, perform multi-level semantic extraction and regional semantic segmentation on the face image, and synthesize them into multi-level regional semantic features; for each level of regional semantic features, adaptively generate one of the two types of watermarks set based on the binary sequence, and embed it into the regional semantic features of the corresponding level, and reconstruct the watermark image by synthesizing all the regional semantic features embedded in the watermark; The watermark extraction and verification module is used to extract two types of watermarks from the input watermark image, trace the source through the proposed watermarks, and determine whether the input watermark image has been deeply forged through cross-comparison.

9. A processing device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images