Deep counterfeiting active detection method based on block chain and robust watermark enhancement

Abstract

The invention discloses a deep counterfeiting active detection method based on block chain and robust watermark enhancement, and relates to the technical field of image processing, and the method is characterized in that the method comprises the following steps: S1, training a watermark embedding extraction network; s2, embedding a 100-bit random binary bit watermark into the original face image through a watermark embedding model, and storing watermark information to a block chain; s3, training a deep forgery detector, and carrying out image enhancement on the watermark image; and S4, performing deep counterfeiting on the enhanced watermark image, and extracting the watermark from the deep counterfeiting image. The technical problem to be solved by the invention is to provide a deep counterfeiting active detection method based on block chain and robust watermark enhancement, and an efficient and reliable solution is provided for deep counterfeiting detection and traceability by embedding and extracting watermark information and combining a block chain technology.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a deep forgery active detection method based on blockchain and robust watermark enhancement. Background Art

[0002] Deepfake, as an image and video generation technology based on deep learning, has developed rapidly in recent years, showing a high degree of generation fidelity and wide application potential. Through deep neural networks, this technology can synthesize virtual images that are extremely similar to the original human face, thereby replacing the face, imitating movements, and reenacting expressions, thereby generating highly deceptive fake videos. These fake videos are not only visually extremely realistic, but also perfectly replicate the behavior and emotional expressions of the characters, making the fake content almost indistinguishable from the real video. With the popularization of Deepfake technology, the potential risks it brings are becoming increasingly apparent, especially in terms of data security, privacy protection, and social ethics.

[0004] To address this problem, researchers have proposed a variety of defense techniques, which can be mainly divided into two categories: passive detection and active defense. Passive detection focuses on post-event evidence collection, relying on features or traces in the video to identify whether it is forged. Current technical methods mostly rely on spatial features, frequency domain features, or forgery traces left by the generative model. For example, classic neural networks such as Xception Net and Efficient Net are used to extract features of video frames, while F3-Net and SPSL methods enhance the detection accuracy of forged content by enhancing frequency domain information. In addition, some methods use attention mechanisms to focus on subtle forgery traces in video frames, thereby further improving the accuracy of detection, such as the Multi-Attention method. Although these passive detection technologies have made progress to a certain extent, due to the uneven quality of data sets and the continuous upgrading of forgery technology, existing methods still face huge challenges, especially in the detection of high-quality forged videos, where the accuracy is still insufficient.

[0005] Unlike passive detection, active defense technology focuses more on taking preventive measures before forgery occurs. The basic idea is to embed secret information, such as watermarks or adversarial disturbances, into video content to prevent forgery or facilitate tracing the source of forgery. Active defense technology can be further divided into two types: active interference and active forensics. Active interference interferes with the training process of the generative model by adding noise or disturbances to the video data to prevent the generation of forged content. For example, defense methods based on adversarial attacks and data poisoning can effectively protect face data from forgery. Active forensics embeds imperceptible fingerprints or keys into face videos to facilitate forgery tracking and tracing. For example, a method of embedding artificial fingerprints in training data is used to trace forged images, and a decentralized attribute scheme based on user-specific keys is used to enable effective tracking after forgery occurs. Through these technologies, active defense can significantly reduce the risk of the spread of forged content and improve the overall protection effect.

[0006] Although existing Deepfake defense technologies have made some progress in some areas, they still face many key problems. For example, existing passive detection methods have obvious shortcomings when dealing with high-quality forged content, especially when facing new generation models (such as diffusion models), the robustness and accuracy of detection are significantly reduced. At the same time, although active defense technology can effectively prevent forgery, its implementation relies on embedding watermarks or perturbations in the image in advance, which may affect the quality of the video and is vulnerable to attacks by attackers. In addition, in the scenario of large-scale applications, how to balance data privacy protection and the performance of anti-counterfeiting technology remains a major challenge.

[0007] The prior art has the following deficiencies: 1. How to embed and maintain imperceptible and highly robust watermark information in image generation, forgery, and subsequent image enhancement processing, so that it can resist various image processing operations and ensure that the watermark information can still be identified after multiple stages of operations, so as to track forgery behavior and trace the source; 2. How to improve the visual quality of the image after embedding the watermark, avoid interference with the original image, and reduce the risk of being misjudged as a forged image by the deep fake detection system; 3. How to securely store watermark information to prevent it from being tampered with or forged, ensure the security and credibility of image traceability data, and at the same time provide a transparent and reliable forged content responsibility tracing mechanism to enhance the credibility and verifiability of the anti-counterfeiting system. Summary of the invention

[0008] The technical problem to be solved by the present invention is to provide an active deep counterfeit detection method based on blockchain and robust watermark enhancement. By embedding and extracting watermark information and combining blockchain technology, an efficient and reliable solution is provided for deep counterfeit detection and traceability.

[0009] The present invention adopts the following technical solutions to achieve the invention objectives: A deep forgery active detection method based on blockchain and robust watermark enhancement, characterized by comprising the following steps: S1: Train a watermark embedding extraction network; S2: embed a 100-bit random binary bit watermark into the original face image through the watermark embedding model, and save the watermark information to the blockchain; S3: Train a deep fake detector and perform image enhancement on the watermarked image; S4: Deep forging the enhanced watermark image, and then extracting the watermark from the deep forged image; S5: Compare and verify the watermark extracted from the deep fake image with the watermark stored in the blockchain.

[0010] As a further limitation of the technical solution, the specific steps of S1 are: S11: Setting a watermark embedding network model E with an encoder-decoder structure and a watermark extraction network model D with a decoder structure; S12: Watermark Embedding Network Model , where E is the watermark embedding network model, For a clean face image, is a binary watermark, To embed a watermarked face image, the model uses a randomly generated 100-bit binary watermark Embedded into clean face image In this paper, adversarial training is used to embed watermarked face images. Add slight disturbances, then input the deep fake model for malicious tampering, and extract the network model with watermarks , where D is the watermark extraction network model, For watermarked face images (including face images with only watermarks, images with slightly disturbed watermarks, and images with maliciously tampered watermarks), To extract the binary watermark, the model extracts the watermark information from the image. Both the watermark embedding network model and the watermark extraction network model are designed as convolutional neural networks. The training goal is to minimize the following two loss functions: binary cross entropy loss And mean square error loss : (1) in: and are the first Bit; n is the length of the watermark; (2) in: It is the image after watermarking through the watermark embedding network model; is the original clean face image; represents the L2 norm, i.e., the Euclidean distance; S13: The final training goal is to minimize the weighted sum of the two: (3) in: is a hyperparameter used to balance the weights of these two loss terms.

[0011] As a further limitation of the technical solution, the specific steps of S2 are: S21: The watermark information is mapped to a suitable size through a fully connected layer and prepared and compared with the clean face image to integrate; S22: Dynamically adjust the embedding strength of the watermark based on the local complexity of the image to ensure that the effect of the watermark embedded in the image can adapt to changes in the image content; By calculating the input clean face image The local complexity of the image is used to quantify the characteristics of the image; Set clean face image For a size of A tensor of , where: B is the batch size, C is the number of channels of the image, H and W are the height and width of the image; The local complexity of an image is calculated using the following formula: (4) in: Indicates the variance calculation in the height and width directions; and are the indices of batch and channel respectively; The adaptive embedding weight is calculated based on the local complexity of the image and the embedding strength embedding_strength. The local complexity is scaled by a Sigmoid function so that higher complexity values ​​correspond to stronger embedding strength. (5) in: represents the adaptive weight of each channel at each image position; is the Sigmoid function; Watermark information According to the adaptive embedding weight, the watermark information It is a tensor of the same size as the input clean face image, representing the secret information to be embedded. The embedding strength is adjusted to an adaptive weight through the following formula: (6) S23: Use multiple convolutional layers and deconvolutional layers to extract and restore image features, and gradually restore the structural details of the image to embed the watermark information into the original image. In this stage, for each clean face image Assign a unique binary watermark Then, use the trained watermark to embed the network model Each clean face image Convert to watermark image and get the image embedded with watermark ; S24: The watermark information is stored in the blockchain through smart contracts.

[0012] As a further limitation of the present technical solution, the specific steps of S3 are: S31: Select face images from the dataset as positive examples and take the following steps to generate negative samples to train the CNN model; Detect faces in the original image and extract face regions using the software package dlib. Align faces to multiple scales and randomly select one scale before smoothing them using a Gaussian blur with a kernel size of 5×5. The smoothed face is affine-warped back to the same size of the original face to simulate the DeepFake generation process. The region of interest is used as the input of the CNN model network. Since the goal is to reveal the artifacts between the pseudo face region and the surrounding area, the RoIs are selected as the rectangular region containing the face and the surrounding area; S32: Generate adversarial perturbations through the fast gradient sign method, setting the watermarked image to , and through the neural network model To classify images, in the fast gradient sign method, perturb It is generated by calculating the gradient of the image on the deep fake detector, the formula is as follows: (7) in: is the size of the disturbance; is the loss function; The deep fake detection model for the watermarked image is Output: is the true label, is the gradient with respect to the input image; After adding adversarial perturbations, adversarial samples are generated , that is, the image enhanced by disturbance: (8) Compute the image generated by adversarial perturbation Loss: (9) in: Denotes the deep fake detection model for the perturbation-enhanced watermark image, i.e., the adversarial sample The output score of It is the loss function that makes the model judge the image as a fake image.

[0013] As a further limitation of the technical solution, the specific steps of S4 are: Generate models using deepfakes For the enhanced watermark image Processing to obtain a forged image ; (10) The forged image is input into the watermark extraction model to extract the watermark; (11) in: It is a watermark extraction model; From the forged image Extract the watermark information.

[0014] As a further limitation of the present technical solution, the specific steps of S5 are: Use bit error rate to measure the extracted watermark information and original watermark information The difference between: (12) in: is the number of bits of watermark information; and They are the original watermark and the extracted watermark. Bit.

[0015] Compared with the prior art, the advantages and positive effects of the present invention are: The present invention proposes a robust watermark embedding and extraction method, which embeds imperceptible watermark information in an image to ensure that the watermark information can still maintain its recognizability even in the process of image forgery or subsequent image enhancement. The watermark has a high degree of robustness and can effectively resist various image processing operations, ensuring that effective traceability and tracking of forgery behavior can still be achieved at all stages of image generation. The present invention combines image enhancement technology to enhance the image embedded with the watermark. This enhancement process is intended to improve the visual quality of the image, and while maintaining the integrity of the image, cleverly avoid the risk of being misjudged as a forged image by the deep forgery detection system. Through this enhancement technology, the interference of the generated watermark embedding model on the original image can be effectively reduced, and the authenticity of the watermark image can be improved. The present invention further combines watermarking with blockchain technology, and uses the decentralized and tamper-proof characteristics of blockchain to encrypt and store watermark information. This not only ensures the security and tamper-proofness of watermark data, but also provides a transparent and credible evidence chain for the traceability of the image. Through blockchain technology, the responsibility for forged content can be traced, effectively preventing forgers from tampering with or forging original image data, thereby enhancing the credibility and verifiability of the entire defense system. This method not only embeds imperceptible watermarks to trace the source of forgery, but also combines image enhancement technology to reduce the risk of being identified by the forgery detection system. The watermark information is encrypted and stored through the blockchain to ensure that it cannot be tampered with and is highly secure, providing a reliable means of tracing the source of forged content. This method can effectively cope with the challenges brought by complex generation models, while ensuring the quality of video images, and providing a transparent and credible chain of evidence for the traceability and responsibility tracing of deep forgery content. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is the overall structure diagram of the present invention. DETAILED DESCRIPTION

[0017] A specific implementation of the present invention is described in detail below in conjunction with the accompanying drawings, but it should be understood that the protection scope of the present invention is not limited by the specific implementation.

[0018] The present invention embeds a 100-bit random binary bit watermark into the original face image, and then uses an image enhancement network to improve the concealment of the watermark to reduce the risk of being detected as a fake image. The enhanced image is sent to the deep fake model for face-changing operation. In the face-changing image, the watermark extraction network extracts the watermark information, which is then compared with the original watermark information stored in the blockchain to verify the authenticity of the image. The entire process combines deep learning, image processing and blockchain technology to achieve effective detection and traceability of deep fake content. The overall flow chart is as follows: Figure 1 As shown. The following steps are included: S1: Train a watermark embedding extraction network.

[0019] The specific steps of S1 are: S11: Setting a watermark embedding network model E with an encoder-decoder structure and a watermark extraction network model D with a decoder structure; S12: Watermark Embedding Network Model , where E is the watermark embedding network model, For a clean face image, is a binary watermark, To embed a watermarked face image, the model uses a randomly generated 100-bit binary watermark Embedded into clean face image In this paper, adversarial training is used to embed watermarked face images. Add slight disturbances, then input the deep fake model for malicious tampering, and extract the network model with watermarks , where D is the watermark extraction network model, For watermarked face images (including face images with only watermarks, images with slightly disturbed watermarks, and images with maliciously tampered watermarks), To extract the binary watermark, the model extracts the watermark information from the image. Both the watermark embedding network model and the watermark extraction network model are designed as convolutional neural networks. The training goal is to minimize the following two loss functions: binary cross entropy loss And mean square error loss : (1) in: and are the first Bit; n is the length of the watermark; (2) in: It is the image after watermarking through the watermark embedding network model; is the original clean face image; represents the L2 norm, i.e., the Euclidean distance; S13: The final training goal is to minimize the weighted sum of the two: (3) in: is a hyperparameter used to balance the weights of these two loss terms.

[0020] S2: Embed a 100-bit random binary bit watermark into the original face image through the watermark embedding model, and save the watermark information to the blockchain.

[0021] The specific steps of S2 are: The embedding process of watermark information (i.e., 100 binary bits) is implemented through an encoder-decoder structure.

[0022] S21: The watermark information is mapped to a suitable size through a fully connected layer and prepared and compared with the clean face image to integrate; S22: Dynamically adjust the embedding strength of the watermark based on the local complexity of the image to ensure that the effect of the watermark embedded in the image can adapt to changes in the image content; By calculating the input clean face image The local complexity of the image is used to quantify the characteristics of the image; Set a clean face image For a size of A tensor of , where: B is the batch size, C is the number of channels of the image, H and W are the height and width of the image; The local complexity of an image is calculated using the following formula: (4) in: Indicates the variance calculation in the height and width directions; after the calculation, we need to know which channel in which batch these variance values ​​belong to. In other words, each channel in each batch will have a calculated local complexity value; and are the indices of batch and channel respectively; The adaptive embedding weight is calculated based on the local complexity of the image and the embedding strength embedding_strength. The local complexity is scaled by a Sigmoid function so that higher complexity values ​​correspond to stronger embedding strength. (5) in: represents the adaptive weight of each channel at each image position; is the Sigmoid function; Watermark information According to the adaptive embedding weight, the watermark information It is a tensor of the same size as the input clean face image, representing the secret information to be embedded. The embedding strength is adjusted to an adaptive weight through the following formula: (6) S23: Use multiple convolutional layers and deconvolutional layers to extract and restore image features, and gradually restore the structural details of the image to embed the watermark information into the original image. In this stage, for each clean face image Assign a unique binary watermark Then, use the trained watermark to embed the network model Each clean face image Convert to watermark image and get the image embedded with watermark ; S24: The watermark information is stored in the blockchain through smart contracts.

[0023] S3: Train a deepfake detector and perform image augmentation on the watermarked image.

[0024] The specific steps of S3 are: S31: Select face images from the dataset as positive examples. Since the purpose here is to detect artifacts introduced by the affine face deformation step in deep fake production, the negative example generation process is simplified by directly simulating the affine face deformation step. The following steps are taken to generate negative samples to train the CNN model. Specifically, faces are detected in the original image and face regions are extracted using the software package dlib. Faces are aligned to multiple scales and one scale is randomly selected and then smoothed by Gaussian blur with a kernel size of 5×5 to better simulate the different resolutions introduced in affine deformed faces. The smoothed face is affine warped back to the same size of the original face to simulate the DeepFake generation process. The region of interest (RoI) is used as the input of the CNN model. Since the goal is to reveal the artifacts between the pseudo face region and the surrounding area, the RoIs are selected as the rectangular region containing the face and the surrounding area; Specifically, facial landmarks are used to determine RoIs, such as ; in: Represents the minimum bounding box that can cover all facial feature points except the cheek contour; variable is in the interval and Random values ​​are taken inside, where h and w are the height and width of the dataset image respectively; The regions of interest (RoIs) were resized to 224×224 for training the convolutional neural network (CNN) model. A CNN model ResNet 50 was trained using the training data. Then, the predictions of all RoIs were averaged as the final false probability. S32: Generate adversarial perturbations through the fast gradient sign method, setting the watermarked image to , and through the neural network model To classify images, in the fast gradient sign method, perturb It is generated by calculating the gradient of the image on the deep fake detector, the formula is as follows: (7) in: is the size of the disturbance; is the loss function; The deep fake detection model for the watermarked image is Output: is the true label, is the gradient with respect to the input image; After adding adversarial perturbations, adversarial samples are generated , that is, the image enhanced by disturbance: (8) Compute the image generated by adversarial perturbation Loss: (9) in: Denotes the deep fake detection model for the perturbation-enhanced watermark image, i.e., the adversarial sample The output score of It is the loss function that makes the model judge the image as a fake image.

[0025] We hope that this loss is as low as possible, As close to 0 (true image) as possible. Need to maximize The possibility of it being a real image is reduced, thereby reducing the probability that the deep fake detection model judges the image as a fake image.

[0026] S4: Deepforge the enhanced watermark image and then extract the watermark from the deepforged image.

[0027] The specific steps of S4 are: Generate models using deepfakes For the enhanced watermark image Processing to obtain a forged image ; (10) The forged image is input into the watermark extraction model to extract the watermark; (11) in: It is a watermark extraction model; From the forged image Extract the watermark information.

[0028] The specific steps of S5 are: Use bit error rate to measure the extracted watermark information and original watermark information The difference between: (12) in: is the number of bits of watermark information; and They are the original watermark and the extracted watermark. Bit.

[0029] The lower the bit error rate, the more accurate the extracted watermark is.

[0030] The present invention significantly improves the detection capability of deep fake content by embedding robust watermarks, combining image enhancement technology and blockchain encrypted storage. First, the embedding of robust watermarks ensures that the watermark remains identifiable even during image forgery or enhancement, thereby effectively tracing and tracking the forgery behavior. Secondly, the image embedded with the watermark is optimized through image enhancement technology, which avoids misjudgment as a forged image while maintaining visual quality, thereby ensuring the generation quality of the watermark embedding model. More importantly, the present invention combines blockchain technology with watermark information, utilizes the decentralization and immutability of blockchain, ensures the security of watermark information, and provides a transparent and credible chain of evidence for image tracing. This invention not only effectively improves the detection capability of deep fakes, but also increases the credibility and verifiability of the detection system.

[0031] The above disclosure is only a specific embodiment of the present invention, but the present invention is not limited thereto, and any changes that can be conceived by those skilled in the art should fall within the protection scope of the present invention.

Claims

1. A deep forgery active detection method based on blockchain and robust watermark enhancement, characterized in that: The following steps are involved: S1: Train a watermark embedding extraction network; S2: embed a 100-bit random binary bit watermark into the original face image through the watermark embedding model, and save the watermark information to the blockchain; S3: Train a deep fake detector and perform image enhancement on the watermarked image; S4: Deep forging the enhanced watermark image, and then extracting the watermark from the deep forged image; S5: Compare and verify the watermark extracted from the deep fake image with the watermark stored in the blockchain.

2. The deep forgery active detection method based on blockchain and robust watermark enhancement according to claim 1 is characterized by: The specific steps of S1 are: S11: Setting a watermark embedding network model E with an encoder-decoder structure and a watermark extraction network model D with a decoder structure; S12: Watermark Embedding Network Model , where E is the watermark embedding network model, For a clean face image, is a binary watermark, To embed a watermarked face image, the model uses a randomly generated 100-bit binary watermark Embedded into clean face image In this paper, adversarial training is used to embed watermarked face images. Add slight disturbances, then input the deep fake model for malicious tampering, and extract the network model with watermarks , where D is the watermark extraction network model, For a face image with embedded watermark, To extract the binary watermark, the model extracts the watermark information from the image. Both the watermark embedding network model and the watermark extraction network model are designed as convolutional neural networks. The training goal is to minimize the following two loss functions: binary cross entropy loss And mean square error loss : （1） in: and are the first Bit; n is the length of the watermark; （2） in: It is the image after watermarking through the watermark embedding network model; is the original clean face image; represents the L2 norm, i.e., the Euclidean distance; S13: The final training goal is to minimize the weighted sum of the two: （3） in: is a hyperparameter used to balance the weights of these two loss terms.

3. The deep forgery active detection method based on blockchain and robust watermark enhancement according to claim 2 is characterized by: The specific steps of S2 are: S21: The watermark information is mapped to a suitable size through a fully connected layer and prepared and compared with the clean face image to integrate; S22: Dynamically adjust the embedding strength of the watermark based on the local complexity of the image to ensure that the effect of the watermark embedded in the image can adapt to changes in the image content; By calculating the input clean face image The local complexity of the image is used to quantify the characteristics of the image; Set a clean face image For a size of A tensor of , where: B is the batch size, C is the number of channels of the image, H and W are the height and width of the image; The local complexity of an image is calculated using the following formula: （4） in: Indicates the variance calculation in the height and width directions; and are the indices of batch and channel respectively; The adaptive embedding weight is calculated based on the local complexity of the image and the embedding strength embedding_strength. The local complexity is scaled by a Sigmoid function so that higher complexity values ​​correspond to stronger embedding strength. （5） in: represents the adaptive weight of each channel at each image position; is the Sigmoid function; Watermark information According to the adaptive embedding weight, the watermark information It is a tensor of the same size as the input clean face image, representing the secret information to be embedded. The embedding strength is adjusted to an adaptive weight through the following formula: （6） S23: Use multiple convolutional layers and deconvolutional layers to extract and restore image features, and gradually restore the structural details of the image to embed the watermark information into the original image. In this stage, for each clean face image Assign a unique binary watermark Then, use the trained watermark to embed the network model Each clean face image Convert to watermark image and get the image embedded with watermark ; S24: The watermark information is stored in the blockchain through smart contracts.

4. The deep forgery active detection method based on blockchain and robust watermark enhancement according to claim 3 is characterized by: The specific steps of S3 are: S31: Select face images from the dataset as positive examples and take the following steps to generate negative samples to train the CNN model; Detect faces in the original image and extract face regions using the software package dlib. Align faces to multiple scales and randomly select one scale before smoothing them using a Gaussian blur with a kernel size of 5×5. The smoothed face is affine-warped back to the same size of the original face to simulate the DeepFake generation process. The region of interest is used as the input of the CNN model network. Since the goal is to reveal the artifacts between the pseudo face region and the surrounding area, the RoIs are selected as the rectangular region containing the face and the surrounding area; S32: Generate adversarial perturbations through the fast gradient sign method, setting the watermarked image to , and through the neural network model To classify images, in the fast gradient sign method, perturb It is generated by calculating the gradient of the image on the deep fake detector, the formula is as follows: （7） in: is the size of the disturbance; is the loss function; The deep fake detection model for the watermarked image is Output: is the true label, is the gradient with respect to the input image; After adding adversarial perturbations, adversarial samples are generated , that is, the image enhanced by disturbance: （8） Compute the image generated by adversarial perturbation Loss: （9） in: Denotes the deep fake detection model for the perturbation-enhanced watermark image, i.e., the adversarial sample The output score of It is the loss function that makes the model judge the image as a fake image.

5. The deep forgery active detection method based on blockchain and robust watermark enhancement according to claim 4 is characterized by: The specific steps of S4 are: Generate models using deepfakes For the enhanced watermark image Processing to obtain a forged image ; （10） The forged image is input into the watermark extraction model to extract the watermark; （11） in: Extract the model for watermark; From the forged image Extract the watermark information.

6. The deep forgery active detection method based on blockchain and robust watermark enhancement according to claim 4 is characterized by: The specific steps of S5 are: Use bit error rate to measure the extracted watermark information and original watermark information The difference between: （12） in: is the number of bits of watermark information; and They are the original watermark and the extracted watermark. Bit.

Images