A distorted image generation method based on perceptual perturbation
Through the distorted image generation method based on perceptual perturbation, the generative model is used to naturally add perturbation to the YCbCr color space, which solves the problem of unstable image reconstruction in the prior art, and effectively prevents the propagation of false information and improves vision Quality and protection success rate.
Patent Information
- Application Number
- CN202410727373.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-06
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2044-06-06
AI Technical Summary
The prior art method of eliminating disturbances through image reconstruction in preventing deep forgery technology is not robust, and it is difficult to effectively prevent the spread of false information.
Using a distorted image generation method based on perceptual perturbation, a generative model is constructed by acquiring the face image set, and an optimized image is generated using the extraction module, identity fusion module and reconstruction module. The specific steps include preprocessing, feature extraction, identity fusion and perturbation. Add to make sure the perturbation is naturally added in the YCbCr color space for visual effects.
The generated perturbation images can effectively prevent the spread of false information, improve visual quality and protection success rate, significantly improve PSNR and SSIM indicators, and enhance the ability to fight against deep forgery.
Smart Images

Figure CN118587754B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image-level deep forgery, and in particular to a method for generating distorted images based on perceptual perturbations. Background Art
[0002] Deep fake is a technology that uses deep learning and artificial intelligence to generate fake videos, audios or images. With the help of deep fake technology, those who spread false information with ulterior motives can be said to be "adding wings to a tiger". It can replace the facial expressions, voices or behaviors of people in existing videos, audios or images, making the faked content look very realistic and difficult to distinguish the authenticity. Deep fake technology has been applied to various scenarios, so it has also caused people to worry about the authenticity of information and digital identity authentication. Passive detection of malicious tampering in recent years is a post-event forensics strategy that has failed to prevent the spread of false information in advance. To address this limitation, some studies have destroyed deep fakes by adding adversarial perturbations to the original images. However, research on active defense by injecting adversarial perturbations is not robust. Summary of the invention
[0003] In order to overcome the deficiencies of the above technologies, the present invention provides a method for generating distorted images based on perceptual disturbances for the purpose of eliminating disturbances through image reconstruction by an adversary and preventing the spread of false information.
[0004] The technical solution adopted by the present invention to overcome the technical problems is:
[0005] A method for generating a distorted image based on perceptual disturbance comprises the following steps:
[0006] a) Obtain n face images and obtain a face image set I, where I = {I1, I2, ..., I i ,...,I n}, where I i is the i-th face image, i∈{1,...,n};
[0007] b) For the i-th face image I i Perform preprocessing to obtain the preprocessed face image I raw,i , the preprocessed face image set is I raw , I raw = {I raw,1 ,I raw,2 ,...,I raw,i ,...,I raw,n};
[0008] c) constructing a generation model G, which consists of an extraction module EIF, an identity fusion module IFM, and a reconstruction module RHI;
[0009] d) Take the i+1th preprocessed face image I raw,i+1 As the target image, it is input into the extraction module EIF of the generation model G, and the image feature F5 is output;
[0010] e) Input feature F5 into the identity fusion module IFM of the generative model G, and output feature F9;
[0011] f) The preprocessed face image I raw,i As the source image, the preprocessed face image I is calculated by ArcFace algorithm. raw,i The identity feature vector N1 is added element by element to the feature F9 to obtain the feature F 10 ;
[0012] g) The feature F 10 Input into the reconstruction module RHI of the generative model G, and output the image I swap,i ;
[0013] h) Image I is obtained by ArcFace algorithm swap,i The identity feature vector N2;
[0014] i) training the generation model G according to the identity feature vector N1 and the identity feature vector N2 to obtain the optimized generation model G;
[0015] j) The preprocessed face image I raw,i and the i+1th preprocessed face image I raw,i+1 Input into the optimized generative model G, and output the distorted image I swap_pert,i .
[0016] Furthermore, in step a), n face images are obtained from the CelebA-HQ dataset;
[0017] Step b) comprises the following steps:
[0018] b-1) Use the transforms.Resize() function in Python to resize the i-th face image I i Image resized to 256×256 I i ′;
[0019] b-2) Use the transforms.ToTensor() function in Python to transform the image I i ′ format is converted to a tensor type image I i ″;
[0020] b-3) Use the transforms.Normalize() function in Python to transform the image I i ″Perform normalization processing to obtain the preprocessed face image I raw,i .
[0021] Further, step d) comprises the following steps:
[0022] d-1) The extraction module EIF of the generative model G consists of RBCBlock block, single-head attention mechanism, first ConvBlock block, second ConvBlock block and third ConvBlock block;
[0023] d-2) The RBCBlock block of the extraction module EIF is composed of a mirror filling layer, a convolution layer, a batch normalization layer, and a LeakyReLU activation function. The i+1th preprocessed face image I raw,i+1 Input into the RBCBlock block and output the image feature F1;
[0024] d-3) Input the image feature F1 into the single-head attention mechanism of the extraction module EIF, and use the nn.MultiheadAttention function in the torch library to obtain the image feature F2;
[0025] d-4) The first ConvBlock of the extraction module EIF is composed of a convolutional layer, a batch normalization layer, and a LeakyReLU activation function in sequence. The image feature F2 is input into the first ConvBlock, and the image feature F3 is output;
[0026] d-5) The second ConvBlock block of the extraction module EIF is composed of a convolutional layer, a batch normalization layer, and a LeakyReLU activation function in sequence. The image feature F3 is input into the second ConvBlock block, and the image feature F4 is output;
[0027] d-6) The third ConvBlock block of the extraction module EIF is composed of a convolutional layer, a batch normalization layer, and a LeakyReLU activation function in sequence. The image feature F4 is input into the third ConvBlock block, and the image feature F5 is output.
[0028] Preferably, in step d-2), the padding of the mirror padding layer of the RBCBlock block is 3, the number of channels of the convolution layer of the RBCBlock block is 64, the convolution kernel is 7, and the padding is 0, and the number of channels of the batch normalization layer of the RBCBlock block is 64; in step d-3), the num_heads of the nn.MultiheadAttention function is set to 1; in step d-4), the convolution kernel size of the convolution layer of the first ConvBlock block is 3, the step size is 2, the padding is 2, and the number of channels is 128; in step d-5), the convolution kernel size of the convolution layer of the second ConvBlock block is 3, the step size is 2, the padding is 2, and the number of channels is 256; in step d-6), the convolution kernel size of the convolution layer of the third ConvBlock block is 3, the step size is 2, the padding is 2, and the number of channels is 512.
[0029] Further, step e) comprises the following steps:
[0030] e-1) generating an identity fusion module IFM of the model G, which is composed of a first CILRBlock block, a second CILRBlock block, a third CILRBlock block, and an RLBlock block;
[0031] e-2) The first CILRBlock block of the identity fusion module IFM is composed of a mirror padding layer, a convolutional layer, an instance normalization layer, a linear layer, and a LeakyReLU activation function in sequence. The image feature F5 is input into the first CILRBlock block, and the feature F6 is output;
[0032] e-3) The second CILRBlock block of the identity fusion module IFM is composed of a mirror padding layer, a convolution layer, an instance normalization layer, a linear layer, and a LeakyReLU activation function in sequence. The image feature F6 is input into the second CILRBlock block, and the feature F7 is output;
[0033] e-4) The third CILRBlock block of the identity fusion module IFM is composed of a mirror padding layer, a convolution layer, an instance normalization layer, a linear layer, and a LeakyReLU activation function in sequence. The image feature F7 is input into the third CILRBlock block, and the feature F8 is output;
[0034] e-5) The RLBlock block of the identity fusion module IFM is composed of a mirror padding layer, a convolutional layer, an instance normalization layer, and a linear layer in sequence. The feature F8 is input into the RLBlock block and the feature F9 is output.
[0035] Preferably, in step e-2), the padding of the mirror padding layer of the first CILRBlock block is 1, the convolution kernel size of the convolution layer of the first CILRBlock block is 3, the padding is 0, the number of channels is 512, and the output node of the linear layer of the first CILRBlock block is 512; in step e-3), the padding of the mirror padding layer of the second CILRBlock block is 1, the convolution kernel size of the convolution layer of the second CILRBlock block is 3, the padding is 0, the number of channels is 512, and the output node of the linear layer of the second CILRBlock block is The output node is 512; in step e-4), the padding of the mirror filling layer of the third CILRBlock block is 1, the convolution kernel size of the convolution layer of the third CILRBlock block is 3, the padding is 0, the number of channels is 512, and the output node of the linear layer of the third CILRBlock block is 512; in step e-5), the padding of the mirror filling layer of the RLBlock block is 1, the convolution kernel size of the convolution layer of the RLBlock block is 3, the padding is 0, the number of channels is 512, and the output node of the linear layer of the RLBlock block is 512.
[0036] Further, step g) comprises the following steps:
[0037] g-1) The reconstruction module RHI of the generated model G consists of the first UPBlock block, the second UPBlock block, the third UPBlock block, the single-head attention mechanism, and the RCTBlock block;
[0038] g-2) The first UPBlock of the reconstruction module RHI is composed of an upsampling layer, a convolutional layer, and a batch normalization layer. 10 Input into the first UPBlock block, and output the feature F 11 ;
[0039] g-3) The second UPBlock of the reconstruction module RHI is composed of an upsampling layer, a convolutional layer, and a batch normalization layer. 11 Input into the second UPBlock block, and output the feature F 12 ;
[0040] g-4) The third UPBlock of the reconstruction module RHI is composed of an upsampling layer, a convolutional layer, and a batch normalization layer. 12 Input into the third UPBlock block, and output the feature F 13 ;
[0041] g-5) The feature F 13 Input to the single-head attention mechanism of the reconstruction module RHI, and use the nn.MultiheadAttention function in the torch library to obtain the feature F 14 ;
[0042] g-6) The RCTBlock block of the reconstruction module RHI is composed of a mirror padding layer, a convolutional layer, and a Tanh activation function. 14 Input into the RCTBlock block and output the feature F 15 ;
[0043] g-7) The feature F 15 Add 1 to each element and divide by 2 to get image I swap,i .
[0044] Preferably, in step g-2), the sampling multiple of the upsampling layer of the first UPBlock block is 2, the upsampling method is to use a bilinear interpolation method, the convolution kernel size of the convolution layer of the first UPBlock block is 3, the step size is 1, the padding is 1, the number of channels is 256, and the number of channels of the batch normalization layer of the first UPBlock block is 256; in step g-3), the sampling multiple of the upsampling layer of the second UPBlock block is 2, the upsampling method is to use a bilinear interpolation method, the convolution kernel size of the convolution layer of the second UPBlock block is 3, the step size is 1, the padding is 1, the number of channels is 128, and the number of channels of the batch normalization layer of the second UPBlock block is 128. The number is 256; in step g-4), the sampling multiple of the upsampling layer of the third UPBlock block is 2, the upsampling method is to use the bilinear interpolation method, the convolution kernel size of the convolution layer of the third UPBlock block is 3, the step size is 1, the padding is 1, the number of channels is 64, and the number of channels of the batch normalization layer of the third UPBlock block is 256; in step g-5), the num_heads of the nn.MultiheadAttention function is set to 1; in step g-6), the padding of the mirror padding layer of the RCTBlock block is 3, the convolution kernel size of the convolution layer of the RCTBlock block is 7, the padding is 0, and the number of channels is 3.
[0045] Further, step i) comprises the following steps:
[0046] i-1) by formula The loss function L is calculated, where torch.sum(·) is the sum function in the torch library, and the dim value of the sum function is set to 1. torch.norm(·) is the norm function in the torch library, and the dim value of the norm function is set to 1. The Adam optimizer is used to train the generative model G using the loss function L to obtain the optimized generative model G.
[0047] Further, step j) comprises the following steps:
[0048] j-1) Use the PDG algorithm to calculate the preprocessed face image I raw,iThe disturbance P i , the disturbance P i The shape and preprocessed face image I raw,i The shape of the perturbation P i The value of each element in is greater than or equal to -0.05 and less than or equal to 0.05;
[0049] j-2) Preprocessed face image I raw,i Perform the denormalization operation to obtain image I raw,i ′; j-3) image I raw,i 'Convert to YCbCr color space through cv2.cvtColor() function in Opencv library, and convert image I to YCbCr color space raw,i Each pixel value on the Cb channel and Cr channel of ′ is consistent with the perturbation P i Add the corresponding elements in to get the perturbed image I in the YCbCr color space YCbCr,i ;
[0050] j-4) Use the cv2.cvtColor() function in the Opencv library to perturb the image I YCbCr,i Convert from YCbCr color space to RGB color space to obtain the perturbed image I in RGB color space RGB,i ;
[0051] j-5) Use the transforms.Normalize() function in Python to perturb the image I RGB,i Perform normalization to obtain the perturbation image I pert,i ;
[0052] j-6) Take the i+1th pre-processed face image I raw,i+1 Replace the pre-processed face image I in step j-1) raw,i Then repeat steps j-1) to j-5) to obtain the disturbed image I pert,i+1 ;
[0053] j-7) The perturbed image I pert,i As the source image, the perturbation image I pert,i+1 The target image is input into the optimized generative model G, and the distorted image I is output. swap_pert,i .
[0054] The beneficial effect of the present invention is that in order to solve the problem of the adversary eliminating disturbances through image reconstruction, in the process of generating a disturbed image, the original image is first converted to the YCbCr color space, disturbances are added to the Cb and Cr channels of the YCbCr color space, and then converted to the RGB color space to obtain a visually natural disturbed image. At this time, the adversary obtains the disturbed image and sends it to the generation model to output an image with distorted content, thereby preventing the spread of false information. BRIEF DESCRIPTION OF THE DRAWINGS
[0055] Figure 1 Flow chart of the method of the present invention
[0056] Figure 2 It is a structural diagram of the generation model G of the present invention. DETAILED DESCRIPTION
[0057] The following is combined with Figure 1 , Attachment Figure 2 The present invention is further described.
[0058] A method for generating a distorted image based on perceptual disturbance comprises the following steps:
[0059] a) Obtain n face images and obtain a face image set I, where I = {I1, I2, ..., I i ,...,I n}, where I i is the i-th face image, i∈{1,...,n}.
[0060] b) For the i-th face image I i Perform preprocessing to obtain the preprocessed face image I raw,i , the preprocessed face image set is I raw , I raw = {I raw,1 ,I raw,2 ,...,I raw,i ,...,I raw,n}.
[0061] c) Construct a generation model G, which consists of an extraction module EIF, an identity fusion module IFM, and a reconstruction module RHI.
[0062] d) Take the i+1th preprocessed face image I raw,i+1 The target image is input into the extraction module EIF of the generation model G, and the image feature F5 is output.
[0063] e) Input feature F5 into the identity fusion module IFM of the generative model G, and output feature F9.
[0064] f) The preprocessed face image Iraw,i As the source image, the preprocessed face image I is calculated by ArcFace algorithm. raw,i The identity feature vector N1 is added element by element to the feature F9 to obtain the feature F 10 .
[0065] g) The feature F 10 Input into the reconstruction module RHI of the generative model G, and output the image I swap,i .
[0066] h) Image I is obtained by ArcFace algorithm swap,i The identity feature vector N2.
[0067] i) The generation model G is trained according to the identity feature vector N1 and the identity feature vector N2 to obtain the optimized generation model G.
[0068] j) The preprocessed face image I raw,i and the i+1th preprocessed face image I raw,i+1 Input into the optimized generative model G, and output the distorted image I swap_pert,i . Distorted image I swap_pert,i The visual effect is very blurry and is different from the image I. swap,i There is a big visual difference, which achieves the purpose of preventing the spread of false information.
[0069] In order to solve the problem of the adversary eliminating disturbances through image reconstruction, the present invention first converts the original image to the YCbCr color space, adds disturbances to the Cb and Cr channels of the YCbCr color space, and then converts it to the RGB color space to obtain a visually natural disturbed image. The adversary then obtains the disturbed image and sends it to the generation model to output an image with distorted content, thereby preventing the spread of false information.
[0070] The experiment uses the CelebA-HQ face image dataset. In order to verify the reliability of this patented method, we conducted a comparative experiment and displayed the experimental results.
[0071] We selected four models with publicly available source code: CMUA, AntiForgery, Disrupting Deepfakes, and initiative-defense-for-deepfake for comparative testing.
[0072] The bold data in the following table are the best results.
[0073] The experimental results are shown in Table 1:
[0074] Table 1 Quantitative visual quality evaluation of perturbed images on the CelebA-HQ dataset
[0075]
[0076] Table 1 shows the quantitative visual quality evaluation of the perturbed images on the CelebA-HQ dataset. The average peak signal-to-noise ratio (PSNR) and the structural similarity index (SSIM) between the original and perturbed images are used to evaluate the visual quality of the perturbed images. As shown in Table 1, the PSNR and SSIM of our proposed method achieve the best visual quality. Our PSNR and SSIM are improved by 0.72 and 0.099 respectively compared with CMUA, 3.74 and 0.003 respectively compared with AntiForgery, 3.83 and 0.207 respectively compared with Disrupting Deepfake, and 5.41 and 0.177 respectively compared with initiative-defense-for-deepfake. ASR represents the success rate of image protection. The higher the ASR value, the more images are successfully protected, and the more blurred the image will be after face swapping. Our ASR is improved by 35%, 15%, 35%, and 23% compared to CMUA, AntiForgery, DisruptingDeepfakes, and initiative-defense-for-deepfake, respectively.
[0077] In one embodiment of the present invention, in step a), n face images are obtained from the CelebA-HQ dataset. The CelebA-HQ dataset consists of 30,000 face images, including 6,217 different face identities, and the resolution of each image is 1024×1024.
[0078] Step b) comprises the following steps:
[0079] b-1) Use the transforms.Resize() function in Python to resize the i-th face image I i Image resized to 256×256 I i ′.
[0080] b-2) Use the transforms.ToTensor() function in Python to transform the image I i ′ format is converted to a tensor type image I i ″.
[0081] b-3) Use the transforms.Normalize() function in Python to transform the image I i ″Perform normalization processing to obtain the preprocessed face image Iraw,i .
[0082] In one embodiment of the present invention, step d) comprises the following steps:
[0083] d-1) The extraction module EIF of the generative model G is composed of RBCBlock block, single-head attention mechanism, first ConvBlock block, second ConvBlock block and third ConvBlock block.
[0084] d-2) The RBCBlock block of the extraction module EIF is composed of a mirror filling layer, a convolution layer, a batch normalization layer, and a LeakyReLU activation function. The i+1th preprocessed face image I raw,i+1 Input into the RBCBlock block and output the image feature F1.
[0085] d-3) Input the image feature F1 into the single-head attention mechanism of the extraction module EIF, and use the nn.MultiheadAttention function in the torch library to obtain the image feature F2.
[0086] d-4) The first ConvBlock block of the extraction module EIF is composed of a convolutional layer, a batch normalization layer, and a LeakyReLU activation function in sequence. The image feature F2 is input into the first ConvBlock block, and the image feature F3 is output.
[0087] d-5) The second ConvBlock block of the extraction module EIF is composed of a convolutional layer, a batch normalization layer, and a LeakyReLU activation function in sequence. The image feature F3 is input into the second ConvBlock block, and the image feature F4 is output.
[0088] d-6) The third ConvBlock block of the extraction module EIF is composed of a convolutional layer, a batch normalization layer, and a LeakyReLU activation function in sequence. The image feature F4 is input into the third ConvBlock block, and the image feature F5 is output.
[0089] In this embodiment, in step d-2), the padding of the mirror padding layer of the RBCBlock block is 3, the number of channels of the convolution layer of the RBCBlock block is 64, the convolution kernel is 7, and the padding is 0, and the number of channels of the batch normalization layer of the RBCBlock block is 64; in step d-3), the num_heads of the nn.MultiheadAttention function is set to 1; in step d-4), the convolution kernel size of the convolution layer of the first ConvBlock block is 3, the step size is 2, the padding is 2, and the number of channels is 128; in step d-5), the convolution kernel size of the convolution layer of the second ConvBlock block is 3, the step size is 2, the padding is 2, and the number of channels is 256; in step d-6), the convolution kernel size of the convolution layer of the third ConvBlock block is 3, the step size is 2, the padding is 2, and the number of channels is 512.
[0090] In one embodiment of the present invention, step e) comprises the following steps:
[0091] e-1) The identity fusion module IFM of the generated model G is composed of a first CILRBlock block, a second CILRBlock block, a third CILRBlock block, and an RLBlock block.
[0092] e-2) The first CILRBlock block of the identity fusion module IFM is composed of a mirror padding layer, a convolutional layer, an instance normalization layer, a linear layer, and a LeakyReLU activation function in sequence. The image feature F5 is input into the first CILRBlock block and the feature F6 is output.
[0093] e-3) The second CILRBlock block of the identity fusion module IFM is composed of a mirror padding layer, a convolutional layer, an instance normalization layer, a linear layer, and a LeakyReLU activation function in sequence. The image feature F6 is input into the second CILRBlock block and the feature F7 is output.
[0094] e-4) The third CILRBlock block of the identity fusion module IFM is composed of a mirror padding layer, a convolutional layer, an instance normalization layer, a linear layer, and a LeakyReLU activation function in sequence. The image feature F7 is input into the third CILRBlock block and the feature F8 is output.
[0095] e-5) The RLBlock block of the identity fusion module IFM is composed of a mirror padding layer, a convolutional layer, an instance normalization layer, and a linear layer in sequence. The feature F8 is input into the RLBlock block and the feature F9 is output.
[0096] In this embodiment, in step e-2), the padding of the mirror padding layer of the first CILRBlock block is 1, the convolution kernel size of the convolution layer of the first CILRBlock block is 3, the padding is 0, the number of channels is 512, and the output node of the linear layer of the first CILRBlock block is 512; in step e-3), the padding of the mirror padding layer of the second CILRBlock block is 1, the convolution kernel size of the convolution layer of the second CILRBlock block is 3, the padding is 0, the number of channels is 512, and the linear layer of the second CILRBlock block is The output node of the third CILRBlock block is 512; in step e-4), the padding of the mirror padding layer of the third CILRBlock block is 1, the convolution kernel size of the convolution layer of the third CILRBlock block is 3, the padding is 0, the number of channels is 512, and the output node of the linear layer of the third CILRBlock block is 512; in step e-5), the padding of the mirror padding layer of the RLBlock block is 1, the convolution kernel size of the convolution layer of the RLBlock block is 3, the padding is 0, the number of channels is 512, and the output node of the linear layer of the RLBlock block is 512. In one embodiment of the present invention, step g) comprises the following steps:
[0097] g-1) The reconstruction module RHI of the generated model G is composed of the first UPBlock block, the second UPBlock block, the third UPBlock block, the single-head attention mechanism, and the RCTBlock block.
[0098] g-2) The first UPBlock of the reconstruction module RHI is composed of an upsampling layer, a convolutional layer, and a batch normalization layer. 10 Input into the first UPBlock block, and output the feature F 11 .
[0099] g-3) The second UPBlock of the reconstruction module RHI is composed of an upsampling layer, a convolutional layer, and a batch normalization layer. 11 Input into the second UPBlock block, and output the feature F 12 .
[0100] g-4) The third UPBlock of the reconstruction module RHI is composed of an upsampling layer, a convolutional layer, and a batch normalization layer. 12 Input into the third UPBlock block, and output the feature F 13 .
[0101] g-5) The feature F 13 Input to the single-head attention mechanism of the reconstruction module RHI, and use the nn.MultiheadAttention function in the torch library to obtain the feature F 14 .
[0102] g-6) The RCTBlock block of the reconstruction module RHI is composed of a mirror padding layer, a convolutional layer, and a Tanh activation function. 14 Input into the RCTBlock block and output the feature F 15 .
[0103] g-7) The feature F 15 Add 1 to each element and divide by 2 to get image I swap,i .
[0104] In this embodiment, in step g-2), the sampling multiple of the upsampling layer of the first UPBlock block is 2, the upsampling method is to use a bilinear interpolation method, the convolution kernel size of the convolution layer of the first UPBlock block is 3, the step size is 1, the padding is 1, the number of channels is 256, and the number of channels of the batch normalization layer of the first UPBlock block is 256; in step g-3), the sampling multiple of the upsampling layer of the second UPBlock block is 2, the upsampling method is to use a bilinear interpolation method, the convolution kernel size of the convolution layer of the second UPBlock block is 3, the step size is 1, the padding is 1, the number of channels is 128, and the number of channels of the batch normalization layer of the second UPBlock block is 256. The number of channels is 256; in step g-4), the sampling multiple of the upsampling layer of the third UPBlock block is 2, and the upsampling method is to use the bilinear interpolation method. The convolution kernel size of the convolution layer of the third UPBlock block is 3, the step size is 1, the padding is 1, and the number of channels is 64. The number of channels of the batch normalization layer of the third UPBlock block is 256; in step g-5), the num_heads of the nn.MultiheadAttention function is set to 1; in step g-6), the padding of the mirror padding layer of the RCTBlock block is 3, and the convolution kernel size of the convolution layer of the RCTBlock block is 7, the padding is 0, and the number of channels is 3.
[0105] In one embodiment of the present invention, step i) comprises the following steps:
[0106] i-1) by formula The loss function L is calculated, where torch.sum(·) is the sum function in the torch library, and the dim value of the sum function is set to 1. torch.norm(·) is the norm function in the torch library, and the dim value of the norm function is set to 1. The Adam optimizer is used to train the generative model G using the loss function L to obtain the optimized generative model G.
[0107] In one embodiment of the present invention, step j) comprises the following steps:
[0108] j-1) Use the PDG algorithm to calculate the preprocessed face image I raw,i The disturbance Pi , the disturbance P i The shape and preprocessed face image I raw,i The shape of the perturbation P i The value of each element in is greater than or equal to -0.05 and less than or equal to 0.05.
[0109] j-2) Preprocessed face image I raw,i Perform the denormalization operation to obtain image I raw,i ′.
[0110] j-3) Image I raw,i 'Convert to YCbCr color space through cv2.cvtColor() function in Opencv library, and convert image I to YCbCr color space raw,i Each pixel value on the Cb channel and Cr channel of ′ is consistent with the perturbation P i Add the corresponding elements in to get the perturbed image I in the YCbCr color space YCbCr,i .
[0111] j-4) Use the cv2.cvtColor() function in the Opencv library to perturb the image I YCbCr,i Convert from YCbCr color space to RGB color space to obtain the perturbed image I in RGB color space RGB,i .
[0112] j-5) Use the transforms.Normalize() function in Python to perturb the image I RGB,i Perform normalization to obtain the perturbation image I pert,i .
[0113] j-6) Take the i+1th pre-processed face image I raw,i+1 Replace the pre-processed face image I in step j-1) raw,i Then repeat steps j-1) to j-5) to obtain the disturbed image I pert,i+1 .
[0114] j-7) The perturbed image I pert,i As the source image, the perturbation image I pert,i+1 The target image is input into the optimized generative model G, and the distorted image I is output. swap_pert,i .
[0115] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A method for generating distorted images based on perceptual disturbance, characterized in that: The steps include: a) Obtain n face images and obtain a face image set I, where I = {I1, I2, ..., I i ,...,I n }, where I i is the i-th face image, i∈{1,...,n}; b) For the i-th face image I i Perform preprocessing to obtain the preprocessed face image I raw,i , the preprocessed face image set is I raw , I raw = {I raw,1 ,I raw,2 ,...,I raw,i ,...,I raw,n }; c) constructing a generation model G, which consists of an extraction module EIF, an identity fusion module IFM, and a reconstruction module RHI; d) Take the i+1th preprocessed face image I raw,i+1 As the target image, it is input into the extraction module EIF of the generation model G, and the image feature F5 is output; e) Input feature F5 into the identity fusion module IFM of the generative model G, and output feature F9; f) The preprocessed face image I raw,i As the source image, the preprocessed face image I is calculated by the ArcFace algorithm. raw,i The identity feature vector N1 is added element by element to the feature F9 to obtain the feature F 10 ; g) The feature F 10 Input into the reconstruction module RHI of the generative model G, and output the image I swap,i ; h) Image I is obtained by ArcFace algorithm swap,i The identity feature vector N2; i) training the generation model G according to the identity feature vector N1 and the identity feature vector N2 to obtain the optimized generation model G; j) The preprocessed face image I raw,i and the i+1th preprocessed face image I raw,i+1 Input into the optimized generative model G, and output the distorted image I swap_pert,i ; Step j) comprises the following steps: j-1) Use the PDG algorithm to calculate the preprocessed face image I raw,i The disturbance P i , the disturbance P i The shape and preprocessed face image I raw,i The shape of the perturbation P i The value of each element in is greater than or equal to -0.05 and less than or equal to 0.05; j-2) Preprocessed face image I raw,i Perform the denormalization operation to obtain image I raw,i ′; j-3) Image I raw,i 'Convert to YCbCr color space through cv2.cvtColor() function in Opencv library, and convert image I to YCbCr color space raw,i Each pixel value on the Cb channel and Cr channel of ′ is consistent with the perturbation P i Add the corresponding elements in to get the perturbed image I in the YCbCr color space YCbCr,i ; j-4) Use the cv2.cvtColor() function in the Opencv library to perturb the image I YCbCr,i Convert from YCbCr color space to RGB color space to obtain the perturbed image I in RGB color space RGB,i ; j-5) Use the transforms.Normalize() function in Python to perturb the image I RGB,i Perform normalization to obtain the perturbation image I pert,i ; j-6) Take the i+1th pre-processed face image I raw,i+1 Replace the pre-processed face image I in step j-1) raw,i Then repeat steps j-1) to j-5) to obtain the disturbed image I pert,i+1 ; j-7) The perturbed image I pert,i As the source image, the perturbation image I pert,i+1 The target image is input into the optimized generative model G, and the distorted image I is output. swap_pert,i .
2. The method for generating distorted images based on perceptual disturbance according to claim 1, characterized in that: In step a), n face images are obtained from the CelebA-HQ dataset; Step b) comprises the following steps: b-1) Use the transforms.Resize() function in Python to resize the i-th face image I i Image resized to 256×256 I i ′; b-2) Use the transforms.ToTensor() function in Python to transform the image I i ′ format is converted to a tensor type image I i ″; b-3) Use the transforms.Normalize() function in Python to transform the image I i ″Perform normalization processing to obtain the preprocessed face image I raw,i .
3. The method for generating distorted images based on perceptual disturbance according to claim 1, characterized in that: Step d) comprises the following steps: d-1) The extraction module EIF of the generative model G consists of RBCBlock block, single-head attention mechanism, first ConvBlock block, second ConvBlock block and third ConvBlock block; d-2) The RBCBlock block of the extraction module EIF is composed of a mirror filling layer, a convolution layer, a batch normalization layer, and a LeakyReLU activation function. The i+1th preprocessed face image I raw,i+1 Input into the RBCBlock block and output the image feature F1; d-3) Input the image feature F1 into the single-head attention mechanism of the extraction module EIF, and use the nn.MultiheadAttention function in the torch library to obtain the image feature F2; d-4) The first ConvBlock of the extraction module EIF is composed of a convolutional layer, a batch normalization layer, and a LeakyReLU activation function in sequence. The image feature F2 is input into the first ConvBlock, and the image feature F3 is output; d-5) The second ConvBlock block of the extraction module EIF is composed of a convolutional layer, a batch normalization layer, and a LeakyReLU activation function in sequence. The image feature F3 is input into the second ConvBlock block, and the image feature F4 is output; d-6) The third ConvBlock block of the extraction module EIF is composed of a convolutional layer, a batch normalization layer, and a LeakyReLU activation function in sequence. The image feature F4 is input into the third ConvBlock block, and the image feature F5 is output.
4. The method for generating distorted images based on perceptual disturbance according to claim 3, characterized in that: In step d-2), the padding of the mirror padding layer of the RBCBlock block is 3, the number of channels of the convolution layer of the RBCBlock block is 64, the convolution kernel is 7, and the padding is 0, and the number of channels of the batch normalization layer of the RBCBlock block is 64; in step d-3), the num_heads of the nn.MultiheadAttention function is set to 1; in step d-4), the convolution kernel size of the convolution layer of the first ConvBlock block is 3, the stride is 2, the padding is 2, and the number of channels is 128; in step d-5), the convolution kernel size of the convolution layer of the second ConvBlock block is 3, the stride is 2, the padding is 2, and the number of channels is 256; in step d-6), the convolution kernel size of the convolution layer of the third ConvBlock block is 3, the stride is 2, the padding is 2, and the number of channels is 512.
5. The method for generating distorted images based on perceptual disturbance according to claim 1, characterized in that: Step e) comprises the following steps: e-1) generating an identity fusion module IFM of the model G, which is composed of a first CILRBlock block, a second CILRBlock block, a third CILRBlock block, and an RLBlock block; e-2) The first CILRBlock block of the identity fusion module IFM is composed of a mirror padding layer, a convolutional layer, an instance normalization layer, a linear layer, and a LeakyReLU activation function in sequence. The image feature F5 is input into the first CILRBlock block, and the feature F6 is output; e-3) The second CILRBlock block of the identity fusion module IFM is composed of a mirror padding layer, a convolution layer, an instance normalization layer, a linear layer, and a LeakyReLU activation function in sequence. The image feature F6 is input into the second CILRBlock block, and the feature F7 is output; e-4) The third CILRBlock block of the identity fusion module IFM is composed of a mirror padding layer, a convolution layer, an instance normalization layer, a linear layer, and a LeakyReLU activation function in sequence. The image feature F7 is input into the third CILRBlock block, and the feature F8 is output; e-5) The RLBlock block of the identity fusion module IFM is composed of a mirror padding layer, a convolutional layer, an instance normalization layer, and a linear layer in sequence. The feature F8 is input into the RLBlock block and the feature F9 is output.
6. The method for generating distorted images based on perceptual disturbance according to claim 5, characterized in that: In step e-2), the padding of the mirror padding layer of the first CILRBlock block is 1, the convolution kernel size of the convolution layer of the first CILRBlock block is 3, the padding is 0, the number of channels is 512, and the output node of the linear layer of the first CILRBlock block is 512; in step e-3), the padding of the mirror padding layer of the second CILRBlock block is 1, the convolution kernel size of the convolution layer of the second CILRBlock block is 3, the padding is 0, the number of channels is 512, and the output node of the linear layer of the second CILRBlock block is The number of nodes is 512; in step e-4), the padding of the mirror filling layer of the third CILRBlock block is 1, the convolution kernel size of the convolution layer of the third CILRBlock block is 3, the padding is 0, the number of channels is 512, and the output node of the linear layer of the third CILRBlock block is 512; in step e-5), the padding of the mirror filling layer of the RLBlock block is 1, the convolution kernel size of the convolution layer of the RLBlock block is 3, the padding is 0, the number of channels is 512, and the output node of the linear layer of the RLBlock block is 512.
7. The method for generating distorted images based on perceptual disturbance according to claim 1, characterized in that: Step g) comprises the following steps: g-1) The reconstruction module RHI of the generated model G consists of the first UPBlock block, the second UPBlock block, the third UPBlock block, the single-head attention mechanism, and the RCTBlock block; g-2) The first UPBlock of the reconstruction module RHI is composed of an upsampling layer, a convolutional layer, and a batch normalization layer. 10 Input into the first UPBlock block, and output the feature F 11 ; g-3) The second UPBlock of the reconstruction module RHI is composed of an upsampling layer, a convolutional layer, and a batch normalization layer. 11 Input into the second UPBlock block, and output the feature F 12 ; g-4) The third UPBlock of the reconstruction module RHI is composed of an upsampling layer, a convolutional layer, and a batch normalization layer. 12 Input into the third UPBlock block, and output the feature F 13 ; g-5) The feature F 13 Input to the single-head attention mechanism of the reconstruction module RHI, and use the nn.MultiheadAttention function in the torch library to obtain the feature F 14 ; g-6) The RCTBlock block of the reconstruction module RHI is composed of a mirror padding layer, a convolutional layer, and a Tanh activation function. 14 Input into the RCTBlock block and output the feature F 15 ; g-7) The feature F 15 Add 1 to each element and divide by 2 to get image I swap,i .
8. The method for generating distorted images based on perceptual disturbance according to claim 7, characterized in that: In step g-2), the sampling multiple of the upsampling layer of the first UPBlock block is 2, the upsampling method is to use the bilinear interpolation method, the convolution kernel size of the convolution layer of the first UPBlock block is 3, the step size is 1, the padding is 1, the number of channels is 256, and the number of channels of the batch normalization layer of the first UPBlock block is 256; in step g-3), the sampling multiple of the upsampling layer of the second UPBlock block is 2, the upsampling method is to use the bilinear interpolation method, the convolution kernel size of the convolution layer of the second UPBlock block is 3, the step size is 1, the padding is 1, the number of channels is 128, and the number of channels of the batch normalization layer of the second UPBlock block is 256; in step g-4), the sampling multiple of the upsampling layer of the third UPBlock block is 2, the upsampling method is to use the bilinear interpolation method, the convolution kernel size of the convolution layer of the third UPBlock block is 3, the step size is 1, the padding is 1, the number of channels is 64, and the number of channels of the batch normalization layer of the third UPBlock block is 256; in step g-5), the num_heads of the nn.MultiheadAttention function is set to 1; in step g-6), the padding of the mirror padding layer of the RCTBlock block is 3, the convolution kernel size of the convolution layer of the RCTBlock block is 7, the padding is 0, and the number of channels is 3.
9. The method for generating distorted images based on perceptual disturbance according to claim 1, characterized in that: Step i) comprises the following steps: i-1) by formula The loss function L is calculated, where torch.sum(·) is the sum function in the torch library, and the dim value of the sum function is set to 1. torch.norm(·) is the norm function in the torch library, and the dim value of the norm function is set to 1. The Adam optimizer is used to train the generative model G using the loss function L to obtain the optimized generative model G.
