A face image restoration method and system based on mask autoencoder

Abstract

The present invention discloses a facial image restoration method and system based on a mask autoencoder, which belongs to the technical field of deep fake detection. The method comprises: using a RetinaFace model to crop a forged facial image to obtain a source facial image and a target facial image corresponding to the forged facial image; extracting features of the source facial image, upsampling high-level semantic features and then splicing them with shading features, and obtaining a segmentation probability map based on the spliced feature map; multiplying the segmentation probability map with the source facial image to obtain source facial information, restoring and reconstructing the source face based on the source facial information to obtain a reconstructed source face; dividing the segmentation probability map into blocks and calculating the mean of each segmentation probability map, calculating a weight based on the mean, and performing a mask based on the weight to obtain a masked image; inputting potential target facial features in the reconstructed source face and the masked image into a masked autoencoder to obtain a facial image restoration image of the forged facial image.

Description

Technical Field

[0001] The present invention relates to the technical field of deep fake detection, and in particular to a facial image restoration method and system based on a mask autoencoder. Background Art

[0002] While the development of artificial intelligence technology brings convenience to people, it also poses potential threats. Deepfake technology can achieve realistic face replacements. The popularity of short video platforms has exacerbated the impact of deepfake content on society, enabling the rapid spread of false information and posing a significant threat to individuals and society. There is an urgent need to develop effective deepfake defense and detection algorithms to technically control and manage current deepfake data.

[0003] Current research primarily approaches deepfake detection as a binary classification task, aiming to improve generalization performance across various types of forged data. Existing detection algorithms can be broadly categorized into data-driven and model-optimization-based approaches. Data-driven approaches simulate the face-swapping process to generate diverse forged data to improve generalization performance. Model-optimization-based approaches, on the other hand, optimize model structure and employ methods such as feature consistency analysis and temporal continuity analysis to detect deepfakes. While these algorithms achieve high detection accuracy, they can only provide classification or localization results and cannot provide a chain of evidence to match the detection results. Therefore, improving the interpretability of detection algorithms is a pressing issue. Summary of the Invention

[0004] In order to solve the above technical problems, the present invention provides a face image restoration method based on a mask autoencoder, the method comprising:

[0005] Step S1: collecting a forged face image, cropping the forged face image using a RetinaFace model, and obtaining a source face image and a target face image corresponding to the forged face image;

[0006] Step S2: extracting shading features and high-level semantic features of the source face image, upsampling the high-level semantic features and then splicing them with the shading features, and obtaining a segmentation probability map based on the spliced feature map;

[0007] Step S3: multiplying the segmentation probability map by the source face image to obtain source face information, and restoring and reconstructing the source face based on the source face information to obtain a reconstructed source face;

[0008] Step S4: Divide the segmentation probability map into blocks and calculate the mean of each segmentation probability map, assign weights based on the mean results and calculate the weights, perform masking based on the weights, and obtain a masked image;

[0009] Step S5: input the potential target facial features in the reconstructed source face and the mask image into the mask autoencoder to obtain a facial image restoration image of the forged facial image.

[0010] Optionally, in step S2, extracting shading features and high-level semantic features of the source face image, upsampling the high-level semantic features and then splicing them with the shading features, and obtaining a segmentation probability map based on the spliced feature map specifically includes:

[0011] Use EfficientNet-B0 to extract low-level texture features and high-level semantic features;

[0012] The high-level semantic features are upsampled using transposed convolution, the upsampled high-level semantic features are spliced with the underlying texture features, and the spliced features are input into a transposed convolution layer to generate a segmentation probability map.

[0013] Optionally, in step S3, the process of multiplying the segmentation probability map by the source face image to obtain source face information, and restoring and reconstructing the source face based on the source face information to obtain the reconstructed source face specifically includes:

[0014] S31, based on the source face information obtained by multiplying the segmentation probability map with the source face image, down-sampling the source face information through four convolution-pooling layers to extract a feature map of high-level semantic features;

[0015] S32, dividing the feature map of the high-level semantic features into source face features and potential target face features, upsampling the source face features using transposed convolution, and convolving the potential target face features using a convolution layer to obtain a feature map with a preset number of channels;

[0016] S33, using residual connection to splice the feature map of the high-level semantic feature with the feature map of the preset number of channels, and using convolution to restore the number of channels of the spliced feature map;

[0017] S34. Repeat S31-S33 4 times on the feature map of the restored channel number to restore the source face image, calculate the identity loss and the perception loss based on the source face image and the source face image in step S1, and obtain the reconstructed source face.

[0018] Optionally, in step S4, dividing the segmentation probability map into blocks and calculating the mean of each segmentation probability map, assigning weights based on the mean results and calculating the weights, and performing masking based on the weights to obtain the masked image specifically includes:

[0019] The mask block sequence number is calculated based on the segmentation probability map, the forged face is divided into blocks based on the mask block sequence number, and the forged face after the block division is masked using the mask block sequence number to obtain a masked image.

[0020] Optionally, in step S5, inputting the potential target facial features in the reconstructed source face and the mask image into the mask autoencoder to obtain a restored image of the target facial image of the forged facial image specifically includes:

[0021] Inputting the potential target facial features in the reconstructed source face and the uncovered image blocks in the masked image into a masked autoencoder for flattening and merging into a tensor, inputting the tensor into a fully connected layer to obtain a tensor image feature vector;

[0022] Input the tensor image feature vector into the Transformer module and calculate the global features of the image through the multi-head attention mechanism;

[0023] The mask block filling is performed on the global features of the image, and the filled global features of the image are input into a decoder to perform an operation of restoring the image block content, thereby obtaining a facial image restoration image of the forged facial image.

[0024] The present invention also discloses a face image restoration system based on a mask autoencoder, the system comprising: an image cropping module, a feature map splicing module, a source face restoration module, an image mask module and an autoencoder restoration module;

[0025] The image cropping module is used to collect a forged face image, crop the forged face image using a RetinaFace model, and obtain a source face image and a target face image corresponding to the forged face image;

[0026] The feature map splicing module is used to extract the shading features and high-level semantic features of the source face image, upsample the high-level semantic features and then splice them with the shading features, and obtain a segmentation probability map based on the spliced feature map;

[0027] The source face restoration module is configured to multiply the segmentation probability map by the source face image to obtain source face information, and restore and reconstruct the source face based on the source face information to obtain a reconstructed source face;

[0028] The image mask module is used to divide the segmentation probability map into blocks and calculate the mean of each segmentation probability map, assign weights based on the mean results and calculate the weights, and perform masking based on the weights to obtain a masked image;

[0029] The autoencoder restoration module is used to input the target facial features in the reconstructed source face and the mask image into the mask autoencoder to obtain a facial image restoration image of the forged facial image.

[0030] Optionally, the workflow of the feature map splicing module specifically includes:

[0031] Use EfficientNet-B0 to extract low-level texture features and high-level semantic features;

[0032] The high-level semantic features are upsampled using transposed convolution, the upsampled high-level semantic features are spliced with the underlying texture features, and the spliced features are input into a transposed convolution layer to generate a segmentation probability map.

[0033] Optionally, the source face restoration module includes: a semantic feature extraction submodule, a channel convolution submodule, a channel restoration submodule and an image reconstruction submodule;

[0034] The semantic feature extraction submodule is used to obtain source face information based on the multiplication of the segmentation probability map and the source face image, downsample the source face information through four convolution-pooling layers, and extract a feature map of high-level semantic features;

[0035] The channel convolution submodule is used to split the feature map of the semantic features into source face features and potential target face features, upsample the source face features using transposed convolution, and convolve the potential target face features using a convolution layer to obtain a feature map with a preset number of channels;

[0036] The channel restoration submodule is used to splice the feature map of high-level semantic features with the feature map of the preset number of channels using residual connections, and restore the number of channels of the spliced feature map using convolution;

[0037] The image reconstruction submodule is used to repeat the feature map of the restored channel number 4 times S31-S33, restore the source face image, calculate the identity loss and perceptual loss based on the source face image and the source face image in step S1, and obtain the reconstructed source face.

[0038] Optionally, the workflow of the image mask module specifically includes: calculating the mask block serial number based on the segmentation probability map, dividing the forged face into blocks based on the mask block serial number, and using the mask block serial number to mask the forged face after blocking to obtain a masked image.

[0039] Optionally, the workflow of the autoencoder recovery module specifically includes:

[0040] Inputting the potential target facial features in the reconstructed source face and the uncovered image blocks in the masked image into a masked autoencoder for flattening and merging into a tensor, inputting the tensor into a fully connected layer to obtain a tensor image feature vector;

[0041] Input the tensor image feature vector into the Transformer module and calculate the global features of the image through the multi-head attention mechanism;

[0042] The mask block filling is performed on the global features of the image, and the filled global features of the image are input into a decoder to perform an operation of restoring the image block content, thereby obtaining a facial image restoration image of the forged facial image.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] The present invention proposes a deep fake face recovery algorithm based on a mask autoencoder, using the recovered source face and target face as evidence to support the detection results. First, an identity information segmentation module is constructed to segment the source / target face information to obtain an identity segmentation map; the source face information is further segmented and input into the source face recovery module to reconstruct the source face and extract potential target identity features; finally, the segmentation map is used to generate a mask to cover the image to be detected, and a target face recovery module based on a mask autoencoder is constructed to restore the masked area and reconstruct the target face. After testing on three deep fake data sets and six types of deep fake algorithms, the target face recovered by the technical solution of the present invention has an identity similarity of up to 75.82% with the original target face, surpassing existing algorithms. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0046] Figure 1 This is a framework diagram of the deep fake face recovery algorithm based on masked autoencoder;

[0047] Figure 2 This is the Transformer module architecture diagram used in the masked self-encoder. Specific implementation methods

[0048] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0049] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0050] Example 1

[0051] A face image restoration method based on mask autoencoder, such as Figure 1 As shown, specifically including:

[0052] Step S1: Acquire a forged face image, crop the forged face image using a RetinaFace model, and obtain a source face image and a target face image corresponding to the forged face image.

[0053] RetinaFace is used to locate and align the forged face images in the dataset, and then a 224×224 face image is cropped based on the coordinates of the facial feature points. Similarly, the corresponding source and target face images are obtained.

[0054] Step S2: extracting texture features and high-level semantic features of the source face image, upsampling the high-level semantic features and concatenating them with the texture features, and obtaining a segmentation probability map based on the concatenated feature map, specifically comprising: using a pre-trained EfficientNet-B0 to extract features, using the first layer features as the underlying texture features, using the second-to-last layer features as the high-level semantic features, upsampling the high-level semantic features using four transposed convolutions, concatenating them with the underlying texture features, generating a segmentation map again through a transposed convolution layer, and constraining it to the range of 0-1 using a Sigmoid function.

[0055] Step S3, multiplying the segmentation probability map with the source face image to obtain source face information, and restoring and reconstructing the source face based on the source face information to obtain a reconstructed source face, including: S31, based on the source face information obtained by multiplying the segmentation probability map with the source face image, downsampling the source face information through 4 convolution-pooling layers to extract a feature map of high-level semantic features; S32, segmenting the high-level semantic features into source face features and potential target face features, and using transposed convolution to downsample the source face features. The features are upsampled, and the potential target facial features are convolved using a convolutional layer to obtain a feature map with a preset number of channels. S33, the feature map of the high-level semantic features is spliced with the feature map with the preset number of channels using a residual connection, and the spliced feature map is restored to the number of channels using convolution. S34, S31-S33 are repeated four times for the feature map with the restored number of channels to restore the source facial image. The identity loss and perceptual loss are calculated based on the source facial image and the source facial image in step S1 to obtain a reconstructed source face. In this embodiment, the number of channels is set to 98.

[0056] Specifically: S31, input the source face information into the source face recovery module, perform downsampling operation through 4 convolution-pooling layers, and extract high-level semantic features of the image;

[0057] S32. Segment the feature map of the high-level semantic features obtained in S31 into source facial features and potential target facial features. Upsample the source facial features using a transposed convolutional layer. Process the potential target facial features using a convolutional layer so that their number of channels is consistent with the number of channels in the target facial image. In this embodiment, the number of channels is set to 98.

[0058] S33. Concatenate the feature map calculated by the downsampling layer operation with the corresponding feature map in the upsampling stage through a residual connection, and use a convolution operation to restore the number of channels. In this embodiment, the number of restored channels is the same as the number of channels in the feature map calculated by the downsampling operation. For example, if the downsampling layer has 512 channels and the corresponding upsampling layer has 512 channels, the concatenation of the two will result in 1024 channels, which are then restored to 512 channels using a convolution operation.

[0059] S34. Restore the source face image through four transposed convolution-residual connection-convolution operations, and then calculate the identity loss and perceptual loss with the original source face image to optimize the source face recovery module.

[0060] Step S4: Divide the segmentation probability map into blocks and calculate the mean of each block. Weights are assigned based on the mean results and calculated. Masking is performed based on the weights to obtain a masked image. This process specifically includes: calculating mask block numbers based on the segmentation probability map, dividing the forged face into blocks based on the mask block numbers, and masking the forged face after the blocks are divided using the mask block numbers to obtain a masked image. Step S41: Divide the segmentation probability map into blocks and calculate the mean of the segmentation probability map. The mean calculation process for the probability map is as follows: Because the values in the probability map are all probabilities between 0 and 1, the values of the probability maps of all segmentation maps are added together to calculate the mean. If the mean is less than 0.5, the weight of each block is set to a random value between 0 and 1. If the mean is greater than or equal to 0.5, the mean of each block is calculated as the weight. Step S42: Sort the weights calculated in Step S41 from largest to smallest, define a mask ratio β, and mask the image blocks with larger weights. In this embodiment, β is set to 0.5.

[0061] Step S5, inputting the potential target facial features in the reconstructed source face and the masked image into the masked autoencoder to obtain a facial image restoration image of the forged facial image, specifically comprising: inputting the potential target facial features in the reconstructed source face and the uncovered image blocks in the masked image into the masked autoencoder for flattening, merging into a tensor, inputting the tensor into a fully connected layer to obtain a tensor image feature vector; inputting the tensor image feature vector into the Transformer module, calculating the image global features through a multi-head attention mechanism; performing mask block filling on the image global features, inputting the filled image global features into the decoder to perform an operation of restoring the image block content, and obtaining a facial image restoration image of the forged facial image.

[0062] S51, dividing the image blocks into blocks, and masking the image blocks according to the mask numbers calculated in S42 (pixels are set to 0); in this embodiment, the block size is set to 16, each image is divided into 196 image blocks, and 98 image blocks are retained after masking;

[0063] S52: Input the uncovered image block into the mask autoencoder to recover the image block content corresponding to the target face image.

[0064] S521. Flatten the uncovered image blocks and merge them into a tensor; input the tensor into a fully connected layer to obtain the corresponding image feature vector;

[0065] S522, input the image feature vector into the Transformer module (such as Figure 2As shown in Figure 2, the global features of the image are calculated through a multi-head self-attention mechanism; the final global features are obtained by repeating the Transformer module calculation L times. The global feature calculation process is as follows:

[0066] f m =F(W·x sv )+b)+p sv );

[0067] Among them, F represents the encoding network composed of L Transformer modules, x sv represents the unmasked image tensor, W and b represent the weight and bias of the fully connected layer respectively, and p sv Indicates that x sv The corresponding position code is set as 7 in this embodiment.

[0068] S523. Use the mask block sequence number to pad the global features with mask blocks so that they correspond to the complete image block sequence; pad the unmasked image block features to match the number of image blocks. Assuming the number of unmasked image blocks is 52 and the number of complete image block features is 64, use empty features with a value of 0 to pad the 52 to 64.

[0069] S524: Input the padded global features into the decoder (composed of n Transformer modules), and restore the image block content through operations such as the multi-head attention mechanism and normalization layer. The identity loss, perceptual loss, attribute loss, and pixel recovery loss are calculated with the original target face image. The process of restoring the image content is as follows:

[0070] x rec =Sigmoid(D(f m ||f t ));

[0071] Where D represents the decoder consisting of n Transformer modules, f m represents the features obtained by encoding the uncovered image, f t represents the potential target face features extracted by the source face restoration module. The Sigmoid function is used to constrain the output to the range of 0 to 1. In this embodiment, n is set to 3.

[0072] Example 2

[0073] The proposed scheme was experimented on three datasets: FaceForensics++, CelebaMegaFS, and FFHQ-E4S. Faceforensics++ includes two existing deepfake face-swapping methods: DeepFake (DF) and FaceShifter (FShi). In addition, another set of forged data was generated using the LCR algorithm. These three types of forged images were split into corresponding training, validation, and test sets according to the partitioning strategy specified by FaceForensics++. CelebaMegaFS includes three deepfake methods: ID Injection (IDI), FTM, and LCR, generating 30,038 FTM images, 30,441 ID Injection images, and 30,010 LCR images. FFHQ-E4S is built on the FFHQ dataset. We randomly selected 10,000 images from FFHQ and generated 10 swapped images for each original image using the E4S algorithm. For CelebaMegaFS and FFHQ-E4S, each dataset is divided into training set, validation set and test set in the ratio of 8:1:1.

[0074] In the experiments, the quality of image restoration was evaluated using FID, and the identity similarity between the restored image and the original target face image was evaluated using IDSim. Table 1 shows the restoration performance on three datasets. The "None" method in the first row shows the FID score between the forged face and the original target face, and the following six rows show the numerical values of the corresponding indicators between the restored face and the original target face. The image restoration results on the FaceForensics++, CelebaMegaFS, and FFHQ-E4S datasets were analyzed. Compared with existing face restoration methods (MAT, Repaint) and deepfake restoration methods (RECCE, Delocate, DFI), our scheme achieved the best FID score and identity similarity in different types of deepfake face restoration, and was able to restore more realistic target face images. It can be seen that our scheme can restore the source face while restoring the target face, and the effect is better than that of existing algorithms. Table 1 shows the target face restoration effect on various types of deepfake data.

[0075] Table 1

[0076]

[0077] Example 3

[0078] A face image restoration system based on a mask autoencoder, the system comprising: an image cropping module, a feature map splicing module, a source face restoration module, an image mask module and an autoencoder restoration module;

[0079] The image cropping module is used to collect fake face images, crop them using the RetinaFace model, and obtain the source and target face images corresponding to the fake face images. RetinaFace is used to locate and align the fake face images in the dataset, and then crop a 224×224 face image based on the coordinates of the facial feature points. Similarly, the corresponding source and target face images are obtained.

[0080] The feature map concatenation module extracts texture features and high-level semantic features from the source facial image. The high-level semantic features are upsampled and concatenated with the texture features, generating a segmentation probability map based on the concatenated feature map. The cropped forged facial image is fed into the identity segmentation module to calculate the segmentation probability map. Features are extracted using the pretrained EfficientNet-B0, with the first-layer features used as the underlying texture features and the second-to-last-layer features as the high-level semantic features. The high-level semantic features are upsampled using four transposed convolutions and concatenated with the underlying texture features. A segmentation map is generated using a transposed convolution layer again, and a sigmoid function is used to constrain the segmentation map to a range of 0–1.

[0081] The source face restoration module is used to multiply the segmentation probability map with the source face image to obtain source face information, and restore and reconstruct the source face based on the source face information to obtain a reconstructed source face. The source face restoration module includes: a semantic feature extraction submodule, a channel convolution submodule, a channel restoration submodule, and an image reconstruction submodule;

[0082] The semantic feature extraction submodule is used to obtain source face information based on the multiplication of the segmentation probability map and the source face image, downsample the source face information through 4 convolution-pooling layers, and extract the feature map of high-level semantic features; the channel convolution submodule is used to divide the feature map of high-level semantic features into source face features and potential target face features, upsample the source face features using transposed convolution, and convolve the potential target face features using a convolution layer to obtain a feature map with a preset number of channels; the channel restoration submodule is used to splice the feature map of high-level semantic features with the feature map with the preset number of channels using residual connection, and restore the number of channels of the spliced feature map using convolution; the image reconstruction submodule is used to repeat S31-S33 of the feature map with the restored number of channels 4 times to restore the source face image, calculate the identity loss and perceptual loss based on the source face image and the source face image, and obtain the reconstructed source face.

[0083] The obtained segmentation probability map is multiplied by the forged face image to obtain the source face information, and the source face information is input into the source face recovery module to restore and reconstruct the source face. The source face information is input into the source face recovery module, and down-sampling operations are performed through 4 convolution-pooling layers to extract high-level semantic features of the image; the obtained high-level semantic features are divided into two parts: source face features and potential target face features, and the source face features are up-sampled using a transposed convolution layer; the potential target face features are processed using a convolution layer so that their number of channels is consistent with the number of channels in the target face recovery module. In this embodiment, we set the number of channels to 98. The feature map calculated by the downsampling layer operation is spliced with the corresponding features in the upsampling stage through residual connection, and the number of channels is restored using convolution operation; the source face image is restored through 4 transposed convolution-residual connection-convolution operations, and then the identity loss and perceptual loss are calculated with the original source face image to optimize the source face recovery module.

[0084] The image mask module is used to divide the segmentation probability map into blocks and calculate the mean of each block of the segmentation probability map. Based on the mean result, weights are assigned and calculated, and masking is performed based on the weights to obtain a masked image. The obtained segmentation probability map is input into the mask generation module, the mask block sequence is calculated, and the forged face is divided into blocks. The corresponding image blocks are masked according to the mask block sequence number. The segmentation probability map is divided into blocks and the mean of the segmentation probability map is calculated. If the mean is less than 0.5, the weight of each block is set to a random value between 0 and 1. If the mean is greater than or equal to 0.5, the mean of each block is calculated as the weight. The calculated weights are sorted from large to small, and the mask ratio is defined as β. The image blocks with larger weights are masked. In this embodiment, β is set to 0.5.

[0085] The autoencoder restoration module is used to input the potential target facial features in the reconstructed source face and the mask image into the mask autoencoder to obtain a facial image restoration image of the forged facial image.

[0086] The image blocks are divided into blocks and masked (pixels are set to 0) according to the mask sequence number. In this embodiment, the block size is set to 16, each image is divided into 196 blocks, and 98 blocks are retained after masking.

[0087] The unmasked image block is input into the mask autoencoder to recover the content of the image block corresponding to the target face image.

[0088] Flatten the uncovered image blocks and merge them into a tensor; input it into the fully connected layer to obtain the corresponding image feature vector;

[0089] Input the image feature vector into the Transformer module (such as Figure 2As shown in Figure 2, the global features of the image are calculated through a multi-head self-attention mechanism; the final global features are obtained by repeating the Transformer module calculation L times. The calculation process is as follows:

[0090] f m =F(W·x sv +b)+p sv );

[0091] Among them, F represents the encoding network composed of L Transformer modules, x sv represents the unmasked image tensor, W and b represent the weight and bias of the fully connected layer respectively, and p sv Indicates that x sv The corresponding position code is set as 7 in this embodiment.

[0092] Using the mask block sequence number, the global feature mask block is filled so that it corresponds to the complete image block sequence;

[0093] The padded global features are input into the decoder (composed of n Transformer modules), and the image block content is restored through multi-head attention mechanism, normalization layer and other operations. The identity loss, perception loss, attribute loss and pixel recovery loss are calculated with the original target face image. The process of restoring the image content is as follows:

[0094] x rec =Sigmoid(D(f m ||f t ));

[0095] Where D represents the decoder consisting of n Transformer modules, f m represents the features obtained by encoding the uncovered image, f t represents the potential target face features extracted by the source face restoration module. The Sigmoid function is used to constrain the output to the range of 0 to 1. In this embodiment, n is set to 3.

[0096] The embodiments described above are merely descriptions of preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by persons skilled in the art should fall within the scope of protection defined by the claims of the present invention.

Claims

1. A face image restoration method based on mask autoencoder, characterized in that: The method comprises: Step S1: collecting a forged face image, cropping the forged face image using a RetinaFace model, and obtaining a source face image and a target face image corresponding to the forged face image; Step S2: extracting shading features and high-level semantic features of the source face image, upsampling the high-level semantic features and then splicing them with the shading features, and obtaining a segmentation probability map based on the spliced feature map; Step S3: multiplying the segmentation probability map by the source face image to obtain source face information, and restoring and reconstructing the source face based on the source face information to obtain a reconstructed source face; Step S4: Divide the segmentation probability map into blocks and calculate the mean of each segmentation probability map, assign weights based on the mean results and calculate the weights, perform masking based on the weights, and obtain a masked image; Step S5: input the potential target facial features in the reconstructed source face and the mask image into the mask autoencoder to obtain a facial image restoration image of the forged facial image.

2. The face image restoration method based on mask autoencoder according to claim 1, characterized in that: In step S2, extracting the shading features and high-level semantic features of the source face image, upsampling the high-level semantic features and then splicing them with the shading features, and obtaining a segmentation probability map based on the spliced feature map specifically includes: Use EfficientNet-B0 to extract low-level texture features and high-level semantic features; The high-level semantic features are upsampled using transposed convolution, the upsampled high-level semantic features are spliced with the underlying texture features, and the spliced features are input into a transposed convolution layer to generate a segmentation probability map.

3. The face image restoration method based on mask autoencoder according to claim 1, characterized in that: In step S3, the source face information is obtained by multiplying the segmentation probability map with the source face image, and the source face is restored and reconstructed based on the source face information to obtain the reconstructed source face. Specifically, the process includes: S31, based on the source face information obtained by multiplying the segmentation probability map with the source face image, down-sampling the source face information through four convolution-pooling layers to extract a feature map of high-level semantic features; S32, dividing the feature map of the high-level semantic features into source face features and potential target face features, upsampling the source face features using transposed convolution, and convolving the potential target face features using a convolution layer to obtain a feature map with a preset number of channels; S33, using residual connection to splice the feature map of the high-level semantic feature with the feature map of the preset number of channels, and using convolution to restore the number of channels of the spliced feature map; S34. Repeat S31-S33 4 times on the feature map of the restored channel number to restore the source face image, calculate the identity loss and the perception loss based on the source face image and the source face image in step S1, and obtain the reconstructed source face.

4. The face image restoration method based on masked autoencoder according to claim 3, characterized in that: In step S4, the segmentation probability map is divided into blocks and the mean of each segmentation probability map is calculated. Weights are assigned based on the mean results and the weights are calculated. Masking is performed based on the weights to obtain a masked image. Specifically, the following steps are performed: The mask block sequence number is calculated based on the segmentation probability map, the forged face is divided into blocks based on the mask block sequence number, and the forged face after the block division is masked using the mask block sequence number to obtain a masked image.

5. The face image restoration method based on masked autoencoder according to claim 4, characterized in that: In step S5, inputting the potential target facial features in the reconstructed source face and the mask image into the mask autoencoder to obtain the target facial image restoration image of the forged facial image specifically includes: Inputting the potential target facial features in the reconstructed source face and the uncovered image blocks in the masked image into a masked autoencoder for flattening and merging into a tensor, inputting the tensor into a fully connected layer to obtain a tensor image feature vector; Input the tensor image feature vector into the Transformer module and calculate the global features of the image through the multi-head attention mechanism; The mask block filling is performed on the global features of the image, and the filled global features of the image are input into a decoder to perform an operation of restoring the image block content, thereby obtaining a facial image restoration image of the forged facial image.

6. A face image restoration system based on a masked autoencoder, the system being used to implement the image restoration method according to any one of claims 1 to 4, characterized in that: The system includes: Image cropping module, feature map splicing module, source face restoration module, image mask module and autoencoder restoration module; The image cropping module is used to collect a forged face image, crop the forged face image using a RetinaFace model, and obtain a source face image and a target face image corresponding to the forged face image; The feature map splicing module is used to extract the shading features and high-level semantic features of the source face image, upsample the high-level semantic features and then splice them with the shading features, and obtain a segmentation probability map based on the spliced feature map; The source face restoration module is configured to multiply the segmentation probability map by the source face image to obtain source face information, and restore and reconstruct the source face based on the source face information to obtain a reconstructed source face; The image mask module is used to divide the segmentation probability map into blocks and calculate the mean of each segmentation probability map, assign weights based on the mean results and calculate the weights, and perform masking based on the weights to obtain a masked image; The autoencoder restoration module is used to input the target facial features in the reconstructed source face and the mask image into the mask autoencoder to obtain a facial image restoration image of the forged facial image.

7. The face image restoration system based on mask autoencoder according to claim 6, characterized in that: The workflow of the feature map splicing module specifically includes: Use EfficientNet-B0 to extract low-level texture features and high-level semantic features; The high-level semantic features are upsampled using transposed convolution, the upsampled high-level semantic features are spliced with the underlying texture features, and the spliced features are input into a transposed convolution layer to generate a segmentation probability map.

8. The face image restoration system based on mask autoencoder according to claim 6, characterized in that: The source face restoration module includes: a semantic feature extraction submodule, a channel convolution submodule, a channel restoration submodule and an image reconstruction submodule; The semantic feature extraction submodule is used to obtain source face information based on the multiplication of the segmentation probability map and the source face image, downsample the source face information through four convolution-pooling layers, and extract a feature map of high-level semantic features; The channel convolution submodule is used to split the feature map of the semantic features into source face features and potential target face features, upsample the source face features using transposed convolution, and convolve the potential target face features using a convolution layer to obtain a feature map with a preset number of channels; The channel restoration submodule is used to splice the feature map of high-level semantic features with the feature map of the preset number of channels using residual connections, and restore the number of channels of the spliced feature map using convolution; The image reconstruction submodule is used to repeat the feature map of the restored channel number 4 times S31-S33, restore the source face image, calculate the identity loss and perceptual loss based on the source face image and the source face image in step S1, and obtain the reconstructed source face.

9. The face image restoration system based on mask autoencoder according to claim 6, characterized in that: The workflow of the image mask module specifically includes: calculating the mask block serial number based on the segmentation probability map, dividing the forged face into blocks based on the mask block serial number, and using the mask block serial number to mask the forged face after the block division to obtain a masked image.

10. The face image restoration system based on mask autoencoder according to claim 6, characterized in that: The workflow of the autoencoder recovery module specifically includes: Inputting the potential target facial features in the reconstructed source face and the uncovered image blocks in the masked image into a masked autoencoder for flattening and merging into a tensor, inputting the tensor into a fully connected layer to obtain a tensor image feature vector; Input the tensor image feature vector into the Transformer module and calculate the global features of the image through the multi-head attention mechanism; The mask block filling is performed on the global features of the image, and the filled global features of the image are input into a decoder to perform an operation of restoring the image block content, thereby obtaining a facial image restoration image of the forged facial image.

Images