Face forgery detection method and system for reducing attribute prejudice

Abstract

The invention discloses a face forgery detection method and system for reducing attribute prejudice. The method comprises the following steps: acquiring a video to be detected; preprocessing a to-be-detected video to obtain a to-be-detected image sequence; obtaining a pre-trained counterfeit detection model; inputting the to-be-detected image sequence into a pre-trained counterfeit detection model, extracting facial attribute features by using the attribute analysis model, and extracting facial texture features by using the texture analysis model; performing fusion processing on the facial attribute features and the facial texture features through a spatial-temporal feature aggregator to obtain a detection result; the detection result represents whether the face in the to-be-detected video is forged or not. Facial attribute features and facial texture features are extracted through an attribute analysis model and a texture analysis model, so that the method has higher resolution and robustness; facial attribute features and facial texture features are fused by using a spatial-temporal feature aggregator, so that prejudice is reduced while standard deviation between attributes is reduced, and accuracy, fairness and universality are improved.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and in particular relates to a method and system for detecting face forgery with reduced attribute bias. Background Art

[0002] In recent years, with the rapid development of face editing and face generation, more and more fake videos have been circulated on social media, attracting great public attention. Existing face forgery detection methods based on spatial texture features have found that forged videos generated by generative adversarial networks have obvious inter-frame inconsistencies in timing (such as texture mutations) and unnatural texture details. However, this method has limitations in processing forged videos of faces with different attributes, and it is often difficult to capture the differences in facial attributes between consecutive frames of forged videos. Facial attributes show the difference between real and fake faces at a high semantic level. Although facial attributes in different frames can remain consistent in real videos, forged videos may show significant differences between real and fake faces at a high semantic level.

[0003] Therefore, detection technology that relies solely on spatial texture features cannot fully address the challenges of temporal coherence and facial attribute consistency in forged videos. We need more advanced methods to comprehensively analyze the spatial and temporal dimensions in videos in order to more accurately identify and distinguish real and fake faces. In addition, we found that existing face forgery detection methods based on spatial texture features have huge attribute "bias", that is, they may not treat data of different attributes in the dataset (such as gender, race, age, etc.) fairly, resulting in bias against certain specific groups; data of different attributes such as gender, race, age, etc. are not treated equally, resulting in unfair treatment of certain specific groups. This bias not only affects the universality and fairness of the detection technology, but also limits its application potential in multiple scenarios and multiple groups.

[0004] Although existing forgery detection technologies based on spatial cues can identify local features and texture anomalies in images, they often ignore the temporal inconsistencies between frames of forged videos and the high semantic differences in facial attributes, resulting in insufficient detection accuracy and robustness. Existing forgery detection technologies based on temporal cues have "attribute bias" when processing face data with different attributes, and fail to treat data with different attributes such as gender, race, and age fairly, which affects the universality and fairness of the detection technology. In addition, although forgery detection technologies based on temporal cues can analyze dynamic behaviors and physiological characteristics in videos, they also fail to fully capture the consistency of attribute features in videos, limiting the application potential of detection methods in multiple scenarios and multiple groups. Summary of the invention

[0005] In order to solve the above problems existing in the prior art, the present invention provides a face forgery detection method and system with reduced attribute bias. The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0006] In a first aspect, the present invention provides a method for detecting face forgery with reduced attribute bias, the method comprising:

[0007] Get the video to be detected;

[0008] Preprocessing the video to be detected to obtain an image sequence to be detected;

[0009] Acquire a pre-trained forgery detection model; the pre-trained forgery detection model includes: an attribute analysis model, a texture analysis model and a spatiotemporal feature aggregator;

[0010] The image sequence to be detected is input into a pre-trained forgery detection model, facial attribute features are extracted using the attribute analysis model, and facial texture features are extracted using the texture analysis model; the facial attribute features and the facial texture features are fused by the spatiotemporal feature aggregator to obtain a detection result of the image sequence to be detected; the detection result indicates whether the face in the video to be detected is forged.

[0011] In one embodiment of the present invention, preprocessing the video to be detected to obtain a sequence of images to be detected includes:

[0012] For each video to be detected, randomly extract N frames of images from the video to be detected;

[0013] Use the face detection model based on the CNN network to identify the face area in each frame and obtain facial feature points;

[0014] The to-be-detected image sequence is obtained based on the facial feature points.

[0015] In one embodiment of the present invention, the image sequence to be detected includes several picture sequences.

[0016] In one embodiment of the present invention, the training process of the forgery detection model includes:

[0017] Obtain a training data set; the training data set includes a number of face videos and identity information of each face; wherein each video includes a number of face attribute labels;

[0018] Segmenting the face images in the training data set into a plurality of non-overlapping image windows as image sequences;

[0019] Inputting the image sequence into the forgery detection model, and loading the pre-trained texture analysis model, attribute analysis model and spatiotemporal feature aggregator to establish a model loss function;

[0020] Processing the model loss function using a stochastic gradient descent algorithm to minimize the model loss function;

[0021] The gradient of the model loss function relative to the model parameters is calculated by a back propagation algorithm, and the parameters of the forgery detection model are updated according to the gradient to obtain a trained forgery detection model.

[0022] In one embodiment of the present invention, the attribute analysis model is obtained by adding a first global average pooling layer and a fully connected layer to a ResNet34 network without loading pre-trained weights, and loading the first pre-trained weights.

[0023] In one embodiment of the present invention, the loss function used by the attribute analysis model is expressed as follows:

[0024]

[0025] Wherein, N represents the number of frames of the image sequence to be detected, F i,a represents the features extracted for the forgery detection task, F i, ' a Represents the features extracted for the facial attribute detection task.

[0026] In one embodiment of the present invention, the texture analysis model is obtained by adding a bottleneck module and a second global average pooling layer to a ResNet101 network without loading pre-trained weights, and loading the second pre-trained weights.

[0027] In one embodiment of the present invention, the loss function used by the texture analysis model is expressed as follows:

[0028]

[0029] Wherein, N represents the number of frames of the image sequence to be detected, P i Represents the original label of the face image, P i, ' t Represents the prediction results of the texture analysis model for forgery detection.

[0030] In one embodiment of the present invention, the loss function used by the spatiotemporal feature aggregator is expressed as follows:

[0031]

[0032] Wherein, N represents the number of frames of the image sequence to be detected, P iRepresents the original label of the face image, P i, ' out Represents the prediction results of the forgery detection model.

[0033] In a second aspect, the present invention provides a face forgery detection system that reduces attribute bias, the system comprising:

[0034] A video acquisition module is used to acquire the video to be detected;

[0035] An image sequence acquisition module is used to preprocess the video to be detected to obtain an image sequence to be detected;

[0036] The pre-trained forgery detection model includes: an attribute analysis model, a texture analysis model and a spatiotemporal feature aggregator; the pre-trained forgery detection model is used to extract facial attribute features from the image sequence to be detected using the attribute analysis model and to extract facial texture features from the texture analysis model; the facial attribute features and the facial texture features are fused by the spatiotemporal feature aggregator to obtain the detection result of the image sequence to be detected; the detection result represents whether the face in the video to be detected is forged.

[0037] Beneficial effects of the present invention:

[0038] In the solution provided by the present invention, facial attribute features and facial texture features are extracted through attribute analysis models and texture analysis models, which are more discriminative and robust; facial attribute features and facial texture features are fused using a spatiotemporal feature aggregator, which can not only identify local texture anomalies in the image, but also detect inconsistencies in facial attributes, thereby reducing the standard deviation between attributes while reducing bias and improving accuracy, fairness and universality. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A schematic diagram of the steps of a face forgery detection method for reducing attribute bias provided by an embodiment of the present invention;

[0040] Figure 2 A schematic flow chart of a method for detecting forged faces with reduced attribute bias provided by an embodiment of the present invention;

[0041] Figure 3 A schematic diagram of the structure of a spatiotemporal feature aggregator of a face forgery detection method that reduces attribute bias provided by an embodiment of the present invention;

[0042] Figure 4 A schematic diagram of the structure of a face forgery detection system for reducing attribute bias provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The following will be combined with the accompanying drawings of the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] The embodiments of the present invention provide a method and system for detecting forged faces with reduced attribute bias.

[0045] It should be noted that the execution subject of the face forgery detection method for reducing attribute bias provided in the embodiment of the present invention may be a face forgery detection system for reducing attribute bias, but is certainly not limited thereto.

[0046] Below, firstly, a face forgery detection method for reducing attribute bias provided by an embodiment of the present invention is introduced.

[0047] like Figure 1 As shown, a face forgery detection method for reducing attribute bias provided by an embodiment of the present invention may include the following steps:

[0048] S1, obtain the video to be detected;

[0049] S2, preprocessing the video to be detected to obtain a sequence of images to be detected;

[0050] S3, obtaining a pre-trained forgery detection model; the pre-trained forgery detection model includes: an attribute analysis model, a texture analysis model, and a spatiotemporal feature aggregator;

[0051] S4, input the image sequence to be detected into the pre-trained forgery detection model, use the attribute analysis model to extract facial attribute features, and use the texture analysis model to extract facial texture features; fuse the facial attribute features and facial texture features through the spatiotemporal feature aggregator to obtain the detection result of the image sequence to be detected; the detection result indicates whether the face in the video to be detected is forged.

[0052] In order to solve the problems existing in the face forgery detection method based on spatial clues, in addition to using the texture analysis model, an attribute analysis model is introduced to extract facial attribute features including gender and race. The embodiment of the present invention proposes a spatiotemporal feature face forgery detection framework based on texture and attributes. The spatiotemporal feature aggregator proposed in the embodiment of the present invention fuses texture and attribute features together, in which the spatial and temporal attention mechanisms interact with each other, thereby improving the accuracy of the model and reducing the attribute bias problem for different data sets. The face forgery detection method proposed in the embodiment of the present invention fuses texture features and attribute features in the spatiotemporal frequency domain for comprehensive analysis, which can not only capture the characteristics of forged videos in spatial and temporal dimensions, but also identify high semantic differences in facial attributes, thereby achieving more accurate and fair face forgery detection.

[0053] A flowchart of a method for detecting face forgery with reduced attribute bias provided by an embodiment of the present invention is shown in FIG. Figure 2 , the structural diagram of the spatiotemporal feature aggregator, see Figure 3 , for ease of understanding, the following will combine Figure 2 and Figure 3 Each step of the face forgery detection method provided in the embodiment of the present invention is described separately.

[0054] from Figure 2 It can be seen that in the first stage,

[0055] Construct a new face image dataset from the YFCC-100M Flickr dataset. Specifically, the dataset contains 108,501 face images that are racially balanced and labeled with race, gender, and age group. These images will be used for subsequent training and testing of attribute analysis models. Get the dataset images by specifying the path.

[0056] The dataset images are input into the attribute analysis model. Specifically, the network uses ResNet34 as the base model and replaces its last fully connected layer with a new layer suitable for attribute prediction.

[0057] The convolutional layer part of the model is used to extract feature representations of the input image, which are then fed into the modified fully connected layer for classification.

[0058] This is achieved through the final fully connected layer of the ResNet34 model. This layer converts features into predictions for specific attributes.

[0059] The attributes predicted by the model include gender, age, and race. Gender is predicted as a binary classification problem to be male or female, age is predicted as a multi-classification problem to be different age groups, and race is predicted as a multi-classification problem to be different racial categories.

[0060] In the second stage,

[0061] The specified face forgery video dataset is processed into a sequence of images to be detected and input into the attribute analysis model and the texture analysis model respectively.

[0062] The facial attribute features of the dataset images are extracted through the attribute analysis model, and the facial texture features are extracted through the texture analysis model. The attribute and texture features are input into the spatiotemporal aggregator for feature fusion.

[0063] Temporal and spatial attention mechanisms are used to enhance the representation of fused features.

[0064] Use a binary classification problem to determine whether the image is forged.

[0065] For S1, this may include:

[0066] Through the specified video folder path, traverse all the videos in the folder as the videos to be detected.

[0067] For S2, this may include:

[0068] S21, for each video to be detected, randomly extract N frames of pictures in the video to be detected.

[0069] S22, using a face detection model based on a CNN network to identify the face area in each frame of the image and obtain facial feature points.

[0070] Specifically, the face detected in each frame is processed using a shape predictor to obtain detailed facial feature points.

[0071] S23, obtaining a sequence of images to be detected based on the facial feature points.

[0072] The image sequence to be detected may include several picture sequences.

[0073] The image sequence here is an unprocessed image sequence when it is input. At this time, it is not known how many images should be processed in a batch or how many frames should be taken in a video. In addition, the height and width of the image sequence are not processed. After entering the model, the image sequence is loaded and further processed. By customizing the MyDatasetWild class in the code, the following process can be achieved: for example, the number of images in each batch, the number of video frames in each image sequence, the number of channels of the image, the height and width of the image can be customized; the number of images can be represented by the batch size, the number of video frames can be set to N frames, and the number of channels can be set to 3 for color images. In the code, the image is first adjusted to 256x256 pixels, and then cropped to 224x224 for subsequent processing. Of course, these customized parameters can be adjusted according to the needs of the user.

[0074] Specifically, the face area is cropped from N frames of pictures according to all facial feature points, the face area is resized and edge-filled, and N frames of face images are generated as the image sequence to be detected. Optionally, an embodiment of the present invention selects a face forgery detection dataset WildDeepfake, which may include 7314 facial sequences extracted from 707 Deepfake videos. Among them, the videos can be collected from bilibili and YouTube, and these videos are synthesized by multiple methods, making detection more difficult.

[0075] For S3, the pre-trained forgery detection model may include: an attribute analysis model, a texture analysis model, and a spatiotemporal feature aggregator. For ease of understanding, each module in the pre-trained forgery detection model is introduced below.

[0076] Attribute analysis model

[0077] Specifically, the attribute analysis model is obtained by adding the first global average pooling layer and the fully connected layer to the ResNet34 network without loading the pre-trained weights, and loading the first pre-trained weights.

[0078] The face forgery detection method proposed in the embodiment of the present invention uses face attribute feature detection to train the attribute analysis model, thereby realizing the attribute feature measurement at the high-level semantic level. In order to extract the attribute features of the face, face attribute detection is used as a pre-training task to extract the attribute information. The attribute analysis model obtained by the face recognition method based on face features is used to extract attribute features such as race, gender and age. Specifically, the attribute analysis model can adopt the classic ResNet34 architecture, which consists of 34 convolutional layers, including multiple residual blocks, each residual block consists of two 3×3 convolutional layers, and can also include 1×1 convolutional layers and identity mapping to facilitate cross-layer connections. This residual learning structure helps to alleviate the problem of gradient disappearance during deep neural network training, making the model easier to train and optimize. The method constructs a new face image dataset from the YFCC-100MFlickr dataset. The dataset contains 108,501 face images that are balanced in race and labeled with race, gender and age groups. N video frames V=I1,I2,I3,...,I N Input attribute analysis model and extract attribute features F a ={f a,1 ,f a,2 ,f a,3 ,…,f a,NThe pre-trained model is also fine-tuned for the downstream task of face forgery detection. The embodiment of the present invention uses a loss function to constrain the attribute features before and after the model is updated, so that the updated attribute feature F a The l1 distance is closer to the original attribute detection feature F a ′.

[0079] The expression of the loss function used by the attribute analysis model is as follows:

[0080]

[0081] Where N represents the number of frames in the image sequence to be detected, F i,a represents the features extracted for the forgery detection task, F i, ' a Represents the features extracted for the facial attribute detection task.

[0082] It can be understood that the features extracted by the forgery detection task may include the features extracted by the forgery detection model, and the features extracted by the facial attribute detection task may include the features extracted by the attribute analysis model.

[0083] The process of extracting facial attribute features using the attribute analysis model may include:

[0084] Use the attribute analysis model to process the image sequence to be detected and obtain the corresponding feature vector;

[0085] The feature vector is mapped and dimensionally increased to output an 18-dimensional feature vector, thereby extracting facial attribute features.

[0086] Texture analysis model

[0087] The texture analysis model is obtained by adding a bottleneck module and a second global average pooling layer to the ResNet101 network without loading pre-trained weights, and loading the second pre-trained weights.

[0088] Considering that a forged face may lack texture details, the embodiment of the present invention uses a texture analysis model to extract texture features, thereby improving the performance of forgery detection. The texture analysis model based on Res2Net is used as the backbone network to obtain texture features F t ={f t,1 ,f t,2 ,f t,3 ,...,f t,N}, which can capture detailed spatial forgery clues that are complementary to temporal attribute features. Texture features are extracted using a texture analysis model based on a multi-scale backbone architecture. The texture analysis model divides the input features into several groups. The current group of filters first extracts features from a set of input feature maps, and then combines the output feature maps of the previous group with another set of input feature maps and sends them to the next group of filters. Repeat the above process until all input feature maps are processed; finally, the feature maps of all groups are connected and sent to another 1×1 filter to complete the integration of information. Along any possible path from the input feature map to the output feature map, when passing through the 3×3 filter, the effective perception increases, and due to the combination effect, many equivalent feature scales are obtained. Utilize the classification results everywhere in the module and design a loss function for it, so that the predicted label P t ′ is closer to the true label P.

[0089] The loss function used by the texture analysis model is expressed as follows:

[0090]

[0091] Where N represents the number of frames in the image sequence to be detected, P i Represents the original label of the face image, P i, ' t Represents the prediction results of the texture analysis model for forgery detection.

[0092] The process of extracting facial texture features using a texture analysis model may include:

[0093] The bottleneck module and global average pooling layer in the texture analysis model are used to process the image sequence to be detected to obtain the corresponding facial texture features.

[0094] These modules can gradually abstract features using depthwise separable convolution and batch normalization techniques, and finally generate high-level feature maps as facial texture features, which can capture the texture and shape information of the face.

[0095] Spatiotemporal Feature Aggregator

[0096] See the schematic diagram of the structure of the spatiotemporal feature aggregator of the face forgery detection method provided by the embodiment of the present invention. Figure 3 . Use spatiotemporal feature aggregator to aggregate facial attribute features F t and facial texture features F a Fusion processing is performed to achieve forgery detection.

[0097] Specifically, from Figure 3 The working process of the spatiotemporal aggregator can be seen in Figure 2 The specific content of the second stage feature fusion part is as follows:

[0098] (1) In the spatiotemporal aggregator, the input texture features and attribute features are first input into the spatiotemporal attention network. After being processed by the temporal attention and spatial attention mechanisms, the attention area of ​​the feature is selectively focused to extract high-level texture features. and And high-level attribute characteristics and

[0099] (2) Then, the texture features and attribute features obtained through the temporal attention mechanism are further processed by the residual block to obtain the temporal fusion features. The texture features and attribute features of the spatial attention mechanism are further processed by the residual block to obtain the spatial fusion features.

[0100] (3) Repeat (1) and (2) to obtain more accurate high-level features by fusing multiple layers of spatiotemporal features. and

[0101] The obtained features are input into the spatiotemporal feature aggregator, and the attention area of ​​the column features is selectively managed through the temporal and spatial attention mechanism. The expression of the features input into each layer of the aggregator is as follows:

[0102] F n,1 =Aggre1(f t,n ,f a,n ),n=1,2,…,N;

[0103] F n,m =Aggre m (F 2n-1,m-1 ,F 2n,m-1 ),1,2,…,N / 2 n ; m = 2, 3, ..., log2M;

[0104] Among them, F n,1 represents the output of the first layer of the spatiotemporal feature aggregator, m represents the mth layer of the spatiotemporal feature aggregator, and through the fusion of multiple layers of spatiotemporal features, the feature F used for forgery detection can be finally obtained. out , where the loss function of the last layer of the spatiotemporal feature aggregator is defined as constraining the final prediction P with the true value P out .

[0105] The loss function used by the spatiotemporal feature aggregator is expressed as follows:

[0106]

[0107] Where N represents the number of frames in the image sequence to be detected, P i Represents the original label of the face image, P′ i,out Represents the prediction results of the forgery detection model.

[0108] Therefore, the loss function in the embodiment of the present invention can be L sum =L attribute +L texture +L out Understandably, L attribute represents the loss function used by the attribute analysis model, L texture represents the loss function used by the texture analysis model, L out Represents the loss function adopted by the spatiotemporal feature aggregator.

[0109] Optionally, the network architecture of the pre-trained forgery detection model can be as shown in Table 1.

[0110] Table 1

[0111]

[0112] Specifically, the training process of the forgery detection model includes:

[0113] Obtain a training data set; the training data set includes a number of face videos and identity information of each face; wherein each video includes a number of face attribute labels;

[0114] The face images in the training data set are divided into several non-overlapping image windows as image sequences;

[0115] Input the image sequence into the forgery detection model, and load the pre-trained texture analysis model, attribute analysis model, and spatiotemporal feature aggregator to establish the model loss function;

[0116] The model loss function is processed using the stochastic gradient descent algorithm to minimize the model loss function;

[0117] The gradient of the model loss function relative to the model parameters is calculated through the back-propagation algorithm, and the parameters of the forgery detection model are updated according to the gradient to obtain a trained forgery detection model.

[0118] It is understandable that the image sequence can be formed by selecting a certain number of continuous image frames in sequence. Each picture includes a face image and the face attribute label, which should correspond to the label information of the corresponding video. After the image sequence is input into the forgery detection model, the forgery detection model converts the image sequence into the format required for model training; when loading the pre-trained texture analysis model and attribute analysis model, the corresponding pre-trained weights also need to be loaded; the capsule network is initialized as a spatiotemporal feature aggregator and used for subsequent classification tasks; in the training cycle, when the stochastic gradient descent algorithm is used to minimize the model loss function, the loss function used by the attribute analysis model can be a cross entropy loss function, the loss function used by the texture analysis model can be an L1 loss function, and the loss function used by the spatiotemporal feature aggregator can be a classification loss function; the parameters of the forgery detection model are updated according to the obtained gradient until the performance of the forgery detection model on the training data set reaches a good effect, thereby obtaining a well-trained forgery detection model.

[0119] For S4, the image sequence to be detected is input into the pre-trained forgery detection model, the facial attribute features are extracted using the attribute analysis model, and the facial texture features are extracted using the texture analysis model; the facial attribute features and facial texture features are fused through the spatiotemporal feature aggregator to obtain the detection results of the image sequence to be detected; the detection results represent whether the face in the video to be detected is forged.

[0120] Specifically, the structural diagram of the spatiotemporal feature aggregator is as follows: Figure 3 As shown in the figure, the spatiotemporal feature aggregator inputs the facial attribute features and facial texture features into the capsule network capnet for feature fusion. First, the facial attribute features and facial texture features are reduced in dimension through a dimensionality reduction layer and adjusted to a shape suitable for subsequent processing. Then, these features are divided into multiple segments in the time dimension, and each segment is input into three different spatiotemporal attention mechanism modules, which can learn the temporal-spatial importance of features in the sequence. Then, the pooling layer and the bottleneck layer are used to further compress the features, and finally the detection results of the image sequence to be detected are output through the classifier to realize the detection of facial features after feature fusion.

[0121] The simulation results of the embodiment of the present invention are as follows:

[0122] In the simulation experiment of the embodiment of the present invention, the existing technology based on spatiotemporal texture features and the face forgery detection method based on spatiotemporal texture features and attribute features proposed in the embodiment of the present invention are compared on the gender attribute of the WildDeepfake dataset. The comparison results are shown in Table 2.

[0123] Table 2

[0124]

[0125] It can be seen from Table 2 that the average accuracy of the method based on spatiotemporal texture features reaches 82.65%, but there is a huge difference between the two attributes of male and female, and the standard deviation reaches 3.38. After using the spatiotemporal fusion features proposed in the embodiment of the present invention, the average accuracy is not only improved to 82.94%, but the difference between the two attributes of male and female is also greatly reduced, and the standard deviation is only 0.03, which shows that the bias between different gender attributes is effectively reduced. Based on the same data set, the average accuracy of the face image forgery detection method based on fuzzy network is 64.47%, the average accuracy of the deep fake video detection method based on face distortion is 55.46%, the average accuracy of the face recognition technology based on Xception is 77.25%, the average accuracy of the deep fake detection method is 76.25%, the average accuracy of the face forgery detection method based on frequency perception is 80.66%, the average accuracy of the face recognition research method based on pseudo face recognition technology is 77.38%, the average accuracy of the deep fake video detection method based on frame reasoning is 75.88%, and the average accuracy of the end-to-end deep fake detection method based on visual converter is 81.35%. It can be seen that the average accuracy of the face forgery detection method based on spatiotemporal texture features and attribute features proposed in the embodiment of the present invention reaches 82.94%, which proves that the spatiotemporal texture and attribute features extracted by the method provided in the embodiment of the present invention are efficient and effective.

[0126] In the simulation experiment of the embodiment of the present invention, the existing technology based on spatiotemporal texture features and the face forgery detection method based on spatiotemporal texture features and attribute features proposed in the embodiment of the present invention are compared on the racial attributes of the WildDeepfake dataset, and the comparison results are shown in Table 3. Among them, the racial attributes may include: Europeans, East Asians, Africans, and South Asians.

[0127] Table 3

[0128]

[0129] It can be seen from Table 3 that the average accuracy of the method based on spatiotemporal texture features reaches 81.93%, but there are huge attribute differences between different racial attributes, and the standard deviation reaches 6.20. After using the spatiotemporal fusion features proposed in the embodiment of the present invention, the average accuracy is improved to 84.12%, and the attribute differences between different racial attributes are also greatly reduced to 3.75, indicating that the prejudice between different racial attributes is effectively reduced. Based on the same data set, the average accuracy of the face image forgery detection method based on fuzzy network is 64.47%, the average accuracy of the deep fake video detection method based on face distortion is 55.46%, the average accuracy of the face recognition technology based on Xception is 77.25%, the average accuracy of the deep fake detection method is 76.25%, the average accuracy of the face forgery detection method based on frequency perception is 80.66%, the average accuracy of the face recognition research method based on pseudo face recognition technology is 77.38%, the average accuracy of the deep fake video detection method based on frame reasoning is 75.88%, and the average accuracy of the end-to-end deep fake detection method based on visual converter is 81.35%. It can be seen that the average accuracy of the face forgery detection method based on spatiotemporal texture features and attribute features proposed in the embodiment of the present invention reaches 84.12%.

[0130] After 100 rounds of testing using the above two methods, the following results were obtained:

[0131] For gender attributes: using the spatial texture method will lead to gender bias problems, the standard deviation of measuring attribute differences ranges from 0.01 to 5.8, and the highest accuracy rate reaches 82.64%; using the spatiotemporal texture + attribute method will weaken the gender bias problem, the standard deviation of measuring attribute differences ranges from 0.01 to 3.155, and the highest accuracy rate reaches 82.94%.

[0132] For racial attributes: using the spatial texture method will lead to racial bias problems, the standard deviation of measuring attribute differences ranges from 1.158 to 10.645, and the highest accuracy rate reaches 81.93%; using the spatiotemporal texture + attribute method will weaken the racial bias problem, the standard deviation of measuring attribute differences ranges from 0.764 to 7.507, and the highest accuracy rate reaches 84.12%.

[0133] In summary, the face forgery detection method proposed in the embodiment of the present invention is based on the enhancement of time-space frequency domain clues, taking into account the frequency domain forgery clues in time and space, and the final features obtained are more discriminative and robust. The fusion technology of texture features and attribute features is used for the first time, which can not only identify local texture anomalies of images, but also detect inconsistencies in facial attributes. The face forgery detection method not only improves the accuracy, but also reduces the standard deviation between attributes, reduces bias, and improves the fairness and universality of the detection method.

[0134] In a second aspect, corresponding to the above-mentioned embodiment of a method for detecting forged faces with reduced attribute bias, an embodiment of the present invention further provides a system for detecting forged faces with reduced attribute bias, such as Figure 4 As shown, the system may include:

[0135] A video acquisition module is used to acquire the video to be detected;

[0136] An image sequence acquisition module is used to pre-process the video to be detected to obtain an image sequence to be detected;

[0137] The pre-trained forgery detection model includes: an attribute analysis model, a texture analysis model and a spatiotemporal feature aggregator; the pre-trained forgery detection model is used to extract facial attribute features of the image sequence to be detected using the attribute analysis model and to extract facial texture features using the texture analysis model; the facial attribute features and facial texture features are fused through the spatiotemporal feature aggregator to obtain the detection result of the image sequence to be detected; the detection result represents whether the face in the video to be detected is forged.

[0138] It can be understood that the working process of the video acquisition module corresponds to step S1 in the method embodiment, the working process of the image sequence acquisition module corresponds to step S2 in the method embodiment, and the working process of the pre-trained forgery detection model corresponds to steps S3-S4 in the method embodiment. For the specific processing process of each module, please refer to the specific method embodiment, which will not be elaborated here.

[0139] The embodiment of the present invention extracts facial attribute features and facial texture features through an attribute analysis model and a texture analysis model, which are more discriminative and robust; and uses a spatiotemporal feature aggregator to fuse facial attribute features and facial texture features, which can not only identify local texture anomalies in the image, but also detect inconsistencies in facial attributes, thereby reducing the standard deviation between attributes while reducing bias and improving accuracy, fairness and universality.

[0140] It should be noted that in the description of the present invention, it should be understood that the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0141] In this specification, the system embodiment is described briefly because it is basically similar to the method embodiment, and the relevant parts can be referred to the partial description of the method embodiment.

[0142] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A face forgery detection method for reducing attribute bias, characterized in that: include: Get the video to be detected; Preprocessing the video to be detected to obtain a sequence of images to be detected; Get a pre-trained forgery detection model; The pre-trained forgery detection model includes: an attribute analysis model, a texture analysis model and a spatiotemporal feature aggregator; The image sequence to be detected is input into a pre-trained forgery detection model, facial attribute features are extracted using the attribute analysis model, and facial texture features are extracted using the texture analysis model; the facial attribute features and the facial texture features are fused by the spatiotemporal feature aggregator to obtain a detection result of the image sequence to be detected; the detection result indicates whether the face in the video to be detected is forged.

2. A method for detecting face forgery with reduced attribute bias according to claim 1, characterized in that: Preprocessing the video to be detected to obtain a sequence of images to be detected includes: For each video to be detected, randomly extract N frames of images from the video to be detected; Use the face detection model based on the CNN network to identify the face area in each frame and obtain facial feature points; The to-be-detected image sequence is obtained based on the facial feature points.

3. The method for detecting face forgery with reduced attribute bias according to claim 1, characterized in that: The image sequence to be detected includes several picture sequences.

4. The method for detecting face forgery with reduced attribute bias according to claim 1, characterized in that: The training process of the forgery detection model includes: Obtain a training data set; the training data set includes a number of face videos and identity information of each face; wherein each video includes a number of face attribute labels; Segmenting the face images in the training data set into a plurality of non-overlapping image windows as image sequences; Inputting the image sequence into the forgery detection model, and loading the pre-trained texture analysis model, attribute analysis model and spatiotemporal feature aggregator to establish a model loss function; Processing the model loss function using a stochastic gradient descent algorithm to minimize the model loss function; The gradient of the model loss function relative to the model parameters is calculated by a back propagation algorithm, and the parameters of the forgery detection model are updated according to the gradient to obtain a trained forgery detection model.

5. The method for detecting face forgery with reduced attribute bias according to claim 1, characterized in that: The attribute analysis model is obtained by adding a first global average pooling layer and a fully connected layer to a ResNet34 network without loading pre-training weights, and loading the first pre-training weights.

6. A method for detecting face forgery with reduced attribute bias according to claim 5, characterized in that: The expression of the loss function used in the attribute analysis model is as follows: Wherein, N represents the number of frames of the image sequence to be detected, F i,a represents the features extracted for the forgery detection task, F i, ' a Represents the features extracted for the facial attribute detection task.

7. The method for detecting face forgery with reduced attribute bias according to claim 1, characterized in that: The texture analysis model is obtained by adding a bottleneck module and a second global average pooling layer to a ResNet101 network without loading pre-training weights, and loading the second pre-training weights.

8. The method for detecting face forgery with reduced attribute bias according to claim 7, characterized in that: The expression of the loss function used by the texture analysis model is as follows: Wherein, N represents the number of frames of the image sequence to be detected, P i Represents the original label of the face image, P i, ' t Represents the prediction results of the texture analysis model for forgery detection.

9. The method for detecting face forgery with reduced attribute bias according to claim 1, characterized in that: The loss function used by the spatiotemporal feature aggregator is expressed as follows: Wherein, N represents the number of frames of the image sequence to be detected, P i Represents the original label of the face image, P i, ' out Represents the prediction results of the forgery detection model.

10. A face forgery detection system that reduces attribute bias, characterized in that: include: A video acquisition module is used to acquire the video to be detected; An image sequence acquisition module is used to preprocess the video to be detected to obtain an image sequence to be detected; The pre-trained forgery detection models include: attribute analysis model, texture analysis model, and spatiotemporal feature aggregator; The pre-trained forgery detection model is used to extract facial attribute features from the image sequence to be detected using the attribute analysis model and to extract facial texture features from the texture analysis model; the facial attribute features and the facial texture features are fused by the spatiotemporal feature aggregator to obtain a detection result of the image sequence to be detected; and the detection result indicates whether the face in the video to be detected is forged.

Images