False face picture detection method based on misleading learning

Abstract

The invention provides a false face image detection method based on misleading learning, and the method comprises the steps: S1, constructing a priori knowledge obtaining module, and constructing a basic counterfeit artifact discrimination capability for a counterfeit feature extractor and a detector; s2, establishing a misleading learning knowledge flow and a bias data knowledge flow; s3, introducing a single-channel attention fusion network, and enabling the model to adaptively select required multi-scale counterfeit latent layer features and multi-scale real picture latent layer features; s4, constructing a high-pass filter specially used for misleading learning, and carrying out preprocessing operation on the forged image; s5, misleading learning loss is introduced, and training constraint is carried out on the forged feature extractor; and S6, performing fine adjustment on the forged feature extractor and an adaptive device. According to the method, the model gets rid of feature dependence bias generated by interference of irrelevant semantic features in the picture on the model as far as possible, the method has excellent detection capability for forged pictures of different race statistical tags, and the optimal performance is achieved under intra-domain and cross-domain tests.

Description

Technical Field

[0001] The present invention relates to the technical field of face image detection, and in particular to a false face image detection method based on misleading learning. Background Art

[0002] Nowadays, with the development of deep learning technology, fake face images generated by deep neural network technology are emerging in an endless stream. These false information can mislead the public through news, advertisements and other social channels, posing serious security risks to society. Especially in the context of rapid iteration and widespread use of face recognition technology, it is particularly important to quickly and accurately identify fake face images. Existing fake face image detection technologies can be roughly divided into the following two categories: data enhancement methods based on original images and methods for convolutional neural networks or multi-head attention networks. However, these methods still face two huge challenges: poor generalization ability of the model and huge unfairness in statistical label detection for different races. Summary of the invention

[0003] In view of the above-mentioned deficiencies of the prior art, the purpose of the present invention is to provide a method for detecting fake face images based on misleading learning, and a new method called misleading learning is proposed. In the training stage, the multi-scale latent features of the fake image are adaptively combined with the multi-scale latent features of the real image, and the detector is misjudged by combining specific fake images with statistical labels of different races, so as to improve the sensitivity of the forged feature extractor to subtle forged textures, and at the same time improve the generalization and fair generalization capabilities of the model.

[0004] A false face image detection method based on misleading learning comprises the following steps:

[0005] S1: Construct a priori knowledge acquisition module to build the basic forgery artifact identification capability of the forgery feature extractor and detector;

[0006] S2: Build misleading learning knowledge flow and biased data knowledge flow;

[0007] S3: Introducing a single-channel attention fusion network to enable the model to adaptively select the required multi-scale forged latent features and multi-scale real image latent features;

[0008] S4: Construct a high-pass filter specifically for misleading learning and perform preprocessing operations on forged images;

[0009] S5: Introduce misleading learning loss to constrain the training of the forged feature extractor;

[0010] S6: Fine-tuning the feature extractor plus the adaptor for forgery.

[0011] Furthermore, S1 specifically includes the following steps:

[0012] S11: For the construction of the prior knowledge acquisition module, two backbone network architectures with the same structural design are selected as face forgery feature extractors D and D e and real face detector D a ;

[0013] S12: Build training dataset:

[0014]

[0015] Where X i Indicates picture, Y i represents the identification label, 0: real image, 1: fake image, n represents the number of images;

[0016] S13: The prior knowledge acquisition module is constructed as follows:

[0017] f i e =D e (X i )

[0018] f i a =D a (X i )

[0019] where f i e 、f i a They represent the latent features extracted from the fake face feature extractor and the real face detector respectively.

[0020] Furthermore, S2 specifically includes the following steps:

[0021] S21: In the misleading learning knowledge flow, a pair of images (fake image, real image) as input, where As the input of the fake feature extractor, As the input of the real image detector; the specific process is as follows:

[0022]

[0023] Where D hy represents the hybrid model of dual knowledge flows, f i hy Represents the mixed features obtained by the misleading learning knowledge flow through the mixed model;

[0024] S22: The paranoid data knowledge flow is as follows:

[0025]

[0026] in represents a pair of images (real image, real image) input to the dual knowledge stream hybrid model, Represents the mixed features obtained by the mixed model of the paranoid data knowledge flow.

[0027] Furthermore, S3 specifically includes the following steps:

[0028] S31: The multi-scale features captured by the forged feature extractor are represented as V e , the multi-scale features captured by the real image detector are represented as V a , through a single-channel attention fusion network F sc The obtained fusion feature is represented as V h ; The details are as follows:

[0029] V h =F hy (V a ,F sc (V e ))

[0030] where F hy is the feature mixing module;

[0031] S32: The single-channel attention fusion network firstly follows V e Compute the attention map sc∈R 1*1*c , through the function θ e and θ a Calculate V e The maximum and average values ​​of each channel are as follows:

[0032] sc=sigmoid(conv(θ e (V e ))+conv(θ a (V e )))

[0033] Where sigmoid represents the activation function, and conv represents the convolution block;

[0034] S33: Use the obtained sc attention map to assign adaptive weights to the channels of the multi-scale forged features, as follows:

[0035] V sc =V e +sc*V e

[0036] Where V sc Represents the multi-scale forged latent features enhanced by the attention map;

[0037] S34: Through the feature mixing module F hy Mixed V sc and V a , as follows:

[0038] V h =F hy (V a ,F sc (V e ))=conv(concat(V sc ,V a ))

[0039] Among them, conv is the convolution block and concat is the feature concatenation operation.

[0040] Furthermore, S4 specifically includes the following steps:

[0041] S41: Preprocess the image that passes through the forged feature extractor using the SRM high-pass filter;

[0042] S42: In the misleading learning stage, the SRM convolution kernel is set to the gradient learnable mode and adaptively updated through the misleading process.

[0043] Furthermore, two loss functions are designed in S5 for the misleading learning stage: misleading classification loss and misleading contrast loss;

[0044] S51: Misleading classification loss, as follows:

[0045] L c =L ce (h c (f i hy ),Y f )

[0046] L c_n =L ce (h c (f i hy ),Y k )

[0047] Where L c refers to the classification loss that misleads the learning knowledge flow, L c_n refers to the classification loss of the biased data knowledge flow, L ce represents the cross entropy loss function, h c represents the classification head, Y f =fake, Y k ∈(real, fake);

[0048] S52: The total misleading classification loss is expressed as follows:

[0049] L c_m =α1*L c +α2*L c_n

[0050] Where α1 and α2 represent hyperparameters respectively;

[0051] S52: Misleading contrast loss is as follows:

[0052] L con =max([m+‖f anchor -f + ‖2-‖f anchor -f - ‖2],0)

[0053] where f anchor Represents the anchor feature of the image, f + represents a positive sample, f - represents negative samples, and m represents the hyperparameter that measures the distance between anchor features, positive sample features, and negative sample features;

[0054] S53: The total loss function is expressed as follows:

[0055] L=L c_m +βL con

[0056] Where β is a hyperparameter.

[0057] Furthermore, S6 specifically includes the following steps:

[0058] S61: Guide a feature adapter by using a face forgery feature extractor D trained with a misleading learning loss L e The extracted forged features are fine-tuned for the entire dataset; the adapter uses the Xception model to reduce the misleading learning high-pass filter β mis_srm Information loss during the filtering process;

[0059] S62: The final detector structure for inference includes a face forgery feature extractor D trained with a misleading learning loss L e And the fine-tuned adaptor.

[0060] The beneficial effects of the present invention are:

[0061] 1. Aiming at the challenges of difficulty in generalization and lack of fair generalization ability faced by existing deep fake image detectors, this paper proposes a misleading learning training strategy for the first time.

[0062] 2. Aiming at the misleading learning training strategy, the present invention proposes a multi-scale latent feature attention fusion module, so that the latent features can adaptively allocate the importance parameters of each channel during the training process.

[0063] 3. Aiming at misleading learning training strategies, the present invention designs two knowledge data streams to prevent the model from relying too much on specific falsification methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 It is the overall block diagram of the present invention;

[0065] Figure 2 It is a single-scale multi-head self-attention processing module;

[0066] Figure 3 It is a dual knowledge stream hybrid model;

[0067] Figure 4 It is a single-channel attention fusion network;

[0068] Figure 5 Fine-tune the graph for the adaptor. DETAILED DESCRIPTION

[0069] The present invention is further described below in conjunction with specific embodiments, but the protection scope of the present invention is not limited thereto.

[0070] A false face image detection method based on misleading learning, the overall block diagram is as follows Figure 1 As shown, the main steps are as follows:

[0071] S1: Construct a priori knowledge acquisition module to build the basic forgery artifact identification capability of the forgery feature extractor and detector.

[0072] The purpose of constructing the prior knowledge acquisition module is to establish a learning goal orientation for the misleading learning process and prevent the overfitting phenomenon caused by the model's excessive reliance on specific misleading features.

[0073] S11: For the construction of the prior knowledge acquisition module, the present invention selects two backbone network architectures with the same structural design as the face forgery feature extractor D e and real face detector D a .

[0074] S12: Build training dataset:

[0075]

[0076] Where X i Indicates the image (real image, fake image), Y i represents the identification label (0: real image, 1: fake image), and n represents the number of images.

[0077] S13: The construction of the prior knowledge acquisition module can be expressed as follows:

[0078] f i e =D e (X i )

[0079] f i a =D a (X i )

[0080] where f i e 、f i a They represent the latent features extracted from the fake face feature extractor and the real face detector respectively.

[0081] S2: Build two knowledge flows: Figure 2 As shown, they are the misleading learning knowledge flow and the biased data knowledge flow respectively.

[0082] S21: In the misleading learning knowledge flow, a pair of images (fake image, real image) as input, where As the input of the fake feature extractor, As the input of the real image detector. The specific process is as follows:

[0083]

[0084] Where D hy Represents a hybrid model of dual knowledge flows, such as Figure 3 As shown. Its main structure is: Prior knowledge acquisition module trained real face detector D a And keep the parameters frozen, the second prior knowledge acquisition module is trained face forgery feature extractor D e , it should be noted that its parameters are trainable. i hy Represents the mixed features obtained by the misleading learning knowledge flow through the mixed model.

[0085] S22: If only the misleading learning knowledge flow is used, the extractor will often rely on a single forgery trace to achieve the purpose of misleading the detector. Therefore, the present invention designs the paranoid data knowledge flow as follows:

[0086]

[0087] in represents a pair of images (real image, real image) input to the dual knowledge stream hybrid model, Represents the mixed features obtained by the mixed model of the paranoid data knowledge flow.

[0088] S3: Construct a single-scale multi-head self-attention processing module and introduce a single-channel attention fusion network to enable the model to adaptively select the required multi-scale forged latent features and multi-scale real image latent features.

[0089] This paper designs a unique single-channel attention fusion network such as Figure 4 As shown, three forged latent features of different scales are captured and adaptively combined with three real image features of the same scale.

[0090] S31: The multi-scale features captured by the forged feature extractor are represented as V e , the multi-scale features captured by the real image detector are represented as V a , through a single-channel attention fusion network F sc The obtained fusion feature is represented as V h The details are as follows:

[0091] V h =F hy (V a ,F sc (V e ))

[0092] where F hy It is a feature mixing module.

[0093] S32: The single-channel attention fusion network firstly follows V e Compute the attention map sc∈R 1*1*c , through the function θ e and θ a Calculate V e The maximum and average values ​​of each channel are as follows:

[0094] sc=sigmoid(conv(θ e (V e ))+conv(θ a (V e )))

[0095] Where sigmoid represents the activation function and conv represents the convolution block.

[0096] S33: Use the obtained sc attention map to assign adaptive weights to the channels of the multi-scale forged features, as follows:

[0097] V sc =V e +sc*V e

[0098] Where V sc Represents the multi-scale forged latent features enhanced by the attention map. Each element of the attention map represents V e The amount of information contained in each channel of the forged features. Therefore, the forged semantic channel with a greater amount of information is assigned a larger weight through the attention map to obtain enhancement, and vice versa.

[0099] S34: Through the feature mixing module F hy Mixed V sc and V a , as follows:

[0100] V h =F hy (V a ,F sc (V e ))=conv(concat(V sc ,V a ))

[0101] Among them, conv is the convolution block and concat is the feature concatenation operation.

[0102] S4: Construct a high-pass filter specifically for misleading learning and perform pre-processing operations on forged images.

[0103] S41: The SRM kernel is used to calculate the residual between the pixel value and the predicted value derived from its neighboring pixels. The image that passes through the forged feature extractor is preprocessed using the SRM high-pass filter.

[0104] S42: In the misleading learning stage, the SRM convolution kernel is set to the gradient learnable mode and adaptively updated through the misleading process. The details are as follows:

[0105]

[0106] Among them, β srm refers to the initialization kernel before adaptive update, and L refers to the total misleading learning loss, which is described in S5. γ is used to weight the regularization term of the loss function. β mis_srm It represents the convolution kernel weight obtained after adaptive update.

[0107] S5: Introduce misleading learning loss to constrain the training of the fake feature extractor.

[0108] The present invention designs two loss functions for the misleading learning stage: misleading classification loss and misleading contrast loss.

[0109] S51: Misleading classification loss, as follows:

[0110] L c =L ce (h c (f i hy ),Y f )

[0111] L c_n =L ce (h c (f i hy ),Y k )

[0112] Where L c refers to the classification loss that misleads the learning knowledge flow, L c_n refers to the classification loss of the biased data knowledge flow, L ce represents the cross entropy loss function, h c represents the classification head, Y f =fake, Y k ∈(real,fake).

[0113] S52: The total misleading classification loss is expressed as follows:

[0114] L c_m =α1*L c +α2*L c_n

[0115] Where α1 and α2 represent hyper parameters.

[0116] S52: A misleading contrast regularization loss is used to emphasize the difference between the misleading mixed features and the original image, thereby enhancing the face forgery feature extractor. The misleading contrast loss is as follows:

[0117] L con =max([m+‖f ancho rf + ‖2-‖f anchor -f - ‖2],0)

[0118] where f anchor Represents the anchor feature of the image, f + represents a positive sample, f - represents a negative sample, and m represents a hyperparameter that measures the distance between anchor features, positive sample features, and negative sample features. + and f anchor represents misleading mixed features, then f - Represents the original image features processed by a real face detector.

[0119] In particular, for both data streams, Lcon The tendency is to make the encoder learn the same but slightly different fake feature representations as much as possible.

[0120] S53: The total loss function can be expressed as follows:

[0121] L=L c_m +βL con

[0122] Where β is a hyperparameter.

[0123] S6: Fine-tuning the feature extractor plus the adaptor for forgery.

[0124] S61: This paper introduces a new feature adapter by using a face forgery feature extractor D trained with a misleading learning loss L e The extracted forged features are fine-tuned for the entire dataset. The adapter uses the Xception model to reduce the misleading learning high-pass filter β mis_srm Information that occurs during the filtering process is lost. Figure 5 shown.

[0125] S62: The final detector structure for inference includes a face forgery feature extractor D trained with a misleading learning loss L e And the fine-tuned adaptor. Figure 1 The misleading learning inference process is shown below. The resulting forged vector has a higher sensitivity to forged textures.

[0126] In order to verify the generalization ability of the model proposed in this invention, it was tested on four widely used large-scale datasets: FaceForensics++, DeepfakeDetection, Deepfake Detection Challenge and Celeb-DF. Among them, FaceForensics++ is the main training dataset, which contains five forgery techniques for generating fake face images: Deepfakes, Face2Face, FaceSwap, NerualTexture and FaceShifter. It should be noted that FaceForensics++ has 3 compressed versions (the high-compression version is more challenging for fake face image detectors): original, high-compression version, and low-compression version. The present invention uses the low-compression version by default.

[0127] Four widely used convolutional neural networks are used to verify the effectiveness of the proposed model (Xception, Resnet-34, Resnet-50, Efficientnet-b4). At the same time, the fairness and generalization ability are compared with existing advanced detectors. By default, all methods use Xception as the backbone network model.

[0128] Previous work has demonstrated excellent generalization performance on in-domain datasets. However, there are still significant challenges in achieving fairness. The main reason is that harmful semantics can have a negative impact on the fairness performance of the detector. Table 1 shows that the method of the present invention can refine the sensitivity of the detector to fake semantics and reduce the hyperparameter problem caused by the overfitting of the model to special fake semantics and demographic semantics. Compared with existing methods, the results show that the method of the present invention can improve the fairness ability of the model.

[0129] Table 1 ff++ domain data set test results

[0130]

[0131] In order to evaluate the generalization ability of the model of the present invention on cross-domain datasets. Table 2 shows the comparison results. All models were trained on FF++ and verified on the other three datasets. The results show that the method of the present invention significantly improves the performance level by 6% and achieves higher generalization performance.

[0132] Table 2 Test results of out-of-domain datasets

[0133]

[0134] The above is only a preferred embodiment of the present invention, which is used to illustrate the technical solution of the present invention, but not to limit the present invention. It should be pointed out that for ordinary technicians in this field, without departing from the concept of the present invention, some of the technical features therein are modified and replaced, and they will not depart from the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be based on the attached claims.

Claims

1. A false face image detection method based on misleading learning, characterized in that: The steps include: S1: Construct a priori knowledge acquisition module to build the basic forgery artifact identification capability of the forgery feature extractor and detector; S2: Build misleading learning knowledge flow and biased data knowledge flow; S3: Introducing a single-channel attention fusion network to enable the model to adaptively select the required multi-scale forged latent features and multi-scale real image latent features; S4: Construct a high-pass filter specifically for misleading learning and perform preprocessing operations on forged images; S5: Introduce misleading learning loss to constrain the training of fake feature extractors; S6: Fine-tuning the feature extractor plus the adaptor for forgery.

2. According to claim 1, a false face image detection method based on misleading learning is characterized in that: S1 specifically includes the following steps: S11: For the construction of the prior knowledge acquisition module, two backbone network architectures with the same structural design are selected as face forgery feature extractors D and D e and real face detector D a ; S12: Build training dataset: Where X i Indicates picture, Y i represents the identification label, 0: real image, 1: fake image, n represents the number of images; S13: The prior knowledge acquisition module is constructed as follows: f i e =D e (X i ) f i a =D a (X i ) where f i e 、f i a They represent the latent features extracted from the fake face feature extractor and the real face detector respectively.

3. The method for detecting false face images based on misleading learning according to claim 2, characterized in that: S2 specifically includes the following steps: S21: In the misleading learning knowledge flow, a pair of images (fake image, real image) as input, where As the input of the fake feature extractor, As the input of the real image detector; the specific process is as follows: Where D hy represents the hybrid model of dual knowledge flows, f i hy Represents the mixed features obtained by the misleading learning knowledge flow through the mixed model; S22: The paranoid data knowledge flow is as follows: in represents a pair of images (real image, real image) input to the dual knowledge stream hybrid model, Represents the mixed features obtained by the mixed model of the paranoid data knowledge flow.

4. The method for detecting false face images based on misleading learning according to claim 3, characterized in that: S3 specifically includes the following steps: S31: The multi-scale features captured by the forged feature extractor are represented as V e , the multi-scale features captured by the real image detector are represented as V a , through a single-channel attention fusion network F sc The obtained fusion feature is represented as V h ; The details are as follows: V h =F hy (V a ,F sc (V e )) where F hy is the feature mixing module; S32: The single-channel attention fusion network firstly follows V e Compute the attention map sc∈R 1*1*c , through the function θ e and θ a Calculate V e The maximum and average values ​​of each channel are as follows: sc=sigmoid(conv(θ e (V e ))+conv(θ a (V e ))) Where sigmoid represents the activation function, and conv represents the convolution block; S33: Use the obtained sc attention map to assign adaptive weights to the channels of the multi-scale forged features, as follows: V sc =V e +sc*V e Where V sc Represents the multi-scale forged latent features enhanced by the attention map; S34: Through the feature mixing module F hy Mixed V sc and V a , as follows: V h =F hy (V a ,F sc (V e ))=conv(concat(V sc ,V a )) Among them, conv is the convolution block and concat is the feature concatenation operation.

5. The method for detecting false face images based on misleading learning according to claim 4, characterized in that: S4 specifically includes the following steps: S41: Preprocess the image that passes through the forged feature extractor using the SRM high-pass filter; S42: In the misleading learning stage, the SRM convolution kernel is set to the gradient learnable mode and adaptively updated through the misleading process.

6. The method for detecting false face images based on misleading learning according to claim 5, characterized in that: In S5, two loss functions are designed for the misleading learning stage: misleading classification loss and misleading contrast loss; S51: Misleading classification loss, as follows: L c =L ce (h c (f i hy ),Y f ) L c_n =L ce (h c (f i hy ),Y k ) Where L c refers to the classification loss that misleads the learning knowledge flow, L c_n refers to the classification loss of the biased data knowledge flow, L ce represents the cross entropy loss function, h c represents the classification head, Y f =fake, Y k ∈(real, fake); S52: The total misleading classification loss is expressed as follows: <h2 style=";text-align:left;direction:ltr">L<h2 style=";text-align:left;direction:ltr"> c_m <h2 style=";text-align:left;direction:ltr"> =a1*L<h2 style=";text-align:left;direction:ltr"> c <h2 style=";text-align:left;direction:ltr"> +a2*L<h2 style=";text-align:left;direction:ltr"> c_n Where α1 and α2 represent hyperparameters respectively; S52: Misleading contrast loss is as follows: L con =max([m+||f anchor -f + ||2-||f anchor -f - ||2],0) where f anchor Represents the anchor feature of the image, f + represents a positive sample, f - represents negative samples, and m represents the hyperparameter that measures the distance between anchor features, positive sample features, and negative sample features; S53: The total loss function is expressed as follows: L=L c_m +βL con Where β is a hyperparameter.

7. The method for detecting false face images based on misleading learning according to claim 6, characterized in that: S6 specifically includes the following steps: S61: Guide a feature adapter by using a face forgery feature extractor D trained with a misleading learning loss L e The extracted forged features are fine-tuned for the entire dataset; the adapter uses the Xception model to reduce the misleading learning high-pass filter β min_srm Information loss during the filtering process; S62: The final detector structure for inference includes a face forgery feature extractor D trained with a misleading learning loss L e And the fine-tuned adaptor.

Images