Incremental Learning Method and System for Detecting Forged Facial Image Videos

Abstract

The present invention proposes an incremental learning method and system for detecting forged facial image videos, comprising: obtaining a forged facial image detection model to be incrementally learned as a base model, the base model comprising a feature extractor for extracting image features and a classifier for assigning forged facial image detection categories to the image features; obtaining training images labeled with forged categories; creating a new feature extractor within the base model; creating a new auxiliary classifier within the base model, obtaining auxiliary classification results and auxiliary losses for the training images, and constructing a classification loss based on the classification results of the training images by the classifier; updating weight parameters of the classifier and the new feature extractor using the classification loss and auxiliary losses; and continuing until the total loss converges or a specified number of iterations is reached. The present invention uses incremental learning to process forged facial data that constantly appears in real-world scenarios, thereby enhancing the classifier's discriminative capabilities.

Description

Technical Field

[0001] The present invention relates to the technical field of face forgery detection in image recognition and classification, and in particular to an incremental learning method and system for detecting forged face image videos. Background Art

[0002] Existing counterfeit detection technologies mainly include but are not limited to the following four categories:

[0003] (1) Methods based on deep neural network feature extraction. This type of method directly feeds real samples and synthetic samples into the neural network for supervised training, hoping that the network will learn the different intrinsic distributions of positive and negative samples and output prediction results through a binary classifier. This type of method often relies on the network itself. Networks with strong feature extraction capabilities usually have better classification results. Currently, the most popular basic networks such as ResNet, CapsuleNet, XceptionNet and EfficientNet have been used for face forgery detection tasks. These methods have achieved good detection results in intra-domain tests, but the detection performance has dropped significantly in inter-domain tests, which is manifested as insufficient generalization performance. Therefore, many works have been proposed to improve the generalization performance of the network. The common idea is to find or strengthen the forgery traces in the synthetic data for discrimination. The generalization ability of the network is improved by mining the intrinsic features in the channel difference image and the spectrum image; the network structure Gram-Net that uses global image texture representation for forgery detection has improved the generalization ability; and the method of multi-level feature global consistency has improved the generalization ability of the network. These works have promoted the development of forgery detection technology, but have not yet solved the problem of insufficient generalization ability.

[0004] (2) Detection methods based on image context space. This type of method assumes that the synthetic image only tampered with the facial area in the image, while other parts of the image (such as the background and torso) have not been changed. In theory, there are inevitable differences between the real image and the synthetic image. The real image is continuous, while the synthetic image is spliced. This allows for authenticity detection. Synthetic face images have a fusion boundary, while real images do not have such a boundary. A Face-X-Ray method is proposed to detect boundary fusion traces; an attention mechanism is used to locate the forged area in the face image and use it to improve the final classification task; a three-branch multi-task learning framework that integrates tampered area location, tampered edge extraction, and authenticity discrimination classification is proposed, and the tampered edge is thickened to assist the final classification. The detection method based on image context space has good detection performance for images with only local tampering, and usually has better generalization performance than the method of directly using neural networks for feature extraction. However, for the overall synthetic image, this type of method has poor detection effect.

[0005] (3) Detection method based on video temporal information. The essence of video is the rapid switching of frames. Most existing face forgery videos forge and process each frame separately, and then compress and encode them into the final video. Therefore, there is often a great correlation between adjacent frames, and forged videos usually have temporal and spatial differences between adjacent frames. The existing technology proposes a time-aware pipeline, which first uses CNN (convolutional neural network) to extract frame-level features, then sends the features to RNN (recurrent neural network) for training, and finally uses the trained RNN for discrimination; proposes the concept of visual flow vector field, combines the inter-frame prediction error with the long short term memory (LSTM) network, and obtains good detection results based on VGG-16; proposes a dual-branch network that combines frequency domain features and RGB image (RGB three-channel) features, uses group convolution to integrate the features of each branch and inputs them into LSTM for detection; the amplitude of facial movement in real and synthetic videos is quite different, and uses InceptionV3 to extract features and combines LSTM to extract temporal information, achieving good detection results. The detection method based on video timing information has promoted the practical application of face forgery detection technology. The common idea is to use information such as inconsistent character motion between adjacent frames, abnormal jitter of color and texture to detect forgery.

[0006] (4) Detection methods based on physiological signals. This type of method assumes that real people have rich physiological characteristics, such as heart rate, pulse, and blinking, while computer-synthesized people often do not have these characteristics or their physiological characteristics are inconsistent. In view of the strong time dependence of human eye blinking, a method based on blink detection is proposed to identify forged videos; a method for monitoring central expression and head posture is proposed; features of the eyes, teeth, and facial contours are extracted to detect the authenticity of the video; a method for monitoring pulse Photoplethysmogram signals is proposed; and a method for using heartbeat signals to detect the authenticity of human objects in the video is proposed. The detection method based on physiological signals can be separated from the video carrier (such as frequency, pixels, etc.) and use the continuous physiological signals of people for authenticity detection. It has achieved competitive cross-database test accuracy among existing methods, but this method has high requirements for the data set and is easily affected by noise and blur. It is mainly used for detecting forged videos of important people.

[0007] As deepfake technology continues to advance, the amount of data from different forgery types is increasing. Existing methods have the following main shortcomings when dealing with new forgery types: 1) They have excellent detection capabilities on data of known forgeries, but poor detection capabilities on data of unknown forgeries, manifesting as insufficient generalization performance; 2) If a model is required to have good detection capabilities on new data (unknown forgeries), it must be trained from scratch using both old data (known forgeries) and new data. This typically consumes significant storage and computing resources and relatively hinders the model's ability to learn from new data in real time. Summary of the Invention

[0008] The present invention aims to overcome the problems of the prior art, such as insufficient generalization performance when faced with new forged data and the high storage and computing resource consumption associated with training models from scratch. The present invention proposes an incremental learning-based face forgery detection technology and system. Specifically, the present invention proposes an incremental learning method for detecting forged facial images and videos, which includes:

[0009] Step 1: Obtain the forged face image detection model to be incrementally learned as a basic model, which includes a feature extractor Φ for extracting image features. t-1 , and a classifier H that assigns fake face image detection categories to image features t ;

[0010] Step 2: Obtain training images that have been labeled with forged categories; create a new feature extractor F in the basic model i ; At the current time t, the feature extractor Φ t-1 and the new feature extractor F t Composition feature extractor Φ t ,in For the input training image x, Φ t The extracted feature υ is:

[0011] υ=Φ t (x)=[Φ t-1 (x),F t (x)]

[0012] Step 3: Create a new auxiliary classifier H′ in the basic model t , H′ t The label space is |Y t |+1, including the incremental learning task Y at time t t The categories in the basic model and all the old categories that can be recognized by the basic model, for the input training image x, the auxiliary classifier H′ t The auxiliary classification results are obtained according to the following formula:

[0013] pH′ t(y|x)=Softmax(H′ t (F t (x)))

[0014] The auxiliary classification result and the forged category corresponding to the training image x are used to construct the auxiliary loss L H't ;

[0015] Input the feature υ into the classifier H t :

[0016] pH t (y|x)=Softmax(H t (υ))

[0017] Get the scores of the training image x under each forgery category, and take the forgery category with the highest score as the classification result

[0018]

[0019] in in For all image categories; through the classification results And the forged category corresponding to the training image x constructs the classification loss L C ; The classification loss L C and auxiliary loss L H't Construct the total loss to update the classifier H t And the new feature extractor F i until the total loss converges or reaches the specified number of iterations, and the current basic model is saved as the final recognition model;

[0020] Step 4: Input the image to be forged face image detection into the final recognition model to obtain its recognition result.

[0021] The incremental learning method for detecting forged face image videos, wherein the H t The initial parameters are inherited from H t-1 , H t The output dimension of Φ is continuously updated to match the increased number of forged categories, and its newly added parameters are randomly initialized; t-1 It includes at least one feature extractor, the feature extractor F i The initial weight parameters are inherited from F i-1 , where i represents the i-th incremental learning task.

[0022] The incremental learning method for detecting forged face image videos, wherein the classifier H t For a binary classifier, the current incremental learning task is in represents the real image and forged image under the k-th forgery method at time t, represents the binary classification label under the k-th forgery method at time t, Indicates the multi-classification label under the k-th forgery method at time t; during the training process, the binary classification label Participate in the calculation of loss function, multi-classification label Used by the model to distinguish new categories from old categories; in binary classification learning, the total loss function is composed of the cross entropy loss L of the binary classifier BC and the multi-classification cross entropy loss L of the auxiliary classifier H't constitute:

[0023]

[0024] λ a is a hyperparameter that controls the effect of the auxiliary classifier.

[0025] The incremental learning method for detecting forged face image videos, wherein the classifier H t For multiple classifiers, the current incremental learning task is in represents the real image and forged image under the k-th forgery method at time t, Represents the corresponding multi-classification label; in multi-classification learning, the loss function is composed of a multi-classifier cross entropy loss L MC and the multi-classification cross entropy loss L of the auxiliary classifier H't constitute:

[0026]

[0027] λ b is a hyperparameter that controls the effect of the auxiliary classifier.

[0028] The incremental learning method for detecting forged face image videos, wherein the classifier H t It is a multi-branch classifier composed of multi-classification and binary classification. The current incremental learning task is in represents the real image and forged image under the k-th forgery method at time t, represents the binary classification label under the k-th forgery method at time t, Represents the multi-classification label under the k-th forgery method at time t; the total loss function is composed of the cross entropy loss L of the two classifiers BC , multi-classification cross entropy loss L of the auxiliary classifier H't and the cross entropy loss L of the multi-classifier MC constitute:

[0029]

[0030] λ d is a hyperparameter that controls the effect of the auxiliary classifier, λ c It is a hyperparameter that balances the binary classification loss and the multi-classification loss.

[0031] The present invention also proposes an incremental learning system for detecting forged facial image videos, which includes:

[0032] The initial module is used to obtain the fake face image detection model to be incrementally learned as a basic model, which includes a feature extractor Φ for extracting image features t-1 , and a classifier H that assigns fake face image detection categories to image features t ;

[0033] A new module is created to obtain training images that have been labeled as forged categories; a new feature extractor F is created in this basic model i ; At the current time t, the feature extractor Φ t-1 and the new feature extractor F t Composition feature extractor Φ t ,in For the input training image x, Φ t The extracted feature υ is:

[0034] υ=Φ t (x)=[Φ t-1 (x),F t (x)]

[0035] Training module, used to create a new auxiliary classifier H′ in the basic model t , H′ t The label space is |Y t |+1, including the incremental learning task Y at time t t The categories in the basic model and all the old categories that can be recognized by the basic model, for the input training image x, the auxiliary classifier H′ t The auxiliary classification results are obtained according to the following formula:

[0036] pH′ t (y|x)=Softmax(H′ t (F t (x)))

[0037] The auxiliary classification result and the forged category corresponding to the training image x are used to construct the auxiliary loss L H't ;

[0038] Input the feature υ into the classifier H t :

[0039] pH t(y|x)=Softmax(H t (υ))

[0040] Get the scores of the training image x under each forgery category, and take the forgery category with the highest score as the classification result

[0041]

[0042] in in For all image categories; through the classification results And the forged category corresponding to the training image x constructs the classification loss L C ; The classification loss L C and auxiliary loss L H't Construct the total loss to update the classifier H t And the new feature extractor F i until the total loss converges or reaches the specified number of iterations, and the current basic model is saved as the final recognition model;

[0043] The recognition module is used to input the image to be forged face image detection into the final recognition model to obtain its recognition result.

[0044] The incremental learning system for detecting forged face image videos, wherein the H t The initial parameters are inherited from H t-1 , H t The output dimension of Φ is continuously updated to match the increased number of forged categories, and its newly added parameters are randomly initialized; t-1 It includes at least one feature extractor, the feature extractor F i The initial weight parameters are inherited from F i-1 , where i represents the i-th incremental learning task.

[0045] The incremental learning system for detecting forged face image videos, wherein the classifier H t For a binary classifier, the current incremental learning task is in represents the real image and forged image under the k-th forgery method at time t, represents the binary classification label under the k-th forgery method at time t, Indicates the multi-classification label under the k-th forgery method at time t; during the training process, the binary classification label Participate in the calculation of loss function, multi-classification label Used by the model to distinguish new categories from old categories; in binary classification learning, the total loss function is composed of the cross entropy loss L of the binary classifier BCand the multi-classification cross entropy loss L of the auxiliary classifier H't constitute:

[0046]

[0047] λ a is a hyperparameter that controls the effect of the auxiliary classifier.

[0048] The incremental learning system for detecting forged face image videos, wherein the classifier H t For multiple classifiers, the current incremental learning task is in represents the real image and forged image under the k-th forgery method at time t, Represents the corresponding multi-classification label; in multi-classification learning, the loss function is composed of a multi-classifier cross entropy loss L MC and the multi-classification cross entropy loss L of the auxiliary classifier H't constitute:

[0049]

[0050] λ b is a hyperparameter that controls the effect of the auxiliary classifier.

[0051] The incremental learning system for detecting forged face image videos, wherein the classifier H t It is a multi-branch classifier composed of multi-classification and binary classification. The current incremental learning task is in represents the real image and forged image under the k-th forgery method at time t, represents the binary classification label under the k-th forgery method at time t, Represents the multi-classification label under the k-th forgery method at time t; the total loss function is composed of the cross entropy loss L of the two classifiers BC , multi-classification cross entropy loss L of the auxiliary classifier H't and the cross entropy loss L of the multi-classifier MC constitute:

[0052]

[0053] λ d is a hyperparameter that controls the effect of the auxiliary classifier, λ c It is a hyperparameter that balances the binary classification loss and the multi-classification loss.

[0054] From the above scheme, it can be seen that the advantages of the present invention are:

[0055] (1) Generalization

[0056] We conduct experiments on the FaceForensics++ extended set (Table 1) defined in the experiment to compare the generalization of the traditional XceptionNet and the XceptionNet using incremental learning (the method of the present invention). The experimental results are as follows: Figure 1 The results show that the traditional XceptionNet has good prediction ability on known types of data, but its ACC (accuracy) drops significantly on unknown types of data, which can be attributed to insufficient generalization performance. However, the XceptionNet using incremental learning (the method in this paper) still maintains good discrimination ability on other tasks.

[0057] (2) Resource usage

[0058] We conducted experiments on the experimentally defined ForgeryNet augmented set (Table 2), comparing the resource usage of traditional XceptionNet and XceptionNet using incremental learning (the method in this paper). The experimental results are shown in Table 3. The results show that the traditional XceptionNet achieved an average ACC of 98.07%. Although it solved the problem of catastrophic forgetting, this method was inefficient, requiring 132 minutes of training time and consuming 210,000 images of memory. In contrast, XceptionNet using incremental learning achieved an average ACC of 96.16% with only 45 minutes of training time and consuming 8,024 images of memory.

[0059]

[0060] Table 1

[0061]

[0062] Table 2

[0063]

[0064] Table 3 BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 This is a comparison chart of generalization experiment results;

[0066] Figure 2 It is the overall flow chart of the method of the present invention;

[0067] Figure 3 Flowchart of feature extraction phase;

[0068] Figure 4 Schematic diagram of the learning process of the classifier of the present invention. DETAILED DESCRIPTION

[0069] While conducting research on face forgery detection, the inventors of the present invention discovered that most existing face forgery detection methods use a single training session to infer or predict a specific "fingerprint" of a forged face for authenticity detection. However, face forgery technology is still advancing, and it is difficult for the detection network to fully learn universally effective discriminant features in a single training session. Therefore, when faced with new forged data, existing methods mostly train the model from scratch, but this consumes a large amount of storage and computing resources and relatively hinders the model's ability to learn new data in real time. Through research on incremental learning, the inventors of the present invention discovered that incremental learning (iCaRL) can be used to process forged face data that constantly appears in real-world scenarios. By combining iCaRL with authenticity detection (real images / images generated by GAN (generative adversarial network)), good detection results were achieved. The present invention improves on the incremental learning algorithm DER, uses the XceptionNet backbone network, and designs three classification learning systems to enhance the classifier's discriminative ability.

[0070] In order to achieve the above technical effects, the present invention proposes the following key technical points:

[0071] Key point 1: Incremental learning; Technical effect: As new fake face data continues to emerge, the model can continuously learn new knowledge from the new data;

[0072] Key point 2, face forgery detection; technical effect, the model can accurately distinguish between forged faces and real faces.

[0073] In order to make the above features and effects of the present invention more clearly understood, embodiments are given below and described in detail with reference to the accompanying drawings.

[0074] The overall process of the method of the present invention is as follows Figure 2 As shown in the figure, we use incremental learning to continuously train on newly emerging forgery types, and ultimately the model can maintain good forgery detection capabilities on both new and old data. Our method is divided into the following two stages:

[0075] (1) Feature extraction stage.

[0076] The process at this stage is as follows Figure 3 As shown. A new recognition task i is constructed for a new set of forged images, and the model creates a new feature extractor F for the task. i , in order to adapt more quickly, F i The weight parameters are inherited from F i-1 The new recognition task i can be a multi-forgery category recognition task. At time t, the old feature extractor Φ t-1 and the new feature extractor F tComposition of super feature extractor Φ t ,in For input x, Φ t The extracted feature υ is:

[0077] υ=Φ t (x)=[Φ t-1 (x),F t (x)] (1)

[0078] The feature υ will be input into the classifier for classification. In order to reduce catastrophic forgetting, the feature extractor Φ is frozen at time t. t-1 , because it captures the intrinsic structure of the previous data. Specifically, at time t, the super feature extractor Φ t-1 The parameters related to batch normalization are not updated.

[0079] (2) Classifier learning stage

[0080] This stage is mainly composed of the classifier H t and auxiliary classifier H′ t Composition, auxiliary classifier is used in the whole system (feature extractor F i , classifier H t ), the labels of the auxiliary classifier are the various categories of the new task and the old categories, and all the old categories are regarded as one category. For example, if the new task has three categories A, B, and C, then the classification criteria of the auxiliary classifier are A, B, C and the old categories. In this way, the auxiliary loss is combined to ensure that the auxiliary classifier H′ t The classification is accurate. But the classifier H t The classification standard is all category spaces That is, the auxiliary classifier treats all old classes as one category, but the classifier H t no.

[0081] Specifically, the auxiliary classifier H′ t Only used in the training phase, it is used to assist H in the training phase t Faster convergence, improved training efficiency, H′ t It is used to constrain the network to focus on learning new features for new tasks. The label space is |Y t |+1, including the new task Y at time t t The categories in and all old categories, where all old categories are regarded as one category. For the input x at time t, H′ t Make the following predictions:

[0082] pH′ t (y|x)=Softmax(H′ t (F t (x))) (2)

[0083] Where p and (y|x) mean the score p of input x in each category y.

[0084] Then formula (2) will be used to calculate the auxiliary loss For the feature υ obtained in the previous stage, the classifier H t Make the following predictions:

[0085] pH t (y|x)=Softmax(H t (υ)) (3)

[0086] Formula (3) obtains the category score of input x, and the category with the highest score is taken as the final prediction of the model:

[0087]

[0088] In formula (4), H t The parameters are inherited from H t-1 , to preserve old knowledge, H t The output dimension of is continuously updated to match the increasing number of categories, and its newly added parameters are randomly initialized, where Indicates the final prediction category of the model. The following will introduce H in detail t There are several forms. In actual operation, you can choose only one of the following three forms according to your needs:

[0089] ① Binary classification learning

[0090] In the binary classification learning system, this paper regards the entire problem as a binary classification problem, with the following structure: Figure 4 (a) As shown in the figure. The label space only has two labels: true and false, representing real faces and forged faces respectively. Since there are only two labels: true and false, it is impossible to perform incremental learning training during training. Therefore, this paper designs multi-classification labels for different data according to different forgery methods to assist training. Specifically, for the upcoming new task Y t , which has a form in represents the real image and forged image under the k-th forgery method at time t, represents the binary classification label under the k-th forgery method at time t, Indicates the multi-classification label under the k-th forgery method at time t. In particular, during the training process, there are only two classification labels Participated in the calculation of loss function, multi-classification label It is only used for the model to distinguish new categories from old categories. In binary classification learning, the loss function is composed of the cross entropy loss L of a binary classifier. BC and the multi-classification cross entropy loss L of the auxiliary classifierH't constitute:

[0091]

[0092] λ a is a hyperparameter that controls the effect of the auxiliary classifier. It is worth noting that at time t = 1, λ a =0.

[0093] ②Multi-classification learning

[0094] In the multi-classification learning system, this paper regards the entire problem as a multi-classification problem, with the following structure: Figure 4 (b) As shown. This paper conducts multi-classification of different data according to different forgery methods. Specifically, for the upcoming new task Y t , which has a form in represents the real image and forged image under the k-th forgery method at time t, Indicates the corresponding multi-classification label. In particular, this paper considers real images from different forgery methods as different categories. In multi-classification learning, the loss function is composed of a cross entropy loss L of a multi-classifier. MC and the multi-classification cross entropy loss L of the auxiliary classifier H't constitute:

[0095]

[0096] As with the binary classification learning system, λ b is a hyperparameter that controls the effect of the auxiliary classifier.

[0097] ③Multi-branch learning

[0098] In the multi-branch learning system, this paper combines binary classification learning and multi-classification learning. The structure is as follows Figure 4 (c) As in the binary classification learning system, each input image has two labels, one binary label and one multi-class label, but they are both managed by a multi-classifier. Multi-branch learning is an extension of multi-classification learning. This paper adds a binary classification loss constraint based on Equation (6), forming the loss function in the multi-branch learning system:

[0099]

[0100] λ d is a hyperparameter that controls the effect of the auxiliary classifier, λ c It is a hyperparameter that balances the binary classification loss and the multi-classification loss.

[0101] The following is a system embodiment corresponding to the above method embodiment. This embodiment can be implemented in conjunction with the above embodiment. The relevant technical details mentioned in the above embodiment are still valid in this embodiment and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiment.

[0102] The present invention also proposes an incremental learning system for detecting forged facial image videos, which includes:

[0103] The initial module is used to obtain the fake face image detection model to be incrementally learned as a basic model, which includes a feature extractor Φ for extracting image features t-1 , and a classifier H that assigns fake face image detection categories to image features t ;

[0104] A new module is created to obtain training images that have been labeled as forged categories; a new feature extractor F is created in this basic model i ; At the current time t, the feature extractor Φ t-1 and the new feature extractor F t Composition feature extractor Φ t ,in For the input training image x, Φ t The extracted feature υ is:

[0105] υ=Φ t (x)=[Φ t-1 (x),F t (x)]

[0106] Training module, used to create a new auxiliary classifier H′ in the basic model t , H′ t The label space is |Y t |+1, including the incremental learning task Y at time t t The categories in the basic model and all the old categories that can be recognized by the basic model, for the input training image x, the auxiliary classifier H′ t The auxiliary classification results are obtained according to the following formula:

[0107] pH′ t (y|x)=Softmax(H′ t (F t (x)))

[0108] The auxiliary classification result and the forged category corresponding to the training image x are used to construct the auxiliary loss L H't ;

[0109] Input the feature υ into the classifier H t :

[0110] pH t (y|x)=Softmax(H t (υ))

[0111] Get the scores of the training image x under each forgery category, and take the forgery category with the highest score as the classification result

[0112]

[0113] in in For all image categories; through the classification results And the forged category corresponding to the training image x constructs the classification loss L C ; The classification loss L C and auxiliary loss L H't Construct the total loss to update the classifier H t And the new feature extractor F i until the total loss converges or reaches the specified number of iterations, and the current basic model is saved as the final recognition model;

[0114] The recognition module is used to input the image to be forged face image detection into the final recognition model to obtain its recognition result.

[0115] The incremental learning system for detecting forged face image videos, wherein the H t The initial parameters are inherited from H t-1 , H t The output dimension of Φ is continuously updated to match the increased number of forged categories, and its newly added parameters are randomly initialized; t-1 It includes at least one feature extractor, the feature extractor F i The initial weight parameters are inherited from F i-1 , where i represents the i-th incremental learning task.

[0116] The incremental learning system for detecting forged face image videos, wherein the classifier H t For a binary classifier, the current incremental learning task is in represents the real image and forged image under the k-th forgery method at time t, represents the binary classification label under the k-th forgery method at time t, Indicates the multi-classification label under the k-th forgery method at time t; during the training process, the binary classification label Participate in the calculation of loss function, multi-classification label Used by the model to distinguish new categories from old categories; in binary classification learning, the total loss function is composed of the cross entropy loss L of the binary classifier BC and the multi-classification cross entropy loss L of the auxiliary classifier H't constitute:

[0117]

[0118] λ a is a hyperparameter that controls the effect of the auxiliary classifier.

[0119] The incremental learning system for detecting forged face image videos, wherein the classifier H t For multiple classifiers, the current incremental learning task is in represents the real image and forged image under the k-th forgery method at time t, Represents the corresponding multi-classification label; in multi-classification learning, the loss function is composed of a multi-classifier cross entropy loss L MC and the multi-classification cross entropy loss L of the auxiliary classifier H't constitute:

[0120]

[0121] λ b is a hyperparameter that controls the effect of the auxiliary classifier.

[0122] The incremental learning system for detecting forged face image videos, wherein the classifier H t It is a multi-branch classifier composed of multi-classification and binary classification. The current incremental learning task is in represents the real image and forged image under the k-th forgery method at time t, represents the binary classification label under the k-th forgery method at time t, Represents the multi-classification label under the k-th forgery method at time t; the total loss function is composed of the cross entropy loss L of the two classifiers BC , multi-classification cross entropy loss L of the auxiliary classifier H't and the cross entropy loss L of the multi-classifier MC constitute:

[0123]

[0124] λ d is a hyperparameter that controls the effect of the auxiliary classifier, λ c It is a hyperparameter that balances the binary classification loss and the multi-classification loss.

Claims

1. An incremental learning method for detecting forged face image videos, characterized in that: include: Step 1: Obtain the forged face image detection model to be incrementally learned as a basic model, which includes a feature extractor Φ for extracting image features. t-1 , and a classifier H that assigns fake face image detection categories to image features t ; Step 2: Obtain training images that have been labeled with forged categories; create a new feature extractor F in the basic model i ; At the current time t, the feature extractor Φ t-1 and the new feature extractor F t Composition feature extractor Φ t ,in For the input training image x, Φ t The extracted feature υ is: u=Φ t (x)=[Φ t-1 (x),F t (x)] Step 3: Create a new auxiliary classifier H in the basic model t ', H t 'The label space is |Y t |+1, including the incremental learning task Y at time t t The categories in the basic model and all the old categories that can be recognized by the basic model, for the input training image x, the auxiliary classifier H t 'According to the following formula, the auxiliary classification results are obtained: pH t '(y|x)=Softmax(H t '(F t (x))) The auxiliary classification result and the forged category corresponding to the training image x are used to construct the auxiliary loss L H't ; Input the feature υ into the classifier H t : pH t (y|x)=Softmax(H t (υ)) Get the scores of the training image x under each forgery category, and take the forgery category with the highest score as the classification result in in For all image categories; through the classification results And the forged category corresponding to the training image x constructs the classification loss L C ; The classification loss L C and auxiliary loss L H't Construct the total loss to update the classifier H t And the new feature extractor F i until the total loss converges or reaches the specified number of iterations, and the current basic model is saved as the final recognition model; Step 4: Input the image to be forged face image detection into the final recognition model to obtain its recognition result; Among them, the H t The initial parameters are inherited from H t-1 , H t The output dimension of Φ is continuously updated to match the increased number of forged categories, and its newly added parameters are randomly initialized; t-1 It includes at least one feature extractor, the feature extractor F i The initial weight parameters are inherited from F i-1 , where i represents the i-th incremental learning task.

2. The incremental learning method for detecting forged face image videos according to claim 1, characterized in that: The classifier H t For a binary classifier, the current incremental learning task is in represents the real image and forged image under the k-th forgery method at time t, represents the binary classification label under the k-th forgery method at time t, Indicates the multi-classification label under the k-th forgery method at time t; during the training process, the binary classification label Participate in the calculation of loss function, multi-classification label Used by the model to distinguish new categories from old categories; in binary classification learning, the total loss function is composed of the cross entropy loss L of the binary classifier BC and the multi-classification cross entropy loss L of the auxiliary classifier H 't constitute: λ a is a hyperparameter that controls the effect of the auxiliary classifier.

3. The incremental learning method for detecting forged facial image videos according to claim 1, wherein: The classifier H t For multiple classifiers, the current incremental learning task is in represents the real image and forged image under the k-th forgery method at time t, Indicates the corresponding multi-classification label; In multi-classification learning, the loss function is composed of a multi-classifier cross entropy loss L MC and the multi-classification cross entropy loss L of the auxiliary classifier H 't constitute: λ b is a hyperparameter that controls the effect of the auxiliary classifier.

4. The incremental learning method for detecting forged facial image videos according to claim 1, wherein: The classifier H t It is a multi-branch classifier composed of multi-classification and binary classification. The current incremental learning task is in represents the real image and forged image under the k-th forgery method at time t, represents the binary classification label under the k-th forgery method at time t, Represents the multi-classification label under the k-th forgery method at time t; the total loss function is composed of the cross entropy loss L of the two classifiers BC , multi-classification cross entropy loss L of the auxiliary classifier H 't and the cross entropy loss L of the multi-classifier MC constitute: λ d is a hyperparameter that controls the effect of the auxiliary classifier, λ c It is a hyperparameter that balances the binary classification loss and the multi-classification loss.

5. An incremental learning system for detecting forged facial image videos, characterized in that: include: The initial module is used to obtain the fake face image detection model to be incrementally learned as a basic model, which includes a feature extractor Φ for extracting image features t-1 , and a classifier H that assigns fake face image detection categories to image features t ; A new module is created to obtain training images that have been labeled as forged categories; a new feature extractor F is created in this basic model i ; At the current time t, the feature extractor Φ t-1 and the new feature extractor F t Composition feature extractor Φ t ,in For the input training image x, Φ t The extracted feature υ is: u=Φ t (x)=[Φ t-1 (x),F t (x)] Training module, used to create a new auxiliary classifier H in the basic model t ', H t 'The label space is |Y t |+1, including the incremental learning task Y at time t t The categories in the basic model and all the old categories that can be recognized by the basic model, for the input training image x, the auxiliary classifier H t 'According to the following formula, the auxiliary classification results are obtained: pH t '(y|x)=Softmax(H t '(F t (x))) The auxiliary classification result and the forged category corresponding to the training image x are used to construct the auxiliary loss L H't ; Input the feature υ into the classifier H t : pH t (y|x)=Softmax(H t (υ)) Get the scores of the training image x under each forgery category, and take the forgery category with the highest score as the classification result in in For all image categories; through the classification results And the forged category corresponding to the training image x constructs the classification loss L C ; The classification loss L C and auxiliary loss L H't Construct the total loss to update the classifier H t And the new feature extractor F i until the total loss converges or reaches the specified number of iterations, and the current basic model is saved as the final recognition model; A recognition module, configured to input the image to be detected as a forged human face into the final recognition model to obtain a recognition result; Among them, the H t The initial parameters are inherited from H t-1 , H t The output dimension of Φ is continuously updated to match the increased number of forged categories, and its newly added parameters are randomly initialized; t-1 It includes at least one feature extractor, the feature extractor F i The initial weight parameters are inherited from F i-1 , where i represents the i-th incremental learning task.

6. The incremental learning system for detecting forged facial image videos according to claim 5, wherein: The classifier H t For a binary classifier, the current incremental learning task is in represents the real image and forged image under the k-th forgery method at time t, represents the binary classification label under the k-th forgery method at time t, Indicates the multi-classification label under the k-th forgery method at time t; during the training process, the binary classification label Participate in the calculation of loss function, multi-classification label Used by the model to distinguish new categories from old categories; in binary classification learning, the total loss function is composed of the cross entropy loss L of the binary classifier BC and the multi-classification cross entropy loss L of the auxiliary classifier H't constitute: λ a is a hyperparameter that controls the effect of the auxiliary classifier.

7. The incremental learning system for detecting forged facial image videos according to claim 5, wherein: The classifier H t For multiple classifiers, the current incremental learning task is in represents the real image and forged image under the k-th forgery method at time t, Indicates the corresponding multi-classification label; In multi-classification learning, the loss function is composed of a multi-classifier cross entropy loss L MC and the multi-classification cross entropy loss L of the auxiliary classifier H't constitute: λ b is a hyperparameter that controls the effect of the auxiliary classifier.

8. The incremental learning system for detecting forged facial image videos according to claim 5, wherein: The classifier H t It is a multi-branch classifier composed of multi-classification and binary classification. The current incremental learning task is in represents the real image and forged image under the k-th forgery method at time t, represents the binary classification label under the k-th forgery method at time t, Represents the multi-classification label under the k-th forgery method at time t; the total loss function is composed of the cross entropy loss L of the two classifiers BC , multi-classification cross entropy loss L of the auxiliary classifier H 't and the cross entropy loss L of the multi-classifier MC constitute: L=L MC +λ c L BC +λ d L Ht ' λ d is a hyperparameter that controls the effect of the auxiliary classifier, λ c It is a hyperparameter that balances the binary classification loss and the multi-classification loss.

Images