A method for detecting a pluggable fake face based on test time domain adaptation

Abstract

The application provides a kind of based on test time domain adaptation's pluggable fake face detection method, using a detector trained in source domain as a basic detector, improve the test accuracy of the basic detector in target domain through pluggable adaptation module;Based on the test time domain adaptation of pluggable deep fake detection module, the generalization performance of the existing detector is improved in the test stage, without knowing the structure of the detector and without retraining the detector, in the test time domain adaptation process, the parameters of the basic detector are frozen and unchanged, only the parameters of the feature conversion layer are updated, the old features can better adapt to new fake data, by weighting the results of its nearest neighbor samples, the prediction variance can be reduced, a prediction consistency constraint is constructed to alleviate the influence of noise samples on the feature conversion layer during updating.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field, and in particular to a pluggable fake face detection method based on test time domain adaptation. BACKGROUND

[0002] With the continuous development of deep generation models, face deepfake images or videos are increasingly popular on information networks and social media platforms; these generation techniques learn to generate realistic face images and videos through neural networks, and even can fake the facial features and expressions of specific characters; although deepfake technology has broad application prospects in the entertainment field such as film production, virtual reality and games, it can reduce production costs and improve visual effects, but its abuse has brought many social problems; specifically, deepfake may infringe the privacy of public figures and accelerate the spread of false information; therefore, the widespread application of deepfake poses new challenges to information security.

[0003] For the detection of deepfake content, existing methods mainly regard deepfake detection as a binary classification task, and achieve it by training neural networks on paired real and fake image datasets; at present, a considerable number of trained detectors can be used for the identification of fake faces; but these detection methods have the problem of weak generalization, which is specifically manifested in that the detector can achieve very high detection accuracy on data with the same distribution as the training set, but the detection accuracy on data with different distributions will be greatly reduced.

[0004] On the one hand, these detection methods are prone to overfitting to features unrelated to fake traces, including image backgrounds and face identity information, etc., and when facing different identity face images, the detection accuracy will be disturbed by new identity information;

[0005] On the other hand, deepfake algorithms are increasingly iteratively updated, and different deepfake algorithms will leave different fake features on fake content, and a detector using a specific fake algorithm will have difficulty in detecting unknown fake images.

[0006] Therefore, a pluggable test time adaptation method is needed, which enables the detector to achieve good detection performance on content with new identity information and fake features without retraining the existing detector, and improves its generalization; in order to solve the above technical problems, the prior art designs the following technical solutions, for example:

[0007] ①Patent literature with publication (publication) number CN118968269A, "A deepfake detection method combining spatial texture difference and frequency domain information", combines spatial information and frequency domain information to improve the generalization of the detector; the texture difference information of the spatial domain, i.e. the texture of the background image and the foreground image in the face-swapped image, may differ, based on which information common to different forgery methods can be mined, and the frequency domain information is further fused to improve the detection accuracy.

[0008] ②Patent literature with publication (publication) number CN118470585A, "A deepfake detection method based on multi-domain fusion", proposes a feature fusion method of spatial domain information and frequency domain information; in the model training stage, a suitable convolutional neural network is selected for feature extraction, and a traditional machine learning model such as SVM is used for classification after feature fusion, improving the generalization of the detector.

[0009] ③Patent literature with publication (publication) number CN118379608B, "A high-robustness deepfake detection method based on adaptive learning", uses adaptive technology to solve the robustness problem of deepfake detection, which is difficult for the detector to identify fake images after quality degradation; this method first generates degraded face images of different qualities based on the degradation generation algorithm, supplementing the quality diversity of existing fake face image datasets, and combining an adaptive sampling network to coordinate the learning signals of face images of different qualities, dynamically capturing the forgery features of unknown quality face images, thereby improving the detection performance of the deepfake detection model for low-quality fake images.

[0010] ④Patent literature with publication (publication) number CN118397440A, "A deepfake detection method and system based on unsupervised domain adaptation", combines domain adaptation technology to improve the generalization of deepfake detection; this method uses unsupervised domain adaptation technology to identify known and unknown deepfake categories in the target domain without the need for target domain labels, thereby improving the generalization ability of the model.

[0011] However, the above technologies still have design deficiencies and scheme defects, for example:

[0012] (1) Over-reliance on existing prior knowledge of forgery methods;

[0013] Existing deepfake detection methods that aim to improve generalization still have difficulty adapting to unknown forgery data, as they require known forgery methods to construct training datasets.

[0014] (2) Low efficiency of domain adaptation and small application scope;

[0015] The deep forgery detection method based on domain adaptation can only process a large batch of to-be-detected data offline and can only be applied to a specific detector structure. SUMMARY

[0016] To solve the above technical problems, the present application provides a test time domain adaptive plug-in forgery face detection method, which improves the generalization of deep forgery, uses a domain adaptation scheme to alleviate the dependence of the detector on training knowledge to solve the problem of excessive dependence on prior knowledge of existing forgery methods, and proposes a test time domain adaptive plug-in method to improve the generalization of different training completed detectors without retraining the detector, which can be used in online testing scenarios to improve testing efficiency.

[0017] In a first aspect, a test time domain adaptive plug-in forgery face detection method uses a source domain trained detector as a basic detector to improve the test accuracy of the basic detector in a target domain by a plug-in adaptive module.

[0018] As an example, the plug-in adaptive module only processes the output results of the basic detector without modifying the structure of the basic detector or retraining.

[0019] The specific design scheme of the detection method includes:

[0020] Step 1: Prepare a basic detector and test samples.

[0021] The basic detector includes a feature extractor and a classifier ; and test samples are set.

[0022] The feature extractor is used to extract deep features of an image .

[0023] The classifier calculates an output score according to the deep features .

[0024] As an example, the basic detector uses a trained ViT model as a basic detector.

[0025] As a preferred example, the ViT model is trained on a dataset of paired real face and fake face images.

[0026] The method used by the fake face image includes Deepfakes and FaceSwap algorithms; the test data set includes faces of other identities, including fake face images generated by using Face2Face fake algorithm.

[0027] As an example, the classifier is a learnable prototype classifier.

[0028] Step 2: input the test sample to the base detector;

[0029] Use the history record to store the output of the base detector and filter;

[0030] The specific scheme includes the following two sub-steps:

[0031] Step 2.1: initialize the history record , wherein the deep feature is initialized as the weight value of the classifier, and the output score is initialized as ;

[0032] Step 2.2: update the history record according to the output of the base detector, and the updated history record ;

[0033] Calculate the entropy of each test sample according to the output score: ;

[0034] Wherein: is a softmax function, and then filter the K samples with the maximum entropy to maintain the size of the history record ;

[0035] Step 3: the classifier calculates the class prototype according to the history record , and calculates the prediction result based on this, and then uses the pseudo label to calculate the standard cross entropy loss to update the feature conversion layer;

[0036] The specific scheme includes the following sub-steps:

[0037] Step 3.1: the formula for calculating the class prototype is designed as follows:

[0038] ;

[0039] Wherein: is an indicator function, and is the element index in the history record; the feature conversion layer is innovatively introduced, including A fully connected network; the feature transformation layer can be dynamically updated during testing, enabling the trained base detector features to adapt to new, unknown forged data;

[0040] Then, the adapted features and class prototypes are calculated based on the feature transformation layer: , ;

[0041] The similarity is calculated as the prediction result using the following formula:

[0042] ;

[0043] ;

[0044] in: It is the cosine similarity function, and It is in the Transformation layer for categories The predicted value; It is the average prediction result of all feature transformation layers;

[0045] Step 3.2: Dynamically update the feature transformation layer based on the prediction results;

[0046] The softmax value of the output score of the base detector. To guide the prediction results At the same time, set a threshold. To ignore samples with low confidence, thereby mitigating the impact of noisy labels:

[0047] ;

[0048] ;

[0049] in: It is the standard cross-entropy loss function.

[0050] Step 4: Select the nearest neighbor sample in the feature space of the test sample from the historical records, and further correct the prediction result of the test sample using the nearest neighbor sample;

[0051] The specific plan includes the following sub-steps:

[0052] Step 4.1: Select based on feature distance The nearest neighbor samples:

[0053] ;

[0054] in: It is a distance function. is a deep feature of the test sample and the distance of the deep feature of the nearest neighbor sample; then the nearest neighbor prediction result ;

[0055] Step 4.2: according to the nearest neighbor prediction result, further correct the prediction result of the test sample to obtain a final prediction result:

[0056] ;

[0057] As an example, to further reduce the influence of the noise label on the feature conversion layer update, a nearest neighbor consistency constraint is introduced to force the test sample and its corresponding nearest neighbor sample to output similar results:

[0058] .

[0059] Step 5: after calculating the final prediction result, calculate the loss according to the following formula and update the feature conversion layer using the back propagation algorithm Step:

[0060] ;

[0061] wherein: is a hyperparameter to balance the importance of the consistency constraint.

[0062] In a second aspect, the present application shows an electronic device, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the method of any of the above aspects.

[0063] In a third aspect, the present application shows a non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by the processor of an electronic device, the electronic device can execute the method of any of the above aspects.

[0064] In a fourth aspect, the present application shows a computer program product, when the instructions in the computer program product are executed by the processor of an electronic device, the electronic device can execute the method of any of the above aspects.

[0065] Advantages of the present application:

[0066] 1. A pluggable deep spoofing detection module based on test-time adaptation: The present application proposal constructs a pluggable deep spoofing detection module based on test-time adaptation; in the test phase, the generalization performance of the existing detector is improved, without knowing the structure of the detector and without retraining the detector. The module works in an online test scene, which improves the test efficiency compared to offline domain adaptation.

[0067] 2. A learnable prototype classifier: The present application proposal introduces a feature transformation layer to adapt the deep features in the base detector in the test time. In the test-time adaptation process, the parameters of the base detector are frozen and unchanged, and only the parameters of the feature transformation layer are updated. Through feature transformation, the old features can better adapt to the new spoofing data, including new identity data and new spoofing algorithms.

[0068] 3. A nearest neighbor corrector: The present application proposal constructs a nearest neighbor-based prediction corrector to correct the prediction results of a certain test sample, and by weighting the results of its nearest neighbor samples, the prediction variance can be reduced. At the same time, a prediction consistency constraint is constructed to alleviate the influence of noise samples on the feature transformation layer when it is updated. BRIEF DESCRIPTION OF DRAWINGS

[0069] Fig. 1 A schematic diagram of the overall structure of the present application, a method for detecting pluggable spoofing faces based on test-time adaptation.

[0070] Fig. 2 A schematic diagram of the overall structure of the present application, a method for detecting pluggable spoofing faces based on test-time adaptation.

[0071] Fig. 3 A schematic diagram of the overall structure of the present application, a method for detecting pluggable spoofing faces based on test-time adaptation. DETAILED DESCRIPTION

[0072] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application, with reference to Figs. 1 to 3 the drawings,

[0073] In a first aspect, a method for detecting pluggable spoofing faces based on test-time adaptation uses a trained detector as a base detector, and improves the test accuracy of the base detector in the target domain through a pluggable adaptation module.

[0074] ​​As an example, the pluggable adaptation module only processes the output of the base detector without modifying the structure of the base detector or retraining.

[0075] The specific design of the detection method includes:

[0076] Step 1: Prepare the base detector and test samples;

[0077] The base detector includes a feature extractor and a classifier ; Set the test samples ;

[0078] The feature extractor is used to extract deep features of the image ;

[0079] The classifier calculates the output score according to the deep features ;

[0080] As an example, the base detector uses a trained ViT model as the base detector.

[0081] As a preferred example, the ViT model is trained on a dataset of paired real and fake face images.

[0082] The method used by the fake face image includes Deepfakes and FaceSwap algorithms; the test dataset includes the faces of other identities, including fake face images generated by the Face2Face fake algorithm.

[0083] As an example, the classifier is a learnable prototype classifier.

[0084] Step 2: Input the test samples to the base detector;

[0085] Use the history record to store the output of the base detector and filter it;

[0086] The specific scheme includes the following two sub-steps:

[0087] Step 2.1: Initialize the history record , where the deep features are initialized as the weight values of the classifier, and the output score is initialized as ;

[0088] Step 2.2: Update the history record according to the output of the base detector, and the updated history record ;

[0089] Calculate the entropy of each test sample based on the output score: ;

[0090] in: It uses the softmax function, then filters the K samples with the highest entropy, maintaining the size of the historical records at a certain level. ;

[0091] Step 3: The classifier bases its data on the historical records. The prototype is calculated, and the prediction result is calculated based on it. Then, the standard cross-entropy loss is calculated using pseudo-labels to update the feature transformation layer.

[0092] The specific plan includes the following sub-steps:

[0093] Step 3.1: The prototype formula for the calculation class is designed as follows:

[0094] ;

[0095] in: It is an indicator function, and It is the element index in the historical record; an innovative feature transformation layer is introduced. ,include A fully connected network; the feature transformation layer can be dynamically updated during testing, enabling the trained base detector features to adapt to new, unknown forged data;

[0096] Then, the adapted features and class prototypes are calculated based on the feature transformation layer: , ;

[0097] The similarity is calculated as the prediction result using the following formula:

[0098] ;

[0099] ;

[0100] in: It is the cosine similarity function, and It is in the Transformation layer for categories The predicted value; It is the average prediction result of all feature transformation layers;

[0101] Step 3.2: Dynamically update the feature transformation layer based on the prediction results;

[0102] The softmax value of the output score of the base detector. To guide the prediction results a threshold is set to ignore samples with low confidence, to mitigate the impact of noisy labels:

[0103] ;

[0104] ;

[0105] wherein: is the standard cross-entropy loss function.

[0106] Step 4: further correct the prediction result of the test sample by selecting samples in the history record that are nearest neighbors of the test sample in feature space, and using the nearest neighbor samples to further correct the prediction result of the test sample;

[0107] The specific scheme includes the following sub-steps:

[0108] Step 4.1: selecting nearest neighbor samples according to feature distance:

[0109] ;

[0110] wherein: is a distance function, is the deep feature of the test sample and the deep feature of the nearest neighbor sample; then nearest neighbor prediction results ;

[0111] Step 4.2: further correct the prediction result of the test sample according to the nearest neighbor prediction results, to obtain a final prediction result:

[0112] ;

[0113] As an example, to further reduce the impact of noisy labels on the update of the feature conversion layer, a nearest neighbor consistency constraint is introduced, to force the test sample and its corresponding nearest neighbor sample to output similar results:

[0114] .

[0115] Step 5: after the final prediction result is calculated, the loss is calculated according to the following formula and the feature conversion layer is updated using the back propagation algorithm Step:

[0116] ;

[0117] wherein: is a hyperparameter to balance the importance of the consistency constraint.

[0118] In a second aspect, this application discloses an electronic device comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to perform the method as described in any of the preceding aspects.

[0119] Thirdly, this application discloses a non-transitory computer-readable storage medium that, when the instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to perform the methods described in any of the preceding aspects.

[0120] Fourthly, this application discloses a computer program product in which, when the instructions in the computer program product are executed by a processor of an electronic device, the electronic device is enabled to perform the method described in any of the preceding aspects.

[0121] To better illustrate the design principle of this invention, the detection method of this invention is now described with specific examples as follows:

[0122] Example 1:

[0123] Application of detection methods

[0124] Step 1: Use the trained ViT model as the base detector. The base detector is trained on a dataset of paired real and fake face images. The fake face images are generated using methods including Deepfakes and FaceSwap algorithms. The test dataset includes faces with other identities, including fake faces generated using the Face2Face forgery algorithm.

[0125] Step 2: Set the input batch size for testing to 32, and input all the data to be tested into the base detector sequentially. For the ViT model, use Class Token as a deep feature. The output of the fully connected layer used for classification is used as the output score. The output results will be saved to the history, and the maximum value of the saved samples will be stored. Set it to 1000.

[0126] Step 3: Calculate the class prototype based on the historical sample output results, and set the number of feature transformation layers. The threshold is set to 20. Then, the deep features of the prototype and test samples are input into the feature transformation layer to obtain the adapted prototype and features, and their similarity is calculated to obtain the prediction result. To adapt to new identity information and forgery algorithms in the test data, the feature transformation layer needs to be dynamically updated, and the filtering threshold is adjusted when calculating the cross-entropy loss. Set it to 0.7.

[0127] Step 4: First, the nearest neighbor samples are filtered from the historical records by using the Euclidean distance function. The number of nearest neighbors can be set to 16. Then, the corrected prediction results can be calculated according to the nearest neighbor samples. The results are the detection results as the basis for identification. Finally, the consistency constraint loss value is calculated.

[0128] Step 5: Set the hyperparameter to 10.0, and calculate the overall loss value . Assuming that the weight parameter of the feature transformation layer is , the parameter can be updated according to the following back propagation formula:

[0129]

[0130] wherein is the learning rate. In the specific implementation, the Adam optimizer can be selected, and the learning rate is set to 0.0001. The update step of the batch is 1. Then, return to step 2 until all samples are tested.

[0131] The present application greatly improves the generalization performance of deep fake detection. To verify the generalization performance improvement of the present application on the existing deep fake detector, the ViT model trained on the FF++ c23 dataset is used as the basic detector, and the test is performed on other datasets including CDF-v2, DFD and DFDC. The AUC is used as the evaluation index. The method in the present application can achieve an average performance improvement of 9%, which improves the efficiency of domain adaptation.

[0132] The existing domain adaptation method of deep fake detection mostly uses the offline test method, that is, multiple iterations are performed on the target test dataset. The method in the present application can work in an online test scenario. A little increase in the time consumption of the detector in each round of reasoning is only 0.035s for the specific embodiment 1. Compared with the offline test domain adaptation, the efficiency is greatly increased.

[0133] It should be noted that, for the method embodiment, in order to simply describe, all are expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited by the order of the described actions, because according to the present application, certain steps can be performed in other order or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to optional embodiments, and the actions involved are not necessarily necessary for the present application.

[0134] ​​Optionally, the embodiment of the present application further provides an electronic device, comprising: a processor, a memory, a computer program stored in the memory and executable in the processor, when the computer program is executed by the processor, each process of the method embodiment is realized, and the same technical effects can be achieved, and here is not repeated.

[0135] The embodiment of the present application further provides a computer readable storage medium, and the computer readable storage medium stores a computer program, when the computer program is executed by a processor, each process of the method embodiment is realized, and the same technical effects can be achieved, and here is not repeated. The computer readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0136] Figure 2 is a block diagram of an electronic device 800 according to an embodiment of the present application. For example, the electronic device 800 can be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, or the like.

[0137] Referring to Fig. 2 The electronic device 800 can include one or more of the following components: a processing component 802, a memory 804, a power supply component 806, a multimedia component 808, an audio component 810, an input / output (I / O) interface 812, a sensor component 814, and a communication component 816.

[0138] The processing component 802 usually controls overall operations of the electronic device 800, such as operations associated with displaying, making phone calls, data communications, camera operations and recording operations. The processing component 802 can include one or more processors 820 to execute instructions to complete all or part of steps of the above method. In addition, the processing component 802 can include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 can include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0139] The memory 804 is configured to store various types of data to support the operation of the electronic device 800; examples of these data include instructions for any application programs or methods operating on the electronic device 800, contact data, phonebook data, messages, images, videos, etc. The memory 804 can be implemented by any type of volatile or nonvolatile memory, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disc, or optical disc.

[0140] The power supply component 806 supplies power for various components of the electronic device 800. The power supply component 806 can include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the electronic device 800.

[0141] The multimedia component 808 includes a screen providing an output interface between the electronic device 800 and a user. In some embodiments, the screen can include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes the touch panel, the screen can be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense a touch, a slide, and a gesture on the touch panel. The touch sensor can not only sense a boundary of a touching or a sliding action, but also detect duration and pressure related to the touching or sliding action. In some embodiments, the multimedia component 808 includes a front camera and / or a back camera. The front camera and / or the back camera can receive external multimedia data when the device 800 is in an operation mode, such as a shooting mode or a video mode. Each of the front camera and the back camera can be a fixed optical lens system or have a focal length and optical zoom capability.

[0142] The audio component 810 is configured to output and / or input an audio signal. For example, the audio component 810 includes a microphone (MIC) configured to receive an external audio signal when the electronic device 800 is in an operation mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory 804 or transmitted via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting an audio signal.

[0143] The I / O interface 812 provides an interface between the processing component 802 and peripheral interface modules, which can be a keypad, a click wheel, buttons, etc. The buttons can include, but are not limited to, a home button, a volume button, a start button, and a lock button.

[0144] The sensor component 814 includes one or more sensors for providing status assessments for various aspects of the electronic device 800. For example, the sensor component 814 can detect an open / closed position of the device 800, relative positioning of components, such as a display and a keypad of the electronic device 800, a change in position of the electronic device 800 or a component of the electronic device 800, presence or absence of user contact with the electronic device 800, orientation or acceleration / deceleration / g-force and temperature of the electronic device 800. The sensor component 814 can include an optical sensor for detecting ambient light, a proximity sensor configured to detect proximity of an object without any physical contact, or a CMOS or CCD image sensor for use in imaging applications. In some embodiments, the sensor component 814 can further include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0145] The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as WiFi, a cellular network (e.g., 2G, 3G, 4G, or 5G), or a combination thereof. In an example embodiment, the communication component 816 receives broadcast signals or broadcast operation information from an external broadcast management system via a broadcast channel. In an example embodiment, the communication component 816 can further include a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technology.

[0146] In an example embodiment, the electronic device 800 can be implemented using one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors, or other electronic elements for performing the above-described methods.

[0147] In an example embodiment, a non-transitory computer-readable storage medium including instructions, such as the memory 804 including instructions, is also provided, which can be executed by the processor 820 of the electronic device 800 to implement the above-described methods. For example, the non-transitory computer-readable storage medium can be a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disc, and an optical data storage device, etc.

[0148] Fig. 3is another block diagram of an electronic device 1900 shown in this application. For example, the electronic device 1900 can be provided as a server.

[0149] Referring to Fig. 3 The electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932, for storing instructions executable by the processing component 1922, such as an application program. The application program stored in the memory 1932 can include one or more than one module each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute the instructions to perform the above-mentioned method.

[0150] The electronic device 1900 can also include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output (I / O) interface 1958. The electronic device 1900 can operate based on an operating system stored in the memory 1932, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM or the like.

[0151] From the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment method can be realized by means of software and the necessary general hardware platform, of course, it can also be realized by hardware, but in many cases the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product in essence or in the form of a contribution to the prior art. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for making a terminal (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) execute the methods described in various embodiments of the present application.

[0152] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working process of the above-described system, device and unit can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0153] In the embodiments of the present application, it should be understood that the disclosed apparatus and method can be implemented in other manners. For example, the embodiments of the apparatus described above are merely schematic; the division of the units is merely logical function division; an actual implementation can be another division manner, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, devices or units, and can be in electrical, mechanical or other forms.

[0154] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place, or can be distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiments.

[0155] In addition, each functional unit in the various embodiments of the present application can be integrated into a processing unit, or each unit can be a physically independent unit, or two or more units can be integrated into one unit.

[0156] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the parts of the technical solutions that make contributions to the prior art or the parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes various media that can store program codes, such as U disk, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc.

[0157] The above description is only the preferred embodiments of the present application, and it should be understood that the above embodiment description is only used to help understand the method of the present application and its core idea, and does not limit the protection scope of the present application. Any modification, equivalent replacement, etc. within the idea and principles of the present application should be included in the protection scope of the present application.

Claims

1. A method for detecting pluggable fake human faces based on test time domain adaptation, using a source domain detector trained as a base detector, and improving the test accuracy of the base detector in a target domain by a pluggable adaptation module; characterized in that The specific detection method comprises: Step 1: preparing a basic detector and a test sample; The base detector comprises a feature extractor and a classifier ; setting a test sample ; the feature extractor is configured to extract deep features of the image ; the classifier: computes an output score from the deep features ; Step 2: inputting the test sample to the base detector; Use history record The output result of the base detector is stored and filtered. Step 3: The classifier updates the feature transformation layer based on the history records Compute class prototypes and based on that compute prediction results, then update the feature transformation layer with pseudo labels using standard cross-entropy loss. Step 3.1: calculating a class prototype formula designed as follows: ； Wherein: is an indicator function, and is an element index in the history record; a feature conversion layer is introduced , including a fully connected network; the feature conversion layer can be dynamically updated during testing, so that the features of the trained base detector can adapt to new unknown fake data; Subsequently, the adapted features and class prototypes are computed according to the feature transformation layer: , ; And calculate the similarity as a prediction result through the following formula design: ； ； wherein: is the cosine similarity function, while is the prediction of the conversion layer for the class ; is the average prediction of all feature conversion layers. Step 3.2: dynamically updating the feature conversion layer according to the prediction result; using softmax values of output scores of the base detector to guide the prediction result while setting a threshold to ignore samples with low confidence, thereby mitigating the impact of noisy labels: ； ； wherein: is the standard cross-entropy loss function; Step 4: selecting a sample closest to the test sample in the feature space in the historical record, and further correcting the prediction result of the test sample by using the closest neighbor sample; Step 5: After the final prediction is computed, the loss is computed according to the following formula and the feature transformation layer is updated using the backpropagation algorithm Step: ； where: is a hyper-parameter to balance the importance of consistency constraints.

2. The method of claim 1, wherein the method is based on testing time domain adaptation. The pluggable adaptive module only processes the output result of the basic detector without modifying the structure of the basic detector or retraining.

3. The method of claim 1, wherein the method is based on testing time domain adaptation. The basic detector adopts a trained ViT model as a basic detector; The ViT model is trained on a dataset of paired real human faces and fake human face images; The method adopted by the fake human face image includes Deepfakes and FaceSwap algorithms; the test dataset includes the faces of other identities, including fake human face images generated by using Face2Face fake algorithm.

4. The method of claim 1, wherein, The classifier is a learnable prototype classifier.

5. The method of claim 1, wherein, The specific scheme of step 2 comprises the following two sub-steps: Step 2.1 : initializing the history where the deep features are initialized to the weight values of the classifier and the output scores are initialized to ; Step 2.2: updating the history record according to the output result of the base detector, the updated history record ; calculating an entropy for each of the test samples based on the output scores: ; where: is a softmax function, followed by filtering the K samples with the highest entropy, maintaining the size of the history at .

6. The method of claim 1, wherein, The specific scheme of step 4 comprises the following sub-steps: Step 4.1: Selection of the nearest neighbor according to the characteristic distance the sample with the smallest distance ； wherein: is a distance function, is a deep feature of the test sample and the distance of the deep feature of the nearest neighbor sample; the nearest neighbor prediction may then be obtained; Step 4.2: further correcting the prediction result of the test sample according to the nearest neighbor prediction result to obtain a final prediction result: 。 7. The method of claim 6, wherein the method is based on testing time domain adaptation. In order to further reduce the influence of noisy labels on the update of the feature conversion layer, a nearest neighbor consistency constraint is introduced to force the test sample and its corresponding nearest neighbor sample to output similar results: 。 8. An electronic device, comprising: The electronic device comprises a processor and a memory for storing processor-executable instructions; wherein the processor is configured to execute the method of any one of claims 1-7.

9. A non-transitory computer-readable storage medium, comprising: When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device can execute the method of any one of claims 1-7.

Images