Image deepfake detection acceleration method and system based on quantization technology
By using a quantization-based method to accelerate deepfake detection, the problem of deploying deepfake detection models on hardware platforms with limited computing resources is solved. This method enables fast and accurate multi-task parallel forgery detection, reducing computational complexity and resource consumption.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- INST OF COMPUTING TECH CHINESE ACAD OF SCI
- Filing Date
- 2022-11-07
- Publication Date
- 2026-06-05
AI Technical Summary
Existing deepfake detection models are difficult to deploy on hardware platforms with limited computing resources, and they are also difficult to quickly and accurately identify the authenticity of images and videos of various formats and qualities, resulting in excessive computational complexity and resource consumption.
A deep fake detection acceleration method based on quantization technology is adopted, including a data processing and compression module, a model quantization acceleration module, and a model fine-tuning and acceleration module. By extracting keyframes, quantizing model weights and activation values, and fine-tuning the model, the computational complexity and resource requirements are reduced.
While ensuring detection accuracy, the efficiency of deepfake detection is improved, and the computational complexity and resource consumption are reduced, enabling the model to perform forgery detection quickly and accurately with limited computing resources.
Smart Images
Figure CN115719520B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of machine learning and computer vision, and particularly to the machine learning problem of deepfake detection in computer vision. Background Technology
[0002] In recent years, deep learning technology has continued to develop and has been widely applied in the field of computer vision. On the one hand, deep learning technology has led a new wave of artificial intelligence, but on the other hand, a series of security issues caused by deep learning have also attracted increasing attention. Currently, image and video recognition technologies based on deep learning are widely used in all aspects of people's lives, such as intelligent supervision of network content, automatic video surveillance and analysis, access control systems based on facial recognition, and facial recognition payment. In these key application areas, the reliability and security of information and data should be valued and guaranteed. Since 2017, some fake images and videos generated based on deepfake technology have attracted widespread attention on the Internet, especially when deepfakes are used on influential figures, they often leverage that person's influence to generate even greater impact. Forums have even seen videos where, without permission, the faces of pornographic video characters have been modified to resemble those of celebrities, causing serious negative impacts. In addition, a large number of "one-click" face-swapping software programs have made deepfake technology, compared with traditional image and video processing technologies, "low threshold, high efficiency, and high quality," bringing huge potential threats and hidden dangers. Fake images and videos have become one of the most significant threats to information and data security, and their detection and regulation face enormous challenges.
[0003] AI-synthesized fake faces pose a significant threat, capable of creating videos that mimic real facial expressions and body movements to create the illusion of a target person doing or saying something, thus overturning people's "seeing is believing" mentality. The industry urgently needs an effective technology to detect fake facial images or videos in the online environment, but this is extremely difficult. Deepfake detection, due to the emergence of new forgery methods and the complexity of the online propagation environment, requires continuous iteration and upgrading of technology to quickly and accurately distinguish between genuine and fake images of various formats and qualities. As the number of network parameters and computational complexity of models increase, as does the consumption of hardware resources, deepfake detection models become difficult to deploy on hardware platforms with limited computing resources. They also struggle to handle high-concurrency, multi-data-stream, and fast-response business application scenarios, thus necessitating effective model acceleration technologies.
[0004] Currently, the following challenges exist in the field of deepfake detection: (1) Technological evolution has led to a wide variety of deepfake methods. While deepfake detection models have improved detection accuracy through methods such as model ensemble, they have also significantly increased the number of model parameters and computational complexity. (2) The diversity of deepfake methods and the complexity of their generation scenarios have broadened the types of deepfake datasets. High-resolution forged videos have increased the computational resource and time consumption for model tuning and retraining. (3) In practical application scenarios, it is necessary to meet industrial detection performance requirements, quickly and accurately identify the authenticity of images and videos of various formats and qualities, and achieve multi-task parallelism. Summary of the Invention
[0005] This invention provides a method for accelerating deep fake detection based on quantization technology, and proposes three acceleration modules: a data processing and compression module, a model quantization acceleration module, and a model fine-tuning and acceleration module.
[0006] The proposed method, "A Deepfake Detection Acceleration Method Based on Quantization Technology," primarily provides quantization acceleration support for deepfake detection algorithms. When a deepfake detection model identifies whether a face image has been deepfake synthesized, it can support the fast and accurate detection requirements of high-concurrency tasks in practical application scenarios, improving algorithm efficiency and reducing computational complexity while ensuring the accuracy of deepfake detection.
[0007] Specifically, this invention proposes an accelerated method for image deep fake detection based on quantization technology, which includes:
[0008] Step 1: Obtain the training video with labeled deep fake detection results, extract the key frames for fake detection from the training video, compress the key frames and remove redundant information, and then extract the frame features that retain the key dimensions of fake detection.
[0009] Step 2: Based on the features of this frame, use a face recognition algorithm to locate the face position information in the key frame. Based on the face position information, the features of this frame, and the labeled deep fake detection results, train an initial detection model.
[0010] Step 3: Quantize the weights and activation values of the initial detection model and verify the balance between inference speed and accuracy loss. Use the quantized initial detection model as an intermediate model. Fine-tune or retrain the intermediate model to obtain the final deepfake detection model.
[0011] Step 4: Input the video to be deepfake detected into the deepfake detection model to obtain the detection result of whether it is a fake video.
[0012] The image deep fake detection acceleration method based on quantization technology includes step 2, which includes: performing identity recognition on the face region in the key frame based on the face location information to obtain identity information; and training the initial detection model based on the identity information, the face location information, the frame features, and the labeled deep fake detection results.
[0013] The image deep fake detection acceleration method based on quantization technology, wherein the quantization conversion includes:
[0014] The weights and activation function of the initial detection model are jointly quantized. The validation set corresponding to the training video is used as calibration data to collect information on the data distribution, including minimum / maximum values, optimal threshold based on entropy theory, and quantization factor based on symmetric quantization. The range of the activation function output is calculated so that the hidden layer output range of the initial detection model can be mapped to [0,1] by passing values through a differential nonlinear function.
[0015] The image deep fake detection acceleration method based on quantization technology, wherein step 1 of extracting key frames for fake detection from the training video specifically includes:
[0016] By training a temporal prediction model, the extracted video frame features are encoded and input into the model for decoding and recovery. The score of each frame is obtained based on the encoding and decoding conversion accuracy, and then the key frames are identified.
[0017] This invention also proposes an accelerated method for image deep fake detection based on quantization technology, including:
[0018] The extraction module is used to obtain training videos with labeled deep fake detection results, extract key frames for fake detection from the training videos, compress the key frames and remove redundant information, and then extract frame features that retain the key dimensions of fake detection.
[0019] The training module is used to locate the face position information in the key frame based on the features of the frame and the face recognition algorithm. Based on the face position information, the features of the frame and the labeled deep fake detection results, the initial detection model is trained.
[0020] The retraining module is used to quantize and transform the weights and activation values of the initial detection model, and to verify the balance between inference speed and accuracy loss. The quantized initial detection model is used as an intermediate model. The intermediate model is then fine-tuned or retrained to obtain the final deepfake detection model.
[0021] The detection module is used to input the video to be deepfake detected into the deepfake detection model and obtain the detection result of whether it belongs to a fake video.
[0022] The image deep fake detection acceleration method based on quantization technology, wherein the training module is used to: identify the face region in the keyframe based on the face location information to obtain the identity information, and train the initial detection model based on the identity information, the face location information, the frame features and the labeled deep fake detection results.
[0023] The image deep fake detection acceleration method based on quantization technology, wherein the quantization conversion includes:
[0024] The weights and activation function of the initial detection model are jointly quantized. The validation set corresponding to the training video is used as calibration data to collect information on the data distribution, including minimum / maximum values, optimal threshold based on entropy theory, and quantization factor based on symmetric quantization. The range of the activation function output is calculated so that the hidden layer output range of the initial detection model can be mapped to [0,1] by passing values through a differential nonlinear function.
[0025] The image deep fake detection acceleration method based on quantization technology, wherein extracting keyframes for fake detection from the training video specifically includes:
[0026] By training a temporal prediction model, the extracted video frame features are encoded and input into the model for decoding and recovery. The score of each frame is obtained based on the encoding and decoding conversion accuracy, and then the key frames are identified.
[0027] The present invention also proposes a storage medium for storing a program that executes any of the image deep pseudo-detection acceleration methods based on quantization techniques.
[0028] This invention also proposes a client for any image deep pseudo-detection acceleration system based on quantization technology.
[0029] As can be seen from the above solutions, the advantages of the present invention are:
[0030] The data processing and compression module proposed in this invention processes and compresses the input suspicious face data. For video data, key frames are extracted, and for high-resolution images, compression is performed while retaining key feature information. Using the features extracted in this part can improve the training efficiency of the deepfake detection model.
[0031] The model quantization acceleration module proposed in this invention addresses the need for smaller, faster, and more powerful models in practical deployments of forgery detection due to hardware computing resources and multi-task parallelism. Choosing a suitable model quantization method to quantize and convert the deepfake detection model can reduce model size, lower computational complexity, and improve computational efficiency.
[0032] The model fine-tuning and acceleration module proposed in this invention reduces the accuracy loss during the quantization process of deep fake detection models through model fine-tuning / retraining, quantizes the model's weights and activation values, reduces the model's storage space requirements, and makes the model easier to deploy on the device; it also reduces the memory bandwidth and data access power consumption of the device, thereby reducing the device's operation and maintenance costs; and it improves the speed of the model in the inference stage. Attached Figure Description
[0033] Figure 1 This is a framework diagram of the deep fake detection acceleration method based on quantization technology of the present invention. Detailed Implementation
[0034] This invention proposes a method to accelerate deepfake detection based on quantization techniques. Specifically, it presents a quantization acceleration method for deepfake detection models. By quantizing and pruning the model's weights and activation values, the computational complexity of the model is reduced while maintaining forgery detection accuracy, thereby improving the speed of deepfake detection models during training and inference. More specifically, this method includes the following parts:
[0035] (1) This invention proposes a deepfake detection acceleration method based on quantization technology. By using model quantization acceleration technology, the weights and activation values of the deepfake detection model are quantized, transforming the network's weights and activation values from high precision to low precision, thereby reducing the computational complexity of the model and achieving fast and accurate deepfake detection under limited computing and hardware resources. This acceleration method can be applied to backbone networks of different scales and structures, solving the problem of balancing speed and accuracy in face forgery detection and recognition.
[0036] (2) When extracting data, the present invention extracts key frames from video data, retains the video frame data that is most important for deepfake detection, and compresses the data to reduce the amount of data while retaining key forgery features.
[0037] (3) In the model quantization acceleration module, this invention accelerates the quantization transformation of network weights, activation values, etc. For model weight quantization, network regularization and other methods are added to make the weight distribution more compact and reduce uneven distribution; for activation value quantization, some outliers are removed to minimize the error caused by quantization.
[0038] (4) In the model fine-tuning and acceleration module, in order to reduce the inevitable loss of model accuracy caused by the quantization process, the model accuracy is kept unchanged while effectively accelerating the model. The deepfake detection model is fine-tuned or retrained to further reduce the accuracy gap between the quantized model and the floating-point model.
[0039] (5) In actual deployment, the quantization acceleration method proposed in this invention uses lower bit data to replace the original floating-point data, reducing the size and computational complexity of the model. This enables rapid forgery detection and identification while ensuring accuracy in authentication, even with limited computing and storage resources on servers and terminal devices.
[0040] To make the above features and effects of the present invention clearer and easier to understand, specific embodiments are described below, and detailed descriptions are provided in conjunction with the accompanying drawings.
[0041] To address the three problems existing in the prior art, this invention proposes a deep fake detection acceleration method based on quantization technology, such as... Figure 1 As shown, the deepfake detection process begins with reading image and video data, performing frame extraction and compression, locating the target person in the data using a face detection algorithm, and identifying the target person's identity using the same algorithm. It then divides into two branches: model training and inference, enabling the training of the deepfake detection model and the detection and identification of forged data. The following sections explain the various quantization acceleration modules in the process.
[0042] (1) Data processing and compression module
[0043] This module processes and compresses the input data. For video data, it extracts keyframes from the video stream using a keyframe extraction algorithm, retrieving frames with high weights for the forgery detection model. For high-resolution data, it uses lossless compression, size scaling, and feature extraction to compress the data while preserving key dimensions of forgery detection features, reducing subsequent computational complexity. This module trains a temporal prediction model, encodes the extracted video frame features, inputs them into the model, decodes them, and obtains a score for each frame based on the encoding / decoding accuracy, thereby identifying keyframes.
[0044] (2) Model Quantization Acceleration Module
[0045] This module accelerates computation by compressing the original network, reducing the number of bits required to represent each weight parameter. Two common quantization methods are half-floating-point precision (FP16) and mixed precision, both of which require underlying computational framework support to achieve the desired speedup. Another method is INT8 quantization, which converts the model's weight parameters from FP32 to INT8 and uses INT8 for inference. Quantization speeds up computation primarily because fixed-point operations are faster than floating-point operations, but converting from FP32 to INT8 results in a loss of model precision. Quantization does not change the distribution of weight parameters; it simply maps them from one range to another, a process similar to numerical normalization.
[0046] This application employs the two quantization methods mentioned above, but in the process of model acceleration, it first compresses the data based on keyframe extraction to achieve acceleration;
[0047] During the model compression process, based on the characteristics of the GPU used and the features of the deep fake detection of people, data outside the face area is discarded, and the acceleration unit in the chip, which is specifically designed for neural network deployment, is used to accelerate and optimize the model and select appropriate quantization techniques.
[0048] The weights and activation function are jointly quantized. A validation set is used as calibration data to collect information on the data distribution, including minimum / maximum values, the optimal threshold based on entropy theory, and the quantization factor based on symmetric quantization. The range of the activation function output is calculated to determine the calculation formula for joint quantization. For example, the hidden layer output range can be mapped to [0,1] by using a differentiable nonlinear function to pass values.
[0049] (3) Model fine-tuning and acceleration module
[0050] Quantization inevitably leads to a loss of model accuracy. To maintain the accuracy of the original model as much as possible, the quantized model is usually fine-tuned or retrained. If the accuracy of the quantized model meets the requirements, the fine-tuning and retraining processes are ignored. For accuracy-sensitive models, accuracy compensation can be used to further optimize the performance of the quantized model.
[0051] In actual deployment, the deepfake detection model can be deployed on a Linux system, with hardware support provided by a server or server cluster containing one or more GPUs. The quantization acceleration module is integrated into the deepfake detection system to provide acceleration services for the model's data processing, training, and inference.
[0052] The data consists of normally usable and openable image and video data. During the inference phase, the video is extracted into images for anti-spoofing. During the testing phase, the images are processed by extracting faces, which are then scaled to a uniform size (e.g., 299×299). Subsequently, the images undergo processes such as face recognition and face correction, and finally, the deep anti-spoofing detection model is used for face anti-spoofing.
[0053] Deepfake systems mainly consist of two phases: training and inference. The steps of the training phase are as follows:
[0054] Step 1: Read the input data to be detected, which can be in various forms such as images and videos.
[0055] Step 2: Perform frame extraction on the input video data. Use a keyframe extraction algorithm to extract key video frame data for forgery detection from the video stream. Compress the images to remove redundant information such as background and retain the key features of forgery detection.
[0056] Step 3: Using a face recognition algorithm, locate the face from the processed and compressed input data, extract the area containing the face, and determine the identity using an identity recognition algorithm. Identity verification can serve as auxiliary information for different deep fake detection tasks, enabling the detection of specific individuals.
[0057] Step 4: Train a deep forgery detection model and integrate multiple forgery detection models for different forgery generation techniques and application scenarios.
[0058] Step 5: Using the model quantization acceleration module, select an appropriate quantization method, calculate quantization parameters based on the quantization method and input range, and perform quantization conversion on the model's weights and activation values according to the conversion formula to verify the balance between inference speed and accuracy loss.
[0059] During the testing phase, the trained model is fine-tuned / retrained to maintain the accuracy of the original model. Through the above steps, fake data is detected and identified.
[0060] The following are system embodiments corresponding to the above method embodiments. This embodiment can be implemented in conjunction with the above embodiments. The relevant technical details mentioned in the above embodiments are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiments.
[0061] This invention also proposes an accelerated method for image deep fake detection based on quantization technology, including:
[0062] The extraction module is used to obtain training videos with labeled deep fake detection results, extract key frames for fake detection from the training videos, compress the key frames and remove redundant information, and then extract frame features that retain the key dimensions of fake detection.
[0063] The training module is used to locate the face position information in the key frame based on the features of the frame and the face recognition algorithm. Based on the face position information, the features of the frame and the labeled deep fake detection results, the initial detection model is trained.
[0064] The retraining module is used to quantize and transform the weights and activation values of the initial detection model, and to verify the balance between inference speed and accuracy loss. The quantized initial detection model is used as an intermediate model. The intermediate model is then fine-tuned or retrained to obtain the final deepfake detection model.
[0065] The detection module is used to input the video to be deepfake detected into the deepfake detection model and obtain the detection result of whether it belongs to a fake video.
[0066] The image deep fake detection acceleration method based on quantization technology, wherein the training module is used to: identify the face region in the keyframe based on the face location information to obtain the identity information, and train the initial detection model based on the identity information, the face location information, the frame features and the labeled deep fake detection results.
[0067] The image deep fake detection acceleration method based on quantization technology, wherein the quantization conversion includes:
[0068] The weights and activation function of the initial detection model are jointly quantized. The validation set corresponding to the training video is used as calibration data to collect information on the data distribution, including minimum / maximum values, optimal threshold based on entropy theory, and quantization factor based on symmetric quantization. The range of the activation function output is calculated so that the hidden layer output range of the initial detection model can be mapped to [0,1] by passing values through a differential nonlinear function.
[0069] The image deep fake detection acceleration method based on quantization technology, wherein extracting keyframes for fake detection from the training video specifically includes:
[0070] By training a temporal prediction model, the extracted video frame features are encoded and input into the model for decoding and recovery. The score of each frame is obtained based on the encoding and decoding conversion accuracy, and then the key frames are identified.
[0071] The present invention also proposes a storage medium for storing a program that executes any of the image deep pseudo-detection acceleration methods based on quantization techniques.
[0072] This invention also proposes a client for any image deep pseudo-detection acceleration system based on quantization technology.
[0073] It should be understood that the above description of the preferred embodiments is quite detailed, but it should not be considered as a limitation on the scope of protection of this invention. Those skilled in the art, under the guidance of this invention, can make substitutions or modifications without departing from the scope of protection of the claims of this invention, and all such substitutions or modifications fall within the scope of protection of this invention. The scope of protection of this invention should be determined by the appended claims.
Claims
1. A method for accelerating image deep fake detection based on quantization technology, characterized in that, include: Step 1: Obtain the training video with labeled deep fake detection results, extract the key frames for fake detection from the training video, compress the key frames and remove redundant information, and then extract the frame features that retain the key dimensions of fake detection. Step 2: Based on the features of this frame, use a face recognition algorithm to locate the face position information in the key frame. Based on the face position information, the features of this frame, and the labeled deep fake detection results, train the initial detection model. Step 3: Quantize the weights and activation values of the initial detection model and verify the balance between inference speed and accuracy loss. Use the quantized initial detection model as an intermediate model. Fine-tuning or retraining the intermediate model yields the final deepfake detection model. The quantization transformation includes: jointly quantizing the weights and activation function of the initial detection model; using the validation set corresponding to the training video as calibration data; collecting information on the data distribution, including minimum / maximum values, the optimal threshold based on entropy theory, and the quantization factor based on symmetric quantization; calculating the range of the activation function output; and mapping the hidden layer output range of the initial detection model to [0,1] by passing values through a differentiable nonlinear function. Step 4: Input the video to be deepfake detected into the deepfake detection model to obtain the detection result of whether it belongs to a fake video; Step 1, extracting keyframes for forgery detection from the training video, specifically includes: By training a temporal prediction model, the extracted video frame features are encoded and input into the model for decoding and recovery. The score of each frame is obtained based on the encoding and decoding conversion accuracy, and then the key frames are identified.
2. The image deep fake detection acceleration method based on quantization technology as described in claim 1, characterized in that, Step 2 includes: performing identity recognition on the face region in the keyframe based on the face location information to obtain identity information; and training the initial detection model based on the identity information, the face location information, the frame features, and the labeled deep fake detection results.
3. The image deep fake detection acceleration method based on quantization technology as described in claim 1, characterized in that, When compressing the keyframes and removing redundant information, based on the characteristics of the GPU used and the features of the deep fake detection of people, data outside the face region is discarded, and the model is accelerated and optimized using acceleration units deployed for neural networks.
4. An image deep fake detection acceleration system based on quantization technology, characterized in that, include: The extraction module is used to obtain training videos with labeled deep fake detection results, extract key frames for fake detection from the training videos, compress the key frames and remove redundant information, and then extract frame features that retain the key dimensions of fake detection. The training module is used to locate the face position information in the keyframe based on the features of the frame and the face recognition algorithm. Based on the face position information, the features of the frame and the labeled deep fake detection results, the initial detection model is trained. The retraining module is used to quantize and transform the weights and activation values of the initial detection model, and to verify the balance between inference speed and accuracy loss. The quantized initial detection model is used as an intermediate model. Fine-tuning or retraining the intermediate model yields the final deepfake detection model; The quantization transformation includes: jointly quantizing the weights and activation function of the initial detection model, using the validation set corresponding to the training video as calibration data, collecting information on the data distribution, including minimum / maximum values, the optimal threshold based on entropy theory, and the quantization factor based on symmetric quantization, calculating the range of the activation function output, and mapping the hidden layer output range of the initial detection model to [0,1] by passing values through a differential nonlinear function. The detection module is used to input the video to be deepfake detected into the deepfake detection model and obtain the detection result of whether it belongs to a fake video; The extraction module extracts keyframes for forgery detection from the training video, specifically including: By training a temporal prediction model, the extracted video frame features are encoded and input into the model for decoding and recovery. The score of each frame is obtained based on the encoding and decoding conversion accuracy, and then the key frames are identified.
5. The image deep fake detection acceleration system based on quantization technology as described in claim 4, characterized in that, The training module is used to: identify the face region in the keyframe based on the face location information, obtain the identity information, and train the initial detection model based on the identity information, the face location information, the frame features, and the labeled deep fake detection results.
6. A storage medium for storing a program that executes the image deep pseudo-detection acceleration method based on quantization techniques as described in any one of claims 1 to 3.
7. A client for the image deep pseudo-detection acceleration system based on quantization technology as described in claim 4 or 5.