A deepfake detection method based on a multi-modal large model

Abstract

The present application relates to a kind of deep fake detection methods based on multi-modal large model, the method includes the following steps: step 1: for the pre-processing of deep fake data, step 2: original data set is classified, according to the deep fake technology used by each picture, original data set is divided into multiple sub data sets for specific deep fake technology, step 3: two routing modules are trained using the data set obtained in step 1 to perform the deep fake technology classification task of picture, step 4: each sub data set obtained in step 2 is used to train the corresponding fine-tuning module for specific method, step 5: fine-tuning module and basic multi-modal large model are combined to perform deep fake detection task.For picture, preliminary deep fake technology category identification is carried out first, then deep fake detection is carried out, different deep fake method is used to carry out deep fake detection task using special fine-tuning module, and the final detection accuracy is improved.

Description

Technical Field

[0001] This invention relates to a detection method, specifically a deepfake detection method based on a multimodal large model, belonging to the field of large model deep detection technology. Background Technology

[0002] With the rapid development of deepfake technology, its types are becoming increasingly diverse. Driven by the advancement of artificial intelligence, deepfake technology is developing even faster, producing increasingly realistic images that pose serious challenges to information security, privacy protection, and social ethics. Current methods for deepfake detection based on fine-tuning multimodal large models suffer from the following problems: 1. The accuracy of this method is affected by the data distribution in the fine-tuned dataset. For training data containing a small number of deepfake methods, the fine-tuned multimodal large model often struggles to accurately identify these methods. 2. This method leads to knowledge conflicts when the model learns from forged data generated by different deepfake methods. During fine-tuning, features from different deepfake methods may conflict and interfere with each other, reducing the model's deepfake detection accuracy. 3. This method cannot adapt to the rapid evolution of new deepfake methods. For any new deepfake method, this method requires re-fine-tuning the entire large model to detect it, resulting in significant economic costs. This approach results in the performance of large models in deepfake detection often being affected by the distribution of the training set data. It is difficult to achieve accurate detection of images processed by deepfake techniques that occur less frequently in the training set, thus preventing the method from achieving a high detection accuracy. Furthermore, the high overhead of directly fine-tuning the model makes it difficult for this type of method to adapt to the rapid updates of deepfake techniques. Summary of the Invention

[0003] This invention addresses the technical problems existing in the prior art by providing a deepfake detection method based on a multimodal large model. This technical solution first performs preliminary deepfake technology type identification on the image, and then performs deepfake detection. For different deepfake methods, a dedicated fine-tuning module is used to perform deepfake detection tasks, thereby improving the final detection accuracy.

[0004] To achieve the above objectives, the technical solution of the present invention is as follows: a deepfake detection method based on a multimodal large model, the method comprising the following steps:

[0005] Step 1: Preprocess the deepfake data to build a dataset for training models to recognize faces and non-faces, as well as for deepfake techniques that recognize face images.

[0006] Step 2: Classify the original dataset. Based on the deepfake technique used for each image, divide the original dataset into multiple subsets targeting specific deepfake techniques. Simultaneously, train a deepfake dataset of non-face images.

[0007] Step 3: Use the dataset obtained in Step 1 to train two routing modules for a deepfake image classification task.

[0008] Step 4: Use each subset of data obtained in Step 2 to train the corresponding fine-tuning module for the specific method.

[0009] Step 5: For image input, first process it using a multimodal large-scale visual encoder and routing module, and then select the corresponding fine-tuning module based on the processing results. Combine the fine-tuning module with the basic multimodal large-scale model to perform a deepfake detection task.

[0010] Step 1 is as follows: Construct two datasets based on the data in the original dataset. , ,in Used to train a model that can distinguish between human face images and non-human face images. The model used to train the classification of deepfake images is constructed for each deepfake image in the original dataset. The process is as follows: Determine whether each image is a face image. If the image is not a face image, generate a non-face label for that image; otherwise, generate a face label. Construct... The process is as follows: For each deepfake face image, a label is generated indicating the deepfake technique used in that image. The two datasets mentioned above are used to train two routing modules that perform the deepfake technique classification task. , .

[0011] In step 2, the original dataset is categorized. Based on the deepfake technique used in the face deepfake images, the images are added to the subset corresponding to that deepfake technique, along with the original images (those not deepfake). This results in multiple subsets, each containing only deepfake images obtained using a specific deepfake technique and their corresponding original images. Simultaneously, a dataset containing both original and deepfake images is constructed for non-face images.

[0012] In step 3, two routing modules are trained using the dataset obtained in step 1. , This system classifies face images and non-face images, as well as deepfake techniques used in face images, providing a basis for selecting appropriate fine-tuning modules. Each routing module is a multilayer perceptron, whose input is the output of a multimodal large-scale visual encoder, and whose output is the classification result. Used to distinguish whether an image is a human face or a non-human face image; Deepfake technology used to distinguish facial images

[0013] For routing module one, the processing procedure is shown in the following formula:

[0014]

[0015] in, The output of the visual encoder, For routing module one, This indicates whether the image is a human face image.

[0016] For routing module two, the processing procedure is shown in the following formula:

[0017]

[0018] in, The output of the visual encoder, For routing module two, If we categorize face deepfake methods into n classes to represent the possibilities of using each deepfake technique on an image, then... It's an n-dimensional vector representing the distribution across n different deepfake methods for face images. By performing initial classification of the input image and then using a specific fine-tuning module in conjunction with a multimodal large model for deepfake detection, it can overcome the bottleneck in detection accuracy of the multimodal large model when faced with multiple deepfake methods due to mutual interference between different deepfake techniques and uneven data distribution. This improves the accuracy of deepfake detection.

[0019] In step 4, using the dataset obtained in step 2, the multimodal large model is fine-tuned using each subset of the dataset to obtain the corresponding fine-tuning module. The fine-tuning module consists of two parts: a visual adapter and a fine-tuning module for the large language model.

[0020] The vision adapter is attached after the vision encoder, as shown in the following formula:

[0021]

[0022] in For the output of the visual encoder, For the visual adapter, k corresponds to a specific depth-spoofing method for a face image or a non-face depth-spoofing method. The visual encoder output is mapped to a tamper-sensitive subspace to obtain the image's representation in the tamper-sensitive subspace. ;

[0023] The large language model fine-tuning module is combined with the large language model part of the multimodal large model to achieve fine-tuning of its large language model. The fine-tuning module is obtained by using LoRa fine-tuning technology, and the update of model parameters is shown in the following formula:

[0024]

[0025] in These are the parameters for each layer of the original large language model. , The two low-rank matrices obtained from the fine-tuning are the large language model fine-tuning module. The result of multiplying the two matrices is used... Adding the parameters to the original model parameters allows for fine-tuning of the model.

[0026] The large model is fine-tuned using each subset of the dataset, with the fine-tuning process only affecting the visual adapter and... , Two matrices are used to update parameters, ultimately yielding a fine-tuning module for each deepfake method. This module simultaneously fine-tunes the deepfake detection capabilities of the multimodal large model on both the visual and linguistic sides, enabling the model to effectively capture the features of different deepfake methods and employ specific inference methods for particular deepfake techniques. Furthermore, this fine-tuning module has a small number of parameters, allowing the model to maintain low training overhead while improving deepfake detection accuracy.

[0027] In step 5, this method is used to perform a deepfake image detection task, with the input image being... First, it is processed by a visual encoder to obtain... Then the routing module If Determine the image For non-face images, the fine-tuned module trained on the non-face deepfake dataset is directly combined with a multimodal large model for deepfake detection, as shown in the following formula:

[0028]

[0029] in For the input image, The output of the image after processing by the visual encoder. The visual adapter trained using a non-human face deepfake dataset is used to process the results of the visual adapter. Input to the large model, These are the original parameters of the model. This is a fine-tuning module for a large language model trained on a non-face dataset. After parameter updates, the large language model is processed to finally output... , To determine the final output of deepfake detection using a fine-tuned module trained on a non-face dataset combined with a base multimodal large model, if... If the image is identified as a face, then the routing module will then... Process the image and output its distribution across each face depth fakery method.

[0030]

[0031] The subsequent processing of the input image is shown in the following formula:

[0032]

[0033] Corresponding to specific deepfake methods, The output of the image after processing by the visual encoder. and These represent the visual adapter and the fine-tuning module of the large language model corresponding to this deepfake method, respectively. This is the output result of combining the fine-tuning module corresponding to this deepfake method with a multimodal large model.

[0034] The final detection result for the face image is shown in the following formula:

[0035]

[0036] The final detection result is obtained by weighted summation of detection results for different depth forgery methods, where the weights are... By routing module The output result is processed The result is obtained after function processing. The weighted synthesis uses the detection results of fine-tuning modules with different depth forgery methods, so that the model can detect specific depth forgery methods in a specific way while ensuring relatively comprehensive detection results.

[0037] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the deepfake detection method based on a multimodal large model.

[0038] A computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the aforementioned deepfake detection method based on a multimodal large model.

[0039] In this scheme, images are first initially identified for the types of deepfake techniques, and then deepfake detection is performed. For different deepfake methods, a dedicated fine-tuning module is used for deepfake detection tasks to improve the final detection accuracy. The framework is flexible and scalable. For each newly emerging deepfake method, a new fine-tuning module can be trained to provide the framework with detection capabilities for that deepfake technique. The training overhead is low, and LoRa technology is used to achieve model fine-tuning, ensuring that the parameters of the fine-tuning method are efficient.

[0040] Compared with existing technologies, this invention has the following advantages: 1. This method proposes a multi-expert large model for deepfake detection. Data on different deepfake techniques are divided into corresponding subsets, and each subset is used to train a separate visual adapter and fine-tuning module. This solves the problem in the original method where uneven data distribution leads to low detection accuracy for deepfake techniques with a small number of data points in the original dataset; 2. This method proposes a multi-expert large model for deepfake detection. This method divides data from different deepfake techniques into corresponding subsets and trains a visual adapter and fine-tuning module separately using each subset. This solves the knowledge conflict problem caused by simultaneously learning data from different deepfake techniques in the original method. Different deepfake techniques generate data with different characteristics and patterns, and learning from different deepfake data simultaneously can lead to the model's inability to accurately capture the features of various deepfake techniques, resulting in a decrease in the model's deepfake detection accuracy. Secondly, the original method requires expanding the original dataset and re-fine-tuning the large multimodal model to recognize newly emerging deepfake techniques, incurring significant overhead. This method only requires training an additional visual adapter, fine-tuning module, and two smaller routing modules during the fine-tuning process, effectively reducing the training overhead for newly emerging deepfake techniques. Thirdly, this method uses the visual adapter and the fine-tuning module of the large language model to simultaneously fine-tune the large multimodal model, enabling the large multimodal model to capture relevant visual features and adopt corresponding inference methods for deepfake detection when facing different deepfake techniques, effectively improving the accuracy of deepfake detection. 5. This method employs a multi-granularity routing module to process the input images. First, it separates face images from non-face images, and then further separates them according to the deepfake method used for the face images. This mechanism effectively avoids interference from non-face images on the accuracy of deepfake detection, improving the accuracy of multimodal large-model deepfake detection. Attached Figure Description

[0041] Figure 1. Flowchart of the deepfake detection method using a multi-expert large model. Detailed Implementation

[0042] To enhance understanding of the present invention, the embodiments will be described in detail below with reference to the accompanying drawings.

[0043] Example 1: See Figure 1 This document describes an embodiment of a deepfake detection method based on a multi-expert large model. In this embodiment, Qwen2.5-VL-Instruct is used as the basic multi-modal large model. The method includes the following steps:

[0044] S1: Construct two datasets based on the data in the original dataset. , .in Used to train a model that can distinguish between human face images and non-human face images. This is used to train a model that classifies deepfake images used to perform face image classification. For each deepfake image in the original dataset, a model is constructed. The process is as follows: Determine whether each image is a face image. If the image is not a face image, generate a non-face label for that image; otherwise, generate a face label. (Construction) The process is as follows: For each deepfake face image, a label is generated indicating the deepfake technique used in that image. The two datasets mentioned above are used to train two routing modules to perform the deepfake technique classification task. , .

[0045] S2: Classify the original dataset. Based on the deepfake face image used, add the image to the corresponding subset of the deepfake technology. Simultaneously, add the original image (the image without deepfake technology) to the subset. After these operations, multiple subsets are obtained, each containing only the deepfake image obtained using a specific deepfake technology and its corresponding original image. A dataset containing both the original and deepfake images is also constructed for non-face images. In this embodiment, face deepfake technologies are divided into four main categories: Face Switching (FS), Face Motion Transfer (FR), Face Synthesis (EFS), and Face Editing (FE). Combined with the non-face (NF) deepfake dataset, five subsets are ultimately obtained. , , , , , respectively represent a deepfake dataset using face switching, a deepfake dataset using facial motion transfer, a deepfake dataset using face synthesis, a deepfake dataset using face editing, and a non-face deepfake dataset.

[0046] S3: Using the dataset obtained in S1 , Two routing modules were obtained by training on two datasets. , These modules are used to classify face images and non-face images, as well as various deepfake techniques used to create face images, providing a basis for selecting the appropriate fine-tuning module. Each routing module is a multilayer perceptron, whose input is the output of the visual encoder, and whose output is the classification result. Specifically, routing module one distinguishes whether an image is a face or not; routing module two distinguishes the deepfake technique used in the face image.

[0047] For routing module one, the processing procedure is shown in the following formula:

[0048]

[0049] in, The output of the visual encoder, For routing module one, This indicates whether the image is a human face.

[0050] For routing module two, the processing procedure is shown in the following formula:

[0051]

[0052] in, The output of the visual encoder, This is the second routing module, used to distinguish between face images and non-face images. Given an input face image, the possibilities for applying each deepfake technique are categorized into n classes. It's an n-dimensional vector representing the distribution across n methods. This method categorizes deep face spoofing techniques into four main classes: FS, FR, EFS, and FE. That is, the distribution based on the above four methods. .

[0053] S4: Using the dataset obtained in step 2 , , , , The corresponding fine-tuning modules are obtained by fine-tuning the multimodal large model using each subset of data.

[0054] The fine-tuning module consists of two parts: the visual adapter and the large language model fine-tuning module.

[0055] The vision adapter is attached after the vision encoder, as shown in the following formula:

[0056]

[0057] in For the output of the visual encoder, For the visual adapter, k corresponds to a specific face depth spoofing technique or non-face depth spoofing technique. The output of the visual encoder is mapped to a tamper-sensitive subspace to obtain its representation in the tamper-sensitive subspace. .

[0058] The large language model fine-tuning module is combined with the large language model of the multimodal large model to achieve fine-tuning of the large language model. This fine-tuning module adopts LoRa technology, and the update of model parameters is shown in the following formula:

[0059]

[0060] in These are the parameters for each layer of the original large language model. , The two low-rank matrices obtained from fine-tuning constitute the large language model fine-tuning module. The result of multiplying these two matrices is then used... The parameters are added to the original model to achieve fine-tuning of the large language model.

[0061] The multimodal large model was fine-tuned using each subset of the dataset, with the fine-tuning process focusing only on the visual adapter and... , The matrix is ​​updated with parameters, ultimately resulting in large model fine-tuning modules corresponding to four deepfake face techniques and non-deepfake face techniques.

[0062] S5: This method employs a multi-expert large model for deepfake detection to perform deepfake image recognition tasks. Input image First, it is processed by a visual encoder to obtain... Then the routing module Processing. If If the image is determined to be a non-face image, then the fine-tuned module trained on the non-face deepfake dataset is directly combined with the multimodal large model to perform deepfake detection. The process is shown in the following formula:

[0063]

[0064] in For the input image, The output of the image after processing by the visual encoder. This is a visual adapter trained using a non-human face deepfake dataset. Input to the large model, These are the original parameters of the model. This is a fine-tuned module trained on a non-face dataset, processed by a larger model with updated parameters, and the final output is... . This refers to the output of deepfake detection after combining the fine-tuned module trained on a non-face dataset with the larger model. If the image is identified as a face, then the routing module will then... Process the image and output its distribution across each deepfake face technique.

[0065]

[0066] In this embodiment, In order to be in Distribution of four deepfake face technologies.

[0067] The subsequent processing of the input image is shown in the following formula:

[0068]

[0069] Corresponding to specific deepfake techniques, The final detection result of the face image processed by the visual encoder is shown in the following formula:

[0070] The final detection result is obtained by weighted summation of detection results for different depth-of-spoofing techniques. The weights representing different depth forgery techniques are determined by the routing module. The output result is processed The result is obtained after function processing.

[0071] The existing dataset exhibits uneven distribution of data for different deepfake techniques. Compared to methods that directly use the entire dataset to fine-tune a large multimodal model for deepfake detection, this method improves detection accuracy across all five deepfake techniques, and achieves more consistent accuracy across each technique. (Analysis follows the output table.) and These represent the visual adapter corresponding to this deepfake technology and the fine-tuning module of the large model, respectively. This is the output result of combining the fine-tuning module corresponding to this deepfake technology with a multimodal large model.

[0072] 1. According to the data in Table 2, the FS and FR methods have relatively few training samples and the features of deep forgery are relatively difficult to capture. Therefore, the accuracy of deep forgery detection obtained by directly fine-tuning the multimodal large model is not high. Our method significantly improves the accuracy of deep forgery detection of the model for the FS and FR methods.

[0073] Table 1:

[0074]

[0075] Table 2

[0076]

[0077] It should be noted that the above embodiments are not intended to limit the scope of protection of the present invention. Equivalent transformations or substitutions made based on the above technical solutions all fall within the scope of protection of the claims of the present invention.

Claims

1. A deepfake detection method based on a multimodal large model, characterized in that, The method includes the following steps: Step 1: Preprocess the deepfake data to build a dataset for training models to recognize faces and non-faces, as well as for deepfake techniques that recognize face images. Step 2: Classify the original dataset. Based on the deepfake technique used for each image, divide the original dataset into multiple subsets targeting specific deepfake techniques. Simultaneously, train a deepfake dataset of non-face images. Step 3: Use the dataset obtained in Step 1 to train two routing modules for a deepfake image classification task. Step 4: Use each subset of data obtained in Step 2 to train the corresponding fine-tuning module for the specific method. Step 5: For image input, first use the visual encoder and routing module of the multimodal large model to process it. Select the corresponding fine-tuning module according to the processing result, combine the fine-tuning module with the basic multimodal large model, and perform the deepfake detection task.

2. The deepfake detection method based on a multimodal large model according to claim 1, characterized in that, Step 1 is as follows: Construct two datasets based on the data in the original dataset. , ,in Used to train a model that can distinguish between human face images and non-human face images. The model used to train the classification of deepfake images is constructed for each deepfake image in the original dataset. The process is as follows: Determine whether each image is a face image. If the image is not a face image, generate a non-face label for that image; otherwise, generate a face label. Construct... The process is as follows: For each deepfake face image, a label is generated indicating the deepfake technique used in that image. The two datasets mentioned above are used to train two routing modules that perform the deepfake technique classification task. , .

3. The deepfake detection method based on a multimodal large model according to claim 2, characterized in that, In step 2, the original dataset is classified. Based on the deepfake method used in the face deepfake image, the image is added to the subset corresponding to that deepfake method. At the same time, the original image corresponding to the image, i.e. the image without deepfake, is added to the subset, resulting in multiple subsets. Each subset contains only the deepfake image obtained using a specific deepfake technique and the original image corresponding to that image. Meanwhile, a dataset containing both the original image and the deepfake image is constructed for non-face images.

4. The deepfake detection method based on a multimodal large model according to claim 3, characterized in that, In step 3, two routing modules are trained using the dataset obtained in step 1. , This system classifies face images and non-face images, as well as deepfake techniques used in face images, providing a basis for selecting appropriate fine-tuning modules. Each routing module is a multilayer perceptron, whose input is the output of a multimodal large-scale visual encoder, and whose output is the classification result. Used to distinguish whether an image is a human face or a non-human face image; Deepfake technology used to distinguish facial images For routing module one, the processing procedure is shown in the following formula: in, The output of the visual encoder, For routing module one, This indicates whether the image is a human face image. For routing module two, the processing procedure is shown in the following formula: in, The output of the visual encoder, For routing module two, If we categorize face deepfake methods into n classes to represent the possibilities of using each deepfake technique on an image, then... It is an n-dimensional vector representing the distribution across n different deepfake methods for face images.

5. The deepfake detection method based on a multimodal large model according to claim 4, characterized in that, In step 4, using the dataset obtained in step 2, the multimodal large model is fine-tuned using each subset of the dataset to obtain the corresponding fine-tuning module. The fine-tuning module consists of two parts: a visual adapter and a fine-tuning module for the large language model. The vision adapter is attached after the vision encoder, as shown in the following formula: in, For the output of the visual encoder, For the visual adapter, k corresponds to a specific depth-spoofing method for a face image or a non-face depth-spoofing method. The visual encoder output is mapped to a tamper-sensitive subspace to obtain the image's representation in the tamper-sensitive subspace. ; The large language model fine-tuning module is combined with the large language model part of the multimodal large model to achieve fine-tuning of its large language model. The fine-tuning module is obtained by using LoRa fine-tuning technology, and the update of model parameters is shown in the following formula: in These are the parameters for each layer of the original large language model. , The two low-rank matrices obtained from the fine-tuning process constitute the fine-tuning module, which utilizes the product of these two matrices. Adding this to the original model parameters allows for fine-tuning of the model, ultimately resulting in equivalent parameters. , The large model is fine-tuned using each subset of the dataset, with the fine-tuning process only affecting the visual adapter and... , The parameters of the two matrices are updated to obtain the fine-tuning module corresponding to each depth forgery method.

6. The deepfake detection method based on a multimodal large model according to claim 5, characterized in that, In step 5, this method is used to perform a deepfake image detection task, with the input image being... First, it is processed by a visual encoder to obtain... Then the routing module If Determine the image For non-face images, the fine-tuned module trained on the non-face deepfake dataset is directly combined with a multimodal large model for deepfake detection, as shown in the following formula: in For the input image, The output of the image after processing by the visual encoder. The visual adapter trained using a non-human face deepfake dataset is used to process the results of the visual adapter. Input to the large model, These are the original parameters of the model. This is a fine-tuning module for a large language model trained on a non-face dataset. After parameter updates, the large language model is processed to finally output... , To determine the final output of deepfake detection using a fine-tuned module trained on a non-face dataset combined with a base multimodal large model, if... If the image is identified as a face, then the routing module will then... Process the image and output its distribution across each face depth fakery method. The subsequent processing of the input image is shown in the following formula: For specific deepfake methods targeting human faces or non-human face deepfake methods, The output of the image after processing by the visual encoder. and These represent the visual adapter and the fine-tuning module of the large language model corresponding to this deepfake method, respectively. This is the output result of combining the fine-tuning module corresponding to this deepfake method with a multimodal large model. The final detection result for the face image is shown in the following formula: The final detection result is obtained by weighted summation of detection results for different depth forgery methods, where the weights are... By routing module The output result is processed The result is obtained after function processing.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the deepfake detection method based on a multimodal large model as described in any one of claims 1 to 6.

8. A computer-readable storage medium storing computer instructions thereon, characterized in that, When executed by the processor, the computer instructions implement the deep forgery detection method based on a multimodal large model as described in any one of claims 1-6.

Images