Multi-mode depth forgery detection method and device

Abstract

The invention provides a multi-mode depth forgery detection method and device, and the method comprises the steps: obtaining a video frame sequence and an audio sequence of a target source video, carrying out the feature extraction of the video frame sequence based on a preset visual feature extraction model, and obtaining a visual feature, performing feature extraction on the audio sequence based on a pre-trained audio spectrogram coding model to obtain audio features, fusing the visual features and the audio features to obtain corresponding fusion features, obtaining a preset target multi-modal deep counterfeiting detection model, performing counterfeiting identification on the fusion features based on the target multi-modal deep counterfeiting detection model, and obtaining a target multi-modal deep counterfeiting detection model. And determining an identification result of the target source video.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and more specifically, to a multimodal deep fake detection method and device. Background Art

[0002] Deepfake is a type of multimedia data that uses deep learning algorithms to regenerate and tamper with original videos. The technologies used to generate deepfake content can be mainly divided into deepfake video technology and deepfake voice technology. Deepfake video technology can generate or tamper with original visual data, such as facial and lip changes; while deepfake voice technology can generate audio information, such as recordings. Combining these two technologies can create quite realistic fake content, which may cause serious violations of personal privacy.

[0003] Currently, there are two methods for deep fake video detection: image-level and video-level. Image-level methods mainly perform single-frame analysis based on visual clues, while video-level methods focus more on cross-frame inconsistencies.

[0004] However, most existing methods only use images or videos for detection and find it difficult to fully utilize information from other modalities. Therefore, in multimodal forgery scenarios, the detection accuracy of deep fake content is average. Summary of the invention

[0005] In view of this, the purpose of the present application is to provide a multimodal deep fake detection method and device, which can obtain visual features and audio features by multimodal feature extraction, that is, extracting features from video frame sequences and audio sequences through a visual feature extraction model and an audio spectrogram encoding model. After the two modal features are fused, forgery identification is performed based on a preset target multimodal deep fake detection model to obtain corresponding identification results. This application pays special attention to the comprehensive analysis of the two modalities of vision and audio, and can make full use of the information of other modalities to improve the detection accuracy of carefully crafted multimodal deep fake content.

[0006] In a first aspect, an embodiment of the present application provides a multimodal deep fake detection method, the method comprising:

[0007] Acquire a video frame sequence and an audio sequence of a target source video; wherein the target source video includes a real source video and a forged source video;

[0008] Extracting features from the video frame sequence based on a preset visual feature extraction model to obtain visual features, and extracting features from the audio sequence based on a pre-trained audio spectrogram coding model to obtain audio features;

[0009] fusing the visual feature and the audio feature to obtain a corresponding fused feature;

[0010] A preset target multimodal deep fake detection model is obtained, and the fusion feature is subjected to forgery identification based on the target multimodal deep fake detection model to determine an identification result of the target source video; wherein the identification result indicates whether the target source video corresponding to the fusion feature is forged, and a forgery method of the target source video when it is determined to be forged; the forgery method includes at least lip editing, lip generation, and face reproduction.

[0011] In a possible implementation, the method further includes:

[0012] Constructing corresponding positive and negative sample pairs based on the visual features and the audio features;

[0013] The multimodal deep fake detection model is trained by cross-modal contrast learning based on the positive and negative sample pairs to obtain the trained multimodal deep fake detection model.

[0014] In a possible implementation, the step of obtaining a video frame sequence and an audio sequence of a target source video includes:

[0015] Obtain a multimodal dataset including a plurality of deep fake contents; wherein each deep fake content includes a video frame sequence and an audio sequence of a corresponding deep fake video;

[0016] From the multimodal dataset, video frame sequences and audio sequences of the target source video are randomly extracted at equal intervals according to preset intervals.

[0017] In a possible implementation, before extracting features from the video frame sequence based on a preset visual feature extraction model to obtain visual features, the method further includes:

[0018] The corresponding random standardized training strategy is determined based on the pre-set normal distribution hyperparameters, the mean and standard deviation of the target image dataset;

[0019] The video frame sequence is randomly standardized based on a preset probability to obtain a mean and a standard deviation of random sampling, the video frame sequence is standardized based on the mean and the standard deviation of the random sampling, and the mean and the standard deviation of the target image data set are used for standardization.

[0020] In a possible implementation, the visual feature extraction model includes an image-level feature extraction sub-model and a temporal feature extraction sub-model; and extracting features from the video frame sequence based on a preset visual feature extraction model to obtain visual features includes:

[0021] For the video frame sequence, sampling a first number of video frame sequence groups from the video frame sequence in equal steps based on a preset sampling step; wherein each video frame sequence group includes a second number of continuous video frame sequences;

[0022] Based on all the sampled video frames, the image-level feature extraction sub-model is used to extract corresponding image-level features; wherein each video frame sequence group corresponds to four image-level features;

[0023] For each image-level feature corresponding to each video frame sequence group, the corresponding group-level feature is obtained based on the temporal feature extraction sub-model, and all the group-level features are average-pooled to obtain the corresponding visual feature.

[0024] In a possible implementation, extracting features from the audio sequence based on a pre-trained audio spectrogram coding model to obtain audio features includes:

[0025] For the audio sequence, convert the audio sequence into a corresponding spectrogram;

[0026] The spectrogram is input into the audio spectrogram encoding model to extract the audio features of the audio sequence.

[0027] In a possible implementation, constructing corresponding positive and negative sample pairs based on the visual features and the audio features includes:

[0028] Acquire positive and negative sample pairs based on the real source video and the forged source video; wherein the positive and negative sample pairs include a positive sample pair and a negative sample pair; the positive sample pair includes visual features and audio features of the same real source video; the negative sample pair includes visual features and audio features of the same forged source video, or visual features and audio features of different target source videos;

[0029] The performing cross-modal contrastive learning training on the multimodal deep fake detection model based on the positive and negative sample pairs includes:

[0030] Based on the positive and negative sample pairs, a preset contrast loss function is used to calculate the contrast loss of the target source video to perform cross-modal contrast learning training on the multimodal deep fake detection model.

[0031] In a second aspect, an embodiment of the present application further provides a multimodal deep fake detection device, the device comprising:

[0032] A first acquisition module is used to acquire a video frame sequence and an audio sequence of a target source video; wherein the target source video includes a real source video and a forged source video;

[0033] An extraction module, configured to extract features from the video frame sequence based on a preset visual feature extraction model to obtain visual features, and to extract features from the audio sequence based on a pre-trained audio spectrogram coding model to obtain audio features;

[0034] A fusion module, used for fusing the visual features and the audio features to obtain corresponding fusion features;

[0035] An identification module is used to obtain a preset target multimodal deep fake detection model, perform forgery identification on the fused features based on the target multimodal deep fake detection model, and determine an identification result of the target source video; wherein the identification result indicates whether the target source video corresponding to the fused features is forged, and the forgery method of the target source video when it is determined to be forged; the forgery method at least includes lip editing, lip generation, and face reproduction.

[0036] In a possible implementation, the multimodal deep fake detection device further includes:

[0037] A construction module, used to construct corresponding positive and negative sample pairs based on the visual features and the audio features;

[0038] A training module is used to perform cross-modal contrast learning training on the multimodal deep fake detection model based on the positive and negative sample pairs to obtain the trained multimodal deep fake detection model.

[0039] In a possible implementation manner, the first acquisition module is specifically configured to:

[0040] Obtain a multimodal dataset including a plurality of deep fake contents; wherein each deep fake content includes a video frame sequence and an audio sequence of a corresponding deep fake video;

[0041] From the multimodal dataset, video frame sequences and audio sequences of the target source video are randomly extracted at equal intervals according to preset intervals.

[0042] In a possible implementation, the multimodal deep fake detection device further includes:

[0043] A determination module, used for determining a corresponding random standardized training strategy based on a preset normal distribution hyperparameter, a mean and a standard deviation of a target image data set before extracting features from the video frame sequence based on a preset visual feature extraction model to obtain visual features;

[0044] A standard module is used to randomly standardize the video frame sequence based on a preset probability to obtain a mean and a standard deviation of random sampling, standardize the video frame sequence based on the mean and the standard deviation of the random sampling, and standardize with the mean and the standard deviation of the target image data set.

[0045] In a possible implementation, the visual feature extraction model includes an image-level feature extraction sub-model and a temporal feature extraction sub-model; the extraction module is specifically used to:

[0046] For the video frame sequence, sampling a first number of video frame sequence groups from the video frame sequence in equal steps based on a preset sampling step; wherein each video frame sequence group includes a second number of continuous video frame sequences;

[0047] Based on all the sampled video frames, the image-level feature extraction sub-model is used to extract corresponding image-level features; wherein each video frame sequence group corresponds to four image-level features;

[0048] For each image-level feature corresponding to each video frame sequence group, the corresponding group-level feature is obtained based on the temporal feature extraction sub-model, and all the group-level features are average-pooled to obtain the corresponding visual feature.

[0049] In a possible implementation, the extraction module is specifically used to:

[0050] For the audio sequence, convert the audio sequence into a corresponding spectrogram;

[0051] The spectrogram is input into the audio spectrogram encoding model to extract the audio features of the audio sequence.

[0052] In a possible implementation manner, the building block is specifically used to:

[0053] Acquire positive and negative sample pairs based on the real source video and the forged source video; wherein the positive and negative sample pairs include a positive sample pair and a negative sample pair; the positive sample pair includes visual features and audio features of the same real source video; the negative sample pair includes visual features and audio features of the same forged source video, or visual features and audio features of different target source videos;

[0054] The construction module is specifically used to: calculate the contrast loss of the target source video based on the positive and negative sample pairs using a preset contrast loss function, so as to perform cross-modal contrast learning training on the multimodal deep fake detection model.

[0055] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium through the bus, and the processor executes the machine-readable instructions to perform the steps of the multimodal deep fake detection method as described in any one of the first aspects.

[0056] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the multimodal deep fake detection method described in any one of the first aspects are executed.

[0057] The embodiment of the present application provides a multimodal deep fake detection method and device, which obtains a video frame sequence and an audio sequence of a target source video, extracts features from the video frame sequence based on a preset visual feature extraction model to obtain visual features, extracts features from the audio sequence based on a pre-trained audio spectrogram coding model to obtain audio features, fuses the visual features and audio features to obtain corresponding fused features, obtains a preset target multimodal deep fake detection model, performs forgery identification on the fused features based on the target multimodal deep fake detection model, and determines the identification result of the target source video. The multimodal deep fake detection method of the present application extracts features from the video frame sequence and the audio sequence through multimodal feature extraction, i.e., extracts visual features and audio features through a visual feature extraction model and an audio spectrogram coding model, and after the two modal features are fused, performs forgery identification based on a preset target multimodal deep fake detection model to obtain a corresponding identification result. The present application pays special attention to the comprehensive analysis of the two modalities of vision and audio, which can make full use of the information of other modalities and improve the detection accuracy of carefully crafted multimodal deep fake content.

[0058] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0060] Figure 1 is a flow chart of a multimodal deep fake detection method according to an embodiment of the present application;

[0061] Figure 2is a schematic diagram of the multimodal deepfake detection and training process;

[0062] Figure 3 is a flow chart of a multimodal deep fake detection method according to another embodiment of the present application;

[0063] Figure 4 is a schematic structural diagram of a multimodal deep fake detection device according to an embodiment of the present application;

[0064] Figure 5 It is a structural diagram of a computer device according to an embodiment of the present application. DETAILED DESCRIPTION

[0065] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the present application, or remove one or more operations from the flowchart.

[0066] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0067] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.

[0068] Considering that deepfake is a type of multimedia data that uses deep learning algorithms to regenerate and tamper with original videos. The technologies used to generate deepfake content can be mainly divided into deepfake video technology and deepfake voice technology. Deepfake video technology can generate or tamper with original visual data, such as facial and lip changes; while deepfake voice technology can generate audio information, such as recording content. Combining these two technologies can create quite realistic fake content, which may cause serious violations of personal privacy.

[0069] Currently, there are two methods for deep fake video detection: image-level and video-level. Image-level methods mainly analyze single frames based on visual clues, while video-level methods focus more on cross-frame inconsistencies. However, most existing methods only use images or videos for detection, and it is difficult to fully utilize information from other modalities. Therefore, in multimodal forgery scenarios, the detection accuracy of deep fake content is average.

[0070] To address this problem, the present application provides a multimodal deep fake detection method and device, which obtains visual features and audio features by extracting features from video frame sequences and audio sequences through multimodal feature extraction, i.e., through a visual feature extraction model and an audio spectrogram encoding model. The two modal features are fused and then forgery identification is performed based on a preset target multimodal deep fake detection model to obtain corresponding identification results. The present application pays special attention to the comprehensive analysis of the two modalities of vision and audio, which can make full use of information from other modalities and improve the detection accuracy of carefully crafted multimodal deep fake content.

[0071] Figure 1 is a flowchart of a multimodal deep fake detection method according to an embodiment of the present application, such as Figure 1 As shown, the multimodal deep fake detection method of the embodiment of the present application may specifically include:

[0072] S101, obtaining a video frame sequence and an audio sequence of a target source video.

[0073] S102, extracting features from a video frame sequence based on a preset visual feature extraction model to obtain visual features, and extracting features from an audio sequence based on a pre-trained audio spectrogram coding model to obtain audio features.

[0074] S103, fusing the visual features and the audio features to obtain corresponding fused features.

[0075] S104, obtaining a preset target multimodal deep fake detection model, performing forgery identification on the fused features based on the target multimodal deep fake detection model, and determining an identification result of the target source video.

[0076] The above-mentioned multimodal deep fake detection method obtains visual features and audio features through multimodal feature extraction, that is, feature extraction of video frame sequences and audio sequences through visual feature extraction model and audio spectrogram encoding model. After the two modal features are fused, forgery identification is performed based on a preset target multimodal deep fake detection model to obtain corresponding identification results. This application pays special attention to the comprehensive analysis of visual and audio modalities, which can make full use of information from other modalities and improve the detection accuracy of carefully crafted multimodal deep fake content.

[0077] The above exemplary steps of the embodiment of the present application are described below with reference to specific examples:

[0078] S101, obtaining a video frame sequence and an audio sequence of a target source video.

[0079] In an embodiment of the present application, the target source video includes a real source video and a forged source video. The forged source video is a deep forged video. The target source video is a source video from which two modal features, namely a video frame sequence and an audio sequence, come. The video frame sequence (also known as a frame sequence) and audio sequence of the target source video are acquired for subsequent processing.

[0080] It should be noted that the present application does not impose too many restrictions on the specific method of obtaining the video frame sequence and audio sequence of the target source video, and can be set according to actual conditions.

[0081] As a possible implementation, a multimodal dataset including multiple deep fake contents is obtained; from the multimodal dataset, the video frame sequence and audio sequence of the target source video are randomly extracted at equal intervals according to a preset interval. Each deep fake content includes the video frame sequence and audio sequence of the corresponding deep fake video, and also includes the audio sample length of the audio sequence. For example, a multimodal dataset containing n deep fakes is obtained. Among them, V i ∈ (F×W×H×X) is the frame sequence (video frame sequence) of the i-th deep fake video, where F, W, H and C represent the number of frames, width, height and number of channels respectively, and a i ∈ L is the audio sequence of the i-th deep fake video, where L represents the length of the audio sample. The video frame sequence and audio sequence of the target source video are randomly extracted from this dataset D as input data, for example, Figure 2 As shown, the data input is randomly extracted from the dataset D for multimodal input.

[0082] Optionally, before extracting features from the video frame sequence based on a preset visual feature extraction model to obtain visual features, a corresponding random standardization training strategy is determined based on preset normal distribution hyperparameters, the mean and standard deviation of the target image dataset; the video frame sequence is randomly standardized based on a preset probability to obtain the mean and standard deviation of random sampling, the video frame sequence is standardized based on the mean and standard deviation of random sampling, and the video frame sequence is standardized with the mean and standard deviation of the target image dataset. Among them, the normal distribution hyperparameters are preset by the user based on work experience and specific application scenarios; the residual network model is the ResNet model, and the target image dataset is the ImageNet dataset. For example, continuing to use the random standardization training strategy to train the above-mentioned video frame sequence number, σ and μ are the mean and standard deviation of the ImageNet dataset respectively. Figure 2 As shown, in the multimodal feature extraction module, random normalization is performed (training phase), and it can be seen that this involves encoding of the sequence.

[0084] It should be noted that the input visual data, i.e., the video frame sequence, is perturbed using a random normalization training strategy based on a preset probability to simulate data input from different domains. That is, in the data preprocessing of the training phase, random normalization is used to simulate data input from different domains according to a certain probability on the input video frame sequence, which can improve the generalization ability of the model detection while also maintaining the learning ability of the original domain data; the input data is first standardized with the mean and standard deviation of random sampling, and then standardized with the mean and standard deviation of ImageNet, which improves the adaptability of the model to new scenarios.

[0085] S102, extracting features from a video frame sequence based on a preset visual feature extraction model to obtain visual features, and extracting features from an audio sequence based on a pre-trained audio spectrogram coding model to obtain audio features.

[0086] In an embodiment of the present application, the visual feature is the visual feature obtained after feature extraction of the video frame sequence of the target source video, the audio spectrogram encoding model can be an audio spectrogram Transformer model, and the audio feature is the audio feature obtained after feature extraction of the audio sequence of the target source video. The visual feature is extracted from the video frame sequence according to the visual feature extraction model, and the audio feature is extracted from the audio sequence according to the audio spectrogram encoding model to obtain the audio feature, thereby obtaining the features of the two modalities of visual feature and audio feature for subsequent processing.

[0087] It should be noted that the visual feature extraction model includes an image-level feature extraction sub-model and a temporal feature extraction sub-model. The image-level feature extraction sub-model may be a residual network (ResNet), a densenet network, etc., and the temporal feature extraction sub-model may be a transformer model.

[0088] Optionally, when extracting features from a video frame sequence based on a preset visual feature extraction model to obtain visual features, for the video frame sequence, a first number of video frame sequence groups are sampled from the video frame sequence at equal steps based on a preset sampling step; based on all the sampled video frames, the corresponding image-level features are extracted using an image-level feature extraction submodel; for each image-level feature corresponding to a video frame sequence group, the corresponding group-level features are obtained based on a temporal feature extraction submodel, and all group-level features are average-pooled to obtain corresponding visual features. Each video frame sequence group includes a second number of continuous video frame sequences, the sampling step is a preset step for sampling the video frame sequence, each video frame sequence group corresponds to four image-level features, and the temporal self-attention model is a temporal Transformer model.

[0089] For example, for the input video frame sequence V i , the first number is 4, 4 groups are sampled from the sequence with equal steps, each group contains 4 consecutive frames, for the sampled 16 frames, an image-level feature extraction sub-model, namely ResNet model I(·) is used to extract the corresponding image-level features, and then the image-level features of each group are extracted through a temporal feature extraction sub-model, namely the temporal Transformer model E V (·) Get a group-level feature, and finally get the visual feature by averaging the four group-level features

[0090] Optionally, when extracting features from an audio sequence based on a pre-trained audio spectrogram coding model to obtain audio features, the audio sequence is converted into a corresponding spectrogram; the spectrogram is input into the audio spectrogram coding model to extract the audio features of the audio sequence. For example, for an audio sequence a i , the model converts it into a spectrogram, and then inputs it into an audio spectrogram encoding model, namely the Transformer model E a (·), and finally get the audio features

[0091] Therefore, in the training phase, random normalization is used to process the video frame sequence, and the image-level feature extraction sub-model and the temporal feature extraction sub-model, namely the ResNet model and the temporal Transformer model, are used to extract frame-level features of the video frame sequence. The pre-trained audio spectrogram coding model, namely the Transformer model, is used to extract audio features. The residual network model and the audio spectrogram coding model are used to extract features of the video frame sequence and the audio sequence to obtain visual features and audio features.

[0092] S103, fusing the visual features and the audio features to obtain corresponding fused features.

[0093] In the embodiment of the present application, feature fusion is performed on the visual features and audio features extracted in step S102 to obtain corresponding fused features for subsequent processing.

[0094] As a possible implementation, the video frame features and the audio frame features are fused based on average pooling, self-attention layer, and cross-attention layer. Optionally, the visual features are processed based on the self-attention layer to obtain processed visual features; and the processed visual features and audio features are fused by the cross-attention layer. In short, the features extracted from the two modalities are fused using the self-attention layer and the cross-attention layer to obtain fused features. For example, Figure 2 As shown in the figure, in the feature information fusion module, the visual features and audio features are first processed by the self-attention layer (Self-Attn) Among them, SA is the self-attention layer. Then both are input into the feature fusion module for feature fusion f (a ,t,v) =MHA(FFN(f (a,t) ),f v ,f v ), obtain the fusion feature f (a,t) =MHA(f t ,f a ,f a ), where MHA is the multi-head attention layer.

[0095] S104, obtaining a preset target multimodal deep fake detection model, performing forgery identification on the fused features based on the target multimodal deep fake detection model, and determining an identification result of the target source video.

[0096] In an embodiment of the present application, the target multimodal deep fake detection model is a multimodal deep fake detection model created by the present application, namely, the R-MFDN model. The identification result characterizes whether the target source video corresponding to the fusion feature is forged, and the forgery method of the target source video when it is determined to be forged. The forgery method includes at least lip editing, lip generation, and face reproduction. That is, the identification result of the target source video may include two parts, one part indicates whether it is a forged video (real or forged), and the other part indicates the category of the forged video (that is, the forgery method cited by the forged video), such as lip editing, lip generation, face reproduction, etc. The target multimodal deep fake detection model is obtained, and the fusion feature obtained in step S103 is forged based on the target multimodal deep fake detection model. The identification result of the target source video is determined, thereby completing the deep fake detection. For example, Figure 2 As shown, the fusion feature is passed to the forgery identification module to classify and judge the deep forgery content. Specifically, the above fusion feature f (a,t,v) Pass it to the forgery identification module to calculate the identification result.

[0097] It can be added that in the training phase, the fused features are classified and the classification loss is calculated using the cross entropy loss function. In the forgery identification phase after training, there is no need to use the cross entropy loss function to calculate the classification loss. In short, the various loss functions and corresponding loss calculations used in this application, as well as the cross-modal contrast learning training strategy are all the processes required for training the target multimodal deep forgery detection model. After the training is completed, these processes are no longer needed, and the trained target multimodal deep forgery detection model can be directly used for the forgery identification phase, that is, reasoning.

[0098] It should be noted that the following Table 1 is the evaluation comparison results of the target multimodal deep fake detection model proposed in this application, namely the R-MFDN model and multiple benchmark models on the IDForge dataset (a large-scale multimodal forgery dataset).

[0099] Table 1

[0100]

[0101]

[0102] It can be seen that the R-MFDN model of this application performed best in the evaluation, achieving an accuracy of 90.63% in the second classification (true or false), which is 1.45%, 4.42% and 10.75% higher than RealForensics, VFD and CDCN respectively. These results demonstrate the effectiveness of the R-MFDN of this application in the multimodal deep fake detection task.

[0103] It should also be added that the target multimodal deep fake detection model of the present application is a model similar to the video-level detection method, which can capture cross-frame inconsistencies and extract multimodal information sensitive to forgery for detection.

[0104] The multimodal deep fake detection method provided in the embodiment of the present application obtains the video frame sequence and audio sequence of the target source video, extracts features from the video frame sequence based on a preset visual feature extraction model to obtain visual features, extracts features from the audio sequence based on a pre-trained audio spectrogram coding model to obtain audio features, fuses the visual features and audio features to obtain corresponding fused features, obtains a preset target multimodal deep fake detection model, performs forgery identification on the fused features based on the target multimodal deep fake detection model, and determines the identification result of the target source video. The multimodal deep fake detection method of the present application extracts features from the video frame sequence and audio sequence through multimodal feature extraction, i.e., extracts visual features and audio features through a visual feature extraction model and an audio spectrogram coding model, and after the two modal features are fused, performs forgery identification based on a preset target multimodal deep fake detection model to obtain a corresponding identification result. The present application pays special attention to the comprehensive analysis of the two modalities of vision and audio, which can make full use of the information of other modalities and improve the detection accuracy of carefully crafted multimodal deep fake content.

[0105] Further, such as Figure 3 As shown, the multimodal deep fake detection method of the embodiment of the present application may also include the following steps:

[0106] S301, construct corresponding positive and negative sample pairs based on visual features and audio features.

[0107] It should be noted that during the training phase, cross-modal contrastive learning is required. The contrastive learning loss is used to encourage the feature extractor to extract features that are sensitive to forgery while aligning the features of the two modalities of video and audio. For example, Figure 2 Obviously, the cross-modal contrastive learning training phase and deep fake identification are two independent and parallel phases, that is, this application not only proposes a multimodal deep fake detection method, but also proposes a cross-modal contrastive learning training strategy.

[0108] In the embodiment of the present application, positive and negative sample pairs are constructed based on the visual features and audio features obtained in the above embodiment for subsequent processing.

[0109] As a possible implementation, positive and negative sample pairs are obtained based on the real source video and the target source video. The positive and negative sample pairs include positive sample pairs and negative sample pairs; the positive sample pairs include visual features and audio features of the same real source video; the negative sample pairs include visual features and audio features of the same forged source video, or visual features and audio features of different target source videos. For example, the visual features and audio features from the same non-forged video are regarded as a positive sample pair, and the visual features and audio features from the same forged video are regarded as a negative sample pair. The visual features and audio features from different videos are regarded as a negative sample pair.

[0110] S302: Perform cross-modal contrast learning training on the multimodal deep fake detection model based on the positive and negative sample pairs to obtain a trained multimodal deep fake detection model.

[0111] In the embodiment of the present application, the multimodal deep fake detection model in the above embodiment is trained by cross-modal contrast learning based on the positive and negative sample pairs obtained in step S301 to obtain a trained multimodal deep fake detection model. Figure 2 shown.

[0112] Optionally, based on the positive and negative sample pairs, a preset contrast loss function is used to calculate the contrast loss of the target source video to perform cross-modal contrast learning training on the multimodal deep fake detection model. The contrast loss function is the InfoNCE loss function (a contrast loss function commonly used in self-supervised learning tasks); the parameters of the contrast loss function include: temperature hyperparameter, the number of positive and negative sample pairs, and the similarity function; the InfoNCE loss function represents the cross-modal contrast learning loss function of visual features and audio features.

[0113] For example, the contrastive loss is calculated using the InfoNCE loss function: in represents the features of the p-mode from the ith sample (video). τ is the temperature hyperparameter. K represents the number of sample pairs; σ is the similarity function, which is calculated using cosine similarity.

[0114] It should be noted that by calculating the contrast loss and comparing the differences between the two modal features from different source videos, the feature learning process can be made sensitive to forgery, the mining of forged features can be enhanced, and the forgery category classification can be facilitated. At the same time, the feature spaces of vision and audio can be aligned to improve the convergence speed of model training. In addition, cross-modal contrastive learning training can effectively improve the accuracy of forgery detection and the training efficiency of the model by strengthening the differences between features, improving the sensitivity to forged content and the alignment of feature spaces.

[0115] Therefore, by performing cross-modal comparative learning training on the multimodal deep fake detection model, a more accurate multimodal deep fake detection model is obtained, and the accuracy of forgery identification based on the trained multimodal deep fake detection model is also higher.

[0116] In general, for example, Figure 2 As shown, the present application can make full use of multimodal information to more accurately detect deep fakes, including three main components: a multimodal feature extraction module, a feature information fusion module, and a fake identification module. It also adopts a training strategy of random normalization and cross-modal contrast learning to enhance the generalization ability and detection accuracy of the target multimodal deep fake detection model. In the feature extraction stage, the image-level feature extraction submodel and the temporal feature extraction submodel in the visual feature extraction model, namely the ResNet model and the temporal Transformer model, are used to perform feature processing on the video frame sequence. The pre-trained audio spectrogram encoding model, namely the Transformer model, is used to perform feature processing on the audio sequence, and the feature fusion of the two modalities is achieved through average pooling, self-attention layer, and cross-attention layer. At the same time, the cross entropy loss function and InfoNCE loss function used in the training target multimodal deep fake detection model are used to guide the learning of the feature extractor.

[0117] Specifically, in the target multimodal deep fake detection model training stage, in order to improve the generalization ability of the model detection and maintain the learning ability of the original domain data, the present application uses a random normalization module to simulate the data input from different domains for the input visual data, i.e., the video frame sequence, according to a preset certain probability. The multimodal feature extraction module includes a video encoding part and an audio encoding part. The video encoding part is implemented by the ResNet model. For the input video frame sequence, 4 groups are sampled from the sequence with equal steps, and each group contains 4 consecutive frames. For the sampled 16 frames, a ResNet is used to extract the corresponding image-level features, and then the features of each group are obtained through a temporal Transformer model to obtain a group-level feature. Finally, the visual features are obtained by averaging the 4 group-level features. The audio encoding part uses an audio spectrum Transformer to extract the high-level features of the audio, and then these features are used as the input of the feature information fusion module. In the feature information fusion module, the visual features are first processed by the self-attention layer, and then the features are fused with the audio features through the cross-attention layer. The final fusion features are input into the forgery identification module for category judgment. During the training phase, this application uses two loss functions to constrain the update of the model. The first loss function is the cross-entropy loss function of the classification results, and the second loss function is the cross-modal comparative learning loss function of the visual features and the audio features. Specifically, the target multimodal deep fake detection model makes the feature learning process sensitive to forgery by comparing the differences between the two modal features from different source videos, and aligns the feature space of vision and audio. This not only increases the model's discriminative ability, but also allows the model to achieve more accurate detection in real-world multimodal deep fake scenarios.

[0118] It should be noted that the multimodal deep fake detection method proposed in this application is a deep fake detection method based on cross-modal contrastive learning, that is, a deep fake detection and training method based on cross-modal contrastive learning.

[0119] Figure 4 is a flow chart of a multimodal deep fake detection device according to an embodiment of the present application, such as Figure 4 As shown, the multimodal deep fake detection device 400 of the embodiment of the present application may specifically include:

[0120] The first acquisition module 401 is used to acquire a video frame sequence and an audio sequence of a target source video; wherein the target source video includes a real source video and a forged source video.

[0121] The extraction module 402 is used to extract features from the video frame sequence based on a preset visual feature extraction model to obtain visual features, and to extract features from the audio sequence based on a pre-trained audio spectrogram coding model to obtain audio features.

[0122] The fusion module 403 is used to fuse the visual features and the audio features to obtain corresponding fusion features.

[0123] The identification module 404 is used to obtain a preset target multimodal deep fake detection model, perform forgery identification on the fused features based on the target multimodal deep fake detection model, and determine the identification result of the target source video; wherein the identification result indicates whether the target source video corresponding to the fused features is forged, and the forgery method of the target source video when it is determined to be forged; the forgery method includes at least lip editing, lip generation, and face reproduction.

[0124] In a possible implementation, the multimodal deep fake detection device further includes:

[0125] A construction module is used to construct corresponding positive and negative sample pairs based on visual features and audio features;

[0126] The training module is used to perform cross-modal contrast learning training on the multimodal deep fake detection model based on positive and negative sample pairs to obtain a trained multimodal deep fake detection model.

[0127] In a possible implementation manner, the first acquisition module is specifically configured to:

[0128] Obtain a multimodal dataset including a plurality of deep fake contents; wherein each deep fake content includes a video frame sequence and an audio sequence of a corresponding deep fake video;

[0129] From the multimodal dataset, the video frame sequence and audio sequence of the target source video are randomly extracted at equal intervals according to the preset intervals.

[0130] In a possible implementation, the multimodal deep fake detection device further includes:

[0131] A determination module, used for determining a corresponding random standardized training strategy based on a preset normal distribution hyperparameter, a mean value and a standard deviation of a target image data set before extracting features from a video frame sequence based on a preset visual feature extraction model to obtain visual features;

[0132] The standard module is used to randomly standardize the video frame sequence based on a preset probability, obtain the mean and standard deviation of the random sampling, standardize the video frame sequence based on the mean and standard deviation of the random sampling, and standardize it with the mean and standard deviation of the target image data set.

[0133] In a possible implementation, the extraction module is specifically used to:

[0134] For a video frame sequence, sampling a first number of video frame sequence groups from the video frame sequence in equal steps based on a preset sampling step; wherein each video frame sequence group includes a second number of continuous video frame sequences;

[0135] Based on all the sampled video frames, the corresponding image-level features are extracted using the image-level feature extraction sub-model; wherein each video frame sequence group corresponds to four image-level features;

[0136] For each image-level feature corresponding to a video frame sequence group, the corresponding group-level feature is obtained based on the temporal feature extraction sub-model, and all group-level features are average-pooled to obtain the corresponding visual feature.

[0137] In a possible implementation, the extraction module is specifically used to:

[0138] For the audio sequence, convert the audio sequence into a corresponding spectrogram;

[0139] The spectrogram is input into the audio spectrogram encoding model to extract the audio features of the audio sequence.

[0140] In a possible implementation, the building module is specifically used to:

[0141] Acquire positive and negative sample pairs based on the real source video and the target source video; wherein the positive and negative sample pairs include a positive sample pair and a negative sample pair; the positive sample pair includes visual features and audio features of the same real source video; the negative sample pair includes visual features and audio features of the same forged source video, or visual features and audio features of different target source videos;

[0142] The construction module is specifically used to: calculate the contrast loss of the target source video based on the positive and negative sample pairs using a preset contrast loss function to perform cross-modal contrast learning training on the multimodal deep fake detection model.

[0143] The multimodal deep fake detection device provided in the embodiment of the present application obtains the video frame sequence and audio sequence of the target source video, extracts features from the video frame sequence based on a preset visual feature extraction model to obtain visual features, extracts features from the audio sequence based on a pre-trained audio spectrogram coding model to obtain audio features, fuses the visual features and audio features to obtain corresponding fused features, obtains a preset target multimodal deep fake detection model, performs forgery identification on the fused features based on the target multimodal deep fake detection model, and determines the identification result of the target source video. The multimodal deep fake detection device of the present application extracts features from the video frame sequence and the audio sequence through multimodal feature extraction, i.e., extracts visual features and audio features through a visual feature extraction model and an audio spectrogram coding model, and after the two modal features are fused, performs forgery identification based on a preset target multimodal deep fake detection model to obtain a corresponding identification result. The present application pays special attention to the comprehensive analysis of the two modalities of vision and audio, which can make full use of the information of other modalities and improve the detection accuracy of carefully crafted multimodal deep fake content.

[0144] like Figure 5 As shown, an electronic device 500 provided in an embodiment of the present application includes: a processor 501, a memory 502 and a bus, wherein the memory 502 stores machine-readable instructions executable by the processor 501. When the electronic device is running, the processor 501 communicates with the memory 502 through the bus, and the processor 501 executes the machine-readable instructions to perform the steps of the above-mentioned multimodal deep fake detection method.

[0145] Specifically, the above-mentioned memory 502 and processor 501 can be general-purpose memory and processor, which are not specifically limited here. When the processor 501 runs the computer program stored in the memory 502, the above-mentioned multimodal deep fake detection method can be executed.

[0146] Corresponding to the above-mentioned multimodal deep fake detection method, an embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned multimodal deep fake detection method are executed.

[0147] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the method embodiment, and will not be repeated in this application. In the several embodiments provided in this application, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0148] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0149] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0150] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the deployment method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks, or optical disks.

[0151] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A multimodal deep fake detection method, characterized in that: The method comprises: Acquire a video frame sequence and an audio sequence of a target source video; wherein the target source video includes a real source video and a forged source video; Extracting features from the video frame sequence based on a preset visual feature extraction model to obtain visual features, and extracting features from the audio sequence based on a pre-trained audio spectrogram coding model to obtain audio features; fusing the visual feature and the audio feature to obtain a corresponding fused feature; A preset target multimodal deep fake detection model is obtained, and the fusion feature is subjected to forgery identification based on the target multimodal deep fake detection model to determine an identification result of the target source video; wherein the identification result indicates whether the target source video corresponding to the fusion feature is forged, and a forgery method of the target source video when it is determined to be forged; the forgery method includes at least lip editing, lip generation, and face reproduction.

2. The method according to claim 1, characterized in that The method further comprises: Constructing corresponding positive and negative sample pairs based on the visual features and the audio features; The multimodal deep fake detection model is trained by cross-modal contrast learning based on the positive and negative sample pairs to obtain the trained multimodal deep fake detection model.

3. The method according to claim 1, characterized in that: The step of obtaining a video frame sequence and an audio sequence of a target source video includes: Obtain a multimodal dataset including a plurality of deep fake contents; wherein each deep fake content includes a video frame sequence and an audio sequence of a corresponding deep fake video; The video frame sequence and the audio sequence of the target source video are extracted from the multimodal data set at equal intervals according to preset intervals.

4. The method according to claim 1, characterized in that Before extracting features from the video frame sequence based on a preset visual feature extraction model to obtain visual features, the method further includes: The corresponding random standardized training strategy is determined based on the pre-set normal distribution hyperparameters, the mean and standard deviation of the target image dataset; The video frame sequence is randomly standardized based on a preset probability to obtain a mean and a standard deviation of random sampling, the video frame sequence is standardized based on the mean and the standard deviation of the random sampling, and the mean and the standard deviation of the target image data set are used for standardization.

5. The method according to claim 1, characterized in that The visual feature extraction model includes an image-level feature extraction sub-model and a temporal feature extraction sub-model; The extracting features of the video frame sequence based on a preset visual feature extraction model to obtain visual features includes: For the video frame sequence, sampling a first number of video frame sequence groups from the video frame sequence in equal steps based on a preset sampling step; wherein each video frame sequence group includes a second number of continuous video frame sequences; Based on all the sampled video frames, the image-level feature extraction sub-model is used to extract corresponding image-level features; wherein each video frame sequence group corresponds to four image-level features; For each image-level feature corresponding to each video frame sequence group, the corresponding group-level feature is obtained based on the temporal feature extraction sub-model, and all the group-level features are average-pooled to obtain the corresponding visual feature.

6. The method according to claim 1, characterized in that The extracting features of the audio sequence based on the pre-trained audio spectrogram coding model to obtain audio features includes: For the audio sequence, convert the audio sequence into a corresponding spectrogram; The spectrogram is input into the audio spectrogram encoding model to extract the audio features of the audio sequence.

7. The method according to claim 2, characterized in that The constructing corresponding positive and negative sample pairs based on the visual features and the audio features includes: Acquire positive and negative sample pairs based on the real source video and the forged source video; wherein the positive and negative sample pairs include a positive sample pair and a negative sample pair; the positive sample pair includes visual features and audio features of the same real source video; the negative sample pair includes visual features and audio features of the same forged source video, or visual features and audio features of different target source videos; The performing cross-modal contrastive learning training on the multimodal deep fake detection model based on the positive and negative sample pairs includes: Based on the positive and negative sample pairs, a preset contrast loss function is used to calculate the contrast loss of the target source video to perform cross-modal contrast learning training on the multimodal deep fake detection model.

8. A multimodal deep fake detection device, characterized in that: The device comprises: A first acquisition module is used to acquire a video frame sequence and an audio sequence of a target source video; wherein the target source video includes a real source video and a forged source video; An extraction module, configured to extract features from the video frame sequence based on a preset visual feature extraction model to obtain visual features, and to extract features from the audio sequence based on a pre-trained audio spectrogram coding model to obtain audio features; A fusion module, used for fusing the visual features and the audio features to obtain corresponding fusion features; An identification module is used to obtain a preset target multimodal deep fake detection model, perform forgery identification on the fused features based on the target multimodal deep fake detection model, and determine an identification result of the target source video; wherein the identification result indicates whether the target source video corresponding to the fused features is forged, and the forgery method of the target source video when it is determined to be forged; the forgery method at least includes lip editing, lip generation, and face reproduction.

9. An electronic device, characterized in that: include: A processor, a memory and a bus, the memory storing machine-readable instructions executable by the processor, when the electronic device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the steps of the multimodal deep fake detection method as described in any one of claims 1 to 7 are performed.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the multimodal deep fake detection method as described in any one of claims 1 to 7.

Images