A method and device for fine-grained detection of deep fake audio and video based on multimodal feature fusion

Abstract

A method and device for fine-grained detection of deepfake audio and video based on multimodal feature fusion includes a neural network model based on multimodal feature fusion supplemented by an attention mechanism. This model integrates and analyzes both short-range feature similarity information and long-range feature difference information from both single-modal and cross-modal dimensions, outputting fine-grained detection results. The method includes: first, obtaining a dataset of speaker-forged audio and video and performing data preprocessing for model training; then, constructing the proposed neural network model and training it to a good fit; then, subjecting the audio and video to be detected to the same preprocessing operations and then inputting them into the trained neural network model for processing. Finally, the model outputs fine-grained detection results. This invention addresses the current lack of accurate and fine-grained detection methods for speaker-forged audio and video.

Description

Technical Field

[0001] The present invention belongs to the field of multimedia forensics technology, and specifically relates to a method and device for fine-grained detection of deep fake audio and video based on multimodal feature fusion. Background Art

[0002] Social media platforms are now essential for acquiring information in daily life, with audio and video being among the most important and crucial carriers of information. Current artificial intelligence technologies enable people to generate and edit audio and video with greater freedom and convenience. Deepfakes, a representative form of audio and video content editing technology, have evolved from single-modality to multimodality with the advancement of generative models and the widespread adoption of multimodal fusion techniques. In other words, in such deepfakes, at least one of the two modalities, audio or video, is manipulated. Perhaps an audio or video is manipulated using voice deepfake technology to render the speech of speaker A as that of speaker B, while lip-syncing techniques such as wave2lip are used to alter the original speech of speaker B to match the speech. Such manipulations are both realistic and difficult to detect, while also undermining the credibility of both audio and video modalities. Furthermore, deepfakes using face-replacement techniques such as FSGAN and FaceSwap can be mixed with the video modality. If this falsification technology is abused illegally to distort news facts and create false information, and these falsified products are widely disseminated on social networks, it will cause immeasurable harm to society.

[0003] To address this situation, deepfake detection methods need to possess multimodal analysis and fine-grained detection capabilities. For example, the "Facial Forgery Detection Method Based on Multimodal Collaborative Learning" disclosed in CN202410555732.0 uses interactive collaborative learning of features in the spatial and frequency domains, and introduces a multi-scale feature enhancement module in the spatial domain to enhance the texture and semantic information of spatial domain features. Finally, an adaptive fusion module is used to fuse the spatial and frequency domain features to obtain feature representations. This method has a good detection effect on facial forgeries, but it can still only detect forgeries in a single visual modality. For example, the "Deepfake Detection Method for Specific Persons Based on Audio and Video Multimodality" disclosed in CN202211185641.X extracts features from multiple modalities such as lips, face, and voice, and fuses these features to detect audio and video forgeries. However, this method does not deeply explore the potential correlation between the two modalities of audio and video in feature fusion, and the feature information mining in a single modality is not sufficient. Therefore, it cannot achieve high-precision detection of multimodal forgeries of speakers' audio and video, nor can it meet the needs of fine-grained detection. Another example is the "Deep fake detection method based on audio-visual joint learning" disclosed in CN202311815982.5. This method selects video clips with the most significant motion content in the input video, and uses a dual-branch network to process the information of these video clips for deep fake detection. This method has good detection efficiency, but it still cannot achieve fine-grained detection of multimodal audio and video.

[0004] In light of the shortcomings of existing methods, this paper proposes a fine-grained method and device for deepfake audio and video detection based on multimodal feature fusion. This method, through multimodal feature fusion and combined with an attention mechanism, effectively integrates multivariate information from audio and video, models information interactions within modalities, and then achieves fine-grained detection of forged audio and video by training and utilizing a specially designed neural network. Summary of the Invention

[0005] To address the current lack of high-precision, fine-grained detection methods for multimodal deepfakes in speaker audio and video, this paper proposes a method and apparatus for fine-grained detection of multimodal deepfakes. Based on multimodal feature fusion and an attention mechanism, this paper achieves high-precision, fine-grained classification of multimodal deepfakes, including those described in the background art.

[0006] The present invention proposes a fine-grained method for detecting deep fake audio and video based on multimodal feature fusion, which includes the following steps:

[0007] S1 Speaker forgery audio and video training dataset preprocessing: Separate the audio and video tracks used for training and align the time dimension. Read the audio and video tracks as tensors and split them into multiple segments. The final input data is

[0008] S2 builds and trains a multimodal authentication model: A builds a feature extraction module and constructs a visual feature encoder E. v and audio feature encoder E a , feature extraction is performed on the visual input and audio input of audio and video. The process can be summarized as follows: B. Constructing a feature fusion module to perform feature fusion. Three attention-based feature fusion blocks designed by this invention are stacked to integrate the feature similarity information of close distances and the feature difference information of long distances, respectively. The features are fused and enhanced from both intra-modal and cross-modal perspectives. C. Constructing a global feature pooling module. Combining the attention mechanism, the features of each segment of the audio and video modalities are aggregated to generate a global feature representation for each modality. D builds a classification module, which combines the global representation of each modality and the hybrid features formed by splicing The classification results are output after processing by the FC layer E model training uses a multi-task learning strategy, compares the classification results with the labels to calculate the model loss, and backpropagates the model loss to train the model.

[0009] S3: Preprocessing of audio and video of the speaker to be detected: The processing method is the same as S1.

[0010] S4 uses the trained multimodal authentication model to analyze and process the data to be detected to obtain fine-grained detection results of the audio and video to be detected.

[0011] The second aspect of the present invention relates to a deep fake audio and video fine-grained detection device based on multimodal feature fusion, including a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the deep fake audio and video fine-grained detection method based on multimodal feature fusion proposed by the present invention.

[0012] The beneficial effects of the present invention are mainly manifested in:

[0013] 1. This paper, embracing the trend toward multimodal and realistic forgery technology, proposes a fine-grained method for detecting deepfakes in audio and video based on multimodal feature fusion. This method is based on a multimodal fine-grained deepfake detection model that analyzes and processes features from two dimensions: within a single modality and across modalities. This model enables high-precision, fine-grained detection of multimodal deepfakes in speaker audio and video.

[0014] 2. The present invention proposes a new feature fusion module that can effectively mine single-modal and cross-modal information of audio and video. For the intra-modal feature fusion part, the present invention proposes an attention-based intra-modal feature fusion mechanism based on the lack of long-term memory in current common counterfeiting methods, which integrates the feature similarity information of close distances and the feature difference information of long distances, thereby modeling the information interaction within the modality in a more effective way, further enriching the feature content, and effectively improving the detection accuracy of the model. For the cross-modal feature fusion part, the present invention adopts a relatively simple fusion strategy while ensuring the accuracy of the model. The structure is lightweight and has strong adaptability and effectiveness in the model of this method. The cross-stacking of the two parts makes the fusion of features more complete and effective.

[0015] 3. For the task of multimodal forgery fine-grained detection, every audio and video clip is an object worthy of attention. The present invention uses a global pooling module that can effectively extract key information, improve feature expression capabilities, reduce redundant information, and integrate the features of each clip into video-level features, providing more refined and effective input for the classification module.

[0016] 4. This invention uses multi-task learning to train the model, which can effectively balance single-modal and multi-modal feature learning, enhance intra-domain and cross-domain detection capabilities, and improve the robustness and generalization capabilities of forgery detection. This multi-level loss constraint mechanism helps to more comprehensively capture forgery clues and improve the accuracy and reliability of detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a flow chart of the method proposed by the present invention;

[0018] Figure 2 Schematic diagram of the intra-modal feature fusion part of the method proposed in the present invention;

[0019] Figure 3 It is a structural diagram of the neural network model in the method proposed by the present invention;

[0020] Figure 4 It is a schematic diagram of the device of the present invention. DETAILED DESCRIPTION

[0021] The technical solution of the present invention is further described below with reference to the accompanying drawings.

[0022] Example 1

[0023] Reference Figures 1 to 3 , a deep fake audio and video fine-grained detection method based on multimodal feature fusion, includes the following steps:

[0024] Preprocessing of S1 speaker forged audio and video training dataset.

[0025] The speaker audio and video in the multimodal deep fake training dataset are preprocessed. First, the video track and the audio track are separated. In order to fully preserve the spatiotemporal characteristics of the audio, the original audio is selected as the input. The soundfile tool is used to read the audio file at a sampling rate of 16kHz and in mono, and it is converted to a tensor type for subsequent processing. Then, according to the number of frames of the video (denoted as T v ), calculate the difference between the audio tensor time dimension stack and the video frame number, if the audio feature is shorter than the video, add some zero padding, on the contrary, if the audio feature is longer, discard the corresponding content. Then cut it evenly into S segments, and finally get the audio tensor Where fps is the frame rate of the video stream, and fre is the sampling rate when reading audio. For video, save the video stream in the form of video frames, use the PIL library to read each video frame in the form of a tensor, and convert it to a fixed size. Then splice each image tensor in the time dimension. The size of a video tensor is T v ×H×W×C. H, W, C are the height, width, and number of channels of each frame. The tensor is evenly divided into S segments along the time dimension. Each segment extracts the tensor corresponding to n consecutive frames. If the number of frames is insufficient, the first segment is filled with n frames, and the subsequent segments reuse the consecutive frames of the previous segment to make up for it. The final video tensor is obtained.

[0026] In summary, the final input data is Where N is the total number of samples in the training data, m is the modality tag, v represents the visual modality, a represents the audio modality, is the label under a certain mode, Among them, 0 means that the current sample is real in the current mode, and 1 means that the current sample is forged in the current mode. get They correspond to four types of forgery situations in each modality.

[0027] S2 builds and trains a multimodal authentication model. The model structure is as follows Figure 2 As shown in the figure, the model includes feature extraction module, feature fusion module, global feature pooling module and classification module. The processing process of each module of the neural network is introduced below.

[0028] A feature extraction module consists of two parts: visual feature extraction and audio feature extraction, which respectively extract features from the visual input and audio input after preprocessing.

[0029] A1 Visual Feature Extraction: This example uses two 2D convolutional layers and three stacked spatiotemporal feature extraction blocks from the STIL (Spatial-Temporal Inconsistency Learning) model as the visual feature encoder E v , which can extract high-level spatiotemporal features that are helpful for forgery detection in the visual modality. It extracts features from each visual input and maps them into a d-dimensional space to obtain a visual feature set for each audio and video. This process can be expressed as:

[0030]

[0031] in Equivalently,

[0032] A2 Audio Feature Extraction: This example uses an encoder similar to RawGAT-ST (Raw GAT with spectro-temporalattention) as the audio feature encoder E a , which consists of 1 Sinc layer and 6 2D residual blocks, is used to extract the features of each audio segment and map it to the space of the same dimension d to obtain the audio feature set of each audio and video. This process can be expressed as:

[0033]

[0034] in Equivalently,

[0035] B. Construct a feature fusion module for feature fusion: The feature fusion module is composed of three stacked feature fusion blocks, and the feature fusion block is composed of an intra-modal feature fusion block and a cross-modal feature fusion block. The processing process of a feature fusion block can be summarized as follows:

[0036]

[0037] Where r,s∈{v,a}; r≠s, the same below; is the feature of the t time segment under a certain mode output by the feature fusion block of the previous layer, assuming its dimension is d', The updated features of the current layer.

[0038] B1 Intra-modal feature fusion block: This part focuses on the feature fusion within a single modality, which is used to capture the intrinsic connection between each segment under the current modality to enhance the representation ability of each segment feature. Specifically, taking the t1 time segment as the current time segment and the first layer feature fusion block as an example, the present invention selects the features of the t1 current time segment and the P time segments before and after to enhance the features of the current time segment. If the number of the previous and next segments is insufficient, the cyclic shift method is used to supplement it, such as Figure 1 For the convenience of introduction, the total set of selected segments is represented by Q, the set of non-displaced segments is represented by Q1, and the set of displacement supplemented segments is represented by Q2. First, a relative time weight is assigned to each segment according to the time interval, as shown in formula (4):

[0039]

[0040] in is the time weight between segments t1 and t2, 1≤t1≤S, t2∈Q. After obtaining the time weight, the present invention generates the query vector and key vector of each segment respectively through two affine transformations, as shown in formula (5):

[0041]

[0042] where q t ∈R d / 2 、k t ∈R d / 2 are the query vector and key vector of each selected segment feature. Next, the attention weights between the current segment and each selected segment are calculated, as shown in formulas (6) and (7):

[0043]

[0044] in is the intrinsic correlation between segments under m mode, is the attention weight between segments. It is worth noting that: for adjacent segments, the present invention uses the content similarity of the query vector and the key vector to represent the intrinsic correlation between the two segments. For distant segments, the content difference between the query vector and the key vector is calculated to represent their intrinsic correlation. This will be used to integrate the distant difference information to enrich the feature content to help the model judge forgery. For this reason, the current segment should pay attention to different content for the features of distant and nearby segments. Therefore, when generating the value representation vector, the present invention uses relatively independent radial transformations for the segment features in different sets of Q1 and Q2, as shown in formula (8):

[0045]

[0046] where v t ∈R d / 2 is the value representation vector of the feature of the time segment t, and finally the feature of the current segment is enhanced according to formula (9).

[0047]

[0048] Among them, β1 and β2 are learnable parameters.

[0049] B2 Cross-modal feature fusion block: This part has a simple structure and mainly consists of two linear layers. Through experiments, we found that the following simple cross-modal fusion strategy combined with the intra-modal feature fusion part also has good results.

[0050]

[0051] Among them, Linear is the linear layer, is the cross-modal feature after linear transformation, and β3 is a learnable parameter.

[0052] In this example, three feature fusion blocks are stacked, and q in the first two feature fusion blocks is t 、k t 、v t The vector dimensions are half of the current layer input segment features, and in the third feature fusion block q t 、k t 、v t The vector dimension is the same as the input features.

[0053] C builds a global feature pooling module: This module aims to aggregate the features of each segment of the audio and video modalities through the attention mechanism to generate a global feature representation of each modality for use by the subsequent classification module.

[0054]

[0055] in: is the feature of a certain time segment under a certain input mode; h gate It is a neural network that generates attention weights. The present invention uses a linear layer to calculate the attention weights of each segment. The attention weights of each segment are normalized by softmax processing and used as the final weights of each segment feature. θ It is a feature transformation network. The present invention uses a linear layer to perform a learnable linear transformation on the features of each input segment. Finally, the weighted summation of the transformed features is performed according to the obtained weights to obtain the global feature representation under the current mode.

[0056] D. Constructing a classification module: After obtaining the global representation of each modality, classify and predict the video modality, audio modality, and mixed modality. For the video modality and audio modality, construct a FC (Fully Connected Layer) layer to process the single modality and output the classification results. and For mixed modalities, the present invention combines the features of the video and audio modalities and then outputs the video-level classification result γ through an FC layer. fusion The process can be expressed as:

[0057]

[0058] Where [·] represents the concatenation operation of tensors, The final fine-grained detection result.

[0059] E Model training: This paper adopts a multi-task learning strategy when training the model, uses multi-level loss constraints, and uses three cross-entropy loss functions to guide the training of the model, as shown below.

[0060]

[0061] Where H represents the cross entropy loss calculation formula.

[0062] S3: Preprocess the audio and video of the speaker to be detected. Similar to step S1, extract the audio and video streams for the audio and video to be detected and align their lengths in the time dimension. Use soundfile to read the audio stream as a tensor and split it into multiple segments. Save the video stream as video frames. Use the PIL library to read each video frame as a tensor and convert it to a fixed size. Then, after concatenating each image tensor in the time dimension, split it into multiple segments in the same way as described in S1.

[0063] S4 uses the trained multimodal authentication model to analyze and process the data to be detected to obtain fine-grained detection results of the audio and video to be detected.

[0064] Example 2

[0065] Reference Figure 4 This embodiment provides a deep fake audio and video fine-grained detection device based on multimodal feature fusion, including a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the deep fake audio and video fine-grained detection method based on multimodal feature fusion of Example 1.

[0066] The above description is only a description of the preferred embodiment of the present invention and does not limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solution of the present invention by ordinary technicians in this field should fall within the scope of protection determined by the claims of the present invention.

Claims

1. A fine-grained method for detecting deepfake audio and video based on multimodal feature fusion, comprising the following steps: S1 speaker forgery audio and video training dataset preprocessing: the audio and video used for training are separated and the time dimension is aligned; the audio and video tracks are read as tensors and divided into multiple segments to obtain the input data ;in, is the total number of samples of training data, is the modal tag, Indicates the visual modality, Indicates the audio mode, is a tensor under a certain mode, is the label under a certain mode, {0,1}, 0 means the current sample is authentic in the current modality, and 1 means the current sample is forged in the current modality; S2 builds and trains a multimodal authentication model: A builds a feature extraction module and constructs a visual feature encoder. and audio feature encoder , feature extraction is performed on the visual input and audio input of audio and video. The process can be summarized as follows: ,in, is the video tensor, The audio tensor; B constructs a feature fusion module to perform feature fusion, stacking 3 attention-based feature fusion blocks. A feature fusion block consists of two parts: the intra-modal feature fusion part and the cross-modal feature fusion part; the intra-modal feature fusion part focuses on the feature fusion within a single modality, which is used to capture different information from different segments in the current modality, and integrates the feature similarity information of close distances and the feature difference information of long distances to enhance the representation ability of each segment feature; the cross-modal feature fusion part uses a simple and lightweight linear layer to fuse multimodal features, and the intra-modal feature fusion part is used in combination with it to further mine forged information; C constructs a global feature pooling module, combines the attention mechanism to aggregate the features of each segment of the audio modality and the video modality, and generates a global feature representation for each modality ; D builds a classification module, which combines the global representation of each modality and the hybrid features formed by splicing , and output classification results after FC layer processing 、 、 ; E model training uses a multi-task learning strategy, compares the classification results with the labels to calculate the model loss, and backpropagates the model loss to train the model; S3 audio and video preprocessing of the speaker to be detected: the processing method is the same as S1; S4 uses the trained multimodal authentication model to analyze and process the data to be detected to obtain fine-grained detection results of the audio and video to be detected.

2. The method for fine-grained detection of deep fake audio and video based on multimodal feature fusion according to claim 1 is characterized by: In S1, the audio and video input to the neural network are separated into audio and video tracks and aligned in time dimension; the audio and video tracks are read into the form of tensors and divided into multiple segments.

3. The method for fine-grained detection of deep fake audio and video based on multimodal feature fusion according to claim 1 is characterized by: In S2, the model uses a visual feature encoder consisting of 2 two-dimensional convolutional layers and 3 stacked STIL blocks to extract features from video clips, and uses an audio feature encoder consisting of 1 Sinc layer and 6 2D residual blocks to extract features from audio clips.

4. The method for fine-grained detection of deep fake audio and video based on multimodal feature fusion according to claim 1 is characterized by: In S2, a global feature pooling module is introduced, which focuses on the connection between each segment, calculates the weight of each segment, and integrates the features of each segment into video-level features.

5. The method for fine-grained detection of deep fake audio and video based on multimodal feature fusion according to claim 1 is characterized by: In S2, a multi-task learning strategy is used to guide the training of the model and balance the feature learning of single modality and multimodality.

6. A fine-grained detection device for deep fake audio and video based on multimodal feature fusion, characterized by: It includes a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, it is used to implement the deep fake audio and video fine-grained detection method based on multimodal feature fusion according to any one of claims 1 to 5.

Images