A multimodal deepfake detection model for temporal forgery localization

Abstract

The present invention relates to fields such as network security technology, and discloses a multimodal deep fake detection model for temporal fake positioning, which aims to solve the problem of insufficient representation of fine-scale fake features in existing detection methods and the problem that existing methods cannot effectively capture image fake artifacts; it includes a feature extraction module composed of a visual feature extraction module and an audio feature extraction module; an adaptive multimodal feature fusion module that fuses visual feature representation and audio feature representation in the feature channel dimension by constructing an attention map; a frame classification module that processes the visual feature representation and audio feature representation to obtain visual frame-level prediction labels and audio frame-level prediction labels; and a boundary positioning module that obtains the final fake boundary prediction by processing the visual feature representation, audio feature representation, multimodal fusion features, visual frame-level prediction labels and audio frame-level prediction labels.

Description

Technical Field

[0001] The present invention relates to the fields of network content security, cyberspace security technology, and more specifically, to a multimodal deep fake detection model for temporal fake positioning. Background Art

[0002] As an efficient information carrier, digital video plays a crucial role in information dissemination in modern society. To better demonstrate and utilize digital video, video editing technologies are rapidly developing. Currently, the most prominent example is deepfake technology. Since its inception, deepfakes have rapidly spread online, with continuous updates and iterations, thanks to their ability to generate high-quality videos in a short period of time with minimal effort and low barriers to entry.

[0003] Traditional deepfake detection aims to determine the authenticity of videos by detecting artifacts in the spatial and frequency domains of images, as well as unnatural biometric features (such as lip movements inconsistent with the audio and unusual blinking). Numerous research methods have demonstrated strong capabilities in determining video authenticity. However, these traditional detection methods assume a binary classification task and focus solely on identifying whether a complete video has been tampered with. This limitation severely impacts their effectiveness in complex scenarios. Current deepfake technology has the ability to precisely manipulate key semantics. For example, in a public speech by a tech company founder, AI synthesis technology replaced the word "safety" with "hazards." This millisecond-level speech-to-text synchronization forgery not only retains the rhythmic characteristics of the original sound but also creates highly misleading false evidence through semantic reversal. This requires detection methods to not only verify the authenticity of a video but also locate the start and end points of the forged segments within the video. Currently, research on temporal forgery localization is still in its infancy. Such methods combine traditional deepfake binary classification detection techniques with localization techniques such as Proposal Relation Blocks (PRBs) and boundary matching losses. While current methods have made some progress, the following issues remain:

[0004] 1. Existing methods tend to analyze overall image semantic features while ignoring subtle forgery artifacts. The forged clips in temporal forgery localization tasks are very short and utilize deep forgery techniques like Expression Swap. The forged areas typically focus on organs like the mouth or eyes, rather than the entire face. In this case, micro-scale features such as edges, textures, and corners are more effective at detecting forgery artifacts than the overall semantics of the image. Furthermore, existing methods, due to their use of continuous convolution, can degrade or even eliminate subtle cross-layer features, leading to insufficient representation of forgery features. This, in turn, affects the judgment of forged frames, causing deviations or even failure in temporal forgery localization.

[0005] 2. Existing methods capture image tampering artifacts only through the RGB color channels, ignoring artifacts in high-frequency regions. Although deepfakes disrupt the correlation and natural optical laws of the RGB color channels, with the continuous advancement of deepfake technology, tampering artifacts in the RGB color channels are becoming increasingly natural. In fact, existing deepfake technology still leaves obvious traces of tampering in high-frequency regions. Extracting these high-frequency features can reveal the statistical differences between forged and authentic images. However, existing methods do not take this into account.

[0006] 3. Existing methods all involve multimodality, which requires the fusion of visual and auditory modal features. In reality, visual and auditory perceptions have different importance in reflecting the fact of forgery at the same moment. However, existing methods cannot dynamically adjust the importance of each modal feature based on the specific input features during the multimodal feature fusion process. This may lead to over-emphasis or neglect of certain modal features during the modal fusion process, resulting in the model failing to capture certain key forgery features, which in turn affects the judgment and location of forged video frames. Summary of the Invention

[0007] The purpose of the present invention is to solve the problem of insufficient representation of fine-scale forgery features in existing detection methods and the problem that existing methods cannot effectively capture image forgery artifacts, and to provide a multimodal deep forgery detection model for temporal forgery positioning, which includes a feature extraction module, an adaptive multimodal feature fusion module, a frame classification module and a boundary positioning module. The feature extraction module is constructed based on the ConvNeXt network, which provides the model with excellent ability to capture subtle forgery features from image frame sequences and audio streams. The adaptive multimodal feature fusion module can better obtain multimodal fusion feature representation by adjusting the fusion weights of different modal features. The frame classification module can generate frame-level authenticity predictions based on the feature representation. The boundary positioning module can generate the final forgery boundary prediction through vision, audio, fusion feature representation and frame classification prediction.

[0008] The present invention is achieved through the following technical solutions: a multimodal deepfake detection model for temporal forgery localization, comprising a feature extraction module, an adaptive multimodal feature fusion module, a frame classification module, and a boundary localization module;

[0009] The feature extraction module is mainly composed of a visual feature extraction module and an audio feature extraction module. The visual feature extraction module is constructed based on the ConvNeXt 3D visual encoder and is used to extract visual feature representations; the audio feature extraction module is constructed based on the ConvNeXt audio encoder and is used to extract audio feature representations;

[0010] The adaptive multimodal feature fusion module fuses the visual feature representation and the audio feature representation in the feature channel dimension by constructing an attention map to obtain a multimodal fusion feature;

[0011] The frame classification module is intended to process the visual feature representation and the audio feature representation obtained by the feature extraction module to obtain a visual frame-level prediction label and an audio frame-level prediction label;

[0012] The boundary localization module aims to obtain the final forged boundary prediction by processing the visual feature representation and audio feature representation obtained by the feature extraction module, the multimodal fusion features obtained by the adaptive multimodal feature fusion module, and the visual frame-level prediction labels and audio frame-level prediction labels obtained by the frame classification module.

[0013] To further better implement the multimodal deepfake detection model for temporal forgery localization described in the present invention, the following configuration is particularly adopted: the visual feature extraction module, when extracting visual feature representations, includes the following steps:

[0014] 1.1.1) Use TorchVision to obtain image frame sequences from the original video;

[0015] 1.1.2) Preprocessing the Image Frame: First, read the original RGB color channel information and convert it into a color channel tensor representation. Then, use the designed high-frequency feature extraction module to obtain the high-frequency component information of the image and convert it into a high-frequency component tensor representation. The high-frequency feature extraction module is composed of a high-pass filter based on the Gaussian Laplacian operator.

[0016] 1.1.3) Applying a ConvNeXt 3D network to extract feature representations of the RGB color channels and high-frequency components of the image; the ConvNeXt 3D network structure includes four sequences: sequence 1, sequence 2, sequence 3, and sequence 4, each of which is composed of 3, 3, 9, and 3 3D ConvNeXt blocks, respectively;

[0017] 1.1.4) The feature representations of the RGB color channels and high-frequency components of the image obtained in step 1.1.3) are fused using an adaptive visual feature fusion module designed based on a weighted parameter matrix to obtain a final visual feature representation.

[0018] To further improve the multimodal deepfake detection model for temporal forgery localization described in the present invention, the following configuration is particularly employed: A ConvNeXt 3D network is employed to extract feature representations of the image's RGB color channels and high-frequency components, respectively, including the following specific steps:

[0019] 1.1.3.1) Input the color channel tensor representation and the high-frequency component tensor representation, respectively, and perform an initial 3D convolutional layer to perform initial image modeling, providing basic features for subsequent network layers;

[0020] 1.1.3.2) After passing through a sequence of three 3D ConvNeXt blocks, layer normalization is performed and the input and output of each block are summed using residual connections; the final tensor shape remains unchanged.

[0021] 1.1.3.3) After a downsampling layer, the feature space dimension is reduced;

[0022] 1.1.3.4) Repeat steps 1.1.3.2) to 1.1.3.3) three times, replacing sequence 1 with sequence 2, sequence 3, and sequence 4 respectively during each repetition to further extract feature information;

[0023] 1.1.3.5) After step 1.1.3.4), map the features to the specified feature representation through a fully connected layer and two convolutional layers to facilitate multimodal fusion operations with subsequent audio features.

[0024] To further better implement the multimodal deepfake detection model for temporal forgery localization described in the present invention, the following configuration is particularly employed: Step 1.1.4) includes the following specific steps:

[0025] 1.1.4.1) Construct two tensor parameters with the same feature representations for the RGB color channel and high-frequency component obtained in step 1.1.3), and train the two tensor parameters to obtain two parameter matrices;

[0026] 1.1.4.2) Perform dot product operations on the two tensor parameter matrices with the feature representations of the RGB color channels and the high-frequency components, respectively, to obtain two weighted feature components;

[0027] 1.1.4.3) The two weighted feature components are then added together to obtain the final visual feature representation.

[0028] To further better implement the multimodal deepfake detection model for temporal forgery localization described in the present invention, the following configuration is particularly adopted: when the audio feature extraction module extracts audio feature representations, the following steps are included:

[0029] 1.2.1) Use TorchVision to obtain raw audio data from the original video;

[0030] 1.2.2) Convert the raw audio data into a Mel-spectrogram using short-time Fourier transform and Mel-filter;

[0031] 1.2.3) Applying a ConvNeXt network to extract audio feature representation; the ConvNeXt network comprises three sequences: sequence 1, sequence 2, and sequence 3, each sequence comprising 3, 3, and 3 ConvNeXt blocks, respectively.

[0032] To further better implement the multimodal deepfake detection model for temporal forgery localization described in the present invention, the following configuration is particularly employed: Step 1.2.3) includes the following steps:

[0033] 1.2.3.1) Input the Mel-spectrogram (audio tensor) and pass it through an initial convolutional layer to perform initial modeling of the audio information, providing basic features for subsequent network layers;

[0034] 1.2.3.2) After passing through a sequence of three ConvNeXt blocks, layer normalization is performed and the input and output of each block are summed using a residual connection; the final tensor shape remains unchanged.

[0035] 1.2.3.3) After a downsampling layer, the feature space dimension is reduced;

[0036] 1.2.3.4) Repeat steps 1.2.3.2) to 1.2.3.3) twice, replacing sequence 1 with sequence 2 and sequence 3 respectively during each repetition to further extract feature information;

[0037] 1.2.3.5) Map the features to the specified feature representation through a fully connected layer and a convolutional layer to obtain the audio feature representation, which facilitates multimodal fusion operations with the previously obtained visual features.

[0038] To further better implement the multimodal deepfake detection model for temporal forgery localization described in the present invention, the following configuration is particularly adopted: the adaptive multimodal feature fusion module obtains multimodal fusion features through the following steps:

[0039] 2.1) Concatenate the visual feature representation and audio feature representation obtained by the feature extraction module along the channel dimension to obtain the original concatenated features;

[0040] 2.2) The original concatenated features are normalized through a 1×1 convolution layer and a batch normalization layer, and the result is normalized through a ReLU Activation function, and then through a 3×3 convolution layer and Sigmoid The activation function is processed to obtain an attention weight that includes visual and audio channels;

[0041] 2.3) The attention weights obtained in step 2.2) are copied and concatenated in the channel dimension so that their size is consistent with the original concatenated features obtained in step 2.1) to obtain the final attention feature map;

[0042] 2.4) Perform weighted calculation on the original concatenated features obtained in step 2.1) and the attention feature map obtained in step 2.3) to obtain the final multimodal fusion features of vision and audio.

[0043] To further better implement the multimodal deepfake detection model for temporal forgery localization described in the present invention, the following configuration is particularly employed: the weighted calculation process in step 2.4) includes the following steps:

[0044] 2.4.1) Perform dot product operation on the original concatenated features and the attention feature map;

[0045] 2.4.2) Add the result obtained in step 2.4.1) to the original concatenated features;

[0046] 2.4.3) The result obtained in step 2.4.2) is further processed through a 1×1 convolutional layer for dimensionality reduction to obtain the final multimodal fusion features of vision and audio.

[0047] To further improve the multimodal deepfake detection model for temporal forgery localization described in the present invention, the following configuration is particularly employed: the frame classification module processes the visual feature representation and the audio feature representation to obtain visual frame-level prediction labels and audio frame-level prediction labels through the following steps:

[0048] 3.1) A 1×1 convolutional layer is used to reduce the dimensionality of the visual feature representation obtained from the visual feature extraction module to obtain a binary visual frame-level prediction label, which is then used in the subsequent boundary localization module processing.

[0049] 3.2) The audio feature representation obtained from the audio feature extraction module is reduced in dimensionality through a 1×1 convolutional layer to obtain a frame-level prediction label for the binary audio. This label continues to participate in the subsequent boundary localization module processing.

[0050] To further implement the multimodal deepfake detection model for temporal forgery localization described in the present invention, the following configuration is particularly employed: the boundary localization module obtains the final forgery boundary prediction through the following processing steps:

[0051] 4.1) Adding the visual feature representation and the visual frame-level predicted label, and then processing them through PRBs to obtain the visual position-aware boundary map and the visual channel-aware boundary map;

[0052] 4.2) The audio feature representation and the audio frame-level predicted label are added together, and then processed through PRBs to obtain audio position-aware boundary maps and audio channel-aware boundary maps.

[0053] 4.3) Adding the visual frame-level prediction labels, audio frame-level prediction labels, and multimodal fusion features, and then processing them through PRBs to obtain fused position-aware boundary maps and fused channel-aware boundary maps;

[0054] 4.4) Add the position-aware boundary map and channel-aware boundary map of the corresponding modality obtained in steps 4.1) to 4.3) and aggregate them through a 2×2 convolutional layer to obtain the position-channel boundary map of the corresponding modality;

[0055] 4.5) Finally, the final boundary map is calculated using the various boundary maps obtained in steps 4.1) to 4.4).

[0056] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0057] (1) To address the problem of insufficient representation of fine-scale forgery features in existing detection methods, this paper proposes a feature encoder based on ConvNeXt. The encoder retains the local advantages of convolutional networks through the strategies of depthwise separable convolution and group convolution, and enhances the ability to capture fine-scale features of images through more efficient parameter allocation. In addition, compared with traditional convolutional networks, the encoder increases the network width by adding more convolution kernels in each layer, further improving the ability to capture fine-scale features of images. The encoder also inherits the skip connection design in the residual network and introduces a new global response normalization layer (GRN) and Gaussian error linear unit (GELU) to help maintain the stability of feature cross-layer flow and reduce the loss of features when they are transferred between layers.

[0058] (2) To address the problem that existing methods cannot effectively capture image forgery artifacts, this paper proposes a high-frequency feature extraction module to enhance the representation of forgery features in the visual modality. This module consists of a high-pass filter based on the Gaussian Laplacian operator, which is specifically used to extract high-frequency component information in the image and highlight its detailed features. In addition, a visual feature adaptive fusion module for image RGB color and high-frequency component features is proposed. This module introduces a weighted parameter matrix to achieve pixel-level adaptive fusion of RGB color and high-frequency component features, which overall improves the model's capture and representation of visual forgery features.

[0059] (3) In response to the problem that existing methods cannot adaptively adjust the importance of different modal features during the multimodal feature fusion process, the present invention proposes an adaptive multimodal feature fusion module to improve the multimodal feature fusion effect. The module first splices the audio features and visual features to form a preliminary fusion feature, and then generates a feature attention matrix by calculation to dynamically evaluate the importance of each modal feature. On this basis, the original features are weighted and combined with the attention matrix to obtain the final multimodal fusion feature. The module can adaptively adjust the fusion weights of visual and audio features to ensure that key forgery feature information is fully retained, thereby significantly improving the accuracy and robustness of multimodal feature fusion and providing a more reliable feature representation for temporal forgery positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 This is a diagram of the overall architecture of the multimodal deep fake detection model described in the present invention.

[0061] Figure 2 The feature extraction module of the present invention

[0062] Figure 3is the filter kernel parameter diagram.

[0063] Figure 4 It is an adaptive visual feature fusion module.

[0064] Figure 5 It is an adaptive multimodal feature fusion module.

[0065] exist Figure 1 middle, For average operation; It is soft non-maximum suppression.

[0066] exist Figure 2 middle, is a sequence; For blocks; is a 3D convolutional layer; is a linear layer; is a 2d convolutional layer; It is a 1d convolution layer; is layer normalization; is the Gaussian error linear unit; Normalize the global response; To rearrange. DETAILED DESCRIPTION

[0067] The present invention will be further described in detail below with reference to the examples, but the embodiments of the present invention are not limited thereto.

[0068] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the invention for which protection is sought, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0069] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature identified as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, "plurality" means two or more, unless otherwise specifically defined.

[0070] Glossary:

[0071] ConvNeXt 3D: The next generation of 3D convolutional networks.

[0072] ConvNeXt: The next generation of convolutional networks.

[0073] TorchVision: A computer vision tool library.

[0074] 3D ConvNeXt: The next generation of 3D convolutional networks.

[0075] PRB: Proposal Relation Block.

[0076] Example 1:

[0077] This paper designs a multimodal deepfake detection model for temporal forgery localization, which solves the problem that existing detection methods are insufficient in representing fine-scale forgery features and the problem that existing methods cannot effectively capture image forgery artifacts. It includes a feature extraction module, an adaptive multimodal feature fusion module, a frame classification module, and a boundary localization module;

[0078] The feature extraction module is mainly composed of a visual feature extraction module and an audio feature extraction module. The visual feature extraction module is constructed based on the ConvNeXt 3D visual encoder and is used to extract visual feature representations; the audio feature extraction module is constructed based on the ConvNeXt audio encoder and is used to extract audio feature representations;

[0079] The adaptive multimodal feature fusion module fuses the visual feature representation and the audio feature representation in the feature channel dimension by constructing an attention map to obtain a multimodal fusion feature;

[0080] The frame classification module is intended to process the visual feature representation and the audio feature representation obtained by the feature extraction module to obtain a visual frame-level prediction label and an audio frame-level prediction label;

[0081] The boundary localization module aims to obtain the final forged boundary prediction by processing the visual feature representation and audio feature representation obtained by the feature extraction module, the multimodal fusion features obtained by the adaptive multimodal feature fusion module, and the visual frame-level prediction labels and audio frame-level prediction labels obtained by the frame classification module.

[0082] Example 2:

[0083] This embodiment is a further optimization based on the above embodiment. The similarities with the above technical solutions are not repeated here. To better implement the multimodal deep fake detection model for temporal fake location described in the present invention, the following configuration is particularly adopted: the visual feature extraction module includes the following steps when extracting visual feature representations:

[0084] 1.1.1) Use TorchVision to obtain image frame sequences from the original video;

[0085] 1.1.2) Preprocessing the Image Frame: First, read the original RGB color channel information and convert it into a color channel tensor representation. Then, use the designed high-frequency feature extraction module to obtain the high-frequency component information of the image and convert it into a high-frequency component tensor representation. The high-frequency feature extraction module is composed of a high-pass filter based on the Gaussian Laplacian operator.

[0086] 1.1.3) Applying a ConvNeXt 3D network to extract feature representations of the RGB color channels and high-frequency components of the image; the ConvNeXt 3D network structure includes four sequences: sequence 1, sequence 2, sequence 3, and sequence 4, each of which is composed of 3, 3, 9, and 3 3D ConvNeXt blocks, respectively;

[0087] 1.1.4) The feature representations of the RGB color channels and high-frequency components of the image obtained in step 1.1.3) are fused using an adaptive visual feature fusion module designed based on a weighted parameter matrix to obtain a final visual feature representation.

[0088] Example 3:

[0089] This embodiment is a further optimization based on any of the above embodiments. Similarities with the above technical solutions are not repeated here. To better implement the multimodal deepfake detection model for temporal forgery localization described in the present invention, the following configuration is specifically adopted: A ConvNeXt 3D network is used to extract feature representations of the image's RGB color channels and high-frequency components, respectively, including the following specific steps:

[0090] 1.1.3.1) Input the color channel tensor representation and the high-frequency component tensor representation, respectively, and perform an initial 3D convolutional layer to perform initial image modeling, providing basic features for subsequent network layers;

[0091] 1.1.3.2) After passing through a sequence of three 3D ConvNeXt blocks, layer normalization is performed and the input and output of each block are summed using residual connections; the final tensor shape remains unchanged.

[0092] 1.1.3.3) After a downsampling layer, the feature space dimension is reduced;

[0093] 1.1.3.4) Repeat steps 1.1.3.2) to 1.1.3.3) three times, replacing sequence 1 with sequence 2, sequence 3, and sequence 4 respectively during each repetition to further extract feature information;

[0094] 1.1.3.5) After step 1.1.3.4), map the features to the specified feature representation through a fully connected layer and two convolutional layers to facilitate multimodal fusion operations with subsequent audio features.

[0095] Example 4:

[0096] This embodiment is a further optimization based on any of the above embodiments. The similarities with the above technical solutions are not repeated here. To better implement the multimodal deep fake detection model for temporal forgery positioning described in the present invention, the following configuration is particularly adopted: Step 1.1.4) includes the following specific steps:

[0097] 1.1.4.1) Construct two tensor parameters with the same feature representations for the RGB color channel and high-frequency component obtained in step 1.1.3), and train the two tensor parameters to obtain two parameter matrices;

[0098] 1.1.4.2) Perform dot product operations on the two parameter matrices with the feature representations of the RGB color channels and the high-frequency components, respectively, to obtain two weighted feature components;

[0099] 1.1.4.3) The two weighted feature components are then added together to obtain the final visual feature representation.

[0100] Example 5:

[0101] This embodiment is a further optimization based on any of the above embodiments. The similarities with the above technical solutions are not repeated here. To better implement the multimodal deep fake detection model for temporal fake positioning described in the present invention, the following configuration is particularly adopted: when the audio feature extraction module extracts audio feature representations, the following steps are included:

[0102] 1.2.1) Use TorchVision to obtain raw audio data from the original video;

[0103] 1.2.2) Convert the raw audio data into a Mel-spectrogram using short-time Fourier transform and Mel-filter;

[0104] 1.2.3) Applying a ConvNeXt network to extract audio feature representation; the ConvNeXt network comprises three sequences: sequence 1, sequence 2, and sequence 3, each sequence comprising 3, 3, and 3 ConvNeXt blocks, respectively.

[0105] Example 6:

[0106] This embodiment is a further optimization based on any of the above embodiments. The similarities with the above technical solutions are not repeated here. To better implement the multimodal deep fake detection model for temporal fake positioning described in the present invention, the following configuration is particularly adopted: Step 1.2.3) includes the following steps:

[0107] 1.2.3.1) Input the Mel-spectrogram (audio tensor) and pass it through an initial convolutional layer to perform initial modeling of the audio information, providing basic features for subsequent network layers;

[0108] 1.2.3.2) After passing through a sequence of three ConvNeXt blocks, layer normalization is performed and the input and output of each block are summed using a residual connection; the final tensor shape remains unchanged.

[0109] 1.2.3.3) After a downsampling layer, the feature space dimension is reduced;

[0110] 1.2.3.4) Repeat steps 1.2.3.2) to 1.2.3.3) twice, replacing sequence 1 with sequence 2 and sequence 3 respectively during each repetition to further extract feature information;

[0111] 1.2.3.5) Map the features to the specified feature representation through a fully connected layer and a convolutional layer to obtain the audio feature representation, which facilitates multimodal fusion operations with the previously obtained visual features.

[0112] Example 7:

[0113] This embodiment is a further optimization based on any of the above embodiments. The similarities with the above technical solutions are not repeated here. In order to better implement the multimodal deep fake detection model for temporal fake positioning described in the present invention, the following setting is particularly adopted: the adaptive multimodal feature fusion module obtains the multimodal fusion feature through the following steps:

[0114] 2.1) Concatenate the visual feature representation and audio feature representation obtained by the feature extraction module along the channel dimension to obtain the original concatenated features;

[0115] 2.2) The original concatenated features are normalized through a 1×1 convolution layer and a batch normalization layer, and the result is normalized through a ReLU Activation function, and then through a 3×3 convolution layer and Sigmoid The activation function is processed to obtain an attention weight that includes visual and audio channels;

[0116] 2.3) The attention weights obtained in step 2.2) are copied and concatenated in the channel dimension so that their size is consistent with the original concatenated features obtained in step 2.1) to obtain the final attention feature map;

[0117] 2.4) Perform weighted calculation on the original concatenated features obtained in step 2.1) and the attention feature map obtained in step 2.3) to obtain the final multimodal fusion features of vision and audio.

[0118] Example 8:

[0119] This embodiment is a further optimization based on any of the above embodiments. The similarities with the above technical solutions are not repeated here. To better implement the multimodal deep fake detection model for temporal fake positioning described in the present invention, the following configuration is particularly adopted: the weighted calculation process in step 2.4) includes the following steps:

[0120] 2.4.1) Perform dot product operation on the original concatenated features and the attention feature map;

[0121] 2.4.2) Add the result obtained in step 2.4.1) to the original concatenated features;

[0122] 2.4.3) The result obtained in step 2.4.2) is further processed through a 1×1 convolutional layer for dimensionality reduction to obtain the final multimodal fusion features of vision and audio.

[0123] Example 9:

[0124] This embodiment is a further optimization based on any of the above embodiments. The similarities with the above technical solutions are not repeated here. To better implement the multimodal deep fake detection model for temporal fake positioning described in the present invention, the following configuration is particularly adopted: the frame classification module processes the visual feature representation and the audio feature representation through the following steps to obtain visual frame-level prediction labels and audio frame-level prediction labels:

[0125] 3.1) A 1×1 convolutional layer is used to reduce the dimensionality of the visual feature representation obtained from the visual feature extraction module to obtain a binary visual frame-level prediction label, which is then used in the subsequent boundary localization module processing.

[0126] 3.2) The audio feature representation obtained from the audio feature extraction module is reduced in dimensionality through a 1×1 convolutional layer to obtain a frame-level prediction label for the binary audio. This label continues to participate in the subsequent boundary localization module processing.

[0127] Example 10:

[0128] This embodiment is a further optimization based on any of the above embodiments. The similarities with the above technical solutions are not repeated here. To better implement the multimodal deep fake detection model for temporal fake positioning described in the present invention, the following configuration is particularly adopted: the boundary positioning module obtains the final fake boundary prediction through the following processing steps:

[0129] 4.1) Add the visual feature representation and the visual frame-level prediction label, and then pass PRB After processing, the boundary map of visual position perception and the boundary map of visual channel perception are obtained;

[0130] 4.2) Add the audio feature representation and the audio frame-level prediction label, and then pass PRB After processing, audio position-aware boundary mapping and audio channel-aware boundary mapping are obtained;

[0131] 4.3) Add the visual frame-level prediction label, audio frame-level prediction label and multimodal fusion features, and then pass PRB After processing, the boundary map of fused position perception and the boundary map of fused channel perception are obtained;

[0132] 4.4) Add the position-aware boundary map and channel-aware boundary map of the corresponding modality obtained in steps 4.1) to 4.3) and aggregate them through a 2×2 convolutional layer to obtain the position-channel boundary map of the corresponding modality;

[0133] 4.5) Finally, the final boundary map is calculated using the various boundary maps obtained in steps 4.1) to 4.4).

[0134] Example 11:

[0135] A multimodal deepfake detection model for temporal forgery localization, combining Figures 1 to 5 As shown, it includes a feature extraction module, an adaptive multimodal feature fusion module, a frame classification module and a boundary positioning module.

[0136] Combine Figure 2As shown in FIG, the feature extraction module is mainly composed of a visual feature extraction module constructed based on a ConvNeXt 3D visual encoder and an audio feature extraction module constructed based on a ConvNeXt audio encoder.

[0137] The visual feature extraction module is used to extract visual feature representation , comprising the following steps:

[0138] S1. Use TorchVision to obtain image frame sequences from the original video .

[0139] S2. Preprocess the image frame: First read the original RGB color channel information and convert it into a color channel tensor representation ; Then the high-frequency component (HF) information of the image is obtained through the designed high-frequency feature extraction module and converted into a high-frequency component tensor representation The high-frequency feature extraction module is composed of a high-pass filter based on the Laplacian of Gaussian operator. The parameters of the filter kernel are as follows: Figure 3 shown.

[0140] S3. Apply ConvNeXt 3D network to extract feature representation of RGB color channels of the image and high-frequency component feature representation ; including the following specific steps:

[0141] S3.1 Input color channel tensor representation separately and high-frequency component tensor representation , after an initial 3D convolution layer, the image is initially modeled to provide basic features for subsequent network layers;

[0142] S3.2) After passing through a sequence of three 3D ConvNeXt blocks, layer normalization is performed and the input and output of each block are summed using a residual connection; the final tensor shape remains unchanged.

[0143] S3.3) After a downsampling layer, the feature space dimension is reduced;

[0144] S3.4) Repeat steps S3.2) to S3.3) three times, replacing sequence 1 with sequence 2, sequence 3, and sequence 4 during each repetition to further extract feature information;

[0145] S3.5) After step S3.4), the features are mapped to the specified feature representation through a fully connected layer and two convolutional layers to obtain the feature representation of the RGB color channel of the image. and high-frequency component feature representation , which facilitates multimodal fusion operations with subsequent audio features.

[0146] The network structure of the ConvNeXt 3D network includes four sequences: sequence 1, sequence 2, sequence 3, and sequence 4. Each sequence is composed of 3D ConvNeXt blocks, and the number of 3, 3, 9, and 3 are respectively.

[0147] S4. Combination Figure 4 As shown, the feature representation of the RGB color channel of the image is obtained by the adaptive visual feature fusion module designed based on the weighted parameter matrix. and high-frequency component feature representation Fusion is performed to obtain the final visual feature representation ; including the following specific steps:

[0148] S4.1. Construct two feature representations of the RGB color channels obtained in step S3. and high-frequency component feature representation The same two tensor parameters and , the two tensor parameters and Two parameter matrices are obtained through training (RGB color image weighted parameter matrix) and (High-frequency image weighted parameter matrix), the calculation formula is as follows:

[0149] ;

[0150] S4.2. Two parameter matrices and Feature representation of RGB color channels respectively and high-frequency component feature representation Perform dot product operation to obtain two weighted feature components;

[0151] S4.3. Then add the two weighted feature components to get the final visual feature representation .

[0152] Audio feature extraction module, used to extract audio feature representation ; comprising the steps of:

[0153] A1. Use TorchVision to obtain raw audio data from the original video .

[0154] A2. Convert the original audio data into a Mel spectrum through short-time Fourier transform and Mel filter .

[0155] A3. Apply ConvNeXt network to extract audio feature representation ; comprising the steps of:

[0156] A3.1. Input Mel Spectrogram (Audio Tensor) , after an initial convolutional layer, the audio information is initially modeled to provide basic features for subsequent network layers;

[0157] A3.2. After a sequence of three ConvNeXt blocks, layer normalization is performed and the input and output of each block are added together using a residual connection; the final tensor shape remains unchanged.

[0158] A3.3, after a downsampling layer, the feature space dimension is reduced;

[0159] A3.4. Repeat steps A3.2 to A3.3 twice, replacing sequence 1 with sequence 2 and sequence 3 respectively during each repetition to further extract feature information;

[0160] A3.5. Map the features to the specified feature representation through a fully connected layer and a convolutional layer to obtain the audio feature representation , which facilitates multimodal fusion operations with previously obtained visual features.

[0161] Among them, the network structure of the ConvNeXt network includes three sequences, sequence one, sequence two, and sequence three. Each sequence is composed of ConvNeXt blocks, and the number of them is 3, 3, and 3 respectively.

[0162] Combine Figure 5 As shown in the figure, the adaptive multimodal feature fusion module fuses visual feature representations in the feature channel dimension by constructing an attention map. and audio feature representation , get multimodal fusion features ; When performing fusion, the following steps are included:

[0163] B1. Represent the visual features obtained by the feature extraction module and audio feature representation Splicing along the channel dimension to obtain the original splicing features .

[0164] B2. Original splicing features After a 1×1 convolution layer and a batch normalization layer for normalization, the result is passed through a ReLU Activation function, and then through a 3×3 convolution layer and SigmoidThe activation function is processed to obtain an attention weight that includes visual and audio channels , the calculation formula is as follows:

[0165] ; Where Z is the enhanced intermediate feature representation.

[0166] ReLU (Rectified Linear Unit): A piecewise linear activation function defined as , used to introduce nonlinear characteristics.

[0167] Batch Normalization (BN): A normalization technique that normalizes the input to each layer of a neural network. It calculates the mean and variance of a mini-batch of data, normalizing the input to a distribution with mean 0 and variance 1.

[0168] Conv 1×1 (1×1 convolution): An operation that extracts local spatial features using a 1×1 convolution kernel to capture local patterns (such as edges, textures) in an image or feature map.

[0169] Sigmoid: A nonlinear activation function that maps the input to the (0,1) interval, defined as , the output can be interpreted as probability and is often used in binary classification problems.

[0170] Conv 3×3 (3×3 Convolution) An operation that extracts local spatial features using a 3×3 convolution kernel, capturing local patterns (such as edges, textures) in an image or feature map.

[0171] B3. Attention weight obtained through step B2 Copy and splice in the channel dimension to make its size the same as the original splicing feature obtained in step B1 Consistent, get the final attention feature map The calculation formula is as follows:

[0172] , where C is the number of repetitions of the attention weight of the audio and video channel splicing, is the attention weight of the audio channel, A v is the attention weight of the video channel.

[0173] B4. The original splicing features obtained in step B1 and the attention feature map obtained in step 2.3) Perform weighted calculation to obtain the final multimodal fusion features of vision and audio .

[0174] In step B4, the weighted calculation process includes the following steps:

[0175] B4.1. Original splicing features and attention feature maps Perform dot product operation;

[0176] B4.2. Combine the result obtained in step B4.1 with the original splicing feature Perform addition operation;

[0177] B4.3. The result obtained in step B4.2 is further processed through a 1×1 convolutional layer to obtain the final multimodal fusion features of vision and audio. .

[0178] The frame classification module (including the visual frame classification module and the audio frame classification module) is designed to represent the visual features obtained by the feature extraction module. and audio feature representation Processing is performed to obtain visual frame-level prediction labels and audio frame-level prediction labels; the processing includes the following specific steps:

[0179] C1, through a 1×1 size convolution layer to obtain the visual feature representation from the visual feature extraction module Perform dimensionality reduction to obtain a binary visual frame-level prediction label , the label continues to participate in the subsequent boundary positioning module processing.

[0180] C2, through a 1×1 size convolution layer to obtain the audio features from the audio feature extraction module Indicates that dimensionality reduction processing is performed to obtain a frame-level prediction label for a binary audio classification , the label continues to participate in the subsequent boundary positioning module processing.

[0181] The boundary localization module (including visual boundary matching layer, fusion boundary matching layer, and audio boundary matching layer) aims to represent the visual features obtained by the feature extraction module. and audio feature representation , multimodal fusion features obtained through the adaptive multimodal feature fusion module , and the visual frame-level prediction labels obtained by the frame classification module and frame-level predicted labels for audio Processing is performed to obtain the final forged boundary prediction .

[0182] The boundary positioning module is processing to obtain the final forged boundary prediction The following specific steps are included:

[0183] D1. Representing visual features and visual frame-level predicted labels Perform addition operation, then pass PRB After processing, we get the boundary map of visual position perception and boundary mapping perceived by the visual channel , the calculation process is as follows:

[0184] .

[0185] D2. Representing audio features and audio frame-level predicted labels Perform addition operation, then pass PRB After processing, we get the audio position-aware boundary map and audio channel-aware boundary mapping , the calculation process is as follows:

[0186] .

[0187] D3, predict the visual frame level label , frame-level predicted labels for audio and multimodal fusion features Perform addition operation and then process it through PRB to obtain the boundary map of fusion position awareness and fused channel-aware boundary mapping , the calculation process is as follows:

[0188] .

[0189] D4, the corresponding mode obtained from step D1 to step D3 Location-aware boundary mapping and channel-aware boundary mapping Perform the addition operation and then aggregate through a 2×2 convolutional layer to obtain the position-channel boundary mapping of the corresponding mode , the calculation process is as follows:

[0190] ;

[0191] D5 finally uses the various boundary mappings obtained in steps D1 to D4 to calculate the final forged boundary prediction , the calculation formula is as follows:

[0192] .

[0193] in, Represents the weights of different boundary mappings, by combining the features of different modalities and boundary mapping It is obtained by stacking 1×1 convolutional layers and then performing an average operation. The calculation formula is as follows:

[0194] .in, Mapping for video boundaries; For audio boundary mapping, ;

[0195] In the inference phase, Soft Non-Maximum Suppression (S-NMS) is used to remove duplicate prediction results and eliminate prediction redundancy. The final boundary map It will be used as the inference result of the model.

[0196] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any simple modification or equivalent change made to the above embodiment based on the technical essence of the present invention shall fall within the scope of protection of the present invention.

Claims

1. A multimodal deepfake detection model for temporal forgery localization, characterized by: It includes feature extraction module, adaptive multimodal feature fusion module, frame classification module and boundary positioning module; The feature extraction module is mainly composed of a visual feature extraction module and an audio feature extraction module. The visual feature extraction module is constructed based on the ConvNeXt 3D visual encoder and is used to extract visual feature representations; The audio feature extraction module is constructed based on the ConvNeXt audio encoder and is used to extract audio feature representations. When performing visual feature representation extraction, the module includes the following steps: 1.1.1) Use TorchVision to obtain image frame sequences from the original video; 1.1.2) Preprocessing the Image Frame: First, read the original RGB color channel information and convert it into a color channel tensor representation. Then, use the designed high-frequency feature extraction module to obtain the high-frequency component information of the image and convert it into a high-frequency component tensor representation. The high-frequency feature extraction module is composed of a high-pass filter based on the Gaussian Laplacian operator. 1.1.3) Applying a ConvNeXt 3D network to extract feature representations of the RGB color channels and high-frequency components of the image; the ConvNeXt 3D network structure includes four sequences: sequence 1, sequence 2, sequence 3, and sequence 4, each of which is composed of 3, 3, 9, and 3 3D ConvNeXt blocks, respectively; 1.1.4) Using the results from step 1.1.3), the feature representations of the image's RGB color channels and high-frequency components are fused using an adaptive visual feature fusion module designed based on a weighted parameter matrix to obtain the final visual feature representation. The adaptive multimodal feature fusion module fuses the visual feature representation and the audio feature representation in the feature channel dimension by constructing an attention map to obtain a multimodal fusion feature; and the adaptive multimodal feature fusion module obtains the multimodal fusion feature by the following steps: 2.1) Concatenate the visual feature representation and audio feature representation obtained by the feature extraction module along the channel dimension to obtain the original concatenated features; 2.2) The original concatenated features are normalized through a 1×1 convolution layer and a batch normalization layer, and the result is normalized through a ReLU Activation function, and then through a 3×3 convolution layer and Sigmoid The activation function is processed to obtain an attention weight that includes visual and audio channels; 2.3) The attention weights obtained in step 2.2) are copied and concatenated in the channel dimension so that their size is consistent with the original concatenated features obtained in step 2.1) to obtain the final attention feature map; 2.4) Perform weighted calculation on the original concatenated features obtained in step 2.1) and the attention feature map obtained in step 2.3) to obtain the final multimodal fusion features of vision and audio; The frame classification module processes the visual feature representation and the audio feature representation obtained by the feature extraction module to obtain a visual frame-level prediction label and an audio frame-level prediction label; The boundary localization module obtains the final forged boundary prediction by processing the visual feature representation and audio feature representation obtained by the feature extraction module, the multimodal fusion features obtained by the adaptive multimodal feature fusion module, and the visual frame-level prediction labels and audio frame-level prediction labels obtained by the frame classification module.

2. A multimodal deepfake detection model for temporal forgery localization according to claim 1, characterized in that: Applying the ConvNeXt 3D network to extract the feature representations of the image's RGB color channels and high-frequency components includes the following specific steps: 1.1.3.1) Input the color channel tensor representation and the high-frequency component tensor representation, and perform an initial 3D convolutional layer to perform initial image modeling; 1.1.3.2) After passing through a sequence of three 3D ConvNeXt blocks, perform layer normalization and sum the input and output of each block using a residual connection; 1.1.3.3) After a downsampling layer, the feature space dimension is reduced; 1.1.3.4) Repeat steps 1.1.3.2) to 1.1.3.3) three times, replacing sequence 1 with sequence 2, sequence 3, and sequence 4 respectively during each repetition. 1.1.3.5) After step 1.1.3.4), map the features to the specified feature representation through a fully connected layer and two convolutional layers.

3. The multimodal deepfake detection model for temporal forgery localization according to claim 1, characterized in that: The step 1.1.4) includes the following specific steps: 1.1.4.1) Construct two tensor parameters with the same feature representations for the RGB color channel and high-frequency component obtained in step 1.1.3), and train the two tensor parameters to obtain two parameter matrices; 1.1.4.2) Perform dot product operations on the two parameter matrices with the feature representations of the RGB color channels and the high-frequency components, respectively, to obtain two weighted feature components; 1.1.4.3) The two weighted feature components are then added together to obtain the final visual feature representation.

4. A multimodal deepfake detection model for temporal forgery localization according to claim 1, 2, or 3, characterized in that: When the audio feature extraction module extracts audio feature representation, it includes the following steps: 1.2.1) Use TorchVision to obtain raw audio data from the original video; 1.2.2) Convert the raw audio data into a Mel-spectrogram using short-time Fourier transform and Mel-filter; 1.2.3) Applying a ConvNeXt network to extract audio feature representation; the ConvNeXt network comprises three sequences: sequence 1, sequence 2, and sequence 3, each sequence comprising 3, 3, and 3 ConvNeXt blocks, respectively.

5. The multimodal deepfake detection model for temporal forgery localization according to claim 4, characterized in that: The step 1.2.3) includes the following steps: 1.2.3.1) Input the Mel-spectrogram and pass it through an initial convolutional layer to perform initial modeling of the audio information; 1.2.3.2) After a sequence of three ConvNeXt blocks, perform layer normalization and sum the input and output of each block using a residual connection; 1.2.3.3) After a downsampling layer, the feature space dimension is reduced; 1.2.3.4) Repeat steps 1.2.3.2) to 1.2.3.3) twice, replacing sequence 1 with sequence 2 and sequence 3 respectively each time; 1.2.3.5) Map the features to the specified feature representation through a fully connected layer and a convolutional layer to obtain the audio feature representation.

6. The multimodal deepfake detection model for temporal forgery localization according to claim 1, characterized in that: The weighted calculation process in step 2.4) includes the following steps: 2.4.1) Perform dot product operation on the original concatenated features and the attention feature map; 2.4.2) Add the result obtained in step 2.4.1) to the original concatenated features; 2.4.3) The result obtained in step 2.4.2) is further processed through a 1×1 convolutional layer for dimensionality reduction to obtain the final multimodal fusion features of vision and audio.

7. A multimodal deepfake detection model for temporal forgery localization according to claim 1, 2, or 3, characterized in that: The frame classification module processes the visual feature representation and the audio feature representation through the following steps to obtain the visual frame-level prediction label and the audio frame-level prediction label: 3.1) Perform dimensionality reduction on the visual feature representation obtained from the visual feature extraction module through a 1×1 convolutional layer to obtain a binary visual frame-level prediction label; 3.2) The audio feature representation obtained from the audio feature extraction module is reduced in dimensionality through a 1×1 convolutional layer to obtain a frame-level prediction label for a binary audio classification.

8. A multimodal deepfake detection model for temporal forgery localization according to claim 1, 2, or 3, characterized in that: The boundary localization module obtains the final forged boundary prediction through the following processing steps: 4.1) Adding the visual feature representation and the visual frame-level predicted label, and then processing them through PRBs to obtain the visual position-aware boundary map and the visual channel-aware boundary map; 4.2) The audio feature representation and the audio frame-level predicted label are added together, and then processed by PRB to obtain the audio position-aware boundary map and the audio channel-aware boundary map; 4.3) Adding the visual frame-level prediction labels, audio frame-level prediction labels, and multimodal fusion features, and then processing them through PRBs to obtain fused position-aware boundary maps and fused channel-aware boundary maps; 4.4) Add the position-aware boundary map and channel-aware boundary map of the corresponding modality obtained in steps 4.1) to 4.3) and aggregate them through a 2×2 convolutional layer to obtain the position-channel boundary map of the corresponding modality; 4.5) Finally, the final boundary map is calculated using the various boundary maps obtained in steps 4.1) to 4.4).

Images