A time sequence forgery positioning method for audio-video multi-modal deep forged content

Abstract

The application discloses a kind of time sequence forgery positioning methods for audio-video multi-modal content, method uses pre-training behavior recognition network and pre-training audio feature representation network to extract the feature sequence of visual and audio content.On this basis, reconstruction learning method is used to train feature reconstruction encoder and feature reconstruction decoder, to learn the latent distribution of real sample feature sequence.Through reconstruction error attention enhancement module, for improving the degree of attention to time sequence forgery segment.At the same time, parallel feature pyramid module is introduced, to enhance the expression ability of small tampering segment feature.Finally, enhanced feature sequence is input into time sequence feature decoder, to determine the start and end time of the forged segment.This method can effectively assist in positioning the forged segment in local forged video, and has good positioning accuracy, AP@0.75 score can reach 95.54%, with practical application value.

Description

Technical Field

[0001] This invention relates to the field of video forensics technology, and in particular to a method for locating temporal forgeries in multimodal audio and video content. Background Technology

[0002] In recent years, with the rapid development of AI-generated technology, concerns about the authenticity of multimedia content have been increasing. Although some progress has been made in binary classification detection methods for entire deepfake content, practical algorithms still lack solutions to prevent the abuse of the newly emerging and more threatening partial deepfake content in real-world network environments. This forgery typically targets partial alterations of multimodal temporal video and audio content, where the main content is genuine, but only a local segment is modified, leading to a complete distortion of the original content. Since the forged content constitutes only a small proportion of the original content, this significantly increases the difficulty of the detection task. For example, the paper "Thinking in Frequency: Face Forgery Detection by Mining Frequency-Aware Clues," published at the 2020 European Conference on Computer Vision (ECCV), achieved a relatively high accuracy of 98.62% AUC in a binary classification task distinguishing between videos generated entirely by deepfake technology and real videos. However, for identifying videos containing only partial alterations from real videos, the method's performance was limited, with an AUC score of only 52.0%. To address the aforementioned issues, the paper "Do You Really Mean That?" published at the 2022 International Conference on Digital Image Computing: Techniques and Applications (DICTA) proposed the first semantically relevant temporal forgery localization dataset and an algorithm to locate tampered segments. However, the algorithm's localization accuracy was low, with an AP@0.75 score of only 38.50%. This suggests that when facing more dangerous local forgery scenarios, it is necessary to design new frameworks to determine the specific start and end times of forgery in multimedia content, thereby promoting the rational application of artificial intelligence generation technology and reducing potential risks and adverse effects. Summary of the Invention

[0003] The purpose of this invention is to overcome the above-mentioned limitations and provide a method for locating temporal forgery of audio and video multimodal content, thereby improving the accuracy of locating locally forged multimedia content.

[0004] The objective of this invention is achieved through the following technical solution:

[0005] A temporal forgery localization method for multimodal deepfake content in audio and video, utilizing a pre-trained behavior recognition network f B Obtain the feature sequence F of visual content b ;Utilizing a pre-trained audio feature representation network f A Obtain the feature sequence F of the audio content a ;Utilizing the feature space mapping module f Y , the feature sequence F b and F a The fusion mapping is a feature sequence F m ; in the characteristic sequence F m Based on this, the encoder f is reconstructed using features. E Obtain the implicit feature Z and use a binary classifier f S Determine the authenticity of the original multimedia content corresponding to implicit features; based on the implicit feature Z, use the feature to reconstruct the decoder f. D Obtain the reconstructed feature sequence In the original feature sequence F m and Based on this, the reconstruction error attention enhancement module f is used. R Improve the feature sequence F m The degree of attention paid to temporal forgery segments is determined by obtaining the feature sequence F enhanced based on reconstruction feature error. crat Introducing a parallel feature pyramid module f PCF Enhanced feature sequence F crat The feature representation capability of small-scale tampered fragments is utilized to obtain feature sequence pyramids P0, P1, P2, P3, and P4; a temporal feature decoder f is constructed. LOC Each includes a decoder classification header f cls and decoder regression head f reg We jointly analyze the feature sequence pyramids P0, P1, P2, P3, and P4 to obtain the trained model f. C The goal is to determine the start and end times of the forged fragments; this includes the following main steps:

[0006] Step 1: Obtain training audio and video data containing both real and fake samples. Real samples are the original, unaltered content, while fake samples are audio and video clips with partially altered segments, including labels for the start and end times of the altered segments. Based on the modality of the data, the visual content is input into the pre-trained behavior recognition network f. B Extracting feature sequence F b The audio content is input into the pre-trained audio feature representation network f. A Extracting feature sequence F a ;

[0007] Step 2: Construct the feature space mapping module f Y The extracted visual content feature sequence F is obtained by using convolutional mapping and feature concatenation. b and the feature sequence F of audio content a Mapping to a unified feature space yields the feature sequence F. m ;

[0008] Step 3: Construct a real sample reconstruction module f based on the reconstruction learning method AE The real sample reconstruction module consists of the feature reconstruction encoder f E and feature reconstruction decoder f D It is used to learn the latent feature distribution of feature sequences of real samples;

[0009] Step 4: Construct the reconstruction error attention enhancement module f R Improve the original feature sequence F m The degree of attention paid to the features of forged fragments is used to obtain the enhanced feature sequence F. crat ;

[0010] Step 5: Construct the parallel feature pyramid module f PCF Enhanced feature sequence F crat To assess the expressive power of features in minutely altered fragments, feature sequence pyramids P0, P1, P2, P3, and P4 were obtained.

[0011] Step 6: Construct a temporal feature decoder f LOC Each includes a decoder classification header f cls and decoder regression head f reg We jointly analyze the abnormal features in the feature sequence pyramids P0, P1, P2, P3, and P4, and determine the start and end times of the forged segments. At this point, we obtain the trained model f. C ;

[0012] Step 7: Extract the multimedia content to be detected into a feature sequence and input it into the trained model f. CThe timestamps of the predicted forged fragments and their corresponding forgery probabilities are obtained. Non-maximum suppression is used to post-process the prediction results, and the start and end times of the forged fragments are given.

[0013] Furthermore, in step 1, the specific method for obtaining the feature sequence is as follows:

[0014] (1) Separate the video content from the audio content of the audio-visual content; decode the video content into video frames, calculate the optical flow of adjacent frames, and save them as optical flow frames;

[0015] (2) Input the extracted video frames and optical flow frames into the pre-trained behavior recognition network f B In the process, visual feature sequence F is obtained. b T×C b T is the length of the feature sequence, C b For feature dimensions;

[0016] (3) Convert the audio content into a Mel spectrogram;

[0017] (4) Input the Mel spectrogram into the pre-trained audio feature representation network f A In the middle, and use interpolation to obtain the audio feature sequence F a Its dimensions are T×C a C a For feature dimensions;

[0018] Furthermore, in step 2, the specific method for mapping features to a unified feature space is as follows:

[0019] (1) Visual feature sequence F b and audio feature sequence F a By concatenating the feature channels in the extended dimension, feature F is obtained. ba The dimensions are T×(C) b +C a );

[0020] (2) The spliced ​​feature sequence F ba The input is fed into a one-dimensional convolution layer to obtain the feature F′. ba The number of channels will be changed from C b +C a Convert to a new feature dimension C ba ;

[0021] (3) F′ ba The input is fed into an N-layer module consisting of one-dimensional convolutions and layer normalization, embedding the features into a unified feature space to obtain the feature sequence F. m Its dimensions are T×C m C m For feature dimensions.

[0022] Furthermore, in step 3, the real sample reconstruction module f based on the reconstruction learning method... AE The specific method is as follows:

[0023] (1) Feature sequence F m The input is fed into the feature reconstruction encoder f. E The low-dimensional implicit feature Z is extracted. The feature reconstruction encoder consists of L one-dimensional convolutions, instance normalization layers, and activation layers. The implicit feature size is... C z For feature dimensions;

[0024] (2) Input the implicit feature Z into a simple binary classifier f, which consists of a one-dimensional convolution, layer normalization and activation layer and a one-dimensional global average pooling layer. S In the middle, the output probability p a The size is 2, used to determine whether a feature sequence contains forged segments, and employs a binary classification cross-entropy loss function. Calculate the corresponding loss;

[0025] (3) Input the implicit feature sequence Z into the feature reconstruction decoder f D Obtain the reconstructed feature sequence Calculate the reconstructed feature sequence The feature sequences belonging to real samples With the original feature sequence F m The feature sequence F belonging to the real sample mr The distance is used as the reconstruction loss of the real sample. The feature reconstruction decoder consists of L one-dimensional transposed convolutions, an instance normalization layer, and an activation layer.

[0026] Furthermore, in step 4, a reconstruction error attention enhancement module f is constructed. R The specific method is as follows:

[0027] (1) Simultaneously, for reconstructing feature sequences and the original feature sequence F m By incorporating positional encoding, the ability to represent temporal features is enhanced, resulting in new position-sensitive feature sequences. and F pe ;

[0028] (2) Apply layer normalization and one-dimensional convolution operations to the feature sequence respectively. and F pe Mapped to feature F k and F q The two features have the same size, T×D, where D is the feature dimension after mapping;

[0029] (3) Calculate the characteristic sequence F k and characteristic sequence Fq The correlation matrix of the transpose The cross-attention matrix F is obtained by applying the Softmax operation to each row element of the matrix. cra Its dimensions are T×T;

[0030] (4) Use layer normalization and one-dimensional convolution operations to transform the feature F pe Mapped to feature F v The dimensions are T×D;

[0031] (5) Using the cross-attention matrix F cra With characteristic sequence F v Multiply to obtain the reconstruction error attention feature F ea The dimensions are T×D;

[0032] (6) Use residual connections to connect feature F ea With F pe The features are added together and processed using a feedforward neural network to obtain feature F based on the reconstruction feature error enhancement. cra t.

[0033] Furthermore, in step 5, a parallel feature pyramid module f is constructed. PCF The specific method is as follows:

[0034] (1) Based on the enhanced feature sequence F cra t, using a Transformer with a scaling factor of 1 to further enhance the feature sequence, outputting features The size is T×Dfpn, where Dfpn is the feature dimension of the feature pyramid;

[0035] (2) Use a Transformer with a scaling factor of 2 to capture features. Deep semantic information to obtain features Size is

[0036] (3) Using the cross-attention module fcr, in and Based on this, obtain high-resolution semantically enhanced feature sequences The dimensions are T×Dfpn;

[0037] (4) Using the cross-attention module fcr, in and Based on this, obtain high-resolution semantically enhanced feature sequences Then use a Transformer with a scaling factor of 4 to obtain features. Size is

[0038] (5) Using the cross-attention module fcr, in and Based on this, obtain high-resolution semantically enhanced feature sequences The dimensions are T×Dfpn;

[0039] (6) Using the cross-attention module fcr, in and Based on this, obtain high-resolution semantically enhanced feature sequences Then use a Transformer with a scaling factor of 8 to obtain features. Size is

[0040] (7) Using the cross-attention module fcr, in and Based on this, obtain high-resolution semantically enhanced feature sequences The dimensions are T×Dfpn;

[0041] (8) Using the cross-attention module fcr, in and Based on this, obtain high-resolution semantically enhanced feature sequences Then use a Transformer with a scaling factor of 16 to obtain features. Size is

[0042] (9) Using the cross-attention module, in and Based on this, obtain high-resolution semantically enhanced feature sequences The dimensions are T×Dfpn;

[0043] (10) The obtained feature sequence set As a parallel feature pyramid, it is sequentially input into the feature pyramid network f. fpn We obtain feature sequence pyramids P0, P1, P2, P3, and P4 at five scales.

[0044] In practice, the input order is P. pl P in Then the cross-attention module f cr The specific construction method is as follows:

[0045] (1) Given input features P pl By using layer normalization and one-dimensional convolution operations, the features are mapped to the query vector Q. p ;

[0046] (2) Given input features P in By using layer normalization and one-dimensional convolution operations, the features are mapped to key vectors K.p ;

[0047] (3) Given input features P in By using layer normalization and one-dimensional convolution operations, the features are mapped to a numerical vector V. p ;

[0048] (4) Based on the feature Q p K p V p Obtain the feature sequence of the multi-scale cross-attention table:

[0049]

[0050] Where g(·) represents the time interpolation function, used to align the time lengths of different features. Used as a scaling factor, Softmax is calculated along the rows of the feature matrix;

[0051] In implementation, the feature pyramid network f fpn The specific calculation method is as follows:

[0052] (1) Given a feature sequence Using one-dimensional convolution to perform a linear transformation on the features, we obtain the features. Size is Then, a linear transformation is performed using one-dimensional convolution to map the features to P4;

[0053] (2) Given features Upsampled using bilinear interpolation to the same level as... Consistent, dimensions are Combine it with features Add to obtain features Use a one-dimensional convolutional weighted mapping as feature P3;

[0054] (3) Given features Upsampled using bilinear interpolation to the same level as... Consistent, dimensions are Combine it with features Add to obtain features Use a one-dimensional convolutional weighted mapping as feature P2;

[0055] (4) Given features Upsampled using bilinear interpolation to the same level as... Consistent, dimensions are Combine it with features Add to obtain features Use a one-dimensional convolutional weighted mapping to form feature P1;

[0056] (5) Given features Upsampled using bilinear interpolation to the same level as... Consistent, dimensions are T×D fpn to associate it with features Add to obtain features Use a one-dimensional convolutional weighted mapping to form feature P0;

[0057] Furthermore, in step 6, the temporal feature decoder f LOC The components include the classification header module f cls Regression Head Module f reg :

[0058] (1) The classification head f cls There are five feature heads in total, corresponding to the feature sequence pyramids P0, P1, P2, P3, and P4 described in the previous step. Each feature head contains three modules, each consisting of a one-dimensional convolution, layer normalization, and an activation function. For the output of the concatenated three modules, Softmax is used to calculate the probability that a given segment is real or fake. The results from the five feature heads are concatenated along the time dimension to obtain the final result p. t The size is

[0059] (2) The regression head f reg There are five in total, each corresponding to one of the classification heads. Each regression head contains three modules, each consisting of a one-dimensional convolution, layer normalization, and an activation function. The output uses two parallel one-dimensional convolutions and simultaneously outputs the fake start time s. t and the corresponding duration d t All sizes

[0060] In practice, after step 6 is completed, the total training loss function is:

[0061]

[0062] in, This is the cross-entropy loss function for binary classification. For the reconstruction loss of real samples, The Focal loss function is used to calculate the loss between the forgery probability of a fragment and the ground truth label. The GIoU loss function is used to calculate the overlap rate between the predicted fake segment and the ground truth segment on the timeline. The goal is to maximize the overlap rate for better performance. This completes the training of the model f. C .

[0063] In step 7, the trained model f is used. CThe feature sequences extracted from multimedia content are predicted. Non-maximum suppression is used to post-process all predicted probabilities and their corresponding time axis coordinates, removing low-probability prediction regions and merging results from similar regions. Finally, based on requirements, the forgery probabilities are sorted from high to low, and a specified number of results are selected as the prediction results for the forged segments.

[0064] This invention utilizes deep learning technology to ensure multimedia content security. It extracts specified feature sequences from video, audio, or audio-visual media, reconstructs genuine feature sequences, amplifies differences in forged sequences, and uses a parallel pyramid structure to mine features from shorter forged segments to predict the start and end times of forged content within the sequence.

[0065] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0066] 1. This invention utilizes a pre-trained behavior recognition network and a pre-trained audio feature representation network to extract feature sequences and map them to a unified feature space. It can adapt to inputs of different modalities such as audio, video, and audio-visual content, and can be easily extended to other types of sequence data.

[0067] 2. Learn the latent feature distribution of real samples by using the real sample reconstruction mechanism, and enhance the salience of fake sample features by combining the reconstruction error of fake samples with the attention enhancement module of reconstruction feature error;

[0068] 3. By utilizing a parallel feature pyramid module, the original features of shorter forged fragments are preserved while enhancing their feature representation capabilities, thus improving the temporal forgery localization effect. The AP@0.75 score is increased from 38.50% in existing algorithms to 95.54%. Attached Figure Description

[0069] Figure 1 This is a flowchart of the implementation steps of the present invention.

[0070] Figure 2 This is a network structure diagram of the implementation steps of the present invention.

[0071] Figure 3 Parallel Feature Pyramid Module f, which is the implementation step of this invention PCF Structural diagram.

[0072] Figure 4 This is a comparison of the experimental results of the present invention with those of existing methods in a publicly available verification example.

[0073] Figure 5 This is a diagram showing the detection results of an embodiment of the present invention. Detailed Implementation

[0074] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0075] like Figures 1-3 As shown, this embodiment of the invention constructs a temporal forgery localization method for multimodal deepfake content in audio and video, including a pre-trained behavior recognition network f. B Pre-trained audio feature representation network f A Feature space mapping module f Y Feature reconstruction encoder f E Feature reconstruction decoder f D Reconstruction error attention enhancement module f R Parallel Feature Pyramid Module f PCF and decoder classification head f cls Decoder regression head f ref The nonmaximum suppression operation module constitutes the entire model framework. Figure 1 The workflow of this invention is described; Figure 2 A specific network structure diagram of one embodiment of the present invention is provided; Figure 3 The parallel feature pyramid module f in one embodiment of the present invention is given. PCF Structural diagram.

[0076] Step 1: Obtain multimedia fake samples that contain partial fakes and corresponding real samples as training data. Multimedia samples refer to multimodal audio and video content.

[0077] Step 2, use the pre-trained behavior recognition network f B and pre-trained audio feature representation network f A Feature sequences F of visual and auditory content are extracted respectively. b and F a Interpolation is used to align the time dimensions of feature sequences from different modalities.

[0078] In some specific implementation scenarios, a TSN network with a stride of 1 is used as the training method for the action recognition network f. B It can extract visual modal features from videos, with video frames and optical flow frames as inputs. It uses a BYOL-A network as a pre-trained audio feature representation network. A It can extract audio modal features;

[0079] Step 3, construct the feature space mapping module f Y Features from different modalities are fused and mapped onto a unified feature space to obtain the feature sequence F. m Its dimensions are T×C m C m For feature dimensions;

[0080] In some specific implementation scenarios, features from different modalities are concatenated along the feature channel dimension, and two feature space mapping modules consisting of one-dimensional convolution and layer normalization are used to map the features; T is 768, and if insufficient, upsampling is used for unification, C m It is 256;

[0081] Step 4: Construct a real sample reconstruction module f based on the reconstruction learning method. AE The encoder f is reconstructed using features. E The feature sequence is mapped to implicit features Z, and the implicit features Z are input into the binary classifier f. S In the output sample forgery probability p a and using cross-entropy Supervision. Constructing a feature reconstruction decoder f D Mapping the latent features back to the original feature space yields the reconstructed feature sequence. And reconstruct the feature sequences The feature sequences belonging to real samples With the original feature sequence F m The feature sequence F belonging to the real sample mr Using the reconstruction loss function Supervision is used to help the model distinguish between real and fake samples, and to improve the accuracy of fake samples.

[0082] In some specific implementation scenarios, the feature reconstruction encoder f E There are 5 modules, each containing a one-dimensional convolution, an instance normalization layer, and an activation layer; the binary classifier consists of a one-dimensional convolution, layer normalization and activation layers, and a one-dimensional global average pooling layer; the feature reconstruction decoder f D It consists of 5 modules, each containing a one-dimensional transposed convolution, an instance normalization layer, and an activation layer; the reconstruction loss function can be calculated using L1 distance;

[0083] Step 5: Construct the reconstruction feature error attention enhancement module f R Given the original feature sequence F m and reconstructed feature sequences The large error in reconstructing the feature sequence due to reliance on forgery enhances the original feature sequence F. m To obtain feature F based on reconstruction feature error enhancement crat ;

[0084] In some specific implementation scenarios, the specific steps for calculating the reconstruction feature error enhancement are as follows: 1) Input feature sequence F m and 2) Add positional encoding to the two feature sequences. The positional encoding uses sinusoidal positional encoding to obtain the features. and F pe3) Construct a transformation module consisting of a layer normalization layer and a one-dimensional convolutional layer to transform the features and f pe Mapped to F respectively k and F q The feature size is 768×256; 4) Calculate the correlation matrix of the feature sequences to obtain 5) Apply the Softmax operation to each row of the matrix to obtain the cross-attention matrix F. cra The size is 768×256; 6) Construct a transformation module consisting of a layer normalization layer and a one-dimensional convolutional layer to transform the feature F pe Mapped to F v 7) Use cross-attention matrix F cra With characteristic sequence F v Multiply to obtain the reconstruction error attention feature F ea 8) Use residual connections to connect feature F. ea With F pe Addition; 9) Use a feedforward neural network consisting of one-dimensional convolution, non-linear activation layers, and one-dimensional convolution to fuse the added features into an error enhancement feature F. crat The dimensions are 768×256;

[0085] In some specific implementation scenarios, the nonlinear activation function is ReLU or Leaky ReLU.

[0086] In some specific implementations, the first one-dimensional convolution of the feedforward neural network maps the features from 256 dimensions to 2048 dimensions. After filtering using a non-linear activation function, a second one-dimensional convolution is used to map them back to the original 256 dimensions.

[0087] Step 6, construct the parallel feature pyramid module f PCF Given the enhanced feature F crat The multi-scale feature sequence pyramids P0, P1, P2, P3, and P4 are obtained.

[0088] In some specific implementation scenarios, such as Figure 3 As shown, the parallel feature pyramid module f PCF The specific calculation steps are as follows: 1) Given the enhanced feature sequence F crat The input is fed into a Transformer with a scaling factor of 1 to further enhance the feature sequence, and the output features are... 2) Given features Input to a Transformer with a scaling factor of 2, output features The dimensions are 384×256; 3) Given features and Input cross-attention module, output The dimensions are 768×256; 4) Given features and Input to cross-attention module f cr Output features Then input it into a Transformer with a scaling factor of 4, and output the features. The dimensions are 192×256; 5) Given features and Input to cross-attention module f cr Output The dimensions are 768×256; 6) Given features and Input to cross-attention module f cr Output features Then input it into a Transformer with a scaling factor of 8, and output the features. The dimensions are 96×256; 7) Given and Input to cross-attention module f cr Output The dimensions are 768×256; 8) Given and Input to cross-attention module f cr Output features Then input it into a Transformer with a scaling factor of 16, and output the features. The dimensions are 48×256; 9) Given and Input to cross-attention module f cr Output The size is 768×256; 10) The obtained feature sequence set As a parallel feature pyramid, the feature pyramid network f is input sequentially. fpn We obtain feature sequence pyramids P0, P1, P2, P3, and P4 at five scales.

[0089] In some specific implementation scenarios, the cross-attention module f cr This can be achieved using a bilinear interpolation module combined with a self-attention model. The specific steps are: 1) Give the input P in sequence. pl and P in ;2) Input P pl 1) Input the self-attention model's mapping module, mapping it to the query vector; 2) Input P in1) Input the mapping module of the self-attention model and map it into a key vector and a numerical vector; 2) Input the key vector and the numerical vector, and use bilinear interpolation to make the output scale consistent with the query vector; 3) Given the query vector and the interpolated key vector and numerical vector, input the self-attention function to obtain the feature output;

[0090] In some specific implementation scenarios, the feature pyramid network f fpn You can use an FPN network or a PAN network.

[0091] Step 7, construct the temporal feature decoder f LOC Decoder classification head f cls and decoder regression head f reg Given a feature sequence pyramid P0, P1, P2, P3, P4, output the time coordinates (s) of the forged fragment. t ,d t ) and the corresponding forgery probability p t ;

[0092] In some specific implementation scenarios, the temporal feature decoder f LOC Five classification heads f are used respectively cls and the corresponding regression head f reg constitute;

[0093] In some specific implementations, both the classification head and the regression head consist of three modules. Each module comprises a one-dimensional convolution, layer normalization, and an activation function in sequence. The output of the classification head is fed into a one-dimensional convolution with two output channels, and then softmax is used to give the probability score p. t The regression head output is fed into two different one-dimensional convolutions, and the output starts at time s. t and the corresponding duration d t ;

[0094] In some specific implementation scenarios, the results of the five classification heads and the regression heads are combined together to form the final result;

[0095] In some specific implementation scenarios, the final loss function of the network constructed in steps 1 to 7 is shown in the following formula:

[0096]

[0097] in The cross-entropy loss function is used for binary classification. To reconstruct the loss function, L1 distance can be used for calculation. For Focal loss, The GIoU loss function can be replaced by similar loss functions such as CIoU and EIoU loss in some specific implementation scenarios.

[0098] Step 8: Test multimedia content on the test set using a pre-trained behavior recognition network f. B and pre-trained audio feature representation network f A Extract the feature sequence and input it into the trained model f. C Post-processing is performed using nonmaximum suppression, and a specified number of results are selected as the prediction results for the forged fragments.

[0099] In some specific implementations, a threshold can be used to determine the specific quantity. 0.5 can be used as the true / false probability threshold, and prediction segments below 0.5 are ignored.

[0100] Example

[0101] This embodiment includes the following steps:

[0102] S1: Collect training samples;

[0103] S1.1: Input video, extract video frames and calculate optical flow frames;

[0104] S1.2: Input audio, extract Mel spectrum;

[0105] S1.2: Using a TSN network with a stride of 1, extract the visual modal feature sequences F of video frames and optical flow frames. b The feature size was upsampled to 768×4096;

[0106] S1.3: Using the BYOL-A network, extract the Mel-spectral audio modal feature sequence F. a Feature upsampling: Feature size upsampled to 768×2048

[0107] S1.4: Feature sequence F of spliced ​​video b and audio feature sequence F a The resulting size is 768×6144;

[0108] S1.5: Using the feature space mapping module f Y Map the features to F m The dimensions are 768×256;

[0109] S1.6: The forgery time period corresponding to each feature sequence label (s) g ,d g The label also includes whether the content contains fake fragments; 1 indicates the content contains fake fragments, and 0 indicates the content does not contain fake fragments.

[0110] S2: Constructing the reconstructed feature sequence

[0111] S2.1: Input feature sequence, use feature reconstruction encoder fE Obtain the implicit feature Z with a feature size of 24×256;

[0112] S2.2: Using a simple binary classifier f S The probability p that implicit Z belongs to content containing forged fragments. a The binary classification cross-entropy loss is calculated using the true label;

[0113] S2.3: Using features to reconstruct the decoder f D Decoding implicit features into reconstructed feature sequences The resulting dimensions are 768×2048;

[0114] S2.4: Reconstruct the feature sequence The feature sequences belonging to real samples With the original feature sequence F m The feature sequence F belonging to the real sample mr Calculate reconstruction losses;

[0115] S3: Construct a reconstruction feature error attention enhancement module f R Obtain the reconstruction error enhancement feature F crat ;

[0116] S3.1: Obtain the positional encoding and compare it with the original feature sequence F. m and reconstructed feature sequences Add them together to obtain feature F. pe and

[0117] S3.2: Construct two mapping modules consisting of layer normalization and one-dimensional convolution to map the features F. pe and Mapped to F respectively q and F k All dimensions are 768×256;

[0118] S3.3: Calculate F q and F k Correlation matrix F r The dimensions are 768×768.

[0119] S3.4: For the correlation matrix F r Each row of elements undergoes a Softmax operation to obtain the attention matrix F. cra ;

[0120] S3.5: Use layer normalization and one-dimensional convolution to transform F pe Mapped to F v ;

[0121] S3.6: Use F cra With Fv Multiplication yields the reconstructed attention feature F ea ;

[0122] S3.7: F ea With F pe The summation is then input into a feedforward neural network to obtain the reconstruction error enhancement feature F. crat

[0123] S4: Constructing a parallel feature pyramid module f PCF Obtain the multi-scale feature sequence pyramids P0, P1, P2, P3, and P4.

[0124] like Figure 3 As shown, the specific steps for obtaining a multi-scale feature pyramid are as follows:

[0125] S4.1: Input feature F crat Using a Transformer with a scaling factor of 1, features are obtained. The dimensions are 768×256;

[0126] S4.2: Input Features Features are obtained using a Transformer with a scaling factor of 2. The dimensions are 384×256.

[0127] S4.3: Input Features and Using the cross-attention module, obtain The dimensions are 768×256;

[0128] S4.4: Input Features and Use the cross-attention module to obtain scale cross features. The dimensions are 768×256;

[0129] S4.5: Input Features Features are obtained using a Transformer with a scaling factor of 4. The dimensions are 192×256;

[0130] S4.6: Input Features and Using the cross-attention module, obtain The dimensions are 768×256;

[0131] S4.7: Input Features and Use the cross-attention module to obtain scale cross features. The dimensions are 768×256;

[0132] S4.8: Input Features Features were obtained using a Transformer with a scaling factor of 8. The dimensions are 96×256;

[0133] S4.9: Input and Using the cross-attention module, obtain The dimensions are 768×256;

[0134] S4.10: Input and Use the cross-attention module to obtain scale cross features. The dimensions are 768×256;

[0135] S4.11: Input Features Using a Transformer with a scaling factor of 16, the output features are... The dimensions are 48×256;

[0136] S4.12: Input and Using the cross-attention module, the output is... The dimensions are 768×256;

[0137] S4.13: The obtained set of feature sequences As a parallel feature pyramid, the feature pyramid network f is input sequentially. fpn We obtain feature sequence pyramids P0, P1, P2, P3, and P4 at five scales.

[0138] S5: Constructing a temporal feature decoder f LOC Input feature sequence pyramids P0, P1, P2, P3, P4, output fake fragment time coordinates (s) t ,d t ) and the corresponding forgery probability p t ;

[0139] S5.1: Input feature P0, and simultaneously use two decoders consisting of one-dimensional convolution, layer normalization, and activation functions to classify the feature into classification heads f. cls and return head f reg ;

[0140] S5.2: The above classification header f cls and return head f reg Classification header f cls The input is a 1D convolution with 2 output channels, followed by Softmax to output true / false probabilities, with a size of 768×2. Correspondingly, the regression head f... regTwo parallel one-dimensional convolutions are used to predict the start time of the forgery and the duration of the forgery, respectively, both with a size of 768×1.

[0141] S5.3: Input feature P1, and simultaneously use two decoders consisting of one-dimensional convolution, layer normalization, and activation functions to classify the feature into classification heads f. cls and return head f reg ;

[0142] S5.4: The above classification header f cls and return head f reg Classification header f cls The input is a 1D convolution with 2 output channels, followed by Softmax to output true / false probabilities, with a size of 384×2. Correspondingly, the regression head f... reg Two parallel one-dimensional convolutions are used to predict the start time of the forgery and the duration of the forgery, respectively, both with a size of 384×1.

[0143] S5.5: Input features P2, and simultaneously use two decoders consisting of one-dimensional convolutions, layer normalization, and activation functions to classify the features into classification heads f. cls and return head f reg ;

[0144] S5.6: The above classification header f cls and return head f reg Classification header f cls The input is a 1D convolution with 2 output channels, followed by Softmax to output true / false probabilities, with a size of 192×2. Correspondingly, the regression head f... reg Two parallel one-dimensional convolutions are used to predict the start time of the forgery and the duration of the forgery, respectively, both with a size of 192×1.

[0145] S5.7: Input feature P3, and simultaneously use two decoders consisting of one-dimensional convolutions, layer normalization, and activation functions to classify the feature into classification heads f. cls and return head f reg ;

[0146] S5.8: The above classification header f cls and return head f reg Classification header f cls The input is a 1D convolution with 2 output channels, followed by Softmax to output true / false probabilities, with a size of 96×2. Correspondingly, the regression head f... reg Two parallel one-dimensional convolutions are used to predict the start time of the forgery and the duration of the forgery, respectively, both with a size of 96×1.

[0147] S5.8: Input feature P4, and simultaneously use two decoders consisting of one-dimensional convolutions, layer normalization, and activation functions to classify the feature into classification heads f. cls and return head f reg ;

[0148] S5.9: The above classification header f cls and return head f reg Classification header f cls The input is a 1D convolution with 2 output channels, followed by Softmax to output true / false probabilities, with a size of 48×2. Correspondingly, the regression head f... reg Two parallel one-dimensional convolutions are used to predict the start time of the forgery and the duration of the forgery, respectively, both with a size of 48×1.

[0149] S5.10: Concatenate all the above classification results into p t The size is 1488×2. Focal loss is calculated with the true labels, and the results of the regression branches are concatenated to form s. t and d t All values ​​are 1488×1, and the GIoU loss is calculated with the truth labels.

[0150] S6: Obtain the content to be detected, extract video frames, optical flow frames, and Mel-spectrum data, and use a pre-trained behavior recognition network f B and pre-trained audio feature representation network f A The feature sequences are extracted separately and input into the trained model to obtain the predicted fake start segment, segment length, and corresponding true / false probability scores. Redundant prediction results are post-processed using non-maximum suppression.

[0151] This embodiment uses the average precision (AP) at Intersection over Union (IoU) ratios of 0.75 and 0.95, denoted as AP@0.75 and AP@0.95, respectively. AP is the area under the curve formed by precision and recall. An IoU of 0.75 means that a predicted value with an IoU greater than or equal to 0.75 is considered a correctly predicted positive sample, and vice versa. Similarly, an IoU of 0.95 means that a predicted value with an IoU greater than or equal to 0.95 is considered a correctly predicted positive sample, and vice versa. The closer the AP@0.75 and AP@0.95 scores are to 1, the better the model performance. These scores are widely used in time-localization tasks and are representative evaluation metrics.

[0152] Figure 4This example compares the experimental results of the present invention with those of existing methods in a publicly available verification instance. The Lav-DF dataset is used in this embodiment, which contains both audio and video content as well as corresponding manipulated segments. The results show that the method proposed in this invention performs well on the publicly available Lav-DF dataset.

[0153] Figure 5 The following diagrams illustrate the detection results of this invention on different datasets, specifically showing a comparison between the ground truth labels and prediction results of the forged fragments, demonstrating the accuracy of the detection method described in this invention.

[0154] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. For those skilled in the art, the present invention can have various modifications and variations. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A temporal forgery localization method for multimodal deepfake content in audio and video, utilizing a pre-trained behavior recognition network f B Obtain the feature sequence F of visual content b ;Utilizing a pre-trained audio feature representation network f A Obtain the feature sequence F of the audio content a ;Utilizing the feature space mapping module f Y , the feature sequence F b and F a The fusion mapping is a feature sequence F m ; in the characteristic sequence F m Based on this, the encoder f is reconstructed using features. E Obtain the implicit feature Z and use a binary classifier f S Determine the authenticity of the original multimedia content corresponding to implicit features; based on the implicit feature Z, use the feature to reconstruct the decoder f. D Obtain the reconstructed feature sequence In the original feature sequence F m and Based on this, the reconstruction error attention enhancement module f is used. R Improve the feature sequence F m The degree of attention paid to temporal forgery segments is determined by obtaining the feature sequence F enhanced based on reconstruction feature error. crat Introducing a parallel feature pyramid module f PCF Enhanced feature sequence F crat The feature representation capability of small-scale tampered fragments is utilized to obtain feature sequence pyramids P0, P1, P2, P3, and P4; a temporal feature decoder f is constructed. LOC Each includes a decoder classification header f cls and decoder regression head f reg We jointly analyze the feature sequence pyramids P0, P1, P2, P3, and P4 to obtain the trained model f. C To determine the target result of obtaining the start and end times of the forged fragment; The main steps include: Step 1: Obtain training audio and video data that includes both real and fake samples. Real samples are the original content that has not been tampered with, while fake samples are audio and video with some segments tampered with, and include the start and end time labels of the tampered segments. Based on the modality of the data, visual content is input into a pre-trained behavior recognition network f B Extracting feature sequence F b The audio content is input into the pre-trained audio feature representation network f. A Extracting feature sequence F a ; Step 2: Construct the feature space mapping module f Y The extracted visual content feature sequence F is obtained by using convolutional mapping and feature concatenation. b and the feature sequence F of audio content a Mapping to a unified feature space yields the feature sequence F. m ; Step 3: Construct a real sample reconstruction module f based on the reconstruction learning method AE f AE The encoder f is mainly composed of features. E and feature reconstruction decoder f D It is used to learn the latent feature distribution of feature sequences of real samples; Step 4: Construct the reconstruction error attention enhancement module f R Improve the original feature sequence F m The degree of attention paid to the features of forged fragments is used to obtain the enhanced feature sequence F. crat ; Step 5: Construct the parallel feature pyramid module f PCF Enhanced feature sequence F crat To assess the expressive power of features in minutely altered fragments, feature sequence pyramids P0, P1, P2, P3, and P4 were obtained. Step 6: Construct a temporal feature decoder f LOC Each includes a decoder classification header f cls and decoder regression head f reg We jointly analyze the abnormal features in the feature sequence pyramids P0, P1, P2, P3, and P4, and determine the start and end times of the forged segments; thus, we obtain the trained model f. C ; Step 7: Extract the multimedia content to be detected into a feature sequence and input it into the trained model f. C The timestamps of the predicted forged fragments and their corresponding forgery probabilities are obtained. Non-maximum suppression is used to post-process the prediction results, and the start and end times of the forged fragments are given.

2. The method for locating temporal forgeries in multimodal deepfake content of audio and video as described in claim 1, characterized in that... In step 1, the specific method for obtaining the feature sequence is as follows: (1) Decode the given video into video frames and extract the corresponding optical flow frames; (2) Input the video frames and optical flow frames into the pre-trained behavior recognition network f B In the process, extract the feature sequence F. b The dimensions are T×C b T is the length of the feature sequence, C b For feature dimensions; (3) Extract the audio corresponding to the video into a Mel spectrogram; (4) Input the Mel spectrum into the pre-trained audio feature representation network f A In the process, an interpolation method is used to correlate the output size with the video feature sequence F. b Alignment, obtaining audio feature sequence F a Its dimensions are T×C a C a For feature dimensions.

3. The method for locating temporal forgeries in multimodal deepfake content of audio and video as described in claim 1, characterized in that, In step 2, the feature space mapping module f is constructed. Y The specific method is as follows: (1) Concatenate the visual feature sequence F along the feature dimension. b and audio feature sequence F a To obtain the feature sequence F ba The dimensions are T×(C) b +C a ); (2) Feature sequence F after splicing ba The feature sequence mapped in a single one-dimensional convolution layer is F′. ba The feature dimension is changed from C b +C a Convert to a new feature dimension C ba ; (3) Given the feature sequence F′ ba The input is fed into an N-layer module consisting of one-dimensional convolutions and layer normalization, embedding the features into a unified feature space to obtain the feature sequence F. m Its dimensions are T×C m C m For feature dimensions.

4. The method for locating temporal forgeries in multimodal deepfake content of audio and video as described in claim 1, characterized in that, In step 3, the real sample reconstruction module f based on the reconstruction learning method... AE The specific method is as follows: (1) Given the feature sequence F m The input is fed into a feature reconstruction encoder f, which consists of L one-dimensional convolutional layers, instance normalization layers, and activation layers. E Extract low-dimensional implicit features Z, with size Z. C z For feature dimensions; (2) Given the implicit feature Z, input a simple binary classifier f S In the middle, the output probability p a This method determines whether a feature sequence contains forged fragments, and is used to learn the difference in feature distribution between real samples and samples containing forged fragments. It employs a binary classification cross-entropy loss function. Calculate the corresponding loss; (3) Given the implicit feature sequence Z, input it into the feature reconstruction decoder f. D Obtain the reconstructed feature sequence Calculate the reconstructed feature sequence The feature sequences belonging to real samples With the original feature sequence F m The feature sequence F belonging to the real sample mr The distance is used as the reconstruction loss of the real sample.

5. The method for locating temporal forgeries in multimodal deepfake content of audio and video as described in claim 1, characterized in that, In step 4, the reconstruction error attention enhancement module f is constructed. R The specific method is as follows: (1) Given the reconstructed feature sequence and the original feature sequence F m By incorporating positional encoding, new position-sensitive feature sequences are obtained. and F pe ; (2) Given a feature sequence and F pe The feature sequences F are mapped using layer normalization and one-dimensional convolution operations, respectively. k and F q All dimensions are T×D, where D is the mapped feature dimension; (3) Given the feature sequence F k and characteristic sequence F q Transpose, calculate the correlation matrix The cross-attention matrix F is obtained by applying the Softmax operation to each row element of the matrix. cra Its dimensions are T×T; (4) Given the feature sequence F pe Layer normalization and one-dimensional convolution operations are used to map the feature sequence F. v The dimensions are T×D; (5) Cross-attention matrix F cra With characteristic sequence F v Multiply to obtain the reconstruction error attention feature sequence F ea The dimensions are T×D; (6) Given the feature sequence F ea With F pe The features are summed and then input into a feedforward neural network to obtain a feature sequence F based on the reconstruction feature error enhancement. crat .

6. The method for locating temporal forgeries in multimodal deepfake content of audio and video as described in claim 1, characterized in that, In step 5, a parallel feature pyramid module f is constructed. PCF The specific method is as follows: (1) Given the enhanced feature sequence F crat The input is fed into a Transformer with a scaling factor of 1, and the output is a feature sequence. The dimensions are T×D fpn D fpn The feature dimensions of the feature pyramid; (2) Given a feature sequence The input is fed into a Transformer with a scaling factor of 2, and the output is a feature sequence. Size is (3) Given and Input to cross-attention module f cr Output feature sequence The dimensions are T×D fpn ; (4) Given and Input to cross-attention module f cr Output feature sequence The dimensions are T×D fpn Then The input is fed into a Transformer with a scaling factor of 4, and the output is a feature sequence. Size is (5) Given and Input to cross-attention module f cr Output feature sequence The dimensions are T×D fpn ; (6) Given and Input to cross-attention module f cr Output feature sequence The dimensions are T×D fpn Then The input is fed into a Transformer with a scaling factor of 8, and the output is a feature sequence. Size is (7) Given and Input to cross-attention module f cr Output feature sequence The dimensions are T×D fpn ; (8) Given and Input to cross-attention module f cr Output feature sequence The dimensions are T×D fpn Then The input is fed into a Transformer with a scaling factor of 16, and the output is a feature sequence. Size is (9) Given and Input to cross-attention module f cr Output feature sequence The dimensions are T×D fpn ; (10) Given a set of feature sequences As a parallel feature pyramid, it is sequentially input into the feature pyramid network f. fpn We obtain feature sequence pyramids P0, P1, P2, P3, and P4 at five scales.

7. A method for locating temporal forgeries in multimodal deepfake content of audio and video as described in claim 6, characterized in that, The cross-attention module f cr The specific construction method is as follows: (1) Given the input feature order as P pl P in ; (2) Given the input feature sequence P pl By using layer normalization and one-dimensional convolution operations, the features are mapped to the query vector Q. p ; (3) Given the input feature sequence P in By using layer normalization and one-dimensional convolution operations, the features are mapped to key vectors K. p ; (4) Given the input feature sequence P in By using layer normalization and one-dimensional convolution operations, the features are mapped to a numerical vector V. p ; (5) Given the feature Q p K p V p Obtain the feature sequence of the multi-scale cross-attention table: Where g(·) represents the time interpolation function, used to align the time lengths of different features. Used as a scaling factor, Softmax is operated on along the rows of the feature matrix.

8. A method for locating temporal forgeries in multimodal deepfake content of audio and video as described in claim 6, characterized in that, The feature pyramid network f fpn The calculation process is as follows: (1) Given a feature sequence Input a one-dimensional convolution, output a feature sequence Size is Then input a one-dimensional convolution to map the feature sequence to P4; (2) Given a feature sequence Upsampled using bilinear interpolation to the same level as... Consistent, dimensions are Combine it with the feature sequence Add them together to obtain the feature sequence. The input is a one-dimensional convolutional weighted mapping to a feature sequence P3; (3) Given a feature sequence Upsampled using bilinear interpolation to the same level as... Consistent, dimensions are Combine it with the feature sequence Add them together to obtain the feature sequence. The input is a one-dimensional convolutional weighted mapping to a feature sequence P2; (4) Given a feature sequence Upsampled using bilinear interpolation to the same level as... Consistent, dimensions are Combine it with the feature sequence Add them together to obtain the feature sequence. The input is a one-dimensional convolutional weighted mapping to a feature sequence P1; (5) Given a feature sequence Upsampled using bilinear interpolation to the same level as... Consistent, dimensions are T×D fpn Combine it with the feature sequence Add them together to obtain the feature sequence. The input is a one-dimensional convolutional weighted mapping to a feature sequence P0.

9. A method for locating temporal forgeries in multimodal deepfake content of audio and video as described in claim 1, characterized in that, In step 6, the temporal feature decoder f LOC The specific construction method is as follows: (1) The temporal feature decoder f LOC Classification head f by decoder cls and decoder regression head f reg constitute; (2) The classification head f cls It consists of one-dimensional convolution, layer normalization, and activation functions; for the output of the three concatenated modules, Softmax is used to calculate the probability value of a certain segment being real or fake; (3) The regression head f reg It consists of one-dimensional convolution, layer normalization, and activation functions; the output uses two parallel one-dimensional convolutions to output the fake start time and the corresponding duration. (4) The number of classification heads and regression heads is the same as the number of feature maps in the feature sequence pyramid. All classification head results are concatenated along the time dimension to obtain the probability p. t The regression head results are concatenated to form the start time s. t and the corresponding duration d t The probability p is calculated using the Focal loss function. t Loss of truth labels Calculate the start time s using the GIoU loss function. t and duration d t Loss corresponding to truth labels 10. A method for locating temporal forgeries in multimodal deepfake content of audio and video as described in claim 1, characterized in that, After step 6 is completed, the total training loss function can be accumulated as follows: in, The cross-entropy loss function is used for binary classification. For the reconstruction loss of real samples, The Focal loss function is used to calculate the loss between the forgery probability of a fragment and the ground truth label. This is the GIoU loss function.

11. The method for locating temporal forgeries in multimodal deepfake content of audio and video as described in claim 1, characterized in that, In step 7, the pre-trained behavior recognition network f B and pre-trained audio feature representation network f A Extract the feature sequences separately and input them into the trained model f. C Post-processing is performed using nonmaximum suppression, and a specified number of results are selected as the prediction results for the forged fragments.

Images