A video inter-frame detection method and system based on CLIP visual encoder and ViT

Abstract

This invention discloses a video inter-frame detection method and system based on CLIP visual encoder and ViT, belonging to the field of image processing technology. The method includes: uniformly sampling the input video and extracting a frame-level feature vector sequence using CLIP visual encoder; performing multi-scale differential and high-frequency domain analysis on the sequence, fusing the obtained anomaly representations into the original sequence to obtain an enhanced frame-level feature sequence; inputting the enhanced sequence into ViT, performing global and local temporal modeling through multi-head attention and contextual attention, and outputting temporal features; and normalizing the result after dimensionality reduction and integration by two fully connected layers to output the detection result. This invention achieves the detection of AIGC videos while maintaining lightweight design through explicit inter-frame anomaly enhancement and dual-layer attention temporal modeling.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and specifically to a video inter-frame detection method and system based on CLIP visual encoder and ViT. Background Technology

[0002] In an era of rapid iteration in artificial intelligence technology, various video generation models are constantly being updated and iterated, resulting in a massive amount of AI-generated video data emerging from online platforms. These AI-generated videos are highly realistic and generated quickly, and are widely circulated on major online platforms. The proliferation of fake videos poses a serious challenge to network information security and content risk control and supervision. Therefore, the accurate identification of AI-generated videos has significant research significance and social value.

[0003] However, traditional video detection methods primarily focus on the static features of a single frame, analyzing information such as texture, lighting, and artifacts in individual images to determine the authenticity of a video. When faced with highly realistic AI-generated videos, traditional single-frame detection methods fall short. This is because currently generated videos are already highly close to real videos at the single-frame visual level, making them difficult for the human eye and static detection models to distinguish. Consequently, the discrimination effectiveness of traditional detection methods is severely limited when processing highly realistic synthetic videos.

[0004] Currently, AIGC video detection based on temporal information has gradually become a research hotspot in the detection field. Most mainstream temporal detection methods rely on deep learning models to mine the temporal dimension features of videos to determine their authenticity. However, existing research unfortunately has significant shortcomings in the temporal modeling stage, often neglecting the subtle dynamic changes between video frames. Real videos possess natural inter-frame motion smoothness, while generated videos generally exhibit subtle inter-frame abrupt changes, semantic discontinuities, and other temporal anomalies. Existing models lack sensitivity to these fine-grained temporal artifacts, failing to fully mine potential inter-frame anomalies, resulting in insufficient detection accuracy and weak generalization ability, severely impacting the reliability of video detection.

[0005] Furthermore, many current high-performance video authenticity detection methods rely on complex 3D convolutional structures and multi-scale Transformer architectures. While these models demonstrate excellent performance and can accurately identify fake videos, their massive computational demands and significant training resource overhead have become bottlenecks restricting their widespread application. On resource-constrained edge devices, such as small smart terminal devices, or in real-world deployment environments, such as real-time video review scenarios requiring rapid response and limited resources, these large models struggle to operate effectively, significantly limiting the adoption of AIGC video detection technology in practical applications. Summary of the Invention

[0006] To address the aforementioned issues, this invention proposes a video inter-frame detection method and system based on CLIP visual encoder and ViT. The method extracts frame-level features through CLIP visual encoder, performs explicit inter-frame anomaly enhancement using multi-scale inter-frame difference and frequency domain high-frequency analysis, and utilizes ViT combined with multi-head attention and contextual attention for global and local temporal modeling. Finally, the method outputs the data through two fully connected layers, achieving accurate detection of AIGC videos.

[0007] On the one hand, a video inter-frame detection method based on CLIP visual encoder and ViT includes:

[0008] S1. The input video is uniformly sampled to obtain a sequence of sampled video frames. The pre-trained CLIP visual encoder is used to encode the features of each frame in the sampled video frame sequence to obtain the frame-level feature vector corresponding to each frame. All frame-level feature vectors are arranged in chronological order to obtain a sequence of frame-level feature vectors.

[0009] S2, perform multi-scale inter-frame differential processing and frequency domain high-frequency discontinuity analysis on the frame-level feature vector sequence to obtain inter-frame anomaly representations; fuse the inter-frame anomaly representations into the frame-level feature vector sequence to obtain the enhanced frame-level feature sequence;

[0010] S3. The enhanced frame-level feature sequence is input into ViT. The global correlation information between video frames in the sequence is modeled through a multi-head attention mechanism to obtain global temporal features. The context key features and value features are extracted from the global temporal features. The key features and global temporal features are concatenated and the context attention weights are calculated. The value features are weighted and aggregated based on the context attention weights. The weighted and aggregated value features and key features are fused to output the temporal features.

[0011] S4. The temporal features are input into the first fully connected layer for dimensionality reduction mapping to obtain the dimensionality-reduced temporal features. The dimensionality-reduced temporal features are then input into the second fully connected layer for feature integration and dimensionality transformation. After normalization by the Softmax function, the probability values ​​of the input video belonging to real videos and generated videos are obtained. Based on the probability values, the video category is determined and the video detection result is output.

[0012] Furthermore, the pre-trained CLIP visual encoder is used to perform feature encoding on each frame in the sampled video frame sequence to obtain the frame-level feature vector corresponding to each frame, specifically including:

[0013] The current frame image in the sampled video frame sequence is obtained, the current frame image is divided into several image blocks, each image block is linearly mapped to obtain the image block feature vector, and then concatenated into an image feature vector sequence.

[0014] The image feature vector sequence is input into the Transformer encoder, and the spatial correlation information between image patches is extracted through the self-attention mechanism to obtain the optimized image patch feature sequence;

[0015] Global average pooling is performed on the optimized image patch feature sequence to obtain the frame-level feature vector corresponding to the current frame.

[0016] Furthermore, multi-scale inter-frame differencing and high-frequency discontinuity analysis are performed on the frame-level feature vector sequence to obtain inter-frame anomaly representations. The calculation formula is as follows:

[0017] ;

[0018] ;

[0019] ;

[0020] ;

[0021] ;

[0022] ;

[0023] in, Indicates the first The CLIP frame-level feature vector corresponding to the frame; Represent real numbers; Indicates the feature dimension; Presentation layer normalization operation; This represents the frame-level feature vector of frame t after layer normalization; Representing time scale The next first-order inter-frame difference feature; Representing time scale Second-order residual difference characteristics; This indicates the interactive mutation feature between adjacent differences; This represents an element-wise multiplication operation; Indicates feature concatenation operation; and Representing time scales respectively The learnable weight matrix and bias terms are as follows. Represents a nonlinear activation function; This indicates an anomaly representation between frames.

[0024] Furthermore, the inter-frame anomaly representations are fused into the frame-level feature vector sequence to obtain the enhanced frame-level feature sequence, calculated as follows:

[0025] ;

[0026] ;

[0027] ;

[0028] ;

[0029] ;

[0030] in, These are learnable parameters; Used for adaptive screening of high-frequency discontinuous components; Indicates the activation function; Represents the inverse discrete cosine transform; This indicates the total number of video frames after sampling; Represents the first in the frequency domain Each frequency component number; This represents a learnable high-frequency filtering threshold parameter used to determine the starting filtering position of high-frequency discontinuous components in the frequency domain; This represents the frequency domain characteristics corresponding to the k-th frequency component; This represents the frame-level feature vector of frame t after layer normalization. The bias vector representing the gated branch; The bias vector representing the branch generated by the enhancement term; The learnable weight matrix representing the gated branches; The learnable weight matrix represents the branch generated by the enhancement term; Indicates the first The frequency domain consistency anomaly representation corresponding to the frame; Represents multi-scale residual frame difference features; This indicates adaptive enhancement gating weights; This represents the enhancement term generated jointly by frame difference features and frequency domain anomaly features; This represents the enhanced frame-level feature vector; This represents an element-wise multiplication operation;

[0031] An enhanced frame-level feature sequence is constructed based on the enhanced frame-level feature vector.

[0032] Furthermore, the step of modeling the global correlation information between video frames in the sequence through a multi-head attention mechanism to obtain global temporal features specifically includes:

[0033] A linear transformation is performed on each frame-level feature vector in the enhanced frame-level feature sequence to obtain the query vector, key vector, and value vector.

[0034] The dot product of the query vector and each key vector is calculated, and after scaling and Softmax normalization, the attention weights between frames are obtained.

[0035] The global temporal features corresponding to each frame are obtained by weighted summation of the value vectors based on the attention weights between each frame.

[0036] Furthermore, the dimensionality-reduced temporal features are input into the second fully connected layer for feature integration and dimensionality transformation, calculated as follows:

[0037] ;

[0038] ;

[0039] in, Indicates temporal characteristics; This represents the first weight matrix; This represents the first bias vector; Indicates the activation function; Represents the temporal features after dimensionality reduction; This represents the second weight matrix; Indicates the second bias vector; This represents the feature vector output by the second fully connected layer.

[0040] On the other hand, a video inter-frame detection system based on CLIP visual encoder and ViT includes:

[0041] The sampling module is used to uniformly sample the input video to obtain a sampled video frame sequence. The pre-trained CLIP visual encoder is used to encode the features of each frame in the sampled video frame sequence to obtain the frame-level feature vector corresponding to each frame. All frame-level feature vectors are arranged in chronological order to obtain a frame-level feature vector sequence.

[0042] The enhancement module is used to perform multi-scale inter-frame differential processing and frequency domain high-frequency discontinuity analysis on the frame-level feature vector sequence to obtain inter-frame anomaly representations; the inter-frame anomaly representations are then fused into the frame-level feature vector sequence to obtain the enhanced frame-level feature sequence.

[0043] The weighted aggregation module is used to input the enhanced frame-level feature sequence into ViT, and to model the global correlation information between video frames in the sequence through a multi-head attention mechanism to obtain global temporal features. The context key features and value features are extracted from the global temporal features. The key features and global temporal features are concatenated and the context attention weights are calculated. The value features are weighted and aggregated based on the context attention weights. The weighted aggregated value features and key features are fused to output the temporal features.

[0044] The detection module is used to input temporal features into the first fully connected layer for dimensionality reduction mapping to obtain dimensionality-reduced temporal features; input the dimensionality-reduced temporal features into the second fully connected layer for feature integration and dimensionality transformation, and normalize them using the Softmax function to obtain the probability values ​​of the input video belonging to real videos and generated videos. Based on the probability values, the video category is determined and the video detection result is output.

[0045] The present invention adopts the above technical solution and has the following beneficial effects:

[0046] (1) Before CLIP frame-level features enter ViT temporal modeling, this invention constructs first-order inter-frame difference, second-order residual difference and interactive mutation features on multiple time scales, and combines frequency domain high-frequency discontinuity component analysis to explicitly model video frame subtle mutations, local flicker, motion drift and semantic jumps, and mine fine-grained temporal anomalies in AIGC videos that are difficult to be captured by single-frame features.

[0047] (2) This invention uses a pre-trained CLIP visual encoder to extract frame-level features and combines ViT with contextual attention mechanism to complete temporal modeling. By embedding the contextual attention mechanism into the second and fourth layers of the ViT network, bidirectional collaborative perception of low-level texture anomalies and high-level semantic anomalies is achieved. While ensuring detection accuracy, the computational complexity and memory consumption are significantly reduced, and it can be effectively deployed on resource-constrained edge devices.

[0048] (3) The present invention uses two complementary fully connected layers to form an MLP classification head. The first layer is responsible for dimensionality reduction mapping and screening of key temporal features with discriminative power. The second layer is responsible for feature integration and dimensionality transformation. The model is trained in conjunction with the cross-entropy loss function, which can effectively improve the classification accuracy of real videos and AIGC videos. It can be widely used in short video platform content review, fake and forged video identification, network risk control supervision and other scenarios. Attached Figure Description

[0049] Figure 1 This is a schematic diagram of the video inter-frame detection process based on CLIP visual encoder and ViT according to an embodiment of the present invention;

[0050] Figure 2 This is a schematic diagram of the video inter-frame detection method based on CLIP visual encoder and ViT according to an embodiment of the present invention.

[0051] Figure 3 This is a structural diagram of the CLIP visual encoder ViT according to an embodiment of the present invention;

[0052] Figure 4 This is a diagram showing the internal structure of the CoT context attention unit in an embodiment of the present invention.

[0053] Figure 5This is a structural diagram of the adaptive enhancement algorithm according to an embodiment of the present invention;

[0054] Figure 6 This is a diagram of a video inter-frame detection system based on CLIP visual encoder and ViT according to an embodiment of the present invention. Detailed Implementation

[0055] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0056] like Figure 1 As shown, the present invention provides a video inter-frame detection method based on CLIP visual encoder and ViT, comprising:

[0057] S1. The input video is uniformly sampled to obtain a sequence of sampled video frames. The pre-trained CLIP visual encoder is used to encode the features of each frame in the sampled video frame sequence to obtain the frame-level feature vector corresponding to each frame. All frame-level feature vectors are arranged in chronological order to obtain a sequence of frame-level feature vectors.

[0058] Specifically, the pre-trained CLIP visual encoder is used to encode features for each frame in the sampled video frame sequence, obtaining a frame-level feature vector for each frame, including:

[0059] The current frame image in the sampled video frame sequence is obtained, the current frame image is divided into several image blocks, each image block is linearly mapped to obtain the image block feature vector, and then concatenated into an image feature vector sequence.

[0060] The image feature vector sequence is input into the Transformer encoder, and the spatial correlation information between image patches is extracted through the self-attention mechanism to obtain the optimized image patch feature sequence;

[0061] Global average pooling is performed on the optimized image patch feature sequence to obtain the frame-level feature vector corresponding to the current frame.

[0062] like Figure 2The diagram illustrates the principle of a video inter-frame detection method based on CLIP visual encoder and ViT, comprising five consecutive processing stages: video frame sampling, CLIP frame-level encoding, inter-frame anomaly enhancement, ViT temporal modeling, and MLP classification output. For the input real or generated video, several video frames are first extracted chronologically, and each frame is converted into frame-level semantic features of a unified dimension using the CLIP visual encoder. Subsequently, a multi-scale residual frame difference-frequency domain consistency adaptive enhancer jointly analyzes the first-order frame difference, second-order residual, multi-scale difference, and high-frequency consistency anomalies in the frame-level feature sequence, generating enhanced features that highlight inter-frame abrupt changes, local flicker, and semantic drift. The enhanced frame-level feature sequence is then fed into the ViT temporal modeling module, where the MHSA Block and CoT Block alternately extract global temporal correlations and contextual detail changes. Finally, the MLP Head performs dimensionality reduction, integration, and normalization classification on the temporal features, outputting the detection result indicating whether the input video belongs to a real or generated video.

[0063] S2 performs multi-scale inter-frame differential processing and frequency domain high-frequency discontinuity analysis on the frame-level feature vector sequence to obtain inter-frame anomaly representations; the inter-frame anomaly representations are then fused into the frame-level feature vector sequence to obtain the enhanced frame-level feature sequence.

[0064] Specifically, multi-scale inter-frame differencing and high-frequency discontinuity analysis are performed on the frame-level feature vector sequence to obtain inter-frame anomaly representations. The calculation formula is as follows:

[0065] ;

[0066] ;

[0067] ;

[0068] ;

[0069] ;

[0070] ;

[0071] in, Indicates the first The CLIP frame-level feature vector corresponding to the frame; Represent real numbers; Indicates feature dimension; Presentation layer normalization operation; This represents the frame-level feature vector of frame t after layer normalization; Representing time scale The next first-order inter-frame difference feature; Representing time scale Second-order residual difference characteristics; This indicates the interactive mutation feature between adjacent differences; This represents an element-wise multiplication operation; Indicates feature concatenation operation; and Representing time scales The learnable weight matrix and bias terms below, Represents a non-linear activation function; This indicates an anomaly representation between frames.

[0072] Specifically, in S2, the inter-frame anomaly representations are fused into the frame-level feature vector sequence to obtain the enhanced frame-level feature sequence. The calculation formula is as follows:

[0073] ;

[0074] ;

[0075] ;

[0076] ;

[0077] ;

[0078] in, These are learnable parameters; Used for adaptive screening of high-frequency discontinuous components; Indicates the activation function; Represents the inverse discrete cosine transform; This indicates the total number of video frames after sampling; Represents the first in the frequency domain Each frequency component number; This represents a learnable high-frequency filtering threshold parameter used to determine the starting filtering position of high-frequency discontinuous components in the frequency domain; This represents the frequency domain characteristics corresponding to the k-th frequency component; This represents the frame-level feature vector of frame t after layer normalization. The bias vector representing the gated branch; The bias vector representing the branch generated by the enhancement term; The learnable weight matrix representing the gated branches; The learnable weight matrix represents the branch generated by the enhancement term; Indicates the first The frequency domain consistency anomaly representation corresponding to the frame; Represents multi-scale residual frame difference features; This indicates adaptive enhancement gating weights; This represents the enhancement term generated jointly by frame difference features and frequency domain anomaly features; This represents the enhanced frame-level feature vector; This represents an element-wise multiplication operation;

[0079] An enhanced frame-level feature sequence is constructed based on the enhanced frame-level feature vector.

[0080] Specifically, in this embodiment, before the frame-level feature vector sequence is input into ViT, a multi-scale residual frame difference-frequency domain consistency adaptive enhancement algorithm is introduced to perform first-order frame difference, second-order residual difference, multi-scale time interval difference, and frequency domain high-frequency consistency analysis on the frame-level feature vector sequence to obtain the enhanced frame-level feature sequence. The enhanced frame-level feature sequence is then input into ViT, and preliminary modeling is performed through a multi-head self-attention (MHSA) mechanism to capture global correlation information between video frames. A contextual transformer block (CoT) mechanism is also introduced to enhance the model's sensitivity to dynamic changes between frames, thereby effectively identifying subtle abnormal changes in the video.

[0081] Specifically, Figure 3 The diagram shows the ViT structure of the CLIP visual encoder according to an embodiment of the present invention. The ViT structure in the CLIP visual encoder is used to convert a single-frame video image into a frame-level feature vector. Specifically, the input image is first divided into multiple image blocks of uniform size. Each image block is linearly projected to form an image block embedding, which, together with positional encoding and CLS markers used to aggregate global information, forms an image feature sequence. This sequence is then input into a multi-layer Transformer encoder, where a self-attention mechanism models the spatial relationships between different image blocks, enabling the model to simultaneously perceive local texture, target contours, and overall semantic information. After encoding, the image feature vector corresponding to the current video frame is obtained based on the CLS markers or the pooled global representation, serving as the basic input for subsequent inter-frame difference analysis and temporal modeling.

[0082] Specifically, the multi-scale residual frame difference-frequency domain consistency adaptive enhancement algorithm is executed before ViT global temporal modeling. First, the frame-level features output by CLIP are layer-normalized. Then, first-order inter-frame differences, second-order residual differences, and interactive mutation features are constructed separately. Adaptive aggregation is then performed at multiple time scales using learnable weights to obtain multi-scale residual frame difference aggregated features. The multi-scale residual frame difference-frequency domain consistency adaptive enhancement algorithm also performs discrete cosine transform on the frame-level features along the time dimension. A learnable high-frequency selection function is used to extract high-frequency anomalous components related to local flicker, periodic jitter, and temporal discontinuities. Finally, a gated residual fusion mechanism is used to fuse the frame difference anomalous features and frequency domain anomalous features back into the original frame-level semantic features, generating an enhanced frame-level feature sequence.

[0083] Specifically, in this embodiment, the CLIP visual encoder uses pre-trained weights to complete video frame feature encoding, performs block mapping, encoding optimization, and pooling compression on the video frames, and generates frame-level feature vectors with regular dimensions and rich semantics. The CLIP visual encoder performs frame-level feature encoding on a single video frame as follows:

[0084] The input complete video frame image is equally divided into several non-overlapping image blocks of the same number and size. A linear mapping process is performed on each independent image block to generate a corresponding image block feature vector. All image block feature vectors are then concatenated to form a complete image feature vector sequence. The expression for the above mapping process is as follows:

[0085] ;

[0086] in Represents a sequence of feature vectors. Indicates the video frame number; Indicates the first The first video frame is divided into the... Image blocks; represents the feature vectors of the 1st, 2nd to nth image blocks obtained from the division of the t-th frame, respectively; n represents the number of image blocks obtained from the division of the current video frame.

[0087] Secondly, the generated image feature vector sequence is input into the Transformer encoder. Relying on the self-attention mechanism, spatial correlation information between image patches is mined, and feature optimization iterations are completed to obtain the optimized image patch feature sequence. The above encoding process is expressed as follows:

[0088] ;

[0089] Finally, a global average pooling operation is performed on the optimized feature sequence to compress the feature dimension and retain the core semantic information, ultimately generating a frame-level feature vector that can represent the content of a single video frame; the expression for the above pooling process is as follows:

[0090] ;

[0091] ;

[0092] in, This represents a sequence of frame-level features arranged in chronological order. This represents the feature mapping function of the Transformer encoder; This represents the optimized image patch feature sequence obtained after processing by the Transformer encoder. (·) represents a pooling operation, used to perform global aggregation on optimized image patch feature sequences; This represents the frame-level feature vector corresponding to frame t; This represents a sequence of frame-level features arranged in chronological order. These represent the frame-level feature vectors corresponding to the 1st frame, the 2nd frame, and the Nth frame, respectively; N represents the number of video frames after sampling; and the comma indicates that the frame-level feature vectors are arranged in chronological order.

[0093] Specifically, the Context Attention (CoT) mechanism is used to mine local detail features within video frames and dynamic change features between frames, as follows:

[0094] First, for the high-dimensional feature vector input to the network, convolution and linear transformation are used in parallel to extract the context key and value features within the feature vector. Convolution operations are used to capture local spatial information of the image, accurately representing the local detail features of different regions within the video frame. The expression for this process is as follows:

[0095] ;

[0096] in The input feature vector; Convolution operation for extracting key features; V represents the linear transformation for extracting value features; V represents the value features extracted from the input features; K represents the context key features extracted from the input features.

[0097] Secondly, the extracted key features are concatenated and fused with the original input features. Then, feature transformation is performed through multi-layer convolution operations combined with a non-linear activation function to calculate and generate exclusive context attention weights. The expression for this process is as follows:

[0098] ;

[0099] in Indicates the context attention weights; It represents a series of convolution operations and non-linear activations.

[0100] Finally, the value features are weighted and aggregated element-wise based on the generated attention weights. The weighted aggregated value features are then fused with the original key features to generate the final output features of the context attention mechanism. The expression for this process is as follows:

[0101] ;

[0102] in This indicates a weighted operation, that is, a feature of aggregated values ​​weighted by elements. .

[0103] Preferably, the MLP classification head consists of two stacked fully connected layers with different structures and complementary functions, used to complete the dimensionality mapping of high-dimensional temporal features, feature selection, and video authenticity classification, as follows:

[0104] First, the high-dimensional temporal features output by the ViT network are input into the first fully connected layer. Using a pre-defined weight matrix and bias vector, feature dimensionality reduction and feature mapping are performed to select key temporal features with strong discriminative power. This process is expressed as follows:

[0105] ;

[0106] in This represents the timing characteristics of the ViT output; This represents the weight matrix of the first layer; represents the bias vector of the first layer; ReLU(·) represents the activation function used to introduce nonlinear mapping; f1 represents the intermediate temporal features obtained after dimensionality reduction mapping of the first fully connected layer.

[0107] Secondly, the intermediate features output from the first fully connected layer are input into the second fully connected layer to further complete feature integration and dimensionality transformation, adapting to the requirements of subsequent classification tasks. This process is expressed as:

[0108] ;

[0109] in This represents the weight matrix of the second layer; f1 represents the bias vector of the second layer; f2 represents the feature vector output by the fully connected second layer.

[0110] Specifically, in this embodiment, the frame difference features are concatenated and then nonlinearly mapped to obtain candidate anomaly features at each time scale:

[0111] ;

[0112] in, This indicates a feature concatenation operation. and They represent the first Learnable weight matrices and bias terms at various time scales Represents a non-linear activation function; Representing time scale The next first-order inter-frame difference feature; Representing time scale Second-order residual difference characteristics; This indicates the interactive mutation feature between adjacent differences; and Representing time scales The learnable weight matrix and bias terms below, Represents a non-linear activation function; This indicates an anomaly representation between frames.

[0113] Further calculation of scale-adaptive weights:

[0114] ;

[0115] in, This represents a learnable scale rating vector. Indicates the first Frames in time scale Differential contribution weights;

[0116] And the multi-scale residual frame difference aggregation features are obtained:

[0117] ;

[0118] in, It can take into account both short-cycle local mutations and long-cycle semantic drift, avoiding the problem of insufficient adaptation of a single time scale to videos with different motion speeds.

[0119] Specifically, to enhance the model's ability to perceive local flicker, periodic jitter, and high-frequency temporal artifacts in AIGC videos, a discrete cosine transform is performed on the normalized frame-level features along the time dimension:

[0120] ;

[0121] in, Indicates the first The frequency domain characteristics corresponding to each frequency component; t represents the video frame number.

[0122] And construct a learnable high-frequency selection function:

[0123] ;

[0124] in, Used to control the slope of the frequency selection function. Used to control the high-frequency screening threshold. This is used for adaptive screening of high-frequency discontinuous components. Subsequently, an inverse discrete cosine transform is performed on the screened frequency domain features to obtain frequency domain consistency anomaly features:

[0125] ;

[0126] in, Represents the inverse discrete cosine transform. Indicates the first The frequency domain consistency anomaly representation corresponding to the frame.

[0127] Specifically, in this embodiment, the model training phase uses the cross-entropy loss function to optimize model parameters, which can accurately measure the deviation between the model's predicted labels and the true labeled labels, and guide the network to update weight parameters.

[0128] Specifically, Figure 5 The adaptive enhancement algorithm structure diagram is as follows: The adaptive enhancement algorithm takes the frame-level feature sequence output by CLIP as input. First, it performs layer normalization on the features of each frame to reduce the impact of feature scale differences between different video frames on anomaly modeling. In the temporal branch, the algorithm constructs first-order inter-frame difference, second-order residual difference, and interactive mutation features according to different time scales. Multi-scale residual frame difference features are obtained through nonlinear mapping and adaptive aggregation to characterize short-period mutations and long-period semantic drift. In the frequency domain branch, the algorithm performs DCT transformation along the time dimension and uses a learnable high-frequency selection function to filter high-frequency components related to local flicker, periodic jitter, and inter-frame discontinuities. Then, frequency domain anomaly features are obtained through inverse IDCT transformation. Finally, the gated residual fusion module generates gate weights and enhancement vectors based on the frame difference anomaly features, frequency domain anomaly features, and original semantic preservation features. The enhancement terms are fused back into the original frame-level features in residual form to obtain the enhanced frame-level feature sequence, which is then fed into the ViT and CoT modules for temporal modeling.

[0129] S3. The enhanced frame-level feature sequence is input into ViT. The global correlation information between video frames in the sequence is modeled through a multi-head attention mechanism to obtain global temporal features. The context key features and value features are extracted from the global temporal features. The key features and global temporal features are concatenated and the context attention weights are calculated. The value features are weighted and aggregated based on the context attention weights. The weighted and aggregated value features and key features are fused to output the temporal features.

[0130] Specifically, the step of modeling the global correlation information between video frames in the sequence through a multi-head attention mechanism to obtain global temporal features includes:

[0131] A linear transformation is performed on each frame-level feature vector in the enhanced frame-level feature sequence to obtain the query vector, key vector, and value vector.

[0132] The dot product of the query vector and each key vector is calculated, and after scaling and Softmax normalization, the attention weights between frames are obtained.

[0133] The global temporal features corresponding to each frame are obtained by weighted summation of the value vectors based on the attention weights between each frame.

[0134] S4. The temporal features are input into the first fully connected layer for dimensionality reduction mapping to obtain the dimensionality-reduced temporal features. The dimensionality-reduced temporal features are then input into the second fully connected layer for feature integration and dimensionality transformation. After normalization by the Softmax function, the probability values ​​of the input video belonging to real videos and generated videos are obtained. Based on the probability values, the video category is determined and the video detection result is output.

[0135] Specifically, the dimensionality-reduced temporal features are input into the second fully connected layer for feature integration and dimensionality transformation. The calculation formula is as follows:

[0136] ;

[0137] ;

[0138] in, Indicates temporal characteristics; This represents the first weight matrix; This represents the first bias vector; Indicates the activation function; Represents the temporal features after dimensionality reduction; This represents the second weight matrix; Indicates the second bias vector; This represents the feature vector output by the second fully connected layer.

[0139] Specifically, in this embodiment, the context attention mechanism CoT is embedded in the second and fourth layers of the ViT network, simultaneously introducing context modeling capabilities into both low-level and high-level features. Low-level features focus on capturing shallow detail anomalies such as video texture and contours, while high-level features focus on mining cross-frame semantic and global logical inconsistency anomalies. This bidirectional hierarchical combination further enhances the model's temporal modeling capabilities. Figure 4As shown, the CoT context attention unit adopts a dual-branch parallel structure, extracting local context information through the convolutional branch and mapping feature parameters through the linear branch. Attention calculation is completed through feature concatenation, convolutional weighting, and feature fusion. This can enhance the model's ability to capture subtle inter-frame changes in video without significantly increasing the computational load.

[0140] Specifically, the hyperparameters for the model training phase in this embodiment are configured as follows: the total number of training iterations is set to 10; the batch size of the input samples for a single training iteration is set to 16; the SAM optimizer is used to optimize and update the network parameters; the initial learning rate is set to 1e-4; and the optimizer momentum parameter is set to 0.9 to ensure smooth convergence of the model training process and avoid gradient oscillation problems.

[0141] Specifically, this invention selects the GenVid self-built dataset and the publicly available GVF video forgery dataset as experimental datasets. The GenVid dataset consists of two parts: real video samples and generated video samples. The real samples mainly come from videos on the YouTube platform and the Kinetics-400 dataset, a commonly used dataset in the video domain. The generated samples include video samples generated by 11 video generation models: Stable Video Diffusion, AnimateDiff, DynamicCrafter, I2VGen-XL, Latte, OpenSora, Pika, SEINE, VideoCrafte, ZeroScope, and CogVideoX. GenVid covers multiple generation methods in both the training and testing sets, providing more challenging discrimination scenarios for the detection model. While ensuring the diversity of generation methods, it achieves a reasonable trade-off between data scale and construction controllability, providing stable and effective data support for the training and evaluation of the detection model. The dataset contains a total of 314,335 videos. The GVF dataset consists of 964 video pairs containing both real and generated videos. The experiment involved 10 training epochs; the batch size was set to 16; the SAM optimizer was used during training, with an initial learning rate of 1e-4 and a momentum of 0.9. The ACC, AP, and AUC values ​​were used to evaluate the detection performance of the method presented in this chapter.

[0142] Eight representative methods in this field were selected as comparative methods to verify the effectiveness and advancement of the present invention.

[0143] (1) TALL++: An inter-frame detection method that introduces a thumbnail layout strategy. By arranging video clips in a specific order, it can learn the spatiotemporal inconsistencies in this layout and thus effectively identify AI-generated videos.

[0144] (2) GenConViT: An inter-frame detection method that achieves significant performance improvement in detecting fake videos by learning visual artifacts and latent data distribution.

[0145] (3) I3D: A three-dimensional convolutional neural network that can jointly model the spatiotemporal features of video frames to improve the detection effect of inter-frame abnormal patterns.

[0146] (4) Slow: An inter-frame detection method that captures the inter-frame dynamic features of video through a slow path and is widely used in video action classification, detection and other fields.

[0147] (5) X3D: An inter-frame detection method that uses a lightweight network extension method to significantly reduce computational costs while maintaining the ability to capture temporal features, thereby achieving effective identification of abnormal motion patterns between frames.

[0148] (6) Mvit: Multiscale visual transformer, which can hierarchically model video features at different spatial and temporal scales, thereby effectively capturing video temporal information for classification.

[0149] (7) EVR: A frame detection method based on a visual pre-trained model. By freezing the pre-trained encoder and using a lightweight Transformer decoder to model the temporal relationship between adjacent frames, it can effectively capture abnormal changes between frames.

[0150] (8) DeCoF: A video detection method based on inter-frame consistency, which focuses on capturing temporal artifacts for real and fake detection by eliminating the influence of spatial artifacts during the feature learning process.

[0151] Table 1. Results of the subset crossover experiment;

[0152]

[0153] As shown in Table 1, the experimental results demonstrate that this invention significantly outperforms other traditional detectors overall. The average values ​​of ACC, AP, and AUC reach 89.62%, 96.33%, and 96.41%, respectively, representing improvements of 9.92%, 5.39%, and 5.07% compared to the second-best TALL++. This indicates that in cross-subset detection tasks, this invention can extract more stable and discriminative temporal features. This is mainly due to the powerful feature representation capabilities of the CLIP visual encoder and the enhancement of inter-frame dependencies by CoT, enabling the model to capture fine-grained changes between video frames and improving the detector's generalization ability in cross-subset detection tasks. Looking at the results for each subset, the accuracy of this invention exceeds 90% on all seven generative model subsets, with particularly significant performance improvements on subsets such as PK, SE, and ZS. This indicates that this invention has strong adaptability and robustness across various video generation models. In contrast, while TALL++ and GenConViT achieve better detection results on some subsets, their detection performance on the ZS and CV subsets still lags behind this invention.

[0154] Furthermore, the method presented in this embodiment significantly outperforms other traditional detectors overall, exhibiting excellent average values ​​for ACC, AP, and AUC. It demonstrates good detection performance across various generative model subsets, proving its strong generalization ability and high adaptability. On publicly available datasets, this method still demonstrates superior detection capabilities, showing greater stability and higher detection accuracy compared to existing mainstream video-level detectors in cross-generative model and cross-subset scenarios. Ablation experiments show that the introduction of the CoT attention module effectively enhances the temporal modeling capability of ViT. The combination of embedding the CoT module in a two-layer network yields the best results, with the low-layer and high-layer attention modules working together to improve the model's ability to capture inter-frame anomalies.

[0155] To verify the impact of hyperparameter configuration on model training performance, this embodiment also includes two types of ablation experiments: learning rate and batch size, to analyze the effects of hyperparameters on detection accuracy and convergence. The experimental results show that when the learning rate is set to 1e-4 and the batch size is set to 16, the model converges fastest, the loss curve is smooth and oscillating, and the detection accuracy reaches its peak. This set of hyperparameters is well-suited to the model structure and exhibits optimal training stability.

[0156] like Figure 6 As shown, this embodiment also discloses a video inter-frame detection system based on CLIP visual encoder and ViT, including:

[0157] The sampling module 61 is used to uniformly sample the input video to obtain a sampled video frame sequence. The pre-trained CLIP visual encoder is used to encode the features of each frame in the sampled video frame sequence to obtain the frame-level feature vector corresponding to each frame. All frame-level feature vectors are arranged in chronological order to obtain a frame-level feature vector sequence.

[0158] Enhancement module 62 is used to perform multi-scale inter-frame differential processing and frequency domain high-frequency discontinuity analysis on the frame-level feature vector sequence to obtain inter-frame anomaly representations; and to fuse the inter-frame anomaly representations into the frame-level feature vector sequence to obtain the enhanced frame-level feature sequence.

[0159] The weighted aggregation module 63 is used to input the enhanced frame-level feature sequence into ViT, model the global correlation information between video frames in the sequence through a multi-head attention mechanism, and obtain global temporal features; extract the context key features and value features from the global temporal features, concatenate the key features and global temporal features to calculate the context attention weight, perform weighted aggregation on the value features based on the context attention weight, fuse the weighted aggregated value features and key features, and output the temporal features;

[0160] The detection module 64 is used to input the temporal features into the first fully connected layer for dimensionality reduction mapping to obtain the dimensionality-reduced temporal features; input the dimensionality-reduced temporal features into the second fully connected layer for feature integration and dimensionality transformation, and normalize them through the Softmax function to obtain the probability values ​​of the input video belonging to real videos and generated videos; determine the video category based on the probability values ​​and output the video detection results.

[0161] A specific implementation of a video inter-frame detection system based on CLIP visual encoder and ViT is described in this embodiment, which is the same as the video inter-frame detection method based on CLIP visual encoder and ViT.

[0162] Although the invention has been specifically shown and described in conjunction with preferred embodiments, those skilled in the art should understand that various changes in form and detail may be made to the invention without departing from the spirit and scope of the invention as defined in the appended claims, all of which shall be within the scope of protection of the invention.

Claims

1. A video inter-frame detection method based on CLIP visual encoder and ViT, characterized in that, Includes the following steps: S1. The input video is uniformly sampled to obtain a sequence of sampled video frames. The pre-trained CLIP visual encoder is used to encode the features of each frame in the sampled video frame sequence to obtain the frame-level feature vector corresponding to each frame. All frame-level feature vectors are arranged in chronological order to obtain a sequence of frame-level feature vectors. S2, perform multi-scale inter-frame differential processing and frequency domain high-frequency discontinuity analysis on the frame-level feature vector sequence to obtain inter-frame anomaly representations; fuse the inter-frame anomaly representations into the frame-level feature vector sequence to obtain the enhanced frame-level feature sequence; S3, the enhanced frame-level feature sequence is input into ViT, and the global correlation information between video frames in the sequence is modeled through the multi-head attention mechanism to obtain global temporal features; Extract context key features and value features from global temporal features. Concatenate the key features and global temporal features and calculate context attention weights. Based on the context attention weights, perform weighted aggregation on the value features. Fuse the weighted aggregated value features and key features to output temporal features. S4. The temporal features are input into the first fully connected layer for dimensionality reduction mapping to obtain the dimensionality-reduced temporal features. The dimensionality-reduced temporal features are then input into the second fully connected layer for feature integration and dimensionality transformation. After normalization by the Softmax function, the probability values ​​of the input video belonging to real videos and generated videos are obtained. Based on the probability values, the video category is determined and the video detection result is output.

2. The video inter-frame detection method based on CLIP visual encoder and ViT according to claim 1, characterized in that, In S1, the pre-trained CLIP visual encoder is used to encode the features of each frame in the sampled video frame sequence, obtaining the frame-level feature vector corresponding to each frame, specifically including: The current frame image in the sampled video frame sequence is obtained, the current frame image is divided into several image blocks, each image block is linearly mapped to obtain the image block feature vector, and then concatenated into an image feature vector sequence. The image feature vector sequence is input into the Transformer encoder, and the spatial correlation information between image patches is extracted through the self-attention mechanism to obtain the optimized image patch feature sequence; Global average pooling is performed on the optimized image patch feature sequence to obtain the frame-level feature vector corresponding to the current frame.

3. The video inter-frame detection method based on CLIP visual encoder and ViT according to claim 1, characterized in that, In S2, multi-scale inter-frame differencing and high-frequency discontinuity analysis are performed on the frame-level feature vector sequence to obtain inter-frame anomaly representations. The calculation formula is as follows: ； ； ； ； ； ； in, Indicates the first The CLIP frame-level feature vector corresponding to the frame; Represent real numbers; Indicates the feature dimension; Presentation layer normalization operation; This represents the frame-level feature vector of frame t after layer normalization; Representing time scale The next first-order inter-frame difference feature; Representing time scale Second-order residual difference characteristics; This indicates the interactive mutation feature between adjacent differences; This represents an element-wise multiplication operation; Indicates feature concatenation operation; and Representing time scales respectively The learnable weight matrix and bias terms are as follows. Represents a nonlinear activation function; This indicates an anomaly representation between frames.

4. The video inter-frame detection method based on CLIP visual encoder and ViT according to claim 1, characterized in that, In S2, the inter-frame anomaly representations are fused into the frame-level feature vector sequence to obtain the enhanced frame-level feature sequence, calculated as follows: ； ； ； ； ； in, These are learnable parameters; Used for adaptive screening of high-frequency discontinuous components; Indicates the activation function; Represents the inverse discrete cosine transform; This indicates the total number of video frames after sampling; Represents the first in the frequency domain Each frequency component number; This represents a learnable high-frequency filtering threshold parameter used to determine the starting filtering position of high-frequency discontinuous components in the frequency domain. This represents the frequency domain characteristics corresponding to the k-th frequency component; This represents the frame-level feature vector of the t-th frame after layer normalization. The bias vector representing the gated branch; The bias vector representing the branch generated by the enhancement term; The learnable weight matrix representing the gated branches; The learnable weight matrix represents the branch generated by the enhancement term; Indicates the first The frequency domain consistency anomaly representation corresponding to the frame; Represents multi-scale residual frame difference features; This indicates adaptive enhancement gating weights; This represents the enhancement term generated jointly by frame difference features and frequency domain anomaly features; This represents the enhanced frame-level feature vector; This represents an element-wise multiplication operation; An enhanced frame-level feature sequence is constructed based on the enhanced frame-level feature vector.

5. The video inter-frame detection method based on CLIP visual encoder and ViT according to claim 1, characterized in that, In S3, the step of modeling the global correlation information between video frames in the sequence through a multi-head attention mechanism to obtain global temporal features specifically includes: A linear transformation is performed on each frame-level feature vector in the enhanced frame-level feature sequence to obtain the query vector, key vector, and value vector. The dot product of the query vector and each key vector is calculated, and after scaling and Softmax normalization, the attention weights between frames are obtained. The global temporal features corresponding to each frame are obtained by weighted summation of the value vectors based on the attention weights between each frame.

6. The video inter-frame detection method based on CLIP visual encoder and ViT according to claim 1, characterized in that, In S4, the dimensionality-reduced temporal features are input into the second fully connected layer for feature integration and dimensionality transformation. The calculation formula is as follows: ； ； in, Indicates temporal characteristics; This represents the first weight matrix; This represents the first bias vector; Indicates the activation function; Represents the temporal features after dimensionality reduction; This represents the second weight matrix; Indicates the second bias vector; This represents the feature vector output by the second fully connected layer.

7. A video inter-frame detection system based on CLIP visual encoder and ViT, characterized in that, include: The sampling module is used to uniformly sample the input video to obtain a sampled video frame sequence. The pre-trained CLIP visual encoder is used to encode the features of each frame in the sampled video frame sequence to obtain the frame-level feature vector corresponding to each frame. All frame-level feature vectors are arranged in chronological order to obtain a frame-level feature vector sequence. The enhancement module is used to perform multi-scale inter-frame differential processing and frequency domain high-frequency discontinuity analysis on the frame-level feature vector sequence to obtain inter-frame anomaly representations; the inter-frame anomaly representations are then fused into the frame-level feature vector sequence to obtain the enhanced frame-level feature sequence. The weighted aggregation module is used to input the enhanced frame-level feature sequence into ViT, and to model the global correlation information between video frames in the sequence through a multi-head attention mechanism to obtain global temporal features; Extract context key features and value features from global temporal features. Concatenate the key features and global temporal features and calculate context attention weights. Based on the context attention weights, perform weighted aggregation on the value features. Fuse the weighted aggregated value features and key features to output temporal features. The detection module is used to input temporal features into the first fully connected layer for dimensionality reduction mapping to obtain dimensionality-reduced temporal features; input the dimensionality-reduced temporal features into the second fully connected layer for feature integration and dimensionality transformation, and normalize them using the Softmax function to obtain the probability values ​​of the input video belonging to real videos and generated videos. Based on the probability values, the video category is determined and the video detection result is output.

Images