Face feature cooperation and unsupervised learning based pseudo video detection method and system

Abstract

The present application belongs to the technical field of video detection, and particularly relates to a pseudo-video detection method and system based on face feature cooperation and unsupervised learning. Real face sample loss functions and fake face sample loss functions are constructed according to hard pseudo labels and soft pseudo labels, a dynamic temperature parameter is designed to adjust the similarity distribution between samples through adaptive learning, and independent feature clusters are formed in the learning feature space through sample pairs, so that the data construction cost is effectively reduced without relying on artificial annotation data; the convolution, residual and other feature extraction methods are used to extract multi-scale face features with different fine granularity, capture multi-scale face information, and effectively improve the accuracy of face detection; through optimization of the initial clustering center selection strategy, the inter-frame feature similarity is calculated by combining the Spearman correlation coefficient, and the real and fake judgment is completed by using the inter-frame feature fluctuation difference between real videos and fake videos, so as to further improve the accuracy of detection.

Description

Technical Field

[0001] This invention belongs to the field of video detection technology, specifically relating to a pseudo-video detection method and system based on facial feature collaboration and unsupervised learning. Background Technology

[0002] In recent years, with the rapid development of deep learning technology, deepfake technology has become increasingly sophisticated. The fake videos and images it generates are widely disseminated online, posing significant and complex threats to personal privacy, public safety, and critical security architectures in specific domains. Through advanced techniques such as autoencoders, generative adversarial networks, and diffusion models, deepfake technology can manipulate and tamper with facial images and video content with high realism, making it almost impossible to distinguish the forged content from the real thing with the naked eye. The abuse of this technology not only disrupts social order and misleads public opinion but also fuels crimes such as online fraud and identity theft, severely impacting the social trust system.

[0003] Currently, most mainstream face forgery detection methods are supervised learning schemes, requiring manual annotation of massive amounts of face video samples frame by frame. However, the internet is flooded with massive amounts of video data, and new AI face-swapping forgery samples are constantly evolving. Manual annotation is time-consuming and labor-intensive, and it is difficult to quickly obtain labels for new forgery data, making it unsuitable for unlabeled real-world business scenarios. Existing methods for extracting single-granularity face features can only capture tiny pixel-level tampering edges or identify global facial lighting distortions, which cannot meet the needs of fake video detection. Furthermore, traditional contrastive learning methods only augment the view of the same image, failing to allow real and fake faces to form separate clusters in the feature space. They lack the ability to distinguish between real and fake faces and cannot directly perform binary classification detection. Summary of the Invention

[0004] The purpose of this invention is to provide a fake video detection method and system based on face feature collaboration and unsupervised learning, so as to solve the problems of existing fake video detection relying on manual annotation, single feature extraction, and low accuracy of video authenticity classification and detection.

[0005] To achieve the above objectives, the technical solution of the present invention is as follows: A pseudo-video detection method based on facial feature collaboration and unsupervised learning includes the following steps: S1. Obtain video frame samples containing forged features, detect face regions and crop full-size frames, extract low-level, mid-level and deep-level multi-scale feature details within the face regions and perform preliminary clustering; S2. Based on the relative distance between the samples and the cluster centers, the reliability of the preliminary clustering results is measured, and high-confidence samples, unknown samples, and outlier samples are divided. Fixed hard pseudo-labels are assigned to high-confidence samples, and dynamically adjustable soft pseudo-labels are assigned to unknown samples. Outlier samples are removed to obtain the initial dataset. S3. Perform data augmentation on each sample in the initial dataset, generating multiple differentiated views for a single face sample using different data augmentation methods, and constructing a contrastive learning dataset containing positive and negative sample pairs; sequentially process each sample in the contrastive learning dataset through convolution, two-level residual separable convolution, four-level residual separable convolution, and convolution to gradually obtain the fourth augmented feature map; input the fourth augmented feature map into the parallel convolution branch for feature extraction at different scales to obtain multi-scale features; concatenate and fuse the multi-scale features to obtain multi-scale discriminative features; Two-level residual separable convolution performs one downsampling operation to extract intermediate features; four-level residual separable convolution performs two downsampling operations to extract high-level semantic features. S4. Map the multi-scale discriminative features to the contrastive learning feature space, calculate the cosine similarity between sample pairs, construct the contrastive loss function through adaptive dynamic temperature parameters, and optimize the learning parameters in reverse based on the loss function until the model converges, thus obtaining the forgery feature contrastive learning model. S5. Input the video to be detected into the fake feature comparison learning model to obtain the high-dimensional feature vector of the corresponding video frame. Calculate the feature correlation of adjacent video frames based on the high-dimensional feature vector, and then calculate the average inter-frame correlation coefficient corresponding to each feature cluster. Determine the feature cluster with the average inter-frame correlation coefficient less than the preset threshold as the real face cluster, and determine the feature cluster with the average inter-frame correlation coefficient not less than the preset threshold as the fake face cluster.

[0006] Preferably, the adaptive dynamic temperature parameter back-optimization learning parameters include the following operations: The dynamic temperature parameter is a learnable parameter that is updated and optimized synchronously during the backpropagation iteration process. The dynamic temperature parameter is used to scale the cosine similarity value, smooth the sample similarity distribution within a batch, and adaptively adjust the feature differentiation between similar and dissimilar samples.

[0007] Preferably, the feature correlation calculation of adjacent video frames is as follows: Let the first l Frame feature vector is , No. l +1 frame feature vector is K is the total dimension of the encoder output features. The difference in the ranking of features between two frames is compared dimension by dimension, and the result is calculated using the following formula. l Frame and the l Spearman correlation coefficient between +1 frames : .

[0008] Preferably, the feature clusters are divided into two categories: true and false. Binary clustering is performed on the two types of feature clusters. By optimizing the initial cluster center selection strategy, the high-dimensional features of all frames in a single video are divided into two independent feature clusters.

[0009] Preferably, reliability is measured in the following ways: The reliability metric is to calculate the Euclidean relative distance from a single sample to its cluster center. Samples with a distance less than the first threshold are classified as high-confidence samples and assigned hard pseudo-labels. Samples with a distance between the first and second thresholds are classified as unknown samples and assigned soft pseudo-labels. Samples with a distance greater than the second threshold are identified as outliers and are directly removed.

[0010] Preferably, multi-scale feature fusion includes the following operations: Multi-scale features are concatenated along the channel dimension to obtain a concatenated feature map. A 1×1 convolution is used to compress the channel and exchange information across channels in the concatenated feature map. Then, through batch normalization and activation functions, multi-scale discriminative features with uniform dimensions are output.

[0011] Preferably, the data augmentation process includes: The images in the initial dataset are subjected to cropping, horizontal flipping, rotation, color dithering, random grayscale, random erasure, and double cropping transformation operations.

[0012] A pseudo-video detection system based on facial feature collaboration and unsupervised learning includes the following modules: The clustering module acquires video frame samples containing forged features, detects face regions and crops full-size frames, extracts low-level, mid-level, and deep-level multi-scale feature details within the face regions, and performs preliminary clustering. The reliability measurement module measures the reliability of the preliminary clustering results based on the relative distance between the samples and the cluster centers, and divides them into high-confidence samples, unknown samples, and outlier samples. It assigns fixed hard pseudo-labels to high-confidence samples, assigns dynamically adjustable soft pseudo-labels to unknown samples, and removes outlier samples to obtain the initial dataset. The multi-scale feature extraction module performs data augmentation on each sample in the initial dataset, generating multiple differentiated views for a single face sample through different data augmentation methods, and constructing a contrastive learning dataset containing positive and negative sample pairs. Each sample in the contrastive learning dataset is then sequentially processed through convolution, two-level residual separable convolution, four-level residual separable convolution, and another convolution operation to gradually obtain a fourth augmented feature map. This fourth augmented feature map is then input into a parallel convolution branch for feature extraction at different scales, resulting in multi-scale features. Finally, these multi-scale features are concatenated and fused to obtain multi-scale discriminative features. Two-level residual separable convolution performs one downsampling operation to extract intermediate features; four-level residual separable convolution performs two downsampling operations to extract high-level semantic features. The contrastive learning module maps multi-scale discriminative features to the contrastive learning feature space, calculates the cosine similarity between sample pairs, constructs a contrastive loss function through adaptive dynamic temperature parameters, and optimizes the learning parameters based on the loss function until the model converges, thus obtaining the forgery feature contrastive learning model. The detection output module inputs the video to be detected into the fake feature comparison learning model to obtain the high-dimensional feature vector of the corresponding video frame. Based on the high-dimensional feature vector, it calculates the feature correlation of adjacent video frames, and then calculates the average inter-frame correlation coefficient corresponding to each feature cluster. Feature clusters with an average inter-frame correlation coefficient less than a preset threshold are identified as real face clusters, and feature clusters with an average inter-frame correlation coefficient not less than a preset threshold are identified as fake face clusters.

[0013] Preferably, the comparative learning module includes: The dynamic parameter adjustment module uses a learnable dynamic temperature parameter that is synchronously updated and optimized during backpropagation iteration. This dynamic temperature parameter is used to scale the cosine similarity value, smooth the similarity distribution of samples within a batch, and adaptively adjust the feature differentiation between similar and dissimilar samples.

[0014] Preferably, the detection output module includes: Calculation module, let the first... l Frame feature vector is , No. l +1 frame feature vector is K is the total dimension of the encoder output features. The difference in the ranking of features between two frames is compared dimension by dimension, and the result is calculated using the following formula. l Frame and the l Spearman correlation coefficient between +1 frames : .

[0015] Compared with the prior art, the technical solution provided by this invention has the following advantages: Based on hard pseudo-labels and soft pseudo-labels, loss functions for real face samples and fake face samples are constructed. An adaptive learning dynamic temperature parameter is designed to adjust the similarity distribution between samples. Independent feature clusters are formed in the learning feature space through sample pairs, thereby eliminating the need for manually labeled data and effectively reducing data construction costs. By employing feature extraction methods such as convolution and residual, multi-scale facial features with different fine granularities are extracted to capture multi-scale facial forgery information, thereby effectively improving the accuracy of face detection. By optimizing the initial cluster center selection strategy and combining the Spearman correlation coefficient to calculate the inter-frame feature similarity, the authenticity of real and fake videos is determined by utilizing the differences in inter-frame feature fluctuations between real and fake videos, thereby further improving the detection accuracy. Attached Figure Description

[0016] Figure 1 This is a heatmap showing the focus of the algorithm for face forgery in this invention. Detailed Implementation

[0017] To further understand the content of this invention, the invention will be described in detail with reference to the embodiments. A pseudo-video detection method based on facial feature collaboration and unsupervised learning includes the following steps: S1. Obtain video frame samples containing forged features, detect face regions and crop full-size frames, extract low-level, mid-level and deep-level multi-scale feature details within the face regions and perform preliminary clustering.

[0018] The system acquires corresponding video frame samples from the video to be detected, performs automatic face region localization and full-size face video frame cropping on the video frames, and adopts an adaptive multi-scale regional feature extraction strategy in the face feature extraction stage. When a face is detected to be in an open-eye or open-mouth state, the system prioritizes focusing on the eye and mouth facial feature regions to extract core fine-grained forgery features such as facial feature deformation and edge tampering traces; if the face is identified as being in an closed-eye or closed-mouth state, the system switches to stable face regions such as the chin and nose for feature capture.

[0019] Because forged content may involve multiple regions of the face, such as the eyes, mouth, chin, and nose, extracting features from a single region may not fully capture the traces of forgery. Fusing features from multiple regions can improve detection accuracy. Features from different regions (such as the eyes and mouth) reflect local forgery information, while features from the chin and nose may provide clues about global structure or lighting inconsistencies. Combining these features helps the model to more comprehensively understand the authenticity of the face.

[0020] Therefore, for features extracted from different facial regions, independent statistical counting is performed and weighted average calculation is used to obtain fused features. Then, the unlabeled fused features are automatically pre-clustered using natural physiological features of the face, so as to realize automatic grouping of samples without manual annotation.

[0021] S2. Based on the preliminary clustering results, a reliability measure is performed on the relative distance from the samples to the cluster centers, dividing the samples into high-confidence samples, unknown samples, and outlier samples. Fixed hard pseudo-labels are assigned to high-confidence samples, and dynamically adjustable soft pseudo-labels are assigned to unknown samples. Outlier samples are removed to obtain the initial dataset.

[0022] After obtaining the initial clustering results, these results need to be transformed into reliable pseudo-labels. However, the initial clustering results may contain uncertainties, especially for samples located near the cluster boundaries. Therefore, the pseudo-label assignment stage aims to optimize the initial clustering results and improve the reliability of the labels.

[0023] This invention employs a three-level confidence assessment mechanism, dividing samples into three categories: high confidence, unknown samples, and outliers, and adopting corresponding labeling strategies for different types of samples.

[0024] Preferably, the reliability measurement in S2 is performed in the following way: The reliability metric is to calculate the Euclidean relative distance from a single sample to its cluster center. Samples with a distance less than the first threshold are classified as high-confidence samples and assigned hard pseudo-labels. Samples with a distance between the first and second thresholds are classified as unknown samples and assigned soft pseudo-labels. Samples with a distance greater than the second threshold are identified as outliers and are directly removed.

[0025] To quantify the reliability of each sample classification, a relative distance ratio is introduced. As a confidence metric, it is expressed as: , in, For the sample To the cluster center The weighted Euclidean distance, For the sample To the cluster center The weighted Euclidean distance; the weight W is determined by the importance of the features.

[0026] The smaller the relative distance ratio, the greater the distance between the sample and one cluster center, and the higher the certainty of the classification. Conversely, when the relative distance ratio is close to 1, it means that the sample is located in the boundary region between two clusters, and the uncertainty of the classification increases.

[0027] Based on the relative distance ratio and the distance to the nearest cluster center, the samples are divided into three categories, and different label assignment strategies are used for different types of samples: A high-confidence sample is a sample that meets two conditions simultaneously. First, its relative distance ratio... <0.8; secondly, its minimum distance to its cluster center is less than twice the standard deviation of the distance from the sample to its cluster center (i.e., 2 High-confidence samples are not only very close to a certain cluster center, but also significantly far from the other cluster centers, thus their classification results have high reliability. Based on this, this invention uses the label of the cluster to which such samples belong as their hard pseudo-label, represented as: , For the sample Its hard fake label It makes the distance The index of the smallest cluster center.

[0028] Specifically, a hard pseudo label of 0 indicates a real face, while a hard pseudo label of 1 indicates a fake face.

[0029] This will satisfy 0.8≤ <1.2 and Samples with certain conditions, considered unknown, lie at the boundary between two clusters, and their classification is uncertain. For these samples, this invention employs a probabilistic soft pseudo-labeling method. A soft pseudo-label is essentially a probability distribution that indicates the probability of a sample belonging to each category. The calculation method is as follows: , in, It uses a fixed temperature parameter to control the smoothness of the probability distribution.

[0030] Will satisfy 1.2 or Samples that deviate from the criteria are considered outliers. These samples may be noisy data or anomalous samples, and they are far from both cluster centers. Since outliers have low classification reliability, they are excluded in subsequent training phases to prevent the introduction of noisy data, thereby ensuring the stability and accuracy of the model.

[0031] S3. Perform data augmentation on each sample in the initial dataset, generating multiple differentiated views for a single face sample using different data augmentation methods, and constructing a contrastive learning dataset containing positive and negative sample pairs; sequentially process each sample in the contrastive learning dataset through convolution, two-level residual separable convolution, four-level residual separable convolution, and convolution to gradually obtain the fourth augmented feature map; input the fourth augmented feature map into the parallel convolution branch for feature extraction at different scales to obtain multi-scale features; concatenate and fuse the multi-scale features to obtain multi-scale discriminative features; Two-level residual separable convolution performs one downsampling operation to extract intermediate features; four-level residual separable convolution performs two downsampling operations to extract high-level semantic features.

[0032] Preferably, the data augmentation process includes: The images in the initial dataset are subjected to cropping, horizontal flipping, rotation, color dithering, random grayscale, random erasure, and double cropping transformation operations.

[0033] Multiple differential views are generated from a single face sample through various transformations, and a dataset of positive and negative sample pairs is constructed.

[0034] Specifically, the positive and negative sample pairs dataset contains RGB face images with a pixel size of 128×128 and 3 channels.

[0035] Specifically, perform the following operations sequentially on each positive and negative sample in the dataset: The sample is input into a 3×3 convolutional layer with a stride of 2 to extract basic visual features such as the low-level contour of the image, and outputs the first feature map with a size of 63×63 and 32 output channels.

[0036] The first feature map is processed through two stages of residual separable convolution, progressively expanding the feature channels while simultaneously adjusting the feature map resolution. The first stage of residual separable convolution has a stride of 1 and is responsible for extracting intermediate image features such as texture, expanding the output channel count to 64 while maintaining the same feature map size. The second stage of residual separable convolution has a stride of 2, expanding the channel count to 128 and outputting the second feature map, which is then compressed to 31×31 pixels.

[0037] By using the residual connection cross-layer identity mapping mechanism, shallow features can skip multiple convolution operations and be directly passed to the deeper layers of the network; during backpropagation, the gradient can be propagated back along the identity mapping branch without decay, effectively alleviating gradient vanishing and enhancing feature reuse capability.

[0038] Furthermore, the second feature map is processed through four levels of residual separable convolution, where the first, second, and fourth levels use single-layer residual concatenation, and the third level stacks eight layers of residual network for deep feature extraction, in order to enhance the network's ability to abstractly represent complex face forgery artifacts.

[0039] The first convolutional layer with a stride of 2 downsamples the 31×31 second feature map to 16×16, expanding the channel dimension from 128 to 256. After the second convolutional layer with a stride of 2, the feature map size is compressed to 8×8, and the number of channels is expanded to 512. After the third and fourth layers, the high-level semantic features of complex forgery traces are extracted and the channel dimension is mapped, resulting in a third feature map with a size of 7×7 and 1024 channels.

[0040] The third feature map is input into a 3×3 convolutional layer with a stride of 2 to obtain a fourth feature map with 2048 channels and unchanged size.

[0041] The fourth feature map is simultaneously input into three parallel convolutional branches, and multi-granularity feature extraction is completed by using asynchronous long convolutions.

[0042] Specifically, the high-resolution sub-branch uses a step size of 1, does not perform downsampling, and outputs a first-scale feature with 256 channels and a size of 7×7. The medium-resolution sub-branch uses a step size of 2 to perform moderate downsampling, outputting 256 channels and reducing the size of the features. Then, it is upsampled to restore the size to 7×7 to obtain the second-scale features. The low-resolution sub-branch uses a step size of 4 for significant downsampling, outputting 256 channels and smaller-sized features. These features are then upsampled back to a 7×7 size to obtain the third-scale features. Finally, the first, second, and third-scale features are fused using a multi-scale feature fusion process.

[0043] Preferably, multi-scale feature fusion includes the following operations: Multi-scale features are concatenated along the channel dimension to obtain a concatenated feature map. A 1×1 convolution is used to compress the channel and exchange information across channels in the concatenated feature map. Then, through batch normalization and activation functions, multi-scale discriminative features with uniform dimensions are output.

[0044] Specifically, by splicing along the channel dimension, a spliced ​​feature map of 768×7×7 is obtained. The spliced ​​feature map is then compressed to 512 channels using a 1×1 convolution, and combined with a pooling layer to complete cross-channel information interaction.

[0045] S4. Map the multi-scale discriminative features to the contrastive learning feature space, calculate the cosine similarity between sample pairs, construct a contrastive loss function through adaptive dynamic temperature parameters, and optimize the learning parameters in reverse based on the loss function until the model converges, thus obtaining the forged feature contrastive learning model.

[0046] Multi-scale discriminative features are linearly or non-linearly mapped to a dedicated contrastive learning feature space via a projection head, filtering redundant visual information and retaining discriminative information that can distinguish between real and fake faces. At the same time, the feature similarity of identical pseudo-label samples is amplified, and the distance between different pseudo-label samples is increased.

[0047] Within the contrastive learning feature space, the cosine similarity of sample pairs is calculated. A loss function is constructed based on this cosine similarity. For samples with hard pseudo-labels of 1 and forged samples with soft pseudo-labels in S2, an intra-group contrastive loss function is constructed. The optimization objective of this loss function is to reduce the distance between the features of all fake samples, so that all fake face samples cluster together in the contrastive learning feature space. The loss function is expressed as follows: , in, This represents the total number of samples containing fake faces. Indicates the first A fake face sample with a pseudo-label of 1 has the following features. Indicates the first A fake face sample with a pseudo-label of 1 has the following features. Indicates except the first Outside of the first sample, the first Features of a fake face sample with a pseudo-label of 1; This is a dynamic temperature parameter used to adaptively adjust the discriminative power of pseudo-labeled samples.

[0048] For S2, where the hard pseudo-label is 0 and all real samples are in the soft pseudo-label, construct an within-group contrastive loss function. The optimization objective is to reduce the feature distance between all real face samples, increase the spatial interval between real face clusters and fake face feature clusters, and strengthen the unified representation capability of real face features. Loss function , is represented as: , in, This represents the total number of real human face samples. Indicates the first Features of real face samples with pseudo-labels of 0 Indicates the first Features of real face samples with pseudo-labels of 0 Indicates except the first Outside of the first sample, the first Features of real face samples with pseudo-labels of 0.

[0049] The total loss function is expressed as : .

[0050] The gradient of the temperature parameter is calculated using the total loss function, and the optimizer automatically and dynamically updates the temperature parameter based on the gradient.

[0051] Preferably, the adaptive dynamic temperature parameter back-optimization learning parameters include the following operations: The dynamic temperature parameter is a learnable parameter that is updated and optimized synchronously during the backpropagation iteration process. The dynamic temperature parameter is used to scale the cosine similarity value, smooth the sample similarity distribution within a batch, and adaptively adjust the feature differentiation between similar and dissimilar samples.

[0052] Specifically, in the early stages of iteration, real / fake sample features are mixed in the contrast space, and the optimizer will drive... The cosine similarity value is automatically increased and decreased, smoothing the similarity between samples and preventing them from getting trapped in local optima. As the iteration progresses, the optimizer drives... The cosine similarity between samples is amplified by adaptive reduction, continuously narrowing the feature gap between real and fake faces until the iteration is completed or the model converges, thus completing the training of the high-dimensional feature encoder.

[0053] S5. Input the video to be detected into the fake feature comparison learning model to obtain the high-dimensional feature vector of the corresponding video frame. Calculate the feature correlation of adjacent video frames based on the high-dimensional feature vector, and then calculate the average inter-frame correlation coefficient corresponding to each feature cluster. Determine the feature cluster with the average inter-frame correlation coefficient less than the preset threshold as the real face cluster, and determine the feature cluster with the average inter-frame correlation coefficient not less than the preset threshold as the fake face cluster.

[0054] Each input video frame of the video to be detected Extracting high-dimensional representations yields high-dimensional video frames. .

[0055] Preferably, the feature correlation calculation of adjacent video frames is as follows: Let the first l The high-dimensional feature vector of the frame is , No. l +1 frame high-dimensional feature vector is K is the total dimension of the encoder output features. The difference in the ranking of features between two frames is compared dimension by dimension, and the result is calculated using the following formula. l Frame and the l Spearman correlation coefficient between +1 frames : , Subsequently, binary clustering was performed on the high-dimensional feature vectors of all video frames in a single video to achieve the division of feature clusters between real and fake faces.

[0056] Preferably, the feature clusters are divided into two categories: true and false. Binary clustering is performed on the two types of feature clusters. By optimizing the initial cluster center selection strategy, the high-dimensional features of all frames in a single video are divided into two independent feature clusters.

[0057] First, select one feature from all frame features of a single video as the first initial cluster center; then calculate the distance between all other features and this center, and select the feature vector with the farthest distance as the second initial cluster center.

[0058] By ensuring that the two initial cluster centers have the maximum feature space interval, the probability of the two initial center points falling into the same feature cluster is reduced, thus guiding the clustering algorithm to segment the video frame features into two independent feature clusters: real faces and fake faces.

[0059] After clustering is completed, the average inter-frame correlation coefficient corresponding to each feature cluster is calculated. This allows for the determination of whether a face is genuine or fake. Inter-frame average correlation coefficient. Represented as: , In the formula, Indicates the first Frame and the + Correlation between 1 frames; It represents the total number of frames in the video clip.

[0060] Naturally shot video is affected by factors such as ambient lighting, camera shake, micro-expressions of people, and natural movement during recording, resulting in subtle and realistic dynamic changes between consecutive frames. These subtle changes are generally smooth, but in a high-dimensional feature space, feature extractors can capture these natural, non-idealized variations, enabling the calculation of correlation indices. Slightly lower.

[0061] Forged videos are often generated through synthesis or splicing. Their generation process frequently employs smoothing strategies or post-processing to eliminate noise, resulting in overly consistent features across consecutive frames and thus affecting the calculated correlation. The correlation coefficient is relatively high. Feature clusters with an inter-frame average correlation coefficient less than a preset threshold are identified as real face clusters, while feature clusters with an inter-frame average correlation coefficient not less than a preset threshold are identified as fake face clusters.

[0062] To verify the effectiveness of the present invention, a simulation experiment was conducted.

[0063] This experiment was conducted in a high-performance computing environment, and the hardware used included: Eight NVIDIA RTX 4090 GPUs, each with 24GB of video memory, can process large-scale data in parallel; Two Intel Xeon Silver 4210 CPUs provide high-performance computing capabilities; Eight 32GB memory modules, for a total memory capacity of 256GB, ensure system stability during data loading and model training.

[0064] Python and PyTorch were used as the programming language and training framework, respectively, to train and improve the model. During model training, the Adam optimizer was employed. The number of training epochs was set to 1000 to improve training efficiency while ensuring convergence. The initial learning rate was set to 3e-4, supplemented by a weight decay strategy of 1e-4.

[0065] The learning rate was gradually decayed to a minimum of 1e-5 using a cosine annealing strategy to ensure model stability in the later stages of training. The batch size was set to 128, and the input image size was uniformly adjusted to 224x224.

[0066] To improve data reading and preprocessing efficiency, the number of processes was set to 8. Furthermore, automatic mixed precision was enabled during training, and the gradient clipping threshold was set to 1.0 to balance computational resource consumption and prevent gradient explosion.

[0067] The datasets used in the experiment include: Dataset 1: CDF, a celebrity deepfake dataset; Dataset 2: F2F, a face replay dataset; Dataset 3: FS, a face transformation dataset; Dataset 4: FSw, a face swap dataset; Dataset 5: NT, a neural texture dataset; and Dataset 6: DF, a deepfake dataset. To ensure consistency across all input samples, frame extraction was performed on the video data, and all images were uniformly resized to 224×224 pixels. Ultimately, each dataset yielded approximately 400,000 images after cropping, and the training and test sets were divided in a 9:1 ratio.

[0068] To further highlight the performance of this invention, three comparison algorithms were used to compare its performance with that of this invention.

[0069] Compared to Algorithm 1, the EfficientNet algorithm is currently a mainstream supervised baseline solution for face forgery detection. Its core is based on neural network scaling, achieving a lightweight yet highly representative backbone network by balancing and coordinating adjustments to the network's depth, width, and input image resolution. In face forgery detection, this method relies on a massive, frame-by-frame manually annotated face dataset for fully supervised training. It uses high-level semantic features extracted layer by layer to perform binary classification of forged and real faces. While this algorithm achieves high detection accuracy thanks to a mature supervised learning framework, its drawbacks include extremely high manual annotation costs, inability to adapt to unannotated real-world business scenarios, and significant limitations in generalizing to novel, unseen AI face-swapping forgery samples.

[0070] Compared to Algorithm 2, the ULF algorithm is a general face forgery detection algorithm for unsupervised scenarios. This algorithm relies solely on single-scale shallow texture features to extract forgery traces and uses a conventional K-Means initialization strategy in the feature clustering stage, without any specific optimization for the inter-frame characteristics of fake videos. This algorithm has weak anti-interference capabilities; its detection performance drops drastically when the detected samples experience perturbations such as changes in lighting or facial pose shifts. It also exhibits instability on multi-class cross-scene forgery datasets and has poor generalization performance.

[0071] Comparing Algorithm 3, the FUDVD algorithm: This algorithm is currently a cutting-edge reference solution in the field of unsupervised deepfake detection. Its core uses a basic single-branch feature extraction network combined with a traditional contrastive learning framework to achieve fake video detection. It constructs sample pairs through basic data augmentation methods, uses a fixed temperature coefficient to construct a contrastive loss to complete model training, and determines the authenticity of faces during the inference phase by simply calculating the inter-frame cosine similarity. This algorithm initially achieves unsupervised fake video detection without manual annotation, but it is limited by the single-scale feature extraction capability and the fixed-temperature contrastive loss design, failing to sufficiently differentiate between real and fake face clusters in the feature space. Therefore, there is still significant room for improvement in detection accuracy under some complex fake scenarios.

[0072] Table 1 shows the accuracy comparison of the present invention and the comparison algorithm on different datasets.

[0073] Table 1. Comparison of accuracy (%) across different datasets As can be clearly seen from Table 1, Algorithm 1, which uses a supervised learning method and employs a large amount of labeled training data, exhibits higher detection accuracy. However, its superior performance relies on high-quality labeled data, and in practical applications, it faces challenges such as high labeling costs and difficulty in dealing with new forgery techniques.

[0074] In the unsupervised learning category, the algorithm of this invention demonstrates significant advantages. Compared to comparative algorithms 2 and 3, the detection accuracy of this invention is significantly improved on all datasets. The accuracy of this invention is 36% higher on dataset 2 and 25% higher than comparative algorithm 1 on dataset 3. This shows that, within the unsupervised learning framework, the algorithm of this invention can still achieve considerable detection results even without a large amount of labeled data, demonstrating its practicality and effectiveness in scenarios with limited data labeling resources.

[0075] As shown in Table 2, the performance of the present invention and the comparison algorithm 3 were tested on different datasets.

[0076] Table 2 Performance metrics for different datasets Table 2 shows that on dataset 1, the proposed algorithm outperforms the comparison algorithm in recall, precision, and F1 score by 1.25%, 0.35%, and 0.78%, respectively. The proposed algorithm also outperforms the comparison algorithm 3 on datasets 2 and 3. Particularly on dataset 4, the accuracy of the comparison algorithm 3 is 70.85%, while the proposed algorithm achieves 76.06%. The recall and precision are also higher than the comparison algorithm 3 by 3.45% and 1.75%, respectively, and the F1 score is 1.26% higher. This indicates that the proposed algorithm achieves a better balance between precision and recall, ensuring high identification ability for fake samples while reducing false positives. Overall, the proposed algorithm outperforms the comparison algorithm 3 on dataset 4. Furthermore, the improved AUC value further demonstrates the proposed algorithm's stronger discriminative ability.

[0077] Figure 1 This image shows heatmaps illustrating the areas of interest for face forgery identified by the algorithm of this invention across different datasets. From left to right, each column represents face data from datasets 1, 2, 3, 4, and 5. To protect facial privacy, the faces in the datasets have been pixelated. The heatmaps use color intensity to reflect the areas of interest identified by this invention when recognizing forged videos. Figure 1 As observed, the model focuses on different regions across different datasets, indicating that it captures texture features from different areas on different datasets. Overall, however, the model not only shows high attention to key facial features (such as eyes and mouth), but also pays attention to more facial edge regions (such as eyebrows, the connection area between the eyes and nose, and facial contours), which are often overlooked in fake video detection.

[0078] A pseudo-video detection system based on facial feature collaboration and unsupervised learning includes the following modules: The clustering module acquires video frame samples containing forged features, detects face regions and crops full-size frames, extracts low-level, mid-level, and deep-level multi-scale feature details within the face regions, and performs preliminary clustering. The reliability measurement module measures the reliability of the preliminary clustering results based on the relative distance between the samples and the cluster centers, and divides them into high-confidence samples, unknown samples, and outlier samples. It assigns fixed hard pseudo-labels to high-confidence samples, assigns dynamically adjustable soft pseudo-labels to unknown samples, and removes outlier samples to obtain the initial dataset. The multi-scale feature extraction module performs data augmentation on each sample in the initial dataset, generating multiple differentiated views for a single face sample through different data augmentation methods, and constructing a contrastive learning dataset containing positive and negative sample pairs. Each sample in the contrastive learning dataset is then sequentially processed through convolution, two-level residual separable convolution, four-level residual separable convolution, and another convolution operation to gradually obtain a fourth augmented feature map. This fourth augmented feature map is then input into a parallel convolution branch for feature extraction at different scales, resulting in multi-scale features. Finally, these multi-scale features are concatenated and fused to obtain multi-scale discriminative features. Two-level residual separable convolution performs one downsampling operation to extract intermediate features; four-level residual separable convolution performs two downsampling operations to extract high-level semantic features. The contrastive learning module maps multi-scale discriminative features to the contrastive learning feature space, calculates the cosine similarity between sample pairs, constructs a contrastive loss function through adaptive dynamic temperature parameters, and optimizes the learning parameters based on the loss function until the model converges, thus obtaining the forgery feature contrastive learning model. The detection output module inputs the video to be detected into the fake feature comparison learning model to obtain the high-dimensional feature vector of the corresponding video frame. Based on the high-dimensional feature vector, it calculates the feature correlation of adjacent video frames, and then calculates the average inter-frame correlation coefficient corresponding to each feature cluster. Feature clusters with an average inter-frame correlation coefficient less than a preset threshold are identified as real face clusters, and feature clusters with an average inter-frame correlation coefficient not less than a preset threshold are identified as fake face clusters.

[0079] Preferably, the comparative learning module includes: The dynamic parameter adjustment module uses a learnable dynamic temperature parameter that is synchronously updated and optimized during backpropagation iteration. This dynamic temperature parameter is used to scale the cosine similarity value, smooth the similarity distribution of samples within a batch, and adaptively adjust the feature differentiation between similar and dissimilar samples.

[0080] Preferably, the detection output module includes: Calculation module, let the first... l Frame feature vector is , No. l +1 frame feature vector is K is the total dimension of the encoder output features. The difference in the ranking of features between two frames is compared dimension by dimension, and the result is calculated using the following formula. l Frame and the l Spearman correlation coefficient between +1 frames : .

Claims

1. A pseudo-video detection method based on facial feature collaboration and unsupervised learning, characterized in that, Includes the following steps: S1. Obtain video frame samples containing forged features, detect face regions and crop full-size frames, extract low-level, mid-level and deep-level multi-scale feature details within the face regions and perform preliminary clustering; S2. Based on the relative distance between the samples and the cluster centers, the reliability of the preliminary clustering results is measured, and high-confidence samples, unknown samples, and outlier samples are divided. Fixed hard pseudo-labels are assigned to high-confidence samples, and dynamically adjustable soft pseudo-labels are assigned to unknown samples. Outlier samples are removed to obtain the initial dataset. S3. Perform data augmentation on each sample in the initial dataset, generate multiple differential views for a single face sample using different data augmentation methods, and construct a contrastive learning dataset containing positive and negative sample pairs. The individual samples in the contrastive learning dataset are sequentially subjected to convolution operations, two-level residual separable convolution operations, four-level residual separable convolution operations, and convolution operations to gradually obtain the fourth enhanced feature map. The fourth enhanced feature map is input into the parallel convolution branch to extract features at different scales, resulting in multi-scale features; the multi-scale features are then concatenated and fused to obtain multi-scale discriminative features. Two-level residual separable convolution performs one downsampling operation to extract intermediate features; four-level residual separable convolution performs two downsampling operations to extract high-level semantic features. S4. Map the multi-scale discriminative features to the contrastive learning feature space, calculate the cosine similarity between sample pairs, construct the contrastive loss function through adaptive dynamic temperature parameters, and optimize the learning parameters in reverse based on the loss function until the model converges, thus obtaining the forgery feature contrastive learning model. S5. Input the video to be detected into the fake feature comparison learning model to obtain the high-dimensional feature vector of the corresponding video frame. Calculate the feature correlation of adjacent video frames based on the high-dimensional feature vector, and then calculate the average inter-frame correlation coefficient corresponding to each feature cluster. Determine the feature cluster with the average inter-frame correlation coefficient less than the preset threshold as the real face cluster, and determine the feature cluster with the average inter-frame correlation coefficient not less than the preset threshold as the fake face cluster.

2. The pseudo-video detection method based on face feature collaboration and unsupervised learning according to claim 1, characterized in that... The adaptive dynamic temperature parameter back-optimization learning parameters in S4 include the following operations: The dynamic temperature parameter is a learnable parameter that is updated and optimized synchronously during the backpropagation iteration process. The dynamic temperature parameter is used to scale the cosine similarity value, smooth the sample similarity distribution within a batch, and adaptively adjust the feature differentiation between similar and dissimilar samples.

3. The pseudo-video detection method based on face feature collaboration and unsupervised learning according to claim 1, characterized in that... Feature correlation calculation of adjacent video frames in S5: Let the first l Frame feature vector is , No. l +1 frame feature vector is K is the total dimension of the encoder output features. The difference in the ranking of features between two frames is compared dimension by dimension, and the result is calculated using the following formula. l Frame and the l Spearman correlation coefficient between +1 frames : 。 4. The pseudo-video detection method based on face feature collaboration and unsupervised learning according to claim 1, characterized in that... S5 divides the feature clusters into two categories: true and false. Binary clustering is performed on the two types of feature clusters. By optimizing the initial cluster center selection strategy, the high-dimensional features of all frames in a single video are divided into two independent feature clusters.

5. The pseudo-video detection method based on face feature collaboration and unsupervised learning according to claim 1, characterized in that... In S2, reliability is measured in the following ways: The reliability metric is to calculate the Euclidean relative distance from a single sample to its cluster center. Samples with a distance less than the first threshold are classified as high-confidence samples and assigned hard pseudo-labels. Samples with a distance between the first and second thresholds are classified as unknown samples and assigned soft pseudo-labels. Samples with a distance greater than the second threshold are identified as outliers and are directly removed.

6. The pseudo-video detection method based on face feature collaboration and unsupervised learning according to claim 1, characterized in that... Multi-scale feature fusion in S3 includes the following operations: Multi-scale features are concatenated along the channel dimension to obtain a concatenated feature map. A 1×1 convolution is used to compress the channel and exchange information across channels in the concatenated feature map. Then, through batch normalization and activation functions, multi-scale discriminative features with uniform dimensions are output.

7. The pseudo-video detection method based on face feature collaboration and unsupervised learning according to claim 1, characterized in that... Data augmentation in S3 includes: The images in the initial dataset are subjected to cropping, horizontal flipping, rotation, color dithering, random grayscale, random erasure, and double cropping transformation operations.

8. A pseudo-video detection system based on facial feature collaboration and unsupervised learning, characterized in that, Includes the following modules: The clustering module acquires video frame samples containing forged features, detects face regions and crops full-size frames, extracts low-level, mid-level, and deep-level multi-scale feature details within the face regions, and performs preliminary clustering. The reliability measurement module measures the reliability of the preliminary clustering results based on the relative distance between the samples and the cluster centers, and divides them into high-confidence samples, unknown samples, and outlier samples. It assigns fixed hard pseudo-labels to high-confidence samples, assigns dynamically adjustable soft pseudo-labels to unknown samples, and removes outlier samples to obtain the initial dataset. The multi-scale feature extraction module performs data augmentation on each sample in the initial dataset, generating multiple differentiated views for a single face sample through different data augmentation methods, and constructing a contrastive learning dataset containing positive and negative sample pairs. The individual samples in the contrastive learning dataset are sequentially subjected to convolution operations, two-level residual separable convolution operations, four-level residual separable convolution operations, and convolution operations to gradually obtain the fourth enhanced feature map. The fourth enhanced feature map is input into the parallel convolution branch to extract features at different scales, resulting in multi-scale features; the multi-scale features are then concatenated and fused to obtain multi-scale discriminative features. Two-level residual separable convolution performs one downsampling operation to extract intermediate features; four-level residual separable convolution performs two downsampling operations to extract high-level semantic features. The contrastive learning module maps multi-scale discriminative features to the contrastive learning feature space, calculates the cosine similarity between sample pairs, constructs a contrastive loss function through adaptive dynamic temperature parameters, and optimizes the learning parameters based on the loss function until the model converges, thus obtaining the forgery feature contrastive learning model. The detection output module inputs the video to be detected into the fake feature comparison learning model to obtain the high-dimensional feature vector of the corresponding video frame. Based on the high-dimensional feature vector, it calculates the feature correlation of adjacent video frames, and then calculates the average inter-frame correlation coefficient corresponding to each feature cluster. Feature clusters with an average inter-frame correlation coefficient less than a preset threshold are identified as real face clusters, and feature clusters with an average inter-frame correlation coefficient not less than a preset threshold are identified as fake face clusters.

9. The pseudo-video detection system based on face feature collaboration and unsupervised learning according to claim 8, characterized in that... The comparative learning module includes: The dynamic parameter adjustment module uses a learnable dynamic temperature parameter that is synchronously updated and optimized during backpropagation iteration. This dynamic temperature parameter is used to scale the cosine similarity value, smooth the similarity distribution of samples within a batch, and adaptively adjust the feature differentiation between similar and dissimilar samples.

10. The pseudo-video detection system based on face feature collaboration and unsupervised learning according to claim 8, characterized in that... The detection output module includes: Calculation module, let the first... l Frame feature vector is , No. l +1 frame feature vector is K is the total dimension of the encoder output features. The difference in the ranking of features between two frames is compared dimension by dimension, and the result is calculated using the following formula. l Frame and the l Spearman correlation coefficient between +1 frames : 。

Images