Generative portrait video attack detection method based on face feature guidance

Abstract

The invention discloses a face feature guidance-based generative portrait video attack detection method, and aims to solve the problems of weak recognition capability, poor generalization, insufficient feature extraction, insufficient robustness and the like existing in an existing detection method when facing a high-fidelity forged video generated by a large model. According to the method, a CLIP image encoder is utilized to extract multi-level semantic features of a face key area, and a FCG loss function is guided through a face component to guide a space module to focus on subtle differences of real and synthetic videos in local texture, illumination and structure; meanwhile, a time module is designed to capture inter-frame dynamic inconsistency, and multi-branch classification fusion is carried out in combination with spatial-temporal characteristics. According to the method, the generalization ability and the detection accuracy of unknown generative attacks are remarkably improved, high robustness is still kept under interference of noise, compression and the like, and the method is suitable for face living body verification of high-security-requirement scenes of insurance, finance and the like.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and in particular to a generative human image video attack detection method guided by facial features. Background Technology

[0002] Facial recognition, as an important technology in the field of artificial intelligence, is widely used in key scenarios such as identity verification in the insurance industry, financial security, and judicial evidence preservation.

[0003] With the evolution of deep learning technology, generative models (such as CLIP and StableDiffusion) have been able to construct a complete image generation system covering data acquisition, feature extraction, and semantic analysis. Their synthesized videos exhibit highly realistic characteristics in dimensions such as texture, lighting, and motion trajectory. Specifically, existing technologies have constructed a detection framework from static feature recognition to dynamic behavior analysis through the collaborative operation of liveness detection algorithms and adversarial example defense mechanisms. Liveness detection mainly relies on micro-expression analysis and motion artifact recognition, while adversarial example defense focuses on robustness optimization of input perturbations. However, existing technologies have systemic defects in detecting generative portrait video attacks: traditional methods only extract local texture features and cannot identify facial motion artifacts (such as unnatural blinking frequencies) in the temporal dimension of the generative model, resulting in insufficient accuracy in recognizing deepfake videos. Specifically, existing technologies typically use multi-head self-attention (MHSA) mechanisms to handle temporal continuity, but have not established a joint analysis framework for spatiotemporal features, resulting in a detection accuracy of less than 65% on the latest attack samples. Among these, the focus loss function ( The lack of spatiotemporal feature tensors makes it difficult for models to capture generative anomalies in facial components (such as lips and skin), while the spatiotemporal feature tensor ( The incomplete representation of facial features further limits the system's ability to detect forged videos. Therefore, the industry urgently needs to build a new detection system that integrates spatiotemporal features and facial attribute guidance to address the security threats posed by generative video technology. Summary of the Invention

[0004] The present invention aims to at least partially solve one of the technical problems in the related art.

[0005] Therefore, the first objective of this invention is to propose a generative facial video attack detection method guided by facial features.

[0006] Another objective of this invention is to propose a generative human image video attack detection device guided by facial features.

[0007] The third objective of this invention is to provide a computer device.

[0008] A fourth objective of this invention is to provide a non-transitory computer-readable storage medium.

[0009] To achieve the above objectives, a first aspect of the present invention proposes a generative facial video attack detection method guided by facial features, comprising: S1: The CLIP image decoder is used to extract the spatiotemporal features of the face in the video. The temporal continuity between video frames is analyzed through a multi-head self-attention mechanism to generate temporal feature embeddings. S2, combining the attention attributes of the CLIP image encoder with the preset facial component attributes, uses a cross-attention module to process spatial distribution features and generate spatial feature embeddings; S3, based on temporal feature embedding and spatial feature embedding, calculates the classification loss of time, space and spatiotemporal joint through a multi-branch classifier, and introduces a facial component guided loss function to optimize feature alignment; S4 weightedly fuses temporal loss, spatial loss, spatiotemporal joint loss, and facial component guidance loss to generate the final video attack detection and classification result.

[0010] In one embodiment of the present invention, S1 includes: S11, video feature tensor The input time module generates a time affinity map through block-temporal multi-head self-attention (PT-MHSA). ; S12, mapping of time affinity The last two dimensions apply 2D convolution kernels Perform feature compression to generate aggregated features. And reshape it into a temporal feature embedding .

[0011] In one embodiment of the present invention, S2 includes: S21, calculated via cross-attention (CA) Generate affinity graph ; S22, Affinity Diagram With video features Combine to generate feature maps Spatial feature embeddings are generated through averaging operations. .

[0012] In one embodiment of the present invention, S3 includes: S31, using the focus loss function Calculate the classification loss for the time branch; S32, using the FCG loss function within the InfoNCE framework. This forces the model to learn the alignment of facial component attributes.

[0013] In one embodiment of the present invention, S4 includes: S41, through the overall objective function Perform weighted fusion, where Adjust the impact weight of FCG loss; S42, utilizing LayerNorm and Time characteristics and spatial features Standardize the process.

[0014] To achieve the above objectives, a second aspect of the present invention provides a generative facial video attack detection device guided by facial features, comprising: The spatiotemporal feature extraction module is used to extract the spatiotemporal features of faces in videos using the CLIP image decoder, and to analyze the temporal continuity between video frames through a multi-head self-attention mechanism to generate temporal feature embeddings. The spatial feature processing module combines the attention attributes of the CLIP image encoder with the preset facial component attributes, and uses a cross-attention module to process spatial distribution features to generate spatial feature embeddings. The classification loss calculation module is used to calculate the classification loss based on temporal feature embedding and spatial feature embedding through a multi-branch classifier, and to optimize feature alignment by introducing a facial component guided loss function. The loss fusion and classification module is used to weightedly fuse temporal loss, spatial loss, spatiotemporal joint loss and facial component guidance loss to generate the final video attack detection and classification result.

[0015] This invention discloses a method and apparatus for detecting generative portrait video attacks based on facial features. It can effectively detect deepfake attacks in generative portrait videos and significantly improve detection accuracy and model generalization ability by extracting and analyzing the temporal-spatial inconsistencies of facial features.

[0016] To achieve the above objectives, a third aspect of this application provides a computer device, including a processor and a memory; wherein the processor reads executable program code stored in the memory to run a program corresponding to the executable program code, for implementing a generative portrait video attack detection method based on facial feature guidance as described in the first aspect embodiment.

[0017] To achieve the above objectives, a fourth aspect of this application provides a non-transitory computer-readable storage medium storing a computer program that, when executed by a processor, implements a generative facial video attack detection method guided by facial features as described in the first aspect embodiment.

[0018] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0019] Figure 1 This is a flowchart of a generative human image video attack detection method based on facial feature guidance according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the overall process of a generative human image video attack detection technology solution based on facial features according to an embodiment of the present invention. Figure 3 This is an image encoder diagram according to an embodiment of the present invention; Figure 4 This is a structural diagram of a generative human image video attack detection device based on facial feature guidance according to an embodiment of the present invention; Figure 5 It is a computer device according to an embodiment of the present invention. Detailed Implementation

[0020] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0021] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0022] The following description, with reference to the accompanying drawings, describes a generative facial video attack detection method and apparatus based on facial feature guidance according to an embodiment of the present invention.

[0023] Example 1 Figure 1 This is a flowchart of a generative facial video attack detection method based on facial feature guidance according to an embodiment of the present invention, as shown below. Figure 1 As shown, it includes: S1 utilizes the CLIP image decoder to extract the spatiotemporal features of faces in the video, and analyzes the temporal continuity between video frames through a multi-head self-attention mechanism to generate temporal feature embeddings.

[0024] Specifically, this step utilizes the CLIP image decoder to extract spatiotemporal features of faces in the video and combines this with a multi-head self-attention mechanism to analyze the temporal continuity between video frames, thereby generating temporal feature embeddings. This process leverages the powerful cross-modal alignment capability of the CLIP model to extract image features with semantic information and models them in the temporal dimension to capture temporal inconsistencies in faces within the video.

[0025] Specifically, the CLIP image decoder at each layer In the Transformer structure, the output video segment features ,in Indicates the number of input video frames. It is the number of spatial locations in the feature map. It's about the number of heads to focus on. These are the feature dimensions of each attention head. In the temporal module, for each fixed spatial location, block-temporal multi-head self-attention (PT-MHSA) is applied to the selected attention attributes. Processing is performed to generate a time affinity map. The mapping is normalized in the time dimension using the softmax function to enhance temporal relevance.

[0026] Furthermore, to compress the time dimension and extract more compact temporal features, a two-dimensional convolutional kernel is employed. right The last two dimensions are processed to generate Subsequently, through the residual layer (Constitutes two fully connected layers and an intermediate ReLU activation) Processing ,generate Its shape is similar to Consistency. To integrate information from spatially adjacent blocks, Remodeling and using convolution kernels Perform two-dimensional convolution operations to obtain aggregated features. Finally, Remodeling into temporal feature embeddings As the first Temporal feature output of the layer decoder.

[0027] Furthermore, it is particularly suitable for attack detection scenarios involving generative facial videos, such as identity verification in the insurance industry and liveness detection in financial transactions. By capturing temporal inconsistencies between video frames, the model can effectively identify fake videos generated by deepfake technology, thereby improving the accuracy and robustness of detection. Regarding parameter settings, The value is usually 8 or 16. It is 64 or 128. The model is configured with 16 or 32 frames to balance computational complexity with temporal modeling accuracy. This technique significantly enhances the model's ability to perceive the temporal dimension, providing crucial temporal feature support for subsequent classification and detection.

[0028] Furthermore, S1 includes: S11, video feature tensor The input time module generates a time affinity map through block-temporal multi-head self-attention (PT-MHSA). .

[0029] Specifically, in this patented technical solution, the step of "temporal feature detection" introduces a block-temporal multi-head self-attention (PT-MHSA) mechanism to process the video feature tensor. Processing is performed to generate a time affinity map. This allows for the capture of inconsistencies in video over time, thereby improving the detection capabilities against generative portrait video attacks.

[0030] Furthermore, this step begins with the first... Extracting video features from layer decoders and select specific attention attributes In other words, query, key, and value. Under the premise of a fixed spatial location, the time dimension... A multi-head self-attention mechanism is applied to form attentional interactions along the block-temporal dimension. Specifically, PT-MHSA generates a temporal affinity map by calculating the attention weights between each time point and other time points. Its shape is ,in Indicates the number of selected attention attributes. For the number of attention heads, For video frame rate, This represents the number of image patches.

[0031] Furthermore, time affinity mapping The dimensional design fully considers the temporal continuity and spatial distribution characteristics of the video. In the generation... Then, a softmax function is applied to normalize the last axis to ensure the attention weights are interpretable and stable over time. Subsequently, a two-dimensional convolutional kernel is used. Attention scores from adjacent time points are compressed and integrated to generate feature maps. Furthermore, through the residual layer (Constitutes two fully connected layers and an intermediate ReLU activation layer) Processing ,generate Its shape is similar to Consistency. To integrate information from spatially adjacent blocks, Remodeling And apply a two-dimensional convolution kernel Ultimately, aggregated features are generated. Finally, Remodeling As the first Temporal feature output of the layer decoder.

[0032] Furthermore, this step is primarily used to detect facial videos synthesized by generative models (such as GPT and StableDiffusion). By analyzing the temporal affinity between video frames, the model can identify common temporal discontinuities in forged videos, such as abnormal motion between frames and unnatural facial expressions. This mechanism is particularly suitable for facial recognition authentication scenarios in the insurance industry, effectively identifying forged videos and preventing fraudulent activities.

[0033] Furthermore, through the PT-MHSA mechanism, the model can more accurately model the temporal dynamics of the video, thereby improving the detection accuracy against generative portrait video attacks. Compared with traditional methods, this step introduces a more fine-grained attention mechanism in the temporal dimension, enhancing the model's sensitivity to temporal inconsistencies and significantly improving the robustness and generalization ability of the detection.

[0034] S12, mapping of time affinity The last two dimensions apply 2D convolution kernels Perform feature compression to generate aggregated features. And reshape it into a temporal feature embedding .

[0035] Specifically, in this patented technical solution, time affinity mapping is performed. The last two dimensions apply 2D convolution kernels To achieve feature compression and generate aggregated features Ultimately, it is reshaped into a temporal feature embedding. This step improves the detection performance against generative facial video attacks by compressing and aggregating attention information along the temporal dimension and extracting discriminative temporal features.

[0036] Furthermore, time affinity mapping It is generated by a block-temporal multi-head self-attention (PT-MHSA) mechanism, and its shape is... ,in Representing feature dimension, This indicates the number of selected attention attributes (query, key, value). For the number of attention heads, The input video frame count is used. This mapping reflects the attention distribution relationship between video frames in the temporal dimension. To further compress and integrate this information, a 2D convolutional kernel is employed. For its last two dimensions (i.e. The operation is performed using this kernel. The number of channels in this convolution kernel is... The number of output channels is 1, and the kernel size is typically 1. or The stride is 1, and zero padding is used to maintain spatial resolution. After the convolution operation, the output feature map is... Its spatial dimensions are compressed to It preserves the attention pattern in the time dimension.

[0037] Furthermore, in order to integrate information from spatially adjacent blocks, Remodeling In the form of, And apply 2D convolution kernels again. Its output channel number is 1, and its input channel number is... The kernel size is Step size is This is to achieve further compression of the spatial dimension. The final output aggregated features Shape It contains compressed and integrated attention features in the time dimension.

[0038] Furthermore, this step is primarily used for temporal feature extraction from generative portrait videos. By compressing the attention distribution along the temporal dimension, the model can capture inconsistencies between frames in the video, such as disjointed movements and abrupt changes in facial expressions commonly seen in forged videos. The temporal feature embedding output from this step... Will be used as a classifier The input is used to predict whether a video is a synthetic video.

[0039] Furthermore, this step effectively reduces the feature dimensionality through 2D convolution operations while preserving key temporal attention patterns, improving the model's computational efficiency and discriminative ability when processing high-dimensional temporal data. By compressing attention information in the temporal dimension into low-dimensional embeddings, the model can focus more on anomalous features in the temporal dimension, thereby enhancing its robustness against generative portrait video attacks.

[0040] S2 combines the attention attributes of the CLIP image encoder with the preset facial component attributes, and uses a cross-attention module to process spatial distribution features to generate spatial feature embeddings.

[0041] Specifically, in this patented technical solution, the step "spatial feature detection" combines the attention attributes of the CLIP image encoder with preset facial component attributes, and uses a cross-attention module to process the spatial distribution features, thereby generating a semantically guided spatial feature embedding. This step utilizes the attention mechanism of the decoder in the CLIP model to extract spatial features related to key facial parts (such as lips, skin, eyes, and nose), and uses the FCG loss function to achieve accurate alignment and optimization of these features.

[0042] Furthermore, the spatial module first extracts the attention attributes of each decoder layer from the CLIP image encoder. ,in This represents three attention attributes: query, key, and value. Through preprocessing stages including facial alignment and random enhancements (such as scaling, cropping, and bidirectional flipping), video frames are input into the CLIP image encoder to extract layer-by-layer attention attributes. Subsequently, using pre-defined facial component coordinate information, the feature locations of each key area (such as lips and eyes) are precisely located, and the attention attributes at these locations are L2 normalized and averaged to generate layer-by-layer facial component attributes. , where 4 represents the number of facial features. For the number of attention heads, The feature dimensions for each head.

[0043] Furthermore, the spatial module employs a cross-attention (CA) mechanism to integrate query attributes. Interact with encoder attention properties to generate affinity graphs ,in For the number of input frames, The number of training samples, This is the number of features. The affinity map is obtained by comparing the encoder features... Combine to generate feature maps Finally, by averaging the first two dimensions, the layer-specific spatial feature embedding is obtained. .

[0044] Furthermore, this step is primarily used in practical applications for attack detection in generative facial videos, especially in scenarios with high identity verification requirements, such as the insurance industry. By introducing a facial component guidance mechanism, the model can more effectively identify facial structural anomalies in forged videos, thereby improving detection accuracy and generalization ability. Technically, this step significantly enhances the semantic representation of spatial features, enabling the model to more accurately capture the feature distribution of key facial parts, effectively countering forgery strategies employed by generative models in the spatial dimension, thus improving the robustness and practicality of the overall detection system.

[0045] Furthermore, S2 includes: S21, calculated via cross-attention (CA) Generate affinity graph

[0046] Specifically, in step three of this patented technical solution, an affinity graph is generated by cross-attention (CA) calculation. A high-dimensional feature correlation matrix is ​​constructed to capture the spatial relationship between key facial features and global features in video frames, thereby improving the detection capability of generative portrait video attacks.

[0047] Furthermore, this step first starts from the decoder's... Extracting query vectors from layers ,in Indicates the number of query points. It's about the number of heads to focus on. These are the feature dimensions of each attention head. These query vectors are related to the attention attributes from the encoder layer. Perform cross-attention calculation, where An index vector representing an attention attribute is used to locate feature regions related to key facial features (such as lips, eyes, and nose). The cross-attention mechanism generates an affinity graph by calculating the similarity between the query vector and the target features. ,in It is the number of input video frames. It is the spatial dimension of the feature map (such as the product of the width and height of the feature map).

[0048] Furthermore, the key parameters involved in this step include: the number of query points. Attention count Feature Dimension Input frame count and the spatial dimension of the feature map By setting A value consistent with the number of key facial features (e.g.) (corresponding to lips, skin, eyes, and nose respectively), ensuring that each query point is accurately aligned to its corresponding facial region. Furthermore, the output dimension of cross-attention... This ensures that the model can simultaneously model the feature relationships in both time series and spatial locations.

[0049] Furthermore, this step is primarily used for spatial feature modeling in generative facial video attack detection. This is achieved by constructing an affinity graph. The model can identify abnormal feature distributions in key facial areas of a video. For example, in fake videos, there may be inconsistencies between lips and speech frames, or the eye movements may not conform to real physiological patterns. These inconsistencies manifest as abnormal attention distributions in the affinity map, thus providing a basis for subsequent classifiers.

[0050] Furthermore, this step significantly enhances the model's ability to perceive key facial features by introducing a cross-attention mechanism. Compared to traditional methods that rely solely on global or local feature extraction, this approach improves the model's robustness and generalization ability in complex backgrounds and dynamic scenes by precisely guiding the alignment of the query vector with facial attributes. In addition, this step provides crucial input for subsequent FCG loss calculation, helping to further optimize the model's accuracy in recognizing generative attacks.

[0051] S22, Affinity Diagram With video features Combine to generate feature maps Spatial feature embeddings are generated through averaging operations. .

[0052] Specifically, in this patented technical solution, step three, "spatial feature detection," involves detecting the affinity graph. With video features Combine to generate feature maps Furthermore, spatial feature embeddings are generated through averaging operations. This is a key step in realizing the detection of spatial inconsistencies in faces in generative portrait videos.

[0053] Furthermore, this step is first based on the selected attributes in the decoder. Affinity graphs are calculated using the cross-attention (CA) mechanism. Its shape is ,in Indicates the number of video frames. Indicates the number of query vectors. This indicates the feature dimension. Subsequently, With the from the Video features of layer encoders Perform fusion to generate new feature maps Its dimensions are ,in For the number of attention heads, For each head, a feature dimension is defined. This fusion process weights video features with attention weights, thereby enhancing the representation of features related to key facial attributes such as lips, eyes, and nose.

[0054] Furthermore, in this step, key parameters include the number of attention heads. Feature Dimension Number of query vectors and time frame count .in, This represents the length of the feature vector output by each attention head, and is typically set to [value] in practical deployments. , To balance model complexity and feature representation capability. Spatial feature embedding. The dimension is ,Right now This is used for subsequent classification tasks.

[0055] Furthermore, this step is applicable to attack detection scenarios involving generative facial videos, particularly in insurance industry operations such as identity verification, remote account opening, and claims review. By accurately extracting the spatial features of faces in videos, the system can identify unnatural facial structures or texture distributions in forged videos, effectively distinguishing between real and synthetic content. In practical deployments, this module can be processed in parallel with the temporal feature detection module, ultimately using a multi-branch classifier for comprehensive judgment.

[0056] Furthermore, this step significantly enhances the discriminative power of spatial features by introducing a facial attribute guidance mechanism. By aligning the attention mechanism with key facial features, the model can more effectively capture facial spatial inconsistencies in generative videos, thereby enhancing robustness against deepfake attacks. In addition, this method generates fixed-dimensional embedding vectors through averaging, facilitating subsequent classifier processing and improving the model's generalization ability and deployment efficiency.

[0057] S3, based on temporal feature embedding and spatial feature embedding, calculates the classification loss of time, space and spatiotemporal joint through a multi-branch classifier, and introduces a facial component guided loss function to optimize feature alignment.

[0058] Specifically, this step constructs a multi-branch classifier, calculates the classification losses for time, space, and spatiotemporal joint, and introduces Facial Component Guided Loss (FCGLoss) to optimize feature alignment, thereby improving the accuracy and robustness of generative portrait video attack detection. In some implementations, this method is based on video features extracted by the CLIP image encoder, performs feature modeling through the temporal and spatial branches in the decoder module, and combines attribute information of key facial features to achieve multi-dimensional discrimination of the authenticity of faces in videos.

[0059] Furthermore, temporal feature embedding extracts temporal continuity features between video frames using a multi-head self-attention mechanism (MHSA). Specifically, in the first... In the layer decoder, for the selected attention attribute Applying block-temporal multi-head self-attention (PT-MHSA) to generate temporal affinity maps Subsequently, a 2D convolution kernel was used. The time dimension is compressed and integrated to generate feature maps. Furthermore, it is achieved through a residual layer consisting of two fully connected layers and ReLU activation. The process is performed to finally output the temporal feature embedding. .

[0060] Furthermore, spatial feature embedding uses a cross-attention (CA) mechanism to embed the query attributes of video segments. With attention attributes Interact to generate affinity graph Further combining encoder features Generate feature maps Spatial feature embeddings are extracted through average pooling. .

[0061] Furthermore, this step is applicable to facial recognition authentication systems in the insurance industry, particularly in detecting forged facial images generated by large models. It effectively identifies temporal and spatial inconsistencies, thereby preventing fraudulent activities. In terms of technical effectiveness, this step enhances the model's ability to identify generative attacks through multi-branch loss design and facial component guidance mechanisms, improving detection accuracy and robustness, and addressing the shortcomings of traditional methods in feature alignment and generalization capabilities.

[0062] Furthermore, S3 includes: S31, using the focus loss function Calculate the classification loss for the time branch.

[0063] Specifically, this patent proposal involves using a focal loss function to calculate the classification loss of the temporal branch. This step, by introducing the focal loss function, optimizes the temporal branch's ability to identify temporal inconsistencies in generative portrait videos, thereby improving the model's accuracy and robustness in fake video detection tasks.

[0064] Furthermore, the output characteristics of the time branch First, through LayerNorm Normalization is performed to eliminate feature distribution bias and enhance model stability. The normalized features are then input into the fully connected layer. In this layer, the high-dimensional features are mapped to the classification space, outputting a prediction vector related to the realism of the video segment. This prediction vector is then compared with the ground truth label. Input together into the focus loss function The loss is calculated during the process. The introduction of the focus loss function aims to alleviate the class imbalance problem, especially when there are few fake video samples. By adjusting the loss weights, the model focuses more on hard-to-classify samples, thereby improving the overall detection performance.

[0065] Furthermore, the focus loss function takes the form of:

[0066] in This indicates the probability that the model's prediction is true or false. Used to balance the weights of positive and negative samples. It is a focusing factor that adjusts the level of attention given to samples of varying difficulty. In this proposal, It is usually set to 2. It can be dynamically adjusted based on the distribution of the dataset to enhance the model's sensitivity to fake videos.

[0067] Furthermore, this step is primarily used for the temporal feature classifier during the training phase, and its output, together with the losses from the spatial and spatiotemporal branches, constitutes the total loss function. This is used for end-to-end optimization of the model. In actual deployment, the classification results of the time branch will be used as part of the final decision to determine whether a video is a generative attack video.

[0068] Furthermore, by introducing the focus loss function, the ability of the temporal branch to discriminate fake videos is significantly improved. Especially when facing highly realistic generated videos, it can effectively capture abnormal patterns in the temporal dimension, thereby enhancing the model's robustness and generalization ability against generative attacks.

[0069] S32, using the FCG loss function within the InfoNCE framework. This forces the model to learn the alignment of facial component attributes.

[0070] Specifically, in this patented technical solution, by introducing the FCG loss function and based on the InfoNCE framework, the model is forced to learn the attribute alignment relationships related to key facial components in the spatial module, thereby improving the accuracy and generalization ability of generative portrait video attack detection. The mathematical expression of this loss function is as follows:

[0071] in, Indicates the number of training samples. Indicates the number of layers in the Transformer decoder. It is a temperature parameter used to control the softness or stiffness of the softmax output distribution. This represents the cosine similarity between two vectors. Indicates the first The first in the layer The query vector for each sample. Indicates the first The first in the layer A vector of facial component attributes for each sample.

[0072] Furthermore, the FCG loss function guides the model to align the query vector with the facial component attribute vector in the spatial dimension through a contrastive learning mechanism. Specifically, during training, the model generates multiple query vectors for each sample and compares them with facial component attribute vectors extracted from real faces. By maximizing the similarity of positive sample pairs while minimizing the similarity of negative sample pairs, the model can learn more discriminative facial feature representations.

[0073] Furthermore, Typically initialized to 0.07, and dynamically adjusted during training via the learning rate scheduler. Facial component attribute vector. The images are extracted using the CLIP image encoder, and the video frames undergo preprocessing before extraction, including random cropping, scaling, and bidirectional flipping to enhance the model's robustness. Furthermore, each facial component (such as lips, skin, eyes, and nose) is independently modeled in the loss function to ensure the model can accurately detect forgery traces in different areas.

[0074] Furthermore, in practical applications, this step is mainly used to improve the feature extraction capabilities of the spatial module, especially when processing high-fidelity portrait videos synthesized by generative models (such as GANs and diffusion models), effectively identifying inconsistencies in the spatial distribution of facial components. For example, in facial recognition systems in the insurance industry, attackers may generate portrait videos with natural expressions and lighting through deepfakes, while the FCG loss function can guide the model to focus on feature alignment of key facial regions, thereby improving detection accuracy and anti-spoofing capabilities.

[0075] Furthermore, the FCG loss function significantly enhances the model's spatial discriminative ability, enabling the spatial module to more effectively identify anomalies in facial components in forged videos. Experiments show that this loss function can bring about a 3% to 5% improvement in detection accuracy on multiple public datasets (such as Celeb-DF and DFDC), effectively alleviate the overfitting problem of the model to specific datasets, and enhance the model's cross-dataset generalization ability.

[0076] S4 weightedly fuses temporal loss, spatial loss, spatiotemporal joint loss, and facial component guidance loss to generate the final video attack detection and classification result.

[0077] Furthermore, this fusion mechanism performs a weighted summation based on the output of the multi-branch classifier. Specifically, the temporal module extracts the continuity features of video frames in the temporal dimension through a multi-head self-attention mechanism (MHSA) to generate temporal embeddings. And by classifier The loss function for predicting the composite score over the time dimension is: The spatial module extracts spatial features of image patches through a cross-attention (CA) mechanism. , and by The classifier performs a composite judgment on spatial dimensions, and its loss function is: The spatiotemporal co-concatenation module then concatenates temporal and spatial features (using the `cat` operation), through... The classifier performs joint judgment, and its loss function is: .

[0078] Furthermore, this fusion mechanism is widely applicable to high-security scenarios that rely on facial recognition, such as insurance, finance, and security. For example, during remote identity verification, the system needs to determine in real time whether the uploaded video is a deepfake. Through the synergistic effect of multiple loss functions, the model can more comprehensively capture anomalies in video at the temporal dynamics, spatial structure, and facial component levels, thereby significantly reducing false positive and false negative rates.

[0079] Furthermore, through joint optimization of multi-dimensional loss functions, the model can more accurately identify forgery traces in generative portrait videos, especially demonstrating stronger adaptability and robustness when facing unknown attack methods. The introduction of facial component-guided loss further enhances the model's ability to perceive local features, providing more granular discrimination criteria for video attack detection.

[0080] Furthermore, S4 includes: S41, through the overall objective function Perform weighted fusion, where Adjust the impact weight of FCG loss.

[0081] Specifically, in this patented technical solution, step four involves the design of the classification target of the detection model, the core of which lies in constructing a comprehensive optimized objective function by weighted fusion of multiple classification loss functions. To improve the accuracy and robustness of generative facial video attack detection.

[0082] Furthermore, this step integrates the outputs of the temporal, spatial, and spatiotemporal joint classifiers, and introduces the Face Component Guided Loss (FCGLoss) as an auxiliary supervision signal. Among these, the temporal loss... Features extracted from the time module via LayerNorm and fully connected layer Then input focus loss function This is used to measure the model's ability to distinguish between real and fake videos over time.

[0083] Furthermore, the fusion weights of the loss function This is a key adjustable parameter used to control the impact of FCG loss on the overall training objective. This parameter is typically optimized during training using cross-validation to balance the learning performance of the model across different feature dimensions.

[0084] Furthermore, this step is applicable to fields such as the insurance industry and security systems, where the authenticity of facial videos is extremely important. By fusing multiple loss functions, the model can more comprehensively capture facial inconsistencies in videos, thus maintaining a high detection accuracy even when faced with complex forgery techniques.

[0085] Furthermore, by introducing FCG loss and combining it with the loss function of multi-class heads, the model's discriminative ability in multi-dimensional feature space is significantly improved, enhancing its robustness and generalization ability against generative human image video attacks, and providing a solid theoretical and practical foundation for subsequent model deployment and practical application.

[0086] S42, utilizing LayerNorm and Time characteristics and spatial features Standardize the process.

[0087] Specifically, in this patented technical solution, the step "utilizing LayerNorm" and Time characteristics and spatial features Normalization is a crucial preprocessing step for generative human image video attack detection. This step introduces two independent LayerNorm modules to normalize the features extracted in the temporal and spatial dimensions, respectively, thereby improving the model's stability and generalization ability in subsequent classification tasks.

[0088] Furthermore, LayerNorm is a normalization technique commonly used in deep learning models. Its function is to standardize the input features at each layer to alleviate gradient vanishing and accelerate model convergence. In this scheme, temporal features... It is extracted from the video frame sequence through a temporal module (such as PT-MHSA), and its dimension is ,in Indicates the number of time frames. Indicates the number of features. Spatial features. The spatial module extracts the data from the image patch, and its dimension is... These two features are respectively passed through LayerNorm layers. and Normalization is performed to eliminate differences in feature distribution between different layers and enhance the model's robustness to input features.

[0089] Furthermore, the parameters of a LayerNorm layer typically include learnable scaling parameters. and offset parameters The calculation formula is as follows:

[0090]

[0091] in and These are the mean and standard deviation of the features, respectively. By introducing these two learnable parameters, the model can adaptively adjust the distribution of features, thereby improving detection performance.

[0092] Furthermore, this step is widely applied in the feature processing stage of video frame sequences, especially in multimodal video analysis systems based on CLIP image encoders. By normalizing temporal and spatial features respectively, the model can more effectively capture inconsistencies in the temporal dimension of faces in videos (such as inter-frame motion anomalies) and structural anomalies in the spatial dimension (such as unnatural distribution of facial components), thereby improving the ability to identify generative facial video attacks.

[0093] Furthermore, this step significantly enhances the model's stability to input features and reduces false positives and false negatives caused by inconsistent feature distributions. Simultaneously, through an independent LayerNorm module, the model can optimize the representation of temporal and spatial features separately, providing a foundation for subsequent classifiers. , and This provides more consistent and interpretable input, thereby improving overall detection accuracy and robustness. Furthermore, this design helps the model maintain high detection performance when facing different resolutions, lighting conditions, or video compression methods, demonstrating good engineering practical value.

[0094] This invention provides a generative portrait video attack detection method based on facial feature guidance, which can effectively identify synthetic traces in generative portrait videos and improve detection accuracy and generalization ability by analyzing the temporal and spatial inconsistencies of facial features.

[0095] Example 2 In another embodiment of the present invention, a generative facial video attack detection technology based on facial feature guidance is proposed, and the overall process is as follows: Figure 2 As shown, the specific technical solution is as follows: Step 1: Feature Extraction. The CLIP image encoder is used to extract general features from the network's spatial and temporal modules that can be used for deepfake video detection. Several symbols and notations representing network configuration are introduced: L represents the number of transformer layers, P is the number of features, T is the number of input frames, H is the number of attention heads in multi-head self-attention, and D is the feature dimension of each attention head.

[0096] like Figure 3 As shown, for each transformer layer of the image encoder, the initial video segment features are obtained. and attention attributes ,in Representation layer index, Specify the attention attribute types (i.e., query, key, and value). These attributes are processed by the decoder blocks of specific layers and analyzed by their respective spatial and temporal modules, thus producing the temporal representation for each layer. and spatial embedding The features of these specific layers are aggregated to form a comprehensive representation, denoted as... and Then use their respective LayerNorm layers and It was then standardized. The framework employs three independent classifiers. , and Each layer is implemented by a fully connected (FC) layer with its own temporal, spatial, and spatiotemporal branches to predict the level of synthesis and the presence of synthesized videos. The final score is the average of the outputs of these classifiers during inference.

[0097] Step 2, Temporal Feature Detection. In the l-th layer decoder block, the temporal module focuses on capturing the temporal continuity of video segments. This is achieved by detecting temporal features from the l-th encoder layer. Select certain attributes To achieve this, initially, for each fixed location, multi-head self-attention (MHSA), specifically block-time multi-head self-attention (PT-MHSA), is applied along the time axis. Each attribute selected. This will generate a time affinity map. The affinity is normalized by applying a softmax function on its last axis. Then, the multi-head affinity scores at adjacent time points are compressed, and the results are compared with... 2D kernel with convolution on the last two dimensions Integrating them produces another feature map. Then through the residual layer Process the result to produce the same as Same shape ,in It consists of two FC layers and a middle ReLU activation layer, used to facilitate activation across all Interaction at specific times is used to better capture subtle temporal differences in the video. To merge spatially adjacent block information, [the following is done / implemented]... Remodeling and use kernels in the last two dimensions. Applying two-dimensional convolution to generate aggregated features Finally, Remodeling As the temporal feature of the l-th decoder layer.

[0098] Step 3, Spatial Feature Detection. The core function of the spatial module in the decoder is to detect complex features from image patches. The model accurately extracts and effectively utilizes generalization-capable cues. These cues are crucial for subsequent image processing tasks and can significantly improve model performance. The spatial module's channels detail how to identify layer-by-layer attributes closely related to each major facial component (such as eyes, nose, and mouth) through deep mining of facial attributes. This layer-by-layer attribute identification not only helps the model understand facial structure more accurately but also improves its accuracy in tasks such as face recognition. Furthermore, the FCG (Facial Component Guided) loss function employs a contrastive objective function to ensure that each query is precisely aligned with the corresponding facial attribute. In this way, the FCG loss function significantly enhances the generalization ability of the spatial module, enabling it to better adapt to different datasets and scenarios. The FCG loss function plays a vital role in the optimization process. It guides the spatial module to focus on key facial features such as lips, skin, eyes, and nose. This targeted focus not only helps improve the model's ability to recognize these features but also effectively prevents the model from overfitting to specific features of the dataset. By avoiding overfitting, the model can exhibit better generalization performance on new data.

[0099] Furthermore, the channel design includes: assuming that training data is available for the model to learn starting from N steps, denoted as... Selected attributes for all video clips Cross-attention (CA) is used to generate affinity graphs. ,in Then, each affinity is mapped to a feature from the video segment from the l-th encoder layer. Combined to obtain a queryable feature map. The final step involves averaging the first two dimensions to generate layer-specific spatial features. .

[0100] Furthermore, facial attribute mining includes: before calculating the FCG loss, first identifying specific layer attributes associated with key facial features (lips, skin, eyes, nose) to guide the spatial module's queries at each layer during training. First, several frames are selected from the preprocessed face alignment video. These frames undergo various random enhancements, such as resizing, cropping, and bidirectional flipping. After enhancement, these frames are fed into the CLIP image encoder to extract layer-by-layer images. property Using the coordinates from the preprocessing stage, the feature locations of each facial region can be precisely located. Then, the data from... The feature attributes corresponding to each facial region are calculated, and after L2 normalization, they are averaged to generate layer-by-layer facial region attributes. For example, the first dimension is aligned with the lips, skin, eyes, and nose respectively.

[0101] Furthermore, facial feature guidance includes: after completing facial attribute mining, calculating the FCG loss using the InfoNCE loss function, which has a reward function. and The attributes of each face component are closely matched. The FCG loss calculation formula is as follows:

[0102] in This indicates the degree of softness or hardness of the output distribution of the softmax function; N=4 represents the number of facial features. This represents the cosine similarity between two vectors. The number of queries is equal to the number of queries for facial features.

[0103] Step four: Design the classification objective of the detection model. The main objective function includes focus loss. They come from time ,space and spacetime branches Three different classification heads:

[0104]

[0105]

[0106] in The join operation represents the merging of features, where y is the true or false fact label. The overall objective of training the model is defined as:

[0107] in Adjusting the impact of FCG loss on the overall training objective.

[0108] Step 5: Model Deployment and Launch. To ensure the model's effectiveness in actual production, stability monitoring is performed to observe whether model iterations and updates are needed, thereby adjusting and optimizing the model accordingly.

[0109] Generative facial video attack detection technology guided by facial features can better adapt to complex environments and changing scenarios by introducing more advanced algorithms and models, and can accurately distinguish between real facial images and attack videos.

[0110] The generative facial video attack method based on facial feature guidance in this invention exhibits higher real-time performance and robustness. In real-time applications, the system needs to respond quickly and accurately determine the authenticity of the video. The generative facial video attack detection technology based on facial feature guidance achieves rapid video processing and accurate judgment through optimized algorithms and accelerated computation, ensuring the real-time performance of the system. Simultaneously, this technology also possesses strong robustness, capable of responding to various forms of attack, including but not limited to video tampering and forgery.

[0111] Example 3 To achieve the above embodiments, such as Figure 4 As shown, this embodiment also provides a generative facial video attack detection device 10 guided by facial features, including: The spatiotemporal feature extraction module 100 is used to extract the spatiotemporal features of the face in the video using the CLIP image decoder, analyze the temporal continuity between video frames through a multi-head self-attention mechanism, and generate temporal feature embeddings. The spatial feature processing module 200 is used to combine the attention attributes of the CLIP image encoder with the preset facial component attributes, and use a cross-attention module to process spatial distribution features to generate spatial feature embeddings. The classification loss calculation module 300 is used to calculate the classification loss based on temporal feature embedding and spatial feature embedding through a multi-branch classifier, and to optimize feature alignment by introducing a facial component guided loss function. The loss fusion and classification module 400 is used to weightedly fuse temporal loss, spatial loss, spatiotemporal joint loss and facial component guidance loss to generate the final video attack detection and classification result.

[0112] Furthermore, the spatiotemporal feature extraction module 100 is also used for: video feature tensor The input time module generates a time affinity map through block-time multi-head self-attention. ; Time affinity mapping The last two dimensions apply 2D convolution kernels Perform feature compression to generate aggregated features. And reshape it into a temporal feature embedding .

[0113] Furthermore, the spatial feature processing module 200 is also used for: Calculated via cross attention (CA) Generate affinity graph ; Affinity diagram With video features Combine to generate feature maps Spatial feature embeddings are generated through averaging operations. .

[0114] This invention discloses a generative portrait video attack detection method based on facial feature guidance, which can effectively detect deepfake attacks in generative portrait videos. By extracting and analyzing the temporal-spatial inconsistency of facial features, it significantly improves the detection accuracy and model generalization ability.

[0115] Example 4 To implement the methods of the above embodiments, the present invention also provides a computer device, such as... Figure 5 As shown, the computer device 600 includes a memory 601 and a processor 602; wherein, the processor 602 reads the executable program code stored in the memory 601 to run a program corresponding to the executable program code, so as to implement the various steps of the generative portrait video attack detection method based on facial features described above.

[0116] Example 5 To implement the above embodiments, this application also proposes a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a generative portrait video attack detection method based on facial features as described in the foregoing embodiments.

[0117] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0118] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

Claims

1. A face feature guide-based generative portrait video attack detection method, characterized in that, include: S1: The CLIP image decoder is used to extract the spatiotemporal features of the face in the video. The temporal continuity between video frames is analyzed through a multi-head self-attention mechanism to generate temporal feature embeddings. S2, combining the attention attributes of the CLIP image encoder with the preset facial component attributes, uses a cross-attention module to process spatial distribution features and generate spatial feature embeddings; S3, based on temporal feature embedding and spatial feature embedding, calculates the classification loss of time, space and spatiotemporal joint through a multi-branch classifier, and introduces a facial component guided loss function to optimize feature alignment; S4 weightedly fuses temporal loss, spatial loss, spatiotemporal joint loss, and facial component guidance loss to generate the final video attack detection and classification result.

2. The method of claim 1, wherein, S1 includes: S11, generating a video feature tensor input time module generates a time affinity map through block-time multi-head self-attention (PT-MHSA) ; S12, map time affinity apply 2D convolution kernel to the last two dimensions perform feature compression to generate aggregated features and reshape them into time feature embeddings .

3. The method of claim 1, wherein, The S2 includes: S21, generating an affinity map by cross-attention (CA) computation ;​ S22, generating an affinity graph with video features in combination, generating feature maps and generating spatial feature embeddings by averaging operations .

4. The method of claim 1, wherein, The S3 includes: S31, employing a focal loss function computing a classification loss for the temporal branch; S32, using the FCG loss function within the InfoNCE framework. Force the model to learn facial component attribute alignment.

5. The method as described in claim 1, characterized in that, The S4 includes: S41, through the overall objective function Weighted fusion is performed, where Adjust the impact weight of FCG loss; S42, utilizing LayerNorm and Time characteristics and spatial features Standardize the process.

6. A generative facial video attack detection device guided by facial features, characterized in that, include: The spatiotemporal feature extraction module is used to extract the spatiotemporal features of faces in videos using the CLIP image decoder, and to analyze the temporal continuity between video frames through a multi-head self-attention mechanism to generate temporal feature embeddings. The spatial feature processing module combines the attention attributes of the CLIP image encoder with the preset facial component attributes, and uses a cross-attention module to process spatial distribution features to generate spatial feature embeddings. The classification loss calculation module is used to calculate the classification loss based on temporal feature embedding and spatial feature embedding through a multi-branch classifier, and to optimize feature alignment by introducing a facial component guided loss function. The loss fusion and classification module is used to weightedly fuse temporal loss, spatial loss, spatiotemporal joint loss and facial component guidance loss to generate the final video attack detection and classification result.

7. The apparatus as claimed in claim 6, characterized in that, The spatiotemporal feature extraction module is also used for: video feature tensor The input time module generates a time affinity map through block-time multi-head self-attention. ; Time affinity mapping The last two dimensions apply 2D convolution kernels Perform feature compression to generate aggregated features. And reshape it into a temporal feature embedding .

8. The apparatus as claimed in claim 6, characterized in that, The spatial feature processing module is also used for: Calculated via cross attention (CA) Generate affinity graph ; Affinity diagram With video features Combine to generate feature maps Spatial feature embeddings are generated through averaging operations. .

9. A computer device, characterized in that, Including processor and memory; The processor reads executable program code stored in the memory to run a program corresponding to the executable program code, so as to implement a generative portrait video attack detection method based on facial features as described in any one of claims 1-5.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements a generative facial video attack detection method based on facial features as described in any one of claims 1-5.

Images