Face forgery detection method and device based on asymmetric orthogonal hybrid expert model

Abstract

The application discloses a face forgery detection method and device based on an asymmetric orthogonal mixed expert model, and belongs to the technical field of image processing. The method comprises the following steps: obtaining a plurality of tokens corresponding to a face image to be detected; calculating the allocation probability of each dynamic expert corresponding to the tokens through a gating network; determining the target dynamic expert corresponding to the tokens based on the allocation probability; inputting the tokens into a frozen semantic subspace, a shared general subspace and the target dynamic expert respectively for processing, so as to obtain a first output, a second output and a third output; aggregating the obtained outputs to obtain an aggregated feature corresponding to the tokens; determining a comprehensive feature vector of the face image to be detected based on the aggregated features of the plurality of tokens; and inputting the comprehensive feature vector into a classification head for detection, so as to obtain a face forgery detection result. The method provided in the application can improve the detection accuracy of a forged face image.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a method and apparatus for detecting face forgery based on an asymmetric orthogonal hybrid expert model. Background Technology

[0002] With the continuous development of deepfake technology, face forgery detection is increasingly focusing on its ability to generalize to unknown manipulations. Recently, efficient parameter fine-tuning based on visual base models or large visual language models has shown significant potential in deepfake detection.

[0003] Existing detectors mainly employ a symmetric binary classification paradigm and rely on heuristic parameter fine-tuning methods, which inevitably leads to feature entanglement and overfitting to artifacts specific to the method, resulting in low accuracy of face forgery detection results. Summary of the Invention

[0004] The technical problem to be solved by this invention is that existing detectors mainly adopt a symmetric binary classification paradigm and rely on heuristic parameter fine-tuning methods, which inevitably leads to feature entanglement and overfitting to artifacts of specific methods, resulting in low accuracy of face forgery detection results. In order to solve the above problems, this invention provides a face forgery detection method and device based on an asymmetric orthogonal hybrid expert model.

[0005] The content of this invention includes: In a first aspect, embodiments of the present invention provide a face forgery detection method based on an asymmetric orthogonal hybrid expert model, comprising: The face image to be detected is segmented and encoded to obtain multiple tokens, each of which is used to characterize a region in the face image to be detected; The token is input into an orthogonal hybrid expert OMSD layer based on subspace decoupling for feature extraction to obtain the aggregated features of the token. The OMSD layer includes non-overlapping frozen semantic subspaces, shared general subspaces, and dynamic orthogonal subspaces. The dynamic orthogonal subspace includes multiple dynamic experts. The frozen semantic subspace is used to anchor the global structure of natural faces. The shared general subspace is used to capture cross-domain general face features. The dynamic experts are used to capture local specific forgery traces. The comprehensive feature vector of the face image to be detected is determined based on the aggregated features of multiple tokens; The comprehensive feature vector is input into the classification head for detection to obtain the face forgery detection result; The step of inputting the token into the OMSD layer for feature extraction to obtain the aggregated features of the token includes: The allocation probability of the token to each dynamic expert is calculated through a gating network; The target dynamic expert corresponding to the token is determined based on the allocation probability; The tokens are input into the frozen semantic subspace, the shared general subspace, and the target dynamic expert respectively for processing to obtain the first output, the second output, and the third output. The first output, the second output, and the third output are aggregated to obtain the aggregated feature corresponding to the token.

[0006] Optionally, before inputting the token into the OMSD layer for feature extraction to obtain the aggregated features of the token, the method further includes: Obtain the pre-trained weight matrix from the visual base model ; For the pre-trained weight matrix Perform singular value decomposition to obtain , and All are orthogonal matrices. Let be a diagonal matrix containing singular values ​​arranged in descending order. for transpose; Based on the results of the singular value decomposition, the largest value is selected. The feature components corresponding to each singular value are used to construct a frozen semantic subspace. It is a positive integer; Select the previous After each singular value component The shared general subspace is constructed by using the feature components corresponding to each singular value. It is a positive integer; Select the last remaining The dynamic orthogonal subspace is constructed by using the feature components corresponding to each singular value. It is a positive integer; The dynamic orthogonal subspace is divided into The aforementioned dynamic expert. It is a positive integer.

[0007] Optionally, calculating the allocation probability of the token for each dynamic expert through the gating network includes: The routing distribution corresponding to the token is calculated through the gating network, and the routing distribution includes... A vector of the nth value, the nth The value is used to characterize the _th _. The allocation probability corresponding to each dynamic expert. , The total number of dynamic experts; The allocation probability of each dynamic expert is determined based on the routing distribution; Among them, the tokens Corresponding route distribution for: ; in, Used to characterize the Softmax function Let be the learnable weight matrix of the gated network.

[0008] Optionally, determining the target dynamic expert corresponding to the token based on the allocation probability includes: Among the multiple dynamic experts, assign the one with the highest probability. One dynamic expert was identified as the target dynamic expert. It is a positive integer.

[0009] Optionally, the token is input into the target dynamic expert for processing to obtain a third output, including: Enter the tokens respectively The target dynamics expert is processed to obtain... One expert output; Based on the allocation probability, the The third output is obtained by weighted summation of the expert outputs.

[0010] Optionally, the face forgery detection model includes the OMSD layer and the classification head, and the loss function used to train the face forgery detection model includes asymmetric routing entropy loss. for: ； ； ; in, For the first tokens Corresponding route distribution Shannon entropy, The sample with the lowest Shannon entropy was the one being fabricated. A set of indexes for each token. As a preset value, The total number of dynamic experts. For the number of tokens, Used to characterize mathematical expectation, For real face sample set, To forge a facial sample set, The loss is for the real samples. Losses due to falsified samples.

[0011] Optionally, the face forgery detection model includes the OMSD layer and the classification head, and the loss function used to train the face forgery detection model includes the orthogonal subspace penalty loss. for: ; ; ; in, Let be the left singular vector matrix of all trainable experts in the residual space. Let be the right singular vector matrix of all trainable experts in the residual space. It is the identity matrix. The overall error used to characterize matrix differences Let be the left singular vector matrix of the shared universal subspace. For the first The left singular vector matrix of a dynamic expert Let be the right singular vector matrix of the shared universal subspace. For the first The right singular vector matrix of a dynamic expert , This represents the total number of dynamic experts.

[0012] Secondly, embodiments of the present invention also provide a face forgery detection device based on an asymmetric orthogonal hybrid expert model, comprising: The processing module is used to perform segmentation and encoding processing on the face image to be detected to obtain multiple tokens, each of which is used to characterize a region in the face image to be detected; The feature extraction module is used to input the token into an orthogonal hybrid expert OMSD layer based on subspace decoupling for feature extraction to obtain the aggregated features of the token. The OMSD layer includes non-overlapping frozen semantic subspaces, shared general subspaces, and dynamic orthogonal subspaces. The dynamic orthogonal subspace includes multiple dynamic experts. The frozen semantic subspace is used to anchor the global structure of natural faces. The shared general subspace is used to capture cross-domain general face features. The dynamic experts are used to capture local specific forgery traces. The determination module is used to determine the comprehensive feature vector of the face image to be detected based on the aggregated features of multiple tokens; The detection module is used to input the comprehensive feature vector into the classification head for detection and obtain the face forgery detection result; The feature extraction module includes: A computing unit is used to calculate the allocation probability of the token for each of the dynamic experts through a gating network; A determining unit is configured to determine the target dynamic expert corresponding to the token based on the allocation probability; The processing unit is used to input the token into the frozen semantic subspace, the shared general subspace, and the target dynamic expert respectively for processing, and obtain a first output, a second output, and a third output; An aggregation unit is used to aggregate the first output, the second output, and the third output to obtain the aggregated feature corresponding to the token.

[0013] Thirdly, embodiments of the present invention provide an electronic device, including: a memory, a processor, and a program stored in the memory and executable on the processor; the processor is configured to read the program in the memory to implement the steps in the face forgery detection method based on an asymmetric orthogonal hybrid expert model as described in the first aspect.

[0014] Fourthly, embodiments of the present invention provide a readable storage medium for storing a program, which, when executed by a processor, implements the steps in the face forgery detection method based on an asymmetric orthogonal hybrid expert model as described in the first aspect.

[0015] The beneficial effects of this invention are as follows: First, by freezing the semantic subspace, the global structure of natural faces can be effectively anchored, preventing the model from forgetting basic prior knowledge of real faces during training. Second, by sharing a general subspace to focus on capturing cross-domain general face features, the robustness of the model in different scenarios is improved. Third, the dynamic orthogonal subspace accurately captures specific local forgery traces through multiple dynamic experts. This decoupled design not only ensures the stability of basic features but also greatly enhances the model's generalization detection capability against unknown or novel forgery techniques. Fourth, by combining a gating network with a dynamic routing mechanism that allocates probabilities, intelligent on-demand allocation of computing resources is achieved. By calculating the allocation probability of each dynamic expert corresponding to a token and determining the target dynamic expert accordingly, the model can adaptively focus on the most discriminative region in the image, avoiding the waste of computing power and noise interference caused by the indiscriminate processing of all regions in traditional methods. Finally, the first output of the frozen semantic subspace, the second output of the shared general subspace, and the third output of the target dynamic expert are aggregated to make the final comprehensive feature vector have the integrity of natural images, the universality of cross-domain features, and the sensitivity to specific forgery traces, thereby significantly improving the accuracy and anti-interference ability of face forgery detection. Attached Figure Description

[0016] Figure 1A flowchart of a face forgery detection method based on an asymmetric orthogonal hybrid expert model provided in an embodiment of the present invention; Figure 2 A schematic diagram of the structure of the Orthogonal MoEvia Subspace Decoupling (OMSD) layer provided in an embodiment of the present invention; Figure 3 A schematic diagram illustrating the calculation of asymmetric routing entropy loss provided in an embodiment of the present invention; Figure 4 A schematic diagram illustrating the calculation of orthogonal subspace penalty loss provided in an embodiment of the present invention; Figure 5 A schematic diagram of the overall architecture for face forgery detection based on an asymmetric orthogonal hybrid expert model provided in an embodiment of the present invention; Figure 6 A schematic diagram of a face forgery detection device based on an asymmetric orthogonal hybrid expert model provided in an embodiment of the present invention; Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0017] In the embodiments of this application, the term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship. In the embodiments of this application, the term "multiple" refers to two or more, and other quantifiers are similar. The terms "first," "second," etc., in the specification of this application are used to distinguish similar objects and are not used to describe a specific order or sequence. It should be understood that such terms can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first" and "second" are usually of the same class, without limiting the number of objects. For example, the first object can be one or multiple.

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0020] This application provides a face forgery detection method and apparatus based on an asymmetric orthogonal hybrid expert model, which aims to improve the accuracy of face forgery detection results and achieve state-of-the-art cross-dataset generalization performance with a very small number of trainable parameters.

[0021] Please see Figure 1 , Figure 1 This is a flowchart illustrating the face forgery detection method based on an asymmetric orthogonal hybrid expert model provided in an embodiment of the present invention. The method specifically includes the following steps: Step 101: Perform segmentation and encoding processing on the face image to be detected to obtain multiple tokens, each of which is used to represent a region in the face image to be detected.

[0022] Step 102: Input the token into an Orthogonal MoE via Subspace Decoupling (OMSD) layer for feature extraction to obtain the aggregated features of the token. The OMSD layer includes non-overlapping frozen semantic subspaces, shared general subspaces, and dynamic orthogonal subspaces. The dynamic orthogonal subspace includes multiple dynamic experts. The frozen semantic subspace is used to anchor the global structure of natural faces. The shared general subspace is used to capture cross-domain general face features. The dynamic experts are used to capture local specific forgery traces.

[0023] Step 103: Determine the comprehensive feature vector of the face image to be detected based on the aggregated features of multiple tokens.

[0024] Step 104: Input the comprehensive feature vector into the classification head for detection to obtain the face forgery detection result.

[0025] In this embodiment, the OMSD layer includes a non-overlapping frozen semantic subspace, a shared general subspace, and a dynamic orthogonal subspace. Specifically, principal components with high-energy singular values ​​are strictly preserved as the frozen semantic basis to anchor the global structure of the natural image, the remaining low-rank residual components are reconstructed into a shared general expert to capture cross-domain forgery commonalities, and multiple dynamic experts serve as catchers of method-specific artifacts.

[0026] Optionally, in some embodiments, before step 102, the method further includes: Obtain the pre-trained weight matrix from the visual base model ; For the pre-trained weight matrix Perform singular value decomposition to obtain , and All are orthogonal matrices. Let be a diagonal matrix containing singular values ​​arranged in descending order. for transpose; Based on the results of the singular value decomposition, the largest value is selected. The feature components corresponding to each singular value are used to construct a frozen semantic subspace. It is a positive integer; Select the previous After each singular value component The shared general subspace is constructed by using the feature components corresponding to each singular value. It is a positive integer; Select the last remaining The dynamic orthogonal subspace is constructed by using the feature components corresponding to each singular value. It is a positive integer; The dynamic orthogonal subspace is divided into The aforementioned dynamic expert. It is a positive integer.

[0027] set up This is a pre-trained weight matrix in the Vision Foundation Model (VFM), whose parameters are obtained by directly loading publicly available pre-trained weights. Its spectral decomposition can be obtained by performing Singular Value Decomposition (SVD) on it. ,in, To output feature dimensions, The input feature dimension.

[0028] It should be understood that VFM is a visual model pre-trained on a large-scale dataset with powerful general feature extraction capabilities, and its specific structure is not limited here. For example, in some embodiments, VFM is a visual model obtained after training based on Contrastive Language-Image Pre-training (CLIP), and in other embodiments, VFM is a visual model obtained after training based on Distillation with no labels (DINO).

[0029] Please see Figure 2 Based on singular value spectra, using index sets , , (ranks are respectively) , , The parameters are divided into three disjoint subspaces: the frozen semantic subspace, the shared general subspace, and the dynamic orthogonal subspace.

[0030] To preserve the basic semantics and topological structure of natural facial features, the largest anterior segment after SVD decomposition is selected. Each singular value component is used to construct a frozen semantic subspace. The largest singular value corresponds to the principal direction of the data variance. Specifically, before extraction... The main components constitute the fixed base. ,in, , , Left singular vector matrix The former List, A right singular vector matrix The former List, For the front A diagonal matrix consisting of the largest singular values.

[0031] It should be understood that during training, the parameters of the frozen semantic subspace are frozen and do not participate in any updates, thus achieving strict freezing during training. During fine-tuning training for face forgery detection, the model does not disrupt the original feature extraction backbone, effectively avoiding catastrophic forgetting. This ensures that when faced with forged images, the model can still accurately understand basic information such as facial structure, lighting, and texture, providing a solid semantic foundation for subsequent discrimination.

[0032] Select the middle size after SVD decomposition Each singular value component is used to construct a shared general subspace. In this embodiment, the subsequent... A singular component constructs a general domain adapter. The physical motivation behind this is to establish a transition buffer that bridges the domain gap between general pre-trained data and general facial priors, while absorbing perturbations unrelated to forgery (such as JPEG compression or lighting variations).

[0033] By constructing the shared general subspace, the model can be appropriately fine-tuned and adapted to the general features based on the data distribution of the current forgery detection task. This ensures the model's adaptability to new tasks and avoids drastic fluctuations in model parameters by limiting the adjustment range to the intermediate frequency band, thus maintaining the stability of training.

[0034] In some embodiments, for input tokens Output of shared common subspace for: ; in, These are trainable parameters initialized from SVD residuals. Left singular vector matrix The middle List, A right singular vector matrix The middle List, as the middle A diagonal matrix consisting of singular values, this part of the data is trainable and universally activates all input tokens.

[0035] Select the last one after SVD decomposition. Each singular value component is used to construct a dynamic orthogonal subspace. This space models high-frequency local variance, making it an ideal submanifold for capturing hidden, method-specific artifacts. Furthermore, the dynamically orthogonal subspace is divided into... A dynamic expert Each dynamic expert consists of trainable low-rank triples. Parameterization. Specifically, Used to characterize the A dynamic expert Left singular vector matrix Assigned to the A dynamic expert's part. A right singular vector matrix Assigned to the A dynamic expert's part. This is the corresponding diagonal matrix.

[0036] In this embodiment, the learning of forged features is restricted to a dynamic orthogonal subspace, which achieves orthogonal decoupling between forged features and basic semantic features. This means that while the model enhances its detection capabilities, it does not interfere with the understanding of normal semantics, thus significantly reducing the false alarm rate.

[0037] In this embodiment, step 102 includes: The allocation probability of the token to each dynamic expert is calculated through a gating network; The target dynamic expert corresponding to the token is determined based on the allocation probability; The tokens are input into the frozen semantic subspace, the shared general subspace, and the target dynamic expert respectively for processing to obtain the first output, the second output, and the third output. The first output, the second output, and the third output are aggregated to obtain the aggregated feature corresponding to the token.

[0038] It should be understood that the forged images are primarily realistic, and the anomalies are usually limited to specific areas (e.g., blending boundaries in face swapping or distorted pupils in images generated by Generative Adversarial Networks (GANs)). Based on the specific characteristics of the forged faces described above, this embodiment introduces a token-level routing mechanism. This mechanism uses a gating network to determine the allocation probability of a token for each dynamic expert, thereby identifying the target dynamic expert that the token should activate.

[0039] It should be understood that the number of target dynamic experts is not limited here. For example, in some embodiments, the dynamic expert with the highest assignment probability is determined as the target dynamic expert. In other embodiments, dynamic experts with an assignment probability greater than a threshold are determined as target dynamic experts.

[0040] Optionally, in other embodiments, determining the target dynamic expert corresponding to the token based on the allocation probability includes: Among the multiple dynamic experts, assign the one with the highest probability. One dynamic expert was identified as the target dynamic expert. It is a positive integer.

[0041] In this embodiment, a Top-k strategy is adopted to select the assignment with the highest probability from multiple dynamic networks. Activating multiple dynamic experts as target dynamic experts, this multi-perspective feature fusion mechanism provides richer information than a single expert, effectively avoiding missed detections due to errors in single expert judgment and significantly improving detection accuracy. Simultaneously, this mechanism enhances the model's robustness against unknown forgery methods or low-quality images, preventing drastic fluctuations in model output.

[0042] Optionally, in some embodiments, calculating the allocation probability of the token corresponding to each dynamic expert through the gating network includes: The routing distribution corresponding to the token is calculated through the gating network, and the routing distribution includes... A vector of the nth value, the nth The value is used to characterize the _th _. The allocation probability corresponding to each dynamic expert. , The total number of dynamic experts; The allocation probability of each dynamic expert is determined based on the routing distribution; Among them, the tokens Corresponding route distribution for: ; in, Used to characterize the Softmax function Let be the learnable weight matrix of the gated network.

[0043] First, through a learnable weight matrix With the entered token Perform matrix multiplication ( This step is equivalent to assigning a raw score (Logits) to the current token in front of each dynamic expert, outputting an unnormalized vector of real numbers. Next, this raw score vector is input into the Softmax function, which outputs... It is a collection A vector of values, where each value represents the probability or importance of the corresponding dynamic expert processing the current token, and the sum of the allocation probabilities of all dynamic experts is 1.

[0044] It should be understood that, The weight matrix is ​​learnable, allowing the gating network to automatically adjust its weight parameters based on feedback from the detection task. This data-driven adaptive learning makes the routing strategy increasingly accurate as training progresses, eliminating the need for manually designing complex routing rules.

[0045] In this embodiment, by calculating the allocation probability, the model can automatically calculate the high matching probability of the corresponding expert for input containing specific forgery features, thereby accurately activating that expert for processing. This content-based dynamic routing mechanism ensures that computing resources are allocated to the module best suited to process the current feature, significantly improving the targeting and effectiveness of feature extraction.

[0046] Optionally, in some embodiments, the token is input into the target dynamic expert for processing to obtain a third output, including: Enter the tokens respectively The target dynamics expert is processed to obtain... One expert output; Based on the allocation probability, the The third output is obtained by weighted summation of the expert outputs.

[0047] In this embodiment, the outputs of multiple dynamic experts are summed based on assigned probabilities. This weighted summation, based on the assigned probabilities, results in a continuous combination of the outputs from multiple experts. This means that even if the input data undergoes minor changes or falls at the boundary between the abilities of two experts, the model's output can maintain a smooth transition. This continuity greatly improves the model's stability during the inference phase and enhances its generalization ability when facing unknown or mutated forged samples. Simultaneously, this mechanism ensures that even with the introduction of... For each target dynamic expert, the noise features generated by those experts who are irrelevant or have low matching degree will be greatly weakened because their corresponding assignment probability is extremely low (close to 0), thus ensuring the purity and accuracy of the third output.

[0048] As a specific embodiment, the first tokens The corresponding third output It can be represented as: ; ; in, For the first tokens , No. The output of a dynamic expert Used for characterization Index of dynamic experts for each target. For the first tokens , No. The allocation probability corresponding to each dynamic expert.

[0049] Furthermore, in this embodiment, the first tokens Corresponding aggregate features for: ; in, This is the first output corresponding to the frozen semantic subspace. To share the second output corresponding to the common subspace, the above method ensures that the real background region and the heterogeneous tampered region in the same image can be projected into completely different orthogonal submanifolds without feature entanglement.

[0050] Deepfake detection requires an imbalanced, asymmetric allocation strategy that depends on the authenticity of the input. To explicitly inject this forensic prior into the gating network, optionally, in some embodiments, the face forgery detection model includes an OMSD layer and a classification head, with the loss function used to train the OMSD layer and classification head including an asymmetric routing entropy loss. .

[0051] like Figure 3 As shown, let For the first Token routing distribution Shannon entropy: ; Real faces are sampled from genuine natural image manifolds and contain no forgery artifacts. Therefore, gating networks should follow the maximum entropy principle, consciously avoiding activation of any specific anomaly detection experts to prevent phantom false positives. Since real faces lack manipulation cues, forcing gating networks to make confident assignments will inevitably lead to overfitting to identity or background features.

[0052] It should be understood that most areas of existing fake face images are real, interspersed with localized anomalous regions. Therefore, the gating network should not exhibit global confidence across the entire image. Instead, in this embodiment, absolute certainty is enforced only on the most suspicious regions, dynamically filtering for regions containing anomalous features by tracking the entropy map of the fake samples. The index set of the lowest entropy tokens (Representing the regions most likely to contain artifacts), then sharpen their distribution to obtain the loss for the fake samples. : ; This local sharpness constraint explicitly allows the network to ignore real background tokens in forged images and forces the gating network to make decisive hard assignments to tampered areas, routing specific anomaly types to their designated experts.

[0053] Total asymmetric routing entropy loss This reconciles the opposing physical requirements of the two domains, formalized as an asymmetric expectation on the data distribution: ; ; in, The loss is for the real samples. The number of tokens. For real face sample set, Used to forge facial sample sets.

[0054] Although spectral decomposition via SVD imparts perfect geometric orthogonality to the parameter space at initialization, standard backpropagation dynamics in unconstrained Euclidean space inevitably undermines this fragile mathematical structure. Gradient updates tend to pull the weight vectors toward the steepest descent directions, which are often correlated across different experts. As training progresses, the weight vectors become entangled and homogenized, a phenomenon known as orthogonal drift. Without constraints, the dynamic experts will lazily collapse into overlapping representations, losing their ability to independently capture heterogeneous forgery cues.

[0055] Please see Figure 4 The orthogonal subspace penalty loss is used to regularize the expert subspace to prevent orthogonal drift and feature overlap during training. Let and The summation basis of all trainable experts in the residual space, and the orthogonal subspace penalty loss. for: ; in, Let be the left singular vector matrix of all trainable experts in the residual space. Let be the right singular vector matrix of all trainable experts in the residual space. It is the identity matrix. The overall error used to characterize matrix differences Let be the left singular vector matrix of the shared universal subspace. For the first The left singular vector matrix of a dynamic expert Let be the right singular vector matrix of the shared universal subspace. For the first The right singular vector matrix of a dynamic expert.

[0056] In this embodiment, the differentiable penalty applies a two-level structure regularization. Firstly, by forcing... This explicitly prevents rank collapse within a single dynamic expert, ensuring that each expert maintains its full multidimensional capacity to model complex generative artifacts without shrinking to a low-rank subspace. Secondly, by penalizing cross terms (e.g.... It guarantees strict orthogonality: This acts as a structural firewall; it ensures that dynamic experts only react to anomalies of specific methods in the orthogonal complement of general facial semantics, thereby preventing false alarms triggered by common degradations such as JPEG compression.

[0057] As a specific implementation, the final training objective unifies the standard cross-entropy classification loss with structure-aware constraints, resulting in a total loss function. for: .

[0058] in, To control asymmetric routing entropy loss The intensity parameter, for example, takes a value of 0.1. For use in controlling orthogonal subspace penalty The intensity parameter, for example, takes a value of 0.1. The standard cross-entropy classification loss is used, and its specific calculation method will not be elaborated here.

[0059] As a specific example, such as Figure 5 As shown, CLIPViT-L / 14 is used as the visual backbone network to leverage its robust cross-modal representation. To ensure efficient parameter fine-tuning, the backbone network parameters are kept frozen; in this embodiment, the OMSD layer is integrated only into the linear projection layer of the self-attention module. In this embodiment, the OMSD layer divides the residual subspace into five parts, including one shared general subspace for capturing general forensic traces and four dynamic experts for multi-scale forgery modeling. For each input token, the routing mechanism dynamically activates the two dynamic experts with the highest gating weights (i.e., the highest probability of allocation) as the target dynamic experts.

[0060] like Figure 5 As shown, the face image to be detected is first processed by the Patch embedding model, which divides the input face image into multiple fixed-size image blocks and maps these image blocks into high-dimensional vectors to obtain multiple tokens. Each token corresponds to a local region in the image and carries the visual information of that region.

[0061] Then, multiple tokens are processed through N OMSD layers to obtain the aggregated features of each token. Based on the subspace decoupling principle, the OMSD layer contains three non-overlapping subspaces: a frozen semantic subspace, used to extract and anchor the global structural features of natural faces, providing stable prior knowledge; a shared general subspace, used to capture general face features across different forgery methods, improving the model's generalization ability; and a dynamic orthogonal subspace, containing multiple dynamic experts, which intelligently select target dynamic experts to capture local, specific types of forgery traces by calculating allocation probabilities through a gating network. The outputs of the above three subspaces are aggregated to obtain aggregated features that integrate global, general, and specific forgery information. The aggregated features are processed through a self-attention layer to capture long-distance dependencies between different image patches (tokens), understanding the relationships between different parts of the face. Finally, a feedforward network is used to further non-linearly transform and enhance the features. After N layers of deep processing, each token contains rich and refined semantic information.

[0062] After deep feature extraction, the information scattered across the various tokens needs to be integrated into a unified image-level representation. Based on the aggregated features of multiple tokens, a comprehensive feature vector representing the entire face image is generated through global average pooling or other aggregation methods.

[0063] Finally, the comprehensive feature vector of the face image to be detected is input into a pre-trained classifier. The classifier analyzes the feature vector and outputs a probability value or directly gives the face forgery detection result, that is, determines whether the input image is "real" or "forged".

[0064] like Figure 6 As shown, this embodiment of the invention also provides a face forgery detection device 600 based on an asymmetric orthogonal hybrid expert model, comprising: Processing module 601 is used to perform segmentation and encoding processing on the face image to be detected to obtain multiple tokens, each of which is used to characterize a region in the face image to be detected; The feature extraction module 602 is used to input the token into an orthogonal hybrid expert OMSD layer based on subspace decoupling for feature extraction to obtain the aggregated features of the token. The OMSD layer includes non-overlapping frozen semantic subspaces, shared general subspaces, and dynamic orthogonal subspaces. The dynamic orthogonal subspaces include multiple dynamic experts. The frozen semantic subspace is used to anchor the global structure of natural faces. The shared general subspace is used to capture cross-domain general face features. The dynamic experts are used to capture local specific forgery traces. The determining module 603 is used to determine the comprehensive feature vector of the face image to be detected based on the aggregated features of multiple tokens; Detection module 604 is used to input the comprehensive feature vector into the classification head for detection and obtain the face forgery detection result; The feature extraction module 602 includes: A computing unit is used to calculate the allocation probability of the token for each of the dynamic experts through a gating network; A determining unit is configured to determine the target dynamic expert corresponding to the token based on the allocation probability; The processing unit is used to input the token into the frozen semantic subspace, the shared general subspace, and the target dynamic expert respectively for processing, and obtain a first output, a second output, and a third output; An aggregation unit is used to aggregate the first output, the second output, and the third output to obtain the aggregated feature corresponding to the token.

[0065] Optionally, the face forgery detection device 600 based on an asymmetric orthogonal hybrid expert model further includes: The acquisition module is used to obtain the pre-trained weight matrix from the visual base model. ; The decomposition module is used to decompose the pre-trained weight matrix. Perform singular value decomposition to obtain , and All are orthogonal matrices. Let be a diagonal matrix containing singular values ​​arranged in descending order. for transpose; The first construction module is used to select the largest [value] based on the result of the singular value decomposition. The feature components corresponding to each singular value are used to construct a frozen semantic subspace. It is a positive integer; The second construction module is used to select the preceding... After each singular value component The shared general subspace is constructed by using the feature components corresponding to each singular value. It is a positive integer; The third building module is used to select the last remaining... The dynamic orthogonal subspace is constructed by using the feature components corresponding to each singular value. It is a positive integer; The partitioning module is used to partition the dynamic orthogonal subspace into The aforementioned dynamic expert. It is a positive integer.

[0066] Optionally, calculating the allocation probability of the token for each dynamic expert through the gating network includes: The routing distribution corresponding to the token is calculated through the gating network, and the routing distribution includes... A vector of the nth value, the nth The value is used to characterize the _th _. The allocation probability corresponding to each dynamic expert. , The total number of dynamic experts; The allocation probability of each dynamic expert is determined based on the routing distribution; Among them, the tokens Corresponding route distribution for: ; in, Used to characterize the Softmax function Let be the learnable weight matrix of the gated network.

[0067] Optionally, determining the target dynamic expert corresponding to the token based on the allocation probability includes: Among the multiple dynamic experts, assign the one with the highest probability. One dynamic expert was identified as the target dynamic expert. It is a positive integer.

[0068] Optionally, the token is input into the target dynamic expert for processing to obtain a third output, including: Enter the tokens respectively The target dynamics expert is processed to obtain... One expert output; Based on the allocation probability, the The third output is obtained by weighted summation of the expert outputs.

[0069] Optionally, the face forgery detection model includes the OMSD layer and the classification head, and the loss function used to train the face forgery detection model includes asymmetric routing entropy loss. for: ; ; ; in, For the first tokens Corresponding route distribution Shannon entropy, The sample with the lowest Shannon entropy was the one being fabricated. A set of indexes for each token. As a preset value, The total number of dynamic experts. For the number of tokens, Used to characterize mathematical expectation, Used to represent a set of real human face samples. Used to characterize a set of fake face samples. The loss is for the real samples. Losses due to falsified samples.

[0070] Optionally, the face forgery detection model includes the OMSD layer and the classification head, and the loss function used to train the face forgery detection model includes the orthogonal subspace penalty loss. for: ; ; ; in, Let be the left singular vector matrix of all trainable experts in the residual space. Let be the right singular vector matrix of all trainable experts in the residual space. It is the identity matrix. The overall error used to characterize matrix differences Let be the left singular vector matrix of the shared universal subspace. For the first The left singular vector matrix of a dynamic expert Let be the right singular vector matrix of the shared universal subspace. For the first The right singular vector matrix of a dynamic expert , This represents the total number of dynamic experts.

[0071] The face forgery detection device 600 based on an asymmetric orthogonal hybrid expert model provided in this application embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, so it will not be described again here.

[0072] It should be noted that the division of units in the embodiments of this application is illustrative and only represents one logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units.

[0073] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a processor-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0074] like Figure 7As shown, this application provides an electronic device 700, including: a memory 702, a processor 701, and a program stored in the memory 702 and executable on the processor 701; the processor 701 is used to read the program in the memory 702 to implement the steps in the face forgery detection method based on the asymmetric orthogonal hybrid expert model as described above.

[0075] This application also provides a readable storage medium storing a program that, when executed by a processor, implements the various processes of the above-described face forgery detection method based on an asymmetric orthogonal hybrid expert model, and achieves the same technical effect. To avoid repetition, it will not be described again here. The readable storage medium can be any available medium or data storage device that the processor can access, including but not limited to magnetic storage (such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MOs), etc.), optical storage (such as compact disks (CDs), digital video discs (DVDs), Blu-ray discs (BDs), high-definition universal discs (HVDs), etc.), and semiconductor storage (such as read-only memory (ROMs), erasable programmable read-only memory (EPROMs), electrically erasable programmable read-only memory (EEPROMs), non-volatile memory (NAND flash), solid-state drives (SSDs)).

[0076] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0077] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0078] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other modifications under the guidance of this application without departing from its spirit, and all of these modifications are within the scope of protection of this application.

Claims

1. A face forgery detection method based on an asymmetric orthogonal hybrid expert model, characterized in that, include: The face image to be detected is segmented and encoded to obtain multiple tokens, each of which is used to characterize a region in the face image to be detected; The token is input into an orthogonal hybrid expert OMSD layer based on subspace decoupling for feature extraction to obtain the aggregated features of the token. The OMSD layer includes non-overlapping frozen semantic subspaces, shared general subspaces, and dynamic orthogonal subspaces. The dynamic orthogonal subspace includes multiple dynamic experts. The frozen semantic subspace is used to anchor the global structure of natural faces. The shared general subspace is used to capture cross-domain general face features. The dynamic experts are used to capture local specific forgery traces. The comprehensive feature vector of the face image to be detected is determined based on the aggregated features of multiple tokens; The comprehensive feature vector is input into the classification head for detection to obtain the face forgery detection result; The step of inputting the token into the OMSD layer for feature extraction to obtain the aggregated features of the token includes: The allocation probability of the token to each dynamic expert is calculated through a gating network; The target dynamic expert corresponding to the token is determined based on the allocation probability; The tokens are input into the frozen semantic subspace, the shared general subspace, and the target dynamic expert respectively for processing to obtain the first output, the second output, and the third output. The first output, the second output, and the third output are aggregated to obtain the aggregated feature corresponding to the token.

2. The face forgery detection method based on an asymmetric orthogonal hybrid expert model according to claim 1, characterized in that, Before inputting the token into the OMSD layer for feature extraction to obtain the aggregated features of the token, the method further includes: Obtain the pre-trained weight matrix from the visual base model ; For the pre-trained weight matrix Perform singular value decomposition to obtain , and All are orthogonal matrices. Let be a diagonal matrix containing singular values ​​arranged in descending order. for Transpose of; Based on the results of the singular value decomposition, the largest value is selected. The feature components corresponding to each singular value are used to construct a frozen semantic subspace. It is a positive integer; Select the previous After each singular value component The shared general subspace is constructed by using the feature components corresponding to each singular value. It is a positive integer; Select the last remaining The feature components corresponding to each singular value are used to construct the dynamic orthogonal subspace. It is a positive integer; The dynamic orthogonal subspace is divided into The aforementioned dynamic expert. It is a positive integer.

3. The face forgery detection method based on an asymmetric orthogonal hybrid expert model according to claim 1, characterized in that, The step of calculating the allocation probability of the token for each dynamic expert through the gating network includes: The routing distribution corresponding to the token is calculated through the gating network, and the routing distribution includes... A vector of the nth value, the nth The value is used to characterize the _th _. The allocation probability corresponding to each dynamic expert. , The total number of dynamic experts; The allocation probability of each dynamic expert is determined based on the routing distribution; Among them, the tokens Corresponding route distribution for: ； in, Used to characterize the Softmax function Let be the learnable weight matrix of the gated network.

4. The face forgery detection method based on an asymmetric orthogonal hybrid expert model according to claim 1, characterized in that, The step of determining the target dynamic expert corresponding to the token based on the allocation probability includes: Among the multiple dynamic experts, assign the one with the highest probability. One dynamic expert was identified as the target dynamic expert. It is a positive integer.

5. The face forgery detection method based on an asymmetric orthogonal hybrid expert model according to claim 4, characterized in that, The token is input into the target dynamic expert for processing, resulting in a third output, including: Enter the tokens respectively The target dynamics expert is processed to obtain... One expert output; Based on the allocation probability, the The third output is obtained by weighted summation of the expert outputs.

6. The face forgery detection method based on an asymmetric orthogonal hybrid expert model according to claim 1, characterized in that, The face forgery detection model includes the OMSD layer and the classification head. The loss function used to train the face forgery detection model includes asymmetric routing entropy loss. for: ； ； ； in, For the first tokens Corresponding route distribution Shannon entropy, The sample with the lowest Shannon entropy was the one being forged. A set of indexes for each token. As a preset value, The total number of dynamic experts. For the number of tokens, Used to characterize mathematical expectation, This is a real face sample set. To forge a facial sample set, The loss is for the real samples. Losses due to falsified samples.

7. The face forgery detection method based on an asymmetric orthogonal hybrid expert model according to claim 1, characterized in that, The face forgery detection model includes the OMSD layer and the classification head. The loss function used to train the face forgery detection model includes the orthogonal subspace penalty loss. for: ； ； ； in, Let be the left singular vector matrix of all trainable experts in the residual space. Let be the right singular vector matrix of all trainable experts in the residual space. It is the identity matrix. The overall error used to characterize matrix differences Let be the left singular vector matrix of the shared universal subspace. For the first The left singular vector matrix of a dynamic expert Let be the right singular vector matrix of the shared universal subspace. For the first The right singular vector matrix of a dynamic expert , This represents the total number of dynamic experts.

8. A face forgery detection device based on an asymmetric orthogonal hybrid expert model, characterized in that, include: The processing module is used to perform segmentation and encoding processing on the face image to be detected to obtain multiple tokens, each of which is used to characterize a region in the face image to be detected; The feature extraction module is used to input the token into an orthogonal hybrid expert OMSD layer based on subspace decoupling for feature extraction to obtain the aggregated features of the token. The OMSD layer includes non-overlapping frozen semantic subspaces, shared general subspaces, and dynamic orthogonal subspaces. The dynamic orthogonal subspace includes multiple dynamic experts. The frozen semantic subspace is used to anchor the global structure of natural faces. The shared general subspace is used to capture cross-domain general face features. The dynamic experts are used to capture local specific forgery traces. The determination module is used to determine the comprehensive feature vector of the face image to be detected based on the aggregated features of multiple tokens; The detection module is used to input the comprehensive feature vector into the classification head for detection and obtain the face forgery detection result; The feature extraction module includes: A computing unit is used to calculate the allocation probability of the token for each of the dynamic experts through a gating network; A determining unit is configured to determine the target dynamic expert corresponding to the token based on the allocation probability; The processing unit is used to input the token into the frozen semantic subspace, the shared general subspace, and the target dynamic expert respectively for processing, and obtain a first output, a second output, and a third output; An aggregation unit is used to aggregate the first output, the second output, and the third output to obtain the aggregated feature corresponding to the token.

9. An electronic device, comprising: A memory, a processor, and a program stored in the memory and executable on the processor; characterized in that the processor is configured to read the program in the memory to implement the steps in the face forgery detection method based on an asymmetric orthogonal hybrid expert model as described in any one of claims 1 to 7.

10. A readable storage medium for storing a program, characterized in that, When the program is executed by the processor, it implements the steps in the face forgery detection method based on an asymmetric orthogonal hybrid expert model as described in any one of claims 1 to 7.

Images