Face detection method and system based on cross-modal prompt tuning

Abstract

The invention belongs to the technical field of artificial intelligence and information security, and particularly relates to a face detection method and system based on cross-modal prompt tuning. Comprising the following steps: generating a class specific prompt on a text branch through synonym and semantic extension through a preprocessed face image, projecting and injecting the class specific prompt into a visual encoder, extracting a high-frequency counterfeit clue on a visual branch, projecting and injecting the high-frequency counterfeit clue into a text encoder, and forming bidirectional prompt interaction between a text and vision; and obtaining the joint representation in the shared discrimination space under the multi-level cooperative constraint, and outputting true / false categories and confidence. According to the method, two attacks of physical presentation and digital forgery can be processed at the same time under a unified framework, and the adaptability and robustness in a complex scene are improved.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and information security technology, and specifically relates to a face detection method and system based on cross-modal prompting optimization. Background Technology

[0002] Facial recognition systems mainly face two types of attack threats: the first is physical representation attacks, such as printed photos, video playback, and 3D masks, which occur before image acquisition; the second is digital deepfake attacks, such as face swapping, attribute editing, and face synthesis, which are generated digitally after image acquisition.

[0003] Traditionally, the two types of problems mentioned above have been handled independently by Face Anti-Spoofing (FAS) and DeepFake Detection (DFD) methods, leading to increased model deployment and computational costs. Furthermore, since both types of attacks originate from different forgery paths of "live faces," they share a common discrimination space. Therefore, exploring Unified Face Attack Detection (UAD) that simultaneously detects both methods is of great significance. However, directly applying a general vision-language pre-trained model (VLM) to UAD often yields unsatisfactory results, mainly due to the unique challenges of the two tasks: (1) Category semantic ambiguity: UAD is usually represented in a binary classification form. The category labels "live / real" and "fake / fake" are not concrete entities but subjective concepts, which makes it difficult to align the general text and image semantics of VLM. (2) Diversity of forgery: There are significant differences in the distribution of evidence and visual cues between physical and digital forgery, making it difficult for linguistic modalities to provide a stable classification decision boundary for all types of forgery.

[0004] Therefore, this invention proposes a face detection method and system based on cross-modal prompting optimization to solve the problems in the prior art, such as separate modeling of face physical representation attacks and digital deepfake attacks, insufficient cross-modal semantic alignment, and weak generalization ability to unknown domains. Summary of the Invention

[0005] The purpose of this invention is to provide a face detection method and system based on cross-modal cue optimization. This invention improves cross-modal alignment performance through semantic enhancement and frequency domain cue modeling, so as to distinguish multiple forgery types in a unified discrimination space and improve robustness across datasets.

[0006] To address the aforementioned technical problems, this invention provides a face detection method based on cross-modal cue optimization, comprising: By acquiring facial images and performing alignment, cropping, and normalization processing; By performing synonym retrieval and aggregation of true and false categories based on the thesaurus and vocabulary at the encoding layer of one or more Transformer modules in the text branch, class-specific text prompts are generated. The class-specific text prompts are mapped via a text-visual cross-modal projection module to an injection signal aligned with the dimensions and semantics of the corresponding visual coding layer, and then injected at one or more levels of the visual encoder. By performing frequency domain transformation on visual features and applying a high-frequency mask in one or more coding layers of the visual branch, high-frequency features associated with forgery are obtained and refined into high-frequency cues by a lightweight network. The high-frequency cues are mapped to bias or additional cues of text branches via a visual-text cross-modal projection module and injected at one or more levels of the text encoder; By obtaining a joint representation in a shared discriminative space under the bidirectional cueing constraints of the class-specific text cue and the high-frequency cue, and by selectively introducing contrast and alignment losses to stabilize cross-modal alignment; The true / false category and confidence level are output based on the joint representation.

[0007] Preferably, the process of forming the text branch includes: Word segmentation and word embedding: The template text is segmented into words to obtain an initial word embedding matrix. ,in For the total length of word embedding, For text embedding dimensions; Hierarchical encoding, i-th layer text Transformer module Output to the previous layer Encode, obtain ; Text representation and projection: The final layer's terminal word embedding or full sequence aggregation is taken as the text representation t, which is then projected through the text-side projection layer. Mapped to the shared space, resulting in .

[0008] Preferably, the formation process of the visual branch includes: Image segmentation and embedding, taking the input image The image is divided into fixed-size blocks, and the initial image block embedding is obtained by linear mapping. , where is the visual embedding dimension and n is the number of image patches; Category and starting label: Learnable class labels are inserted before the visual input at each layer. The i-th layer visual input is obtained by embedding and splicing it with the previous layer block. ; Encoding and Reading, i-th Layer Visual Transformer Module right Encode to obtain the (i+1)th layer visual Transformer module. Input The class label output of the last layer is taken as the visual feature. ; Projected into the shared space, through the visual side projection layer Mapping V to a shared graph-text embedding space yields... .

[0009] Preferably, the synonym retrieval and aggregation is performed by selecting the top h most similar synonyms from a fixed vocabulary, pre-trained word vectors, a knowledge base, or candidates generated by a language model and then weighting and aggregating them; where h is an adjustable hyperparameter.

[0010] Preferably, both the text-visual cross-modal projection module and the visual-text cross-modal projection module include one or more of linear mapping, multilayer perceptron (MLP), convolution, attention or cross-attention, and adapter or LoRA, with parameters that can be shared or hierarchically independent.

[0011] Preferably, the injection is performed at one or more levels of the visual encoder or the text encoder, and the injection method includes one or more of vector concatenation, additive bias, scaling-translation, attention bias, or cue word embedding.

[0012] Preferably, the frequency domain transformation includes one or more of the following: Fast Fourier Transform (FFT), Discrete Cosine Transform (DCT), Wavelet Transform, Bandpass or Highpass Filtering, and Spatial Domain Highpass Convolution.

[0013] Preferably, the retention ratio α of the high-frequency mask can be adjusted within [0,1] and can adapt to different layers; the injection location is one or more of the key, value or query of the text encoder, feedforward layer input and residual branch.

[0014] Preferably, the contrast and alignment loss includes a cross-entropy main loss and at least one auxiliary regularization, the auxiliary regularization including synonymous uniformity loss and neighborhood diversity loss; The function of the cross-entropy main loss is:

[0015] in, For the visual characteristics of unidirectional vision, For matching text features, For similarity function, For temperature parameters and , For the downstream dataset, I and These are the samples and labels in the downstream dataset, respectively. Let θ be the number of image categories, and θ be the parameters that the model needs to optimize. The function of the synonymous uniformity loss is:

[0016] in, For KL divergence constraints, h The number of candidate synonyms. i For the number of floors, y As a category, For the first i Layers, Categories y The weight vector and , For uniform distribution and ; The function for the neighborhood diversity loss is:

[0017] in, K The total number of encoder layers is represented by the set of synonym embeddings selected for the i-th layer. ; The combined objective of the contrast and alignment losses is:

[0018] in The weights are adjustable or adaptive within the range [0,1].

[0019] This invention also provides a face detection system based on cross-modal cue optimization, employing a face detection method based on cross-modal cue optimization as described above, including: The data preprocessing module is used to perform face detection, alignment, cropping, and normalization. The synonym semantic enhancement module is used to perform synonym retrieval and aggregation, and generate class-specific text prompts; The text-visual cross-modal projection module is used to map class-specific text cues to visual-side injection signals and complete hierarchical injection; The frequency domain high-frequency enhancement module is used to perform frequency domain transformation, masking, and refinement of visual features to obtain high-frequency cues; The visual-text cross-modal projection module is used to map high-frequency cues to text side offsets or additional cues and complete hierarchical injection; The fusion and discrimination module is used to compute joint representations in a shared discrimination space and output true / false categories and confidence levels.

[0020] Compared with the prior art, the present invention has the following beneficial effects: 1. Construct a unified discrimination space: Overcome the deployment complexity and knowledge fragmentation caused by the separation of face anti-spoofing (FAS) and deep fake detection (DFD), and realize integrated detection of multiple types of forgery based on VLM.

[0021] 2. Alleviating category semantic ambiguity: To address the problem that the semantic abstraction of "live / real" and "fake / forged" is difficult to be stably recognized by the VLM model, we propose semantic augmentation of synonyms (SSA) to generate class-specific prompts on the text side, and inject them into the visual branch through the cross-modal projection module Proj_t2v to improve the separability and transferability of category boundaries.

[0022] 3. Enhance the modeling of forged evidence: To address the issues of large differences in the distribution of forged clues across different types and weak details, a frequency domain high-frequency enhancement (FHFA) is introduced to highlight attack-related high-frequency features at the visual end. These features are then projected by the cross-modal projection module Proj_v2t as bias injection branches into the text, enabling collaborative discrimination of multiple types of forgery within a unified discrimination space.

[0023] 4. Improve cross-domain generalization and robustness: By using bidirectional cue transfer and cross-modal projection, the coupling and alignment of visual and text branches are achieved, reducing overfitting to the training domain / device and improving generalization performance for unknown scenarios and novel attacks.

[0024] 5. Reduced implementation costs and easier engineering deployment: Alignment and enhancement can be completed without or with minimal changes to the backbone encoder, reducing training overhead and parameter size, making it easier to deploy as a method and system in practical applications. Attached Figure Description

[0025] Figure 1 This is a flowchart of a face detection method based on cross-modal prompting optimization provided by the present invention. Detailed Implementation

[0026] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The advantages and features of the present invention will become clearer from the following description. It should be noted that the drawings are all in a very simplified form and use non-precise proportions, and are only used to facilitate and clarify the illustration of the embodiments of the present invention.

[0027] like Figure 1 As shown, this embodiment of the invention provides a face detection method based on cross-modal cue tuning, including: By acquiring facial images and performing alignment, cropping, and normalization processing; By performing synonym retrieval and aggregation of true and false categories based on the thesaurus and vocabulary at the encoding layer of one or more Transformer modules in the text branch, class-specific text prompts are generated. The class-specific text prompts are mapped via a text-visual cross-modal projection module to an injection signal aligned with the dimensions and semantics of the corresponding visual coding layer, and then injected at one or more levels of the visual encoder. By performing frequency domain transformation on visual features and applying a high-frequency mask in one or more coding layers of the visual branch, high-frequency features associated with forgery are obtained and refined into high-frequency cues by a lightweight network. The high-frequency cues are mapped to bias or additional cues of text branches via a visual-text cross-modal projection module and injected at one or more levels of the text encoder; By obtaining a joint representation in a shared discriminative space under the bidirectional cueing constraints of the class-specific text cue and the high-frequency cue, and by selectively introducing contrast and alignment losses to stabilize cross-modal alignment; The true / false category and confidence level are output based on the joint representation.

[0028] Preferably, the synonym retrieval and aggregation is performed by selecting the top h most similar synonyms from a fixed vocabulary, pre-trained word vectors, a knowledge base, or candidates generated by a language model and then weighting and aggregating them; where h is an adjustable hyperparameter.

[0029] Preferably, both the text-visual cross-modal projection module and the visual-text cross-modal projection module include one or more of linear mapping, multilayer perceptron (MLP), convolution, attention or cross-attention, and adapter or LoRA, with parameters that can be shared or hierarchically independent.

[0030] Preferably, the injection is performed at one or more levels of the visual encoder or the text encoder, and the injection method includes one or more of vector concatenation, additive bias, scaling-translation, attention bias, or cue word embedding.

[0031] Preferably, the frequency domain transformation includes one or more of the following: Fast Fourier Transform (FFT), Discrete Cosine Transform (DCT), Wavelet Transform, Bandpass or Highpass Filtering, and Spatial Domain Highpass Convolution.

[0032] Preferably, the retention ratio of the high-frequency mask It can be adjusted within [0,1] and can adapt to different layers; the injection position is one or more of the key, value or query of the text encoder, feedforward layer input and residual branch.

[0033] Preferably, the contrast and alignment loss includes a cross-entropy main loss and at least one auxiliary regularization, the auxiliary regularization including synonymous uniformity loss and neighborhood diversity loss; The function of the cross-entropy main loss is:

[0034] Where V represents the homogeneous visual feature. For matching text features, For similarity function, For temperature parameters and , For the downstream dataset, I and These are the samples and labels in the downstream dataset, respectively. For the number of image categories, These are the parameters that the model needs to optimize; The function of the synonymous uniformity loss is:

[0035] in, For KL divergence constraints, h The number of candidate synonyms. i For the number of floors, y As a category, For the first i Layers, Categories y The weight vector and , For uniform distribution and ; The function for the neighborhood diversity loss is:

[0036] in, K The total number of encoder layers is represented by the set of synonym embeddings selected for the i-th layer. ; The combined objective of the contrast and alignment losses is:

[0037] in The weights are adjustable or adaptive within the range [0,1].

[0038] The system of this invention is built on a pre-trained large-scale visual-language model, specifically CLIP (Contrastive Language-Image Pre-training) as an example. CLIP belongs to the visual-language pre-trained model (VLM). Through contrastive learning on large-scale image-text pairs, it aligns the representations of the visual branch and the text branch to a shared embedding space, thereby supporting downstream tasks such as image-text matching and zero-shot classification. The following describes its two branches and alignment method (where the numerical dimensions are for illustrative purposes only and do not limit the invention).

[0039] The construction process of the visual branch includes: 1) Image segmentation and embedding: This involves dividing the input image into segments and embedding them into blocks. The image is divided into fixed-size patches, and the initial image patch embedding is obtained by linear mapping. ,in Where n is the visual embedding dimension (e.g., 768), and n is the number of image patches.

[0040] 2) Category / Starting Label: Learnable class labels can be inserted before the visual input at each layer. The i-th layer visual input is obtained by embedding and splicing it with the previous layer block. .

[0041] 3) Encoding and Reading: The i-th Layer Visual Transformer Module right Encode to obtain the (i+1)th layer visual Transformer module. Input The class label output (or its pooled representation) of the last layer is taken as the visual feature. .

[0042] 4) Projection to shared space: via visual side projection layer Mapping V to a shared graph-text embedding space yields... (like The projection layer can be a linear mapping, a multilayer perceptron, an attention module, or a combination thereof, and can be shared or decoupled from the main body without limitation.

[0043] The process of building text branches includes: 1) Word segmentation and word embedding: The template text (e.g., "a photo of a") is segmented and embedded. <class>(or its Chinese equivalent) is segmented to obtain an initial word embedding matrix. ,in is the total length of word embeddings, and m is the dimension of text embeddings (e.g., 512).

[0044] 2) Hierarchical encoding: The i-th level text Transformer module Output to the previous layer Encode, obtain .

[0045] 3) Text Representation and Projection: The final layer's end-word embedding or full sequence aggregation is taken as the text representation t, which is then projected through the text-side projection layer. Mapped to the shared space, resulting in The template can be a fixed template, a learnable template, or a prompt word embedding; the implementation of the projection layer is also not limited.

[0046] In a preferred embodiment of the present invention, the shared space alignment and training objective further includes: To align visual features in a shared space Text features that match it (Text hints for category y), optimized using a contrastive cross-entropy objective. Given a similarity function. (e.g., normalized dot product / cosine similarity) and temperature parameters For downstream datasets Sample I and its label The cross-entropy loss function can be written as: ,in The number of image categories is represented by "True" and "False". These are the parameters that the model needs to optimize.

[0047] In a preferred embodiment of the invention, CLIP provides a shareable graphic embedding space and pluggable graphic branches: 1) For text branches, we construct class-specific text prompts using the Semantic Synonym Enhancement (SSA) module, and then inject them into the visual branches using the text → visual cross-modal projection module (Proj_t2v). 2) Visual Branch: We extract attack-related high-frequency clues through the Frequency Domain High Frequency Enhancement Module (FHFA) and then inject them into the text branch using the visual-to-text cross-modal projection module (Proj_v2t). 3) This enables coupling alignment and unified discrimination within a shared space, which can be used for joint detection of face physical representation attacks and digital deepfake attacks.

[0048] The dimensions, tags, template formats, similarity, and loss functions mentioned above are all exemplary technical features and can be substituted equivalently as needed without affecting the core ideas and scope of protection of this invention.

[0049] As a further illustration of the embodiments of the present invention, an embodiment of the present invention also constructs a Cross-Modal Prompt Tuning (CMPT) framework to establish bidirectional prompting channels between text and vision (t2v) and vision and text (v2t) in the Visual-Language Model (VLM), realizing hierarchical coupling and alignment of the two branches in a shared discriminant space. Without modifying or only slightly modifying the backbone encoder, the framework uniformly handles physical presentation attacks and digital deepfake attacks by injecting cross-modal prompts into various or several layers.

[0050] The aforementioned CMPT includes at least the following functional modules: 1) Text (word) embedding and text encoder: Generate category semantic representations; support fixed / learnable word embedding templates and prompts.

[0051] 2) Visual image block embedding and visual encoder: Generate visual representations of images / sequences; the backbone is preferably frozen or only a few layers are fine-tuned.

[0052] 3) Synonym Semantic Augmentation (SSA) module: retrieves and aggregates synonyms / near-synonyms from the thesaurus and outputs class-specific text prompts.

[0053] 4) Frequency Domain High-Frequency Enhancement Module (FHFA): Performs frequency domain decomposition, masking and refinement of visual features, and outputs attack-related high-frequency cues.

[0054] 5) Cross-modal projection / mapping modules: The Proj_t2v module maps text-side cues to injected signals that are alignable with the visual branch dimensions and semantics; the Proj_v2t module maps visual-side cues to bias / additional cues that the text branch can accept. Both types of projections can be implemented using linear mapping, multilayer perceptrons, convolution, attention / cross-attention, or combinations thereof, with either parameter sharing or hierarchical independence.

[0055] 6) Cross-modal fusion and discrimination module: Aggregates the multimodal representation after bidirectional hints at a specified layer, and outputs the liveness / spoofing category and confidence level.

[0056] As a further illustration of the embodiments of the present invention, the following hierarchical injection and bidirectional coupling mechanism is also included. To establish cross-modal coupling between each Transformer module in the visual and text branches, CMPT constructs two types of coupling functions at level i: 1) Text-to-visual coupling function. The input is the i-th layer text-side cue (e.g., SSA output), projected by the Proj_t2v module to obtain the injected representation of the i-th layer visual branch from the Transformer module. ; 2) : Visual-to-text coupling function. The input is the visual side cue of the i-th layer (such as the FHFA output), which is projected by the Proj_v2t module to obtain the injected representation of the text branch of the i-th layer from the Transformer module. .

[0057] Injection methods include, but are not limited to, at least one of the following: vector concatenation, additive bias, scale-shift, attention key / value / bias injection, feedforward layer residual injection, and learnable cue word embedding insertion; the injection level is at least one or more layers from shallow / medium / deep. This design ensures that the text branch obtains a visual bias related to the forged evidence, and the visual branch obtains a text bias related to the category semantics, thereby forming a stable discrimination boundary within the shared space.

[0058] This invention also provides a face detection system based on cross-modal cueing optimization, comprising: The data preprocessing module is used to perform face detection, alignment, cropping, and normalization. The Synonym Semantic Enhancement Module (SSA) is used to perform synonym retrieval and aggregation, and generate class-specific text suggestions. The text-visual cross-modal projection module Proj_t2v is used to map class-specific text cues to visual-side injection signals and complete hierarchical injection. The Frequency Domain High Frequency Enhancement Module (FHFA) is used to perform frequency domain transformation, masking, and refinement of visual features to obtain high-frequency cues. The visual-text cross-modal projection module Proj_v2t is used to map high-frequency cues to text side offsets or additional cues and perform hierarchical injection. The fusion and discrimination module is used to compute joint representations in a shared discrimination space and output true / false categories and confidence levels.

[0059] As a further explanation of the embodiments of the present invention, the above-mentioned SSA module is used to address the problem of coarse semantic granularity and unstable boundaries of the "true / false" category label. The SSA module generates class-specific text prompts for each category in the text branch and injects them into the visual branch through text-to-visual cross-modal projection (t2v) to enhance the separability and transferability of visual representation to category semantics.

[0060] The SSA module described above is used to input text branches and a thesaurus / vocabulary; where the text branch is the text embedding at the i-th encoding layer (which can be the terminal word embedding or sequence aggregation representation); the thesaurus / vocabulary (containing word vectors or semantic resources that can generate synonyms); it is used to output class-specific text cues at the i-th layer; and the visual injection signal (as cues / biases) that is aligned with the corresponding visual layer dimension and semantics after being projected by the Proj_t2v module.

[0061] The SSA module mentioned above specifically includes the following processing flow: 1) Category representation extraction: In the text branch... i Layer, obtain target category y Text embedding representations (such as template-terminal word embeddings).

[0062] 2) Query construction: The embedding is lightly transformed (e.g., multilayer perceptron / linear / normalization) to obtain the query vector.

[0063] 3) Synonym retrieval and filtering: Calculate semantic similarity (e.g., cosine / dot product) between query vector and word vector, select the top h most similar synonym candidates, and perform weight normalization (e.g., softmax / temperature scaling).

[0064] 4) Synonym aggregation and hint generation: Weight h synonym embeddings according to weight to obtain class-specific text hints (which can be concatenated or added to the original category embeddings).

[0065] As a further explanation of the embodiments of the present invention, the following key technical features are also described: 1) Sources of synonyms: The thesaurus can come from a fixed vocabulary, pre-trained word vectors, a knowledge base, or be generated by a language model according to category semantics; a hybrid approach of retrieval and generation can also be used.

[0066] 2) Filtering and weighting: h is an adjustable hyperparameter; the similarity measurement and normalization method are not limited.

[0067] 3) Cue Form: The cue can be a vector, matrix, or several learnable word embeddings; it can be concatenated / added with category word embeddings, or used as an attention bias injection.

[0068] 4) t2v projection family: linear mapping, multilayer perceptron, convolution, attention / cross-attention, adapter / LoRA, gating unit or any combination thereof; parameters can be shared or hierarchically independent; dimensionality matching can be accomplished by dimensionality increase / decrease.

[0069] 5) Injection location: At least one of the shallow / medium / deep layers of the visual encoder, or multiple layers combined.

[0070] The SSA module described above can achieve the following technical effects: 1) Alleviating semantic ambiguity: Refining the category expression of "true / false" through synonym aggregation, and stabilizing category boundaries; 2) Enhanced discrimination ability: T2V injection enables visual branches to be guided by the category semantics of synonym aggregation during representation learning, thereby improving the ability to distinguish between true and false forgery attacks. 3) Improve model generalization ability: Hierarchical prompts constrain text and image alignment at different levels, reducing the risk of overfitting to the training domain / device.

[0071] Regarding the equivalent variations and optional implementation schemes of the SSA module: 1) Synonym candidates can be obtained through methods such as nearest neighbor retrieval, template expansion, and generation by large language models; 2) Weighted average can be replaced by gating / attention allocation / sparse selection; 3) Text prompts can also be directly embedded as visual side prompts, or they can be concatenated first and then unified in dimension through a projection layer; 4) If the task contains more than two types of tags, SSA generates hints for each type and injects them in parallel, keeping the logic consistent.

[0072] As a further explanation of the embodiments of the present invention, the SSA module also includes the following formula process for construction: In the i Layers, categories y The last text embedding is The query vector is obtained through a lightweight transformation. ,in It can be a linear / multilayer perceptron / normalized system or a combination thereof, and is not limited to these. Let the thesaurus be... The total vocabulary is e.

[0073] Define similarity function (Such as cosine / dot product, any equivalent metric is acceptable), normalization operator (e.g., softmax / temperature scaling / gated normalization). Select h most similar candidate indices to form a set. Its weight is .

[0074] Generate class-specific text hints (weighted aggregation or linear transformation after concatenation, both are equivalent): , or defined as: ,in This can be a linear / multilayer perceptron / attention mapping. Subsequently, an injection signal aligned with the corresponding visual layer dimension and semantics is obtained through text-to-visual cross-modal projection. .in This is the i-th layer text-to-visual cross-modal projection module, which can be implemented using linear, multilayer perceptron, convolution, attention / cross-attention, Adapter / LoRA, or combinations thereof, without limitation. The overall process can be summarized as follows: Finally, the corresponding visual features are injected into the visual layer to obtain the visual features. ,in and These represent true and false semantic alignment injection signals, respectively.

[0075] As a further explanation of the embodiments of the present invention, the above-mentioned FHFA module is used to address the problem that physical representation attacks and digital deepfakes generally have discriminative clues in high-frequency details. FHFA extracts high-frequency cues related to forgery in the visual branch and maps them to the bias / additional cues of the text branch through visual-to-text cross-modal projection (v2t), thereby guiding the text branch to be more sensitive to forgery traces and promoting the coupling and alignment of image and text semantics in a unified discrimination space.

[0076] The aforementioned FHFA module is used to input the features (image features, denoted as ) of the visual encoder at the i-th level. ), used to output high-frequency cues (visual domain) of the i-th layer, which are cross-modal bias / additional cues that are aligned with the corresponding text layer dimensions and semantics after being projected by the Proj_v2t module.

[0077] The FHFA module mentioned above also includes the following processing flow: 1) Frequency domain transformation: for Perform frequency domain transformation (such as FFT / DCT / wavelet, but not limited to these) to obtain the spectral representation.

[0078] 2) High-frequency mask and retention: Construct a high-frequency mask of the same size as the feature, according to proportional parameters (e.g., ...). Selecting high-frequency regions allows for the suppression of low-frequency components while preserving high-frequency components; the mask can be regularized or learnable.

[0079] 3) Amplitude / phase refinement: The retained high-frequency spectrum is decomposed into amplitude and phase branches, and then refined and denoised using networks such as lightweight convolution / separable convolution / multilayer perceptron / attention.

[0080] 4) Inverse Transformation and Reconstruction: Perform inverse transformation and fusion on the refined spectrum to obtain the enhanced high-frequency features (i.e., high-frequency hints) of the i-th layer.

[0081] The aforementioned FHFA module also includes detailed explanations of the following key technical features: 1) Frequency domain transform family: FFT, DCT, wavelet or equivalent frequency domain / high-pass filtering process; supports spatial domain high-pass convolution approximation as an equivalent implementation.

[0082] 2) Mask / Scale: The mask can be defined regularly or learned; high-frequency retention scale. It can be selected within (0,1) and can adapt to different layers.

[0083] 3) Refined network: Amplitude / phase branches can use convolution, separable convolution, multilayer perceptron, attention or a combination thereof; they can be combined with residual / normalization units.

[0084] 4) Proj_v2t projection family: linear mapping, multilayer perceptron, convolution, attention / cross-attention, Adapter / LoRA, gating unit or any combination thereof; parameters can be shared or hierarchically independent; dimensionality matching can be accomplished by dimensionality aligner.

[0085] 5) Injection location: At least one of the shallow / medium / deep layers of the text encoder, or multiple layers combined.

[0086] 6) Hint format: High-frequency hints can be vectors, matrices, or several word embeddings; injection can be applied to key / value / query / feedforward residuals, etc.

[0087] The aforementioned FHFA module also includes the following technical effects: 1) Amplify forgery clues: Highlight high-frequency differences such as texture, edges, and periodic noise commonly found in face attacks, and reduce background and lighting interference.

[0088] 2) Semantic alignment enhancement: The bias generated by the Proj_v2t module directly affects the learnable word embeddings of the text branches, making them pay more attention to the semantic dimensions related to high-frequency forgery, thus stabilizing the image-text alignment.

[0089] 3) Unified discrimination space: Complementary to the class-specific hints generated by SSA, the two form a robust cross-modal boundary in a unified space, improving cross-domain generalization and robustness.

[0090] The FHFA module mentioned above also includes the following equivalent variations and optional implementation schemes: 1) Frequency domain / high-pass implementation: FFT, DCT, wavelet, bandpass / high-pass filtering or their approximations; 2) Mask and scaling: The mask is regular or learnable; high-frequency retention scaling. Take any value within (0,1) (preferably 0.25), and can be adaptive with the layer or driven by data.

[0091] 3) Amplitude / phase processing: The refined network for amplitude and phase can be convolution, separable convolution, multilayer perceptron, attention, frequency domain convolution or a combination thereof; it can be combined with residual, normalization, and skip connections.

[0092] 4) Reconstruction strategy: Before the inverse transform, amplitude / phase weighted fusion, gated fusion or multi-scale reconstruction can be performed; or only amplitude or only phase branch can be used.

[0093] 5) Cue Form and Injection: Generate vector / matrix / cue word embeddings as text side biases; the injection location can be the key / value / query of the text encoder, feedforward layer input, residual branch or word embedding layer.

[0094] The FHFA module mentioned above also includes the implementation process of the following formula: In the i-th layer, the frequency domain transformation is denoted as and its inverse transformation The high-frequency mask is (Same size as feature):

[0095] Alternatively, a learnable mask / bandpass filter can be used, where U and V are the frequency domain plot dimensions. The mask spectrum is obtained and decomposed into amplitude / phase channels: ,in and For amplitude and phase, and For amplitude / phase refinement networks (convolutional / separable convolutional / multilayer perceptron / attention, etc.), extract high-frequency features. Projected via Proj_v2t As the text-side bias / additional hint injection text branch at level i, the overall process can be described as follows: The word embeddings of the i-th level of the text branch are expressed as follows: The injection method can be splicing, additive bias, scaling-translation, or attention bias, without limitation.

[0096] As a further explanation of the embodiments of the present invention, cross-modal projection and injection (Proj_t2v / Proj_v2t) also includes the following specific descriptions: 1) Definition: Used to map one-sided cue representations to injectable signals that are alignable with the dimensions and semantics of the other-sided branch.

[0097] 2) Implementation families: linear mapping, multilayer perceptron, convolution, attention / cross-attention, adapter / LoRA, gating unit or a combination thereof; parameter sharing or hierarchical independence are both acceptable; dimensionality matching can use dimensionality aligners.

[0098] 3) Injection method: at least one of splicing, additive bias, scaling-translation, attention key / value / bias, feedforward residual injection, and cue word embedding; the injection level is at least one shallow / medium / deep or multiple layers combined.

[0099] 4) Cross-modal projection and injection using Proj_t2v: Class-specific text cues are mapped via the Proj_t2v projection module to cue vectors / matrices that are isomorphic or alignable to the corresponding visual encoding layer. These cue vectors / matrices are then injected at one or more layers corresponding to the visual encoder using vector concatenation, additive bias, scaling-translation, attention bias, or cue word embedding. Furthermore, these steps can be performed at each or several layers to create hierarchical text-visual coupling.

[0100] 5) Cross-modal projection and injection Proj_v2t: High-frequency cues are mapped to bias / additional cues receptible to the text branch via the vision-to-text cross-modal projection module (Proj_v2t), and injected at one or more levels of the text encoder using additive bias, scaling-translation, attention bias, or cue word embedding. The above process can be performed at each or several layers to form a hierarchical vision-to-text coupling.

[0101] As a further explanation of the embodiments of the present invention, the following loss function and training objective are also included: 1) Design Goals: To stably align image and text representations within a unified discrimination space and avoid overly simplistic or redundant synonym selection, this invention introduces two auxiliary regularizations in addition to the main cross-entropy loss: synonym uniformity loss and neighborhood diversity loss. Joint optimization of these three regularizations helps improve the coverage and discriminative power of semantic-like cues and enhances the robustness and generalization of cross-modal alignment.

[0102] 2) Synonym Uniformity Loss: To prevent the SSA module from collapsing due to selecting only a single synonym at each layer, Kullback-Leibler Divergence is used to constrain the distribution of the h closest synonym weights to approximate a uniform distribution. Let the weight vector of the i-th layer and category y be... Evenly distributed Example format: .

[0103] 3) Neighbor Diversity Loss: To avoid selecting highly similar synonyms and causing redundancy, the algorithm guides the selection of diversity among h candidate synonyms. Let the set of selected synonym embeddings in the i-th layer be denoted as . ,but , Where K is the total number of encoder layers. Minimize The penalty for excessively high similarity between synonyms prompts the model to select a more dispersed set of synonyms with complementary information.

[0104] 4) Overall Objectives and Weights; The comprehensive objective is: .

[0105] in The weights (which can take any value in the range [0,1], or be learned adaptively with each layer) are not limited.

[0106] The following is a specific, non-limiting embodiment illustrating how to implement the face detection method and system (CMPT) of the present invention.

[0107] Step 1: Data preparation; 1) Data sources: Collect publicly available or self-collected datasets containing physical representation attacks and digital deepfakes; divide training / validation / test sets according to protocols (e.g., across datasets / across devices / across scenarios).

[0108] 2) Preprocessing: Perform face detection and alignment on the image / video, crop it to a uniform size (e.g., 224×224), and then perform normalization and data augmentation (horizontal flip, color jitter, slight Gaussian noise, etc.).

[0109] 3) Text templates: Prepare Chinese or English templates for each category (such as "This is a real face" "This is a fake face"), and allow learnable cue words to be embedded.

[0110] Step 2: Model building; 1) Backbone: Load a pre-trained visual-language model (VLM), such as "ViT-B / 16 + Text Transformer". It is preferable to freeze the backbone or only fine-tune a few layers at the end.

[0111] 2) SSA (Text → Visual Channel): Construct a thesaurus (which can be generated by word vectors or a language model). At one or more levels of the text branch, retrieve h synonyms / near-synonyms most similar to the most similar words in the thesaurus from the last word embedding of the text feature, and aggregate them with weights to obtain class-specific text cues. This cue is mapped to a cue signal isomorphic to the corresponding visual layer by the Proj_t2v cross-modal projection module, and injected into the image features using methods such as splicing / additive bias / scaling-translation / attention bias / cue word embedding.

[0112] 3) FHFA (Visual-to-Text Channel): At one or more levels in the visual branch, the image features are transformed in the frequency domain (e.g., Fourier transform). A high-frequency mask is applied to highlight the high-frequency components of the image features, and the amplitude / phase is refined using a lightweight network. The refined amplitude / phase is then transformed in the inverse frequency domain to obtain enhanced high-frequency image features. These enhanced high-frequency image features are mapped to the bias / additional cue of the text branch by the Proj_v2t cross-modal projection module and injected into the text features in the corresponding layer of the text encoder.

[0113] 4) Discriminator: In the shared embedding space, calculate the similarity between the visual representation and the category text representation and complete the binary classification; a linear classification head or direct contrastive classification can be used.

[0114] Step 3: Model training; Forward computation: 1) Text branch: Feed the category template (including learnable word embeddings) into the text encoder to obtain text embeddings at each layer; perform SSA at the selected layer and obtain class-specific text hints; 2) Visual branch: The input image / frame sequence is fed into the visual encoder to obtain features of each layer; FHFA is performed on the selected layer to obtain high-frequency cues; 3) Bidirectional injection and alignment: The cues generated by SSA are injected into the visual layer via t2v, and the cues generated by FHFA are injected into the text layer via v2t, to obtain a joint representation that is coupled across modalities; 4) Loss: The main loss is cross-entropy, with optional additions of synonymous uniformity loss and neighborhood diversity loss; the total loss is defined as: .

[0115] Reverse and update: Only update learnable parameters such as SSA, FHFA, t2v / v2t, and cue / projection; the backbone is preferably frozen or partially fine-tuned.

[0116] Step 4: Model Inference; 1) Input a single image or video frame sequence and perform the same forward process as training (excluding gradient and parameter updates). 2) Fusion: For video, scores from multiple frames are averaged over time or determined by majority vote; 3) Output true / false categories and confidence levels.

[0117] It also includes the following key parameters and conditions (example range, not limited); 1) Number of synonyms h: 10 (usually 8-32); 2) High-frequency mask ratio : 0.25 (0-1), adaptable to different layers; 3) Injection level: at least one or more layers from shallow / medium / deep; 4) T2V / V2T implementation family: linear, multilayer perceptron, convolution, attention / cross-attention, Adapter / LoRA, gating, etc. or combinations thereof; 5) Loss weights: and : 0.01 (0-1), grid or adaptive search; 6) Optimization and training: SGD optimizer; learning rate 0.02; batch size 1; training for 100 epochs (with cosine annealing / linear warm-up).

[0118] Through the above specific implementation methods, those skilled in the art can fully understand and implement the present invention, thereby achieving the goal of improving the generalization ability of cross-domain face anti-counterfeiting systems.

[0119] In summary, the beneficial effects of the present invention are as follows: 1. Unified task and unified discrimination space, simplifying deployment. Through CMPT's bidirectional cue transfer (text-to-visual SSA and visual-to-text FHFA), physical attacks and digital deep fakes are modeled uniformly in the shared space of the pre-trained VLM. Both types of attacks can be detected in the same model, reducing the maintenance cost and deployment complexity of multiple models.

[0120] 2. The model alleviates semantic ambiguity and strengthens its ability to distinguish between genuine and fake evidence. SSA expands the abstract meanings of "live / true, fake / false" into a task-related set of synonyms in the text branch and injects it into the visual branch, alleviating "semantic ambiguity." FHFA extracts high-frequency amplitude / phase cues in the visual branch and feeds them back to the text branch, providing decision biases aligned with forgery traces. These two-way cue-closed-loop constraints significantly improve cross-modal alignment stability.

[0121] 3. Cross-domain robustness and leading unified detection performance: On the JFSFDB benchmark, CMPT achieves an average EER of 17.29%, superior to CLIP's 21.86%, and surpasses the second-best method (with a 1.69% reduction in EER) under the unified detection protocol setting. On UniAttackData, UAD-CMPT achieves the best ACER with an average of 2.71%; among which, protocols P1.1 and P2 show ACER reductions of 4.34% and 4.18% respectively compared to the second-best method. In FAS domain generalization (M / C / I / O cross-domain), CMPT achieves an average HTER of 2.21%, superior to all compared methods, and all protocols achieve optimal HTER, demonstrating stronger cross-domain robustness.

[0122] 4. The components are complementary and their contributions are clear, and the results can be ablation-verified. Introducing either SSA or FHFA brings significant benefits, and when the two are combined, the ACER is reduced by 2.23%. In terms of regularization, after adding synonym uniformity constraints and neighborhood diversity constraints on the basis of cross-entropy, the ACER is further reduced by 1.33%, indicating that "uniform distribution + neighborhood diversity" can stabilize synonym selection and improve discrimination coverage.

[0123] 5. Low overhead and easy to deploy, with a freeze-through backbone. During training, only SSA, FHFA, and cross-modal projection layers are learned. The visual / text encoder can be kept frozen or fine-tuned, significantly reducing parameter and computing power requirements, making it easy to deploy on edge devices or in resource-constrained scenarios.

[0124] 6. Interpretable, extensible, and easy to tune and transfer parameters. The synonym set of SSA and the frequency domain cues of FHFA can both be visualized and manually reviewed; hyperparameters (such as h, ...) are also readily available. , and It can be migrated with minor adjustments to different protocols, and has good engineering controllability and scalability.

[0125] The above description is merely a description of preferred embodiments of the present invention and is not intended to limit the scope of the present invention in any way. Any changes or modifications made by those skilled in the art based on the above disclosure shall fall within the protection scope of the claims.< / class>

Claims

1. A face detection method based on cross-modal cue tuning, characterized in that, include: By acquiring facial images and performing alignment, cropping, and normalization processing; By performing synonym retrieval and aggregation of true and false categories based on the thesaurus and vocabulary at the encoding layer of one or more Transformer modules in the text branch, class-specific text prompts are generated. The class-specific text prompts are mapped via a text-visual cross-modal projection module to an injection signal aligned with the dimensions and semantics of the corresponding visual coding layer, and then injected at one or more levels of the visual encoder. By performing frequency domain transformation on visual features and applying a high-frequency mask in one or more coding layers of the visual branch, high-frequency features associated with forgery are obtained and refined into high-frequency cues by a lightweight network. The high-frequency cues are mapped to bias or additional cues of text branches via a visual-text cross-modal projection module and injected at one or more levels of the text encoder; By obtaining a joint representation in a shared discriminative space under the bidirectional cueing constraints of the class-specific text cue and the high-frequency cue, and by selectively introducing contrast and alignment losses to stabilize cross-modal alignment; The true / false category and confidence level are output based on the joint representation.

2. The face detection method based on cross-modal cueing optimization as described in claim 1, characterized in that, The process of forming the text branch includes: Word segmentation and word embedding: The template text is segmented into words to obtain an initial word embedding matrix. ,in For the total length of word embedding, For text embedding dimensions; Hierarchical encoding, i-th layer text Transformer module Output to the previous layer Encode, obtain ; Text representation and projection: The final layer's terminal word embedding or full sequence aggregation is taken as the text representation t, which is then projected through the text-side projection layer. Mapped to the shared space, resulting in .

3. The face detection method based on cross-modal cueing optimization as described in claim 1, characterized in that, The formation process of the visual branch includes: Image segmentation and embedding, taking the input image The image is divided into fixed-size blocks, and the initial image block embedding is obtained by linear mapping. ,in Where n is the visual embedding dimension, and n is the number of image patches; Category and starting label: Learnable class labels are inserted before the visual input at each layer. The i-th layer visual input is obtained by embedding and splicing it with the previous layer block. ; Encoding and Reading, i-th Layer Visual Transformer Module right Encode to obtain the (i+1)th layer visual Transformer module Input The class label output of the last layer is taken as the visual feature. ; Projected into the shared space, through the visual side projection layer Mapping V to a shared graph-text embedding space yields... .

4. The face detection method based on cross-modal cueing optimization as described in claim 1, characterized in that, The synonym retrieval and aggregation are performed by selecting the top h most similar synonyms from a fixed vocabulary, pre-trained word vectors, a knowledge base, or candidates generated by a language model and then weighting and aggregating them; where h is an adjustable hyperparameter.

5. The face detection method based on cross-modal cueing optimization as described in claim 1, characterized in that, Both the text-visual cross-modal projection module and the visual-text cross-modal projection module include one or more of the following: linear mapping, multilayer perceptron (MLP), convolution, attention or cross-attention, and adapter or LoRA. The parameters can be shared or hierarchically independent.

6. The face detection method based on cross-modal cueing optimization as described in claim 1, characterized in that, Injection is performed at one or more levels of the visual encoder or the text encoder, and the injection methods include vector concatenation, additive bias, scaling-translation, and attention bias or cue word embedding insertion, or one or more of these.

7. The face detection method based on cross-modal cueing optimization as described in claim 1, characterized in that, The frequency domain transformation includes one or more of the following: Fast Fourier Transform (FFT), Discrete Cosine Transform (DCT), Wavelet Transform, Bandpass or Highpass Filtering, and Spatial Domain Highpass Convolution.

8. The face detection method based on cross-modal cueing optimization as described in claim 1, characterized in that, The retention ratio α of the high-frequency mask can be adjusted within [0,1] and can adapt to different layers; the injection position is one or more of the key, value or query of the text encoder, feedforward layer input and residual branch.

9. The face detection method based on cross-modal cueing optimization as described in claim 1, characterized in that, The contrast and alignment loss includes a cross-entropy main loss and at least one auxiliary regularization, which includes synonymous uniformity loss and neighborhood diversity loss. The function of the cross-entropy main loss is: in, For the visual characteristics of unidirectional vision, For matching text features, For similarity function, It is a temperature parameter and it is a temperature parameter and it is a temperature parameter and , For the downstream dataset, I and These are the samples and labels in the downstream dataset, respectively. Let θ be the number of image categories, and θ be the parameters that the model needs to optimize. The function of the synonymous uniformity loss is: in, For KL divergence constraints, h The number of candidate synonyms. i For the number of floors, y As a category, For the first i Layers, Categories y The weight vector and , For uniform distribution and ; The function for the neighborhood diversity loss is: in, K The total number of encoder layers is represented by the set of synonym embeddings selected for the i-th layer. ; The combined objective of the contrast and alignment losses is: in The weights are adjustable or adaptive within the range [0,1].

10. A face detection system based on cross-modal cue optimization, employing a face detection method based on cross-modal cue optimization as described in any one of claims 1 to 9, characterized in that, include: The data preprocessing module is used to perform face detection, alignment, cropping, and normalization. The synonym semantic enhancement module is used to perform synonym retrieval and aggregation, and generate class-specific text prompts; The text-visual cross-modal projection module is used to map class-specific text cues to visual-side injection signals and complete hierarchical injection; The frequency domain high-frequency enhancement module is used to perform frequency domain transformation, masking, and refinement of visual features to obtain high-frequency cues; The visual-text cross-modal projection module is used to map high-frequency cues to text side offsets or additional cues and complete hierarchical injection; The fusion and discrimination module is used to compute joint representations in a shared discrimination space and output true / false categories and confidence levels.

Images