Cross-domain face anti-spoofing detection method and device based on multi-modal text enhancement

Abstract

The application discloses a cross-domain face anti-counterfeiting detection method and device based on multi-modal text enhancement, and relates to the technical field of network information security.The method comprises the following steps: inputting two types of description texts into a pre-trained text encoder to extract text category features representing authenticity / fraud, and inputting an image into a pre-trained visual encoder to extract visual features; adding trainable text prompts to each layer of the text encoder, and adding trainable visual prompts to each layer of the visual encoder, wherein the visual prompt of each layer of the visual encoder is obtained by converting the text prompt of the current layer through a full connection layer; embedding a PFT module and a TIM module into the middle layer of each layer of the text encoder and the visual encoder to realize feature interaction and fusion, obtain the cosine similarity and the mask between the text category features and the visual features, and perform face true / false classification.The application simultaneously completes the modal feature interaction in the feature extraction process based on the PFT module and the TIM module, and improves the cross-domain detection performance.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of network information security, and particularly relates to a cross-domain face anti-forgery detection method and device based on multi-modal text enhancement. BACKGROUND

[0002] As an important part of security detection, face detection (FR) has been widely used in various security identification and electronic payment fields, and has become an important technology for protecting the privacy field involving personal property and information security. Among them, face anti-forgery (FAS) has become an important part of face detection. The task of face anti-forgery is to identify whether the detected face is a real face or a fake physical presentation attack (paper printing, electronic screen playback, 3D mask), and after filtering out the fake physical presentation attack face, the real face is sent to the subsequent face feature extraction system to realize the security protection in the face recognition process.

[0003] For the earlier FAS task, researchers use interactive face anti-forgery [1] , which uses the motion information of the face to make a judgment. The idea of this method is that the face can perform actions such as blinking, opening the mouth, shaking the head, or other actions according to the thinking, but the fake face cannot perform these random simple actions. The system prompts the user to perform a random specific action sequence, and only after successfully completing the action sequence can the user pass the security check and proceed with the subsequent operation. However, this method has been proven to be ineffective for 3D mask fake faces.

[0004] Early methods [2-4] Researchers observe the differences between real and fake faces and manually design features based on these differences to distinguish real and fake faces. The texture of the paper material used for printing and the screen texture in the video are different from the skin texture of the real face. Therefore, both two-dimensional and three-dimensional counterfeit faces exhibit unique texture differences compared to real faces. One study [4] uses local binary pattern (LBP) to evaluate the correlation between adjacent pixels in real and counterfeit data. Another study [5] detects the impact of different color spaces and finds that features extracted from the YCbCr color space are more abundant than data in the gray space.

[0005] The early face anti-forgery algorithms described above rely on manually designed features, which are then used to train classification models using support vector machines (SVM) [6] and other classifiers. However, these manually designed features have relatively shallow discrimination information capabilities and are highly dependent on expert-provided professional discrimination knowledge, and are not very effective in capturing subtle detail differences between live faces and fake faces. In 2018, Liu [7]A depth framework designed for end-to-end predicted depth information map is introduced for the first time, with threshold decision based on feature similarity distance. These methods using deep learning [7-9] and methods using single modality visual features [10-14] Significant results have been achieved in the intra-domain test, but when applied to cross-domain test, the performance will be greatly affected, leading to the failure of real-world application.

[0006] Since there is no solution to face anti-spoofing combined with multi-modal text and trainable prompt information guidance in China at present, in order to solve the above problems.

[0007] Reference:

[0008] [1] Yang Jianwei. A face anti-spoofing method based on local motion pattern: CN105447432A[P / OL]. 2016-03-30 [2024-08-29]. https: / / patents.google.com / patent / CN105447432A / zh.

[0009] [2] Komulainen J, Hadid A, M. Context based face anti-spoofing[C / OL] / / 2013IEEE sixth international conference on biometrics: theory, applications and systems (BTAS). IEEE, 2013: 1-8 [2024-08-29].

[0010] [3] Boulkenafet Z, Komulainen J, Hadid A. Face anti-spoofing based on color texture analysis[C / OL] / / 2015IEEE international conference on image processing (ICIP). IEEE, 2015: 2636-2640 [2024-08-29].

[0011] [4] DE FREITAS PEREIRA T, ANJOS A, DE MARTINO J M, et al. LBP-TOP Based Countermeasure against Face Spoofing Attacks [M / OL] / / PARK J I, KIM J. Computer Vision-ACCV 2012 Workshops: volume 7728. Berlin, Heidelberg: Springer Berlin Heidelberg, 2013: 121-132 [2024-08-29].

[0012] [5] LI J, WANG Y, TAN T, et al. Live face detection based on the analysis of fourier spectra [C / OL] / / Biometric technology for human identification: volume 5404. SPIE, 2004: 296-303 [2024-08-29].

[0013] [6] BURGES C J C. A tutorial on support vector machines for pattern recognition [J / OL]. Data Mining and Knowledge Discovery, 1998, 2(2): 121-167. DOI: 10.1023 / A:1009715923555.

[0014] [7] LIU Y, JOURABLOO A, LIU X. Learning deep models for face anti-spoofing: Binary or auxiliary supervision [C / OL] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 389-398 [2024-08-29].

[0015] [8] MU D, LI T. Face anti-spoofing with multi-color double-stream CNN [C / OL] / / Proceedings of the 13th International Conference on Distributed Smart Cameras. Trento Italy: ACM, 2019: 1-4 [2024-08-29].

[0016] [9] WANG Z, ZHAO C, QIN Y, et al. Exploiting temporal and depth information for multi-frame face anti-spoofing [M / OL]. arXiv, 2019 [2024-08-29].

[0017]

[10] JIA Y, ZHANG J, SHAN S, et al. Single-side domain generalization for face anti-spoofing [C / OL] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2020: 8484-8493 [2024-05-31].

[0018]

[11] HU C, ZHANG K Y, YAO T, et al. Rethinking Generalizable Face Anti-spoofing via Hierarchical Prototype-guided Distribution Refinement in Hyperbolic Space [C / OL] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2024: 1032-1041 [2024-06-20].

[0019]

[12] SUN Y, LIU Y, LIU X, et al. Rethinking domain generalization for face anti-spoofing: Separability and alignment[C / OL] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2023:24563-24574[2024-06-01].

[0020]

[13] LIN X, WANG S, CAI R, et al. Suppress and Rebalance: Towards Generalized Multi-Modal Face Anti-Spoofing[M / OL]. arXiv, 2024[2024-06-01]. http: / / arxiv.org / abs / 2402.19298.

[0021]

[14] ZHOU Q, ZHANG K Y, YAO T, et al. Generative Domain Adaptation for Face Anti-Spoofing[M / OL] / / AVIDAN S, BROSTOW G, M, et al. Computer Vision–ECCV 2022: Volume 13665. Cham: Springer Nature Switzerland, 2022:335-356[2024-06-01]. Summary of the Invention

[0022] To solve the problems of the prior art, the present application aims to provide a cross-domain face anti-counterfeiting detection method and device based on multi-modal text enhancement, which introduces trainable prompts on the basis of the multi-modal method CLIP, uses the prompts to collect the correlation between different face features and corresponding text features, and introduces two modal fusion modules, namely, a prompt fusion transfer module (PFT) and a text-guided image masking module (TIM), in the middle layers of the encoders of the two modalities to solve the problem of insufficient modal feature interaction caused by post-alignment in previous multi-modal correlation methods, so that the modal feature interaction is completed at the same time in the feature extraction process, and the training efficiency and detection performance of the entire model are improved.

[0023] The present application adopts the following technical solutions:

[0024] In one aspect, a cross-domain face anti-counterfeiting detection method based on multi-modal text enhancement includes:

[0025] Two types of description texts including real face description texts and fake face description texts are input into a pre-trained text encoder, and an image is input into a pre-trained visual encoder, and at each layer of the text encoder and the visual encoder, text category features f i T and visual features f i I are extracted, respectively; the visual features include visual category features f i C and image block features; 0≤i≤N-1, and N represents the number of layers of the text encoder and the visual encoder;

[0026] A trainable text prompt is added at each layer of the text encoder A trainable visual prompt is added at each layer of the visual encoder The visual prompt of each layer of the visual encoder is obtained by converting the text prompt of the current layer through a full connection layer;

[0027] The text prompt is spliced with the text category features, the visual prompt is spliced with the visual features, and forward propagation is performed in the corresponding encoder layer; in the forward propagation process, a prompt fusion transfer module and a text-guided image masking module are embedded into the middle layer of each layer of the text encoder and the visual encoder to realize feature interaction fusion, obtain the cosine similarity between the text category features and the visual features, and obtain a mask based on the cosine similarity and the mask, and perform face true / false category discrimination based on the cosine similarity and the mask;

[0028] In the prompt fusion transfer module of the i-th layer, the text prompt converts the text prompt feature into a visual prompt feature converts the text prompt feature into a visual prompt feature, and the text prompt feature of the current layer is fused with the visual prompt feature of the current layer through a multi-head cross-attention mechanism to obtain a multimodal-guided text prompt feature, and the multimodal-guided text prompt feature is combined with the text category feature of the corresponding layer to serve as an input of a next layer of the text encoder;

[0029] In the text-guided image mask module of the i th layer, the current similar text category label is calculated using the visual category feature, and the image block feature output by the current visual encoder layer is subjected to similarity masking using the current similar text label to obtain a text-guided mask; an image block random mask is introduced, and the intersection of the text-guided mask and the image block random mask is obtained to obtain a final mask.

[0030] Preferably, a trainable text prompt is obtained by random initialization, representing text information aggregation from the i th layer.

[0031] Preferably, the visual prompt vector of the first layer is obtained by performing nonlinear transformation on the text prompt of the first layer, and is represented as follows:

[0032]

[0033] wherein MLP(.) represents a multi-layer perceptron.

[0034] Preferably, the text prompt and the text category feature are spliced, the visual prompt and the visual feature are spliced, and forward propagation is performed in the corresponding encoder layer, and is represented as follows:

[0035]

[0036] wherein represents the visual feature of the i+1 th layer; the text category feature of the i+1 th layer; Encoder i v (.) represents the i th layer of the visual encoder, (.) represents the i th layer of the text encoder, represents the visual prompt feature refined by the i th layer of the visual encoder, represents the text prompt feature refined by the i th layer of the text encoder.

[0037] Preferably, the text prompt feature of the current layer is fused with the visual prompt feature of the current layer through a multi-head cross-attention mechanism to obtain a multimodal-guided text prompt feature, and is represented as follows:

[0038]

[0039] where Mean(·) denotes the mean of the batch visual cues; MLP(.) denotes a multi-layer perceptron; denotes the current averaged visual cue feature; denotes the i+1th layer of the textual cue feature; Softmax(·) denotes the exponential normalization of the feature to form a probability distribution; D denotes the dimension number of the feature.

[0040] Preferably, in the text-guided image mask module, the current similar text class label is calculated using the visual class feature, which is represented as follows:

[0041]

[0042] where θ denotes the label determined based on the cosine similarity, θ ∈ {spoof, real}, spoof denotes the real label, and real denotes the fake label; f i θ ∈ {f i real ,f i spoof} denotes an element in the set of all real and fake discriminant feature vectors.

[0043] Preferably, the similar text label performs similarity masking on the image block feature output by the current visual encoder layer to obtain a text-guided mask; an image block random mask is introduced, and the intersection of the text-guided mask and the image block random mask is obtained to obtain a final mask, including:

[0044] The selected text feature f i θ is calculated by matrix multiplication. The attention map between all visual markers

[0045]

[0046] where, denotes the relative score; denotes the length of the image marker block divided by the image; softmax denotes the exponential normalization of the text feature guided image block score to form a probability distribution; f i θ denotes an element in the set of all real and fake discriminant feature vectors.

[0047] The text-guided mask is represented as follows:

[0048]

[0049] where sij denotes the jth fraction in s i τ is the threshold fraction of the fixed proportion obtained after the fractions are sorted in descending order;

[0050] generate a random mask of the same proportion as the text-guided mask the intersection of the text-guided mask and the random mask as the final mask, denoted as follows:

[0051]

[0052] In another aspect, a cross-domain face anti-counterfeiting detection device based on multi-modal text enhancement includes:

[0053] An original feature acquisition module is configured to input two types of description texts including real face description texts and counterfeit face description texts into a pre-trained text encoder, input an image into a pre-trained visual encoder, and extract text category features f i T and visual features f i O representing real / fraudulent at each layer of the text encoder and the visual encoder, respectively; the visual features include visual category features f i C and image block features; 0≤i≤N-1, and N represents the number of layers of the text encoder and the visual encoder.

[0054] A prompt module is configured to add trainable text prompts at each layer of the text encoder add trainable visual prompts at each layer of the visual encoder Each layer of the visual encoder is converted from the text prompts of the current layer through a fully connected layer.

[0055] A fusion and detection module is configured to splice the text prompts with the text category features, splice the visual prompts with the visual features, and perform forward propagation in the corresponding encoder layers; during the forward propagation, a prompt fusion transmission module and a text-guided image mask module are embedded into the middle layer of each layer of the text encoder and the visual encoder to realize feature interaction fusion, obtain the cosine similarity between the text category features and the visual features, and obtain a mask, and based on the cosine similarity and the mask, perform face true / false category discrimination.

[0056] In the prompt fusion transmission module of the i-th layer, the text prompts f are converted into text prompt features, and the visual prompts f ​The text prompt feature of the current layer is fused with the visual prompt feature of the current layer through a multi-head cross attention mechanism to obtain a multi-modal guided text prompt feature, and the multi-modal guided text prompt feature is combined with the text category feature of the corresponding layer to serve as an input of a next layer of the text encoder;

[0057] In the text-guided image mask module of the i-th layer, the current similar text category label is calculated using the visual category feature, and the image block feature output by the current visual encoder layer is subjected to similarity masking using the current similar text label to obtain a text-guided mask; an image block random mask is introduced, and the text-guided mask is taken as an intersection to obtain a final mask.

[0058] Compared with the prior art, the present application has the following beneficial effects:

[0059] (1) The cross-domain face anti-counterfeiting detection method and device based on multi-modal text enhancement of the present application, based on the multi-modal text enhancement cross-domain face anti-counterfeiting model (ME-FAS), increases the trainable text prompt and visual prompt on the basis of the text encoder and the visual encoder, and increases two intermediate layer modules of the encoder to perform feature fusion and interaction process in the feature extraction process, so as to realize the performance improvement of cross-domain face detection.

[0060] (2) The two intermediate layer modules of the encoder of the present application are respectively a prompt fusion transmission module (PFT) and a text-guided image mask module (TIM), which are respectively responsible for the mutual feature interaction transmission between the two modalities; the PFT uses the unique visual feature to guide the learning of the text feature, so that the text is adapted to the input face image, while the TIM uses the text feature mask image block to make the model focus on the features most relevant to the task discrimination and ignore the domain-related style information, so that the model focuses on the task-related area and enhances the robustness. BRIEF DESCRIPTION OF DRAWINGS

[0061] Figure 1 The flowchart of the cross-domain face anti-counterfeiting detection method based on multi-modal text enhancement of the embodiment of the present application;

[0062] Figure 2 The framework flowchart of the cross-domain face anti-counterfeiting model (ME-FAS) of the multi-modal text enhancement of the embodiment of the present application;

[0063] Figure 3For various visualization diagrams of model effect performance; wherein, (a) shows that the initial distribution clustering is not clear due to the lack of pre-trained facial features in the face field; (b) shows the distribution of the trained feature, and the distribution of the real and fake face is obviously close to the text description of it; (c) shows the visualization of the attention area on the image by Grad-CAM; (d) shows the visual representation of the attention of the text features in different levels in the TIM module;

[0064] Figure 4 The structural block diagram of the cross-domain face anti-counterfeiting detection device based on multi-modal text enhancement of the embodiment of the application. DETAILED DESCRIPTION

[0065] The application will be further described below in connection with specific embodiments. It should be understood that these embodiments are only used to illustrate the application and not to limit the scope of the application. Furthermore, it should be understood that those skilled in the art can make various modifications or changes to the application after reading the content of the application, and these equivalent forms also fall within the scope of the appended claims.

[0066] In order to more clearly understand the above-mentioned purposes, features and advantages of the application, the application will be described in detail below in connection with the drawings and specific embodiments. It should be noted that the embodiments of the application and the features in the embodiments can be combined with each other without conflict.

[0067] In the following description, a large number of specific details are set forth in order to facilitate a thorough understanding of the application. The described embodiments are merely some of the embodiments of the application, rather than all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the application.

[0068] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the application belongs. The terms used in the specification of the application are only for the purpose of describing the specific embodiments and are not intended to limit the application.

[0069] The term "comprising" and any variation thereof in the specification and claims of the application and the above-mentioned drawings is intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units not listed, or optionally also includes other steps or units inherent to the process, method, product or device.

[0070] In order to solve the generalization cross-domain problem in the face anti-counterfeiting task, the application discloses a cross-domain face anti-counterfeiting detection method and device based on multi-modal text enhancement, a cross-domain face anti-counterfeiting model (ME-FAS) based on multi-modal text enhancement, and the cross-domain face anti-counterfeiting model (ME-FAS) based on multi-modal text enhancement is increased on the basis of a text encoder and a visual encoder, and a trainable text prompt and a visual prompt are added, and two modal interaction modules are proposed to process information fusion between the modes, so that the performance of cross-domain face anti-counterfeiting detection is improved, and the two modal interaction modules are a prompt fusion transmission module (PFT) and a text-guided image mask module (TIM), respectively. The information of the text and the image mode is used to guide the feature learning of the other mode, so that the cross-domain face anti-counterfeiting model (ME-FAS) based on multi-modal text enhancement can easily complete the cross-domain detection task.

[0071] Referring to Figure 1 and Figure 2 , the application discloses a cross-domain face anti-counterfeiting detection method based on multi-modal text enhancement, which comprises the following steps:

[0072] S101, two types of description texts including real face description texts and fake face description texts are input into a pre-trained text encoder, and images are input into a pre-trained visual encoder, and text category features and visual features representing real / fraud are extracted at each layer of the text encoder and the visual encoder, respectively; the visual features comprise visual category features and image block features;

[0073] S102, a trainable text prompt is added at each layer of the text encoder, and a trainable visual prompt is added at each layer of the visual encoder; the visual prompt of each layer of the visual encoder is obtained by converting the text prompt of the current layer through a full connection layer;

[0074] S103, the text prompt is spliced with the text category feature, the visual prompt is spliced with the visual feature, and forward propagation is performed in the corresponding encoder layer; in the forward propagation process, a prompt fusion transmission module and a text-guided image mask module are embedded into the middle layer of each layer of the text encoder and the visual encoder to realize feature interaction fusion, and the cosine similarity between the text category feature and the visual feature and the mask are obtained, and based on the cosine similarity and the mask, face true / false category discrimination is performed;

[0075] In the prompt fusion transmission module of the i-th layer, the text prompt is converted into a text prompt feature, and the visual prompt The text prompt feature of the current layer is fused with the visual prompt feature of the current layer through a multi-head cross-attention mechanism to obtain a multimodal-guided text prompt feature, and the multimodal-guided text prompt feature is combined with the text category feature of the corresponding layer to serve as an input of a next layer of the text encoder; 0≤i≤N-1, and N represents the number of layers of the text encoder and the visual encoder;

[0076] In the i-th layer text-guided image mask module, the current similar text category feature is calculated by using the visual category feature, and the similar text feature is used to perform similarity masking on the image block feature output by the current visual encoder layer to obtain a text-guided mask; an image block random mask is introduced, and an intersection is taken with the text-guided mask to obtain a final mask.

[0077] Overall, given an image two descriptive texts T r and T s (real face description text and fake face description text) are used for judgment, and a “real / fake face image photo” is judged. The image and the two descriptive texts are sent into a pre-aligned multimodal backbone network to extract features, and the cosine similarity between the features is used as a judgment standard.

[0078] Specifically, the prompt fusion transmission module (PFT) and the text-guided image mask module (TIM) will be described in detail below in combination with trainable text prompts and visual prompts.

[0079] (1) Prompt fusion transmission module (PFT)

[0080] The prompt fusion transmission module (PFT) aims to quickly and regularly adjust the knowledge of a pre-trained visual-linguistic model to adapt to a face anti-fake task. The prompt fusion transmission module (PFT) is inserted into each layer of the encoder to customize the classification text of various visual features.

[0081] Specifically, given an image is sent into a pre-trained visual encoder, and two descriptive texts T r and T s are sent into a pre-trained text encoder to extract features. For the i-th encoder layer, the visual features and the text category features representing real / fraud representing real / fraud are extracted, respectively. The visual features include visual category features f i C and image block features; 0≤i≤N-1, and N represents the number of layers of the text encoder and the visual encoder. I and l TC and D denote the number of channels for visual feature and text category feature respectively. Then, the text cue vector of the i-th layer i l (.) is obtained by a nonlinear transformation of initialized randomly, representing the aggregation of text information from the i-th layer. To align the visual cues with the text cues, the visual cue vector of the first layer (l P is the length of the cue, representing the aggregation of text information from the i-th layer) is obtained by a nonlinear transformation of

[0082]

[0083] where MLP(.) denotes a multi-layer perceptron.

[0084] Then, these cues are concatenated with the original features and undergo the normal forward propagation process in the i-th encoder layer as follows:

[0085]

[0086] where denotes the visual feature of the i+1-th layer; denotes the text category feature of the i+1-th layer; Encoder i v (.) denotes the i-th layer of the visual encoder, denotes the i-th layer of the text encoder, denotes the visual cue feature refined by the i-th layer of the visual encoder, denotes the text cue feature refined by the i-th layer of the text encoder.

[0087] It can be seen that the initial text cue of the current layer is concatenated with the original features, and the next layer of the initial text cue is obtained after the encoder layer.

[0088] Subsequently, and represent the set of multi-modal information from the i-th layer. The PFT model utilizes various visual cue features of different samples to customize the classification text features for each instance by the process of unidirectional injection of visual modalities into language features.

[0089] At this time, there is a problem, and are not consistent in dimension, so the two dynamic features are aggregated in some way first, and in this case, the aggregation of ρ along the feature dimension is selected. Specifically, the model projects to ​fusion computation of the two prompts is performed:

[0090]

[0091] where Mean(·) denotes the mean of the batch visual prompts; MLP(.) denotes a multi-layer perceptron; denotes the current averaged visual prompt feature; denotes the text prompt feature of the i+1-th layer; Softmax(·) denotes the exponential normalization of the feature to form a probability distribution; D denotes the dimensionality of the feature.

[0092] The main purpose of PFT is to assist classification with visual information, providing customized classification text features for each different image sample to improve the generalization ability. Therefore, the output of formula (3) is set as the text prompt of the next layer to maintain distinguishability, while the visual prompt of the next layer is randomly initialized again.

[0093] PFT realizes the pre-fusion process in the pre-trained model (including the visual encoder and the text encoder) through prompt embedding interaction. The dual-modal prompts summarize information from each layer and help each other through prompt interaction. With the help of independent visual prompt information of each layer, the text prompt learns step by step through layer-by-layer interaction, adapting different classification text features for each different image information, thereby improving the generalization ability of the model.

[0094] (2) Text-guided image mask module (TIM)

[0095] PFT provides instance-aware level generalization for the classification text embedding of sample-related visual information, but the visual information lacks sufficient guidance. For the original pre-trained model, the focus is on the global information of the image, without clear distinction of details. However, for the face anti-spoofing task, irrelevant factors such as background, lighting, and environment will cause great interference. In order to refocus the visual encoder of the visual-language model on the face anti-spoofing task, a text-guided image mask (TIM) is proposed to inject task-related information into the visual interaction through text guidance, making the visual encoder more focused on the content related to face anti-spoofing. Specifically, given the i-th text encoder f i s and f i r The classification label feature represents the spoof text and the real text, respectively. First, the cosine similarity between the text classification label and the visual classification label f i c is calculated to determine the most likely label θ∈{spoof,real}, as follows:

[0096]

[0097] Where spoof represents a real tag, and real represents a fake tag; f i θ ∈{f i real ,f i spoof} represents the set f of all current real and fake feature vectors. i T The elements in.

[0098] Specifically, the cosine similarity calculated above can be used to obtain the current similar text category features. In fact, according to equation (4), the most suitable label for each trial feature layer is different, which brings greater flexibility. After obtaining the best label, this is a good source of additional information that can help locate visual information regions related to face tasks. Specifically, the selected text features f are calculated by matrix multiplication. i θ With all visual features f i I The attention map between the two is used, and a fixed percentage, such as 50%, is used to mask low-scoring tags. This process is described as follows:

[0099] s i =softmax(f i θ ×f i I (5)

[0100] in, Indicates a relative fraction; l I The length of the patch tokens representing the image segments is indicated; softmax represents the exponential normalization of the image patch scores guided by text features to form a probability distribution.

[0101] Text guide mask It is expressed as follows:

[0102]

[0103] Among them, s ij s i The j-th score in τ is a fixed percentage of the threshold score obtained after the scores are sorted in descending order.

[0104] The above formula utilizes textual information for learning assistance, but the selection of categories and the method of generating masks are directly coupled with the training process, which can lead to overfitting of the image masking process to textual features. Therefore, this invention also considers introducing randomness to help stabilize the masks during training.

[0105] More specifically, the mask is generated randomly with the same proportion as the language mask M i The intersection of the text-guided mask and the random mask is taken as the final mask M i As shown in the following formula:

[0106]

[0107] After that, the visual features will be masked by the text-guided M i , which will better guide the visual encoder layers to pay more attention to the areas that should be focused on.

[0108] Further, the training loss of the multi-modal text enhanced cross-domain face anti-spoofing model (ME-FAS) will be described as follows.

[0109] The method uses multiple loss functions to optimize the multi-modal text enhanced cross-domain face anti-spoofing model (ME-FAS). Initially, the method calculates the similarity p using the final output features of the image-text pair. Given that the text represents real and fake samples, the method implements a binary cross-entropy loss to supervise the final score:

[0110]

[0111] where, and y j represent the labels (real or fake) corresponding to the training data source and the training sample, respectively.

[0112] The model's ability to distinguish image features is enhanced by using the SimCLR loss in self-supervised learning. During training, each image undergoes two different image enhancement processes, labeled σ 1 and σ 2 , which can respectively design changes in image brightness, contrast, and size. These methods produce two different enhanced attempts of the image, from which features of these enhanced views are extracted. The alignment of the features extracted from the two enhanced views is achieved by using the SimCLR loss, which is described as follows:

[0113]

[0114] In addition, the method aims to align the enhanced image features with the corresponding text features to ensure the best classification accuracy. The alignment loss calculated is shown as follows:

[0115]

[0116] where, f t ​​denotes the I i corresponding category knowledge text features; and denotes the features corresponding to the two images enhanced by the original face image during the training process via the image encoder to enhance the training process of the model.

[0117] In summary, the total loss is:

[0118]

[0119] The performance comparison experiment will be performed as follows.

[0120] The present application is trained and tested on MSU-MFSD (M), CASIA-MFSD (C), Idiap Replay-Attack (I) and OULU-NPU (O), and tested using the Leave-One-Out (LOO) method, with three datasets used for training each time and the remaining dataset used as the test set for cross-domain testing.

[0121] The present application uses the half total error rate (HTER) and the area under the receiver operating characteristic curve (AUC) as the model performance measurement standard and compares it with other existing methods, and the comparison results are shown in Table 1 below.

[0122] Table 1. Comparison of ME-FAS method and other methods

[0123]

[0124] By comparing the main HTER indicator, the method proposed in this paper achieves the best performance in half of the settings. Specifically, in terms of average HTER, the proposed method improves 1.46% compared to the baseline model FLIP, and improves 1.43% compared to the current best model CFPL-FAS. Observations also show that the test results of FLIP under different settings fluctuate significantly, while the method proposed in this paper maintains a consistent and stable level. This stability shows that the method in this paper aligns the text and images from different fields early, provides better adaptation and more robust text features, and thus continuously improves the performance in different settings through early visual-linguistic model alignment.

[0125] At the same time, in order to further compare the cross-domain performance under limited data, only M and I are used as the source domain for training while the remaining two datasets are used as the test domain, and the comparison results are shown in Table 2 below.

[0126] Table 2. Performance comparison of methods under limited conditions

[0127]

[0128] The proposed method is further validated under extremely limited source domain testing conditions as shown in Table 2. The proposed method is also significantly superior to the current state-of-the-art method on limited source data. This shows that the proposed method is still effective for unseen target domains due to the introduction of generalized text and enhanced inter-modal interaction, even under strict conditions.

[0129] In order to clarify the performance of the model and the basis for judgment, the features generated by the model are t-SNE visualized, Grad-CAM maps are drawn, and the focal regions of the model in the TIM module are shown. As shown in Figure 3 (a) and 3(b), due to the lack of pre-trained facial features in the face domain, the clustering of the initial distribution is not clear. Finally, the distributions of real and fake faces are obviously close to their corresponding texts. In Figure 3 (c) Grad-CAM analysis, the gradient of the real face (green) is uniformly concentrated throughout the face, while the gradient of the paper mask (left red) and the printed attack (right red) obviously highlights the general attack area around the eyes and mouth. In addition, the visual representation of the text feature attention of the TIM module at different levels is additionally supplemented, as shown in Figure 3 (d). The figure illustrates that at lower levels, the text emphasizes global detail features, while at higher levels, it focuses on task-related features such as eyes and nose, which are beneficial for generalization judgment.

[0130] In summary, the proposed ME-FAS method has good results in performance and explainability, and achieves the best detection performance in the current FAS task in terms of the main evaluation index HTER.

[0131] Referring to Figure 4 The application further discloses a multi-modal text enhancement-based cross-domain face anti-forgery detection device, which comprises:

[0132] The original feature acquisition module 401 is used for inputting two types of description texts including real face description texts and fake face description texts into a pre-trained text encoder, inputting an image into a pre-trained visual encoder, and extracting text class features f i T and visual features f i I representing real / fraud in each layer of the text encoder and the visual encoder; the visual features include visual class features f i C and image block features; 0≤i≤N-1, and N represents the number of layers of the text encoder and the visual encoder;

[0133] The prompt module 402 is used for adding trainable text prompts in each layer of the text encoder Adding trainable visual cues at each layer of the visual encoder Each layer of visual cues of the visual encoder is converted from the textual cues of the current layer by a fully connected layer;

[0134] The fusion and detection module 403 is configured to splice the textual cues and the textual category features, splice the visual cues and the visual features, and perform forward propagation in the corresponding encoder layer; in the process of forward propagation, the prompt fusion transmission module and the text-guided image mask module are embedded into the intermediate layer of each layer of the text encoder and the visual encoder to realize feature interaction fusion, obtain the cosine similarity between the textual category features and the visual features, and obtain the mask, and based on the cosine similarity and the mask, the true or false category of the face is determined;

[0135] In the prompt fusion transmission module of the i-th layer, the textual cues are converted into textual cue features, the visual cues are converted into visual cue features, the textual cue features of the current layer are fused with the visual cue features of the current layer by a multi-head cross-attention mechanism to obtain multi-modal guided textual cue features, and the multi-modal guided textual cue features and the textual category features of the corresponding layer are combined to serve as the input of the next layer of the text encoder. In the prompt fusion transmission module of the i-th layer, the textual cues are converted into textual cue features, the visual cues are converted into visual cue features, the textual cue features of the current layer are fused with the visual cue features of the current layer by a multi-head cross-attention mechanism to obtain multi-modal guided textual cue features, and the multi-modal guided textual cue features and the textual category features of the corresponding layer are combined to serve as the input of the next layer of the text encoder.

[0136] In the text-guided image mask module of the i-th layer, the visual category features are used to calculate a similar textual category label, the similar textual label is used to perform similarity masking on the image block features output by the current visual encoder layer to obtain a text-guided mask, and a random image block mask is introduced to take the intersection with the text-guided mask to obtain a final mask.

[0137] A specific implementation of a cross-domain face anti-counterfeiting detection system based on multi-modal text enhancement is the same as a multi-modal text enhancement-based cross-domain face anti-counterfeiting detection method, and the embodiment will not be repeated.

[0138] It is apparent for those skilled in the art that the present application is not limited to the details of the above exemplary embodiments, and can be implemented in other specific forms without departing from the spirit or essential characteristics of the present application. Therefore, the embodiments should be regarded as exemplary and non-limiting, and the scope of the present application is defined by the appended claims rather than the above description, and all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be included in the present application. Any reference signs in the claims should not be regarded as limiting the claims involved. In addition, it is clear that the word "comprising" does not exclude other units or, singular does not exclude plural. The plurality of units or systems stated in the system claims can also be implemented by one unit or system through software or hardware.

[0139] ​Finally, it should be noted that the above examples are merely intended to illustrate the technical solutions of the present application and not to limit the present application. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or equivalently replaced without departing from the spirit of the technical solutions of the present application.

Claims

1. A cross-domain face anti-spoofing detection method based on multi-modal text enhancement, characterized in that, Comprising: inputting two types of description texts including real face description texts and fake face description texts into a pre-trained text encoder, inputting an image into a pre-trained visual encoder, at each layer of the text encoder and the visual encoder, respectively extracting text category features f i T representing real / fraudulent and visual features f i I ; the visual features include visual category features f i C and image block features; 0≤i≤N-1, N represents the number of layers of the text encoder and the visual encoder; Adding trainable textual cues at each layer of the textual encoder Adding trainable visual cues at each layer of the visual encoder Each visual cue of the visual encoder is converted from the textual cue of the current layer by a fully connected layer The text prompt is spliced with the text category feature, the visual prompt is spliced with the visual feature, and forward propagation is performed in the corresponding encoder layer; during the forward propagation process, the prompt fusion transmission module and the text-guided image mask module are embedded into the middle layer of each layer of the text encoder and the visual encoder to realize feature interaction and fusion, obtain the cosine similarity between the text category feature and the visual feature, and the mask, and based on the cosine similarity and the mask, the face true or false category is discriminated; In the prompt fusion transmission module of the i-th layer, the text prompt is converted into a text prompt feature, and the visual prompt is converted into a visual prompt feature. The text prompt feature of the current layer and the visual prompt feature of the current layer are fused through a multi-head cross-attention mechanism to obtain a multimodal guided text prompt feature, and the multimodal guided text prompt feature is combined with the text category feature of the corresponding layer to serve as an input of a next layer of the text encoder. In the i-th layer text-guided image mask module, the current similar text category label is calculated using the visual category feature, and the image block feature output by the current visual encoder layer is masked using the current similar text label to obtain a text-guided mask; a random image block mask is introduced, and the intersection of the text-guided mask is taken to obtain a final mask. In the text-guided image mask module, the current similar text category label is calculated using the visual category feature, and is represented as follows: wherein θ represents a label determined based on the cosine similarity, θ ∈ {spoof, real}, spoof represents a real label, and real represents a fake label; f i θ ∈ {f i real ,f i spoof} represents an element in a set of all real and fake discrimination feature vectors at present. The image block feature output by the current visual encoder layer is masked using the similar text label to obtain a text-guided mask; a random image block mask is introduced, and the intersection of the text-guided mask is taken to obtain a final mask, comprising: The selected text features f are computed by matrix multiplication i θ The attention map between all visual markers f i I The attention map between all visual markers f is represented as follows: s i = softmax(f i θ x f i I ) wherein, denotes a relative score; denotes the length of the image label block split out by the image; softmax denotes an exponential normalization of the image block scores guided by the text features to form a probability distribution; f i θ denotes an element in the set of all real and fake discrimination feature vectors at present; Text-guided mask is represented as follows: wherein s ij denotes the jth fraction in s i is a fixed proportion of the threshold fraction obtained after the fractions are sorted in descending order;​ generate a random mask of the same proportions as the text-guided mask the intersection of the text-guided mask and the random mask as the final mask, represented as follows:

2. The cross-domain face anti-spoofing detection method based on multi-modal text enhancement according to claim 1, characterized in that, Each layer of the text encoder adds a trainable text hint with initial values obtained by random initialization, representing the aggregation of text information from the i-th layer.

3. The cross-domain face anti-spoofing detection method based on multi-modal text enhancement according to claim 1, characterized in that, The visual cue vector of the first layer is obtained by performing a nonlinear transformation on the textual cue of the first layer and is represented as follows: Wherein, MLP(.) represents a multi-layer perceptron.

4. The cross-domain face anti-spoofing detection method based on multi-modal text enhancement according to claim 1, characterized in that, The text prompt is spliced with the text category feature, the visual prompt is spliced with the visual feature, and forward propagation is performed in the corresponding encoder layer, and is represented as follows: wherein, represents the visual feature of the i+1th layer; represents the text class feature of the i+1th layer; represents the i-th layer of the visual encoder, represents the i-th layer of the text encoder, represents the visual cue feature refined by the i-th layer of the visual encoder, represents the text cue feature refined by the i-th layer of the text encoder.

5. The cross-domain face anti-spoofing detection method based on multi-modal text enhancement according to claim 4, characterized in that, The text prompt feature of the current layer is fused with the visual prompt feature of the current layer through a multi-head cross-attention mechanism to obtain a multi-modal guided text prompt feature, and is represented as follows: where Mean(·) denotes the mean of the batch visual cues; MLP(.) denotes a multi-layer perceptron; denotes the current averaged visual cue feature; denotes the i+1th layer of the textual cue feature; Softmax(·) denotes the exponential normalization of the feature to form a probability distribution; D denotes the dimensionality of the feature.

6. A cross-domain face anti-spoofing detection device based on multi-modal text enhancement, characterized in that, Based on the method of any one of claims 1-5, comprising: The original feature acquisition module is configured to input two types of description texts including real face description texts and fake face description texts into a pre-trained text encoder, input an image into a pre-trained visual encoder, and extract text category features f representing real / fraud at each layer of the text encoder and the visual encoder i T and visual features f i I ; the visual features include visual category features f i C and image block features; 0≤i≤N-1, and N represents the number of layers of the text encoder and the visual encoder. a prompt module for adding trainable textual prompts at each layer of the textual encoder adding trainable visual prompts at each layer of the visual encoder the visual prompts for each layer of the visual encoder are converted from the textual prompts of the current layer by a fully connected layer The fusion and detection module is used for splicing the text prompt with the text category feature, splicing the visual prompt with the visual feature, and performing forward propagation in the corresponding encoder layer; during the forward propagation process, the prompt fusion transmission module and the text-guided image mask module are embedded into the middle layer of each layer of the text encoder and the visual encoder to realize feature interaction and fusion, obtain the cosine similarity between the text category feature and the visual feature, and the mask, and based on the cosine similarity and the mask, the face true or false category is discriminated; In the prompt fusion transmission module of the i-th layer, the text prompt is converted into a text prompt feature, and the visual prompt is converted into a visual prompt feature. The text prompt feature of the current layer is fused with the visual prompt feature of the current layer through a multi-head cross-attention mechanism to obtain a multimodal guided text prompt feature, and the multimodal guided text prompt feature is combined with the text category feature of the corresponding layer to serve as an input of a next layer of the text encoder. In the i-th layer text-guided image mask module, the current similar text category label is calculated using the visual category feature, and the image block feature output by the current visual encoder layer is masked using the current similar text label to obtain a text-guided mask; a random image block mask is introduced, and the intersection of the text-guided mask is taken to obtain a final mask.

Images