Cross-domain face anti-counterfeiting detection method and device based on multi-modal text enhancement

Abstract

The invention discloses a cross-domain face anti-counterfeiting detection method and device based on multi-modal text enhancement, and relates to the technical field of network information security, and the method comprises the steps: inputting two types of description texts into a pre-trained text encoder, and extracting text type features representing reality / deception, inputting the image into a pre-trained visual encoder to extract visual features; a trainable text prompt is added to each layer of the text encoder, a trainable visual prompt is added to each layer of the visual encoder, and the visual prompt of each layer of the visual encoder is obtained by converting the text prompt of the current layer through a full connection layer; and embedding the PFT module and the TIM module into a middle layer of each layer of a text encoder and a visual encoder to realize feature interaction fusion, obtaining cosine similarity and masks between text category features and visual features, and performing face true and false classification. Based on the PFT module and the TIM module, modal feature interaction is completed in the feature extraction process at the same time, and the cross-domain detection performance is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of network information security, and in particular to a cross-domain face anti-counterfeiting detection method and device based on multimodal text enhancement. Background Art

[0002] As an important part of security detection, face detection (FR) has been widely used in various security identification and electronic payment fields, and has become an important technology to protect the privacy of personal property and information security. Among them, face anti-counterfeiting (FAS) has become an important part of face detection. The task of face anti-counterfeiting is to identify whether the face currently detected is a real face or a forged physical presentation attack (paper printing, electronic screen playback, 3D mask). After filtering out the forged physical presentation attack face, the real face is sent to the subsequent face feature extraction system to achieve security protection in the face recognition process.

[0003] For the earlier FAS task, researchers used an interactive facial anti-counterfeiting approach [1] , using the motion information of the face for judgment. The idea of ​​this method is that the face can perform actions such as blinking, opening the mouth, shaking the head or other actions according to the mind, but the fake face cannot perform these random simple actions. The system prompts the user to perform a random specific action sequence. Only after successfully completing the action sequence can the security check be passed and subsequent operations can be performed. However, this method has been proven to be ineffective for fake faces with 3D masks.

[0004] Early methods [2-4] The researchers observed the differences between real and fake faces and manually designed features based on these differences to distinguish between real and fake faces. The paper material used for printing and the screen texture in the video have different texture variations from the skin texture of the real face. Therefore, both the two-dimensional and three-dimensional fake faces show unique texture differences compared to the real face. One of the studies [4] Local Binary Patterns (LBP) are used to evaluate the correlation between adjacent pixels in real and counterfeit data. [5] By examining the impact of different color spaces, it is found that the features extracted from the YCbCr color space are richer than the data in the grayscale space.

[0005] The early facial anti-counterfeiting algorithms mentioned above relied on manually designed features, which were then used to detect the presence of a fake face using a support vector machine (SVM). [6] However, these manually designed features are relatively shallow in their ability to discern information and are highly dependent on the professional knowledge provided by experts, so they are not very effective in capturing the subtle differences between live faces and fake faces. [7]For the first time, a deep framework designed for end-to-end prediction of depth information graphs is introduced, using threshold decisions based on feature similarity distance. These deep learning methods [7-9] and methods using unimodal visual features [10-14] Significant results have been achieved in intra-domain tests, but when applied to cross-domain tests, the performance is greatly affected and the application fails in the real world.

[0006] Since there is currently no solution in China that combines multimodal text and trainable prompt information to guide face anti-counterfeiting, in order to solve the above-mentioned problems.

[0007] References:

[0008] [1] Yang Jianwei. A face anti-counterfeiting method based on local motion pattern: CN105447432A[P / OL]. 2016-03-30[2024-08-29]. https: / / patents.google.com / patent / CN105447432A / zh.

[0009] [2] KOMULAINEN J, HADID A, M.Context based face anti-spoofing[C / OL] / / 2013IEEE sixth international conference on biometrics:theory, applications and systems(BTAS).IEEE,2013:1-8[2024-08-29].

[0010] [3]BOULKENAFET Z,KOMULAINEN J,HADID A.Face anti-spoofing based on color texture analysis[C / OL] / / 2015IEEE international conference on imageprocessing(ICIP).IEEE,2015:2636-2640[2024-08-29].

[0011] [4]DE FREITAS PEREIRA T,ANJOS A,DE MARTINO J M,et al..LBP-TOP BasedCountermeasure against Face Spoofing Attacks[M / OL] / / PARK J I,KIM J.ComputerVision-ACCV 2012Workshops:volume 7728.Berlin,Heidelberg:Springer BerlinHeidelberg,2013:121-132[2024-08-29].

[0012] [5]LI J,WANG Y,TAN T,et al.Live face detection based on the analysisof fourier spectra[C / OL] / / Biometric technology for human identification:volume 5404.SPIE,2004:296-303[2024-08-29].

[0013] [6]BURGES C J C.A tutorial on support vector machines for patternrecognition[J / OL].Data Mining and Knowledge Discovery,1998,2(2):121-167.DOI:10.1023 / A:1009715923555.

[0014] [7]LIU Y,JOURABLOO A,LIU X.Learning deep models for face anti-spoofing:Binary or auxiliary supervision[C / OL] / / Proceedings of the IEEEconference on computer vision and pattern recognition.2018:389-398[2024-08-29].

[0015] [8]MU D,LI T.Face anti-spoofing with multi-color double-stream CNN[C / OL] / / Proceedings of the 13th International Conference on Distributed SmartCameras.Trento Italy:ACM,2019:1-4[2024-08-29].

[0016] [9]WANG Z,ZHAO C,QIN Y,et al.Exploiting temporal and depthinformation for multi-frame face anti-spoofing[M / OL].arXiv,2019[2024-08-29].

[0017]

[10] JIA Y,ZHANG J,SHAN S,et al.Single-side domain generalization forface anti-spoofing[C / OL] / / Proceedings of the IEEE / CVF Conference on ComputerVision and Pattern Recognition.2020:8484-8493[2024-05-31].

[0018]

[11] HU C,ZHANG K Y,YAO T,et al.Rethinking Generalizable Face Anti-spoofing via Hierarchical Prototype-guided Distribution Refinement inHyperbolic Space[C / OL] / / Proceedings of the IEEE / CVF Conference on ComputerVision and Pattern Recognition.2024:1032-1041[2024-06-20].

[0019]

[12] SUN Y, LIU Y, LIU X, et al. Rethinking domain generalization for face anti-spoofing: Separability and alignment[C / OL] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2023:24563-24574[2024-06-01].

[0020]

[13] LIN X, WANG S, CAI R, et al. Suppress and Rebalance: Towards Generalized Multi-Modal Face Anti-Spoofing[M / OL]. arXiv, 2024[2024-06-01]. http: / / arxiv.org / abs / 2402.19298.

[0021]

[14] ZHOU Q, ZHANG K Y, YAO T, et al. Generative Domain Adaptation for Face Anti-Spoofing[M / OL] / / AVIDAN S, BROSTOW G, M, et al. Computer Vision-ECCV 2022: Volume 13665. Cham: Springer Nature Switzerland, 2022:335-356[2024-06-01]. Summary of the Invention

[0022] In view of the problems of the prior art, the purpose of the present invention is to provide a cross-domain face anti-counterfeiting detection method and device based on multimodal text enhancement. On the basis of the multimodal method CLIP, trainable prompts are introduced, and the prompts are used to collect the correlation between different facial features and corresponding text features; at the same time, in order to solve the problem of insufficient modal feature interaction caused by the late alignment adopted in previous multimodal correlation methods, two modal fusion modules are introduced in the middle layer of the encoders of the two modalities, namely, Prompt Fusion Transfer (PFT) and Text-guided Image Masking (TIM). Through the mutual modal guidance of the two modules, the modal feature interaction is completed simultaneously in the process of feature extraction, thereby improving the training efficiency and detection performance of the entire model.

[0023] The present invention adopts the following technical solution:

[0024] On the one hand, a cross-domain face anti-forgery detection method based on multimodal text enhancement includes:

[0025] Two types of description texts, including real face description texts and fake face description texts, are input into the pre-trained text encoder, and the image is input into the pre-trained visual encoder. In each layer of the text encoder and the visual encoder, the text category features representing the real / deceptive are extracted respectively. i T and visual features i I ; The visual features include visual category features f i C and image block features; 0≤i≤N-1, N represents the number of layers of text encoder and visual encoder;

[0026] Add trainable text hints to each layer of the text encoder Adding trainable visual cues to each layer of the visual encoder Each layer of visual cues in the visual encoder is obtained by converting the text cues in the current layer through a fully connected layer;

[0027] The text prompts are concatenated with the text category features, and the visual prompts are concatenated with the visual features, and forward propagated in the corresponding encoder layer; during the forward propagation process, the prompt fusion transfer module and the text-guided image mask module are embedded in the middle layer of each layer of the text encoder and the visual encoder to realize feature interactive fusion, obtain the cosine similarity and mask between the text category features and the visual features, and perform true and false face category discrimination based on the cosine similarity and mask;

[0028] Among them, in the prompt fusion transfer module of the i-th layer, the text prompt Convert into text prompt features and convert visual prompts Converted into visual cue features, the text cue features of the current layer are fused with the visual cue features of the current layer through a multi-head cross-attention mechanism to obtain multi-modal guided text cue features, and the multi-modal guided text cue features are combined with the text category features of the corresponding layer as the input of the next layer of the text encoder;

[0029] In the i-th layer text-guided image mask module, the visual category features are used to calculate the current similar text category label, and the current similar text label is used to perform similarity masking on the image block features output by the current visual encoder layer to obtain the text-guided mask; a random mask of the image block is introduced and intersected with the text-guided mask to obtain the final mask.

[0030] Preferably, each layer of the text encoder adds trainable text cues The initial value of is obtained by random initialization and represents the aggregation of text information from the i-th layer.

[0031] Preferably, the first layer of visual cue vector Through the text prompt of the first layer The nonlinear transformation is obtained as follows:

[0032]

[0033] Here, MLP(.) represents a multi-layer perceptron.

[0034] Preferably, the textual hint is concatenated with the textual category feature, the visual hint is concatenated with the visual feature, and forward propagated in the corresponding encoder layer, which is expressed as follows:

[0035]

[0036]

[0037] in, Represents the visual features of the i+1th layer; Text category features at the i+1th layer; represents the i-th layer of the visual encoder, represents the i-th layer of the text encoder, represents the visual cue feature after refinement by the i-th layer of the visual encoder, Represents the text prompt features after refinement by the i-th layer of the text encoder.

[0038] Preferably, the text prompt features of the current layer are fused with the visual prompt features of the current layer through a multi-head cross attention mechanism to obtain the multi-modal guided text prompt features, which are expressed as follows:

[0039]

[0040]

[0041] Where Mean(·) represents the average value of the batch visual cues; MLP(.) represents multi-layer perceptron; Represents the current averaged visual cue feature; represents the text prompt feature of the i+1th layer; Softmax(·) represents exponential normalization of the feature to form a probability distribution; D represents the number of dimensions of the feature.

[0042] Preferably, in the text-guided image mask module, the visual category features are used to calculate the current similar text category label, which is expressed as follows:

[0043]

[0044] Among them, θ represents the label determined based on cosine similarity, θ∈{spoof,real}, spoof represents the real label, and real represents the forged label; f i θ ∈{f i real , f i spoof} represents an element in the set of all current real and fake discriminant feature vectors.

[0045] Preferably, the similar text features perform similarity masking on the image block features output by the current visual encoder layer to obtain a text-guided mask; introduce an image block random mask and intersect it with the text-guided mask to obtain a final mask, including:

[0046] Calculate the selected text features f by matrix multiplication i θ With all visual markers f i I The attention map between is represented as follows:

[0047] s i =softmax(f i θ ×f i I )

[0048] in, Indicates relative score; l I represents the length of the image marker block from which the image is segmented; softmax represents the exponential normalization of the image block scores guided by text features to form a probability distribution; f i θRepresents the elements in the set of all current real and fake discriminant feature vectors;

[0049] Text guide mask It is expressed as follows:

[0050]

[0051] Among them, s ij Indicates i The jth score in I ]; τ is the fixed ratio threshold score obtained after the scores are sorted in descending order;

[0052] Generate a random mask of the same proportions as the text-guided mask Direct the text to the mask and random mask The intersection of is used as the final mask, which is expressed as follows:

[0053]

[0054] On the other hand, a cross-domain face anti-forgery detection device based on multimodal text enhancement includes:

[0055] The original feature acquisition module is used to input two types of description texts, including real face description texts and fake face description texts, into the pre-trained text encoder, and input the image into the pre-trained visual encoder. In each layer of the text encoder and the visual encoder, the text category features representing the real / deceptive are extracted respectively. i T and visual features i I ; The visual features include visual category features f i C and image block features; 0≤i≤N-1, N represents the number of layers of text encoder and visual encoder;

[0056] Hint module, which adds trainable text hints to each layer of the text encoder Adding trainable visual cues to each layer of the visual encoder Each layer of visual cues in the visual encoder is obtained by converting the text cues in the current layer through a fully connected layer;

[0057] The fusion and detection module is used to splice the text prompts with the text category features, the visual prompts with the visual features, and perform forward propagation in the corresponding encoder layer; during the forward propagation process, the prompt fusion transfer module and the text-guided image mask module are embedded in the middle layer of each layer of the text encoder and the visual encoder to realize feature interactive fusion, obtain the cosine similarity and mask between the text category features and the visual features, and perform true and false face category discrimination based on the cosine similarity and mask;

[0058] Among them, in the prompt fusion transfer module of the i-th layer, the text prompt Convert into text prompt features and convert visual prompts Converted into visual cue features, the text cue features of the current layer are fused with the visual cue features of the current layer through a multi-head cross-attention mechanism to obtain multi-modal guided text cue features, and the multi-modal guided text cue features are combined with the text category features of the corresponding layer as the input of the next layer of the text encoder;

[0059] In the i-th layer text-guided image mask module, the visual category features are used to calculate the current similar text category label, and the current similar text label is used to perform similarity masking on the image block features output by the current visual encoder layer to obtain the text-guided mask; a random mask of the image block is introduced and intersected with the text-guided mask to obtain the final mask.

[0060] Compared with the prior art, the present invention has the following beneficial effects:

[0061] (1) The present invention provides a cross-domain face anti-counterfeiting detection method and device based on multimodal text enhancement, a cross-domain face anti-counterfeiting model (ME-FAS) based on multimodal text enhancement, which adds trainable text prompts and visual prompts on the basis of text encoders and visual encoders, and adds two encoder intermediate layer modules to perform feature fusion interaction in the feature extraction process, so as to achieve performance improvement of cross-domain face detection;

[0062] (2) The two encoder intermediate layer modules of the present invention are the Prompt Fusion Transfer Module (PFT) and the Text-guided Image Mask Module (TIM), which are responsible for the mutual feature interaction and transfer between the two modalities respectively; PFT uses unique visual features to guide the learning of text features so that the text adapts to the input face image, while TIM uses text feature mask image blocks to make the model focus on the features most relevant to task discrimination and ignore domain-related style information, so that the model focuses on task-related areas and enhances robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 It is a flow chart of a cross-domain face anti-counterfeiting detection method based on multimodal text enhancement according to an embodiment of the present invention;

[0064] Figure 2 A framework flow chart of the multimodal text-enhanced cross-domain face anti-forgery model (ME-FAS) implemented by the present invention;

[0065] Figure 3 Various visualizations showing the performance of the model; (a) shows the unclear initial distribution clustering due to the lack of pre-trained facial features in the face field; (b) shows the distribution of features after training, and the distribution of real and fake faces is obviously close to their corresponding text descriptions; (c) shows the visualization of the focus area on the image by Grad-CAM; (d) shows the visual representation of the attention of text features at different levels in the TIM module;

[0066] Figure 4 This is a structural block diagram of a cross-domain face anti-counterfeiting detection device based on multimodal text enhancement according to an embodiment of the present invention. DETAILED DESCRIPTION

[0067] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.

[0068] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.

[0069] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. The embodiments described are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0070] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0071] The term "comprising" and any variations thereof in the specification and claims of the present invention and the above drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products or devices.

[0072] In order to solve the generalization cross-domain problem in face anti-counterfeiting tasks, the present invention provides a cross-domain face anti-counterfeiting detection method and device based on multimodal text enhancement, a cross-domain face anti-counterfeiting model (ME-FAS) based on multimodal text enhancement, adds trainable text prompts and visual prompts on the basis of a text encoder and a visual encoder, and proposes two modal interaction modules to process information fusion between modalities to achieve performance improvement of cross-domain face anti-counterfeiting detection. The two modal interaction modules are a prompt fusion transfer module (PFT) and a text-guided image mask module (TIM), which use the information of text and image modalities to guide the feature learning of the other modality, so that the constructed multimodal text-enhanced cross-domain face anti-counterfeiting model (ME-FAS) can easily complete the cross-domain detection task.

[0073] See also Figure 1 and Figure 2 As shown, the present invention provides a cross-domain face anti-counterfeiting detection method based on multimodal text enhancement, comprising:

[0074] S101, inputting two types of description texts including real face description texts and fake face description texts into a pre-trained text encoder, inputting an image into a pre-trained visual encoder, and extracting text category features and visual features representing authenticity / deception at each layer of the text encoder and the visual encoder, respectively; the visual features include visual category features and image block features;

[0075] S102, adding trainable text prompts to each layer of the text encoder, and adding trainable visual prompts to each layer of the visual encoder; each layer of the visual prompt of the visual encoder is obtained by converting the text prompt of the current layer through the fully connected layer;

[0076] S103, concatenating the text prompt with the text category feature, concatenating the visual prompt with the visual feature, and performing forward propagation in the corresponding encoder layer; during the forward propagation process, embedding the prompt fusion transfer module and the text-guided image mask module into the middle layer of each layer of the text encoder and the visual encoder to realize feature interactive fusion, obtaining the cosine similarity and mask between the text category feature and the visual feature, and performing true and false face category discrimination based on the cosine similarity and the mask;

[0077] Among them, in the prompt fusion transfer module of the i-th layer, the text prompt Convert into text prompt features and convert visual prompts Converted into visual cue features, the text cue features of the current layer are fused with the visual cue features of the current layer through a multi-head cross-attention mechanism to obtain multimodal guided text cue features, and the multimodal guided text cue features are combined with the text category features of the corresponding layer as the input of the next layer of the text encoder; 0≤i≤N-1, N represents the number of layers of the text encoder and the visual encoder;

[0078] In the i-th layer text-guided image mask module, the current similar text category feature is calculated using the visual category feature, and the similar text feature performs similarity masking on the image block feature output by the current visual encoder layer to obtain a text-guided mask; a random mask of the image block is introduced and intersected with the text-guided mask to obtain the final mask.

[0079] Overall, given an image Through two descriptive texts T r and T s The image and the two descriptive texts will be fed into a pre-aligned multimodal backbone network to extract features respectively, and the cosine similarity between the features will be used as the judgment criterion.

[0080] Specifically, the prompt fusion transfer module (PFT) and the text-guided image mask module (TIM) are described in detail below in combination with trainable text prompts and visual prompts.

[0081] (1) Prompt Fusion Transfer Module (PFT)

[0082] The Prompt Fusion Transfer Module (PFT) is designed to quickly and routinely adapt the knowledge of pre-trained visual-language models to face anti-forgery tasks. The Prompt Fusion Transfer Module (PFT) is inserted into the encoder layers to classify text tailored to various visual features.

[0083] Specifically, given an image is fed into the pre-trained visual encoder, two descriptive texts T r and T s is fed into a pre-trained text encoder to extract features. For the i-th encoder layer, visual features are extracted respectively and text category features representing truth / deception The visual features include visual category features f i C and image block features; 0≤i≤N-1, N represents the number of layers of text encoder and visual encoder.I and l T is the corresponding feature length, C and D represent the number of channels of visual features and text category features respectively. i l (.) Text hint vector is randomly initialized to represent the aggregation of text information from the i-th layer. In order to align the visual and textual cues, the visual cue vector of the first layer (l P is the prompt length, indicating the aggregation of text information from the i-th layer) is obtained by The nonlinear transformation is as follows:

[0084]

[0085] Here, MLP(.) represents multi-layer perceptron.

[0086] These hints are then concatenated with the original features and undergo a normal forward propagation process in the i-th encoder layer as follows:

[0087]

[0088] in, Represents the visual features of the i+1th layer; Text category features at the i+1th layer; Encoder i v (.) represents the i-th layer of the visual encoder, Encoder i l (.) represents the i-th layer of the text encoder, represents the visual cue features after refinement by the visual encoder layer, Represents the text hint features after being refined by the text encoder layer.

[0089] It can be seen that the initial text prompt of the current layer is concatenated with the original feature, and after passing through the encoder layer together, the initial text prompt of the next layer is obtained.

[0090] Then, and Represents the collection of multimodal information from the i-th layer. The PFT model uses various visual cue features of different samples to customize the classification text features for each instance by unidirectionally injecting the visual modality into the language features.

[0091] There is a problem at this time. and The dimensions of are inconsistent, so the two dynamic features are aggregated in some way. Here, we choose to aggregate along the feature dimension ρ. Specifically, the model uses an MLP layer to aggregate Project to Dimensions, then perform the fusion calculation of the two prompts:

[0092]

[0093] Where Mean(·) represents the average value of the batch visual cues; MLP(.) represents multi-layer perceptron; Represents the current averaged visual cue feature; represents the text prompt feature of the i+1th layer; Softmax(·) represents exponential normalization of the feature to form a probability distribution; D represents the number of dimensions of the feature.

[0094] The main purpose of PFT is to use visual information to assist classification and provide customized classification text features for each different image sample to improve generalization ability. Therefore, the output of formula (3) is set as the text prompt of the next layer to maintain distinguishability, while the visual prompt of the next layer is randomly initialized again.

[0095] PFT implements the pre-fusion process in the pre-trained model (including visual encoder and text encoder) through prompt embedding interaction. The bimodal prompts aggregate information from each layer and assist each other through prompt interaction. With the independent visual prompt information of each layer, the text prompts are gradually learned through layer-by-layer interaction, adapting different classification text features for each different image information, thereby improving the generalization ability of the model.

[0096] (2) Text-guided Image Mask Module (TIM)

[0097] PFT provides instance-aware generalization for the classified text embedding of sample-related visual information, but the visual information lacks sufficient guidance. For the original pre-trained model, the focus is on the global information of the image without clear distinction of details. However, for the face anti-counterfeiting task, irrelevant factors such as background, lighting and environment can cause great interference. In order to refocus the visual encoder of the vision-language model on the face anti-counterfeiting task, a text-guided image mask (TIM) is proposed. Task-related information is injected into the visual interaction through text guidance, so that the visual encoder is more focused on content related to face anti-counterfeiting. Specifically, given the i-th text encoder f i s and f i r The classification mark features represent the fake text and the real text respectively. First, the text classification mark and the visual classification mark f are calculated. i cThe cosine similarity between them determines the most likely label θ∈{spoof,real}, as follows:

[0098]

[0099] Among them, spoof represents the real label, real represents the forged label; i θ ∈{f i real , f i spoof} represents the set of all current real and fake discriminant feature vectors f i T The elements in .

[0100] Specifically, the cosine similarity calculated above can be used to obtain the current similar text category features. In fact, according to formula (4), the most suitable label for each trial feature layer is different, which brings more convenient flexibility. After obtaining the best label, this is a good source of additional information that can help locate the visual information area related to the face task. Specifically, the selected text feature f is calculated by matrix multiplication i θ With all visual features f i I The attention map between them is used, and a fixed ratio, such as 50%, is taken to mask the markers with lower scores. This process is expressed as:

[0101] s i =softmax(f i θ ×f i I ) (5)

[0102] in, Indicates relative score; l I Indicates the length of the patch token labeled by the image segmentation; softmax represents the exponential normalization of the image patch scores guided by text features to form a probability distribution.

[0103] Text guide mask It is expressed as follows:

[0104]

[0105] Among them, s ij Indicates i The jth score in I ]; τ is the fixed ratio threshold score obtained after the scores are sorted in descending order.

[0106] The above formula uses text information to assist learning, but the way of selecting categories and generating masks is directly coupled with the training process, which will cause the image mask process to overfit with text features. Therefore, the present invention also considers introducing randomness to assist in stabilizing the mask during the training process.

[0107] In more detail, the mask is randomly generated in the same proportion as the language mask M i , the text guide mask and random mask The intersection of i , as shown in the following formula:

[0108]

[0109] After this, the visual features will be used by the text-guided M i Masking, which will better guide the visual encoder layer on which areas should be paid more attention.

[0110] Furthermore, the training loss of the multimodal text-enhanced cross-domain face anti-forgery model (ME-FAS) is explained as follows.

[0111] This method uses multiple loss functions to optimize the multimodal text-enhanced cross-domain face anti-forgery model (ME-FAS). Initially, the method uses the final output features of the image-text pair to calculate the similarity p. Given that the text represents real and fake samples, the method implements a binary cross-entropy loss to supervise the final score:

[0112]

[0113] in, and j They represent the training data source and the labels (real or fake) corresponding to the training samples respectively.

[0114] The model’s ability to distinguish image features is enhanced by adopting the SimCLR loss in self-supervised learning. During training, each image undergoes two different image enhancement processes, which are labeled as σ 1 and σ 2 , these enhancements may respectively design changes in image brightness, contrast, and size. These methods produce two different enhanced views of the image, from which features of these enhanced views are extracted. The alignment of features extracted from the two enhanced views is achieved by using the SimCLR loss, which is described as follows:

[0115]

[0116] In addition, the goal of the method is to align the enhanced image features with the corresponding text features to ensure the best classification accuracy. The calculated alignment loss is shown below:

[0117]

[0118] Among them, f t Indicates that i The corresponding category knowledge text features; and Refers to the features corresponding to the two images enhanced by the original face images during the training process, which are passed through the image encoder to enhance the training process of the model.

[0119] To sum up, the total loss is:

[0120]

[0121] A performance comparison experiment will be conducted as follows.

[0122] The present invention is trained and tested on MSU-MFSD (M), CASIA-MFSD (C), Idiap Replay-Attack (I) and OULU-NPU (O), and the Leave-One-Out (LOO) method is adopted for testing. Three data sets are used for training each time, and the remaining data set is used as a test set for cross-domain testing.

[0123] The present invention uses the half total error rate (HTER) and the area under the receiver operating characteristic curve (AUC) as model performance metrics to compare with other existing methods. The comparison results are shown in Table 1 below.

[0124] Table 1. Comparison of ME-FAS method with other methods

[0125]

[0126] By comparing the most important HTER metric, the proposed method achieves the best performance on half of the settings. Specifically, in terms of average HTER, the proposed method improves by 1.46% over the baseline model FLIP and by 1.43% over the current best model CFPL-FAS. Observations also show that the test results of FLIP under different settings fluctuate significantly, while the proposed method maintains a consistent level of stability. This stability suggests that the proposed method aligns text from different domains with images early on, providing better adaptation and more robust text features, thereby continuously improving performance in different settings through early visual-language model alignment.

[0127] At the same time, in order to further compare the cross-domain performance under restricted data, only M and I are used as source domains for training and the remaining two datasets are used as test domains. The comparison results are shown in Table 2 below.

[0128] Table 2. Comparison of method performance under restricted conditions

[0129]

[0130] As shown in Table 2, the proposed method is further validated under extremely limited source domain test conditions. The proposed method also significantly outperforms the current state-of-the-art methods on limited source data. This shows that even under strict conditions, the proposed method is still effective for unseen target domains due to the introduction of generalization text and enhanced interaction between modalities.

[0131] In order to clarify the performance of the model and the basis for judgment, the features generated by the model were visualized by t-SNE, a Grad-CAM diagram was drawn, and the focus area of ​​the model in the TIM module was displayed. Figure 3 As shown in (a) and 3(b), the lack of pre-trained facial features in the face domain leads to unclear clustering of the initial distribution. Ultimately, the distributions of real and fake faces are clearly close to their corresponding texts. Figure 3 In the Grad-CAM analysis shown in (c), the gradient of the real face (green) is evenly concentrated on the entire face, while the gradient of the paper mask (red on the left) and the printed attack (red on the right) clearly highlights the common attack areas around the eyes and mouth. In addition, a visual representation of the attention of text features at different levels of the TIM module is also added, such as Figure 3 (d) shows that at lower levels, the text emphasizes global detail features, while at higher levels, it focuses on task-related features such as eyes and noses, which is beneficial for generalization judgment.

[0132] In summary, the proposed ME-FAS method has good results in both performance and interpretability. From the perspective of the main evaluation indicator HTER, it has achieved the best detection performance of the current FAS task.

[0133] See also Figure 4 As shown, the present invention also discloses a cross-domain face anti-counterfeiting detection device based on multimodal text enhancement, comprising:

[0134] The original feature acquisition module 401 is used to input two types of description texts including real face description texts and fake face description texts into a pre-trained text encoder, input the image into a pre-trained visual encoder, and extract text category features representing real / deceptive texts at each layer of the text encoder and the visual encoder. i T and visual featuresi I ; The visual features include visual category features f i C and image block features; 0≤i≤N-1, N represents the number of layers of text encoder and visual encoder;

[0135] Hint module 402, for adding trainable text hints to each layer of the text encoder Adding trainable visual cues to each layer of the visual encoder Each layer of visual cues in the visual encoder is obtained by converting the text cues in the current layer through a fully connected layer;

[0136] The fusion and detection module 403 is used to splice the text prompt with the text category feature, and the visual prompt with the visual feature, and perform forward propagation in the corresponding encoder layer; during the forward propagation process, the prompt fusion transfer module and the text-guided image mask module are embedded in the middle layer of each layer of the text encoder and the visual encoder to realize feature interactive fusion, obtain the cosine similarity and mask between the text category feature and the visual feature, and perform true and false face category discrimination based on the cosine similarity and the mask;

[0137] Among them, in the prompt fusion transfer module of the i-th layer, the text prompt Convert into text prompt features and convert visual prompts Converted into visual cue features, the text cue features of the current layer are fused with the visual cue features of the current layer through a multi-head cross-attention mechanism to obtain multi-modal guided text cue features, and the multi-modal guided text cue features are combined with the text category features of the corresponding layer as the input of the next layer of the text encoder;

[0138] In the i-th layer text-guided image mask module, the current similar text category label is calculated using the visual category features. The similar text label performs similarity masking on the image block features output by the current visual encoder layer to obtain a text-guided mask. A random mask of the image block is introduced and intersected with the text-guided mask to obtain the final mask.

[0139] A specific implementation of a cross-domain face anti-counterfeiting detection system based on multimodal text enhancement. The same cross-domain face anti-counterfeiting detection method based on multimodal text enhancement is not repeated in this embodiment.

[0140] It is obvious to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-restrictive, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any figure mark in the claims should not be regarded as limiting the claims involved. In addition, it is obvious that the word "comprising" does not exclude other units or, and the singular does not exclude the plural. Multiple units or systems stated in the system claim can also be implemented by one unit or system through software or hardware.

[0141] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A cross-domain face anti-counterfeiting detection method based on multimodal text enhancement, characterized in that: include: Two types of description texts, including real face description texts and fake face description texts, are input into the pre-trained text encoder, and the image is input into the pre-trained visual encoder. In each layer of the text encoder and the visual encoder, the text category features representing the real / deceptive are extracted respectively. i T and visual features i I ; The visual features include visual category features f i C and image patch features; 0≤i≤N-1, N represents the number of layers of text encoder and visual encoder; Add trainable text hints to each layer of the text encoder Adding trainable visual cues to each layer of the visual encoder Each layer of visual cues in the visual encoder is obtained by converting the text cues in the current layer through a fully connected layer; The text prompts are concatenated with the text category features, and the visual prompts are concatenated with the visual features, and forward propagated in the corresponding encoder layer; during the forward propagation process, the prompt fusion transfer module and the text-guided image mask module are embedded in the middle layer of each layer of the text encoder and the visual encoder to realize feature interactive fusion, obtain the cosine similarity and mask between the text category features and the visual features, and perform true and false face category discrimination based on the cosine similarity and mask; Among them, in the prompt fusion transfer module of the i-th layer, the text prompt Convert into text prompt features and convert visual prompts Convert the text prompt features into visual cue features. The text prompt features of the current layer are fused with the visual cue features of the current layer through a multi-head cross-attention mechanism to obtain multi-modal guided text prompt features. The multi-modal guided text prompt features are combined with the text category features of the corresponding layer as the input of the next layer of the text encoder. In the i-th layer text-guided image mask module, the current similar text category label is calculated using the visual category features, and the current similar text label is used to perform similarity masking on the image block features output by the current visual encoder layer to obtain the text-guided mask; A random mask of image blocks is introduced and intersected with the text-guided mask to obtain the final mask.

2. According to claim 1, the cross-domain face anti-counterfeiting detection method based on multimodal text enhancement is characterized in that: Each layer of the text encoder adds trainable text hints The initial value of is obtained by random initialization and represents the aggregation of text information from the i-th layer.

3. The cross-domain face anti-counterfeiting detection method based on multimodal text enhancement according to claim 1 is characterized in that: The first layer of visual cue vectors Through the text prompt of the first layer The nonlinear transformation is obtained as follows: Here, MLP(.) represents multi-layer perceptron.

4. The cross-domain face anti-counterfeiting detection method based on multimodal text enhancement according to claim 1 is characterized in that: The text hints are concatenated with the text category features, the visual hints are concatenated with the visual features, and forward propagated in the corresponding encoder layers, as shown below: in, Represents the visual features of the i+1th layer; Text category features at the i+1th layer; represents the i-th layer of the visual encoder, represents the i-th layer of the text encoder, represents the visual cue feature after refinement by the i-th layer of the visual encoder, Represents the text prompt features after refinement by the i-th layer of the text encoder.

5. The cross-domain face anti-counterfeiting detection method based on multimodal text enhancement according to claim 4 is characterized in that: The text prompt features of the current layer are fused with the visual prompt features of the current layer through a multi-head cross-attention mechanism to obtain the multi-modal guided text prompt features, which are expressed as follows: Where Mean(·) represents the average value of the batch visual cues; MLP(.) represents multi-layer perceptron; Represents the current averaged visual cue feature; represents the text prompt feature of the i+1th layer; Softmax(·) represents exponential normalization of the feature to form a probability distribution; D represents the number of dimensions of the feature.

6. The cross-domain face anti-counterfeiting detection method based on multimodal text enhancement according to claim 1 is characterized in that: In the text-guided image mask module, the visual category features are used to calculate the current similar text category label, which is expressed as follows: Among them, θ represents the label determined based on cosine similarity, θ∈{spoof,real}, spoof represents the real label, and real represents the forged label; f i θ ∈{f i real , f i spoof } represents an element in the set of all current real and fake discriminant feature vectors.

7. The cross-domain face anti-counterfeiting detection method based on multimodal text enhancement according to claim 6 is characterized in that: The similar text features perform similarity masking on the image block features output by the current visual encoder layer to obtain a text guidance mask; The random mask of the image block is introduced and intersected with the text-guided mask to obtain the final mask, including: Calculate the selected text features f by matrix multiplication i θ With all visual markers f i I The attention map between is represented as follows: s i =softmax(f i θ ×f i I ) in, Indicates relative score; l I represents the length of the image marker block from which the image is segmented; softmax represents the exponential normalization of the image block scores guided by text features to form a probability distribution; f i θ Represents the elements in the set of all current real and fake discriminant feature vectors; Text guide mask It is expressed as follows: Among them, s ij Indicates i The jth score in I ]; τ is the fixed ratio threshold score obtained after the scores are sorted in descending order; Generate a random mask of the same proportions as the text-guided mask Direct the text to the mask and random mask The intersection of is used as the final mask, which is expressed as follows:

8. A cross-domain face anti-counterfeiting detection device based on multimodal text enhancement, characterized in that: include: The original feature acquisition module is used to input two types of description texts, including real face description texts and fake face description texts, into the pre-trained text encoder, and input the image into the pre-trained visual encoder. In each layer of the text encoder and the visual encoder, the text category features representing the real / deceptive are extracted respectively. i T and visual features i I ; The visual features include visual category features f i C and image patch features; 0≤i≤N-1, N represents the number of layers of text encoder and visual encoder; Hint module, which adds trainable text hints to each layer of the text encoder Adding trainable visual cues to each layer of the visual encoder Each layer of visual cues in the visual encoder is obtained by converting the text cues in the current layer through a fully connected layer; The fusion and detection module is used to splice the text prompts with the text category features, the visual prompts with the visual features, and perform forward propagation in the corresponding encoder layer; during the forward propagation process, the prompt fusion transfer module and the text-guided image mask module are embedded in the middle layer of each layer of the text encoder and the visual encoder to realize feature interactive fusion, obtain the cosine similarity and mask between the text category features and the visual features, and perform true and false face category discrimination based on the cosine similarity and mask; Among them, in the prompt fusion transfer module of the i-th layer, the text prompt Convert into text prompt features and convert visual prompts Convert the text prompt features into visual cue features. The text prompt features of the current layer are fused with the visual cue features of the current layer through a multi-head cross-attention mechanism to obtain multi-modal guided text prompt features. The multi-modal guided text prompt features are combined with the text category features of the corresponding layer as the input of the next layer of the text encoder. In the i-th layer text-guided image mask module, the current similar text category label is calculated using the visual category features, and the current similar text label is used to perform similarity masking on the image block features output by the current visual encoder layer to obtain the text-guided mask; A random mask of image blocks is introduced and intersected with the text-guided mask to obtain the final mask.

Images