Multimodal face liveness detection methods, media, and equipment based on image-text comparison learning.

Abstract

This invention provides a multimodal face liveness detection method, medium, and device based on image-text comparison learning. The method employs a CLIP network architecture for face liveness detection. The CLIP network architecture includes a text encoder, an image encoder, and a multimodal information fusion module. Real and fake face labels are expanded into a text prompt set, which is then fed into the text encoder to obtain text features. The three modalities of the face image are input into the image encoder to extract image features. The cosine similarity between the three modal image features and the text features is calculated to obtain classification probabilities. If the confidence level of any modality's classification probability is high, the detection result is directly obtained. If the confidence levels of all three modalities are low, the three modal image features are fused and then calculated with the text features to obtain the detection result. This method fully utilizes the cross-modal knowledge inherent in the CLIP pre-trained model, extracting the basic features of the face image and text prompts, thus improving the model's generalization ability.

Description

Technical Field

[0001] This invention relates to the field of liveness detection technology, and more specifically, to a multimodal face liveness detection method, medium, and device based on image-text comparison learning. Background Technology

[0002] Face liveness detection is a crucial part of facial recognition, ensuring its reliability in financial and security scenarios, such as facial recognition for ATM deposits and withdrawals, and facial payment. However, the types of attacks against facial recognition are rapidly increasing, and unimodal face liveness detection is relatively weak against such attacks. Multimodal face images can provide additional and complementary information, significantly improving the robustness of face liveness detection. Specifically, 2D attacks based on printing are easily distinguishable in the depth modality, but difficult to distinguish in the visible light modality. Under complex lighting conditions, RGB images become unclear, overexposed, or underexposed, while infrared images are unaffected by complex lighting.

[0003] However, current multimodal face liveness detection has the following limitations, preventing its widespread use in certain application scenarios: 1) Limited generalization performance: Current multimodal face liveness detection models use a single feature fusion method, which is basically feature concatenation or feature weighting, or single-modal attention mechanism weighting, resulting in limited recognition effectiveness; 2) When existing single-modal methods are applied to multimodal applications, the recognition pass rate in multimodal applications is lower than that in single-modal applications; 3) Existing multimodal methods still use basic image processing methods, including the use of classic neural network models, which is limited in scope and leads to poor recognition accuracy and universality.

[0004] Orbbec's 2021 invention patent, "A Multimodal Face Liveness Detection Method and System" (Publication No.: CN112487922A), describes a method that performs face detection on three modalities (color image, infrared image, and depth image) separately, feeds them into three neural networks for face feature extraction, and then merges the features to determine whether the face is alive. Sun Yat-sen University's late 2021 invention patent, "A Single-Modal Face Liveness Detection Method Based on Multimodal Face Training" (Publication No.: CN113705400A), proposes a method that generates infrared and depth maps from visible light images and then stitches together features from the three modalities. However, this method exhibits poor performance in actual testing, and the generator is unstable. The main reason is the significant difference between the image generated by the generative adversarial network and the image actually captured by the camera, making this method impractical. The invention patent "A Multimodal Fusion Face Liveness Detection Model Generation Method and Device, Electronic Equipment" (Publication No.: CN116311451A) proposed by Hangzhou Qiyuan Vision Company in 2023 only includes two modalities: color image and infrared image. This method achieves multimodal detection through cross-comparison, generating an infrared image from visible light and a visible light image from the infrared image. The authenticity is determined by comparing the generated image with the original image. The biggest drawback of this method is the instability of the generation network. The invention patent "A Multimodal Face Anti-Fraud Detection Method Based on AR-MLP" (Publication No.: CN117011911A) proposed by Zhejiang Sci-Tech University in 2023 processes the input face RGB image, depth image, and infrared image to achieve multimodal face feature fusion. The fusion method involves feature fusion of patches of the same face region from the three modalities. However, actual testing revealed limited recognition performance. The invention patent "A Multimodal Face Liveness Detection Method Based on Attention Mechanism" (Publication No.: CN117894082A) proposed by Beijing University of Technology in 2024; this method performs image preprocessing, feature extraction, and multi-level feature fusion on three modalities (color image, infrared image, and depth image); according to the experimental results given in the patent, the BPCER is 1.82% in a single database, which is quite good, but it is still far from practical application, because practical application will definitely encounter multiple domains, that is, cross-domain testing, and the cross-domain performance is still not good in actual testing.

[0005] In summary, the unknown deployment environment, test samples, and data distribution of face liveness detection algorithms in real-world applications pose significant challenges to the model's domain generalization ability. Even with the help of multimodal information from multimodal sensors, advanced algorithms proposed in recent years, and high-quality training datasets, it is still impossible to apply the model to real-world applications, especially in scenarios with complex environments and varying lighting conditions.

[0006] With the emergence of large visual-text modeling techniques, learning feature representations directly from the raw text of images is a promising option because it leverages a wider range of supervision sources. The CLIP algorithm demonstrates that a simple pre-trained model predicting which text matches which image on a dataset of 400 million (image, text) pairs collected from the internet is an effective and scalable method for learning image representations from scratch. After pre-training, using natural language as a reference to learn new information from visual images allows for direct zero-shot transfer to downstream tasks. However, in face liveness detection tasks, how to leverage textual information to improve model performance and domain generalization remains a gap. Summary of the Invention

[0007] To overcome the shortcomings and deficiencies of the existing technology, the present invention aims to provide a multimodal face liveness detection method, medium and device based on image-text comparison learning. The method introduces the CLIP pre-trained model, makes full use of the cross-modal knowledge contained in the CLIP pre-trained model in large-scale cross-modal scenarios, extracts the basic features of face images and text prompts, and improves the generalization ability of the face liveness detection model.

[0008] To achieve the above objectives, the present invention provides a multimodal face liveness detection method based on image-text comparison learning, employing a CLIP network architecture for face liveness detection. The CLIP network architecture refers to a CLIP network architecture for face liveness detection obtained through fine-tuning training based on a CLIP pre-trained model. The CLIP network architecture includes a text encoder for extracting features from text, three image encoders for extracting image features from the RGB, IR, and Depth modalities of the face image, and a multimodal information fusion module for fusing and stitching the image features from the three modalities.

[0009] The multimodal face liveness detection method based on image-text comparison learning includes the following steps:

[0010] Step S1: Expand the real face labels and fake face labels of the face image dataset into a text cue set, and feed it into a text encoder to obtain text features of real face types. Text features of fake faces

[0011] Step S2: Input the RGB, IR, and Depth modalities of the face image into the image encoder corresponding to the CLIP network architecture, and extract the image features of the three modalities.

[0012] Step S3: Extract image features from the three modalities. Respectively with text features Calculate the cosine similarity between real and fake faces to obtain the classification probability of real and fake faces;

[0013] Step S4: If the classification probability confidence of a real or fake face in any modality is high, then the type corresponding to the high confidence classification probability of that modality is taken as the face liveness detection result.

[0014] If the classification probability confidence scores for real and fake faces in all three modalities are low, the image features from the three modalities are input into the multimodal information fusion module to obtain fused image features. Fusion image features The cosine similarity between the text features and the real face and fake face is calculated to obtain the classification probability of real face and fake face. The type with the higher classification probability is taken as the face liveness detection result.

[0015] Preferably, step S2 refers to: inputting the RGB, IR, and Depth modalities of the face image into the image encoder corresponding to the CLIP network architecture; each image encoder is a ViT network model composed of multiple Transformer coding layers; for each modality of the face image, the image encoder first inputs the modality-corresponding image I∈R H×W×C Where H, W, and C represent the height, width, and number of channels of the image, respectively, and the image is uniformly divided into b image blocks, each with a width and height of s; where, The segmented image patches are represented as sequence E by a single layer of two-dimensional convolution. I ={e1 I e2 I ,…e b I}∈R b ×(k-1) Then in sequence E I Add a category tag e to the header cls After adding positional encoding, the image is input into the image encoder; the last Transformer encoding layer of the image encoder outputs the image features. Among them, f I Indicates category tag e cls The corresponding output is considered as global image features; The j-th image patch represents the local feature; the image features of the RGB, IR, and Depth modalities are denoted as follows:

[0016] Preferably, steps S3 and S4, calculating the classification probabilities of real faces and fake faces, refer to:

[0017]

[0018] in, The type is The classification probability; r represents the real face type, s represents the fake face type; I∈{RGB,IR,Depth,fuse}; sim(·,·) represents the cosine similarity score, and τ is the temperature coefficient; This represents the i-th image feature of the i-th modality. Text features representing the true face type, Textual features representing fake face types Textual features representing real and fake face types.

[0019] Preferably, the multimodal information fusion module refers to the method of fusing and stitching together image features from three modalities using a multimodal stitching method or a multimodal fusion cross-attention mechanism to obtain fused image features.

[0020] Preferably, the multimodal splicing method refers to:

[0021]

[0022] Here, contact represents feature connection; conv represents convolution computation.

[0023] Preferably, the multimodal fusion cross-attention mechanism refers to:

[0024] First, we use three parameters, W... q , W k and W v Linear layers are used to extract features separately. and Where m∈(IR,Depth):

[0025]

[0026] in, A classification token for the RGB modality; Image block tokens for IR modes; For the image block token of the Depth modality;

[0027] Subsequently, the attention matrix A is generated using the softmax function to extract attention features.

[0028] in, is the scaling factor, where D and h are the dimension of the feature vector and the number of heads in the multi-head attention, respectively; A is a one-dimensional vector, and n is the length of the one-dimensional vector A;

[0029] Attention features RGB modal image features and attention features The fused image features are obtained by stitching together.

[0030]

[0031] Where conv represents convolution calculation.

[0032] Preferably, step S1 includes: constructing a text cue set of real face labels and fake face labels for the face image dataset;

[0033] Align each face image sample in the face image dataset with the text prompt T in the text prompt set;

[0034] For each text prompt T, extract text features: First, perform word segmentation and truncation / padding operations, and insert text start and end markers to unify the text prompt length to k words; then, use word embedding methods to transform the text prompt from a word sequence into a vector sequence E. T ={e1 T e2 T ,…e k T}∈R k×512 ; the vector sequence E T After adding positional encoding, the text encoder is fed into the CLIP network architecture to learn the vector sequence E through a self-attention mechanism. T The semantic information contained within; the text features output by the last encoding layer of the text encoder. r represents the real face type, s represents the fake face type; text features Among them, f T The output corresponding to the start marker of the text is considered a global feature of the text. This represents the local text features corresponding to the j-th word.

[0035] A readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the multimodal face liveness detection method based on image-text comparison learning.

[0036] A computer device includes a processor and a memory for storing a processor-executable program, wherein when the processor executes the program stored in the memory, it implements the multimodal face liveness detection method based on image-text comparison learning.

[0037] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0038] To address the issue of insufficient generalization ability of current face liveness detection technology when dealing with unknown attack patterns, multiple types of camera sensors, and complex environmental changes, this invention introduces the CLIP pre-trained model. By fully utilizing the cross-modal knowledge contained in the CLIP pre-trained model in large-scale cross-modal scenarios, the basic features of face images and text prompts are extracted, thereby improving the generalization ability of the face liveness detection model. Furthermore, a cross-modal fusion attention mechanism is proposed to obtain more effective attention features. Attached Figure Description

[0039] Figure 1 This is a flowchart of the multimodal face liveness detection method based on image-text comparison learning according to the present invention;

[0040] Figure 2 This is a schematic diagram illustrating the calculation of cosine similarity between the three modal image features and the text features of this invention;

[0041] Figure 3 This is a schematic diagram illustrating the calculation of cosine similarity between the three modal image features fused using a multimodal stitching method and the text features of the present invention;

[0042] Figure 4 This is a schematic diagram illustrating the calculation of cosine similarity between the three modal image features of this invention, which are fused using a multimodal fusion cross-attention mechanism, and the text features.

[0043] Figure 5 This is a schematic diagram of the cross-attention mechanism of the present invention;

[0044] Figure 6 This is a schematic diagram illustrating how existing methods classify real and fake faces by weighting the output scores of three modalities of image features through an MLP. Detailed Implementation

[0045] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0046] Example 1

[0047] This implementation presents a multimodal face liveness detection method based on image-text comparison learning, employing a CLIP network architecture for face liveness detection. The CLIP network architecture refers to a network architecture for face liveness detection obtained through fine-tuning training based on a CLIP pre-trained model. The CLIP pre-trained model has demonstrated powerful semantic information mining capabilities, zero-shot capability, and modality alignment capabilities on multiple downstream tasks. Face liveness detection is also an image-text matching task. This invention uses the CLIP network architecture to extract basic features from face images and retrieve text, thereby fully utilizing the pre-trained knowledge inherent in the model. The CLIP network architecture includes a text encoder for feature extraction from text, three image encoders for extracting image features from the RGB, IR, and Depth modalities of the face image, respectively, and a multimodal information fusion module for fusing and concatenating the image features from the three modalities.

[0048] Multimodal face liveness detection methods based on image-text comparison learning, such as Figure 1 As shown, it includes the following steps:

[0049] Step S1: Expand the real face labels and fake face labels of the face image dataset into a text cue set; and feed it into a text encoder to obtain text features of the real face type. Text features of fake faces .

[0050] Since the face anti-fraud task lacks the text cues required by the CLIP network architecture, a text cue set needs to be constructed. Based on the real face labels and fake face labels from the face image dataset, all text cues are constructed as "a photo of a...". <class>"face", among which <class>Let Real and Spoof represent real and fake faces, respectively. Consider that each class can have multiple distinct text cues, and compute text representation features for each cue. The average features of these representations give the overall features of the context in the embedding space. Aligning the images with a large set of natural language text cues enables the model to learn class-specific cues; examples are shown in Table 1.

[0051] Table 1. Examples of text hint sets extended from real face and fake face tags.

[0052]

[0053] Align each face image sample in the face image dataset with the text prompt T in the text prompt set;

[0054] For each text prompt T, extract text features: First, perform word segmentation and truncation / padding operations, and insert text start and end markers to unify the text prompt length to k words, for example, k=77; then, use word embedding methods to transform the text prompt from a word sequence into a vector sequence E. T ={e1 T e2 T ,…e k T }∈R k×512 ; the vector sequence E T After adding positional encoding, the text encoder is fed into the CLIP network architecture to learn the vector sequence E through a self-attention mechanism. T The semantic information contained within; the text features output by the last encoding layer of the text encoder. r represents the real face type, s represents the fake face type; text features Among them, f T The output corresponding to the start marker of the text is considered a global feature of the text. This represents the local text features corresponding to the j-th word.

[0055] Step S2: Input the RGB, IR, and Depth modalities of the face image into the image encoder corresponding to the CLIP network architecture, and extract the image features of the three modalities.

[0056] Specifically, the RGB, IR, and Depth modalities of the face image are input into the image encoders corresponding to the CLIP network architecture; each image encoder is a ViT network model composed of multiple Transformer coding layers; for each modality of the face image, the image encoder first converts the modality-corresponding image I∈R... H×W×C Where H, W, and C represent the height, width, and number of channels of the image, respectively, and the image is uniformly divided into b image blocks, each with a width and height of s; where, The segmented image patches are represented as sequence E by a single layer of two-dimensional convolution. I ={e1 I e2 I ,…e b I }∈R b×(k-1) Then in sequence E I Add a category tag e to the header cls After adding positional encoding, the image is input into the image encoder; the last Transformer encoding layer of the image encoder outputs the image features. Among them, f I Indicates category tag e cls The corresponding output is considered as global image features; The j-th image patch represents the local feature; the image features of the RGB, IR, and Depth modalities are denoted as follows:

[0057] Step S3: Extract image features from the three modalities. Respectively with text features Calculate the cosine similarity between real and fake faces to obtain the classification probability of real and fake faces.

[0058] Step S4: If the classification probability confidence of a real or fake face in any modality is high, the loss contribution of samples in the other two modalities can be reduced, and the type corresponding to the high-confidence classification probability of that modality is used as the face liveness detection result. Figure 2 As shown. High confidence means that the classification probability is greater than or equal to a set threshold, for example, the set threshold is 0.9.

[0059] If the classification probability confidence scores for real and fake faces in all three modalities are low, the image features from the three modalities are input into the multimodal information fusion module to obtain fused image features. Fusion image features The cosine similarity between the text features and the real face and fake face is calculated to obtain the classification probability of real face and fake face. The type with the higher classification probability is taken as the face liveness detection result.

[0060] Steps S3 and S4, calculating the classification probabilities of real faces and fake faces, refer to:

[0061]

[0062] in, The type is The classification probability; r represents the real face type, s represents the fake face type; I∈{RGB,IR,Depth,fuse}; sim(·,·) represents the cosine similarity score, and τ is the temperature coefficient; This represents the i-th image feature of the i-th modality. Text features representing the true face type, Textual features representing fake face types Textual features representing real and fake face types.

[0063] The multimodal information fusion module refers to the process of fusing and stitching image features from three modalities using a multimodal stitching method or a multimodal fusion cross-attention mechanism to obtain fused image features.

[0064] The multimodal splicing methods include, for example Figure 3 What is shown is:

[0065]

[0066] Here, contact represents feature connection; conv represents convolution computation.

[0067] Multimodal fusion cross-attention mechanisms, such as Figure 4 and Figure 5 As shown.

[0068] Effectively fusing multimodal information to generate more robust feature representations is the primary task of the Multi-modal Fusion-Attention (MFA) module. Since the classification (CLS) token is essentially a proxy for an image, it summarizes all image patch tokens used for classification. Inspired by cross-attention, this invention employs a simple yet effective multimodal fusion strategy that uses the classification token of an RGB modality sequence as a query to exchange information with image patch tokens from other modality sequences. MFA is similar to self-attention but differs in the following ways: In terms of input form, unlike self-attention which takes a single modality sample as input, MFA takes a pair of samples from different modalities as input, where the query matrix is ​​obtained from the current modality sequence, while the key-value pair matrix comes from other modality sequences. Regarding the fusion strategy, self-attention aims to mine the relationships between all tokens (including classification tokens and image patch tokens) in the input samples, while MFA fuses other modality information from the current modality sequence by calculating the dependencies between classification tokens (from the current modality) and image patch tokens (from another modality).

[0069] Specifically, firstly, three parameters are used, namely W q W k and W v Linear layers are used to extract features separately. and Where m∈(IR,Depth):

[0070]

[0071] in, A classification token for the RGB modality; Image block tokens for IR modes; For the image block token of the Depth modality;

[0072] Subsequently, the attention matrix A is generated using the softmax function to extract attention features.

[0073] in, is the scaling factor, where D and h are the dimension of the feature vector and the number of heads in the multi-head attention, respectively; A is a one-dimensional vector, and n is the length of the one-dimensional vector A;

[0074] Attention features RGB modal image features and attention features The fused image features are obtained by stitching together.

[0075]

[0076] Where conv represents convolution calculation.

[0077] During the fine-tuning training phase, after calculating the cosine similarity, it is treated as class logits and used for cross-entropy loss calculation to obtain the class loss L. cls .

[0078] To verify the beneficial effects of the method of the present invention, experiments were conducted on four public datasets, CASIA-CeFA (C), PADISI-Face (P), CASIA-SURF (S), and WMCA (W), comparing the method of the present invention with existing methods.

[0079] The CASIA-SURF dataset contains 21,000 multimodal videos recorded by 1,000 volunteers. The data is sourced from multiple channels (visible light, depth map, and near-infrared), employing either flattening or curling printing attacks to randomly remove areas such as eyes, nose, and mouth. An upgraded version, CASIA-SURF CeFA, was released in 2020, adding cross-ethnic volunteer identities to test the algorithm's cross-ethnic generalization. The Multi-Channel Rendering Attack (WMCA) dataset contains 1,941 short video recordings from 72 different identities of real people and rendering camouflage attacks. The data is recorded from several channels, including color, depth, infrared, and thermal imaging. The PADISI-Face dataset contains 1,105 real-person videos and 924 rendering attack videos, with as many as 37 attack types.

[0080] Existing methods refer to the Baseline: Representations generated by the CLIP model have demonstrated impressive plug-and-play performance in many downstream vision applications based on natural images, such as classification, object detection, and segmentation. However, these features cannot be directly used for face liveness detection tasks that require recognizing subtle variations between similar face images. Therefore, we first fine-tune the visual backbone only for face liveness detection tasks, and refer to this approach as the Baseline. In this approach, a pre-trained CLIP model is used, employing only its image encoder and discarding the text encoder. This provides us with a simple visual transformer (ViT) initialized with pre-trained language-image weights. Given a balanced batch of images from N source domains, the image encoder is used to transform the last transformer block (v... K Extract class tags (c) K The image encoder then passes it to a multilayer perceptron (MLP) classification head to determine whether the input image is fake or real. The image encoder and MLP head use standard cross-entropy loss Lf. ce The update is performed. The final classification values ​​for the three modalities are obtained by weighting the scores to arrive at the final score, which is then used to obtain the face liveness detection result, such as... Figure 6 As shown.

[0081] The experimental comparison results of the method of this invention with existing methods on four public datasets CASIA-CeFA (C), PADISI-Face (P), CASIA-SURF (S), and WMCA (W) are shown in Table 2. Wherein, CPS->W represents training sets C, P, and S, and test set W; CPW->S represents training sets C, P, and W, and test set S; CSW->P represents training sets C, S, and W, and test set P; and PSW->C represents training sets P, S, and W, and test set C.

[0082] Table 2 Comparison of experiments involving attention fusion mechanisms (unit: %)

[0083]

[0084] As shown in Table 2, the method of using a single-modal + multi-modal fusion cross-attention mechanism to fuse data obtained the best HTER (half-error rate) and the highest AUC (accuracy) by testing on four public datasets using the leave-one-out method (selecting three datasets as training sets and the other as test sets to simulate cross-dataset testing), which means it has the best test performance.

[0085] Example 2

[0086] This embodiment provides a readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the multimodal face liveness detection method based on image-text comparison learning as described in Embodiment 1.

[0087] Example 3

[0088] This embodiment discloses a computer device, including a processor and a memory for storing processor-executable programs. When the processor executes the program stored in the memory, it implements the multimodal face liveness detection method based on image-text comparison learning as described in Embodiment 1.

[0089] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.< / class> < / class>

Claims

1. A multimodal face liveness detection method based on image-text comparison learning, characterized in that: The CLIP network architecture is used to achieve face liveness detection. The CLIP network architecture refers to the CLIP network architecture for face liveness detection obtained by fine-tuning the CLIP pre-trained model. The CLIP network architecture includes a text encoder for extracting features from text, three image encoders for extracting image features from the RGB, IR, and Depth modalities of the face image, and a multimodal information fusion module for fusing and stitching the image features from the three modalities. The multimodal face liveness detection method based on image-text comparison learning includes the following steps: Step S1: Expand the real face labels and fake face labels of the face image dataset into a text cue set, and feed it into a text encoder to obtain text features of real face types. Text features of fake faces Step S2: Input the RGB, IR, and Depth modalities of the face image into the image encoder corresponding to the CLIP network architecture, and extract the image features of the three modalities. Step S3: Extract image features from the three modalities. Respectively with text features Calculate the cosine similarity between real and fake faces to obtain the classification probability of real and fake faces; Step S4: If the classification probability confidence of a real or fake face in any modality is high, then the type corresponding to the high confidence classification probability of that modality is taken as the face liveness detection result. If the classification probability confidence scores for real and fake faces in all three modalities are low, the image features from the three modalities are input into the multimodal information fusion module to obtain fused image features. Fusion image features The cosine similarity between the text features and the real face and fake face is calculated to obtain the classification probability of real face and fake face. The type with the higher classification probability is taken as the face liveness detection result.

2. The multimodal face liveness detection method based on image-text comparison learning according to claim 1, characterized in that: Step S2 refers to: inputting the RGB, IR, and Depth modalities of the face image into the image encoder corresponding to the CLIP network architecture; each image encoder is a ViT network model composed of multiple Transformer coding layers; for each modality of the face image, the image encoder first inputs the modality-corresponding image I∈R H×W×C Where H, W, and C represent the height, width, and number of channels of the image, respectively, and the image is uniformly divided into b image blocks, each with a width and height of s; where, The segmented image patches are represented as sequence E by a single layer of two-dimensional convolution. I ={e1 I e2 I ,…e b I }∈R b×(k-1) Then in sequence E I Add a category tag e to the header cls After adding positional encoding, the data is input into the image encoder; the last Transformer encoding layer of the image encoder outputs the image features. Among them, f I Indicates category tag e cls The corresponding output is considered as global image features; j = 1, 2, ..., b, representing the local features corresponding to the j-th image patch; the image features of the RGB, IR, and Depth modalities are denoted as follows:

3. The multimodal face liveness detection method based on image-text comparison learning according to claim 1, characterized in that: Steps S3 and S4, calculating the classification probabilities of real faces and fake faces, refer to: in, The type is The classification probability; r represents the real face type, s represents the fake face type; I∈{RGB,IR,Depth,fuse}; sim(·,·) represents the cosine similarity score, and τ is the temperature coefficient; This represents the i-th image feature of the i-th modality. Text features representing the true face type, Textual features representing fake face types Textual features representing real and fake face types.

4. The multimodal face liveness detection method based on image-text comparison learning according to claim 1, characterized in that: The multimodal information fusion module refers to the method of fusing and stitching image features from three modalities using a multimodal stitching method or a multimodal fusion cross-attention mechanism to obtain fused image features.

5. The multimodal face liveness detection method based on image-text comparison learning according to claim 4, characterized in that: The multimodal splicing method refers to: Here, contact represents feature connection; conv represents convolution computation.

6. The multimodal face liveness detection method based on image-text comparison learning according to claim 4, characterized in that: The aforementioned multimodal fusion cross-attention mechanism refers to: First, we use three parameters, W... q W k and W v Linear layers are used to extract features separately. and Where m∈(IR,Depth): in, A classification token for the RGB modality; Image block tokens for IR modes; For the image block token of the Depth modality; Subsequently, the attention matrix A is generated using the softmax function to extract attention features. in, is the scaling factor, where D and h are the dimension of the feature vector and the number of heads in the multi-head attention, respectively; A is a one-dimensional vector, and n is the length of the one-dimensional vector A; Attention features RGB modal image features and attention characteristics The fused image features are obtained by stitching together. Where conv represents convolution calculation.

7. The multimodal face liveness detection method based on image-text comparison learning according to claim 1, characterized in that: Step S1 includes: constructing a text prompt set for real face labels and fake face labels of the face image dataset; Align each face image sample in the face image dataset with the text prompt T in the text prompt set; For each text prompt T, extract text features: First, perform word segmentation and truncation / padding operations, and insert text start and end markers to unify the text prompt length to k words; then, use word embedding methods to transform the text prompt from a word sequence into a vector sequence E. T ={e1 T e2 T ,…e k T }∈R k×512 ; the vector sequence E T After adding positional encoding, the text encoder is fed into the CLIP network architecture to learn the vector sequence E through a self-attention mechanism. T The semantic information contained within; the text features output by the last encoding layer of the text encoder. r represents the real face type, s represents the fake face type; text features Among them, f T The output corresponding to the start marker of the text is considered a global feature of the text. j = 1, 2, ..., k represents the local text features corresponding to the j-th word.

8. A readable storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, causes the processor to perform the multimodal face liveness detection method based on image-text comparison learning as described in any one of claims 1-7.

9. A computer device comprising a processor and a memory for storing a processor-executable program, characterized in that, When the processor executes the program stored in the memory, it implements the multimodal face liveness detection method based on image-text comparison learning as described in any one of claims 1-7.

Images