Multi-modal face in-vivo detection method, medium and equipment based on image-text contrast learning

Abstract

The invention provides a multi-modal face in-vivo detection method for image-text contrast learning, a medium and equipment. The method comprises the following steps of: realizing human face living body detection by adopting a CLIP network architecture; the CLIP network architecture comprises a text encoder, an image encoder and a multi-modal information fusion module; expanding the true and false face labels into a text prompt set, and sending the text prompt set into a text encoder to obtain text features; respectively inputting the three modals of the face image into an image encoder to extract image features; calculating cosine similarity between the three modal image features and the text features to obtain classification probabilities; if any modal classification probability confidence coefficient is high, a detection result is directly obtained; and if the confidence coefficients of the three modalities are not high, fusing the image features of the three modalities, and calculating the cosine similarity with the text features to obtain a detection result. According to the method, cross-modal knowledge contained in a CLIP pre-training model is fully utilized, basic features of face images and text prompts are extracted, and the generalization ability of the model is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of liveness detection, and more specifically to a multimodal face liveness detection method, medium and device based on image-text contrast learning. Background Art

[0002] Face liveness detection is a very important part of face recognition tasks. It can ensure the reliability of face recognition in the financial industry or security scenarios, such as face-swiping deposits and withdrawals at self-service machines, face payment, etc. Due to the rapid increase in the types of attacks on facial recognition, the ability of single-modal face liveness detection to resist attacks is weak. Multimodal face images can provide additional and supplementary information, which can greatly improve the robustness of face liveness detection. Specifically, print-based 2D attacks are very easy to distinguish in the depth mode, but difficult to distinguish in the visible light mode. Under complex light interference, RBG images are not clear, overexposed or too dark, while infrared images are not affected by complex light.

[0003] However, the current multimodal face liveness detection has the following limitations, which prevent it from being widely used in certain application scenarios: 1) Limited generalization performance: The current multimodal face liveness detection model has a single feature fusion method, which is basically feature splicing or feature weighting, or weighted attention mechanism of a single modality, and the recognition effect is limited; 2) The existing single-modal method is applied to multimodality, and the recognition pass rate of this method in multimodality is lower than that of single modality; 3) The existing multimodal method still uses basic image processing methods, including the use of classic neural network models for processing, with a single means, resulting in poor recognition accuracy and universality.

[0004] The invention patent "A multimodal face liveness detection method and system" (publication number: CN112487922A) proposed by Orbbec in 2021; this method performs face detection on three modalities (color image, infrared image and depth image) respectively, and sends them into three neural networks for face feature extraction, and then merges the features to determine whether it is alive. The invention patent "A single-modal face liveness detection method based on multimodal face training" (publication number: CN113705400A) proposed by Sun Yat-sen University at the end of 2021; the idea of ​​this method is to generate infrared and depth maps through visible light images, and then use the features of the three modalities for splicing; however, the actual test performance of this method is very poor, and the generator is unstable. The main reason is that there is a big gap between the image generated by the generative adversarial network and the image actually collected by the camera, and this method is not practical. The invention patent "A multimodal fusion face liveness detection model generation method and device, electronic equipment" (publication number: CN116311451A) proposed by Hangzhou Qiyuan Vision Company in 2023; the multimodality of this method only includes two modes, namely color image and infrared image, and the method is implemented by cross-comparison, that is, visible light generates infrared light image, infrared light image generates visible light image, and the authenticity is identified by generating images and original images. The biggest defect of this method is the instability of the generation network. The invention patent "A multimodal face anti-fraud detection method based on AR-MLP" (publication number: CN117011911A) proposed by Zhejiang University of Technology in 2023; this method performs face image processing on the input face RGB image, depth image and infrared image to realize face multimodal feature fusion, and the fusion method is the feature fusion of the same face area patch of the three modalities, but actual testing found that the recognition performance is limited. The invention patent "A multimodal face liveness detection method based on attention mechanism" (publication number: CN117894082A) proposed by Beijing University of Technology in 2024; this method performs image preprocessing, feature extraction, and multi-level feature fusion on three modalities (color images, infrared images, and depth images) respectively; judging from the experimental results given in the patent, the BPCER obtained in a single database is 1.82%, which is a good effect, but it is still out of practical application, because practical applications must encounter multiple domains, that is, cross-domain testing, and the actual test of cross-domain performance is still not good.

[0005] In summary, the unknown deployment environment, test samples, and data distribution of face liveness detection algorithms in practical applications pose great challenges to the domain generalization of the model. Even with the help of multimodal information from multimodal sensors, advanced algorithms proposed in recent years, and high-quality training data sets, the model still cannot be applied to practical applications, especially in scenes with complex environments and changing lighting backgrounds.

[0006] With the advent of large visual-text model technology, learning feature representations directly from the original text of an image is a promising option because it leverages a wider range of supervision sources. The CLIP algorithm shows that a simple pre-trained model that predicts which text matches which image is an effective and scalable way to learn image representations from scratch on a dataset of 400 million (image, text) pairs collected from the Internet. After pre-training, using natural language as a reference to learn new information in visual images can be directly transferred to downstream tasks in a zero-shot manner. However, in the task of face liveness detection, how to use text information to improve the performance of the model and domain generalization performance is still a blank. Summary of the invention

[0007] In order to overcome the shortcomings and deficiencies in the prior art, the purpose of the present invention is to provide a multimodal face liveness detection method, medium and device based on image-text contrast learning; the method introduces the CLIP pre-training model, fully utilizes the cross-modal knowledge contained in the CLIP pre-training model in large-scale cross-modality, extracts the basic features of face images and text prompts, and improves the generalization ability of the face liveness detection model.

[0008] In order to achieve the above object, the present invention is implemented by the following technical scheme: a multimodal face liveness detection method based on image-text contrast learning, adopting CLIP network architecture to realize face liveness detection; the CLIP network architecture refers to: fine-tuning training based on CLIP pre-training model to obtain CLIP network architecture for face liveness detection; the CLIP network architecture includes a text encoder for feature extraction of text, three image encoders for extracting image features of three modes of face image RGB, IR, and Depth respectively, and a multimodal information fusion module for fusing and splicing the image features of the three modes;

[0009] The multimodal face liveness detection method based on image-text contrast learning includes the following steps:

[0010] Step S1: Expand the real face labels and fake face labels of the face image dataset into a text prompt set and send it to the text encoder to obtain the text features of the real face type. and text features of fake face types

[0011] Step S2: Input the three modes of face image RGB, IR, and Depth into the image encoder corresponding to the CLIP network architecture respectively to extract the image features of the three modes.

[0012] Step S3: The image features of the three modalities Respectively with text features Calculate the cosine similarity of the real face and the fake face to get the classification probability of the real face and the fake face;

[0013] Step S4: If the classification probability confidence of the real face or the fake face of any modality is high, the type corresponding to the high confidence classification probability of the modality is used as the face liveness detection result;

[0014] If the classification probability confidence of the true face and the fake face of the three modalities is not high, the image features of the three modalities are input into the multimodal information fusion module to obtain the fused image features. Fusion image features The cosine similarity of the real face and the fake face is calculated with the text features to obtain the classification probability of the real face and the fake face, and the type with higher classification probability is taken as the face liveness detection result.

[0015] Preferably, step S2 refers to: inputting the three modes of the face image RGB, IR, and Depth into the image encoder corresponding to the CLIP network architecture respectively; each image encoder is a ViT network model composed of multiple layers of Transformer encoding layers; for each mode of the face image, the image encoder first converts the mode corresponding image I∈R H×W×C , where H, W, and C represent the height, width, and number of channels of the image, respectively. The image is evenly divided into b image blocks, and the width and height of each image block are s; where And through a layer of two-dimensional convolution, the segmented image blocks are represented as a sequence E I ={e1 I ,e2 I ,…e b I}∈R b ×(k-1) ; Then in sequence E I Add a category tag to the header cls , and add the position code and input it into the image encoder; the last layer of the image encoder, the Transformer encoding layer, outputs the image features Among them, f I Indicates the category label e cls The corresponding output is regarded as the global feature of the image; represents the local features corresponding to the jth image block; the image features of the three modalities of RGB, IR, and Depth are recorded as

[0016] Preferably, the steps S3 and S4, calculating the classification probabilities of real faces and fake faces, refer to:

[0017]

[0018] in, Indicates the type The classification probability of r represents the real face type, s represents the fake face type; I∈{RGB,IR,Depth,fuse}; sim(·,·) represents the cosine similarity score, τ is the temperature coefficient; represents the i-th image feature of the i-th modality, Text features representing true face types, Text features indicating the type of fake face, Text features indicating real or fake face types.

[0019] Preferably, the multimodal information fusion module refers to a module that uses a multimodal splicing method or a multimodal fusion cross attention mechanism to fuse and splice the image features of the three modalities to obtain a fused image feature.

[0020] Preferably, the multimodal splicing method refers to:

[0021]

[0022] Among them, contact represents feature connection; conv represents convolution calculation.

[0023] Preferably, the multimodal fusion cross attention mechanism method refers to:

[0024] First, three parameters are used: W q , W k and W v The linear layers are used to extract features respectively and Where m∈(IR,Depth):

[0025]

[0026] in, is the classification token of RGB modality; is the image patch token of IR modality; It is the image block token of Depth mode;

[0027] Then, the softmax function is used to generate the attention matrix A to extract the attention features.

[0028] in, is the scaling factor, where D and h are the dimension of the feature vector and the number of heads in the multi-head attention, respectively; A is a one-dimensional vector, and n is the length of the one-dimensional vector A;

[0029] Attention Features RGB modality image features and attention features Stitching to get fused image features

[0030]

[0031] Among them, conv represents convolution calculation.

[0032] Preferably, the step S1 comprises: constructing a text prompt set of real face labels and fake face labels of the face image data set;

[0033] Align each face image sample of the face image dataset with the text prompt T of the text prompt set;

[0034] For each text prompt T, extract text features: first perform word segmentation and truncation / lengthening operations, and insert text start and end tags to unify the length of the text prompt to k words; then use the word embedding method to convert the text prompt from a word sequence to a vector sequence E T ={e1 T ,e2 T ,…e k T}∈R k×512 ; The vector sequence E T After adding the position encoding, it is fed into the text encoder of the CLIP network architecture, and the vector sequence E is learned through the self-attention mechanism. T The semantic information contained in the text; the last encoding layer of the text encoder outputs text features r represents the real face type, s represents the fake face type; text features Among them, f T The output corresponding to the text start mark is regarded as the global feature of the text; Represents the local text feature corresponding to the jth word.

[0035] A readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the multimodal face liveness detection method of image-text contrast learning.

[0036] A computer device comprises a processor and a memory for storing a program executable by the processor. When the processor executes the program stored in the memory, the multimodal face liveness detection method of image-text contrast learning is implemented.

[0037] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0038] In view of the problem that the current face liveness detection technology has insufficient generalization ability when dealing with unknown attack modes, multiple types of camera sensors and complex environmental changes, the present invention introduces the CLIP pre-training model, fully utilizes the cross-modal knowledge contained in the CLIP pre-training model in large-scale cross-modality, extracts the basic features of face images and text prompts, and improves the generalization ability of the face liveness detection model; and proposes a cross-modal fusion attention mechanism to obtain more effective attention features. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a flow chart of the multimodal face liveness detection method of the present invention based on image-text contrast learning;

[0040] Figure 2 It is a schematic diagram of calculating the cosine similarity between the three modal image features and the text features of the present invention;

[0041] Figure 3 It is a schematic diagram of calculating the cosine similarity between the three modal image features of the present invention and the text features after being fused in a multimodal splicing manner;

[0042] Figure 4 It is a schematic diagram of calculating the cosine similarity between the three modal image features of the present invention and the text features after being fused by adopting the multimodal fusion cross attention mechanism;

[0043] Figure 5 is a schematic diagram of the cross attention mechanism of the present invention;

[0044] Figure 6 It is a schematic diagram of the existing method of achieving true and false face classification by weighting the image features of the three modalities through the MLP output scores. DETAILED DESCRIPTION

[0045] The present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0046] Embodiment 1

[0047] This embodiment of a multimodal face liveness detection method based on image-text contrast learning adopts CLIP network architecture to realize face liveness detection; the CLIP network architecture refers to: fine-tuning training based on CLIP pre-training model to obtain CLIP network architecture for face liveness detection. The CLIP pre-training model has demonstrated its powerful semantic information mining ability, zero-sample ability and modality alignment ability in multiple downstream tasks. The face liveness detection task is also a picture-text matching task. The present invention adopts CLIP network architecture to extract basic features of face images and retrieve text, so as to make full use of the pre-training knowledge contained in the model. The CLIP network architecture includes a text encoder for feature extraction of text, three image encoders for extracting image features of three modalities of face image RGB, IR and Depth, and a multimodal information fusion module for fusing and splicing image features of the three modalities.

[0048] Multimodal face liveness detection method based on image-text contrast learning, such as Figure 1 As shown, the following steps are included:

[0049] Step S1: Expand the real face labels and fake face labels of the face image dataset into a text prompt set; and send it to the text encoder to obtain the text features of the real face type and text features of fake face types .

[0050] Since the face anti-spoofing task lacks the textual prompts required by the CLIP network architecture, it is necessary to construct a textual prompt set. According to the real face labels and fake face labels of the face image dataset, all textual prompts are constructed as “aphoto of a <class>face", where <class>Real and Spoof represent real faces and fake faces, respectively. Consider that each class can have multiple different text prompts, and calculate text representation features for each text prompt. The average features of these representations give the overall features of the context in the embedding space. Aligning images with a large set of natural language text prompts enables the model to learn class-specific clues; examples are shown in Table 1.

[0051] Table 1 Examples of text prompt sets extended from real face labels and fake face labels

[0052]

[0053] Align each face image sample of the face image dataset with the text prompt T of the text prompt set;

[0054] For each text prompt T, extract text features: first perform word segmentation and truncation / lengthening operations, and insert text start and end markers to unify the length of the text prompt to k words, for example, k = 77; then use the word embedding method to convert the text prompt from a word sequence into a vector sequence E T ={e1 T ,e2 T ,…e k T }∈R k×512 ; The vector sequence E T After adding the position encoding, it is fed into the text encoder of the CLIP network architecture, and the vector sequence E is learned through the self-attention mechanism. T The semantic information contained in the text; the last encoding layer of the text encoder outputs text features r represents the real face type, s represents the fake face type; text features Among them, f T The output corresponding to the text start mark is regarded as the global feature of the text; Represents the local text feature corresponding to the jth word.

[0055] Step S2: Input the three modes of face image RGB, IR, and Depth into the image encoder corresponding to the CLIP network architecture respectively to extract the image features of the three modes.

[0056] Specifically, the three modes of face image RGB, IR, and Depth are input into the image encoder corresponding to the CLIP network architecture respectively; each image encoder is a ViT network model composed of multiple layers of Transformer encoding layers; for each mode of the face image, the image encoder first converts the mode corresponding image I∈R H×W×C , where H, W, and C represent the height, width, and number of channels of the image, respectively. The image is evenly divided into b image blocks, and the width and height of each image block are s; where And through a layer of two-dimensional convolution, the segmented image blocks are represented as a sequence E I ={e1 I ,e2 I ,…e b I }∈R b×(k-1) ; Then in sequence E I Add a category tag to the header cls , and add the position code and input it into the image encoder; the last layer of the image encoder, the Transformer encoding layer, outputs the image features Among them, f I Indicates the category label e cls The corresponding output is regarded as the global feature of the image; Represents the local features corresponding to the jth image block; the image features of the three modalities of RGB, IR, and Depth are recorded as

[0057] Step S3: The image features of the three modalities Respectively with text features The cosine similarity of the real face and the fake face is calculated to obtain the classification probability of the real face and the fake face.

[0058] Step S4: If the classification probability confidence of the real face or fake face in any modality is high, the loss contribution of the samples in the other two modalities can be reduced, and the type corresponding to the high confidence classification probability of the modality is used as the face liveness detection result, such as Figure 2 As shown in Figure 2, high confidence means that the classification probability is ≥ the set threshold. For example, the set threshold is 0.9.

[0059] If the classification probability confidence of the true face and the fake face of the three modalities is not high, the image features of the three modalities are input into the multimodal information fusion module to obtain the fused image features. Fusion image features The cosine similarity of the real face and the fake face is calculated with the text features to obtain the classification probability of the real face and the fake face, and the type with higher classification probability is taken as the face liveness detection result.

[0060] Step S3 and step S4, calculating the classification probability of real face and fake face, means:

[0061]

[0062] in, Indicates the type The classification probability of r represents the real face type, s represents the fake face type; I∈{RGB,IR,Depth,fuse}; sim(·,·) represents the cosine similarity score, τ is the temperature coefficient; represents the i-th image feature of the i-th modality, Text features representing true face types, Text features indicating the type of fake face, Text features indicating real or fake face types.

[0063] The multimodal information fusion module refers to the use of multimodal splicing or multimodal fusion cross-attention mechanism to fuse and splice the image features of the three modalities to obtain fused image features.

[0064] Among them, the multimodal splicing methods, such as Figure 3 As shown, it means:

[0065]

[0066] Among them, contact represents feature connection; conv represents convolution calculation.

[0067] Multimodal fusion cross attention mechanism, such as Figure 4 and Figure 5 shown.

[0068] How to effectively fuse multimodal information to generate more powerful feature representations is the primary task of the multimodal fusion module (Multi-modal Fusion-Attention, MFA). Since the classification (CLS) token is essentially a proxy for an image, it summarizes all the image patch tokens used for classification. Inspired by cross attention, the present invention adopts a simple and effective multimodal fusion strategy, which uses the classification token of the RGB modality sequence as a query to exchange information with the image patch tokens of other modality sequences. MFA is similar to self-attention, but with the following differences: in terms of input form, unlike self-attention which takes a single modality sample as input, the input of MFA is a pair of samples of different modalities, where the query matrix is ​​obtained from the current modality sequence, and the key-value pair matrix comes from other modality sequences. In terms of fusion strategy, the purpose of self-attention is to mine the relationship between all tokens (including classification tokens and image patch tokens) in the input sample, while MFA fuses the other modality information of the current modality sequence by calculating the dependency between the classification token (from the current modality) and the image patch token (from another modality).

[0069] Specifically, first, three parameters are used: W q , W k and W v The linear layers are used to extract features respectively and Where m∈(IR,Depth):

[0070]

[0071] in, is the classification token of RGB modality; is the image patch token of IR modality; It is the image block token of Depth mode;

[0072] Then, the softmax function is used to generate the attention matrix A to extract the attention features.

[0073] in, is the scaling factor, where D and h are the dimension of the feature vector and the number of heads in the multi-head attention, respectively; A is a one-dimensional vector, and n is the length of the one-dimensional vector A;

[0074] Attention Features RGB modality image features and attention features Stitching to get fused image features

[0075]

[0076] Among them, conv represents convolution calculation.

[0077] In the fine-tuning training stage, after calculating the cosine similarity, the cosine similarity is regarded as the category logits and used for cross entropy loss calculation to obtain the category loss L cls .

[0078] In order to verify the beneficial effects of the method of the present invention, experiments were carried out on four public datasets: CASIA-CeFA (abbreviated as C), PADISI-Face (abbreviated as P), CASIA-SURF (abbreviated as S) and WMCA (abbreviated as W) by comparing the method of the present invention with the existing methods.

[0079] The CASIA-SURF dataset has a total of 1,000 volunteers participating in the recording, totaling 21,000 multimodal videos. The data comes from multiple channels (visible light, depth map, and near infrared), using flat printing attacks or curled printing attacks, and randomly removing areas such as eyes, noses, and mouths. The upgraded version of this dataset, CASIA-SURF CeFA, was released in 2020, adding cross-racial volunteer identities to test the generalization of the algorithm across races. The Multi-Channel Presentation Attack (WMCA) dataset contains 1,941 short video records of real people and presentation disguise attacks from 72 different identities. The data is recorded from several channels, including color, depth, infrared, and thermal imaging. The PADISI-Face dataset has a total of 360 volunteers participating in the recording, totaling 1,105 real-person videos, 924 presentation attack videos, and up to 37 types of attacks.

[0080] Existing methods are referred to as Baselines: The representations generated by the CLIP model have shown impressive plug-and-play performance in many downstream vision applications based on natural images, such as classification, object detection, and segmentation. However, these features cannot be directly used for face liveness detection tasks that require identifying subtle changes between similar face images. Therefore, we first fine-tune the visual backbone only for the face liveness detection task, and this method baseline is referred to as Baseline. In this method, a pre-trained CLIP model is adopted, and only its image encoder is used, and the text encoder is discarded. This provides us with a simple visual transformer (ViT) initialized with language-image pre-trained weights. Given a balanced batch of images from N source domains, the image encoder is used from the last transformer block (v K ) to extract the class label (c K ), which is then passed to a multi-layer perceptron (MLP) classification head to determine whether the input image is fake or real. The image encoder and MLP head use a standard cross entropy loss L ce The final classification values ​​of the three modalities are weighted by scores to obtain the final score value, and then the face liveness detection result is obtained, such as Figure 6 shown.

[0081] The experimental comparison results of the method of the present invention and the existing methods on four public data sets CASIA-CeFA (abbreviated as C), PADISI-Face (abbreviated as P), CASIA-SURF (abbreviated as S) and WMCA (abbreviated as W) are shown in Table 2; wherein, CPS->W represents the training set is C, P, S, and the test set is W; CPW->S represents the training set is C, P, W, and the test set is S; CSW->P represents the training set is C, S, W, and the test set is P; PSW->C represents the training set is P, S, W, and the test set is C.

[0082] Table 2 Comparison of experiments on adding attention fusion mechanism (unit: %)

[0083]

[0084] As can be seen from Table 2, the method of fusion of single modality + multimodal fusion cross-attention mechanism was tested in 4 public datasets through the leave-one-out method (select 3 datasets as training sets and the other one as test set to simulate cross-dataset testing), and the best HTER (half error rate) was minimized and AUC (accuracy) was highest, that is, the test performance was the best.

[0085] Embodiment 2

[0086] This embodiment provides a readable storage medium, wherein the readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the multimodal face liveness detection method based on image-text contrast learning described in Embodiment 1.

[0087] Embodiment 3

[0088] The present embodiment provides a computer device, including a processor and a memory for storing a program executable by the processor. When the processor executes the program stored in the memory, the multimodal face liveness detection method based on image-text contrast learning described in the first embodiment is implemented.

[0089] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.< / class> < / class>

Claims

1. A multimodal face liveness detection method based on image-text contrast learning, characterized by: A CLIP network architecture is used to realize face liveness detection; the CLIP network architecture refers to: fine-tuning training based on a CLIP pre-trained model to obtain a CLIP network architecture for face liveness detection; the CLIP network architecture includes a text encoder for extracting features from text, three image encoders for extracting image features from three modalities of RGB, IR, and Depth of a face image, and a multimodal information fusion module for fusing and splicing the image features of the three modalities; The multimodal face liveness detection method based on image-text contrast learning includes the following steps: Step S1: Expand the real face labels and fake face labels of the face image dataset into a text prompt set and send it to the text encoder to obtain the text features of the real face type. and text features of fake face types Step S2: Input the three modes of face image RGB, IR, and Depth into the image encoder corresponding to the CLIP network architecture respectively to extract the image features of the three modes. Step S3: The image features of the three modalities Respectively with text features Calculate the cosine similarity of the real face and the fake face to get the classification probability of the real face and the fake face; Step S4: If the classification probability confidence of the real face or the fake face of any modality is high, the type corresponding to the high confidence classification probability of the modality is used as the face liveness detection result; If the classification probability confidence of the true face and the fake face of the three modalities is not high, the image features of the three modalities are input into the multimodal information fusion module to obtain the fused image features. Fusion image features The cosine similarity of the real face and the fake face is calculated with the text features to obtain the classification probability of the real face and the fake face, and the type with higher classification probability is taken as the face liveness detection result.

2. The multimodal face liveness detection method based on image-text contrast learning according to claim 1 is characterized by: The step S2 refers to: inputting the three modes of the face image RGB, IR, and Depth into the image encoder corresponding to the CLIP network architecture respectively; each image encoder is a ViT network model composed of multiple layers of Transformer encoding layers; for each mode of the face image, the image encoder first converts the mode corresponding image I∈R H×W×C , where H, W, and C represent the height, width, and number of channels of the image, respectively. The image is evenly divided into b image blocks, and the width and height of each image block are s; where And through a layer of two-dimensional convolution, the segmented image blocks are represented as a sequence E I ={e1 I ,e2 I ,…e b I }∈R b×(k-1) ; Then in sequence E I Add a category tag to the header cls , and add the position code and input it into the image encoder; the last layer of the image encoder, the Transformer encoding layer, outputs the image features Among them, f I Indicates the category label e cls The corresponding output is regarded as the global feature of the image; j=1,2,…,b, represents the local features corresponding to the jth image block; the image features of the three modalities of RGB, IR, and Depth are recorded as 3. The multimodal face liveness detection method based on image-text contrast learning according to claim 1 is characterized in that: The steps S3 and S4, calculating the classification probabilities of real faces and fake faces, refer to: in, Indicates the type The classification probability of r represents the real face type, s represents the fake face type; I∈{RGB,IR,Depth,fuse}; sim(·,·) represents the cosine similarity score, τ is the temperature coefficient; represents the i-th image feature of the i-th modality, Text features representing true face types, Text features indicating the type of fake face, Text features indicating real or fake face types.

4. The multimodal face liveness detection method based on image-text contrast learning according to claim 1 is characterized in that: The multimodal information fusion module refers to a module that uses a multimodal splicing method or a multimodal fusion cross attention mechanism to fuse and splice the image features of the three modalities to obtain a fused image feature.

5. The multimodal face liveness detection method based on image-text contrast learning according to claim 4 is characterized in that: The multimodal splicing method refers to: Among them, contact represents feature connection; conv represents convolution calculation.

6. The multimodal face liveness detection method based on image-text contrast learning according to claim 4 is characterized by: The multimodal fusion cross attention mechanism method refers to: First, three parameters are used: W q , W k and W v The linear layers are used to extract features respectively and Where m∈(IR,Depth): in, is the classification token of RGB modality; is the image patch token of IR modality; It is the image block token of Depth mode; Then, the softmax function is used to generate the attention matrix A to extract the attention features. in, is the scaling factor, where D and h are the dimension of the feature vector and the number of heads in the multi-head attention, respectively; A is a one-dimensional vector, and n is the length of the one-dimensional vector A; Attention Features RGB modality image features and attention features Stitching to get fused image features Among them, conv represents convolution calculation.

7. The multimodal face liveness detection method based on image-text contrast learning according to claim 1, characterized in that: The step S1 comprises: constructing a text prompt set of real face labels and fake face labels of the face image data set; Align each face image sample of the face image dataset with the text prompt T of the text prompt set; For each text prompt T, extract text features: first perform word segmentation and truncation / lengthening operations, and insert text start and end tags to unify the length of the text prompt to k words; then use the word embedding method to convert the text prompt from a word sequence to a vector sequence E T ={e1 T ,e2 T ,…e k T }∈R k×512 ; The vector sequence E T After adding the positional encoding, it is fed into the text encoder of the CLIP network architecture and the vector sequence E is learned through the self-attention mechanism. T The semantic information contained in the text; the last encoding layer of the text encoder outputs text features r represents the real face type, s represents the fake face type; text features Among them, f T The output corresponding to the text start mark is regarded as the global feature of the text; j=1,2,…,k, represents the local text feature corresponding to the jth word.

8. A readable storage medium, characterized in that: The storage medium stores a computer program, which, when executed by a processor, enables the processor to execute the multimodal face liveness detection method based on image-text contrast learning according to any one of claims 1 to 7.

9. A computer device comprising a processor and a memory for storing a program executable by the processor, characterized in that: When the processor executes the program stored in the memory, the multimodal face liveness detection method based on image-text contrast learning described in any one of claims 1 to 7 is implemented.

Images