A liveness detection method and system based on language-guided image feature extraction

Abstract

This invention discloses a liveness detection method and system based on language-guided image feature extraction. The method includes the following steps: preprocessing a video dataset for face fraud detection; constructing an image encoder and a text encoder; constructing an image-to-image projector and a text-to-image projector; constructing a contrastive learning module to shorten the distance between corresponding projected features; and a text design module to generate multiple sets of text prompts. A text filtering module is constructed, and a frozen text encoder module freezes the pre-trained weights of the text encoder. The text prompts are calculated using cosine similarity and binary cross-entropy to determine the most accurate match to the face. A classifier judges the authenticity of the face based on the text prompts. Supervised training is performed on the outputs of the contrastive learning module, the image-text feature alignment module, and the classifier. Based on the trained model, a prediction of the face's authenticity is obtained. This invention achieves higher accuracy and better generalization performance in determining face authenticity.

Description

Technical Field

[0001] This invention relates to the field of face liveness detection technology, specifically to a liveness detection method and system based on language-guided image feature extraction. Background Technology

[0002] Compared to fingerprint and iris recognition, facial features are easier to obtain and are widely used for identity verification. With the development of deep learning, facial recognition technology is becoming increasingly mature, but it also faces new challenges such as facial spoofing. Facial anti-spoofing has become crucial, primarily addressing attacks using photos, masks, video playback, and paper covers to ensure system security. Liveness detection, as a pre-emptive security system for facial recognition, treats facial authenticity as a binary or multi-class classification task. Currently, facial anti-spoofing detection methods can generally be divided into two categories: traditional machine learning methods utilizing manual feature extraction and methods relying on deep learning.

[0003] Traditional machine learning methods utilize facial texture, dynamics, 3D structure, and physiological signals. Early algorithms relied on manually designed features and used support vector machines for binary classification, such as using features like Local Binary Pattern (LBP) and Difference of Gaussians (DoG). However, traditional methods over-rely on feature design, resulting in small differences between fraudulent and real faces in representation attacks, leading to poor generalization.

[0004] In contrast, deep learning algorithms with superior feature representations have become the mainstream face anti-fraud algorithms. These strategies include domain adaptation, domain generalization, anomaly detection, and multimodal fusion strategies. However, due to differences in image capture devices, attack methods, lighting conditions, and data resolution, there is a huge domain gap between the source and target distributions, which hinders the effective feature extraction of fraudulent features and thus limits the generalization performance of the model. In addition, the FAS benchmark dataset usually has limited training data, which leads to the model overfitting to the source domain features.

[0005] Therefore, the most important challenge currently facing face fraud detection systems is how to maintain excellent detection performance when facing unknown attacks that are different from the source domain. Summary of the Invention

[0006] To overcome the shortcomings and deficiencies of existing technologies, this invention provides a liveness detection method and system based on language-guided image feature extraction. This invention utilizes a pre-trained image encoder and text encoder on a large-scale dataset, effectively overcoming the problem of insufficient feature extraction in existing methods. It also introduces a contrastive learning module and an image-text alignment module, enabling the network model to learn general fraud clues. Text design and filtering strategies enable the pre-trained model to better adapt to the field of face anti-spoofing detection. A frozen text encoder module is constructed, which ensures robust generalization performance even when trained on a small dataset.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] This invention provides a liveness detection method based on language-guided image feature extraction, comprising the following steps:

[0009] A video dataset preprocessing module is constructed to preprocess the acquired video dataset for face fraud detection, extract face images, and construct a text file containing face images and corresponding true / false labels;

[0010] Construct an image encoder and a text encoder. The image encoder extracts image features from face images, and the text encoder extracts text features from text prompts.

[0011] Construct an image-image projector and a text-image projector. The image-image projector projects different image features to the same feature space, and the text-image projector projects image and text features to the same feature space.

[0012] A contrastive learning module is constructed. Each face image is augmented to obtain two sub-views. The corresponding feature representations are obtained through an image encoder and mapped to another feature space based on an image-to-image projector. The contrastive learning module calculates the similarity and differences of the two image projection features based on the contrastive learning loss function.

[0013] An image-text feature alignment module is constructed. Based on a text-image projector, image features and text features are mapped to obtain corresponding projected features. Based on alignment loss, the distance between the corresponding projected features is shortened to achieve image and text feature alignment.

[0014] The text design module is constructed to generate multiple sets of text prompts based on a large language text model and preset constraints;

[0015] A text filtering module is built to filter multiple sets of text prompts;

[0016] Construct a frozen text encoder module to freeze the pre-trained weights of the text encoder;

[0017] A classifier is constructed, which calculates the text prompt word that best matches the face by calculating cosine similarity and binary cross-entropy, and judges the authenticity of the face based on the truthfulness of the text prompt word description.

[0018] By supervising the training of the contrastive learning module, the image-text feature alignment module, and the three outputs of the classifier, and inputting the face image into the trained model, the real and fake face prediction results are obtained.

[0019] As a preferred technical solution, the image encoder includes a patch embedding layer, a position embedding layer, a multi-layer Transformer encoder, and a linear projection layer;

[0020] The patch embedding layer segments the face image into multiple patches and flattens them into a one-dimensional patch vector.

[0021] The location embedding layer adds location information to each one-dimensional patch vector, preserving the spatial structure and order of the image, as shown below:

[0022] z0 = [x1E; x2E; ...x N E]+E pos

[0023] Where z0 represents the initial embedding features, x i Let E be the one-dimensional patch vector flattened from the i-th patch, and let E be the linear projection matrix. pos For the position embedding matrix;

[0024] The Transformer encoder extracts global features from the initial embedded features through a self-attention mechanism;

[0025] Each Transformer encoder sublayer is followed by a Layer Normalization layer, represented as follows:

[0026]

[0027] Where x represents the input feature vector, μ represents the mean of each feature in the batch, σ represents the standard deviation of each feature in the batch, γ and β represent the affine parameters to be learned, and ∈ is a constant;

[0028] The linear projection layer maps the output features of the Transformer encoder to the target dimension, thus obtaining an image feature representation.

[0029] As a preferred technical solution, the text encoder includes a word embedding layer, a multi-layer Transformer encoder, and a linear projection layer;

[0030] The word embedding layer maps each word of the input text to a high-dimensional feature space, generating initial features, represented as follows:

[0031] e i =W embed x i

[0032] Where, x i Let W be the one-hot vector of the i-th word. embed For embedding matrix;

[0033] The Transformer encoder extracts global features from the initial features through a self-attention mechanism;

[0034] Each Transformer encoder sublayer is followed by a Layer Normalization layer, represented as follows:

[0035]

[0036] Where x represents the input feature vector, μ represents the mean of each feature in the batch, σ represents the standard deviation of each feature in the batch, γ and β represent the affine parameters to be learned, and ∈ is a constant;

[0037] The linear projection layer maps the output features of the Transformer encoder to the target dimension, thus obtaining a text feature representation.

[0038] As a preferred technical solution, the image-image projector and the text-image projector have the same network structure, including three fully connected layers. Each layer contains a weight matrix and a bias vector, and a nonlinear transformation is performed using the ReLU activation function, which is expressed as:

[0039] ReLU(x) = max(0,x)

[0040] Where x is the input feature vector, representing either the image feature vector or the text feature vector;

[0041] The first fully connected layer receives the input feature vector and transforms it using the ReLU activation function, setting negative values ​​to zero.

[0042] The transformed output is used as the input to the second fully connected layer and processed by the ReLU activation function;

[0043] The features are then mapped to the target output dimension through a third fully connected layer to obtain the final output features.

[0044] As a preferred technical solution, each face image undergoes data augmentation to obtain two sub-views, and the corresponding feature representation is obtained through an image encoder, specifically as follows:

[0045] v1 = E image(View1)

[0046] v2 = E image (View2)

[0047] Where View1 and View2 represent two subviews, and v1 and v2 represent the feature representations corresponding to the two subviews, E image Indicates an image encoder;

[0048] Based on the image-image projector mapping to another feature space, it can be represented as:

[0049] v′1=Proj1(v1)

[0050] v′2=Proj2(v2)

[0051] Where Proj1 represents the image-to-image projector, and v′1 and v′2 represent the projection features corresponding to two image feature representations;

[0052] The contrastive learning module calculates the similarity and differences of the projected features of two images based on the contrastive learning loss function, which is expressed as:

[0053]

[0054] Where sim represents the cosine similarity between two feature vectors, τ is the temperature parameter, and B is the batch size.

[0055] As a preferred technical solution, the image features and text features are mapped to obtain the corresponding projected features based on the text-image projector, as follows:

[0056] x′=Proj2(x)

[0057] x = E image (image)

[0058] t′=Proj2(t)

[0059] t = E text (text)

[0060] Where image represents image features, text represents text features, Proj2 represents the text-image projector, and E image E represents the image encoder. text This represents a text encoder, where x′ and t′ represent the corresponding projected features obtained by mapping image features and text features;

[0061] The distance between the corresponding projected features is shortened based on the alignment loss, specifically expressed as follows:

[0062]

[0063] Among them, L inter This represents the L2 norm distance between two feature vectors, where B is the batch size.

[0064] As a preferred technical solution, multiple sets of text prompt words are generated based on a large language text model and preset constraints, specifically including:

[0065] Three constraints were imposed on the text for real and fake faces, including: prohibiting the use of obscure words, prohibiting the use of clauses, and prohibiting descriptions of human features, resolution, environmental background, and lighting.

[0066] To detect fraudulent faces, attack type alerts are provided, including printing attacks, replay attacks, 3D mask attacks, and adversarial sample attacks.

[0067] As a preferred technical solution, a text filtering module is constructed to filter multiple sets of text prompt words, specifically including:

[0068] Calculate the cosine similarity between the image and each text prompt word, and use the text with the highest cosine similarity as the unique prompt word corresponding to the image;

[0069] Calculate the matching frequency of each text and each image, and retain a set number of texts based on the matching frequency;

[0070] Retain the text prompts describing the attack type.

[0071] As a preferred technical solution, the text prompt words that best match the face are calculated using cosine similarity and binary cross-entropy. The authenticity of the face is determined based on the truthfulness of the text prompt words. Specifically, this is expressed as follows:

[0072] Each type of text is encoded using a text encoder to obtain the embedding representation of each text, and these representations are stacked to form a new tensor.

[0073] The cropped face image is encoded using an image encoder to obtain the image features of the image;

[0074] Normalize the image features and text features, calculate the cosine similarity between the image features and text features, select the most similar text and determine its category. If the text describes a real face, the face image is judged as a real face; if the text describes a fake face, the face image is judged as a fake face.

[0075] Supervision is performed using the true and false labels of the samples. The specific loss function formula is as follows:

[0076]

[0077] Among them, y iLet y' represent the true label of the i-th sample, where 0 indicates the face image is fraudulent and 1 indicates the face image is real. i The predicted probability of the i-th sample represents the probability that the model predicts the sample to be true.

[0078] This invention also provides a liveness detection system based on language-guided image feature extraction, used to implement the above-mentioned liveness detection method based on language-guided image feature extraction. The system includes: a video dataset preprocessing module, an image encoder, a text encoder, an image-to-image projector, a text-to-image projector, an image projection feature construction module, a contrast learning module, an image-to-text feature alignment module, a text design module, a text filtering module, a frozen text encoder module, a classifier, a supervised training module, and a prediction module.

[0079] The video dataset preprocessing module is used to preprocess the acquired video dataset for face fraud detection, extract face images, and construct a text file containing face images and corresponding true / false labels;

[0080] The image encoder is used to extract image features from human face images;

[0081] The text encoding is used to extract the text features of text prompt words;

[0082] The image-to-image projector is used to project different image features onto the same feature space;

[0083] The text-image projector is used to project image and text features onto the same feature space;

[0084] The image projection feature construction module is used to perform data augmentation on each face image to obtain two sub-views, obtain the corresponding feature representations through an image encoder, and then map the image to another feature space based on the image-image projector mapping to obtain the corresponding image projection features.

[0085] The contrastive learning module is used to calculate the similarity and differences of the projection features of two images based on the contrastive learning loss function;

[0086] The image-text feature alignment module is used to map image features and text features to obtain corresponding projected features based on a text-image projector, and shorten the distance of the corresponding projected features based on alignment loss to achieve image and text feature alignment.

[0087] The text design module generates multiple sets of text prompts based on a large language text model and preset constraints;

[0088] The text filtering module is used to filter multiple sets of text prompts;

[0089] The frozen text encoder module is used to freeze the pre-trained weights of the text encoder;

[0090] The classifier is used to calculate the text prompt word that best matches the face by calculating cosine similarity and binary cross-entropy, and to determine the authenticity of the face based on the authenticity of the text prompt word description.

[0091] The supervised training module is used to supervise the training of the three outputs of the contrastive learning module, the image-text feature alignment module, and the classifier.

[0092] The prediction module is used to input face images into the trained model to obtain the prediction results of whether the face is real or fake.

[0093] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0094] (1) The present invention constructs an image-text feature alignment module. By projecting the image features and text features extracted by the image encoder and text encoder, the features are brought closer in the new feature space, thereby reducing the distance between features of different modalities and improving their alignment stability in large text language models. This not only reduces the differences between feature modalities but also improves the alignment accuracy of the model, ensuring semantic consistency between text and images. In addition, through this feature alignment method, the model can more effectively handle cross-modal tasks, thereby greatly improving its generalization ability in the face of invisible attacks.

[0095] (2) This invention constructs a text design module and a text filtering module. Specifically, it generates text that is easier for the text encoder to understand through GPT-4, thereby enhancing the text feature extraction capability. It provides text prompts with clear attack methods, enabling the model to learn different attack category information during training. By limiting the length, complexity, and obscure words of the text prompts, it further enhances the feature extraction capability of the text encoder. The text filtering strategy increases the ambiguity of the text by selecting texts with a large weight in the model optimization during training and discarding texts with a small weight. Too many texts can lead to more accurate text-image matching, resulting in overfitting within the library. Text filtering can increase the ambiguity of the text prompts to a certain extent, thereby improving the generalization capability.

[0096] (3) The present invention constructs a frozen text encoder module. By freezing the pre-trained weights, the text encoder will not be optimized due to the preference of text prompt words. Only the image encoder, image-to-image projector and text-to-image projector will be updated. This forces the network model to focus the training optimization on the image encoder, making fuller use of the existing image dataset to learn more general face fraud clues. At the same time, freezing the pre-trained weights reduces the parameter optimization time during the training process, thereby reducing the time complexity of the model while keeping the number of model parameters unchanged.

[0097] (4) The network proposed in this invention has excellent generalization performance when facing invisible attacks. Compared with traditional machine learning methods and other deep learning algorithms that do not use large text and image models, it has higher accuracy in judging the authenticity of faces, shorter training time, and better generalization performance. It only uses text and image dual-modality, is suitable for deployment on mobile phones, embedded terminals and other devices, and has high practical value. Attached Figure Description

[0098] Figure 1 This is a flowchart illustrating the liveness detection method based on language-guided image feature extraction according to the present invention.

[0099] Figure 2 This is a schematic diagram of the overall implementation framework of the liveness detection method based on language-guided image feature extraction of the present invention;

[0100] Figure 3 This is a schematic diagram of the network structure of the image encoder of the present invention;

[0101] Figure 4 This is a schematic diagram of the network structure of the text encoder of the present invention;

[0102] Figure 5 This is a schematic diagram of the network structure of the image-image projector and text-image projector of the present invention;

[0103] Figure 6 This is a schematic diagram illustrating the process by which the comparative learning module of this invention learns the similarities and differences between two subviews.

[0104] Figure 7 This is a schematic diagram of the network structure of the image-text feature alignment module of the present invention;

[0105] Figure 8 This is a text illustration designed for the text design module of the present invention;

[0106] Figure 9 This is a visual diagram of the experimental results of the present invention. Detailed Implementation

[0107] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0108] Example 1

[0109] This embodiment uses four publicly available liveness detection video datasets—Replay-Attack, CASIA-MFSD, MSU_MFSD, and OULU—for training and testing as examples to detail the implementation process. The Replay-Attack database, released by the IDIAP Institute in Switzerland in 2012, contains a total of 1200 videos taken under different lighting conditions with zero different participants. The dataset is divided into four categories: training set, validation set, test set, and registration set (videos not used), corresponding to 15, 15, and 20 participants respectively. The videos were recorded using the built-in webcam of a Macbook laptop and QuickTime software. The CASIA-MFSD dataset, released by the Institute of Automation, Chinese Academy of Sciences in 2012, contains a total of 600 face videos taken with three different devices. The video recording devices were a Sony NEX-5 (video resolution 1280×720), a new USB webcam (video resolution 640×480), and an older USB webcam (video resolution 480×640). The MSU-MFSD dataset, released in 2015 by the Pattern Recognition and Image Processing Lab at Michigan State University, contains 280 videos by 35 photographers, with a real-to-fake ratio of 1:3. The videos were filmed on a Nexus 5 (720×480 resolution) and a MacBook Air 13 (640×480 resolution). The OULU dataset, released in 2017 by the University of Oulu in Finland and Northwestern Polytechnical University in China, contains 4950 videos by 55 photographers, with a real-to-fake ratio of 1:4. The videos were filmed using the front-facing cameras of six mobile devices (Samsung Galaxy S6 edge, HTC Desire Eye, MEIZU X5, ASUS Zenfone Selfie, Sony XPERIA C5 Ultra Dual, and OPPO N3), with three different lighting conditions and background scenes.

[0110] This embodiment runs on a Linux Ubuntu system, implemented using the PyTorch deep learning framework, with an NVIDIA GeForce RTX 4090 GPU, Python version 3.8.10, and PyTorch version 2.12+cu121.

[0111] like Figure 1 , Figure 2 As shown, this embodiment provides a liveness detection method based on language-guided image feature extraction, including the following steps:

[0112] S1. Construct a video dataset preprocessing module, divide the public face fraud detection video dataset, perform frame extraction on the video dataset, detect face regions and extract face images, and construct a text file containing face images and corresponding true / false labels.

[0113] In this embodiment, ffmpeg is used to process the video dataset into frames, and the MTCNN algorithm is used to detect and crop faces. Finally, the Lanczos interpolation algorithm is used to unify the resolution of the face image, making it H×W×C (256×256×3 in this embodiment), which represent the height, width and number of channels of the cropped face image, respectively.

[0114] S2. Construct an image encoder;

[0115] like Figure 3 As shown, the image encoder is used to extract image features. The network structure is mainly built based on the ViT architecture, including a 16×16 patch embedding layer, a position embedding layer, a multi-layer Transformer encoder, and a linear projection layer.

[0116] The patch embedding layer divides the image into 16×16 pixel patches and flattens them into one-dimensional vectors for easier subsequent processing. The position embedding layer adds position information to each patch vector to preserve the spatial structure and order of the image.

[0117] The multi-layer Transformer encoder uses a self-attention mechanism and a feedforward neural network to deeply process patch features, capturing the relationships and contextual information between patches. The linear projection layer maps the output features of the Transformer encoder to the target dimension, providing the final feature representation for downstream tasks.

[0118] The specific process is as follows: The input image resolution is 224×224. First, the input image is divided into 16×16 patches using a patch embedding layer, resulting in 14×14 patches. Each patch is flattened into a vector and mapped to a 768-dimensional high-dimensional feature space through a linear projection layer, yielding initial features of size 14×14×768. Then, a position embedding is added to each patch vector to preserve spatial location information. The specific formula is as follows:

[0119] z0 = [x1E; x2E; ...x N E]+E pos

[0120] Where z0 represents the initial embedding features, x i Let N be the flattened vector of the i-th patch, N = 196, and E be the linear projection matrix. pos The initial features are then passed through a 12-layer Transformer encoder. Each layer includes a multi-head self-attention layer and a feedforward neural network layer. Each sub-layer is followed by a Layer Normalization layer. The Transformer encoder extracts global features of the image through a self-attention mechanism, resulting in a feature map of size 14×14×768. The formula for the self-attention mechanism is as follows:

[0121]

[0122] Where Q (Query), K (Key), and V (Value) are obtained from the input features through different linear transformations, d k It is the dimension of K.

[0123] The formula for Layer Normalization is as follows:

[0124]

[0125] Where x represents the input feature vector, μ represents the mean of each feature in the batch, σ represents the standard deviation of each feature in the batch, γ and β represent the affine parameters to be learned, and ∈ is a very small constant to prevent division by zero;

[0126] S3. Construct a text encoder;

[0127] In this embodiment, the text encoder is used to extract text features, such as... Figure 4 As shown, the network structure is mainly built based on the Transformer architecture, specifically including a word embedding layer, a multi-layer Transformer encoder, and a linear projection layer;

[0128] The specific process is as follows: The input text first passes through a word embedding layer, mapping each word to a high-dimensional feature space to generate initial features of size 197×768, where 197 is the context length and 768 is the embedding dimension. The formula for word embedding is as follows:

[0129] e i =W embed x i

[0130] Where, x i Let W be the one-hot vector of the i-th word. embedThe embedding matrix is ​​used. Initial features are processed by a 12-layer Transformer encoder, each layer containing a multi-head self-attention layer and a feedforward neural network layer. Each sub-layer is followed by a Layer Normalization layer. The Transformer encoder extracts global features of the text through a self-attention mechanism, generating a feature map of size N×D. The formula for the self-attention mechanism is as follows:

[0131]

[0132] Where Q (Query), K (Key), and V (Value) are obtained from the input features through different linear transformations, d k This is the dimension of K. Finally, the feature map is passed through a linear projection layer to obtain a text feature vector of size 768. The LayerNormalization layer formula is as follows:

[0133]

[0134] Where μ represents the mean of each feature in the batch, σ represents the standard deviation of each feature in the batch, γ and β represent the affine parameters to be learned, and ∈ is a very small constant to prevent division by zero;

[0135] S4. Construct image-image projectors and text-image projectors;

[0136] Image-image projectors project different image features onto the same feature space, while text-image projectors project different text and image features onto the same feature space for subsequent optimization operations, such as... Figure 5 As shown, the two projectors have the same structure, but during training, the weight matrix and bias vector are optimized according to different data to achieve different goals.

[0137] The specific process is as follows: Image or text features of dimension in_dim are input into a fully connected layer, mapping them to a feature space of dimension mlp_dim. The fully connected layer contains a weight matrix and a bias vector, which work together during the forward propagation to map the input features to the output features, as shown in the following formula:

[0138] y = W·x + b

[0139] Where W is the weight matrix, x is the input feature vector, b is the bias vector, and y is the output of the linear projection;

[0140] The feature distribution is standardized using a batch normalization layer, and a nonlinear transformation is introduced through the ReLU activation function to set negative values ​​to zero, as shown in the following formula:

[0141] ReLU(x) = max(0,x)

[0142] Where ReLU(x) represents the maximum value of 0 and x;

[0143] The features activated by the activation function are passed through a fully connected layer, maintaining the dimension of `mlp_dim`, followed by batch normalization and ReLU activation. Finally, the features are passed through a fully connected output layer, mapping them to the output dimension of `out_dim` to obtain the final output features. This layer-by-layer linear transformation and non-linear activation enables the model to learn complex feature representations.

[0144] S5. Construct a comparative learning module;

[0145] like Figure 6 As shown, the contrast learning module projects images through an image-to-image projector and then uses contrast learning to identify the similarities and differences between two subviews. By bringing together different data from the same image to enhance view features, it solves the problem of traditional methods being sensitive to environmental factors such as lighting.

[0146] The specific process is as follows: The images preprocessed by the data preprocessing module are then augmented with data, such as through random cropping, brightness and chroma changes, and noise addition. This results in two sub-views, View1 and View2, for each image. These two sub-views, View1 and View2, are then processed by an image encoder to obtain their feature representations, v1 and v2. The output feature dimension of the image encoder is 512, as shown in the following formula:

[0147] v1 = E image (View1), v2 = E image (View2)

[0148] Here, v1 and v2 have a dimension of B×512, where B is the batch size. The image-to-image projector Proj1 is used to map them to another feature space, resulting in projected features v′1 and v′2. The image-to-image projector reduces the feature dimension from 512 to 256, as shown in the following formula:

[0149] v′1=Proj1(v1), v′2=Proj1(v2)

[0150] Here, v′1 and v′2 have dimensions of B×256. The similarity and differences of the projected features of the two images are calculated using a contrastive learning loss function to supervise the optimization process of contrastive learning. The gradient of the loss function with respect to the model parameters is calculated using the backpropagation algorithm, and the model parameters are updated using an optimization algorithm to minimize the contrastive loss, thereby bringing the view features of the same image with different data augmentations closer together. The contrastive learning loss function L... simCLRThe specific formula is as follows:

[0151]

[0152] Where sim(v′1,v′2) represents the cosine similarity between two feature vectors, τ is a temperature parameter used to adjust the smoothness of the similarity distribution, and the sum in the denominator represents the sum of the similarities of all samples in the batch;

[0153] S6. Construct an image-text feature alignment module to better align images and text by narrowing the distance between image features and text features with the same label.

[0154] like Figure 7 As shown, the image-text feature alignment module projects text onto an image using a text-image projector, ensuring semantic consistency between the text and image by narrowing the distance between text and image features. Feature alignment is achieved by calculating the mean difference between image and text features, while preserving content information. This process does not require explicit image synthesis; it only needs to be performed at the feature level, making it simple to implement and computationally inexpensive.

[0155] The specific process is as follows: the input image is named "image" and the input text is named "text". These are then processed by the image encoder and text encoder mentioned in S2 and S3 respectively to obtain the image feature x and text feature t. The specific formulas are as follows:

[0156] x = E image (image), t = E text (text)

[0157] Where x and t have dimensions B×512, and B is the batch size. Using the text-image projector Proj2 from step S4, they are mapped to a new feature space to obtain projected features x′ and t′, while maintaining the feature dimensions. The specific formula is as follows:

[0158] x′=Proj2(x), t′=Proj2(t)

[0159] Here, x′ and t′ have dimensions of B×512, where B is the batch size. Mean squared error is used to narrow the distance between x′ and t′, and an alignment loss function L is designed. inter This is used to monitor the optimization process of alignment loss. inter The specific formula is as follows:

[0160]

[0161] L interThis represents the L2 norm distance between two feature vectors. It measures the similarity between image and text features by calculating the mean distance of each sample in the image and text feature spaces.

[0162] S7. Construct a text design module to design text prompts that are more conducive to the understanding of the text model through GPT-4 and related prompts and constraints;

[0163] In this embodiment, the text design module designs text prompts to address the problem in existing methods where text prompts are too generic and singular, causing the text encoder to fail to extract more valuable text features, resulting in poor alignment between text and image features and poor generalization performance. For example... Figure 8 As shown, the text design module designed two sets of texts describing faces, with 12 texts in each set, for a total of 24 texts;

[0164] The specific process is as follows: First, preprocess the face video dataset, performing face recognition and cropping. Then, arbitrarily select a portion of the preprocessed faces from each dataset to input a multimodal version of GPT-4, and use a large language text model to generate text prompts. Simultaneously, certain constraints are imposed, such as prohibiting the use of obscure words and prohibiting the generation of complex, incomprehensible sentences.

[0165] Specifically, three constraints were applied to the text containing real and fake faces:

[0166] 1. The use of obscure words is prohibited; only simple everyday words and specialized terms in the field of facial fraud detection should be used.

[0167] 2. Clauses are prohibited, and sentences must not exceed 20 words;

[0168] 3. Descriptions of human features, resolution, environmental background, and lighting are prohibited; only declarative descriptions of whether a face is real or fake are allowed.

[0169] For fraudulent faces, additional attack type hints are provided. Four sentences contain specific attack method descriptions: printing attack, replay attack, 3D mask attack, and adversarial sample attack. The remaining eight fraudulent sentences are generated according to the above constraints.

[0170] S8. Construct a text filtering module to further select text prompts that have a significant impact on model optimization by expanding and discarding text, so that the model can converge faster.

[0171] In this embodiment, the text filtering module reduces the pre-designed 24 texts to 12 texts (6 for real faces and 6 for fake faces). Increasing the number of texts makes the model more prone to overfitting. While the model performs well within the database, it performs poorly in cross-database experiments against invisible attacks. Therefore, a text filtering strategy is used to select texts that contribute more to model optimization and discard the remaining texts. The ambiguity of the text prompts is increased to improve the model's generalization ability.

[0172] Specifically, input text and images Figure 2 In the constructed network, the cosine similarity between the image and each text prompt is calculated, and the text with the highest cosine similarity is used as the unique prompt corresponding to the image. For the 12 text prompts describing real faces, the matching frequency of each text and each image is counted, and the top six texts with the highest matching frequency are retained, while the bottom six are discarded. For the 12 text prompts describing fraudulent faces, the matching frequency of each text and each image is counted. Four text prompts describing printing attacks, replay attacks, 3D mask attacks, and adversarial sample attacks are retained, and the two text prompts with the highest matching frequency are additionally retained, while the other text prompts are discarded. The text filtering module transforms a 24-class classification problem into a 12-class classification problem, while increasing the stability of the model, making it easier to align image features with text features, and making the model easier to classify.

[0173] S9. Construct a frozen text encoder module; by freezing, the text encoder retains its pre-trained generalization features, solving the problem of weakened generalization ability in traditional methods during training and optimization.

[0174] In this embodiment, the text encoder module is frozen, meaning the pre-trained weights of the text encoder are frozen, allowing the model to update only the weights of the image feature extractor and the image-to-image and text-to-image projectors. This enhances the model's generalization ability in face fraud detection by leveraging generalized features obtained from pre-training on a large-scale dataset. Specifically, the text encoder is frozen before model training, and the parameters that need to be updated are printed, ensuring that the text encoder transformer module is not present, but the image encoder transformer module is.

[0175] S10. Construct a classifier, calculate the text prompt word that best matches the face by calculating cosine similarity and binary cross-entropy, and then judge the authenticity of the face based on the truth or falsehood of the text prompt word description.

[0176] In this embodiment, the classifier outputs the probability value of whether a sample is real or fake. The classifier includes two fully connected layers, a ReLU activation function layer, and a Dropout layer;

[0177] Specifically, each type of text (6 deceptive texts and 6 real texts) is encoded using a text encoder to obtain an embedding representation for each text. The embedding representations of all 12 texts are stacked together to form a new tensor, `text_features`. The cropped face image is encoded using an image encoder to obtain the image features. The image and text features are normalized to unitize each vector, ensuring that the feature vectors are in one dimension. The cosine similarity between the image and text features is calculated, treating the model as a twelve-class classification problem. By selecting the most similar text and determining its category, if the text describes a real face, the face image is classified as a real face; if the text describes a fake face, the face image is classified as a fake face. Supervision is performed using the real / fake labels of the samples; the specific loss function formula is as follows:

[0178]

[0179] Among them, y i y represents the true label of the i-th sample, where 0 indicates the face image is fraudulent and 1 indicates the face image is real. i ′ represents the predicted probability of the i-th sample, indicating the probability that the model predicts that the sample is a positive class (true).

[0180] S11. Construct a network model training module. By supervising the three outputs of the contrastive learning module, the image-text feature alignment module, and the classifier, we obtain the contrastive learning loss, the alignment loss, and the classification loss, and then obtain the total loss function. The model training is supervised by the total loss function. At the end of the training, the optimal network model and the best weights are retained according to the benchmark index.

[0181] In this embodiment, the network model training module inputs a face image and text prompts. Figure 2 The constructed network updates its weight coefficients through constraints imposed by the total loss function, and the network model and optimal weights are saved after training is complete.

[0182] Specifically, firstly, pre-trained weights are loaded into the image encoder and text encoder modules, and then RGB face images and 12 text prompts are input into the network model for end-to-end training. The Adam optimizer is used for optimization, with an initial learning rate set to 1×10⁻⁶. -6 The weight decay is set to 1×10. -6 The batch size was set to 1, and 5000 iterations were performed for each protocol group. The total loss function L was minimized. total Update the weight parameters of the network model, and save the network model and the optimal weights based on the performance of the benchmark test results.

[0183] The total loss function L involved in this processtotal The specific formula is as follows:

[0184] L total =αL simCLR +βL inter +γL ce

[0185] Where L simCLR L represents the contrastive learning loss. inter L represents the image-text feature alignment loss. ce Representing the classifier loss, α, β, and γ are the weight coefficients of the three loss functions. In this embodiment, α, β, and γ are set to 1, 0.1, and 1, respectively.

[0186] S12. Test using the saved optimal model and weights;

[0187] In this embodiment, the optimal model and weights obtained from training are loaded to construct a test network. Images from the test dataset are input to test the network model. The performance of the network model is evaluated based on benchmark metrics, and visualization analysis, parameter count, and time complexity are explored to jointly investigate the practical possibility of the model's deployment.

[0188] The specific process is as follows: Frames are extracted from the test dataset videos, and faces are cropped. These are then input into the trained model with optimal weights. Feature extraction is performed on the facial images and text prompts. Cosine similarity matching between the images and features is conducted. Based on the prediction results, common benchmark metrics in the field of face fraud detection are calculated. Finally, the model is visualized and analyzed, and the number of parameters and time complexity are calculated. The benchmark metrics used include False Positive Rate (FPR), False Negative Rate (FNR), True Positive Rate (TPR), Half Total Error Rate (HTER), and Area Under Curve (AUC).

[0189] The calculation of false positive rate and false negative rate involves four indicators: TP, FP, FN, and TN. TP is recorded when the label is true and the prediction is true; FP is recorded when the label is false and the prediction is true; FN is recorded when the label is true and the prediction is false; and TN is recorded when the label is false and the prediction is false.

[0190] The false positive rate is calculated by identifying the percentage of faces labeled as fake but mistakenly identified as real faces among all faces labeled as fake. The specific calculation formula is as follows:

[0191]

[0192] The false negative rate is calculated by identifying the percentage of faces labeled as real but misidentified as fake among all faces labeled as real. The specific calculation formula is as follows:

[0193]

[0194] The true positive rate is calculated by determining the percentage of all faces labeled as real and also judged as real faces. The specific calculation formula is as follows:

[0195]

[0196] The semi-total error rate (SCR) is the average of the false positive and false negative rates. A lower SCR indicates lower false positive and false negative rates, signifying higher model accuracy. The specific calculation formula is as follows:

[0197]

[0198] The receiver operating characteristic (ROC) curve is plotted with the false positive rate on the x-axis and the true positive rate on the y-axis. The larger the area under the curve, the better the overall performance of the model.

[0199] This embodiment conducted cross-database experiments on four publicly available video databases: CASIA-MFSD, Replay-Attack, MSU-MFSD, and OULU. Specifically, three databases were selected as the training set, and another dataset was used for testing to verify the model's generalization ability. The four sets of experiments were configured according to the protocol: OCI database across M database, OMI database across C database, OCM database across I database, and ICM database across O database. The optimal experimental results are shown in Tables 1 and 2 below.

[0200] Table 1 Cross-database testing HTER

[0201]

[0202] Table 2 Cross-library test AUC

[0203]

[0204] This embodiment compares with the PatchNet, ViT, and DiVT-M methods. The PatchNet algorithm, published at CVPR 2022, proposes a simple framework for face fraud detection through refined patch recognition. The ViT algorithm, published at ECCV 2022, proposes an adaptive transformer to improve system robustness through few-shot cross-domain face fraud detection. DiVT-M, published at WACV 2023, proposes a domain-invariant visual transformer learning method to improve its generalization ability across different domains. These three top-tier conferences and journals are highly representative in the field of face fraud detection and have achieved excellent performance.

[0205] Analysis of Tables 1 and 2 shows that the present invention achieved the best half-total error rate (WTR) in the OMI library across C library, OCM library across I library, and ICM library across O library experiments, with WTRs of 1.33%, 2.85%, and 5.27%, respectively. It also achieved a competitive result in the OCI library across M library experiment, with a WTR of 4.52%. Correspondingly, the area under the receiver operating characteristic (AUC) curve achieved the best results in the OMI library across C library, OCM library across I library, and ICM library across O library experiments, with AUCs of 99.78%, 99.67%, and 98.76%, respectively. It also achieved a competitive result in the OCI library across M library experiment, with an AUC of 98.72%. Experiments demonstrate that the method of this embodiment possesses excellent generalization performance when facing invisible attacks.

[0206] like Figure 9 As shown, the visualization of the experimental detection errors is obtained. After each test, the misclassified face images are saved: fraudulent faces misclassified as real faces are marked with black boxes, and real faces misclassified as fraudulent faces are marked with red boxes. Experiments show that in the OMI library cross-C library experiment, only one real face was misclassified, while all other real faces were correctly detected. In the OCM library cross-I library experiment, only one fraudulent face was misclassified, while all other fraudulent faces were correctly detected. Visual analysis of the experiments further demonstrates the robustness of this embodiment. The parameter count of this embodiment is 190.56M, which is small and easy to implement in practice.

[0207] Training time was investigated. All experiments were conducted on an Intel Xeon Platinum 8336C CPU and an NVIDIA RTX 4090 GPU, with 5000 iterations under no load. The time test results were taken as the average of five runs to reduce the impact of fluctuations on the same device. The experimental results are shown in Table 3 below:

[0208] Table 3 Training Time

[0209]

[0210] As can be seen, the present invention has a fast convergence time, which is significantly faster than other algorithms that take a day to converge, thus further verifying the feasibility of the present invention.

[0211] Example 2

[0212] This invention provides a liveness detection system based on language-guided image feature extraction, used to implement the liveness detection method based on language-guided image feature extraction in Embodiment 1 above. The system includes: a video dataset preprocessing module, an image encoder, a text encoder, an image-to-image projector, a text-to-image projector, an image projection feature construction module, a contrast learning module, an image-to-text feature alignment module, a text design module, a text filtering module, a frozen text encoder module, a classifier, a supervised training module, and a prediction module.

[0213] In this embodiment, the video dataset preprocessing module is used to preprocess the acquired video dataset for face fraud detection, extract face images, and construct a text file containing face images and corresponding true / false labels;

[0214] In this embodiment, the image encoder is used to extract image features of a human face image;

[0215] In this embodiment, text encoding is used to extract the text features of text prompt words;

[0216] In this embodiment, the image-image projector is used to project different image features onto the same feature space;

[0217] In this embodiment, the text-image projector is used to project image and text features onto the same feature space;

[0218] In this embodiment, the image projection feature construction module is used to perform data augmentation on each face image to obtain two sub-views, obtain the corresponding feature representations through an image encoder, and obtain the corresponding image projection features based on the image-image projector mapping to another feature space.

[0219] In this embodiment, the contrastive learning module is used to calculate the similarity and differences of the projection features of two images based on the contrastive learning loss function;

[0220] In this embodiment, the image-text feature alignment module is used to map image features and text features to obtain corresponding projected features based on a text-image projector, and shorten the distance of the corresponding projected features based on alignment loss to achieve image and text feature alignment.

[0221] In this embodiment, the text design module generates multiple sets of text prompts based on a large language text model and preset constraints;

[0222] In this embodiment, the text filtering module is used to filter multiple sets of text prompts;

[0223] In this embodiment, the frozen text encoder module is used to freeze the pre-trained weights of the text encoder;

[0224] In this embodiment, the classifier is used to calculate the text prompt word that best matches the face by calculating cosine similarity and binary cross-entropy, and to determine the authenticity of the face based on the truthfulness of the text prompt word description.

[0225] In this embodiment, the supervised training module is used to perform supervised training on the three outputs of the contrastive learning module, the image-text feature alignment module, and the classifier;

[0226] In this embodiment, the prediction module is used to input the face image into the trained model to obtain the prediction result of whether the face is real or fake.

[0227] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A liveness detection method based on language-guided image feature extraction, characterized in that, Includes the following steps: A video dataset preprocessing module is constructed to preprocess the acquired video dataset for face fraud detection, extract face images, and construct a text file containing face images and corresponding true / false labels; Construct an image encoder and a text encoder. The image encoder extracts image features from face images, and the text encoder extracts text features from text prompts. Construct an image-image projector and a text-image projector. The image-image projector projects different image features into the same feature space, and the text-image projector projects image and text features into the same feature space. The image-image projector and the text-image projector have the same network structure, including three fully connected layers. Each layer contains a weight matrix and a bias vector, and performs nonlinear transformation through the ReLU activation function. The first fully connected layer receives the input feature vector and transforms it sequentially through batch normalization and ReLU activation function, setting negative values ​​to zero; The transformed output is used as the input to the second fully connected layer, and is processed sequentially through batch normalization and the ReLU activation function; The third fully connected layer then maps the features to the target output dimension, yielding the final output features. A contrastive learning module is constructed. Each face image is augmented to obtain two sub-views. The corresponding feature representations are obtained through an image encoder and mapped to another feature space based on an image-to-image projector. The contrastive learning module calculates the similarity and differences of the two image projection features based on the contrastive learning loss function. An image-text feature alignment module is constructed. Based on a text-image projector, image features and text features are mapped to obtain corresponding projected features. Based on alignment loss, the distance between the corresponding projected features is shortened to achieve image and text feature alignment. The text design module is constructed to generate multiple sets of text prompts based on a large language text model and preset constraints, specifically including: Three constraints were imposed on the text for real and fake faces, including: prohibiting the use of obscure words, prohibiting the use of clauses, and prohibiting descriptions of human features, resolution, environmental background, and lighting; To combat fraudulent faces, attack type alerts are provided, including printing attacks, replay attacks, 3D mask attacks, and adversarial sample attacks. A text filtering module is built to filter multiple sets of text prompts, specifically including: Calculate the cosine similarity between the image and each text prompt word, and use the text with the highest cosine similarity as the unique prompt word corresponding to the image; Calculate the matching frequency of each text and each image, and retain a set number of texts based on the matching frequency; Retain text prompts describing the attack type; Construct a frozen text encoder module to freeze the pre-trained weights of the text encoder; A classifier is constructed, which calculates the text prompt word that best matches the face by calculating cosine similarity and binary cross-entropy, and judges the authenticity of the face based on the truthfulness of the text prompt word description. By supervising the training of the contrastive learning module, the image-text feature alignment module, and the three outputs of the classifier, and inputting the face image into the trained model, the real and fake face prediction results are obtained.

2. The liveness detection method based on language-guided image feature extraction according to claim 1, characterized in that, The image encoder includes a patch embedding layer, a position embedding layer, a multi-layer Transformer encoder, and a linear projection layer; The patch embedding layer segments the face image into multiple patches and flattens them into a one-dimensional patch vector. The location embedding layer adds location information to each one-dimensional patch vector, preserving the spatial structure and order of the image, as shown below: z0 = [x1E; x2E; ...x N E]+E pos Where z0 represents the initial embedding features, x i Let E be the one-dimensional patch vector flattened from the i-th patch, and let E be the linear projection matrix. pos For the position embedding matrix; The Transformer encoder extracts global features from the initial embedded features through a self-attention mechanism; Each Transformer encoder sublayer is followed by a Layer Normalization layer, represented as follows: Where x represents the input feature vector, μ represents the mean of each feature in the batch, σ represents the standard deviation of each feature in the batch, γ and β represent the affine parameters to be learned, and ∈ is a constant; The linear projection layer maps the output features of the Transformer encoder to the target dimension, thus obtaining an image feature representation.

3. The liveness detection method based on language-guided image feature extraction according to claim 1, characterized in that, The text encoder includes a word embedding layer, a multi-layer Transformer encoder, and a linear projection layer; The word embedding layer maps each word of the input text to a high-dimensional feature space, generating initial features, represented as follows: have been i =W embed x i Where, x i Let W be the one-hot vector of the i-th word. embed For embedding matrix; The Transformer encoder extracts global features from the initial features through a self-attention mechanism; Each Transformer encoder sublayer is followed by a Layer Normalization layer, represented as follows: Where x represents the input feature vector, μ represents the mean of each feature in the batch, σ represents the standard deviation of each feature in the batch, γ and β represent the affine parameters to be learned, and ∈ is a constant; The linear projection layer maps the output features of the Transformer encoder to the target dimension, thus obtaining a text feature representation.

4. The liveness detection method based on language-guided image feature extraction according to claim 1, characterized in that, Each face image undergoes data augmentation to obtain two sub-views, and the corresponding feature representations are obtained through an image encoder, specifically as follows: v1 = E image (View1) v2=E image (View2) Where View1 and View2 represent two subviews, and v1 and v2 represent the feature representations corresponding to the two subviews, E image Indicates an image encoder; Based on the image-image projector mapping to another feature space, it can be represented as: v′1=Proj1(v1) v′2=Proj1(v2) Where Proj1 represents the image-to-image projector, and v′1 and v′2 represent the projection features corresponding to two image feature representations; The contrastive learning module calculates the similarity and differences of the projected features of two images based on the contrastive learning loss function, which is expressed as: Where sim represents the cosine similarity between two feature vectors, τ is the temperature parameter, and B is the batch size.

5. The liveness detection method based on language-guided image feature extraction according to claim 1, characterized in that, The text-image projector maps image features and text features to obtain corresponding projected features, represented as follows: x′=Proj2(x) x=E image (image) t=E text (text) Where image represents image features, text represents text features, Proj2 represents the text-image projector, and E image E represents the image encoder. text Indicates a text encoder, x ′ and t ′ This represents the mapping between image features and text features to obtain the corresponding projected features; The distance between the corresponding projected features is shortened based on the alignment loss, specifically expressed as follows: Among them, L inter This represents the L2 norm distance between two feature vectors, where B is the batch size.

6. The liveness detection method based on language-guided image feature extraction according to claim 1, characterized in that, The text prompts that best match the face are calculated using cosine similarity and binary cross-entropy. The authenticity of the face is determined based on the truthfulness of the text prompts, specifically as follows: Each type of text is encoded using a text encoder to obtain the embedding representation of each text, and these representations are stacked to form a new tensor. The cropped face image is encoded using an image encoder to obtain the image features of the image; Normalize the image features and text features, calculate the cosine similarity between the image features and text features, select the most similar text and determine its category. If the text describes a real face, the face image is judged as a real face; if the text describes a fake face, the face image is judged as a fake face. Supervision is performed using the true and false labels of the samples. The specific loss function formula is as follows: Among them, y i Let y' represent the true label of the i-th sample, where 0 indicates the face image is fraudulent and 1 indicates the face image is real. i The predicted probability of the i-th sample represents the probability that the model predicts the sample to be true.

7. A liveness detection system based on language-guided image feature extraction, characterized in that, To implement the liveness detection method based on language-guided image feature extraction as described in any one of claims 1-6, the system comprises: a video dataset preprocessing module, an image encoder, a text encoder, an image-to-image projector, a text-to-image projector, an image projection feature construction module, a contrast learning module, an image-to-text feature alignment module, a text design module, a text filtering module, a frozen text encoder module, a classifier, a supervised training module, and a prediction module. The video dataset preprocessing module is used to preprocess the acquired video dataset for face fraud detection, extract face images, and construct a text file containing face images and corresponding true / false labels; The image encoder is used to extract image features from human face images; The text encoding is used to extract the text features of text prompt words; The image-to-image projector is used to project different image features onto the same feature space; The text-image projector is used to project image and text features onto the same feature space; The image projection feature construction module is used to perform data augmentation on each face image to obtain two sub-views, obtain the corresponding feature representations through an image encoder, and then map the image to another feature space based on the image-image projector mapping to obtain the corresponding image projection features. The contrastive learning module is used to calculate the similarity and differences of the projection features of two images based on the contrastive learning loss function; The image-text feature alignment module is used to map image features and text features to obtain corresponding projected features based on a text-image projector, and shorten the distance of the corresponding projected features based on alignment loss to achieve image and text feature alignment. The text design module generates multiple sets of text prompts based on a large language text model and preset constraints; The text filtering module is used to filter multiple sets of text prompts; The frozen text encoder module is used to freeze the pre-trained weights of the text encoder; The classifier is used to calculate the text prompt word that best matches the face by calculating cosine similarity and binary cross-entropy, and to determine the authenticity of the face based on the authenticity of the text prompt word description. The supervised training module is used to supervise the training of the three outputs of the contrastive learning module, the image-text feature alignment module, and the classifier. The prediction module is used to input face images into the trained model to obtain the prediction results of whether the face is real or fake.

Images