Living body detection method and system based on language guide picture feature extraction

Through the language-guided image feature extraction method, combined with the comparison learning and image text alignment module, the problem of insufficient detection performance of the face anti-fraud detection system under attacks from different source domains is solved, and higher generalization ability and accuracy are achieved .

CN119028027AActive Publication Date: 2024-11-26SOUTH CHINA UNIV OF TECH

Patent Information

Application Number
CN202411002470.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-25
Publication Date
2024-11-26
Estimated Expiration
2044-07-25

AI Technical Summary

Technical Problem

The existing face anti-fraud detection system has poor detection performance when facing unknown attacks in different source domains, and traditional methods rely on feature design and have poor generalization effects. The deep learning model has overfitted due to limited data.

Method used

Using a language-guided picture feature extraction method, using pre-trained picture encoder and text encoder, combining the comparison learning module and the picture text alignment module, the frozen text encoder module is constructed by freezing the pre-training weight of the text encoder, and enhancing the Generalization performance of the model.

Benefits of technology

It improves the generalization ability of the model when facing invisible attacks, enhances feature extraction ability, reduces time complexity, and improves the accuracy of face authenticity and model robustness.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119028027A_ABST
    Figure CN119028027A_ABST
Patent Text Reader

Abstract

The invention discloses a living body detection method and system based on language guide picture feature extraction, and the method comprises the following steps: carrying out the preprocessing of a video data set for face fraud detection, and constructing a picture encoder and a text encoder; constructing a picture-picture projector and a text-picture projector, constructing a contrast learning module to shorten the distance of corresponding projection features, and generating a plurality of groups of text cue words by a text design module; the method comprises the following steps: constructing a text screening module, constructing a frozen text encoder module, freezing a pre-training weight of a text encoder, calculating a text cue word which is most matched with a human face through cosine similarity calculation and binary cross entropy, and judging whether the human face is true or false by a classifier according to whether the text cue word is described. And supervising and training three output results of the comparison learning module, the picture text feature alignment module and the classifier, and obtaining a face true and false prediction result based on the trained model. The method is higher in face authenticity judgment accuracy and better in generalization performance.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of face liveness detection, and in particular to a liveness detection method and system based on language-guided image feature extraction. Background Art

[0002] Compared with fingerprint and iris recognition, facial features are easier to obtain and are widely used for identity authentication. With the development of deep learning, face recognition technology has become more and more mature, but it also faces new challenges such as face spoofing. Facial anti-spoofing has become crucial, mainly to address attacks in the form of photos, masks, video playback, and paper covers to ensure system security. Liveness detection, as a front-end security system for face recognition, treats face authenticity as a binary or multi-class task. At present, face anti-spoofing detection methods can generally be divided into two categories: traditional machine learning methods using manual feature extraction and methods relying on deep learning.

[0003] Traditional machine learning methods use facial texture, dynamics, three-dimensional structure, and physiological signals. Early algorithms relied on manually designed features and used support vector machines for binary classification, such as using features such as Local Binary Pattern (LBP) and Difference of Gaussians (DoG). However, traditional methods rely too much on feature design, and the difference between fraudulent faces and real faces in representation attacks is small, resulting in poor generalization effects.

[0004] In contrast, deep learning algorithms with superior feature representation have now become the mainstream of face anti-fraud algorithms. These strategies include domain adaptation, domain generalization, anomaly detection, and multimodal fusion strategies. However, due to differences in image capture devices, attack methods, lighting conditions, and data resolution, there is a huge domain gap between the source distribution and the target distribution, which hinders the effective feature extraction of fraud features and thus limits the model generalization performance. In addition, the FAS benchmark dataset usually has limited training data, which causes the model to overfit the source domain features.

[0005] Therefore, the most important challenge currently facing face fraud detection systems is how to maintain excellent detection performance when facing unknown attacks that are different from the source domain. Summary of the invention

[0006] In order to overcome the defects and shortcomings of the prior art, the present invention provides a liveness detection method and system based on language-guided image feature extraction. The present invention utilizes image encoders and text encoders pre-trained on large-scale data sets, effectively overcoming the problem of insufficient feature extraction in existing methods. A contrastive learning module and an image-text alignment module are introduced to enable the network model to learn common fraud clues. The text design and screening strategy enable the pre-trained model to better adapt to the field of face anti-spoofing detection, and a frozen text encoder module is constructed to ensure that it has robust generalization performance even when trained on a smaller data set.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] The present invention provides a method for detecting a living body based on language-guided image feature extraction, comprising the following steps:

[0009] Construct a video dataset preprocessing module to preprocess the acquired video dataset for face fraud detection, capture face images, and construct a text file containing face images and corresponding authenticity labels;

[0010] Construct an image encoder and a text encoder. The image encoder extracts the image features of the face image, and the text encoder extracts the text features of the text prompt word.

[0011] Constructing a picture-to-picture projector and a text-to-picture projector, wherein the picture-to-picture projector projects different picture features into the same feature space, and the text-to-picture projector projects picture and text features into the same feature space;

[0012] Construct a contrastive learning module. Each face image is enhanced to obtain two sub-views. The corresponding feature representation is obtained through the image encoder. The sub-views are mapped to another feature space based on the image-image projector. The contrastive learning module calculates the similarity and difference of the projection features of the two images based on the contrastive learning loss function.

[0013] Construct an image-text feature alignment module, map image features and text features to obtain corresponding projection features based on the text-image projector, shorten the distance of corresponding projection features based on the alignment loss, and achieve image and text feature alignment;

[0014] Build a text design module to generate multiple sets of text prompt words based on the language text model and preset constraints;

[0015] Construct a text filtering module to filter multiple groups of text prompt words;

[0016] Build a frozen text encoder module to freeze the pre-trained weights of the text encoder;

[0017] Build a classifier, calculate the text prompt word that best matches the face through cosine similarity calculation and binary cross entropy, and judge the authenticity of the face based on the truth or falsehood of the text prompt word;

[0018] By performing supervised training on the three output results of the contrastive learning module, the image-text feature alignment module, and the classifier, the face image is input into the trained model to obtain the true or false prediction result of the face.

[0019] As a preferred technical solution, the picture encoder includes a patch embedding layer, a position embedding layer, a multi-layer Transformer encoder and a linear projection layer;

[0020] The patch embedding layer divides the face image into multiple patches and flattens them into a one-dimensional patch vector;

[0021] The position embedding layer adds position information to each one-dimensional patch vector, preserving the spatial structure and order of the image, expressed as:

[0022] z0=[x1E;x2E;...x N E]+E pos

[0023] Among them, z0 represents the initial embedding feature, x i is the one-dimensional patch vector after the i-th patch is flattened, E is the linear projection matrix, E pos is the position embedding matrix;

[0024] The Transformer encoder extracts global features of the initial embedded features through the self-attention mechanism;

[0025] Each Transformer encoder sublayer is followed by a Layer Normalization layer, expressed as:

[0026]

[0027] Among them, x represents the input feature vector, μ represents the mean of each feature in the batch, σ represents the standard deviation of each feature in the batch, γ and β represent the affine parameters to be learned, and ∈ is a constant;

[0028] The linear projection layer maps the output features of the Transformer encoder to the target dimension to obtain a picture feature representation.

[0029] As a preferred technical solution, the text encoder includes a word embedding layer, a multi-layer Transformer encoder and a linear projection layer;

[0030] The word embedding layer maps each word of the input text to a high-dimensional feature space and generates initial features, which are expressed as:

[0031] e i =W embed x i

[0032] Among them, x i is the one-hot vector of the i-th word, W embed is the embedding matrix;

[0033] The Transformer encoder extracts global features of the initial features through the self-attention mechanism;

[0034] Each Transformer encoder sublayer is followed by a Layer Normalization layer, expressed as:

[0035]

[0036] Among them, x represents the input feature vector, μ represents the mean of each feature in the batch, σ represents the standard deviation of each feature in the batch, γ and β represent the affine parameters to be learned, and ∈ is a constant;

[0037] The linear projection layer maps the output features of the Transformer encoder to the target dimension to obtain text feature representation.

[0038] As a preferred technical solution, the network structure of the picture-picture projector and the text-picture projector is the same, including three fully connected layers, each layer contains a weight matrix and a bias vector, and performs nonlinear transformation through a ReLU activation function. The ReLU activation function is expressed as:

[0039] ReLU(x)=max(0,x)

[0040] Among them, x is the input feature vector, which represents the image feature vector or text feature vector;

[0041] The first fully connected layer receives the input feature vector and transforms it through the ReLU activation function, setting negative values ​​to zero;

[0042] The transformed output is used as the input of the second fully connected layer and processed by the ReLU activation function;

[0043] The features are then mapped to the target output dimension through the third fully connected layer to obtain the final output features.

[0044] As a preferred technical solution, each face image is enhanced to obtain two sub-views, and the corresponding feature representation is obtained through the image encoder, which is specifically expressed as:

[0045] v1=E image(View1)

[0046] v2=E image (View2)

[0047] Among them, View1 and View2 represent two subviews respectively, v1 and v2 represent the feature representations corresponding to the two subviews, and E image represents a picture encoder;

[0048] Based on the image-image projector mapping to another feature space, it is expressed as:

[0049] v′1=Proj1(v1)

[0050] v′2=Proj2(v2)

[0051] Among them, Proj1 represents the image-to-image projector, v′1 and v′2 represent the corresponding projection features of two image features;

[0052] The contrastive learning module calculates the similarities and differences between the projection features of two images based on the contrastive learning loss function. The contrastive learning loss function is expressed as:

[0053]

[0054] Among them, sim represents the cosine similarity between two feature vectors, τ is the temperature parameter, and B is the batch size.

[0055] As a preferred technical solution, the image features and text features are mapped to obtain corresponding projection features based on the text-image projector, which is expressed as:

[0056] x′=Proj2(x)

[0057] x=E image (image)

[0058] t′=Proj2(t)

[0059] t=E text (text)

[0060] Among them, image represents image features, text represents text features, Proj2 represents text-image projector, E image represents the picture encoder, E text represents the text encoder, x′ and t′ represent the corresponding projection features obtained by mapping the image features and text features;

[0061] The distance of the corresponding projection features is shortened based on the alignment loss, which is specifically expressed as:

[0062]

[0063] Among them, L inter represents the L2 norm distance between two feature vectors, and B is the batch size.

[0064] As a preferred technical solution, multiple groups of text prompt words are generated based on the language text model and preset constraints, specifically including:

[0065] Three constraints were imposed on the real and fake face texts, including: prohibiting the use of uncommon words, prohibiting the use of clauses, and prohibiting the description of character features, resolution, environmental background, and lighting;

[0066] For fraudulent faces, set attack type prompts, including printing attacks, replay attacks, 3D mask attacks, and adversarial sample attacks.

[0067] As a preferred technical solution, a text screening module is constructed to screen multiple groups of text prompt words, specifically including:

[0068] Calculate the cosine similarity between the image and each text prompt word, and use the text with the highest cosine similarity as the unique prompt word corresponding to the image;

[0069] Count the matching frequency between each text and each image, and retain a set number of texts based on the matching frequency;

[0070] Keep textual hints describing the type of attack.

[0071] As a preferred technical solution, the text prompt word that best matches the face is calculated by cosine similarity calculation and binary cross entropy, and the authenticity of the face is judged according to the authenticity of the text prompt word description, which is specifically expressed as:

[0072] Encode each type of text through a text encoder to obtain the embedded representation of each text, and stack them to form a new tensor;

[0073] Encode the cropped face image through an image encoder to obtain the image features of the image;

[0074] Normalize the image features and text features, calculate the cosine similarity between the image features and the text features, select the most similar text and determine its category. If the text describes a real face, the face image is judged as a real face; if the text describes a fake face, the face image is judged as a fake face.

[0075] The true and false labels of the samples are used for supervision. The specific loss function formula is as follows:

[0076]

[0077] Among them, y irepresents the true label of the i-th sample, 0 means the face image is fraudulent, 1 means the face image is real, y′ i Represents the predicted probability of the i-th sample, indicating the probability that the model predicts that the sample is true.

[0078] The present invention also provides a liveness detection system based on language-guided image feature extraction, which is used to implement the above-mentioned liveness detection method based on language-guided image feature extraction, and the system includes: a video data set preprocessing module, a picture encoder, a text encoder, a picture-picture projector, a text-picture projector, a picture projection feature construction module, a contrast learning module, a picture-text feature alignment module, a text design module, a text screening module, a frozen text encoder module, a classifier, a supervised training module and a prediction module;

[0079] The video data set preprocessing module is used to preprocess the acquired video data set for face fraud detection, intercept the face image, and construct a text file containing the face image and the corresponding authenticity label;

[0080] The image encoder is used to extract image features of a face image;

[0081] The text encoding is used to extract text features of text prompt words;

[0082] The picture-to-picture projector is used to project different picture features into the same feature space;

[0083] The text-image projector is used to project image and text features into the same feature space;

[0084] The image projection feature construction module is used to perform data enhancement on each face image to obtain two sub-views, obtain the corresponding feature representation through the image encoder, and map it to another feature space based on the image-image projector to obtain the corresponding image projection feature

[0085] The contrastive learning module is used to calculate the similarities and differences of the projection features of two images based on the contrastive learning loss function;

[0086] The image-text feature alignment module is used to map the image features and the text features based on the text-image projector to obtain corresponding projection features, shorten the distance of the corresponding projection features based on the alignment loss, and realize image and text feature alignment;

[0087] The text design module generates multiple groups of text prompt words based on the language text model and preset constraints;

[0088] The text screening module is used to screen multiple groups of text prompt words;

[0089] The frozen text encoder module is used to freeze the pre-trained weights of the text encoder;

[0090] The classifier is used to calculate the text prompt word that best matches the face through cosine similarity calculation and binary cross entropy, and judge the authenticity of the face according to the authenticity of the text prompt word description;

[0091] The supervised training module is used to perform supervised training on three output results of the contrastive learning module, the image-text feature alignment module, and the classifier;

[0092] The prediction module is used to input the face image into the trained model to obtain the true or false prediction result of the face.

[0093] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0094] (1) The present invention constructs an image-text feature alignment module, which projects the image features and text features extracted by the image encoder and the text encoder, and brings the features closer in the new feature space, thereby reducing the distance between features of different modalities and improving their alignment stability between large text language models. This not only reduces the difference between feature modalities, but also improves the alignment accuracy of the model, ensuring the semantic consistency between text and images. In addition, through this feature alignment method, the model can handle cross-modal tasks more effectively, thereby greatly improving its generalization ability when facing invisible attacks.

[0095] (2) The present invention constructs a text design module and a text screening module. Specifically, GPT-4 is used to generate text that is easier to be understood by the text encoder, thereby enhancing the ability to extract text features. By providing text prompt words that clearly indicate the attack method, the model can learn different attack category information during the training process. By restricting the length, complexity, and uncommon words of the text prompt words, the feature extraction ability of the text encoder is further enhanced. The text screening strategy selects texts with a larger weight for model optimization during the training process, and discards texts with a smaller weight to increase the ambiguity of the text. Too much text can easily lead to more accurate text-image matching, thereby causing the problem of overfitting in the library. Text screening can increase the ambiguity of the text prompt words to a certain extent, thereby improving the generalization ability.

[0096] (3) The present invention constructs a frozen text encoder module. By freezing the pre-trained weights, the text encoder will not be optimized due to the text prompt word preference. Only the picture encoder, picture-to-picture projector, and text-to-picture projector are updated, forcing the network model to focus on the picture encoder for training optimization, making fuller use of the existing picture data set to learn more general face fraud clues. At the same time, freezing the pre-trained weights reduces the parameter optimization time during the training process, thereby reducing the time complexity of the model while keeping the model parameter quantity unchanged.

[0097] (4) The network proposed in the present invention has excellent generalization performance when facing invisible attacks. Compared with traditional machine learning methods and other deep learning algorithms that do not use large text and picture models, it has higher accuracy in judging the authenticity of faces, shorter training time, and better generalization performance. It only uses text and picture dual modalities and is suitable for deployment in devices such as mobile phones and embedded terminals, and has high practical value. BRIEF DESCRIPTION OF THE DRAWINGS

[0098] Figure 1 It is a flow chart of the method for liveness detection based on language-guided image feature extraction of the present invention;

[0099] Figure 2 It is a schematic diagram of the overall implementation framework of the living body detection method based on language-guided image feature extraction of the present invention;

[0100] Figure 3 A schematic diagram of the network structure of the picture encoder of the present invention;

[0101] Figure 4 A schematic diagram of the network structure of the text encoder of the present invention;

[0102] Figure 5 A schematic diagram of the network structure of the picture-picture projector and the text-picture projector of the present invention;

[0103] Figure 6 A schematic diagram of a process in which a comparative learning module of the present invention learns similarities and differences between two sub-views;

[0104] Figure 7 This is a schematic diagram of the network structure of the image text feature alignment module of the present invention;

[0105] Figure 8 A text schematic diagram designed for the text design module of the present invention;

[0106] Fig. 9 It is a schematic diagram for visualizing the experimental results of the present invention. DETAILED DESCRIPTION

[0107] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0108] Example 1

[0109] This embodiment uses the four public liveness detection video datasets Replay-Attack, CASIA-MFSD, MSU_MFSD and OULU for training and testing as examples to introduce the implementation process of this embodiment in detail. The Replay-Attack database was released by the IDIAP Institute in Switzerland in 2012 and contains a total of 1200 videos shot of 0 different experimental personnel under different lighting conditions. The data set is divided into four categories: training set, validation set, test set and registration set (video does not need to be used), and the corresponding number of people is 15, 15, and 20 respectively. The video is recorded by the built-in camera of a Macbook laptop and QuickTime software. The CASIA-FASD dataset was released by the Institute of Automation, Chinese Academy of Sciences in 2012 and contains a total of 600 face videos shot by three different devices. The video shooting equipment is Sony NEX-5 (video resolution is 1280×720), a new USB camera (video resolution is 640×480), and an old USB camera (video resolution is 480×640). The MSU-MFSD dataset was released by the Pattern Recognition and Image Processing Laboratory of Michigan State University in 2015, and contains a total of 280 videos shot by 35 people, with a true-to-false ratio of 1:3. The shooting devices are Nexus 5 mobile phones (resolution 720×480) and MacBook Air 13 (resolution 640×480). The OULU dataset was released by the University of Oulu in Finland and Northwestern Polytechnical University in China in 2017, and contains a total of 4950 videos shot by 55 people, with a true-to-false ratio of 1:4. The videos were shot by the front cameras of 6 mobile devices (Samsung Galaxy S6 edge, HTC Desire EYE, MEIZU X5, ASUS Zenfone Selfie, Sony XPERIA C5 Ultra Dual and OPPO N3), with three different lighting conditions and background scenes.

[0110] This embodiment runs on the Linux Ubuntu system and is implemented using the Pytorch deep learning framework. The graphics card model is NVIDIA GeForce RTX 4090GPU, the Python version is 3.8.10, and the Pytorch version is 2.12+cu121.

[0111] like Figure 1 , Figure 2 As shown, this embodiment provides a method for liveness detection based on language-guided image feature extraction, comprising the following steps:

[0112] S1. Construct a video dataset preprocessing module, divide the public face fraud detection video dataset, perform frame extraction on the video dataset, detect the face area and capture the face image, and construct a text file containing the face image and the corresponding authenticity label;

[0113] In this embodiment, ffmpeg is used to perform frame processing on the video data set, and the MTCNN algorithm is used to detect and crop the face. Finally, the Lanczos interpolation algorithm is used to unify the resolution of the face image and correct it to H×W×C (256×256×3 in this embodiment), which respectively represent the height, width and number of channels of the cropped face image;

[0114] S2, build a picture encoder;

[0115] like Figure 3 As shown in the figure, the image encoder is used to extract image features. The network structure is mainly built based on the ViT architecture, including a 16×16 patch embedding layer, a position embedding layer, a multi-layer Transformer encoder and a linear projection layer;

[0116] The patch embedding layer divides the image into small patches of 16×16 pixels and flattens them into one-dimensional vectors for easy subsequent processing. The position embedding layer adds position information to each patch vector to preserve the spatial structure and order of the image.

[0117] The multi-layer Transformer encoder deeply processes patch features through self-attention mechanism and feed-forward neural network to capture the relationship and context information between patches. The linear projection layer maps the output features of the Transformer encoder to the target dimension, providing the final feature representation for downstream tasks;

[0118] The specific process is as follows: the resolution of the input image is 224×224. First, the patch embedding layer is used to divide the input image into 16×16 patches, resulting in a total of 14×14 patches. Each patch is flattened into a vector and mapped to a 768-dimensional high-dimensional feature space through a linear projection layer to obtain an initial feature of size 14×14×768. Then, position embedding is added to each patch vector to retain the spatial position information. The specific formula is as follows:

[0119] z0=[x1E;x2E;...x N E]+E pos

[0120] Among them, z0 represents the initial embedding feature, x i is the flattened vector of the ith patch, N = 196, E is the linear projection matrix, E pos is the position embedding matrix; the initial features are then passed through a 12-layer Transformer encoder, each layer includes a multi-head self-attention layer and a feedforward neural network layer, and each sublayer is followed by a Layer Normalization layer. The Transformer encoder extracts the global features of the image through the self-attention mechanism to obtain a feature map of size 14×14×768. The formula of the self-attention mechanism is as follows:

[0121]

[0122] Among them, Q (Query), K (Key) and V (Value) are obtained by different linear transformations of input features, d k is the dimension of K.

[0123] The Layer Normalization layer formula is as follows:

[0124]

[0125] Among them, x represents the input feature vector, μ represents the mean of each feature in the batch, σ represents the standard deviation of each feature in the batch, γ and β represent the affine parameters to be learned, and ∈ is a very small constant to prevent division by zero;

[0126] S3, build a text encoder;

[0127] In this embodiment, the text encoder is used to extract text features, such as Figure 4 As shown in the figure, the network structure is mainly built based on the Transformer architecture, which includes a word embedding layer, a multi-layer Transformer encoder and a linear projection layer;

[0128] The specific process is as follows: the input text first passes through the word embedding layer to map each word into a high-dimensional feature space, generating an initial feature of size 197×768, where 197 is the context length and 768 is the embedding dimension. The formula for word embedding is as follows:

[0129] e i =W embed x i

[0130] Among them, x i is the one-hot vector of the i-th word, W embedis the embedding matrix. The initial features are processed by a 12-layer Transformer encoder. Each layer contains a multi-head self-attention layer and a feedforward neural network layer. Each sub-layer is followed by a Layer Normalization layer. The Transformer encoder extracts the global features of the text through the self-attention mechanism and generates a feature map of size N×D. The formula of the self-attention mechanism is as follows:

[0131]

[0132] Among them, Q (Query), K (Key) and V (Value) are obtained by different linear transformations of input features, d k is the dimension of K. Finally, the feature map is passed through a linear projection layer to obtain a text feature vector of size 768. The LayerNormalization layer formula is as follows:

[0133]

[0134] Among them, μ represents the mean of each feature in the batch, σ represents the standard deviation of each feature in the batch, γ and β represent the affine parameters to be learned, and ∈ is a very small constant to prevent division by zero;

[0135] S4, constructing a picture-picture projector and a text-picture projector;

[0136] The image-image projector projects different image features into the same feature space, and the text-image projector projects different text features and image features into the same feature space for subsequent optimization operations, such as Figure 5 As shown, the two projectors have the same structure, but during the training process, the weight matrix and bias vector are optimized according to different data to achieve different goals.

[0137] The specific process is as follows: Input the image features or text features with the input dimension in_dim into a fully connected layer and map them to the feature space with the dimension mlp_dim. The fully connected layer contains the weight matrix and the bias vector, which work together in the forward propagation process to map the input features to the output features. The formula is as follows:

[0138] y=W·x+b

[0139] Where W is the weight matrix, x is the input feature vector, b is the bias vector, and y is the output of the linear projection;

[0140] The batch normalization layer is used to standardize the feature distribution, and the nonlinear transformation is introduced through the ReLU activation function to set the negative values ​​to zero. The formula is as follows:

[0141] ReLU(x)=max(0,x)

[0142] Among them, ReLU(x) represents the maximum value of 0 and x;

[0143] The features that have passed the activation function are passed through a fully connected layer again to maintain the dimension of mlp_dim, followed by batch normalization and ReLU function activation. Finally, they pass through a fully connected output layer to map the features to the output dimension of out_dim to obtain the final output features. This layer-by-layer linear transformation and nonlinear activation enables the model to learn complex feature representations.

[0144] S5, build contrastive learning module;

[0145] like Figure 6 As shown in the figure, the contrastive learning module projects through a picture-to-picture projector, and then uses contrastive learning to learn the similarities and differences between the two sub-views. By bringing together the features of different data-enhanced views of the same picture, it solves the problem that traditional methods are sensitive to environmental factors such as lighting.

[0146] The specific process is as follows: perform data enhancement on the images preprocessed by the data preprocessing module, such as random cropping, brightness change, chromaticity change, adding noise, etc., so that each image obtains two sub-views View1 and View2. The two sub-views View1 and View2 are passed through the image encoder to obtain their feature representations v1 and v2. The output feature dimension of the image encoder is 512. The specific formula is as follows:

[0147] v1=E image (View1), v2 = E image (View2)

[0148] The dimensions of v1 and v2 are B×512, where B is the batch size. The image-image projector Proj1 is used to map them to another feature space to obtain projected features v′1 and v′2. The image-image projector reduces the feature dimension from 512 to 256. The specific formula is as follows:

[0149] v′1=Proj1(v1), v′2=Proj1(v2)

[0150] Among them, the dimensions of v′1 and v′2 are B×256. The contrastive learning loss function is used to calculate the similarities and differences between the projection features of the two images to supervise the optimization process of contrastive learning. The gradient of the loss function to the model parameters is calculated through the back propagation algorithm, and the model parameters are updated using the optimization algorithm to minimize the contrastive loss, thereby bringing the view features of different data enhancements of the same image closer. Contrastive learning loss function L simCLRThe specific formula is as follows:

[0151]

[0152] Among them, sim(v′1,v′2) represents the cosine similarity between two feature vectors, τ is the temperature parameter used to adjust the smoothness of the similarity distribution, and the sum in the denominator represents the sum of the similarities of all samples in the batch;

[0153] S6. Build an image-text feature alignment module to better align images and texts by shortening the distance between image features and text features with the same label.

[0154] like Figure 7 As shown in the figure, the image text feature alignment module is projected through the text-image projector, and the semantic consistency of text and image is ensured by shortening the distance between text features and image features. Feature alignment is achieved by calculating the mean difference between image and text features, while retaining content information. This process does not require explicit image synthesis, but only requires operations at the feature level, which is simple to implement and has low computational complexity.

[0155] The specific process is as follows: the input image is named image and the input text is named text. They are respectively passed through the image encoder and text encoder mentioned in S2 and S3 to obtain the image feature x and text feature t. The specific formula is as follows:

[0156] x=E image (image), t = E text (text)

[0157] The dimensions of x and t are B×512, and B is the batch size. The text-image projector Proj2 in step S4 is used to map it to a new feature space to obtain the projected features x′ and t′, and the feature dimensions remain unchanged. The specific formula is as follows:

[0158] x′=Proj2(x), t′=Proj2(t)

[0159] The dimensions of x′ and t′ are B×512, and B is the batch size. The mean square error is used to narrow the distance between x′ and t′, and the alignment loss function L is designed. inter , to supervise the optimization process of alignment loss. inter The specific formula is as follows:

[0160]

[0161] L interRepresents the L2 norm distance between two feature vectors. The similarity between image features and text features is measured by calculating the mean of the distance between each sample in the image and text feature space;

[0162] S7. Build a text design module to design text prompts that are more conducive to text model understanding through GPT-4 and related prompts and constraints;

[0163] In this embodiment, the text design module is designed for text prompt words to solve the problem that the text prompt words in the existing method are too common and single, so that the text encoder fails to extract more valuable text features, resulting in poor alignment of text features and image features and poor generalization performance. Figure 8 As shown, the text design module designs 2 groups of texts describing faces, each group has 12 texts, a total of 24 texts;

[0164] The specific process is as follows: preprocess the face video dataset, perform face recognition and cropping. From each dataset, select some preprocessed faces and input them into the multimodal version of GPT-4, and use the language text model to generate text prompt words. At the same time, give it certain constraints, such as prohibiting the use of uncommon words, prohibiting the generation of complex and long sentences that are difficult to understand, etc.

[0165] Specifically, three constraints are imposed on real and fake face texts:

[0166] 1. Do not use uncommon words. Only use simple daily words and specific terms in the field of face fraud detection.

[0167] 2. Clauses are prohibited and sentences are prohibited from exceeding 20 words;

[0168] 3. It is forbidden to describe the characteristics of people, resolution, environmental background, and lighting. Only declarative sentences describing the authenticity of the face are allowed;

[0169] For fraudulent faces, additional attack type prompts are given, of which 4 sentences contain specific attack method descriptions - printing attack, replay attack, 3D mask attack and adversarial sample attack. The remaining 8 fraudulent sentences are still generated according to the above constraints;

[0170] S8. Build a text screening module to expand and discard texts, further select text prompt words that have a great impact on model optimization, and make the model converge faster;

[0171] In this embodiment, the text screening module screens the pre-designed 24 texts into 12 texts (6 for each of the real and fake face description texts). Because the increase in the number of texts is more likely to lead to overfitting of the model, the performance is good within the library, but poor in the cross-library experiment facing invisible attacks. Therefore, the text screening strategy is used to select texts that contribute more to model optimization, and the remaining texts are discarded, and the fuzziness of the text prompt words is increased to improve the generalization ability of the model;

[0172] Specifically, text and images are entered into Figure 2 In the constructed network, the cosine similarity between the image and each text prompt word is calculated, and the text with the highest cosine similarity is used as the unique prompt word corresponding to the image. For the 12 text prompt words describing real faces, the matching frequency of each text and each image is counted, the top six texts with matching frequencies are retained, and the bottom six texts with matching frequencies are discarded. For the 12 text prompt words describing fraudulent faces, the matching frequency of each text and each image is counted. The four text prompt words describing printing attacks, replay attacks, 3D mask attacks, and adversarial sample attacks are retained, and the two text prompt words with the highest matching frequencies are additionally retained, and the other text prompt words are discarded. The text filtering module is used to convert a 24-classification problem into a 12-classification problem, while increasing the stability of the model, making it easier to align image features with text features, and making the model easier to classify;

[0173] S9. Construct a frozen text encoder module; by freezing, the text encoder retains its pre-trained generalization features, solving the problem of weakened generalization ability of traditional methods during training optimization;

[0174] In this embodiment, the frozen text encoder module freezes the pre-trained weights of the text encoder, so that the model only updates the weights of the image feature extractor and the image-image and text-image projectors. By utilizing the generalized features pre-trained on a large-scale data set, the generalization ability of the model in face fraud detection is enhanced. Specifically, the text encoder is frozen before model training, and the parameters that need to be updated in the model are printed to ensure that there is no transformer module of the text encoder, but there is a transformer module of the image encoder;

[0175] S10, construct a classifier, calculate the text prompt word that best matches the face through cosine similarity calculation and binary cross entropy, and then judge the authenticity of the face based on the authenticity of the text prompt word description;

[0176] In this embodiment, the classifier is used to output the probability value of whether the sample is real or fraudulent. The classifier includes two fully connected layers, a ReLU activation function layer and a Dropout layer;

[0177] Specifically, each type of text (6 deceptive texts and 6 real texts) is encoded through a text encoder to obtain the embedded representation of each text. The embedded representations of all 12 texts are stacked together to form a new tensor text_features. The cropped face image is encoded through an image encoder to obtain the image features of the image. The image features and text features are normalized to unitize each vector and ensure that the feature vector is in one dimension. The cosine similarity between the image features and the text features is calculated, and the model is regarded as a twelve-classification problem. By selecting the most similar text and judging its category, if it is a text describing a real face, the face image is judged as a real face, and if it is a text describing a fake face, the face image is judged as a fake face; the authenticity label of the sample is used for supervision, and the specific loss function formula is as follows:

[0178]

[0179] Among them, y i Represents the true label of the i-th sample, 0 means the face image is fraudulent, and 1 means the face image is real. i ′ represents the predicted probability of the i-th sample, indicating the probability that the model predicts that the sample is positive (true);

[0180] S11, constructing a network model training module, supervising the three output results of the contrastive learning module, the image text feature alignment module, and the classifier, obtaining the contrastive learning loss, the alignment loss, and the classification loss, and then obtaining the total loss function. The model training is supervised by the total loss function, and at the end of the training, the optimal network model and the best weight are retained according to the benchmark indicators;

[0181] In this embodiment, the network model training module inputs the face image and the text prompt word into Figure 2 The constructed network updates the network weight coefficients through the constraints of the total loss function, and saves the network model and optimal weights after training is completed;

[0182] Specifically, we first load the pre-trained weights into the image encoder module and the text encoder module, and then input the RGB face image and 12 text prompts into the network model for end-to-end training. We use the Adam optimizer for optimization, and the initial learning rate is set to 1×10 -6 , weight decay is set to 1×10 -6 , the batch size is set to 1, and 5000 iterations are performed for each set of protocols. total Update the weight parameters of the network model and save the network model and the optimal weights according to the performance of the benchmark test indicators;

[0183] The total loss function L involved in this processtotal The specific formula is as follows:

[0184] L total =αL simCLR +βL inter +γL ce

[0185] Where L simCLR represents contrastive learning loss, L inter represents the image-text feature alignment loss, L ce represents the classifier loss, α, β, and γ are weight coefficients of the three loss functions. In this embodiment, α, β, and γ are set to 1, 0.1, and 1;

[0186] S12, test using the saved optimal model and weights;

[0187] In this embodiment, the optimal model and weights obtained through training are loaded to build a test network, and images of the test data set are input to test the network model. The pros and cons of the network model are measured according to the benchmark indicators, and visualization analysis, parameter quantity, and time complexity are explored to jointly explore the actual feasibility of the model implementation.

[0188] The specific process is as follows: extract frames and crop faces from the video of the test data set, input the trained model and load the optimal weights, extract features from the face image and text prompts, perform cosine similarity matching between the image and the features, calculate common benchmark indicators in the field of face fraud detection based on the prediction results, and finally perform visualization analysis of the model and calculation of parameter quantity and time complexity. The benchmark indicators used include False Positive Rate (FPR), False Negative Rate (FNR), True Positive Rate (TPR), Half Total Error Rate (HTER), and Area Under Curve (AUC);

[0189] The calculation of false positive rate and false negative rate involves four indicators TP, FP, FN, TN; when the label is true and the prediction is true, it is recorded as TP, when the label is false and the prediction is true, it is recorded as FP, when the label is true and the prediction is false, it is recorded as FN, when the label is false and the prediction is false, it is recorded as TN;

[0190] The false positive rate is calculated by calculating the proportion of faces labeled as fake but misclassified as real faces among all faces labeled as fake. The specific calculation formula is as follows:

[0191]

[0192] The false negative rate is calculated by calculating the proportion of all faces labeled as real people but misclassified as fake people. The specific calculation formula is as follows:

[0193]

[0194] The true positive rate is calculated by calculating the proportion of faces that are labeled as real faces and judged as real faces among all faces that are labeled as real faces. The specific calculation formula is as follows:

[0195]

[0196] The half-total error rate is the average of the false positive rate and the false negative rate. When the half-total error rate is smaller, the false alarm rate and false negative rate of the model are smaller, which proves that the accuracy of the model is higher. The specific calculation formula is as follows:

[0197]

[0198] The receiver operating characteristic curve is plotted with the false positive rate as the horizontal axis and the true positive rate as the vertical axis. The larger the area under the curve, the better the overall performance of the model.

[0199] This embodiment conducts cross-database experiments on four public video databases: CASIA-MFSD, Replay-Attack, MSU-MFSD, and OULU. Specifically, three databases are selected as training sets, and another data set is used for testing to verify the generalization ability of the model. The four groups of experiments are set according to the protocol as OCI database cross M database, OMI database cross C database, OCM database cross I database, and ICM database cross O database. The optimal experimental results are shown in Tables 1 and 2 below;

[0200] Table 1 Cross-library test HTER

[0201]

[0202] Table 2 Cross-library test AUC

[0203]

[0204] This embodiment is compared with the PatchNet, ViT, and DiVT-M methods. The PatchNet algorithm was published in CVPR in 2022, and proposed a simple framework PatchNet for facial anti-counterfeiting detection through refined patch recognition; the ViT algorithm was published in ECCV in 2022, and proposed an adaptive transformer to improve the robustness of the system through few-sample cross-domain facial anti-counterfeiting detection; DiVT-M was published in WACV in 2023, and proposed a domain-invariant visual transformer learning method to improve its generalization ability between different domains. These three top conferences and journals are very representative in the field of face fraud detection, and have also achieved excellent performance.

[0205] Through the analysis of Tables 1 and 2, it can be obtained that the present invention has achieved the best results in the semi-total error rate in the three experiments of OMI library across C library, OCM library across I library, and ICM library across O library, which are 1.33%, 2.85%, and 5.27% respectively. At the same time, it has also achieved quite competitive results in the experiment of OCI library across M library, with a semi-total error rate of 4.52%. Correspondingly, the area under the receiver operating characteristic curve has achieved the best results in the three experiments of OMI library across C library, OCM library across I library, and ICM library across O library, which are 99.78%, 99.67%, and 98.76% respectively. At the same time, it has also achieved quite competitive results in the experiment of OCI library across M library, with an area under the receiver operating characteristic curve of 98.72%. It has been proved through experiments that the method of this embodiment has excellent generalization performance when facing invisible attacks.

[0206] like Fig. 9 As shown, the visualization of experimental detection errors is obtained, and the misclassified face images are saved after each test: the fraudulent faces misclassified as real faces are marked with black frames, and the real faces misclassified as fraudulent faces are marked with red frames. Through experiments, it is found that in the experiment of OMI library across C library, only one real face is misclassified, and all other real faces are correctly detected. In the experiment of OCM library across I library, only one fraudulent face is misclassified, and all other fraudulent faces are correctly detected. The robustness of this embodiment is further demonstrated through the visualization analysis of the experiment. The parameter amount of this embodiment is 190.56M, and the small parameter amount facilitates the actual implementation of this embodiment.

[0207] The training time was explored. All experiments were conducted on Intel Xeon Platinum 8336C CPU and NVIDIA RTX 4090 GPU. 5000 iterations were performed without load. The time test results were averaged over five runs to reduce the impact of fluctuations on the same device. The experimental results are shown in Table 3 below:

[0208] Table 3 Training time

[0209]

[0210] It can be seen that the convergence time of the present invention is fast, and compared with other algorithms that take one day to converge, it has obvious advantages, which further verifies the feasibility of the implementation of the present invention.

[0211] Example 2

[0212] The present invention provides a liveness detection system based on language-guided image feature extraction, which is used to implement the liveness detection method based on language-guided image feature extraction of the above-mentioned embodiment 1, and the system comprises: a video data set preprocessing module, a picture encoder, a text encoder, a picture-picture projector, a text-picture projector, a picture projection feature construction module, a contrast learning module, a picture-text feature alignment module, a text design module, a text screening module, a frozen text encoder module, a classifier, a supervised training module and a prediction module;

[0213] In this embodiment, the video data set preprocessing module is used to preprocess the acquired video data set for face fraud detection, capture face images, and construct a text file containing face images and corresponding authenticity labels;

[0214] In this embodiment, the picture encoder is used to extract picture features of the face image;

[0215] In this embodiment, text encoding is used to extract text features of text prompt words;

[0216] In this embodiment, the picture-to-picture projector is used to project different picture features into the same feature space;

[0217] In this embodiment, the text-image projector is used to project the image and text features into the same feature space;

[0218] In this embodiment, the image projection feature construction module is used to perform data enhancement on each face image to obtain two sub-views, obtain the corresponding feature representation through the image encoder, and map it to another feature space based on the image-image projector to obtain the corresponding image projection feature.

[0219] In this embodiment, the contrastive learning module is used to calculate the similarities and differences between the projection features of two images based on the contrastive learning loss function;

[0220] In this embodiment, the image-text feature alignment module is used to map the image features and the text features based on the text-image projector to obtain corresponding projection features, shorten the distance of the corresponding projection features based on the alignment loss, and realize the alignment of the image and text features;

[0221] In this embodiment, the text design module generates multiple groups of text prompt words based on the language text model and preset constraints;

[0222] In this embodiment, the text screening module is used to screen multiple groups of text prompt words;

[0223] In this embodiment, the frozen text encoder module is used to freeze the pre-trained weights of the text encoder;

[0224] In this embodiment, the classifier is used to calculate the text prompt word that best matches the face through cosine similarity calculation and binary cross entropy, and judge the authenticity of the face according to the authenticity of the text prompt word description;

[0225] In this embodiment, the supervised training module is used to perform supervised training on three output results of the contrastive learning module, the image-text feature alignment module, and the classifier;

[0226] In this embodiment, the prediction module is used to input the face image into the trained model to obtain the true or false prediction result of the face.

[0227] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A method for liveness detection based on language-guided image feature extraction, characterized in that: The steps include: Construct a video dataset preprocessing module to preprocess the acquired video dataset for face fraud detection, capture face images, and construct a text file containing face images and corresponding authenticity labels; Construct an image encoder and a text encoder. The image encoder extracts the image features of the face image, and the text encoder extracts the text features of the text prompt word. Constructing a picture-to-picture projector and a text-to-picture projector, wherein the picture-to-picture projector projects different picture features into the same feature space, and the text-to-picture projector projects picture and text features into the same feature space; Construct a contrastive learning module. Each face image is enhanced to obtain two sub-views. The corresponding feature representation is obtained through the image encoder. The sub-views are mapped to another feature space based on the image-image projector. The contrastive learning module calculates the similarity and difference of the projection features of the two images based on the contrastive learning loss function. Construct an image-text feature alignment module, map image features and text features to obtain corresponding projection features based on the text-image projector, shorten the distance of corresponding projection features based on the alignment loss, and achieve image and text feature alignment; Build a text design module to generate multiple sets of text prompt words based on the language text model and preset constraints; Construct a text filtering module to filter multiple groups of text prompt words; Build a frozen text encoder module to freeze the pre-trained weights of the text encoder; Build a classifier, calculate the text prompt word that best matches the face through cosine similarity calculation and binary cross entropy, and judge the authenticity of the face based on the truth or falsehood of the text prompt word; By performing supervised training on the three output results of the contrastive learning module, the image-text feature alignment module, and the classifier, the face image is input into the trained model to obtain the true or false prediction result of the face.

2. The method for liveness detection based on language-guided image feature extraction according to claim 1, characterized in that: The image encoder includes a patch embedding layer, a position embedding layer, a multi-layer Transformer encoder and a linear projection layer; The patch embedding layer divides the face image into multiple patches and flattens them into a one-dimensional patch vector; The position embedding layer adds position information to each one-dimensional patch vector, preserving the spatial structure and order of the image, expressed as: z0=[x1E;x2E;...x N And]+And pos Among them, z0 represents the initial embedding feature, x i is the one-dimensional patch vector after the i-th patch is flattened, E is the linear projection matrix, E pos is the position embedding matrix; The Transformer encoder extracts global features of the initial embedded features through the self-attention mechanism; Each Transformer encoder sublayer is followed by a Layer Normalization layer, expressed as: Among them, x represents the input feature vector, μ represents the mean of each feature in the batch, σ represents the standard deviation of each feature in the batch, γ and β represent the affine parameters to be learned, and ∈ is a constant; The linear projection layer maps the output features of the Transformer encoder to the target dimension to obtain a picture feature representation.

3. The method for liveness detection based on language-guided image feature extraction according to claim 1, characterized in that: The text encoder includes a word embedding layer, a multi-layer Transformer encoder and a linear projection layer; The word embedding layer maps each word of the input text to a high-dimensional feature space and generates initial features, which are expressed as: have been i =W embed x i Among them, x i is the one-hot vector of the i-th word, W embed is the embedding matrix; The Transformer encoder extracts global features of the initial features through the self-attention mechanism; Each Transformer encoder sublayer is followed by a Layer Normalization layer, expressed as: Among them, x represents the input feature vector, μ represents the mean of each feature in the batch, σ represents the standard deviation of each feature in the batch, γ and β represent the affine parameters to be learned, and ∈ is a constant; The linear projection layer maps the output features of the Transformer encoder to the target dimension to obtain text feature representation.

4. The method for liveness detection based on language-guided image feature extraction according to claim 1, characterized in that: The network structure of the picture-picture projector and the text-picture projector is the same, including three fully connected layers, each layer contains a weight matrix and a bias vector, and performs nonlinear transformation through the ReLU activation function. The ReLU activation function is expressed as: ReLU(x)=max(0,x) Among them, x is the input feature vector, which represents the image feature vector or text feature vector; The first fully connected layer receives the input feature vector and transforms it through the ReLU activation function, setting negative values ​​to zero; The transformed output is used as the input of the second fully connected layer and processed by the ReLU activation function; The features are then mapped to the target output dimension through the third fully connected layer to obtain the final output features.

5. The method for liveness detection based on language-guided image feature extraction according to claim 1, characterized in that: Each face image is enhanced to obtain two sub-views, and the corresponding feature representation is obtained through the image encoder, which is specifically expressed as: v1=E image (View1) <h2 style=";text-align:left;direction:ltr">v2=E<h2 style=";text-align:left;direction:ltr"> image <h2 style=";text-align:left;direction:ltr"> (View2) Among them, View1 and View2 represent two subviews respectively, v1 and v2 represent the feature representations corresponding to the two subviews, and E image Represents a picture encoder; Based on the image-image projector mapping to another feature space, it is expressed as: v1′=Proj1(v1) v′2=Proj1(v2) Among them, Proj1 represents the image-to-image projector, v′1 and v′2 represent the corresponding projection features of two image features; The contrastive learning module calculates the similarities and differences between the projection features of two images based on the contrastive learning loss function. The contrastive learning loss function is expressed as: Among them, sim represents the cosine similarity between two feature vectors, τ is the temperature parameter, and B is the batch size.

6. The method for liveness detection based on language-guided image feature extraction according to claim 1, characterized in that: Based on the text-image projector, the image features and text features are mapped to obtain the corresponding projection features, which are expressed as: x′=Proj2(x) x=E image (image) t′=Proj2(t) t=E text (text) Among them, image represents image features, text represents text features, PrQj2 represents text-image projector, E image represents the picture encoder, E text represents the text encoder, x′ and t′ represent the corresponding projection features obtained by mapping the image features and text features; The distance of the corresponding projection features is shortened based on the alignment loss, which is specifically expressed as: Among them, L inter represents the L2 norm distance between two feature vectors, and B is the batch size.

7. The method for liveness detection based on language-guided image feature extraction according to claim 1, characterized in that: Generate multiple sets of text prompt words based on the language text model and preset constraints, including: Three constraints were imposed on the real and fake face texts, including: prohibiting the use of uncommon words, prohibiting the use of clauses, and prohibiting the description of character features, resolution, environmental background, and lighting; For fraudulent faces, set attack type prompts, including printing attacks, replay attacks, 3D mask attacks, and adversarial sample attacks.

8. The method for liveness detection based on language-guided image feature extraction according to claim 1, characterized in that: Construct a text filtering module to filter multiple groups of text prompt words, including: Calculate the cosine similarity between the image and each text prompt word, and use the text with the highest cosine similarity as the unique prompt word corresponding to the image; Count the matching frequency of each text and each image, and retain a set number of texts based on the matching frequency; Keep textual hints describing the attack type.

9. The method for liveness detection based on language-guided image feature extraction according to claim 1, characterized in that: The text prompt word that best matches the face is calculated by cosine similarity calculation and binary cross entropy, and the authenticity of the face is judged based on the authenticity of the text prompt word description. The specific expression is: Encode each type of text through a text encoder to obtain the embedded representation of each text, and stack them to form a new tensor; Encode the cropped face image through an image encoder to obtain the image features of the image; Normalize the image features and text features, calculate the cosine similarity between the image features and the text features, select the most similar text and determine its category. If the text describes a real face, the face image is judged as a real face; if the text describes a fake face, the face image is judged as a fake face. The true and false labels of the samples are used for supervision. The specific loss function formula is as follows: Among them, y i represents the true label of the i-th sample, 0 means the face image is fraudulent, 1 means the face image is real, y i ′ represents the predicted probability of the i-th sample, indicating the probability that the model predicts that the sample is true.

10. A liveness detection system based on language-guided image feature extraction, characterized in that: Used to implement the method for liveness detection based on language-guided image feature extraction as described in any one of claims 1 to 9, the system comprises: a video data set preprocessing module, an image encoder, a text encoder, an image-image projector, a text-image projector, an image projection feature construction module, a contrast learning module, an image-text feature alignment module, a text design module, a text screening module, a frozen text encoder module, a classifier, a supervised training module and a prediction module; The video data set preprocessing module is used to preprocess the acquired video data set for face fraud detection, intercept the face image, and construct a text file containing the face image and the corresponding authenticity label; The image encoder is used to extract image features of a face image; The text encoding is used to extract text features of text prompt words; The picture-to-picture projector is used to project different picture features into the same feature space; The text-image projector is used to project image and text features into the same feature space; The image projection feature construction module is used to perform data enhancement on each face image to obtain two sub-views, obtain the corresponding feature representation through the image encoder, and map it to another feature space based on the image-image projector to obtain the corresponding image projection feature The contrastive learning module is used to calculate the similarities and differences of the projection features of two images based on the contrastive learning loss function; The image-text feature alignment module is used to map the image features and the text features based on the text-image projector to obtain corresponding projection features, shorten the distance of the corresponding projection features based on the alignment loss, and realize image and text feature alignment; The text design module generates multiple groups of text prompt words based on the language text model and preset constraints; The text screening module is used to screen multiple groups of text prompt words; The frozen text encoder module is used to freeze the pre-trained weights of the text encoder; The classifier is used to calculate the text prompt word that best matches the face through cosine similarity calculation and binary cross entropy, and judge the authenticity of the face according to the authenticity of the text prompt word description; The supervised training module is used to perform supervised training on three output results of the contrastive learning module, the image-text feature alignment module, and the classifier; The prediction module is used to input the face image into the trained model to obtain the true or false prediction result of the face.

Citation Information

Patent Citations

  • Comparative learning-based unsupervised pre-training-fine tuning type radar target identification method

    CN115047423A

  • Living body detection network, living body detection model, living body detection method and device

    CN118196863A

  • Face living body detection method and device

    CN118314611A

  • Living body detection method and apparatus, and computer device

    WO2023109551A1

Cited By

  • Multi-modal face in-vivo detection method, medium and equipment based on image-text contrast learning

    CN119763199A