Multi-modal face anti-counterfeiting model training method and system, and face anti-counterfeiting detection method

Abstract

The present application provides a multi-modal face anti-counterfeiting model training method and system, and a face anti-counterfeiting detection method, comprising: obtaining a multi-modal face image training set; randomly discarding modal for each face image sample to simulate the scene of modal loss, determining the discarding strategy for different missing modal scenes according to the training difficulty, and performing batch-level and sample-level random modal discarding; based on the modal loss scene of the multi-modal face image and the batch-level and sample-level random discarding, determining positive and negative samples; constructing a multi-modal face anti-counterfeiting model, using the multi-modal face image under the missing modal scene and the positive and negative samples as input to obtain a face anti-counterfeiting result, comparing and training using the positive and negative samples to determine the multi-modal face anti-counterfeiting network model. The present application can be used for images collected by different modalities or modal combinations of natural light, infrared and depth cameras and other devices, adapt to different counterfeit face methods, and has the characteristics of low cost, high precision and high robustness.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, specifically to a multimodal face anti-spoofing model training method, system, and face anti-spoofing detection method. Background Technology

[0002] With the rapid development of artificial intelligence, computer vision technology is becoming increasingly popular and widely applied in people's daily lives and production activities. Among these, facial recognition technology, as a mature biometric method, has been widely used in various intelligent interaction scenarios, such as smartphone unlocking, access control systems, and financial payments. However, current facial recognition systems face many security risks, especially being vulnerable to spoofing attacks. Criminals may deceive the system by using printed photos, video playback, or 3D-printed masks. Therefore, facial anti-spoofing technology aims to distinguish genuine faces from fake ones. In its early stages, this technology primarily relied on hand-designed feature extraction algorithms to analyze facial texture, facial movement, three-dimensional structure, and image quality to determine whether a face was real. However, these traditional methods are often based on fixed prior knowledge, lacking robustness and struggling to adapt to the complex and ever-changing application environments of the real world.

[0003] With the rise of deep learning, convolutional neural networks (CNNs) with stronger feature extraction capabilities have been proposed for end-to-end learning of face anti-spoofing tasks. This involves modeling the authenticity problem as a supervised binary classification problem and using labeled neural networks to extract features. Furthermore, researchers have utilized multiple modalities, such as natural light images, infrared images, and depth images, for cross-modal information interaction and utilization to compensate for the lack of information in natural light images and improve the classification accuracy of face anti-spoofing networks. However, in real-world scenarios, due to data privacy concerns, sensor damage, and other factors, some modal images may be incomplete, leading to modality loss. The aforementioned methods struggle to perform efficient and accurate classification in scenarios with missing modalities. In recent years, training with missing modalities has gained increasing attention due to its wide range of applications. The MMANet model disclosed by Wei et al. in their 2023 paper "MMANet: Margin-aware distillation and modality-aware regularization for incomplete multimodal learning" at the Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, the mmFormer model disclosed by Zhang et al. in their 2022 paper "mmFormer: Multimodal medical transformer for incomplete multimodal learning-ing of brain tumor segmentation" at the International Conference on Medical Image Computing and Computer-Assisted Intervention, and the mmFormer model disclosed by Yu et al. in their 2024 paper "Rethinking vision transformer and masked autoencoder in multimodal face anti-spoofing" at the International Journal of Computer Vision, all use a single neural network model to train multiple different missing scenarios, but they focus on learning the general features between multiple modalities and ignore the specific features of each modality.In their 2023 paper "Multimodal prompting with missing modalities for visual recognition" published in the Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, Lee et al. published MAP, which uses a special network for each missing scenario. It focuses on learning the specific features between modalities but ignores the general features between modalities. The FM-ViT paper "FM-ViT: Flexible Modal Vision Transformers for Face Anti-Spoofing" published by Liu et al. in IEEE Transactions on Information Forensics and Security in 2023, and the MA-ViT paper "MA-ViT: Modality-Agnostic Vision Transformers for Face Anti-Spoofing" published by Liu and Liang in Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence in 2022, both propose an adapter scheme to extract and learn general features for the missing modality scenario of face anti-spoofing tasks. However, neither of the above methods can efficiently utilize general features and specific features at the same time, resulting in a sharp decline in model performance and affecting applications in real complex scenarios. Summary of the Invention

[0004] To address the shortcomings of existing technologies, the purpose of this invention is to provide a multimodal face anti-spoofing model training method, system, face anti-spoofing detection method, as well as media and terminal.

[0005] To achieve the above objectives, according to a first aspect of the present invention, a method for training a multimodal face anti-spoofing model is provided, comprising:

[0006] A multimodal face image training set is obtained, wherein each face image sample in the multimodal face image training set includes natural light images, infrared images, and depth map face images;

[0007] For each face image sample, a modality loss scenario is simulated by randomly discarding modalities. Based on the training difficulty of different modality loss scenarios, a discarding strategy for different modality loss scenarios is determined.

[0008] Random modal discarding is performed at both the batch and sample levels according to the discarding strategy;

[0009] Based on the modality loss scenario of the current multimodal face image and the random discarding at the batch level and the sample level, positive and negative samples are determined;

[0010] A multimodal face anti-spoofing model is constructed by using multimodal face images in scenarios with missing modalities and the positive and negative samples as inputs to obtain face anti-spoofing results. The multimodal face anti-spoofing model is then trained by comparing the positive and negative samples to determine the multimodal face anti-spoofing network model for face anti-spoofing detection.

[0011] Preferably, the step of simulating modality loss scenarios by randomly discarding modalities for each face image sample, and determining the discard strategy for different modality loss scenarios based on the training difficulty of different modality loss scenarios, includes:

[0012] The missing scenario is simulated by randomly discarding elements, resulting in a probability list, which represents the probability of each modality occurring.

[0013] Calculate the distribution distance between the predicted distribution for different modal missing scenarios and the predicted distribution for the full modal scenario;

[0014] The probability of different modality missing scenarios is adjusted according to the distribution distance, so that the smaller the distribution distance, the lower the probability of modality missing scenarios, and the larger the distribution distance, the higher the probability of modality missing scenarios.

[0015] We obtain an adjusted list of probabilities, which serves as the discard strategy.

[0016] Preferably, the step of performing batch-level and sample-level random modal discarding according to the discarding strategy includes:

[0017] Random discarding is performed at the batch level or at the sample level based on the probability list;

[0018] In this process, batch-level random dropping occurs when a batch consists of the same missing modality scenario, resulting in batch-level randomly dropped samples to ensure the existence of positive samples for contrastive learning; sample-level random dropping occurs when each sample consists of a different missing modality scenario, resulting in sample-level randomly dropped samples to ensure the existence of negative samples for contrastive learning.

[0019] Preferably, determining positive and negative samples based on the modality loss scenario of the current multimodal face image and the random discarding at the batch level and the sample level includes:

[0020] Search for multimodal face image samples of the same missing modality scene and the same category in the batch-level randomly discarded samples, and identify them as positive samples;

[0021] Search for multimodal face image samples with different modal missing scenarios or multimodal face image samples with the same modal missing scenario but different categories in the randomly discarded samples at the sample level, and identify them as negative samples.

[0022] Preferably, the multimodal face anti-spoofing model uses a multimodal Transformer network block as the main skeleton, and adds a modal decoupling adapter to fine-tune the main skeleton;

[0023] The implementation process of the modal decoupling adapter is as follows:

[0024] The low-frequency components of the input are extracted and analyzed using the low-pass filter in the modal decoupling adapter to determine the common characteristics of different modes;

[0025] The high-frequency components of the input are extracted and analyzed using the high-pass filter in the modal decoupling adapter to determine the unique characteristics of different modes;

[0026] The modality decoupling adapter uses multi-layer convolution to fuse general features and proprietary features.

[0027] Preferably, the step of using the positive and negative samples to comparatively train the multimodal face anti-spoofing model to determine the multimodal face anti-spoofing network model for face anti-spoofing detection includes:

[0028] The classification loss function is determined based on the cross-entropy between the predicted classification values ​​of the current missing modality face image samples and the actual values ​​of the current missing modality face image samples.

[0029] The contrast loss function is determined based on the cosine similarity between the higher-order normalized features of the positive samples and the higher-order normalized features of the current missing modality face image samples, and the cosine similarity between the higher-order normalized features of the negative samples and the higher-order normalized features of the current missing modality face image samples.

[0030] The cosine similarity between the LBP higher-order features of the current missing modality face image sample and the LBP higher-order features of the positive sample is used to adjust the ratio loss function to remove face structure information.

[0031] The classification loss function and the contrastive loss function are weighted to obtain the overall loss function, specifically:

[0032] L = L cls +λ1·L con

[0033] Where L represents the overall loss function, L cls Let L represent the classification loss function. con Let λ1 represent the contrastive loss function, and let λ1 represent the weighting weight of the contrastive loss function.

[0034] The model parameters are updated using the backpropagation algorithm based on the overall loss function.

[0035] According to a second aspect of the present invention, a training system for a multimodal face anti-spoofing model is provided, comprising:

[0036] Face image acquisition module: acquires a multimodal face image training set, wherein each face image sample in the multimodal face image training set includes a natural light image, an infrared image, and a depth map face image;

[0037] Missing Scene Simulation Module: Simulates a missing modality scenario by randomly discarding modalities for each face image sample. Based on the training difficulty of different missing modalities, a discarding strategy is determined for different missing modalities. Random modalities are discarded at both the batch and sample levels according to the discarding strategy.

[0038] Positive and negative sample acquisition module: Determines positive and negative samples based on the modality loss scenario of the current multimodal face image and the random discarding at the batch level and the sample level;

[0039] Model training module: Constructs a multimodal face anti-spoofing model, using multimodal face images in missing modality scenarios and the positive and negative samples as input to obtain face anti-spoofing results, and uses the positive and negative samples to compare and train the multimodal face anti-spoofing model to determine the multimodal face anti-spoofing network model for face anti-spoofing detection.

[0040] According to a second aspect of the present invention, a training system for a multimodal face anti-spoofing model is provided, comprising:

[0041] Face image acquisition module: acquires a multimodal face image training set, wherein each face image sample in the multimodal face image training set includes three modal images: natural light image, infrared image and depth map face image acquired by the acquisition;

[0042] Missing Scene Simulation Module: Simulates a missing modality scenario by randomly discarding modalities from each face image sample. Based on the training difficulty of different missing modalities, a discarding strategy is determined for different missing modalities. Random modalities are discarded at both the batch and sample levels according to the discarding strategy.

[0043] Positive and negative sample acquisition module: Determines positive and negative samples based on the modality loss scenario of the current multimodal face image and the random discarding at the batch level and the sample level;

[0044] Model training module: Constructs a multimodal face anti-spoofing model, using multimodal face images in missing modality scenarios and the positive and negative samples as input to obtain face anti-spoofing results, and uses the positive and negative samples to compare and train the multimodal face anti-spoofing model to determine the multimodal face anti-spoofing network model for face anti-spoofing detection.

[0045] According to a third aspect of the present invention, a multimodal face anti-spoofing detection method is provided, comprising:

[0046] The input face image can be any one modality or a combination of multiple modalities of a natural light image, an infrared image, and a depth map image.

[0047] Convolutional networks are applied to each modality of face image to obtain block embeddings of multimodal face images;

[0048] The block is embedded into the multimodal face anti-spoofing model to obtain the face anti-spoofing detection result; wherein, the multimodal face anti-spoofing model is obtained by training using any of the multimodal face anti-spoofing model training methods or the multimodal face anti-spoofing model training systems described above.

[0049] According to a fourth aspect of the present invention, a terminal is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, can be used to perform any of the methods described herein, or to run the system described herein.

[0050] According to a fifth aspect of the invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, can be used to perform any of the methods described herein, or to run the system described herein.

[0051] Compared with the prior art, the embodiments of the present invention have at least one of the following beneficial effects:

[0052] The training method of the multimodal face anti-spoofing model in this embodiment of the invention uses a large amount of missing modality data to train the multimodal face anti-spoofing network model, which has strong generalization ability and strong adaptability to different modality missing scenarios, and achieves low-cost and highly robust face anti-spoofing recognition.

[0053] The training method of the multimodal face anti-spoofing model in this embodiment of the invention uses positive and negative samples for training, uses contrastive loss to constrain samples under different missing modalities, and weights the LBP features in the contrastive loss to eliminate the influence of face structural features on face anti-spoofing. It can generalize on different face structures and can be applied to any face scene, achieving high-precision anti-spoofing detection under different face image shooting conditions, different face forgery attack methods, and different faces.

[0054] The multimodal face anti-spoofing detection method in this invention introduces a modal decoupling adapter to explicitly model and analyze general and proprietary features of different modalities. This enhances the information interaction capability of different modalities and the information utilization rate of the multimodal face anti-spoofing network model. It is applicable to face anti-spoofing based on natural light, infrared, depth and other acquisition devices, and effectively improves the accuracy of face anti-spoofing detection when using any one, two or all three devices. Attached Figure Description

[0055] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0056] Figure 1 This is a flowchart of a multimodal face anti-spoofing model training method according to an embodiment of the present invention;

[0057] Figure 2 This is a schematic diagram of the structure of a multimodal face anti-spoofing model training method according to an embodiment of the present invention;

[0058] Figure 3 This is a flowchart of model training for a multimodal face anti-spoofing network in a preferred embodiment of the present invention;

[0059] Figure 4 This is a visualization of the face anti-counterfeiting recognition result in a specific embodiment of the present invention.

[0060] Figure 5 This is a network model block diagram of the modal decoupling adapter in a preferred embodiment of the present invention;

[0061] Figure 6 This is a flowchart of a multimodal face anti-counterfeiting detection method according to an embodiment of the present invention;

[0062] Figure 7 This is a block diagram of a training system for a multimodal face anti-spoofing model according to an embodiment of the present invention. Detailed Implementation

[0063] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention. These all fall within the scope of protection of the present invention.

[0064] In one embodiment of the present invention, a method for training a multimodal face anti-spoofing model is provided, such as... Figure 1 As shown, the specific steps include the following:

[0065] S11, Obtain a multimodal face image training set. Each face image sample in the multimodal face image training set includes three modal images: natural light image, infrared image, and depth map face image.

[0066] S12, the modality missing scenario is simulated by randomly discarding modalities from each face image sample in S11. The discarding strategy for different modality missing scenarios is determined according to the training difficulty of different modality missing scenarios.

[0067] S13, perform random modal discarding at the batch and sample levels according to the discarding strategy in S12;

[0068] S14, determine positive and negative samples based on the modality loss scenario of the current multimodal face image and the random discarding at the batch level and sample level in S13;

[0069] S15, construct a multimodal face anti-spoofing model, using multimodal face images in missing modality scenarios and positive and negative samples as input to obtain face anti-spoofing results, and use positive and negative samples to compare and train the multimodal face anti-spoofing model to determine the multimodal face anti-spoofing network model for face anti-spoofing detection.

[0070] The above embodiments use a large amount of missing modality data to train a multimodal face anti-spoofing network model, which has strong generalization ability and strong adaptability to different modality missing scenarios, achieving low-cost and highly robust face anti-spoofing recognition.

[0071] In a preferred embodiment of the present invention, the multimodal face image training set in S11 includes multimodal face image samples collected by different acquisition devices or environments, and each face image sample includes three modalities: natural light image, infrared image and depth map image.

[0072] In some specific embodiments, the determined face image sample {X i ,Y i}, where X i The input image X has three modalities. i = Yi The labels are used for binary classification to predict whether a face is real or fake.

[0073] In a preferred embodiment of the present invention, a preferred implementation scheme for step S12 is provided, specifically:

[0074] The method of randomly discarding modes typically uses a mask Δ = {δ1, δ2, δ3} to mask the three modal inputs, where δ i =0 indicates that the mode is discarded, δ i =1 indicates that the mode is retained. This random discarding method yields a probability list showing the probability of each mode combination occurring.

[0075] When calculating the training difficulty of different modality missing scenarios, some preferred embodiments calculate the distribution distance (MSE) between the predicted distributions of different modality missing scenarios and the predicted distributions of the full modality missing scenario. The distribution distance between the missing modality and the full modality represents the training difficulty of this missing scenario; the larger the distribution distance, the more difficult it is to apply the model of the full modality missing scenario. Therefore, the probability allocation in the probability list of different modality missing scenarios is adjusted according to the distribution distance, so that the probability of missing modalities with smaller distribution distances is lower, and the probability of missing modalities with larger distribution distances is higher. In this way, the training process of different missing modalities can be balanced, achieving more training for missing modalities that are difficult to train, and easier training for missing modalities. of Missing modalities result in insufficient training.

[0076] In a preferred embodiment of the present invention, S13 is implemented to perform random modal discarding at the batch level and sample level according to the discarding strategy set in S12.

[0077] In this process, batch-level random dropping occurs when a batch consists of the same missing modality scenario, resulting in batch-level randomly dropped samples to ensure the existence of positive samples for contrastive learning; sample-level random dropping occurs when each sample consists of a different missing modality scenario, resulting in sample-level randomly dropped samples to ensure the existence of negative samples for contrastive learning.

[0078] In a preferred embodiment of the present invention, step S14 is performed to determine positive and negative samples based on the modality missing scenario of the current multimodal face image and the random discarding at the batch level and sample level.

[0079] Among them, the current multimodal face image samples are used for face anti-spoofing recognition.

[0080] In some preferred embodiments, multimodal face image samples of the same missing modality and category are searched for in batch-level randomly discarded samples and identified as positive samples; multimodal face image samples of different missing modalities or multimodal face images of the same missing modality but different categories are searched for in sample-level randomly discarded samples and identified as negative samples. Specifically, multimodal face image samples with the same mask Δ are considered as face image samples under the same missing modality scenario.

[0081] Furthermore, in some specific embodiments, for the current face image sample x, the category label y, the normalized high-order feature z corresponding to the current face image sample x, and the mask Δ, a set of samples in the multimodal face image training set belonging to the same missing modality scene as the current face image sample x is constructed:

[0082] b + ={z i},y i =y,Δ i =Δ

[0083] Among them, y i =y represents the set of samples in the face image training set that belong to the same category as the current face image sample x, Δ i =Δ represents the set of samples in the face image training set that belong to the same missing modality scene as the current face image sample x.

[0084] Construct a set of samples that do not belong to the same missing modality scene as the current face image sample:

[0085] b - ={z i},Δ i ≠Δ

[0086] Where, Δ i ≠Δ represents the set of samples in the face image training set that belong to different missing modalities than the current face image sample x.

[0087] Construct a set of samples that do not belong to the same category as the current face image sample:

[0088] b - ={z i},y i ≠y

[0089] Among them, y i ≠y represents the set of samples in the face image training set that do not belong to the same category as the current face image sample x.

[0090] In a preferred embodiment of the present invention, a preferred implementation of S15 is provided. In a multimodal face anti-spoofing model for scenarios lacking a modality, a multimodal Transformer network block is used as the main skeleton, and a modality decoupling adapter is added to fine-tune the skeleton. Specifically, the modality decoupling adapter is placed in parallel on one side of the multi-head attention layer and the forward propagation layer of the Transformer network block. For example... Figure 5 As shown, the implementation process in the modal decoupling adapter is as follows:

[0091] S21, The low-frequency components of the input are extracted and analyzed by the low-pass filter in the mode decoupling adapter to determine the general characteristics of different modes;

[0092] S22, the high-frequency components of the input are extracted and analyzed by the high-pass filter in the mode decoupling adapter to determine the unique characteristics of different modes;

[0093] S23 utilizes multi-layer convolution in the modal decoupling adapter to deeply fuse general features and proprietary features.

[0094] Based on the above embodiments, another embodiment of the present invention provides a complete multimodal face anti-spoofing model, which specifically consists of convolutional layers, multi-layer multimodal Transformer network blocks finely tuned by a modal decoupling adapter, and a classifier. The processing procedure of this multimodal face anti-spoofing model is as follows:

[0095] First, multimodal face images in missing modality scenarios are used as input, and a convolutional layer is applied to each modality as a block embedding network to obtain the block embedding of the multimodal face images.

[0096] Then, based on the block embedding of the multimodal face image, multimodal features are obtained through multiple layers of multimodal Transformer network blocks and modal decoupling adapter;

[0097] Finally, a face anti-counterfeiting classifier is used to classify the multimodal features to obtain the face anti-counterfeiting detection results.

[0098] Furthermore, the process of acquiring multimodal features is as follows:

[0099] S31, the multi-head self-attention layer in the multimodal Transformer network block is used to extract and fuse intermodal features of the input image, and to extract cross-modal interaction features;

[0100] S32 utilizes the forward propagation layer in the multimodal Transformer network block to further fuse cross-modal interaction features.

[0101] Meanwhile, the above S21-S23 processes are implemented in the modal decoupling adapter, which runs parallel to the multi-head self-attention layer and the forward propagation layer. Based on the distinguishability of general features and specific features in the frequency domain, the modal general features and specific features are explicitly modeled, analyzed, and uniformly utilized. This enhances the information interaction capability of different modalities and the information utilization rate of the multimodal face anti-spoofing network model. It is applicable to face anti-spoofing based on natural light, infrared, depth and other acquisition devices, and effectively improves the accuracy of face anti-spoofing detection when using any one, two or all three devices.

[0102] Furthermore, in a preferred embodiment, positive and negative samples are used to comparatively train the multimodal face anti-spoofing model to determine the multimodal face anti-spoofing network model, such as... Figure 2 As shown, the training process of the face anti-spoofing network under the preset modality missing scenario includes batch-level random discarding, sample-level random discarding, feature extraction from the benchmark model (multimodal face anti-spoofing model), and contrastive loss constraints.

[0103] The baseline model uses three modalities (natural light image, infrared image, and depth map face image) as input and consists of three parts: a convolutional network layer, a feature extractor, and a classifier. The initial parameters of the network are randomly generated and optimized according to the contrastive loss function and the classification prediction loss function, specifically:

[0104] L = L cls +λ1·L con

[0105] L represents the overall loss function, L cls Let L represent the classification loss function. con Let λ represent the contrastive loss function, and λ1 represent the weighting weight of the contrastive loss function.

[0106] Furthermore, in some specific embodiments, the classification loss function is determined based on the cross-entropy between the classification prediction value of the current missing modality face image sample and the actual value of the current missing modality face image sample.

[0107] Furthermore, in some specific embodiments, such as Figure 3 The diagram shows a flowchart of a method for training a pre-defined multimodal face anti-spoofing model. A contrastive loss function is used to constrain the similarity between the current face sample and positive and negative samples, optimizing the parameters of the pre-defined multimodal face anti-spoofing network model to determine the model, including steps S31 to S32.

[0108] S31, input the randomly discarded multimodal face images into the multimodal face anti-spoofing network model and output the face coding features;

[0109] S32, the contrastive loss function is used to constrain the similarity between the current face sample and positive and negative samples, optimize the preset parameters of the multimodal face anti-spoofing network model, and determine the multimodal face anti-spoofing network model.

[0110] The consistency between the normalized higher-order features of the current face image sample and the normalized higher-order features of the positive sample is constrained by the contrastive loss function.

[0111] The contrastive loss function constrains the dissimilarity between the normalized higher-order features of the current face image sample and the normalized higher-order features of the negative sample.

[0112] In some specific embodiments, the high-order features corresponding to the current face image sample are normalized:

[0113] z i =z i‘ / ||z i’ ||2

[0114] Among them, z i z represents the high-order feature corresponding to the current face image sample after normalization. i’ This represents the high-order features corresponding to the face image samples before normalization.

[0115] Based on the normalized high-order features corresponding to the current face image sample and the normalized high-order features of the positive and negative samples, the contrast loss function is determined:

[0116]

[0117] Where N is the number of training samples. The number of positive samples. τ represents the number of negative samples, and τ is a temperature coefficient used to control the effect of contrast loss. It is the normalized high-order features corresponding to the current face image sample and the cosine similarity of the positive sample. It is the normalized high-order features corresponding to the current face image sample and the cosine similarity of the negative sample. The weighted term is calculated based on the similarity between the normalized high-order features corresponding to the current face image sample and the LBP features of the positive sample, and is used to remove irrelevant face structural features.

[0118] Based on the contrast loss function, gradient descent is used to optimize the parameters of the multimodal face anti-spoofing network until the parameters of the multimodal face anti-spoofing network converge, thus forming a multimodal face anti-spoofing network model.

[0119] The above embodiments eliminate the influence of facial structural features on face anti-spoofing, can be generalized on different facial structures, and can be applied to any face scene, achieving high-precision anti-spoofing detection under different face image shooting conditions, different face spoofing attack methods, and different faces.

[0120] Based on the same inventive concept, other embodiments of the present invention provide a training system for a multimodal face anti-spoofing model, such as... Figure 7 As shown, it includes:

[0121] Face image acquisition module 110: acquires a multimodal face image training set, wherein each face image sample in the multimodal face image training set includes three modal images: natural light image, infrared image and depth map face image acquired by the acquisition;

[0122] Missing Scene Simulation Module 120: Simulates a missing modality scenario by randomly discarding modalities from each face image sample; determines a discarding strategy for different missing modalities based on the training difficulty of different missing modalities; and performs batch-level and sample-level random modal discarding according to the discarding strategy.

[0123] Positive and negative sample acquisition module 130: Determines positive and negative samples based on the modality missing scenario of the current multimodal face image and the random discarding at the batch level and the sample level;

[0124] Model training module 140: Constructs a multimodal face anti-spoofing model, using multimodal face images in missing modality scenarios and the positive and negative samples as inputs to obtain face anti-spoofing results, and uses the positive and negative samples to compare and train the multimodal face anti-spoofing model to determine the multimodal face anti-spoofing network model for face anti-spoofing detection.

[0125] The specific implementation techniques of each module / unit in the above examples of the present invention can be referred to the steps of the training method of the multimodal face anti-spoofing model in the above embodiments, and will not be repeated here.

[0126] Based on the same inventive concept, another embodiment of the present invention provides a multimodal face anti-spoofing detection method, such as... Figure 6 As shown, the process includes the following steps: acquiring an input face image, which is an image of any modality or a combination of multiple modalities, such as a natural light image, an infrared image, and a depth map image; inputting the input face image into a multimodal face anti-spoofing model to obtain the face anti-spoofing detection result; wherein, the multimodal face anti-spoofing model is trained using any of the training methods or training systems for multimodal face anti-spoofing models.

[0127] To better illustrate the technology of this invention, another preferred embodiment of this invention provides a complete process for a multimodal face anti-spoofing detection method, such as... Figure 6 As shown, specifically:

[0128] S41, using multimodal face images in missing modality scenarios as input, the block embedding of multimodal face images is obtained. Using multimodal face images in missing modality scenarios as input, a convolutional layer is applied to each modality as a block embedding network to obtain the block embedding of multimodal face images.

[0129] S42, based on the block embedding of the multimodal face image, multimodal features are obtained through multiple layers of multimodal Transformer network blocks and modal decoupling adapter.

[0130] The multimodal Transformer network block includes:

[0131] Multi-head self-attention layers in the multimodal Transformer network block are used for feature extraction and fusion between modalities to extract cross-modal interaction features;

[0132] Further fusion of cross-modal interaction features is achieved by utilizing the forward propagation layer in the multimodal Transformer network block;

[0133] Low-frequency components are extracted and analyzed using a low-pass filter in a modal decoupling adapter to determine the common characteristics of different modes;

[0134] High-frequency components are extracted and analyzed using a high-pass filter in a modal decoupling adapter to determine the unique characteristics of different modes;

[0135] The modal decoupling adapter utilizes multi-layer convolution to deeply fuse general features and specific features.

[0136] S43 uses a face anti-counterfeiting classifier to classify multimodal features and obtain face anti-counterfeiting detection results.

[0137] The technical solutions described in the above embodiments perform data augmentation processing on face image samples in the multimodal face image training set; simulate actual modality missing scenarios, and adaptively determine the discard strategy for different missing modality scenarios based on the training difficulty of different missing modality scenarios to balance the model training process; use positive and negative samples for comparative learning to eliminate face features that affect face generalization and enhance the learning ability for different modality missing scenarios; introduce a modality decoupling adapter to explicitly model and analyze modality general features and proprietary features, thereby enhancing the information utilization rate of the multimodal face anti-spoofing network model; and train the multimodal face anti-spoofing network model with a large amount of missing modality data, resulting in strong generalization and strong adaptability to different modality missing scenarios, achieving low-cost and highly robust face anti-spoofing recognition.

[0138] To verify the feasibility and effectiveness of the multimodal face anti-spoofing model training method, system, and face anti-spoofing detection method in the above embodiments, some specific embodiments of the present invention collected and acquired multimodal face anti-spoofing datasets WMCA, CAISA-SURF, and CASIA-SURF CeFA with different scenarios and face spoofing attack types. Each dataset contains natural light images, infrared images, and depth map face images. Among them, WMCA consists of 1941 video data collected from 72 people, including 2D and 3D face spoofing attack types; CAISA-SURF consists of 21000 video data collected from 1000 people, including two face spoofing attack types: printing and cropping; CASIA-SURF CeFA consists of 23538 video data collected from 1607 people, including printing, playback, and some 3D face spoofing attack types.

[0139] The multimodal face anti-spoofing model training method was experimentally validated on three datasets: WMCA, CAISA-SURF, and CASIA-SURF CeFA. Three testing protocols were used: (1) the training and test sets were the same dataset with the same face attack type; (2) the training and test sets were different datasets with the same face attack type; and (3) the training and test sets were the same dataset with different face attack types. Each testing protocol was tested on different missing modalities to evaluate its classification accuracy.

[0140] Experiments were conducted on three datasets: WMCA, CAISA-SURF, and CASIA-SURF CeFA. The model evaluation metric included ACER, which represents the average classification error rate, i.e., the average of the false acceptance rate and the false rejection rate. The lower the ACER, the better the model's performance. Specifically, the ACER on the CAISA-SURF and CASIA-SURF CeFA datasets was calculated using the EER of the validation set, while the ACER on the WMCA dataset was calculated using the BPCER of the validation set at 1%. Here, EER is the equal error rate, and BPCER is the misclassification rate of real face samples.

[0141] The specific comparison results are shown in Tables 1, 2, and 3 below. Table 1 shows the comparison of the classification accuracy of the first test protocol on the WMCA and CAISA-SURF datasets of an embodiment of the present invention. Table 1 shows that the multimodal face anti-spoofing network model (MMA-FAS) of the present invention greatly improves the classification performance in seven missing modality scenarios, indicating that the multimodal face anti-spoofing network model of the embodiment of the present invention can work normally in different missing modality scenarios, overcoming the problem of poor model generalization caused by modality missingness.

[0142] Table 2 shows a comparison of the classification accuracy results of an embodiment of the present invention trained on the WMCA dataset and tested on the CAISA-SURF dataset to perform a second test protocol evaluation. Table 2 shows that the multimodal face anti-spoofing network model (MMA-FAS) of the embodiment of the present invention can still achieve the best classification performance when encountering dataset distribution shift, and its ACER is reduced by 1.83% compared with AMA.

[0143] Table 3 shows a comparative evaluation of the classification accuracy of a third test protocol across face attack types on the WMCA and CASIA-SURF CeFA datasets, based on an embodiment of the present invention. Table 3 demonstrates that the Multimodal Face Anti-spoofing Network Model (MMA-FAS) of this embodiment still achieves excellent classification performance when encountering unknown face attack types, exhibiting strong generalization ability.

[0144]

[0145] Table 1

[0146]

[0147] Table 2

[0148]

[0149] Table 3

[0150] Figure 4 This is a visualization of the face anti-counterfeiting recognition result in a specific embodiment. For example... Figure 4 As shown, the multimodal face anti-spoofing method of this invention focuses on the facial region to predict the real face, and can use its unique information in each modality to perform comprehensive analysis and judgment of face anti-spoofing.

[0151] Through the above technical solutions, the multimodal face anti-spoofing method for dealing with missing modal scenarios proposed in this invention can achieve a lower average error rate in various scenarios with different missing modal scenarios and with the same or different distributions of test set data and training set data. This shows that the multimodal face anti-spoofing method of this invention can learn the corresponding features with better out-of-distribution generalization according to different missing modal scenarios, thus having a wider range of application scenarios.

[0152] Based on the same inventive concept, in other embodiments of the present invention, a terminal is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it can be used to perform the above-described method or to run the above-described system.

[0153] Optionally, the memory is used to store programs; the memory may include volatile memory, such as random-access memory (RAM), such as static random-access memory (SRAM), double data rate synchronous dynamic random-access memory (DDR SDRAM), etc.; the memory may also include non-volatile memory, such as flash memory. The memory is used to store computer programs (such as application programs, functional modules, etc. that implement the above methods), computer instructions, etc., and the above-mentioned computer programs, computer instructions, etc., may be partitioned and stored in one or more memories.

[0154] A processor is used to execute a computer program stored in memory to implement the various steps of the methods involved in the above embodiments. For details, please refer to the relevant descriptions in the preceding method embodiments.

[0155] The processor and memory can be separate structures or integrated structures. When the processor and memory are separate structures, they can be coupled together via a bus.

[0156] Based on the same inventive concept, in other embodiments of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, can be used to perform the above-described method or to run the above-described system.

[0157] Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transfer of computer programs from one place to another. Storage media can be any available medium accessible to a general-purpose or special-purpose computer. An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and storage medium can reside in an ASIC. Alternatively, the ASIC can reside in a user device. Of course, the processor and storage medium can also exist as discrete components in a communication device.

[0158] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0159] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0160] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0161] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0162] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A training method for a multimodal face anti-spoofing model, characterized in that, include: A multimodal face image training set is obtained, wherein each face image sample in the multimodal face image training set includes natural light images, infrared images, and depth map face images; For each face image sample, a modality loss scenario is simulated by randomly discarding modalities. Based on the training difficulty of different modality loss scenarios, a discarding strategy for different modality loss scenarios is determined. Random modal discarding is performed at both the batch and sample levels according to the discarding strategy; Based on the modality loss scenario of the current multimodal face image and the random discarding at the batch level and the sample level, positive and negative samples are determined; A multimodal face anti-spoofing model is constructed by using multimodal face images in scenarios with missing modalities and the positive and negative samples as inputs to obtain face anti-spoofing results. The multimodal face anti-spoofing model is then trained by comparing the positive and negative samples to determine the multimodal face anti-spoofing network model for face anti-spoofing detection.

2. The training method for the multimodal face anti-spoofing model according to claim 1, characterized in that, The process of randomly discarding modalities for each face image sample simulates a modality loss scenario. Based on the training difficulty of different modality loss scenarios, a discarding strategy is determined for each scenario, including: The missing scenario is simulated by randomly discarding elements, resulting in a probability list, which represents the probability of each modality occurring. Calculate the distribution distance between the predicted distribution for different modal missing scenarios and the predicted distribution for the full modal scenario; The probability of different modality missing scenarios is adjusted according to the distribution distance, so that the smaller the distribution distance, the lower the probability of modality missing scenarios, and the larger the distribution distance, the higher the probability of modality missing scenarios. We obtain an adjusted list of probabilities, which serves as the discard strategy.

3. The training method for the multimodal face anti-spoofing model according to claim 2, characterized in that, The step of performing batch-level and sample-level random modal discarding according to the discarding strategy includes: Random discarding is performed at the batch level or at the sample level based on the probability list; In this process, batch-level random dropping occurs when a batch consists of the same missing modality scenario, resulting in batch-level randomly dropped samples to ensure the existence of positive samples for contrastive learning; sample-level random dropping occurs when each sample consists of a different missing modality scenario, resulting in sample-level randomly dropped samples to ensure the existence of negative samples for contrastive learning.

4. The training method for the multimodal face anti-spoofing model according to claim 1, characterized in that, Based on the modality loss scenario of the current multimodal face image and the random discarding at the batch level and the sample level, positive and negative samples are determined, including: Search for multimodal face image samples of the same missing modality scene and the same category in the batch-level randomly discarded samples, and identify them as positive samples; Search for multimodal face image samples with different modal missing scenarios or multimodal face image samples with the same modal missing scenario but different categories in the randomly discarded samples at the sample level, and identify them as negative samples.

5. The training method for the multimodal face anti-spoofing model according to claim 1, characterized in that, The multimodal face anti-spoofing model uses a multimodal Transformer network block as the main skeleton, and adds a modal decoupling adapter to fine-tune the main skeleton; The implementation process of the modal decoupling adapter is as follows: The low-frequency components of the input are extracted and analyzed using the low-pass filter in the modal decoupling adapter to determine the common characteristics of different modes; The high-frequency components of the input are extracted and analyzed using the high-pass filter in the modal decoupling adapter to determine the unique characteristics of different modes; The modality decoupling adapter uses multi-layer convolution to fuse general features and proprietary features.

6. The training method for the multimodal face anti-spoofing model according to claim 1, characterized in that, The step of using the positive and negative samples to comparatively train the multimodal face anti-spoofing model, and determining the multimodal face anti-spoofing network model for face anti-spoofing detection, includes: The classification loss function is determined based on the cross-entropy between the classification prediction value of the current missing modality face image sample and the actual value of the current missing modality face image sample; The contrast loss function is determined based on the cosine similarity between the higher-order normalized features of the positive samples and the higher-order normalized features of the current missing modality face image samples, and the cosine similarity between the higher-order normalized features of the negative samples and the higher-order normalized features of the current missing modality face image samples. The cosine similarity between the LBP higher-order features of the current missing modality face image sample and the LBP higher-order features of the positive sample is used to adjust the ratio loss function to remove face structure information. The classification loss function and the contrastive loss function are weighted to obtain the overall loss function, specifically: L=L cls +λ1·L con Where L represents the overall loss function, L cls Let L represent the classification loss function. con Let λ1 represent the contrastive loss function, and let λ1 represent the weighting weight of the contrastive loss function. The model parameters are updated using the backpropagation algorithm based on the overall loss function.

7. A training system for a multimodal face anti-spoofing model, characterized in that, include: Face image acquisition module: acquires a multimodal face image training set, wherein each face image sample in the multimodal face image training set includes a natural light image, an infrared image, and a depth map face image; Missing Scene Simulation Module: Simulates a missing modality scenario by randomly discarding modalities for each face image sample. Based on the training difficulty of different missing modalities, a discarding strategy is determined for different missing modalities. Random modalities are discarded at both the batch and sample levels according to the discarding strategy. Positive and negative sample acquisition module: Determines positive and negative samples based on the modality loss scenario of the current multimodal face image and the random discarding at the batch level and the sample level; Model training module: Constructs a multimodal face anti-spoofing model, using multimodal face images in missing modality scenarios and the positive and negative samples as input to obtain face anti-spoofing results, and uses the positive and negative samples to compare and train the multimodal face anti-spoofing model to determine the multimodal face anti-spoofing network model for face anti-spoofing detection.

8. A multimodal face anti-spoofing detection method, characterized in that, include: Acquire an input face image, wherein the input face image is an image of any one modality or a combination of multiple modalities of a natural light image, an infrared image, and a depth map image; The input face image is input into the multimodal face anti-spoofing model to obtain the face anti-spoofing detection result; The multimodal face anti-spoofing model is obtained by training the multimodal face anti-spoofing model training method according to any one of claims 1-6 or the multimodal face anti-spoofing model training system according to claim 7.

9. A terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it can be used to perform the method of any one of claims 1-6 and 8, or to run the system of claim 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program can be used to perform the method of any one of claims 1-6 and 8, or to run the system of claim 7.

Images