A robust face spoofing attack detection method suitable for multi-modal instance misaligned data

Abstract

The application discloses a robust face spoofing attack detection method suitable for multi-modal instance misalignment data, and proposes an instance aggregation modal alignment learning framework which is composed of three technologies of class center aggregation, gradient enhancement and boundary relaxation. By independently introducing class center constraints in each modal, the same class instance features are forced to be aggregated to a shared representation, and the distance between different class features is enlarged. A gradient enhancement factor is introduced to dynamically adjust the optimization strength, so that the problem of gradient disappearance in the later aggregation stage is solved. And by using the boundary relaxation mechanism, the constraint is weakened after the features reach the similarity threshold, and the necessary instance diversity is reserved. The application breaks through the limitation that multi-modal learning must rely on expensive instance-level paired data, greatly reduces the data construction cost, fundamentally alleviates the feature deviation between training and reasoning, and significantly improves the robustness and generalization ability of the system in the cross-race and unknown attack scene.

Description

Technical Field

[0001] This invention belongs to the field of face recognition technology, and in particular relates to a robust face spoofing attack detection method suitable for multimodal instance misalignment data. Background Technology

[0002] In recent years, facial recognition technology has made significant progress in both algorithm performance and application scale, and has been widely deployed in high-security scenarios such as mobile payment, remote account opening, and identity verification. However, relying solely on facial recognition models is insufficient to effectively defend against presentation attacks such as photo replay, screen capture, and video playback. Therefore, in practical systems, in addition to identity verification, it is also necessary to simultaneously determine whether the input face comes from a real, living individual. In other words, facial deception (FAS) has become an indispensable component of ensuring system security.

[0003] In existing research, most FAS methods are based on single-modal RGB data for modeling. Although these methods have advantages in terms of deployment cost and engineering feasibility, their ability to characterize geometric structures and physiological properties is limited by RGB imaging, resulting in significant limitations in dealing with high-fidelity replay attacks or 3D printing attacks. Furthermore, the model's generalization ability is easily affected by environmental changes and the diversity of attack methods. To alleviate these problems, in recent years, research has gradually introduced auxiliary modalities such as depth maps and infrared maps to improve the accuracy and robustness of face anti-spoofing detection through multimodal feature complementarity.

[0004] However, multimodal face deception prevention methods face a key constraint in practical applications: the difficulty in obtaining high-quality, instance-aligned multimodal training data. An ideal multimodal face spoofing (FAS) training process requires simultaneously acquiring RGB, depth, and infrared multimodal liveness and attack samples corresponding to the same physical instance. This not only relies on expensive multi-sensor acquisition equipment but also necessitates a strictly synchronized acquisition process and extensive manual annotation, resulting in extremely high data construction costs. This significantly limits the scalability of multimodal methods in large-scale and consumer-grade scenarios.

[0005] Against this backdrop, a more realistic and feasible direction is to utilize existing, independently collected unimodal face anti-spoofing data to construct a training set for multimodal learning. In this setup, data from different modalities only share semantic labels at the category level (e.g., liveness or attack), but do not correspond to the same specific face instance, resulting in multimodal training data with misaligned modal instances. While this setup offers significant advantages in data acquisition, it also introduces new learning challenges: during training, features from different modalities come from different individuals; however, during actual inference, multimodal inputs are often collected synchronously and aligned with instances. This leads to a significant difference in the joint feature distribution between training and inference phases, making traditional multimodal fusion strategies prone to severe feature shifts, resulting in performance degradation and even failing to surpass unimodal baseline models.

[0006] Existing research on cross-modal misalignment mainly focuses on two types of methods: one type restores the correspondence between modalities through explicit realignment mechanisms (such as optimal transmission, dynamic temporal warping, offset estimation, or deformable convolution); the other type leverages the global modeling capabilities of self-attention mechanisms to alleviate the mismatch between modalities without explicit alignment. However, these methods generally imply an important premise: that data from different modalities are alignable at the instance level, or at least originate from the same physical object. This assumption is reasonable in tasks such as video-to-caption alignment or cross-view perception, but it does not hold true in the multimodal instance misalignment face anti-spoofing scenario that this paper focuses on, and therefore cannot be directly applied.

[0007] In summary, how to learn cross-modal representations with good generalization ability to instance-aligned inputs during the inference stage under training data conditions where modal instances are completely misaligned is a key issue that has not yet been fully studied in the current field of multimodal face anti-spoofing.

[0008] To address this issue, this invention proposes a robust face spoofing attack detection method for multimodal instance misalignment data. This method effectively alleviates the distribution offset problem during the training and inference stages without requiring instance-level cross-modal pairing, and mines complementary information between different modalities at the category discrimination level, thereby significantly improving the robustness and practicality of multimodal face anti-spoofing systems in real-world applications. Summary of the Invention

[0009] To address the issue of insufficient robustness of existing multimodal face deception detection methods in scenarios where training samples from different modalities cannot be matched one-to-one at the instance level, leading to significant differences in the joint feature distribution between the training and inference stages, this invention proposes an instance-aggregated modality alignment learning framework. This framework, without requiring instance-level cross-modal pairing, effectively narrows the distribution gap between the training and inference stages by constraining and regulating the feature distribution, thereby improving the stability and generalization ability of the multimodal face deception detection system.

[0010] The specific technical solution of the present invention is as follows:

[0011] A robust face spoofing attack detection method suitable for multimodal instance misalignment data includes the following steps:

[0012] Step 1: Construct instance misaligned sample pairs: Randomly pair face image samples from different modalities only within the same category based on the category label, without requiring them to correspond to the same face instance;

[0013] Step 2: Obtain the feature representation of the modality: For each modality, use an independent feature encoder to extract features from the instance misaligned sample pairs constructed in Step 1, and obtain the feature representation:

[0014] Step 3: Class center aggregation constraint for modality representation: For the feature representation of the misaligned sample pairs obtained in Step 2, class center aggregation loss constraint is applied to each modality to force intra-class features to be highly compact and inter-class features to be clearly separated, so that the features of different instances of the same class tend to share the class center representation.

[0015] Step 4: Gradient enhancement for center aggregation constraints: An adjustable gradient enhancement factor is introduced into the class center aggregation loss function to dynamically increase the loss response strength of sample pairs that are close to the optimization target, so that they can still provide effective backpropagation signals for the feature encoder in the later stages of model training.

[0016] Step 5: Boundary relaxation of center aggregation constraint: On the basis of gradient enhancement in step 4, add a mask function based on similarity truncation;

[0017] Step 6: Build a complete multimodal network and train and optimize it using the total loss function.

[0018] Furthermore, step one is described in detail as follows:

[0019] Obtain a multimodal face image dataset, including RGB modality, IR modality, and depth modality; let the two input modalities be denoted as the first modality. Second mode The dataset is represented as ,in This represents the samples corresponding to two input modalities, where N is the number of samples and the class label. M represents the total number of categories; for modalities samples The correct pairing should be modal. samples However, in the case of instance mismatch, the actual pairing becomes ,in and .

[0020] Furthermore, the class center aggregation loss is defined as follows:

[0021]

[0022] in, It is a feature representation of sample pairs from the same category but different instances. This represents the category labels of the i-th and j-th samples. Let be the cosine similarity.

[0023] The class center aggregation loss formula after gradient boosting and boundary relaxation is as follows:

[0024]

[0025]

[0026] in, As a gradient enhancement factor, This is a mask function, where the function variable x is... , The threshold similarity hyperparameter is defined as:

[0027]

[0028] Furthermore, the formula for calculating the total loss function is as follows:

[0029]

[0030] in, For cross-entropy loss, The class center aggregation loss for the first mode. For the class center aggregation loss of the second mode, For hyperparameters

[0031] Furthermore, the gradient enhancement factor Threshold similarity hyperparameter =0.95.

[0032] Furthermore, during the inference phase, multimodal data is collected from the same target instance, and features of each modality are extracted using a trained feature encoder. A fusion representation is obtained through a fusion module, and then a linear classifier is applied to output the final face classification decision.

[0033] Compared with the prior art, the present invention has the following significant advantages:

[0034] It breaks through the limitation of traditional multimodal learning, which must rely on expensive instance-level paired data. It can directly use independently collected single-modal datasets for random intra-class paired training, which greatly reduces data preparation costs and has extremely high engineering application value.

[0035] It fundamentally suppresses modality-specific feature shift, significantly reducing the joint distribution gap between the training phase (instance mismatch) and the inference phase (instance alignment), thus achieving truly robust multimodal fusion.

[0036] By employing a synergistic design of gradient enhancement and boundary relaxation, the model maintains necessary instance diversity while ensuring strong intra-class consistency, thus avoiding feature collapse and overfitting. The model maintains stable and excellent performance even in extreme generalization scenarios such as cross-race and unseen attacks. Attached Figure Description

[0037] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0038] To better understand the purpose, structure, and function of this invention, the following description, in conjunction with the accompanying drawings, provides a more detailed account of a robust face spoofing attack detection method suitable for multimodal instance misalignment data.

[0039] This embodiment provides a robust face spoofing attack detection method suitable for multimodal instance misalignment data, which mainly includes the following three core technologies: class center aggregation technology, gradient enhancement technology, and boundary relaxation technology.

[0040] First, addressing the modality-specific feature shift inevitably introduced by instance misalignment, which leads to erroneous joint distributions across different modalities during training and significantly degrades fusion performance during inference, this invention proposes a class center aggregation technique. By independently introducing class center constraints within each modality, features from different instances of the same category aggregate towards a shared class center representation in the feature space, while simultaneously widening the feature margins between different categories. This technique effectively mitigates the modality-specific shift caused by differences in specific instances, enabling different modalities to form a consistent discriminative structure at the category semantic level. This fundamentally alleviates the inconsistency in training-inference distribution caused by instance misalignment and significantly improves the robustness of cross-modal fusion.

[0041] Secondly, addressing the issue that during class center aggregation, as intra-class features become increasingly compact and inter-class features gradually separate, the learning difficulty of some samples decreases and gradient signals weaken, thus affecting the continuous optimization of the model, this invention further proposes a gradient enhancement technique. This technique introduces a learnable gradient enhancement factor to adaptively enhance the discriminative constraint strength of highly aggregated or clearly separated samples, dynamically increasing their optimization difficulty. This ensures that effective gradient signals are continuously provided even in the later stages of training, guaranteeing stable model convergence and further strengthening intra-class consistency and inter-class discriminative ability.

[0042] Finally, to address the issues of excessively strong intra-class aggregation constraints potentially leading to over-compression of the feature space, loss of instance diversity, and model overfitting, this invention proposes a boundary relaxation technique. This technique introduces a similarity threshold and a mask modulation mechanism. When the feature similarity between samples of the same class exceeds a preset threshold, the corresponding gradient constraint strength is automatically reduced, thereby preserving necessary instance-specific information such as pose, expression, and lighting while maintaining intra-class compactness. This mechanism effectively prevents feature collapse and overfitting, improving the model's generalization ability and training stability in complex application scenarios such as those with unknown attack types. Figure 1 As shown, the method specifically includes the following steps:

[0043] Step 1: Construct unaligned training samples. Samples of different modalities, such as RGB, IR, and depth, are randomly paired within the same category based solely on their class labels, without requiring them to correspond to the same face instance.

[0044] Specifically, the two input modes are denoted as the first mode. Second mode The dataset is represented as ,in This represents the samples corresponding to two input modalities, where N is the number of samples and the class label. M represents the total number of categories. For example, in the CeFA-SURF(RGB-IR) dataset, the RGB modalities using the CeFA dataset are... The IR modes of the +SURF dataset are For modal samples The correct pairing should be modal. samples However, in the case of instance mismatch, the actual pairing becomes ,in and Therefore, it is possible to perform multimodal training using independently collected single-modal data, significantly reducing the cost of instance-level manual pairing.

[0045] Step 2: Obtain the feature representation of the mode. For each mode... and Use separate feature encoders and Misaligned sample pairs for instances constructed in step one Extract features to obtain feature representations:

[0046]

[0047]

[0048] in, and These are parameters of the modal encoder. This represents the first modality feature. This represents the second modality feature.

[0049] Step 3: Class center aggregation constraints for modal representation. This involves feature representations of the instance misalignment data obtained in Step 2. , For each modality, a class center aggregation loss constraint is applied, forcing highly compact intra-class features and significant inter-class feature separation, so that features of different instances of the same class tend to share a common class center representation. Here, we take the first modality as an example. For example, its class center aggregation loss is defined as follows:

[0050]

[0051] in, It is the feature representation obtained from pairs of samples of the same class but different instances using the independent feature encoder mentioned in step two, where This represents the category labels of the i-th and j-th samples. Cosine similarity. The first term enforces the constraint on the first mode. The second constraint on the first modality represents different samples belonging to the same class, reducing intra-class differences and minimizing cross-instance feature shift. The sample representations come from different classes, increasing the inter-class distance and preventing inter-class features from collapsing into the same representation.

[0052] Ideally, once the class center aggregation loss is fully optimized, the first mode... Medium category All samples will converge to the shared representation Even if the training data is mismatched, the second modality... samples Matched modal samples Instead Inference performance is also unaffected because... They share the same feature representation.

[0053] Step 4: Gradient enhancement for the central aggregation constraint. This addresses the issue in Step 3... By introducing gradient enhancement techniques and incorporating an adjustable gradient enhancement factor into the class center aggregation loss function, the loss response strength of sample pairs that are close to the optimization target is dynamically increased, thereby providing effective backpropagation signals for the feature encoding network even in the later stages of model training. As shown in the following equation, for the first mode... The original gradient is :

[0054]

[0055] It can be observed that during training, as intra-class similarity increases or inter-class differences increase, the gradient amplitude decreases, thereby weakening the optimization signal. Therefore, this invention introduces a gradient enhancement factor. To increase the learning difficulty of samples close to the target optimization, the formula for class center aggregation loss is modified to solve the gradient vanishing problem. The formula is as follows:

[0056]

[0057] At the same time, gradient guarantees The lower bound of the algorithm ensures effective optimization even when intra-class similarity approaches 1 or inter-class similarity approaches 0. Specifically, when the cosine similarity of a pair of samples of the same class is already high (e.g., 0.99, close to the original target 1), the original intra-class term loss is small ((0.99-1)²=0.0001), the gradient almost disappears, and the optimization signal is weak. After introducing α, the intra-class term becomes (cos-α-1)². For example, in practice, when α=0.3, (0.99-0.3-1)²=(-0.31)²=0.0961, the loss is amplified by about 960 times, the gradient increases significantly, and the network is forced to continue to "try to get closer" to the features (even if they are already very close). Similarly, when the cosine similarity of inter-class samples is very low (e.g., 0.01), the original term loss is 0.0001; after adding α, (0.01+0.3)²=0.0961, the loss is also amplified, continuously pushing away features from the different classes. Therefore, regardless of how close the intra-class similarity is to 1 or how close the inter-class similarity is to 0, the gradient magnitude will always guarantee at least one constant. The lower bound that is directly proportional (approximately) Or a multiple thereof, depending on the number of sample pairs in the batch, will not disappear completely.

[0058] Step 5: Boundary relaxation of center aggregation constraints. Although center aggregation can compress the offset to near zero, excessively forcing samples of the same class but different physical instances to have completely identical features can lead to feature space collapse, loss of instance-specific information such as pose and lighting, impairing generalization ability, and also easily causing overfitting. Therefore, this invention implements gradient enhancement in Step 4. Add a mask function on top of that. This approach softens excessive intra-class constraints, preserves instance-private information such as pose and lighting, prevents feature overfitting, and maintains optimization stability. Similarly, it uses modal... For example, the class center aggregation loss with boundary relaxation, added on top of gradient boosting, is:

[0059]

[0060]

[0061] The mask function Defined as:

[0062]

[0063] in Here, x is the threshold similarity hyperparameter, and the function variable x is... This function performs similarity truncation. Specifically, in practice... =0.95, when a pair of similar samples < hour, = The loss is normally calculated as The strong gradient drives the features to converge further. ≥ hour, Forced to be fixed as The loss becomes a fixed value. At this point, the gradient contributes zero to the cosine similarity of the sample pair because the output of R is constant. The similarity no longer changes with cosine similarity, which is equivalent to "relaxing" the further aggregation constraints on these already sufficiently similar samples. This avoids overly forcing all similar samples to have completely identical features (collapse to a single point), preserves instance-specific variation information such as pose, lighting, and expression, and prevents feature space collapse and overfitting.

[0064] Step Six: Build a complete multimodal network and train it using the total loss function.

[0065]

[0066] The first item The first term is the cross-entropy loss for face spoofing attack detection, and the second term is the bimodal instance aggregation loss, obtained using the optimization from step five. Constrained modes and This constrains similar samples to be closer together and dissimilar samples to be farther apart, thereby mitigating feature shift between matched and mismatched multimodal data pairs. These are hyperparameters used to balance the strength of any loss and the regularization loss; in practice... =2.

[0067] During the inference phase, multimodal data is collected from the same target instance, meaning there is no instance mismatch. Features for each modality are extracted using a trained feature encoder. and And through the fusion module The fusion representation is obtained It follows the standard form Subsequently, the application parameters are: and A linear classifier is used to obtain the logarithmic value. The final multimodal prediction result can be expressed as... Based on the final classification judgment of the numerical output, the input face is determined to be either "real live" or "spoofing".

[0068] Table 1 shows the composition of the dataset used in this embodiment for multimodal learning tasks with instance mismatch.

[0069] Table 1: Composition of CeFA-SURF (RGB-IR) and CeFA-SURF (IR-RGB) datasets.

[0070]

[0071] For the CeFA-SURF dataset, ResNet18 was used as the backbone network, and parameter sharing was performed on all methods. Data augmentation included random flipping, rotation, and cropping. All models were trained using SGD for 100 epochs with a batch size of 64 and an initial learning rate of 1×10⁻⁶. -3 The learning rate was linearly warmed up for the first 5 cycles, and then decayed by a factor of 10 in the 33rd and 50th cycles. The weight decay and momentum were set to 0.0005 and 0.9, respectively.

[0072] Table 2: Performance comparison of different methods on CeFA-SURF (RGB-IR) and CeFA-SURF (IR-RGB).

[0073]

[0074] BPCER: Liveness Rendering Classification Error Rate. This refers to the proportion of genuine live face samples incorrectly identified as forgeries during detection. ACER: Average Classification Error Rate. It is the arithmetic mean of APCER and BPCER, used to comprehensively evaluate the performance of a face anti-spoofing system; a lower value indicates better overall system performance. RGB-only and IR-only are unimodal models relying solely on RGB and IR inputs; RGB-IR-Paired is a model relying on matching multimodal inputs; RGB-IR-UnPaired is a model relying on matching multiple modal inputs; Concat and Parameter-Share are traditional multimodal fusion algorithms; PMR is an attention-based modality correction method; ODAF, Norton, and UCPM are realignment-based modality correction methods.

[0075] Table 2 shows that the proposed method significantly improves multimodal performance in noisy instance scenarios. Its performance in cross-tests on two public datasets surpasses all comparative methods and is significantly better than the single-modal baseline, approaching the upper bound of instance-aligned training. This fully verifies its superiority in mitigating training-inference feature shift and achieving robust cross-modal fusion.

[0076] It is understood that the present invention has been described through some embodiments, and those skilled in the art will recognize that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the invention. Furthermore, under the teachings of the present invention, these features and embodiments can be modified to adapt to specific situations and materials without departing from the spirit and scope of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the protection scope of the present invention.

Claims

1. A robust face spoofing attack detection method suitable for multimodal instance misalignment data, characterized in that, Includes the following steps: Step 1: Construct instance misaligned sample pairs: Randomly pair face image samples from different modalities only within the same category based on the category label, without requiring them to correspond to the same face instance; Step 2: Obtain the feature representation of the modality: For each modality, use an independent feature encoder to extract features from the instance misaligned sample pairs constructed in Step 1, and obtain the feature representation: Step 3: Class center aggregation constraint for modality representation: For the feature representation of the misaligned sample pairs obtained in Step 2, class center aggregation loss constraint is applied to each modality to force intra-class features to be highly compact and inter-class features to be clearly separated, so that the features of different instances of the same class tend to share the class center representation. Step 4: Gradient enhancement for center aggregation constraints: An adjustable gradient enhancement factor is introduced into the class center aggregation loss function to dynamically increase the loss response strength of sample pairs that are close to the optimization target, so that they can still provide effective backpropagation signals for the feature encoder in the later stages of model training. Step 5: Boundary relaxation of center aggregation constraint: On the basis of gradient enhancement in step 4, add a mask function based on similarity truncation; Step 6: Build a complete multimodal network and train and optimize it using the total loss function.

2. The robust face spoofing attack detection method suitable for multimodal instance misalignment data according to claim 1, characterized in that, Step one is described in detail as follows: Obtain a multimodal face image dataset, including RGB modality, IR modality, and depth modality; denot the two input modalities as the first modality. Second mode The dataset is represented as ,in This represents the samples corresponding to two input modalities, where N is the number of samples and the class label. M represents the total number of categories; for modalities samples The correct pairing should be modal. samples However, in the case of instance mismatch, the actual pairing becomes ,in and .

3. The robust face spoofing attack detection method suitable for multimodal instance misalignment data according to claim 2, characterized in that, The class center aggregation loss is defined as follows: in, It is a feature representation of sample pairs from the same category but different instances. This represents the category labels of the i-th and j-th samples. Let be the cosine similarity.

4. A robust face spoofing attack detection method suitable for multimodal instance misalignment data according to claim 3, characterized in that, The class center aggregation loss formula after gradient boosting and boundary relaxation is as follows: in, As a gradient enhancement factor, Here, the function variable x is a mask function. , The threshold similarity hyperparameter is defined as:

5. A robust face spoofing attack detection method suitable for multimodal instance misalignment data according to claim 4, characterized in that, The formula for calculating the total loss function is as follows: in, For cross-entropy loss, The class center aggregation loss for the first mode. For the class center aggregation loss of the second mode, This is a hyperparameter.

6. A robust face spoofing attack detection method suitable for multimodal instance misalignment data according to claim 5, characterized in that, The gradient enhancement factor Threshold similarity hyperparameter =0.

95.

7. A robust face spoofing attack detection method suitable for multimodal instance misalignment data according to claim 6, characterized in that, During the inference phase, multimodal data is collected from the same target instance, and features of each modality are extracted using a trained feature encoder. The fusion module then obtains a fused representation, and a linear classifier is applied to output the final face classification decision.

Images