Adaptive cross-modal face liveness detection method and device, electronic equipment and medium

Abstract

The application provides a self-adaptive cross-modal face living body detection method and device, electronic equipment and medium, the method comprises the steps of: processing the received three modal images by using a lightweight backbone network to obtain the primary feature representation of each mode; the primary feature representation is processed by a lightweight cross-modal attention module to obtain the corresponding spatial-modal weight map of each mode; based on the spatial-modal weight map, the primary feature representation is reconstructed and completed by a lightweight cross-modal fusion module through cross-modal attention, and a fusion feature is obtained; under the constraint of joint loss, the fusion feature is predicted by a lightweight classification head to complete face living body detection.

Description

Technical Field

[0001] This invention relates to the field of face liveness detection technology, specifically to an adaptive cross-modal face liveness detection method, device, electronic device, and medium. Background Technology

[0002] Facial recognition technology, with its advantages of being contactless, easy to operate, and highly efficient, has been widely applied in engineering scenarios such as identity authentication, financial payments, public security, and smart terminals, becoming one of the most representative biometric identification technologies at present. However, with the large-scale deployment of facial recognition technology, the risk of facial attack is becoming increasingly prominent. Attackers can forge facial information through methods such as printed photos, video playback, and 3D masks, easily bypassing the identity verification process of traditional facial recognition systems. This not only poses a direct threat to personal privacy and property security but also seriously reduces the reliability of facial recognition systems in high-security scenarios. Therefore, facial liveness detection, as a core component in ensuring the security of facial recognition systems, has significant engineering value and practical implications for its research and optimization.

[0003] Early face liveness detection methods were primarily based on a single red-green-blue (RGB) modality, distinguishing real faces from forged samples by extracting texture features, frequency domain information, or temporal behavioral cues from face images. While these methods could meet basic detection needs in controlled environments, their reliance on only two-dimensional appearance information made them susceptible to variations in lighting, differences in imaging equipment, and high-quality forgery materials. Their robustness and generalization capabilities were limited in complex real-world scenarios. To overcome the inherent limitations of single-modal methods, multimodal face liveness detection methods emerged. These methods incorporate auxiliary sensing information such as depth and infrared sensors, fusing complementary features from different modalities to enhance the ability to detect various presentation attacks. Related research and public benchmarks have verified that multimodal methods significantly outperform single-modal methods in both detection accuracy and stability, making them the mainstream development direction in the field of face liveness detection.

[0004] However, existing multimodal face liveness detection methods still face two major technical challenges in practical applications, making it difficult to meet the deployment requirements of resource-constrained scenarios and the detection requirements of complex attack scenarios:

[0005] High computational complexity and difficulty in lightweight deployment: The performance improvement of existing multimodal methods largely depends on multi-branch deep network structures or high-dimensional feature splicing strategies, which leads to a significant increase in the number of model parameters and floating-point operations. Although the detection accuracy has been improved, the computational overhead is too large, making it impossible to deploy efficiently on resource-constrained edge devices such as mobile devices and embedded devices, thus limiting the engineering application of multimodal face liveness detection technology. Cross-modal fusion strategies lack robustness and have poor adaptability to unknown attacks: The discriminative contributions of different sensing modalities in different spatial regions of the face, different attack types, and different imaging conditions are not constant. Existing multimodal methods mostly adopt static or coarse-grained fusion strategies, failing to explicitly model the dynamic differences between modalities in spatial dimensions and attack scenarios. They cannot adaptively adjust the discriminative weights of each modality according to the feature distribution of the input samples, resulting in insufficient generalization ability and a significant drop in detection performance when facing unknown attack types, modal noise, or modal missingness.

[0006] Meanwhile, existing lightweight networks in face liveness detection are mostly focused on single-modal feature extraction. Although strategies such as depthwise separable convolution, channel rearrangement, and inexpensive feature generation have reduced model complexity, when directly applied to multimodal scenarios, they suffer from low modal feature utilization and weak cross-modal information interaction capabilities, making it difficult to balance the complementary mining of multimodal features and the computational efficiency of the model. Furthermore, existing cross-modal attention and fusion mechanisms are mostly designed for general computer vision tasks and have not been adapted to the lightweight nature of face liveness detection tasks. Their introduction can easily lead to a significant increase in model computational overhead, failing to meet the needs of resource-constrained scenarios.

[0007] Therefore, how to build a computationally efficient and lightweight face liveness detection framework while maintaining the complementary advantages of multimodal features, and at the same time achieve adaptive modeling of input correlation for the discrimination contributions of different modalities, so as to improve the robustness and generalization ability of the model under complex attack scenarios, has become a key technical problem that urgently needs to be solved in the field of multimodal face liveness detection. Summary of the Invention

[0008] In view of this, the present disclosure provides an adaptive cross-modal face liveness detection method, apparatus, electronic device and medium, which at least partially solves the problems existing in the prior art.

[0009] In a first aspect, embodiments of this disclosure provide an adaptive cross-modal face liveness detection method, which includes: A lightweight backbone network is used to process the received trimodal images to obtain the primary feature representations of each modality; The primary feature representation is processed by a lightweight cross-modal attention module to obtain the spatial-modal weight map corresponding to each modality; Based on the aforementioned spatial-modal weight graph, a lightweight cross-modal fusion module is used to perform cross-modal attention reconstruction and completion on the primary feature representation to obtain fused features; Under the constraint of joint loss, a lightweight classification head is used to perform liveness prediction on the fused features to complete face liveness detection.

[0010] According to a specific implementation of this disclosure, the step of processing the received trimodal image using a lightweight backbone network to obtain primary feature representations of each modality includes: Through convolutional layers, the original input image of each modality of the three modal images is initially spatially downsampled to obtain shallow features of each modality with the same number of feature channels; The shallow features of each modality are progressively downsampled by the group shuffling module and the squeeze excitation channel attention module to complete the high semantic feature encoding and obtain the primary feature representation.

[0011] According to a specific implementation of this disclosure, the step of processing the primary feature representation through a lightweight cross-modal attention module to obtain the space-modal weight map corresponding to each modality includes: The primary feature representations of each modality are stacked along the modality dimension to obtain a cross-modality feature tensor; The number of channels in the cross-modal feature tensor is compressed using 1×1 convolution to obtain a preliminary joint representation of the cross-modal features; By grouping one-dimensional convolutions, parallel response modeling is performed on the preliminary joint features to obtain a three-modal attention scoring feature tensor; The mean of the attention score feature tensor of the three modalities is calculated, and the mean is normalized to obtain the space-modality weight map. Wherein, the sum of the three modal weights at any spatial location on the spatial-modal weight graph is 1.

[0012] According to a specific implementation of this disclosure, the step of performing cross-modal attention reconstruction and completion on the primary feature representation based on the spatial-modal weight graph using a lightweight cross-modal fusion module to obtain fused features includes: The primary feature representation of each modality is multiplied pixel-by-pixel with the weight of the corresponding modality in the spatial-modal weighting graph to complete spatial adaptive weighting; The weighted main modal features are mapped to obtain the query matrix; The weighted first and second auxiliary modal features are mapped to the corresponding key and value matrices, respectively. By performing cross-modal correlation distribution calculation on the main modal feature, the first auxiliary modal feature, and the second auxiliary modal feature through dot product attention, information reconstruction at the semantic level is completed, and the first modal fusion feature and the second modal fusion feature are obtained. The first modality fusion feature, the second modality fusion feature, and the weighted main modality feature are concatenated along the channel dimension, and then the concatenated features are mapped to a unified feature space through convolution to obtain the fusion feature.

[0013] According to a specific implementation of this disclosure, the step of using a lightweight classification head to perform liveness prediction on the fused features under the constraint of the joint loss function to complete face liveness detection includes: mapping and increasing the dimensionality of the fused features through 1×1 convolution; Global average pooling is performed on the fused features after dimensionality upscaling to obtain a global feature representation; The global feature representation is subjected to overfitting suppression processing; Using a fully connected layer, the global feature representation after overfitting suppression is mapped to a 2D feature space. The global feature representation after the 2D feature space mapping is then processed by a normalization operation to obtain the probability distributions corresponding to the two classes: liveness and attack. Based on the liveness probability distribution, the error between the predicted value and the true label of the sample is calculated to obtain the binary classification cross-entropy loss. The difference between the fused feature map corresponding to the fused feature and the structured target map is measured at the pixel level to obtain the pixel-level fusion weighted loss; For the space-modal weight graph, the modal-level weight vector is obtained by taking the mean value in the spatial dimension. The binary cross-entropy loss of the three-modal branches is calculated. The binary cross-entropy loss is dynamically weighted and summed based on the modal-level weight vector to obtain the modal-weighted loss. The binary classification cross-entropy loss, the pixel-level fusion weighted loss, and the modality weighted loss are weighted and summed to obtain the joint loss; During training, the joint loss is minimized using the backpropagation algorithm, and the detection model parameters are iteratively updated to obtain the trained detection model. The trained detection model is used to predict liveness using the fused features, thus completing face liveness detection.

[0014] Secondly, embodiments of this disclosure provide an adaptive cross-modal face liveness detection device, which includes: The feature extraction module is configured to use a lightweight backbone network to perform feature extraction processing on the received trimodal images and output the primary feature representations of each modality. The attention weight reconstruction module is configured to perform cross-modal attention modeling processing on the primary feature representation to generate a space-modal weight map corresponding to each modality. The cross-modal fusion module is configured to perform cross-modal attention reconstruction and completion on the primary feature representation based on the space-modal weight graph, and output fused features; The detection and decision module is configured to perform liveness prediction on the fused features using a lightweight classification head under the constraint of joint loss, thereby completing face liveness detection.

[0015] According to a specific implementation of an embodiment of this disclosure, the feature extraction module includes: an initial downsampling unit and a progressive encoding unit; The initial downsampling unit is a convolutional layer, used to perform initial spatial downsampling on the original input image of each modality in the three modal images, and output shallow features of each modality with the same number of feature channels; The progressive encoding unit includes a group shuffling module and a squeezed excitation channel attention module, which are used to progressively downsample the shallow features of each modality and complete the high semantic feature encoding, and output the primary feature representation; The group shuffling module is a lightweight feature extraction module that combines ghost feature generation mechanism, residual connection and channel shuffling. It completes feature extraction through channel mapping, feature expansion, channel splicing and shuffling.

[0016] According to a specific implementation of an embodiment of this disclosure, the attention weight reconstruction module includes: a feature stacking unit, a channel compression unit, a response modeling unit, and a weight normalization unit; The feature stacking unit is used to stack the primary feature representations of each modality in the modality dimension to generate a cross-modality feature tensor; The channel compression unit is used to compress the number of channels of the cross-modal feature tensor by a preset ratio using 1×1 convolution to obtain a preliminary joint representation of the cross-modal features; The response modeling unit is used to perform parallel response modeling on the preliminary joint representation through grouped one-dimensional convolution with a preset number of groups, and generate a three-modal attention score. The weight normalization unit is used to calculate the mean and normalize the attention scores of the three modalities, and output the spatial-modal weight map, wherein the sum of the three modal weights at any spatial location on the spatial-modal weight map is 1.

[0017] Thirdly, embodiments of this disclosure provide an electronic device, the electronic device comprising: At least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the adaptive cross-modal face liveness detection method described above.

[0018] Fourthly, embodiments of this disclosure provide a non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the aforementioned adaptive cross-modal face liveness detection method.

[0019] In summary, compared with the prior art, this embodiment has the following advantages: With a high degree of lightweight design and extremely low resource consumption, the backbone network is built based on the improved ShuffleNetV2 and Ghost feature generation mechanism. Combined with the lightweight module design, the model has only 0.14M parameters and 0.0065G floating-point operations, which is far lower than conventional multimodal face liveness detection networks. It can be efficiently deployed in resource-constrained scenarios such as mobile devices and edge devices.

[0020] Strong cross-modal fusion adaptability: A lightweight cross-modal attention module and a weight-guided fusion module are proposed, which can dynamically allocate the weights of each modality at different spatial locations, automatically enhance discriminative modal features and suppress noise information, and solve the problem that traditional fixed fusion strategies cannot adapt to complex attacks and scene changes.

[0021] Outstanding detection accuracy and robustness: Achieves an average classification error rate of 0.064% on the CASIA-SURF dataset, with a true positive rate of 99.86% under a fixed low false positive rate. It maintains stable performance under complex protocols such as cross-race, cross-attack type, and cross-modal, and has stronger generalization ability to unknown attacks.

[0022] Good feature discriminativeness and interpretability: The joint loss simultaneously supervises the classification results, spatial response distribution and modality weight allocation, enabling the model to focus on key facial regions; t-SNE visualization shows high feature separability between live and attack samples, the feature learning process is clear and the results are more reliable.

[0023] Strong modal compatibility and scalability: Supports RGB, depth and infrared three-modal input, with optimal performance under multimodal conditions, while also being compatible with single-modal RGB input and maintaining good detection effect, adapting to actual application scenarios with different sensor configurations. Attached Figure Description

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings. Here, the illustrative embodiments and descriptions of this invention are used to explain the invention, but are not intended to limit the invention.

[0025] The term "comprising" and its variations as used herein signify open inclusion, i.e., "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "at least partially based on". The terms "one example embodiment" and "one embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0026] Figure 1 This application provides a flowchart illustrating an adaptive cross-modal face liveness detection method. Figure 2 A schematic diagram of the structure of an adaptive cross-modal face liveness detection device provided in this application embodiment; Figure 3 An exemplary structural diagram of a device capable of implementing the method according to an embodiment of the present invention is shown. Detailed Implementation

[0027] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0028] The following specific examples illustrate the implementation of this disclosure. Those skilled in the art can easily understand other advantages and effects of this disclosure from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. This disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this disclosure. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0029] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.

[0030] It should also be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this disclosure. The drawings only show the components related to this disclosure and are not drawn according to the number, shape and size of the components in actual implementation. In actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0031] Furthermore, specific details are provided in the following description to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that the described aspects can be practiced without these specific details.

[0032] Please see Figure 1 The present application provides a flowchart of an adaptive cross-modal face liveness detection method, which can be executed by an electronic device, specifically by one or more processors of the electronic device, and implements the following steps: S101 uses a lightweight backbone network to process the received trimodal image and obtain the primary feature representation of each modality.

[0033] Optionally, through convolutional layers, the original input image of each modality of the three-modal image is initially spatially downsampled to obtain shallow features of each modality with the same number of feature channels; The shallow features of each modality are progressively downsampled by the group shuffling module and the squeeze excitation channel attention module to complete the high semantic feature encoding and obtain the primary feature representation.

[0034] Specifically, to achieve channel dimension unification and shallow spatial feature extraction for multimodal features, the original input image of each modality of the three-modal images (e.g., RGB color image, depth image, infrared image) is first subjected to initial spatial downsampling and channel dimension upsampling operations through a convolutional layer (e.g., 3×3): after the original images of each modality are uniformly adjusted to the standard input size (e.g., 64×64), they are processed by the convolutional layer and output tensor features with the same number of channels (e.g., all 24), thus obtaining shallow features of each modality with shape [B,24,32,32] (B is the batch size).

[0035] For the shallow features of each modality with a uniform number of channels, a strategy of independent parallel processing of each modality is adopted. By stacking feature extraction units with Ghost-Generated Shuffle BlockA (GGS-BlockA) as the core, and embedding SE (Squeeze-and-Excitation) channel attention modules in each downsampling stage, a progressive downsampling and feature encoding from shallow texture to mid-level structure and then to high-level semantics is completed. This process is divided into three consecutive stages: stage 2, stage 3, and stage 4. The RGB, Depth, and IR modal branches share the same network architecture. The specific process is as follows: Phase 2 processing: Cascade two GGS-BlockA units with the SE2 channel attention module to shrink the spatial resolution of shallow features from 32×32 to 16×16 and adjust the number of channels to 16, thereby achieving the initial extraction of mid-level structural features; The SE2 module performs lightweight recalibration of intramodal features by explicitly modeling channel dependencies, thereby suppressing the interference of background noise on effective features.

[0036] Phase 3 processing: Six GGS-BlockA units are cascaded with an SE3 channel attention module to downsample the feature space resolution to 8×8 and expand the number of channels to 32, further mining fine-grained discriminative features within the modality; during this process, GGS-BlockA improves feature utilization with low computational overhead and avoids parameter redundancy through the collaborative design of Ghost feature generation mechanism, grouped convolution and channel shuffling.

[0037] Stage 4 processing: cascade 3 GGS-BlockA units with SE4 channel attention modules to finally downsample the feature space resolution to 4×4 and expand the number of channels to 48, completing high semantic feature encoding; the feature map output in this stage integrates the core semantic information required for face liveness detection (such as RGB texture, Depth concave-convex structure, IR heat distribution features) and maintains extremely low computational complexity.

[0038] After the aforementioned progressive downsampling and high semantic feature encoding, the RGB, Depth, and IR modalities each obtain a high semantic feature tensor of shape [B, 48, 4, 4], denoted as follows: This feature tensor is the primary feature representation of each mode.

[0039] The core design advantage of this feature extraction process is that it achieves three-level feature extraction of shallow texture, mid-level structure, and high-level semantics with only a small number of inexpensive convolution operations and feature reuse strategies. The final output primary feature representation maintains a compact tensor dimension and has strong discriminativeness and modality specificity.

[0040] S102 processes the primary feature representation through a lightweight cross-modal attention module to obtain the spatial-modal weight map corresponding to each modality.

[0041] Optionally, a 1×1 convolution is used to compress the number of channels of the cross-modal feature tensor to obtain a preliminary joint representation of the cross-modal features; parallel response modeling is performed on the preliminary joint features through grouped one-dimensional convolution to obtain a trimodal attention score feature tensor; the mean of the trimodal attention score feature tensor is calculated and normalized to obtain the spatial-modal weight map; wherein the sum of the trimodal weights at any spatial location on the spatial-modal weight map is 1.

[0042] Specifically, the primary features of the RGB, Depth, and IR modes output by the lightweight backbone network will be represented... As input to the LCMA (Lightweight Cross-Modal Attention Module), in order to achieve pixel-level cross-modal feature modeling, the features of the three modalities are first stacked and concatenated in the modal dimension to form a complete cross-modal feature tensor. This allows the lightweight cross-modal attention module to simultaneously capture the feature correlations and differences between the three modalities, laying the foundation for subsequent attention score calculation.

[0043] To significantly reduce computational overhead and avoid feature dimensionality redundancy while ensuring joint modeling of cross-modal features, a 1×1 convolution operation is performed on the stacked cross-modal feature tensors, compressing their channel count to 1 / 8 of the original number. This operation achieves preliminary fusion and joint representation of cross-modal features through dimensionality reduction, preserving the core feature association information of the three modalities while allowing the subsequent attention modeling process to be performed in a low-dimensional feature space, which aligns with the lightweight design principle of the entire model.

[0044] Based on the preliminary joint representation of cross-modal features in low dimension, the lightweight cross-modal attention module uses a grouped one-dimensional convolution with 4 groups to perform parallel response modeling on features in the modality-pixel dual dimension. In the output of this grouped one-dimensional convolution, the first three response channels are precisely mapped to the attention scores corresponding to the RGB, Depth, and IR modalities, which are used to quantify the discriminative contribution of each modality at each spatial location. Subsequently, the attention score response maps corresponding to these three modalities are dimensionally reshaped to obtain a feature tensor of shape B×3×K×H×W (B is the batch size, K is the parallel response dimension within the modality, and K×H×W are the height and width of the feature map), completing the spatial modeling of the three-modal attention scores and obtaining the three-modal attention score feature tensor.

[0045] For the reshaped trimodal attention score feature tensor, the mean is calculated on the independent branch of each modality to eliminate the interference of the parallel response dimension K within the modality, resulting in trimodal attention score features that only include batch, modality, and spatial location dimensions, so that the score can more accurately reflect the relative importance of each modality at each spatial location. Softmax normalization is performed on the mean-calculated trimodal attention scores to generate a pixel-level joint weight distribution, i.e., a spatial-modal weight map Q={Q RGB Q Depth Q IR (These correspond to the spatial weight subgraphs for RGB, Depth, and IR modes, respectively).

[0046] This normalization operation strictly constrains the numerical distribution of the weight map, ensuring that for any spatial location (h, w) on the feature map, the sum of the weights of the three modes RGB, Depth, and IR at that location is always 1. This allows the weight map to dynamically allocate the contribution of the three modes to the discrimination. In other words, the effective modal weights at important spatial locations will be increased, while the weights of redundant and noisy modes will be suppressed.

[0047] The final generated spatial-modal weight map is a dynamic weight representation of the input, meaning that different face samples (live / attack samples, samples with different attack types) will correspond to different weight distributions: the module will adaptively adjust the relative contribution of each modality at different spatial locations based on the cross-modal feature distribution of the input sample. For example, in liveness-sensitive areas such as nasolabial folds and eye sockets, the weights corresponding to RGB texture, Depth concave-convex structure, and IR thermal distribution features will be specifically increased, while the modal weights of background and illumination noise areas will be suppressed.

[0048] S103, Based on the spatial-modal weight graph, the primary feature representation is reconstructed and completed through cross-modal attention using a lightweight cross-modal fusion module to obtain fused features.

[0049] Optionally, the primary feature representations of each modality are multiplied pixel-by-pixel with the corresponding modal weights in the spatial-modal weighting graph to achieve spatial adaptive weighting; the weighted primary modal features are mapped to obtain a query matrix; the weighted first and second auxiliary modal features are mapped to their corresponding key and value matrices, respectively; cross-modal correlation distribution calculations are performed on the primary modal features, the first auxiliary modal features, and the second auxiliary modal features using dot product attention to complete semantic-level information reconstruction and obtain the first and second modal fusion features; the first and second modal fusion features are concatenated with the weighted primary modal features along the channel dimension, and then the concatenated features are mapped to a unified feature space through convolution to obtain the fusion features.

[0050] Specifically, firstly, the spatial-modal weight map output by the LCMA module (containing the adaptive weights of the three modes RGB, Depth, and IR at each spatial location) is combined with the trimodal primary feature representation obtained by the feature extraction network. Perform dimensional matching and alignment. Ensure that both are completely consistent in spatial size (H×W) and batch dimension (B), laying the foundation for subsequent pixel-by-pixel weighting operations.

[0051] To highlight the features of each modality in key spatial regions and suppress interference from noise regions, the primary features of each modality are first spatially adaptively weighted: the primary feature representation of each modality is multiplied pixel-by-pixel with the weight of the corresponding modality in the spatial-modal weight map. For example, the feature F of the RGB modality... RGB Q in the spatial-modal weighting graph RGB Multiplication, F of the Depth mode Depth With Q Depth Multiplication, F of IR mode IR With Q IR Multiplication. Specifically, it is represented as: ⊙ , ⊙ , ⊙ The symbol ⊙ represents pixel-wise multiplication. Spatial adaptive weighting is achieved, enabling dynamic filtering: in highly discriminative regions such as facial features and texture details, the corresponding modal has a higher weight, and the features are significantly enhanced; in ineffective regions such as background and lighting noise, the weight is lower, and the features are weakened, thereby improving the feature quality of subsequent fusion.

[0052] Based on the weighted trimodal features, a query matrix, a key matrix, and a value matrix are constructed to simulate the core logic of the attention mechanism and realize the calculation of cross-modal relevance distribution at the semantic level. The weighted features of the dominant modality (usually the most informative and discriminative modality, such as Depth) are selected and mapped to a query matrix through a linear mapping or convolutional layer. .

[0053] Key / value matrix generation: Perform linear mapping of the same dimension on the weighted features of the first auxiliary mode (e.g., RGB) and the second auxiliary mode (e.g., IR) respectively to obtain the key matrix and the value matrix.

[0054] The key matrix is ​​used to provide semantic references across modalities, while the value matrix is ​​used to carry the feature information that needs to be fused in the end.

[0055] Next, the dot product of the query matrix and the two secondary modal key matrices is calculated to obtain the cross-modal correlation score distribution between the primary modality and the first and second secondary modalities, as follows: Where C is the channel dimension, used to normalize the scale of attention weights. Through this dominant mode (Depth)-driven attention computation, query features are... It can guide the reconstruction of features from the first auxiliary modality (e.g., RGB) and the second auxiliary modality (e.g., IR) at the semantic level, thereby achieving cross-modal structured complementarity. The correlation score distribution quantifies the degree of semantic association between the main modality features and each auxiliary modality feature; regions with higher correlation have greater weights.

[0056] Then, the relevance scores obtained by the dot product are weighted and summed with the value matrices of the corresponding auxiliary modalities to complete the semantic-level information reconstruction, ultimately obtaining the first modality fusion feature (main modality + first auxiliary modality) and the second modality fusion feature (main modality + second auxiliary modality). This step realizes the transfer and fusion of auxiliary modality information into the main modality semantic space, making up for the information deficiency of single modality.

[0057] To further integrate the core information of the three modalities and obtain the final fused features, channel-dimensional concatenation and convolution mapping operations are performed. The first modality fused features, the second modality fused features, and the weighted main modality features are concatenated along the channel dimension, specifically: In other words, the features enhanced by the lightweight cross-modal fusion module and with deep branch features After concatenation, the data is restored to a unified 64-dimensional feature space via 1×1 convolution, yielding the final fused output. This fused output now includes the original enhanced information of the main modality, as well as cross-modal semantic fusion information with the two auxiliary modalities, further expanding the number of channels and enriching the feature dimensions.

[0058] Furthermore, to avoid computational redundancy caused by an excessive number of channels, and to map the concatenated fusion output to a unified feature space, a 1×1 convolutional layer (or a lightweight convolutional layer) is used to perform dimensionality reduction and nonlinear transformation on the concatenated features. This operation filters and integrates all redundant information, generating the final, compact, and highly discriminative fusion features.

[0059] The final fused feature output retains the unique advantages of each modality (such as the structural information of texture detail depth in RGB and the anti-interference ability of IR), while achieving information complementarity and enhancement through a cross-modal attention mechanism. This effectively suppresses noise and redundancy between modalities, significantly improving the robustness and discriminativeness of the features. This feature can be directly used for subsequent tasks such as face liveness classification and attack type recognition, and is a key component in ensuring model performance.

[0060] Proceed to step S104. S104, under the constraint of joint loss, uses a lightweight classification head to perform liveness prediction on the fused features, thus completing face liveness detection.

[0061] Optionally, a global average pooling operation is performed on the dimensionality-upgraded fused features to obtain a global feature representation; overfitting suppression is applied to the global feature representation; a fully connected layer is used to map the overfitting-suppressed global feature representation to a 2D feature space, and a normalization operation is performed on the mapped global feature representation to obtain probability distributions corresponding to the live and attack classes; based on the live probability distribution, the error between the predicted value and the true label of the sample is calculated to obtain the binary classification cross-entropy loss; at the pixel level, the difference between the fused feature map corresponding to the fused features and the structured target map is measured to obtain the pixel-level fusion weighted loss. The process involves: For the space-modal weight map, taking the mean value in the spatial dimension to obtain a modal-level weight vector; calculating the binary cross-entropy loss of the three modal branches; dynamically weighting and summing the binary cross-entropy loss based on the modal-level weight vector to obtain the modal-weighted loss; weighting and summing the binary cross-entropy loss, the pixel-level fusion weighted loss, and the modal-weighted loss to obtain the joint loss; minimizing the joint loss during training using a backpropagation algorithm to iteratively update the detection model parameters to obtain the trained detection model; and using the trained detection model to perform liveness prediction on the fused features to complete face liveness detection.

[0062] Specifically, firstly, regarding fusion features A 1×1 convolution is used for channel mapping and dimensionality increase, mapping the number of channels from 64 to 128. This enhances the non-linear expressive power of features without expanding the receptive field. The calculation form is as follows: This provides a more discriminative feature foundation for subsequent classification. Then, a global average pooling operation is performed on the upgraded fused features to aggregate spatial dimensionality information into a global feature representation, as follows: This approach enables the model to gain a holistic semantic understanding of the entire face region while significantly reducing feature dimensionality. To prevent overfitting during training, Dropout (rate = 0.3) is applied to the global feature representation to suppress overfitting and enhance the model's generalization ability in unknown samples and complex scenes. Next, a fully connected layer is used to map the processed global features to a two-dimensional output space. These correspond to two categories: living beings and attackers; then, through Softmax normalization, the output is converted into a probability distribution. The model predicts whether the current sample is a live or an attack.

[0063] For loss optimization, a joint loss consisting of three sub-losses is used for end-to-end training: Binary Cross-Entropy (BCE): Calculates the classification error based on the predicted probability and the true label of the sample, directly supervising the model's liveness detection ability. Its definition is as follows: (12) Where n represents the total number of training samples. The true label for the i-th sample is 1 for a live sample and 0 for a fake sample. This represents the liveness probability predicted by the model. The binary cross-entropy loss effectively guides the model to learn a stable and clear classification decision boundary by accumulating the prediction errors of each sample.

[0064] Pixel-Wise Fusion Loss (PWL): Measures the difference between the fused feature map and the pre-defined structured target map at the pixel level, constraining the fused features to form a stable and separable response distribution in space, and enhancing the discriminative power of key regions. Specifically: Where H and W represent the height and width of the feature map, respectively, and the target map Y is set according to the sample type: the target map Y is set as a structured reference response to constrain the overall activation distribution of the fused features in the spatial dimension and avoid overly strong responses in irrelevant regions.

[0065] Modality-Weighted Loss (LAF): The spatial-modality weight map output by the lightweight cross-modality attention module is averaged along its spatial dimension to obtain a modality-level weight vector. The binary cross-entropy loss for the RGB, Depth, and IR modal branches is calculated separately. This weight vector is then used to dynamically weight and sum the binary cross-entropy losses of each branch to obtain the modality-weighted loss. Specifically, let the batch size be n. For the i-th sample, the liveness prediction probabilities of the RGB, Depth, and IR modalities at their respective classifiers are denoted as... Its corresponding real label is First, calculate the average binary cross-entropy (BCE) loss for each modality within the current batch:

[0066] At the same time, the LCMA module outputs a modal attention weight vector for each sample. This describes the relative importance of the three modes in the current sample. To improve training stability, a weighted smoothing process is applied to the batch dimension, i.e., the attention vectors of all samples are averaged to obtain scalarized mode weights:

[0067] Based on the above formula The three-modal classification losses can be weighted and summed according to their respective weights to obtain the modality-weighted loss:

[0068] The modality-weighted loss term enhances the gradient contribution of the modes that LCMA focuses on, making modality weight learning more physically interpretable without increasing additional parameter overhead, and further improving the recognition ability and robustness of multimodal fusion.

[0069] During model training, with the goal of minimizing the joint loss, the backpropagation algorithm is used to iteratively update all parameters of the detection model, enabling the model to gradually learn the optimal feature extraction, cross-modal attention, and fusion strategies, ultimately resulting in a convergent and stable detection model. In the inference phase, the trained model is used to process the input trimodal images, and the final prediction result is output based on the fused features, thereby accurately distinguishing between real faces and forgeries, and completing face liveness detection.

[0070] Specifically, a four-stage iterative training process can be formulated based on joint loss. Through multiple rounds of learning rate decay and small batch training, the model's convergence stability and detection performance can be guaranteed: First, the initial parameters of the model, the stochastic gradient descent optimizer (e.g., momentum 0.9, weight decay 5×10-4), the initial learning rate 0.01, and the loss tradeoff coefficient, batch size (e.g., 32) and other hyperparameters can be determined. A preset (e.g., 4) rounds of training can be executed, with the learning rate decayed before each round, and each round containing 100 training cycles. In each training cycle, the training set samples are randomly shuffled and input into the model in batches. Forward propagation is completed to obtain fused features and classification predictions. The three losses are calculated sequentially and summed to obtain the total loss. The model parameters are updated through backpropagation. After each round of training, the average classification error rate of the model is evaluated on the validation set. After all preset rounds of training are completed, the final trained model parameters are obtained, and the overall optimization is completed.

[0071] The second embodiment of the present invention provides an adaptive cross-modal face liveness detection device, see [link to relevant documentation]. Figure 2 , Figure 2 This is a schematic diagram of the adaptive cross-modal face liveness detection device provided in this embodiment, which includes: The feature extraction module 210 is configured to use a lightweight backbone network to perform feature extraction processing on the received trimodal image and output the primary feature representation of each modality.

[0072] The attention weight reconstruction module 220 is configured to perform cross-modal attention modeling on the primary feature representation and generate spatial-modal weight maps corresponding to each modality.

[0073] The cross-modal fusion module 230 is configured to perform cross-modal attention reconstruction and completion on the primary feature representation based on the spatial-modal weight graph, and output the fused features.

[0074] The detection and decision module 240 is configured to perform liveness prediction on the fused features using a lightweight classification head under the constraint of joint loss, thereby completing face liveness detection.

[0075] First, the feature extraction module 210 will be introduced.

[0076] Specifically, for each modality of the three-modal image (e.g., RGB color image, depth image, infrared image), the original input image is first processed through a convolutional layer (e.g., 3×3) to perform initial spatial downsampling and channel upscaling operations. This resizes the original images of each modality to a standard input size (e.g., 64×64). After processing by this convolutional layer, tensor features with the same number of channels (e.g., all 24) are output, resulting in shallow features of each modality with a shape of [B, 24, 32, 32] (where B is the batch size). See also... Figure 2 The processing completed in the middle stage 1.

[0077] For the shallow features of each modality with a uniform number of channels, a strategy of independent parallel processing of each modality is adopted. By stacking feature extraction units with Ghost-Generated Shuffle BlockA (GGS-BlockA) as the core, and embedding SE (Squeeze-and-Excitation) channel attention modules in each downsampling stage, a progressive downsampling and feature encoding from shallow texture to mid-level structure and then to high-level semantics is completed. This process is divided into three consecutive stages: stage 2, stage 3, and stage 4, and the RGB, Depth, and IR modal branches share the same network architecture.

[0078] After the aforementioned progressive downsampling and high semantic feature encoding, the RGB, Depth, and IR modalities each obtain a high semantic feature tensor of shape [B, 48, 4, 4], denoted as follows: This feature tensor is the primary feature representation of each mode.

[0079] The attention weight reconstruction module 220 will be described below.

[0080] Specifically, the RGB, Depth, and IR three-modal primary features output by the feature extraction module 210 are represented... As input to the LCMA (Lightweight Cross-Modal Attention Module), in order to achieve pixel-level cross-modal feature modeling, the features of the three modalities are first stacked and concatenated in the modal dimension to form a complete cross-modal feature tensor. This allows the lightweight cross-modal attention module to simultaneously capture the feature correlations and differences between the three modalities, laying the foundation for subsequent attention score calculation.

[0081] To significantly reduce computational overhead and avoid feature dimensionality redundancy while ensuring joint modeling of cross-modal features, a 1×1 convolution operation is performed on the stacked cross-modal feature tensors, compressing their channel count to 1 / 8 of the original number. This operation achieves preliminary fusion and joint representation of cross-modal features through dimensionality reduction, preserving the core feature association information of the three modalities while allowing the subsequent attention modeling process to be performed in a low-dimensional feature space, which aligns with the lightweight design principle of the entire model.

[0082] Based on the preliminary joint representation of cross-modal features in low dimension, the lightweight cross-modal attention module uses a grouped one-dimensional convolution with 4 groups to perform parallel response modeling on features in the modality-pixel dual dimension. In the output of this grouped one-dimensional convolution, the first three response channels are precisely mapped to the attention scores corresponding to the RGB, Depth, and IR modalities, which are used to quantify the discriminative contribution of each modality at each spatial location. Subsequently, the attention score response maps corresponding to these three modalities are dimensionally reshaped to obtain a feature tensor of shape B×3×K×H×W (B is the batch size, K is the parallel response dimension within the modality, and K×H×W are the height and width of the feature map), completing the spatial modeling of the three-modal attention scores and obtaining the three-modal attention score feature tensor.

[0083] For the reshaped trimodal attention score feature tensor, the mean is calculated on the independent branch of each modality to eliminate the interference of the parallel response dimension K within the modality, resulting in trimodal attention score features that only include batch, modality, and spatial location dimensions, so that the score can more accurately reflect the relative importance of each modality at each spatial location. Softmax normalization is performed on the mean-calculated trimodal attention scores to generate a pixel-level joint weight distribution, i.e., a spatial-modal weight map Q={Q RGB Q Depth Q IR (These correspond to the spatial weight subgraphs for RGB, Depth, and IR modes, respectively).

[0084] The cross-modal fusion module 230 will be described below.

[0085] Specifically, the spatial-modal weight map output by the LCMA module (containing the adaptive weights of the RGB, Depth, and IR modes at each spatial location) is combined with the trimodal primary feature representation obtained by the feature extraction network. Perform dimensional matching and alignment. Ensure that both are completely consistent in spatial size (H×W) and batch dimension (B), laying the foundation for subsequent pixel-by-pixel weighting operations.

[0086] To highlight the features of each modality in key spatial regions and suppress interference from noise regions, the primary features of each modality are first spatially adaptively weighted: the primary feature representation of each modality is multiplied pixel-by-pixel with the weight of the corresponding modality in the spatial-modal weight map. For example, the feature F of the RGB modality... RGB Q in the spatial-modal weighting graph RGB Multiplication, F of the Depth mode Depth With Q Depth Multiplication, F of IR mode IR With Q IR Multiplication. Specifically, it is represented as: ⊙ , ⊙ , ⊙ The symbol ⊙ represents pixel-wise multiplication. Spatial adaptive weighting is achieved, enabling dynamic filtering: in highly discriminative regions such as facial features and texture details, the corresponding modal has a higher weight, and the features are significantly enhanced; in ineffective regions such as background and lighting noise, the weight is lower, and the features are weakened, thereby improving the feature quality of subsequent fusion.

[0087] Based on the weighted trimodal features, a query matrix, a key matrix, and a value matrix are constructed to simulate the core logic of the attention mechanism and realize the calculation of cross-modal relevance distribution at the semantic level.

[0088] Then, the relevance scores obtained by the dot product are weighted and summed with the value matrices of the corresponding auxiliary modalities to complete the semantic-level information reconstruction, ultimately obtaining the first modality fusion feature (main modality + first auxiliary modality) and the second modality fusion feature (main modality + second auxiliary modality). This step realizes the transfer and fusion of auxiliary modality information into the main modality semantic space, making up for the information deficiency of single modality.

[0089] To further integrate the core information of the three modalities and obtain the final fused features, channel-dimensional concatenation and convolution mapping operations are performed. The first modality fused features, the second modality fused features, and the weighted main modality features are concatenated along the channel dimension, specifically: In other words, the features enhanced by the lightweight cross-modal fusion module and with deep branch features After concatenation, the data is restored to a unified 64-dimensional feature space via 1×1 convolution, yielding the final fused output. This fused output now includes the original enhanced information of the main modality, as well as cross-modal semantic fusion information with the two auxiliary modalities, further expanding the number of channels and enriching the feature dimensions.

[0090] The detection and decision module 240 will be described below.

[0091] Specifically, firstly, regarding fusion features A 1×1 convolution is used for channel mapping and dimensionality increase, mapping the number of channels from 64 to 128. This enhances the non-linear expressive power of features without expanding the receptive field. The calculation form is as follows: This provides a more discriminative feature foundation for subsequent classification. Then, a global average pooling operation is performed on the upgraded fused features to aggregate spatial dimensionality information into a global feature representation, as follows: This approach enables the model to gain a holistic semantic understanding of the entire face region while significantly reducing feature dimensionality. To prevent overfitting during training, Dropout (rate = 0.3) is applied to the global feature representation to suppress overfitting and enhance the model's generalization ability in unknown samples and complex scenes. Next, a fully connected layer is used to map the processed global features to a two-dimensional output space. These correspond to two categories: living beings and attackers; then, through Softmax normalization, the output is converted into a probability distribution. The model predicts whether the current sample is a live or an attack.

[0092] For loss optimization, a joint loss consisting of three sub-losses is used for end-to-end training.

[0093] During model training, with the goal of minimizing the joint loss, the backpropagation algorithm is used to iteratively update all parameters of the detection model, enabling the model to gradually learn the optimal feature extraction, cross-modal attention, and fusion strategies, ultimately resulting in a convergent and stable detection model. In the inference phase, the trained model is used to process the input trimodal images, and the final prediction result is output based on the fused features, thereby accurately distinguishing between real faces and forgeries, and completing face liveness detection.

[0094] Furthermore, the adaptive cross-modal face liveness detection method provided in this embodiment was simulated and verified. Specifically, in terms of experimental setup, three publicly available authoritative datasets—CASIA-SURF, CASIA-SURF CeFA, and CASIA-FASD—were selected for testing. The average classification error rate, false positive rate for attack samples, false positive rate for real samples, true positive rate under a fixed false positive rate, error rate, and semi-total error rate were used as performance evaluation metrics. The model's lightweight nature was measured by the number of parameters and floating-point operations. The experiments were conducted using the PyTorch framework, with input images uniformly scaled to 64×64. A stochastic gradient descent optimizer and a cosine annealing learning rate strategy were employed for end-to-end training under joint loss constraints.

[0095] In terms of effectiveness verification and comparative analysis, three sets of comparative experiments were conducted on input resolution, modality combination, and fusion strategy to determine that 64×64 input, trimodal input, and feature-level fusion were the optimal configurations. Module ablation experiments showed that the ghost generation shuffling module A, squeezed excitation channel attention, lightweight cross-modal attention module, and weight-guided lightweight cross-modal fusion module all continuously improved model performance while maintaining extremely low computational overhead. On the CASIA-SURF dataset, compared with mainstream lightweight liveness detection methods such as MobileNetV2, GhostNet, ShuffleNetV2, FeatherNet, and FaceBagNet, this embodiment achieved an average classification error rate of 0.064% and a true positive rate of 99.86% with a fixed false positive rate, demonstrating the best overall performance, with an ultra-lightweight complexity of 0.14M parameters and 0.0065G floating-point operations.

[0096] In terms of generalization and robustness testing, the proposed method was evaluated on the CASIA-SURF CeFA dataset under challenging protocols involving cross-race and cross-attack types. The results show that the proposed method maintains a stable low error rate even in complex cross-domain scenarios. On the CASIA-FASD dataset, which only contains the RGB modality, the model also achieved excellent results with an error rate of 1.08% and a half-total error rate of 1.27%, validating the effectiveness of the backbone network in single-modality scenarios. Furthermore, t-SNE feature visualization visually demonstrates that as training progresses, the feature distributions of live samples and attack samples gradually form a clearly separable cluster structure, further confirming that the model can effectively learn discriminative feature representations.

[0097] In summary, multiple experimental results fully verify the comprehensive advantages of the lightweight adaptive cross-modal face liveness detection method proposed in this embodiment in terms of detection accuracy, computational efficiency, generalization ability and robustness. It can achieve reliable face liveness detection in resource-constrained environments and has strong engineering practical value.

[0098] In summary, compared with the prior art, this embodiment has the following advantages: With a high degree of lightweight design and extremely low resource consumption, the backbone network is built based on the improved ShuffleNetV2 and Ghost feature generation mechanism. Combined with the lightweight module design, the model has only 0.14M parameters and 0.0065G floating-point operations, which is far lower than conventional multimodal face liveness detection networks. It can be efficiently deployed in resource-constrained scenarios such as mobile devices and edge devices.

[0099] Strong cross-modal fusion adaptability: A lightweight cross-modal attention module and a weight-guided fusion module are proposed, which can dynamically allocate the weights of each modality at different spatial locations, automatically enhance discriminative modal features and suppress noise information, and solve the problem that traditional fixed fusion strategies cannot adapt to complex attacks and scene changes.

[0100] Outstanding detection accuracy and robustness: Achieves an average classification error rate of 0.064% on the CASIA-SURF dataset, with a true positive rate of 99.86% under a fixed low false positive rate. It maintains stable performance under complex protocols such as cross-race, cross-attack type, and cross-modal, and has stronger generalization ability to unknown attacks.

[0101] Good feature discriminativeness and interpretability: The joint loss simultaneously supervises the classification results, spatial response distribution and modality weight allocation, enabling the model to focus on key facial regions; t-SNE visualization shows high feature separability between live and attack samples, the feature learning process is clear and the results are more reliable.

[0102] Strong modal compatibility and scalability: Supports RGB, depth and infrared three-modal input, with optimal performance under multimodal conditions, while also being compatible with single-modal RGB input and maintaining good detection effect, adapting to actual application scenarios with different sensor configurations.

[0103] A third embodiment of the present invention also provides an electronic device, the electronic device comprising: At least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the adaptive cross-modal face liveness detection method of any of the foregoing embodiments.

[0104] The fourth embodiment of the present invention also provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the adaptive cross-modal face liveness detection method described in any of the foregoing embodiments.

[0105] The fifth embodiment of the present invention also provides a computer program product, which includes a computing program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions that, when executed by a computer, cause the computer to perform the adaptive cross-modal face liveness detection method of any of the foregoing embodiments.

[0106] The sixth embodiment of the present invention also provides a computer program, which includes program instructions that, when executed by a computer, cause the computer to perform the adaptive cross-modal face liveness detection method of any of the foregoing embodiments.

[0107] Figure 3 The diagram illustrates a method or device 1000 that can implement embodiments of the present invention. In some embodiments, it may include more or fewer devices than illustrated. In some embodiments, it may be implemented using a single or multiple devices. In some embodiments, it may be implemented using cloud-based or distributed devices.

[0108] like Figure 3 As shown, device 1000 includes a processor 1001, which can perform various appropriate operations and processes based on programs and / or data stored in read-only memory (ROM) 1002 or programs and / or data loaded from storage portion 1008 into random access memory (RAM) 1003. Processor 1001 may be a multi-core processor or may contain multiple processors. In some embodiments, processor 1001 may include a general-purpose main processor and one or more special coprocessors, such as a central processing unit (CPU), graphics processing unit (GPU), neural network processor (NPU), digital signal processor (DSP), etc. Various programs and data required for the operation of device 1000 are also stored in RAM 1003. Processor 1001, ROM 1002, and RAM 1003 are interconnected via bus 1004. Input / output (I / O) interface 1005 is also connected to bus 1004.

[0109] The processor and memory described above are used together to execute programs stored in the memory. When the program is executed by a computer, it can implement the methods, steps, or functions described in the above embodiments.

[0110] The following components are connected to I / O interface 1005: an input section 1006 including a keyboard, mouse, touchscreen, etc.; an output section 1007 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN card, modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to I / O interface 1005 as needed. A removable medium 1011, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 1010 as needed so that computer programs read from it can be installed into storage section 1008 as needed. Figure 3 The diagram only shows a portion of the components and does not imply that the device 1000 only includes... Figure 3 The components shown.

[0111] The systems, devices, modules, or units described in the above embodiments can be implemented by a computer or its associated components. The computer may be, for example, a mobile terminal, smartphone, personal computer, laptop computer, in-vehicle human-machine interface device, personal digital assistant, media player, navigation device, game console, tablet computer, wearable device, smart TV, Internet of Things system, smart home, industrial computer, server, or a combination thereof.

[0112] Although not shown, in this embodiment of the invention, a computer-readable storage medium is provided having a computer program / instruction stored thereon, which, when executed by a processor, implements the adaptive cross-modal face liveness detection method described in Embodiment 1.

[0113] Storage media in embodiments of the present invention include articles that are permanent and non-permanent, removable and non-removable, capable of storing information by any method or technology. Examples of storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0114] Although not shown, embodiments of the present invention also provide a computer program product, including: a computer program / instructions that, when executed by a processor, implement the adaptive cross-modal face liveness detection method described in Embodiment 1.

[0115] The methods, programs, systems, apparatuses, etc., in embodiments of the present invention can be executed or implemented in one or more networked computers, or practiced in a distributed computing environment. In the embodiments of this specification, in these distributed computing environments, tasks can be performed by remote processing devices connected via a communication network.

[0116] Those skilled in the art will understand that the embodiments described in this specification can be provided as methods, systems, or computer program products. Therefore, those skilled in the art will realize that the functional modules / units or controllers and related method steps described in the above embodiments can be implemented in software, hardware, or a combination of both.

[0117] Unless explicitly stated otherwise, the actions or steps of the methods and procedures described in the embodiments of the present invention do not necessarily have to be performed in a specific order and can still achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0118] This document describes several embodiments of the present invention; however, for the sake of brevity, the descriptions of the embodiments are not exhaustive, and identical or similar features or parts between the embodiments may be omitted. In this document, "one embodiment," "some embodiments," "example," "specific example," or "some examples" refers to embodiments applicable to at least one, but not all, of the present invention. The above terms do not necessarily refer to the same embodiments or examples. Without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described herein, as well as the features of the different embodiments or examples.

[0119] The exemplary systems and methods of the present invention have been specifically shown and described with reference to the above embodiments, which are merely examples of the best mode for implementing the systems and methods. Those skilled in the art will understand that various changes can be made to the embodiments of the systems and methods described herein without departing from the spirit and scope of the invention as defined in the appended claims when implementing the systems and / or methods.

Claims

1. An adaptive cross-modal face liveness detection method, characterized in that, include: A lightweight backbone network is used to process the received trimodal images to obtain the primary feature representations of each modality; The primary feature representation is processed by a lightweight cross-modal attention module to obtain the spatial-modal weight map corresponding to each modality; Based on the aforementioned spatial-modal weight graph, a lightweight cross-modal fusion module is used to perform cross-modal attention reconstruction and completion on the primary feature representation to obtain fused features; Under the constraint of joint loss, a lightweight classification head is used to perform liveness prediction on the fused features to complete face liveness detection.

2. The adaptive cross-modal face liveness detection method according to claim 1, characterized in that, The process of using a lightweight backbone network to process the received trimodal images to obtain primary feature representations for each modality includes: Through convolutional layers, the original input image of each modality of the three modal images is initially spatially downsampled to obtain shallow features of each modality with the same number of feature channels; The shallow features of each modality are progressively downsampled by the group shuffling module and the squeeze excitation channel attention module to complete the high semantic feature encoding and obtain the primary feature representation.

3. The adaptive cross-modal face liveness detection method according to claim 1, characterized in that, The step of processing the primary feature representation through a lightweight cross-modal attention module to obtain the space-modal weight map corresponding to each modality includes: The primary feature representations of each modality are stacked along the modality dimension to obtain a cross-modality feature tensor; The number of channels in the cross-modal feature tensor is compressed using 1×1 convolution to obtain a preliminary joint representation of the cross-modal features; By grouping one-dimensional convolutions, parallel response modeling is performed on the preliminary joint features to obtain a three-modal attention scoring feature tensor; The mean of the attention score feature tensor of the three modalities is calculated, and the mean is normalized to obtain the space-modality weight map. Wherein, the sum of the three modal weights at any spatial location on the spatial-modal weight graph is 1.

4. The adaptive cross-modal face liveness detection method according to claim 1, characterized in that, Based on the aforementioned spatial-modal weight graph, a lightweight cross-modal fusion module is used to perform cross-modal attention reconstruction and completion on the primary feature representation to obtain fused features, including: The primary feature representation of each modality is multiplied pixel-by-pixel with the weight of the corresponding modality in the spatial-modal weighting graph to complete spatial adaptive weighting; The weighted main modal features are mapped to obtain the query matrix; The weighted first and second auxiliary modal features are mapped to the corresponding key and value matrices, respectively. By performing cross-modal correlation distribution calculation on the main modal feature, the first auxiliary modal feature, and the second auxiliary modal feature through dot product attention, information reconstruction at the semantic level is completed, and the first modal fusion feature and the second modal fusion feature are obtained. The first modality fusion feature, the second modality fusion feature, and the weighted main modality feature are concatenated along the channel dimension, and then the concatenated features are mapped to a unified feature space through convolution to obtain the fusion feature.

5. The adaptive cross-modal face liveness detection method according to claim 1, characterized in that, Under the constraint of the joint loss function, the lightweight classification head is used to perform liveness prediction on the fused features to complete face liveness detection, including: mapping and increasing the dimensionality of the fused features through 1×1 convolution; Global average pooling is performed on the fused features after dimensionality upscaling to obtain a global feature representation; The global feature representation is subjected to overfitting suppression processing; Using a fully connected layer, the global feature representation after overfitting suppression is mapped to a 2D feature space. The global feature representation after the 2D feature space mapping is then processed by a normalization operation to obtain the probability distributions corresponding to the two classes: liveness and attack. Based on the liveness probability distribution, the error between the predicted value and the true label of the sample is calculated to obtain the binary classification cross-entropy loss. The difference between the fused feature map corresponding to the fused feature and the structured target map is measured at the pixel level to obtain the pixel-level fusion weighted loss; For the space-modal weight graph, the modal-level weight vector is obtained by taking the mean value in the spatial dimension. The binary cross-entropy loss of the three-modal branches is calculated. The binary cross-entropy loss is dynamically weighted and summed based on the modal-level weight vector to obtain the modal-weighted loss. The binary classification cross-entropy loss, the pixel-level fusion weighted loss, and the modality weighted loss are weighted and summed to obtain the joint loss; During training, the joint loss is minimized using the backpropagation algorithm, and the detection model parameters are iteratively updated to obtain the trained detection model. The trained detection model is used to predict liveness using the fused features, thus completing face liveness detection.

6. An adaptive cross-modal face liveness detection device, characterized in that, include: The feature extraction module is configured to use a lightweight backbone network to perform feature extraction processing on the received trimodal images and output the primary feature representations of each modality. The attention weight reconstruction module is configured to perform cross-modal attention modeling processing on the primary feature representation to generate a space-modal weight map corresponding to each modality. The cross-modal fusion module is configured to perform cross-modal attention reconstruction and completion on the primary feature representation based on the space-modal weight graph, and output fused features; The detection and decision module is configured to perform liveness prediction on the fused features using a lightweight classification head under the constraint of joint loss, thereby completing face liveness detection.

7. The adaptive cross-modal face liveness detection device according to claim 6, characterized in that, The feature extraction module includes: an initial downsampling unit and a progressive encoding unit; The initial downsampling unit is a convolutional layer, used to perform initial spatial downsampling on the original input image of each modality in the three modal images, and output shallow features of each modality with the same number of feature channels; The progressive encoding unit includes a group shuffling module and a squeezed excitation channel attention module, which are used to progressively downsample the shallow features of each modality and complete the high semantic feature encoding, and output the primary feature representation; The group shuffling module is a lightweight feature extraction module that combines ghost feature generation mechanism, residual connection and channel shuffling. It completes feature extraction through channel mapping, feature expansion, channel splicing and shuffling.

8. The adaptive cross-modal face liveness detection device according to claim 6, characterized in that, The attention weight reconstruction module includes: a feature stacking unit, a channel compression unit, a response modeling unit, and a weight normalization unit; The feature stacking unit is used to stack the primary feature representations of each modality in the modality dimension to generate a cross-modality feature tensor; The channel compression unit is used to compress the number of channels of the cross-modal feature tensor by a preset ratio using 1×1 convolution to obtain a preliminary joint representation of the cross-modal features; The response modeling unit is used to perform parallel response modeling on the preliminary joint representation through grouped one-dimensional convolution with a preset number of groups, and generate a three-modal attention score. The weight normalization unit is used to calculate the mean and normalize the attention scores of the three modalities, and output the spatial-modal weight map, wherein the sum of the three modal weights at any spatial location on the spatial-modal weight map is 1.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the adaptive cross-modal face liveness detection method according to any one of claims 1 to 5.

10. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer instructions for causing the computer to perform the adaptive cross-modal face liveness detection method according to any one of claims 1 to 5.

Images