Face living body detection method and device, electronic equipment and medium

Abstract

The invention provides a human face living body detection method and device, electronic equipment and a medium, and the method comprises the steps: carrying out the modal image obtaining of a human face image, and obtaining an RGB modal image, an infrared modal image and a depth modal image of a human face; respectively carrying out feature processing on each modal image through a self-adaptive multi-modal Transform feature extractor to obtain key feature expression of the intensified modal image; performing selective enhancement on the key feature expression by using an expert enhancement projection module to obtain optimization features of each mode; performing cross-modal feature interaction on the optimization features among different modals by using the optimization features of the preset priority modals as guidance to obtain modal fusion features, and splicing the modal fusion features to obtain global fusion features; and judging whether the human face is a living body or not according to the global fusion feature to obtain a judgment result.

Description

Technical Field

[0001] This invention relates to the field of face liveness detection technology, specifically to a face liveness detection method, device, electronic device, and medium. Background Technology

[0002] With the deep integration of artificial intelligence and computer vision technologies, facial recognition systems have been widely applied in key scenarios such as identity authentication, financial payments, public security, and smart terminal unlocking, becoming a core technological support for ensuring the security and convenience of these scenarios. However, the security boundaries of facial recognition systems are facing a severe challenge from forgery attacks. Attackers can use low-cost methods such as printing facial photos, playing facial videos, and creating 3D silicone masks to simulate real facial features to deceive the recognition system, leading to security incidents such as identity theft, financial losses, and information leaks. Therefore, Face Anti-Spoofing (FAS), as a key technology for distinguishing between real and fake faces, directly determines the security level of a facial recognition system based on its performance.

[0003] Early face liveness detection technologies primarily relied on a single modality (especially the visible light RGB modality). These methods extract texture details (such as light spots and moiré patterns) or color distribution features from RGB images to determine authenticity, offering advantages such as low data acquisition costs and simple algorithm implementation. However, in complex real-world scenarios, the single RGB modality has significant limitations: firstly, it is greatly affected by lighting conditions; in environments with strong backlighting, low illumination, or non-uniform lighting, texture features are easily distorted, leading to a significant drop in detection accuracy; secondly, when faced with new forgery methods such as high-definition printed photos and highly realistic screen playback, the RGB modality struggles to capture the essential differences in physical properties (such as three-dimensional structure and thermal radiation characteristics) between forgeries and real faces, resulting in severely insufficient detection robustness.

[0004] To overcome the limitations of single-modality facial liveness detection, multimodal face liveness detection technology has gradually become a research hotspot. This type of technology integrates multi-source modal data such as RGB, depth, and infrared (IR) to construct facial feature representations from multiple dimensions. RGB modality provides surface texture and color information, aiding in the identification of planar attacks such as printed or screen-based attacks; depth modality accurately depicts the three-dimensional geometric structure of the face, effectively distinguishing the spatial morphological differences between planar forgeries and genuine faces; IR modality possesses illumination invariance, allowing stable operation in low-light or no-light environments, while also capturing facial thermal radiation distribution characteristics, further enhancing the ability to detect three-dimensional forgeries such as 3D masks. Theoretically, the complementarity of multimodal data can significantly improve the robustness of liveness detection, but existing multimodal detection methods still face three major technical bottlenecks: First, the intramodal feature representation capability is insufficient. Existing methods mostly use traditional convolutional neural networks or fixed-weight attention mechanisms to extract single-modal features. They cannot dynamically adjust the feature weight allocation according to the quality of modal data (such as IR image noise intensity, depth data accuracy) or scenario requirements (such as low-power terminal scenarios, high-precision security scenarios). As a result, key discrimination information (such as abnormal local thermal radiation of the face, three-dimensional structural details) is masked by redundant information, and the feature discrimination power is limited.

[0005] Secondly, intermodal fusion is inefficient. Current multimodal fusion methods often employ simple strategies such as early concatenation followed by later weighting, failing to fully consider the differences in physical characteristics between different modalities (e.g., the feature scale differences between RGB and IR, and the semantic alignment difficulties between Depth and RGB). This easily leads to modal information conflicts or redundant enhancements. For example, when directly concatenating RGB and Depth features, the fusion results in semantic confusion due to the mismatch between the dimensions and semantic spaces of the two types of features, which actually reduces detection accuracy. Although some methods introduce cross-modal attention, they fail to establish deep dependencies between modalities, making it difficult to fully exploit complementary information.

[0006] Third, it is difficult to balance the generalization ability and inference efficiency of the model. On the one hand, existing high-performance multimodal models mostly rely on complex network structures (such as deep Transformers and multi-stream branch CNNs), which require large-scale labeled data for training. Their performance degrades significantly in generalization scenarios such as cross-race, cross-device, and unknown attack types. On the other hand, redundant computing modules (such as multiple sets of parallel feature extractors) introduced in pursuit of high accuracy result in a large number of model parameters and high inference latency, making it difficult to adapt to low-computing-power application scenarios such as smart terminals and embedded devices.

[0007] Therefore, how to overcome the existing technological bottlenecks and provide a face liveness detection technology that accurately extracts intra-modal features and efficiently fuses inter-modal features has become a key technical problem that urgently needs to be solved. Summary of the Invention

[0008] In view of this, the present disclosure provides a face liveness detection method, apparatus, electronic device and medium, which at least partially solves the problems existing in the prior art.

[0009] In a first aspect, embodiments of this disclosure provide a face liveness detection method, which includes: Modal image acquisition is performed on the face image to obtain the RGB modal image, infrared modal image and depth modal image of the face respectively; The key feature representations of the enhanced modal images are obtained by performing feature processing on each modal image through an adaptive multimodal Transformer feature extractor. The key feature representations are selectively enhanced using an expert-enhanced projection module to obtain optimized features for each modality; Guided by the optimization features of the preset priority modalities, cross-modal feature interaction is performed on the optimization features between different modalities to obtain modal fusion features. Then, the modal fusion features are spliced ​​together to obtain global fusion features. The global fusion features are used to determine whether a face is a live object, and the determination result is obtained.

[0010] According to a specific implementation of this disclosure, the step of performing feature processing on each modal image using an adaptive multimodal Transformer feature extractor to obtain key feature representations of the enhanced modal image includes: The query, key, and value vectors of each modality image are subjected to dimensional transformation and feature rearrangement. The weights of the query, the key, and the value vector are adjusted using learnable parameters; The attention value of each modality image is obtained by performing a dot product operation on the weighted query vector and key vector. The attention values ​​are normalized and residuals are connected using the Softmax function to obtain the attention score for each modality image; The key feature representation is obtained based on the attention score and the value vector.

[0011] According to a specific implementation of this disclosure, the selective enhancement of the key feature representation using an expert-enhanced projection module to obtain optimized features for each modality includes: By using a learnable router network, the matching degree between each key feature representation vector and each expert network is calculated, and a subset of target expert networks is selected according to a preset strategy. The key feature representation vectors assigned by the expert network subset are processed in parallel, and the outputs of the expert network subset are weighted and summed according to the weights assigned by the router to obtain the optimized features.

[0012] According to a specific implementation of this disclosure, the step of using the optimization features of a preset priority modality as guidance to perform cross-modal feature interaction on the optimization features between different modalities to obtain modal fusion features, and then concatenating the modal fusion features to obtain global fusion features, includes: Using infrared modes as guidance, the input feature X is processed respectively. IR X rgb X Depth Perform linear transformation Obtain the query vector Q IR Key vector KRGB、 K Depth AND value vector V RGB V Depth ;

[0013] Next, the attention weights of the infrared mode on the RGB mode are calculated using a dot product attention mechanism, where d represents the dimension of the key vector:

[0014] Obtain the fusion result of IR mode guiding RGB mode ; Fusion results of infrared mode-guided depth modes ,

[0015] right and The fusion process is performed to obtain the final fusion characteristics:

[0016] According to a specific implementation of an embodiment of this disclosure, the step of extracting features from the image block using a residual Swing Transformer module to obtain a first-stage feature map, extracting features from the first-stage feature map using the residual Swing Transformer module to obtain a second-stage feature map, extracting features from the second-stage feature map using the residual Swing Transformer module to obtain a third-stage feature map, and extracting features from the third-stage feature map using the residual Swing Transformer module to obtain a fourth-stage feature map includes: use A basic Swin Transformer module extracts features from the image patch to obtain the first-stage feature map, and then uses... A basic Swing Transformer module extracts features from the first-stage feature map to obtain the second-stage feature map, using... The basic Swin Transformer module extracts features from the second-stage feature map to obtain the third-stage feature map, using... A basic Swin Transformer module performs feature extraction on the third-stage feature map to obtain the fourth-stage feature map; in,

[0017]

[0018]

[0019] ; in, For the image block, This is the first stage feature map. This is the second-stage feature map. This is the third stage feature map. This is the feature map for the fourth stage.

[0020] According to a specific implementation of an embodiment of this disclosure, the expert-enhanced projection module includes: Router unit, multiple expert network units, and weighted fusion unit; The router unit is used to calculate the matching degree between the key feature representation and each expert network, and to select and activate a subset of expert networks using a Top-k strategy; The multiple expert network units are independent fully connected networks used to process the assigned key feature representations; The weighted fusion unit performs a weighted summation of the outputs of the activated expert network subset based on the normalized weights output by the router unit.

[0021] According to a specific implementation of an embodiment of this disclosure, the collaborative fusion module includes: a dual-branch fusion unit and a feature splicing unit; The dual-branch fusion unit constructs cross-modal attention interaction channels for IR-RGB and IR-Depth respectively, calculates inter-modal attention weights, and completes feature enhancement; The feature splicing unit is used to splice the dual-branch fusion results along the channel dimension to generate global fusion features.

[0022] According to a specific implementation of this disclosure, the router unit uses linear transformation to calculate the matching degree, and the expert network unit is a two-layer or higher feedforward neural network, with the parameters of each expert network being independent of each other.

[0023] Secondly, embodiments of this disclosure provide a face liveness detection device, which includes: The modal image acquisition module is used to acquire modal images of a face, obtaining RGB modal images, infrared modal images, and depth modal images of the face respectively; The adaptive multimodal Transformer feature extraction module is used to perform feature processing on each modal image to obtain the key feature representation of the enhanced modal image; An expert-enhanced projection module is used to selectively enhance the key feature representations to obtain optimized features for each modality. The collaborative fusion module is used to use the optimization features of the preset priority modal as a guide to perform cross-modal feature interaction on the optimization features between different modalities to obtain modal fusion features, and then splice the modal fusion features to obtain global fusion features; The detection output module is used to determine whether a face is a live object based on the global fusion features and output the determination result.

[0024] Thirdly, embodiments of this disclosure provide an electronic device, the electronic device comprising: At least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the above-described face liveness detection method.

[0025] Fourthly, embodiments of this disclosure provide a non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the aforementioned face liveness detection method.

[0026] In summary, compared with the prior art, this embodiment has the following advantages: 1. By introducing learnable attention weights through the Adaptive Multimodal Transformer Feature Extractor (AMTE), the attention level of different feature channels in each modality (RGB, IR, Depth) can be dynamically adjusted, which can accurately focus on key information that is strongly related to liveness detection.

[0027] 2. The collaborative fusion module constructs a dual-branch fusion architecture guided by IR modality, realizing directional feature interaction of IR-RGB and IR-Depth respectively. Compared with traditional simple splicing fusion, it can effectively eliminate semantic inconsistencies and information redundancy between modalities.

[0028] 3. While ensuring high accuracy, the model achieves efficient inference through multi-stage optimization, and the expert network selection mechanism of the EAP module reduces invalid calculations.

[0029] In summary, the embodiments of this application achieve high-precision, high-robustness, and high-efficiency multimodal face liveness detection through modular innovation and architecture optimization. It can be widely used in security-sensitive scenarios such as identity authentication, financial payment, and public security. While improving system security, it also takes into account user experience and deployment costs, and has significant technical and practical value. Attached Figure Description

[0030] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings. Here, the illustrative embodiments and descriptions of this invention are used to explain the invention, but are not intended to limit the invention.

[0031] The term "comprising" and its variations as used herein signify open inclusion, i.e., "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "at least partially based on". The terms "one example embodiment" and "one embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0032] Figure 1 This application provides a flowchart illustrating a face liveness detection method. Figure 2 A schematic diagram of an adaptive multimodal Transformer extractor provided in an embodiment of this application; Figure 3 This is a schematic diagram of collaborative fusion attention calculation provided in an embodiment of this application; Figure 4 This is a schematic diagram of the face liveness detection device provided in this embodiment; Figure 5 A schematic diagram of a collaborative fusion module provided in an embodiment of this application; Figure 6 An exemplary structural diagram of a device capable of implementing the method according to an embodiment of the present invention is shown. Detailed Implementation

[0033] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0034] The following specific examples illustrate the implementation of this disclosure. Those skilled in the art can easily understand other advantages and effects of this disclosure from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. This disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this disclosure. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0035] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.

[0036] It should also be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this disclosure. The drawings only show the components related to this disclosure and are not drawn according to the number, shape and size of the components in actual implementation. In actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0037] Furthermore, specific details are provided in the following description to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that the described aspects can be practiced without these specific details.

[0038] Please see Figure 1 The present application provides a flowchart of a face liveness detection method, which can be executed by an electronic device, specifically by one or more processors of the electronic device, and implements the following steps: S101, Modal image acquisition is performed on the face image to obtain the RGB modal image, infrared modal image and depth modal image of the face respectively.

[0039] First, electronic devices acquire multi-dimensional modal images of the target face to construct a comprehensive facial feature data source. The specific process is as follows: Three types of modal images are acquired for the three core dimensions of the face: visual appearance, thermal radiation characteristics, and spatial geometry. Firstly, RGB modal images are captured using visible light imaging devices (such as conventional RGB cameras). These images, based on the three primary colors of red, green, and blue light, can fully present the surface texture details of the face (such as skin texture, pore distribution, facial blemishes), color distribution characteristics (such as skin color differences, lip color variations), and local visual markers (such as reflections from glasses, moiré patterns or printing marks from forgeries). This provides visual texture evidence for subsequently distinguishing real faces from planar forgery attacks (such as printed photos or screen replays). Secondly, infrared imaging devices (such as near-infrared or thermal infrared cameras) are used to acquire... This device integrates infrared modal images, enabling it to sense infrared radiation signals from the face itself and the environment, unaffected by visible light intensity (such as strong light, backlight, and low illumination). It can clearly present the thermal distribution contours of the face (such as the thermal radiation differences of the eyes, nose, and lips), effectively avoiding interference from lighting changes on feature extraction. It can also identify the essential differences in thermal conduction characteristics between 3D fake masks and real faces. Thirdly, by acquiring depth modal images through depth sensing devices (such as structured light cameras and ToF cameras), it calculates the spatial distance between each point on the face and the device, generating a depth image containing the three-dimensional coordinate information of the face. This can accurately depict the three-dimensional geometric structure of the face (such as the height of the bridge of the nose, the contour of the cheekbone, and the jawline), thereby quickly distinguishing the spatial morphological differences between planar fakes (such as printed photos and screen images) and real faces, providing key geometric feature support for combating three-dimensional fake attacks such as 3D masks.

[0040] S102, the adaptive multimodal Transformer feature extractor is used to perform feature processing on each modal image to obtain the key feature representation of the enhanced modal image.

[0041] First, we define the feature vector of the electronic device as x for the RGB modal image, y for the infrared modal image, and z for the depth modal image. Let Q represent the Query, Key, and Value of the feature vector x. x , K x V x Let Q represent the Query, Key, and Value of the feature vector y. y , K y V y Let Q represent the Query, Key, and Value of the feature vector z. z , K z V z .

[0042] Optionally, the query, key, and value vectors of each modality image undergo dimensionality transformation and feature rearrangement; the weights of the query, key, and value vectors are adjusted using learnable parameters; a dot product operation is performed on the weighted query vector and key vector to obtain the attention value for each modality image; the attention value is normalized and residual-connected using the Softmax function to obtain the attention score for each modality image; and key feature representations are obtained based on the attention score modulus and value vector. It should be noted that the key feature representations are feature tensors of each enhanced modality image.

[0043] See details Figure 2 , Figure 2 A schematic diagram of an adaptive multimodal Transformer extractor provided in an embodiment of this application is shown below. Figure 2 As shown, the electronic device responds to Q x , K x V x Perform a dimensional transformation to match subsequent calculations, first on Q. x , K x V x The following formula is used for feature rearrangement and expansion:

[0044] The input dimensions are (B, N, C), where B is the batch size, N is the sequence length, and C is the number of channels. For dimension rearrangement operations, Reshape the view. For Q x , K x V x The purpose of feature rearrangement and expansion is to make computation more efficient.

[0045] Next, we initialize a learnable parameter. ,in They correspond to Q respectively x , K x V x The weighting parameters for Q x , K x V x Adaptive adjustments are made to optimize the importance of intramodal features, and this optimization is performed during training using gradient descent. During model training, the network automatically calculates gradients for these parameters and updates them via backpropagation. This allows the model to adaptively learn task-relevant intramodal interaction weights, more accurately capturing the importance of different features within a modality.

[0046] Then, the attention weights are normalized using the Softmax function to ensure that the sum of the weights is 1. The Softmax operation makes the feature weights a probability distribution, where each element represents the importance of the corresponding attention value in the fusion. Next, the weights of each feature component are... With Q x , K x V x Vector multiplication is performed, followed by a residual connection to adaptively enhance the feature representation within the modality. The calculation method is as follows:

[0047] The residual adjustment mechanism can not only dynamically control the contribution of each feature within a mode, but also enhance the interaction between features while preserving the original information, thereby optimizing the overall attention computation effect.

[0048] Next calculation Each vector in The attention value for a single modality is obtained by taking the dot product of each vector, which is essentially matrix multiplication. and The transpose of, where d k yes The dimension is then transformed into a matrix distributed in the interval [0,1] using the Softmax function, and then compared with... Performing matrix multiplication yields the final attention score for the adaptive multimodal Transformer. The attention score matrix is ​​then normalized using the Softmax function to obtain the normalized attention weight matrix. The specific calculation formula is as follows:

[0049] Finally, the normalized attention weight matrix is ​​multiplied with the value vector V to filter key features in V by weight, thereby obtaining the feature tensors of each modal image after attention enhancement, which is the key feature representation of the enhanced modal image.

[0050] After calculating the result of mode x through the adaptive multimodal Transformer extractor, Q is... y , K y V y and Q z ,K z V z Following the above steps, calculate the results of mode y and mode z after passing through the adaptive multimodal Transformer extractor.

[0051] S103 utilizes an expert-enhanced projection module to selectively enhance the representation of key features, thereby obtaining optimized features for each modality.

[0052] Electronic devices can utilize learnable router networks to calculate the matching degree between each key feature representation vector and each expert network, and select a subset of target expert networks according to a preset strategy. The assigned key feature representation vectors are processed in parallel by a subset of the expert network, and the optimized features are obtained by weighted summation of the outputs of the subset of the expert network according to the weights assigned by the router.

[0053] Specifically, Expert Augmented Projection (EAP) is a highly efficient modular architecture that dynamically selects and activates the most relevant expert subnetworks through a gating mechanism, thereby enhancing the model's expressive power while maintaining computational efficiency. Electronic devices utilize EAP modules to replace the fixed projection layers in standard attention mechanisms, solving the problem that traditional static projection struggles to adapt to dynamic attention distributions. EAP routes input features to different expert projection matrices through a gating network, enabling feature transformations to adaptively adjust according to the attention context. This significantly enhances the feature expressive power of the attention mechanism while retaining the parameter efficiency advantages of EAP.

[0054] After processing by the adaptive multimodal Transformer extractor, the electronic device obtains a feature tensor containing spatial importance weights for image regions. Next, the electronic device inputs this feature tensor into the Expert Augmentation Projection (EPA) module for processing. In EPA, the input features first pass through a learnable router network to calculate the matching degree between each feature vector and each expert network, and dynamically selects the most relevant subset of expert networks based on a preset strategy (e.g., a Top-k strategy). The selected expert networks then process the assigned feature vectors in parallel, and finally, the outputs of each expert network in the subset are weighted and summed according to the weights assigned by the routers to obtain the optimized features.

[0055] In the EPA module, the routing mechanism makes independent routing decisions for each feature vector. First, the router uses a linear transformation to evaluate the expert matching degree of the input feature vector x, calculating the matching degree score g of the expert network. x ):

[0056] The input data is a feature tensor. Where B is the batch size, S is the sequence length, and d is the dimension of the feature vector. is the router's weight matrix, where d is the dimension of the feature vectors and N is the total number of expert networks. Let N be the bias vector, where N is the total number of expert networks.

[0057] To optimize computational efficiency while maintaining model expressiveness, electronic devices can employ a Top-k routing strategy to select experts. Specifically, the router selects experts based on the matching scores of all N expert networks. The top k experts with the highest scores are selected through Top-k operations to form an activated expert network subset I = top-k(g( x Next, the original scores of the selected expert networks are subjected to softmax normalization to obtain the final routing weights:

[0058] Among them, a i For the first i The normalized weights for each expert are normalized only in the Top-k set to ensure that only a subset of the expert networks are activated. Ultimately, these weights will be used for the weighted fusion of the subsequent expert network outputs.

[0059] The expert network is the core component of the EPA structure. Each expert network consists of a parameter-independent subnetwork, and each expert network is implemented using a two-layer feedforward neural network (MLP). The calculation process is as follows:

[0060] in, and They represent experts respectively. i The first and second layer weight matrices, h For the hidden layer dimension, and is the bias term, ReLU is the nonlinear activation function, and N is the total number of expert networks.

[0061] For each input feature vector x, the router first calculates its matching score with all experts, then selects only the top-k experts with the highest scores for computation. The final output is a weighted sum of the outputs of these activated experts:

[0062] Among them, E i ( x ) is the first i The output of an expert network, a i These are the expert weights normalized by the router using softmax.

[0063] In summary, the EPA module processes the input feature vector through an expert network and selection mechanism. Its output is composed of a weighted fusion of Top-k experts, forming a complementary structure with AMTE. AMTE enhances the expressive power within a modality by adjusting the contribution of different features. The EPA module processes the features after AMTE and enhances the adaptability and accuracy control of the attention mechanism through an expert routing mechanism, ultimately outputting the feature processing results of the attention stage.

[0064] S104. Using the optimization features of the preset priority modality as a guide, cross-modal feature interaction is performed on the optimization features between different modalities to obtain modal fusion features. Then, the modal fusion features are spliced ​​together to obtain global fusion features.

[0065] Specifically, traditional multimodal fusion methods typically employ simple feature stitching when processing RGB, infrared, and depth modal images. This approach fails to adequately consider the differences in physical properties and perceptual preferences among the different modalities. Specifically, RGB modal images excel at capturing surface forgery features such as light spot reflections, depth modal images effectively identify two-dimensional planar attacks through depth information, and IR modal images, with their illumination invariance, can focus on biometric features such as facial pose and expression. However, existing methods that directly stitch and fuse these three modalities often lead to key issues such as semantic inconsistencies and redundant enhancements in the fused features.

[0066] To address this limitation, in this embodiment, the electronic device utilizes a Collaborative Fusion module (CFM) with modal awareness guidance capabilities. This module leverages the unique structural representation advantages of infrared (IR) modal images in liveness detection, establishing feature dependencies among the RGB, IR, and Depth modalities through a directed cross-modal attention mechanism. Using the IR modality as a guiding source, its attention weight distribution with the visible light and depth modalities is calculated, thereby achieving differentiated fusion of multimodal features.

[0067] CFM employs a dual-branch fusion architecture, constructing feature interactions for IR-RGB and IR-Depth respectively. It fully leverages the illumination-invariant properties of the IR modality, using it as a semantically guided feature to dynamically adjust the feature extraction process of other modalities, effectively enhancing feature discriminative power. CFM automatically aligns modal semantic features, eliminating inconsistencies in heterogeneous feature spaces, and adaptively enhances the representational capabilities of each modality under the guidance of contextual information, thereby achieving selective fusion of complementary information between modalities. The output features of the two fusion paths are concatenated along the channel dimension to obtain a global fusion representation. This fusion mechanism enables fine-grained feature selection and ensures contextual consistency of multimodal features at the semantic level. This process, through a structured interaction method, significantly improves the efficiency of utilizing complementary information between different modalities.

[0068] The implementation process of CFM is as follows: Electronic devices represent the three modal characteristics of IR, RGB, and Depth as follows: , , For details on the collaborative integration process, please refer to [link / reference]. Figure 3 , Figure 3 This is a schematic diagram illustrating a collaborative fusion attention calculation method provided in an embodiment of this application. Figure 3 As shown. In the IR-RGB collaborative fusion process, the electronic device uses the IR mode as a guide to process the input feature X respectively. IR X rgb X Depth Perform a linear transformation to obtain the query vector Q. IR Key vector K RGB、 K Depth AND value vector V RGB、 V Depth :

[0069] Next, the attention weights of the IR mode on the RGB mode are calculated using a dot product attention mechanism, where d represents the dimension of the key vector and is used for scaling operations to stabilize the gradient.

[0070] The final result is the fusion of IR mode guided by RGB mode. Next, the electronic device calculates the fusion result of the IR mode-guided Depth mode according to the above operation. , .

[0071] After collaborative fusion enhancement, two sets of modal fusion features are obtained. and These represent the effective information extraction from the RGB and Depth modes by the IR mode, respectively. Next, the electronic device will... and The fusion process is performed to obtain the final fusion characteristics:

[0072] in, For global fusion features, Indicates a fusion operation. To guide the fusion features of RGB modes for IR modes, Characteristics of IR modes, The Cooperative Fusion Module (CFM) guides the fusion of Depth modes using IR mode-guided bidirectional cross-modal interaction, effectively promoting the completion and fusion of feature information between RGB and Depth modes. This enhances the complementarity and discriminative power of multimodal features while maintaining computational efficiency.

[0073] S105, determine whether the face is a live object based on the global fusion features, and obtain the judgment result.

[0074] In this embodiment, the electronic device feeds the globally fused features into a classification network for discrimination. The classification network typically consists of fully connected layers and activation functions (such as the sigmoid function). First, the high-dimensional globally fused features are mapped into a one-dimensional feature vector through the fully connected layers. Then, the sigmoid function compresses the vector output value to the [0,1] interval. If the output value is close to 1, the current face is determined to be a live object; if the output value is close to 0, it is determined to be an attack sample (such as a printed photo, a 3D mask, etc.).

[0075] The second embodiment of the present invention provides a face liveness detection device, see [link to documentation]. Figure 4 , Figure 4 This is a schematic diagram of the face liveness detection device provided in this embodiment, which includes: The modal image acquisition module 410 is used to acquire modal images of a face image, and obtain the RGB modal image, infrared modal image and depth modal image of the face respectively.

[0076] The adaptive multimodal Transformer feature extraction module 420 is used to perform feature processing on each modal image to obtain the key feature representation of the enhanced modal image.

[0077] The expert-enhanced projection module 430 is used to selectively enhance the representation of key features to obtain optimized features for each modality. The collaborative fusion module 440 is used to use the optimization features of a preset priority mode as a guide to perform cross-modal feature interaction on the optimization features between different modes to obtain modal fusion features, and then splice the modal fusion features to obtain global fusion features; The detection output module 450 is used to determine whether a face is a live object based on global fusion features and output the determination result.

[0078] First, the modal image acquisition module 410.

[0079] Specifically, the modal image acquisition module collects facial images in three key modalities: RGB modal images, infrared modal images, and depth modal images, providing high-quality data sources for subsequent feature extraction and fusion. RGB modal images can be acquired using ordinary visible light cameras, capturing surface details such as facial texture, color, light spots, and moiré patterns. This adapts to normal lighting environments, is low-cost, and simple to implement. Infrared modal images can be acquired using infrared cameras, are unaffected by visible light intensity, and can work stably in low-light or no-light environments. They also sense facial thermal radiation characteristics, possessing natural anti-interference capabilities. Depth modal images are acquired using depth cameras (such as structured light or TOF technologies), accurately modeling the three-dimensional geometric structure of the face, providing crucial geometric information for distinguishing planar forgery attacks (such as printed photos).

[0080] The adaptive multimodal Transformer feature extraction module 420 will be described below.

[0081] Specifically, the adaptive multimodal Transformer feature extraction module first converts the original features of each modality into Query, Key, and Value vectors respectively. By introducing learnable weight parameters, it dynamically adjusts the attention level of different feature channels to highlight information within the modality that is strongly correlated with liveness detection. Then, it calculates the quantified feature correlation through attention scores and combines them with the Value vector for weighted fusion to generate an enhanced intramodal feature tensor. At the same time, the module is compatible with multimodal parallel processing, providing a highly discriminative single-modal feature foundation for subsequent EAP expert optimization and CFM cross-modal fusion, effectively solving the problems of weakened key information and insufficient dynamic adaptability in traditional feature extraction.

[0082] The following is an introduction to the expert-enhanced projection module 430.

[0083] Optionally, the expert enhancement projection module includes: a router unit, multiple expert network units, and a weighted fusion unit; the router unit is used to calculate the matching degree between the key feature representation and each expert network, and to select an active expert network subset using a Top-k strategy; the multiple expert network units are independent fully connected networks used to process the assigned key feature representation; the weighted fusion unit performs a weighted summation of the outputs of the active expert network subset based on the normalized weights output by the router unit.

[0084] Specifically, the EPA module consists of three key components: multiple expert network units, a router unit, and a weighted fusion unit. The expert network unit comprises multiple independent fully connected networks, each focusing on processing different input feature patterns. The router unit calculates the matching probability between the input features and each expert through a learnable gating network and selects the most relevant subset of expert networks based on a Top-k strategy. The weighted fusion unit dynamically weights and fuses the outputs of the selected expert networks according to the probability distribution calculated by the router. This design allows the EPA module to adaptively allocate computational resources based on the input features while maintaining computational efficiency.

[0085] Optionally, the router unit uses linear transformation to calculate the matching degree, and the expert network unit is a two-layer or higher feedforward neural network, with the parameters of each expert network being independent of each other.

[0086] Specifically, the routing unit makes independent routing decisions for each feature vector. First, the router unit uses a linear transformation to evaluate the expert matching degree of the input feature vector x, calculating the expert matching degree score g(x):

[0087] The input data is a tensor. Where B is the batch size, S is the sequence length, and d is the dimension of the feature vector. is the router's weight matrix, where d is the dimension of the feature vectors and N is the total number of experts. Let be the bias vector, where N is the total number of experts.

[0088] Expert networks are the core component of the EPA structure. Each expert network consists of a parameter-independent subnetwork. Each expert network is implemented using two or more layers of feedforward neural networks, and the parameters of each expert network are independent of each other.

[0089] The following is an introduction to the collaborative integration module 440.

[0090] Optionally, the collaborative fusion module includes: a dual-branch fusion unit and a feature splicing unit; The dual-branch fusion unit constructs cross-modal attention interaction channels for IR-RGB and IR-Depth respectively, calculates inter-modal attention weights, and completes feature enhancement; The feature splicing unit is used to splice the dual-branch fusion results along the channel dimension to generate global fusion features.

[0091] See details Figure 5 , Figure 5 This is a schematic diagram of a collaborative fusion module provided in an embodiment of this application. Figure 5 As shown, infrared modality is used as semantic guide. Directional fusion is completed through two symmetrical branch fusion units: IR (infrared features)-RGB (visible light features) and IR-Depth (depth features). Two types of fused features are generated and spliced ​​along the channel. The final output is a global fused representation that has both modal complementarity and semantic consistency, providing efficient feature support for subsequent liveness detection.

[0092] The detection output module 450 will be described below.

[0093] Specifically, the detection output module is the unit that transforms the global fusion features into the final liveness detection result. It is mainly responsible for completing the transformation of high-dimensional features into binary decisions and outputting the results. The input of the detection output module is the global fusion features output by the collaborative fusion module. First, the high-dimensional fusion features are compressed into a one-dimensional discriminant vector by a fully connected layer. Then, the vector value is mapped to the [0,1] interval by the Sigmoid function (the closer the value is to 1, the more likely it is to be a liveness sample; the closer it is to 0, the more likely it is to be an attack sample). Finally, the liveness or attack binary label is output according to the preset threshold to complete the face detection.

[0094] See Figure 5 In this process, a 1×1 convolutional equivalent fully connected layer (feedforward network) is used to map and filter the channel dimensions of the globally fused features, highlighting features related to liveness detection (such as RGB texture, IR thermal radiation differences, and depth geometric details) and suppressing redundant information. Global average pooling is used to perform global mean and maximum pooling on the globally fused features after dual-branch fusion, transforming the two-dimensional feature map into a one-dimensional feature vector, eliminating spatial dimensional redundancy, and retaining global discriminative information for each modality. The multilayer perceptron first maps the one-dimensional features after global pooling to a low-dimensional discriminative space through a fully connected layer, and then compresses the output to the [0,1] interval (close to 1 for liveness, close to 0 for attack samples) through the Softmax function, completing the transformation from features to classification results.

[0095] To verify the performance of the face liveness detection method in this application, simulation verification was also performed on the face liveness detection method provided in this embodiment. The details are as follows: The proposed multimodal liveness detection method was evaluated on the CASIA-SURF dataset and compared with several existing representative methods. The results are shown in Table 1. Table 1 shows that the method in this application achieves the best performance across all three metrics. Specifically, the ACER (Average Classification Error Rate) reaches 0.82%, significantly outperforming other comparative methods. Compared to ViT based on visual Transformer and SEF based on structural enhancement, the method provided in this application improves the ACER by 1.55% and 1.58%, respectively, demonstrating a significant advantage in overall classification accuracy. While the MS-SEF method performs well on this dataset with an ACER of 1.00%, it is still slightly lower than our method. Furthermore, compared to MAP and MA-ViT, our model exhibits a lower false positive rate and stronger generalization ability. The experimental results clearly demonstrate that our proposed method can effectively improve the model's ability to distinguish between attack samples and real samples, exhibiting superior performance on large-scale real-world datasets.

[0096] Table 1

[0097] Furthermore, to fully verify the effectiveness of the proposed method, this application also tested it on Protocol 4 of the CASIA-SURFCeFA dataset and compared it with existing methods. As shown in Table 2, in Protocol 4_1, the method provided by this application achieved an ACER of 0.40%, which is better than the compared methods and demonstrates stronger attack detection capabilities. Among them, BOBO's ACER is 0.42%, which is similar to our method, while PipeNet and Super's ACERs are 1.62% and 0.72%, respectively, which are significantly lower than the method provided by this application. In Protocol 4_2, the method provided by this application also leads existing methods with an ACER of 0.65%, which is much lower than PipeNet's 2.80% and Super's 2.26%, and also better than BOBO's 1.07%, demonstrating stronger robustness. In the most challenging Protocol 4_3, the method presented in this application still achieved an ACER of 0.91%, outperforming PipeNet's 2.40% and BOBO's 1.60%, further demonstrating its robustness and adaptability in complex cross-ethnic scenarios. The average results across the three sub-protocols show that the method presented in this application exhibits the best overall performance compared to BOBO, PiPNet, and Super. Notably, while maintaining a low APCER (Attack Presentation Classification Error Rate), our method significantly reduces the BPCER (Bonafide Presentation Classification Error Rate), demonstrating the practicality and robustness of the method in real-world applications.

[0098] Table 2

[0099] In summary, compared with the prior art, this embodiment has the following advantages: 1. By introducing learnable attention weights through the Adaptive Multimodal Transformer Feature Extractor (AMTE), the attention level of different feature channels in each modality (RGB, IR, Depth) can be dynamically adjusted, which can accurately focus on key information that is strongly related to liveness detection.

[0100] 2. The collaborative fusion module constructs a dual-branch fusion architecture guided by IR modality, realizing directional feature interaction of IR-RGB and IR-Depth respectively. Compared with traditional simple splicing fusion, it can effectively eliminate semantic inconsistencies and information redundancy between modalities.

[0101] 3. While ensuring high accuracy, the model achieves efficient inference through multi-stage optimization, and the expert network selection mechanism of the EAP module reduces invalid calculations.

[0102] In summary, the embodiments of this application achieve high-precision, high-robustness, and high-efficiency multimodal face liveness detection through modular innovation and architecture optimization. It can be widely used in security-sensitive scenarios such as identity authentication, financial payment, and public security. While improving system security, it also takes into account user experience and deployment costs, and has significant technical and practical value.

[0103] The third embodiment of the present invention also provides an electronic device, the electronic device comprising: At least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the face liveness detection method of any of the foregoing embodiments.

[0104] The fourth embodiment of the present invention also provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the face liveness detection method described in any of the foregoing embodiments.

[0105] The fifth embodiment of the present invention also provides a computer program product, which includes a computing program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions that, when executed by a computer, cause the computer to perform the face liveness detection method of any of the foregoing embodiments.

[0106] The sixth embodiment of the present invention also provides a computer program, which includes program instructions that, when executed by a computer, cause the computer to perform the face liveness detection method of any of the foregoing embodiments.

[0107] Figure 6 The diagram illustrates a method or device 1000 that can implement embodiments of the present invention. In some embodiments, it may include more or fewer devices than illustrated. In some embodiments, it may be implemented using a single or multiple devices. In some embodiments, it may be implemented using cloud-based or distributed devices.

[0108] like Figure 6As shown, device 1000 includes a processor 1001, which can perform various appropriate operations and processes based on programs and / or data stored in read-only memory (ROM) 1002 or programs and / or data loaded from storage portion 1008 into random access memory (RAM) 1003. Processor 1001 may be a multi-core processor or may contain multiple processors. In some embodiments, processor 1001 may include a general-purpose main processor and one or more special coprocessors, such as a central processing unit (CPU), graphics processing unit (GPU), neural network processor (NPU), digital signal processor (DSP), etc. Various programs and data required for the operation of device 1000 are also stored in RAM 1003. Processor 1001, ROM 1002, and RAM 1003 are interconnected via bus 1004. Input / output (I / O) interface 1005 is also connected to bus 1004.

[0109] The processor and memory described above are used together to execute a program stored in the memory. When the program is executed by a computer, it can implement the methods, steps, or functions described in the above embodiments.

[0110] The following components are connected to I / O interface 1005: an input section 1006 including a keyboard, mouse, touchscreen, etc.; an output section 1007 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN card, modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to I / O interface 1005 as needed. A removable medium 1011, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 1010 as needed so that computer programs read from it can be installed into storage section 1008 as needed. Figure 6 The diagram only shows a portion of the components and does not imply that the device 1000 only includes... Figure 6 The components shown.

[0111] The systems, devices, modules, or units described in the above embodiments can be implemented by a computer or its associated components. The computer may be, for example, a mobile terminal, smartphone, personal computer, laptop computer, in-vehicle human-machine interface device, personal digital assistant, media player, navigation device, game console, tablet computer, wearable device, smart TV, Internet of Things system, smart home, industrial computer, server, or a combination thereof.

[0112] Although not shown, in this embodiment of the invention, a computer-readable storage medium is provided that stores a computer program / instructions thereon, which, when executed by a processor, implements the face liveness detection method described in Embodiment 1.

[0113] Storage media in embodiments of the present invention include articles that are permanent and non-permanent, removable and non-removable, capable of storing information by any method or technology. Examples of storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0114] Although not shown, embodiments of the present invention also provide a computer program product, including: a computer program / instructions that, when executed by a processor, implement the face liveness detection method described in Embodiment 1.

[0115] The methods, programs, systems, apparatuses, etc., in the embodiments of the present invention can be executed or implemented in one or more networked computers, or practiced in a distributed computing environment. In the embodiments of this specification, in these distributed computing environments, tasks can be performed by remote processing devices connected via a communication network.

[0116] Those skilled in the art will understand that the embodiments described in this specification can be provided as methods, systems, or computer program products. Therefore, those skilled in the art will realize that the functional modules / units or controllers and related method steps described in the above embodiments can be implemented in software, hardware, or a combination of both.

[0117] Unless explicitly stated otherwise, the actions or steps of the methods and procedures described in the embodiments of the present invention do not necessarily have to be performed in a specific order and can still achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0118] This document describes several embodiments of the present invention; however, for the sake of brevity, the descriptions of the embodiments are not exhaustive, and identical or similar features or parts between the embodiments may be omitted. In this document, "one embodiment," "some embodiments," "example," "specific example," or "some examples" refers to embodiments applicable to at least one, but not all, of the present invention. The above terms do not necessarily refer to the same embodiments or examples. Without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described herein, as well as the features of the different embodiments or examples.

[0119] The exemplary systems and methods of the present invention have been specifically shown and described with reference to the above embodiments, which are merely examples of the best mode for implementing the systems and methods. Those skilled in the art will understand that various changes can be made to the embodiments of the systems and methods described herein without departing from the spirit and scope of the invention as defined in the appended claims when implementing the systems and / or methods.

Claims

1. A face liveness detection method, characterized in that, Comprise: The modal image acquisition is carried out to the face image, and the RGB modal image, infrared modal image and depth modal image of the face are obtained respectively; Each modal image is respectively processed by an adaptive multi-modal Transformer feature extractor to obtain the key feature expression of the reinforced modal image; The key feature expression is selectively enhanced by an expert enhanced projection module to obtain the optimized feature of each modal; The optimized features of the preset priority modal are used as the guide to interact the optimized features between different modalities to obtain modal fusion features, and then the modal fusion features are spliced to obtain global fusion features; Whether the face is a living body is judged according to the global fusion features to obtain a judgment result.

2. The face liveness detection method of claim 1, wherein, The adaptive multi-modal Transformer feature extractor is used to process each modal image to obtain the key feature expression of the reinforced modal image, comprising: The dimension transformation and feature rearrangement are performed on the query, key and value vectors of each modal image; The weights of the query, key and value vectors are adjusted by learnable parameters; The dot product operation is performed on the query vector and the key vector after weight adjustment to obtain the attention value of each modal image; The attention value is normalized and residual connected by using Softmax function to obtain the attention score of each modal image; The key feature expression is obtained according to the attention score and the value vector.

3. The face liveness detection method of claim 1, wherein, The key feature expression is selectively enhanced by an expert enhanced projection module to obtain the optimized feature of each modal, comprising: The matching degree of each key feature expression vector and each expert network is calculated by a learnable router network, and a target expert network subset is selected according to a preset strategy; The key feature expression vector allocated by the expert network subset is processed in parallel, and the output of the expert network subset is weighted and summed according to the weight allocated by the router to obtain the optimized feature.

4. The face liveness detection method of claim 1, wherein, The optimized features of the preset priority modal are used as the guide to interact the optimized features between different modalities to obtain modal fusion features, and then the modal fusion features are spliced to obtain global fusion features, comprising: With the infrared modal as the guide, the input features X IR、 X rgb、 X Depth Perform linear transformation , obtain query vector Q IR , key vector K RGB、 K Depth And value vector V RGB、 V Depth ; Then, the attention weight of the infrared modal to the RGB modal is calculated by dot product attention mechanism, wherein d represents the dimension of the key vector. obtaining a fusion result of the infrared modal guiding the RGB modal ; Computing a fusion result of an infrared modality guiding a depth modality , ; Fusion is performed on the and global fusion features: wherein, is a global fusion feature, denotes a fusion operation.

5. A face liveness detection apparatus, characterized in that, Comprise: The modal image acquisition module is used for modal image acquisition of face image, and the RGB modal image, infrared modal image and depth modal image of the face are obtained respectively; The adaptive multi-modal Transformer feature extraction module is used for processing each modal image to obtain the key feature expression of the reinforced modal image; The expert enhanced projection module is used for selectively enhancing the key feature expression to obtain the optimized feature of each modal; The cooperative fusion module is configured to perform cross-modal feature interaction on the optimization features between different modalities by using the optimization features of the preset priority modalities for guidance, obtain modal fusion features, and then perform splicing on the modal fusion features to obtain global fusion features. The detection output module is configured to determine whether the face is a living body according to the global fusion features and output a determination result.

6. The face liveness detection apparatus according to claim 5, characterized in that, The expert enhanced projection module comprises: a router unit, a plurality of expert network units, and a weighted fusion unit. The router unit is configured to calculate matching degrees of the key feature expression and each expert network, and select an activated expert network subset through a Top-k strategy. The plurality of expert network units are independent fully connected networks, and are configured to process the assigned key feature expression. The weighted fusion unit is configured to perform weighted summation on outputs of the activated expert network subset according to normalized weights output by the router unit.

7. The face liveness detection apparatus of claim 5, wherein, The cooperative fusion module comprises a double-branch fusion unit and a feature splicing unit. The double-branch fusion unit respectively constructs cross-modal attention interaction channels of IR-RGB and IR-Depth, calculates inter-modal attention weights, and completes feature enhancement. The feature splicing unit is configured to splice double-branch fusion results along a channel dimension to generate global fusion features.

8. The face liveness detection apparatus of claim 6, wherein, The router unit adopts linear transformation to calculate the matching degrees, the expert network unit is a two-layer or more feedforward neural network, and parameters of each expert network are independent of each other.

9. An electronic device, comprising: The electronic device comprises: at least one processor; and a memory connected to the at least one processor in communication; wherein The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the face living body detection method of any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium, comprising: The non-transitory computer readable storage medium stores computer instructions for causing the computer to perform the face living body detection method of any one of claims 1 to 7.

Images