Face living body detection method and system for multi-modal feature fusion

Abstract

The application provides a face living body detection method and system for multi-modal feature fusion, and relates to the technical field of face living body detection. The specific method comprises the following steps: synchronously acquiring RGB, infrared and depth images, performing Gabor texture enhancement on the RGB image and completing pixel-level registration with the other two modalities; sending the three modalities of data into a shallow convolutional network respectively to extract basic features; implementing double-branch processing on the basic features of each modality, refining through light-weight convolution compression, highlighting the difference regions through multi-scale maximum pooling, and obtaining dimension-reduced features through fusion; constructing a cross-modal affinity matrix based on cosine similarity to realize feature migration and reinforcement between modalities; after channel and spatial attention weighting, splicing and fusion are performed on the reinforced feature map, and a living body or non-living body discrimination result is output through a classifier. Through cross-modal feature migration and double-branch feature dimension reduction mechanism, the application effectively improves the accuracy and stability of face living body detection in complex attack scenes.

Description

Technical Field

[0001] This invention relates to the field of face liveness detection technology, specifically to a face liveness detection method and system oriented towards multimodal feature fusion. Background Technology

[0002] Face liveness detection technology, as a core security barrier for face recognition systems, aims to effectively distinguish between real, live faces and forgeries such as photos, video playback, and 3D masks. Currently, single-modal detection technologies, due to limitations in information dimensions, often fall short in the face of complex and ever-changing deception methods. For example, traditional RGB image detection is highly susceptible to interference from changes in ambient lighting, moiré patterns on high-definition screens, and reflections from printed photos, resulting in insufficient defense against high-resolution replay attacks. Infrared thermal imaging has low resolution and lacks the ability to describe microscopic texture details, making it difficult to identify complex prosthetics made of highly realistic thermal materials. Depth image data is easily affected by measurement distance and ambient reflected light, and is prone to voids and noise at facial edges or occluded areas, limiting its ability to distinguish high-precision 3D-printed head molds. Therefore, relying solely on single-modal information is insufficient to build a robust and universal liveness detection defense.

[0003] To overcome the inherent limitations of single-modal data, multimodal liveness detection techniques that fuse RGB, infrared, and depth images have emerged. Existing fusion methods often employ simple channel stitching or decision-level weighting, which, while improving detection accuracy to some extent, still have significant shortcomings. On one hand, many methods handle and fuse features from different modalities in a coarse manner, failing to fully consider the significant differences in feature distribution, semantic representation, and noise patterns among the various modalities. This results in superficial feature fusion, and the complementarity between modalities is not deeply explored or effectively utilized. On the other hand, existing methods often neglect the semantic affinity between modalities, treating the three modalities as completely isolated feature sources for independent processing before fusion, lacking an efficient cross-modal information transfer and enhancement mechanism. Summary of the Invention

[0004] The purpose of this invention is to provide a face liveness detection method and system oriented towards multimodal feature fusion, so as to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: A face liveness detection method based on multimodal feature fusion, comprising the following steps: S1: Simultaneously acquire RGB image, infrared image and depth image of the face to be detected, apply multiple different Gabor filter kernels to the RGB image for texture enhancement, and perform pixel-level registration on the filtered RGB image, infrared image and depth image to obtain spatially aligned input data for the three modalities. S2: Input data from the three modalities are fed into a shallow convolutional network. Channel expansion, local feature extraction, and channel attention weighting are performed on the input data of each modality in sequence to obtain the basic feature maps of the three modalities. S3: Perform dual-branch processing on the basic feature maps of each modality. The first branch applies lightweight convolution to the basic feature maps of each modality to obtain compressed and refined first-branch features. The second branch performs max pooling on the basic feature maps of each modality to obtain second-branch features that highlight the difference regions. The first-branch features and second-branch features of each modality are fused element by element and then convolved with equal-sized features to obtain the dimensionality-reduced features of each modality. S4: Perform global average pooling on the dimensionality reduction features of the three modalities respectively to obtain the global feature vectors of each modality. Based on the cosine similarity between the global feature vectors of the three modalities, construct a cross-modal affinity matrix, and calculate the transfer features of each modality based on the cross-modal affinity matrix. Then, fuse the transfer features of each modality with its dimensionality reduction features element by element to obtain the enhanced feature map of each modality. S5: Perform channel and spatial attention weighting on the augmented feature maps of the three modalities respectively, and concatenate them according to the channels to obtain the face fusion feature map. Perform dimensionality reduction, pooling and fully connected processing on the obtained face fusion feature map in sequence to obtain the fusion feature vector. Input the obtained fusion feature vector into the preset binary classifier and output the live or non-live face discrimination result.

[0006] Furthermore, the first branch in the dual-branch processing applies lightweight convolution to the basic feature maps of each modality in the following specific way: a lightweight convolution structure is constructed that sequentially includes a depthwise convolutional layer, a first stabilizing layer, a first activation function, a pointwise convolutional layer, a second stabilizing layer, and a second activation function; For any modality's base feature map, firstly, the depthwise convolutional layer independently performs spatial convolution on each input channel of the base feature map to extract the spatial features of all channels. Then, the first stabilization layer performs batch normalization on the spatial features of all channels and uses the first activation function to perform nonlinear transformation on all normalized spatial features to form a multi-channel feature map. The obtained multi-channel feature maps are input into a pointwise convolutional layer for cross-channel linear combination and channel dimension compression to form compressed feature maps. Then, the second stabilizing layer performs batch normalization on the compressed feature maps and uses the second activation function to perform nonlinear transformation on the normalized compressed feature maps, outputting the first branch features after compression and refinement.

[0007] Furthermore, the second branch in the dual-branch processing performs max pooling screening on the basic feature maps of each modality in the following specific way: constructing a max pooling screening structure consisting of a multi-scale parallel pooling layer and an attention calibration layer; For any modality's base feature map, the multi-scale parallel pooling layer first performs multi-scale spatial downsampling on the base feature map in parallel. Multiple spatial pooling windows of different sizes are used to select the maximum response value in the local area they cover, generating multiple sets of pooling feature maps with different spatial resolutions. The attention calibration layer concatenates the multiple pooling feature maps according to the channel dimension, and performs convolutional compression and attention weighting on the concatenated features to map the multiple pooling features to a unified channel dimension and highlight key spatial regions, outputting a second branch feature that highlights the differences in the regions.

[0008] Furthermore, the specific method for constructing the cross-modal affinity matrix based on the cosine similarity between the global feature vectors of the three modalities is as follows: For the global feature vectors of the three modalities, calculate the cosine similarity between each pair; construct a third-order matrix with the three modalities as rows and columns respectively, so that the diagonal elements are 1 and the off-diagonal elements are the cosine similarity between the global feature vectors of the corresponding row mode and column mode, and use this third-order matrix as the cross-modal affinity matrix.

[0009] Furthermore, the specific method for calculating the modal transfer features based on the cross-modal affinity matrix is ​​as follows: set a similarity threshold, and select all off-diagonal elements in the cross-modal affinity matrix whose values ​​are greater than the similarity threshold as transferable elements; For any transferable element, the global feature vector of the column mode of the element is mapped to the feature space of the global feature vector of the row mode of the element through the learnable projection matrix corresponding to the current transfer direction, so as to obtain the transfer component of the row mode of the element; in the cross-modal affinity matrix, all transfer components of the same row mode are weighted and summed to obtain the transfer features of each mode.

[0010] Furthermore, the specific method for weighting the augmented feature maps of the three modalities with channel and spatial attention is as follows: a hybrid weighted structure containing a channel attention module and a spatial attention module is constructed, wherein the channel attention module consists of a global pooling unit, a fully connected network layer and a first weight function, and the spatial attention module consists of a channel pooling unit, a channel convolutional layer and a second weight function. For any modality's augmented feature map, it is input into the channel attention module. The global pooling unit performs global max pooling and global average pooling operations on the augmented feature map of the current modality, respectively. The two pooling results are then input into the fully connected network layer to obtain two pooling vectors corresponding to the two pooling results. The two pooling vectors are added and fused element by element, and the channel attention weights are generated on the fused vector through the first weight function. The obtained channel attention weights are multiplied element by element with the augmented feature map of the current modality to obtain the channel weighted feature map of the current modality. The channel-weighted feature map of the current modality is input into the spatial attention module. The channel pooling unit performs global max pooling and global average pooling operations on the channel-weighted feature map along the channel dimension. The two pooling results are concatenated according to the channel. The concatenated result is input into the channel convolutional layer for feature mapping. The spatial attention weights are generated on the convolutional feature vector through the second weight function. The obtained spatial attention weights are multiplied element-wise with the channel-weighted feature map of the current modality to obtain the attention-weighted feature map of the current modality.

[0011] Furthermore, the specific method for obtaining the face fusion feature map is as follows: perform simple linear transformations on the attention-weighted feature maps of the three modalities respectively to align the channel dimensions of the three modalities, and then stitch the aligned attention-weighted feature maps of each modality according to the order of RGB modality, infrared modality and depth modality, according to the channels to obtain the face fusion feature map.

[0012] Furthermore, the specific method for inputting the fused feature vector into a preset binary classifier and outputting the result of live or non-live face discrimination is as follows: Construct a binary classifier containing a fully connected network layer and a Softmax activation function. Input the fused feature vector obtained from the face fusion features into the binary classifier. After passing through the fully connected network layer, the fused feature vector is mapped into a two-dimensional vector. Then, the Softmax activation function is used to convert the obtained two-dimensional vector into a probability distribution of live and non-live faces. Compare the probability values ​​of the two results and take the one with the larger probability value as the final discrimination result.

[0013] The present invention also provides a face liveness detection system oriented towards multimodal feature fusion, wherein the face liveness detection system oriented towards multimodal feature fusion is used to execute the above-described face liveness detection method oriented towards multimodal feature fusion, comprising: The preprocessing module is used to simultaneously acquire RGB, infrared, and depth images of the face to be detected. Multiple different Gabor filter kernels are used to enhance the texture of the RGB image. The filtered RGB, infrared, and depth images are then registered at the pixel level to obtain spatially aligned input data for the three modalities. The feature extraction module is used to feed the input data of the three modalities into the shallow convolutional network respectively, and to perform channel expansion, local feature extraction and channel attention weighting on the input data of each modality in sequence to obtain the basic feature maps of the three modalities. The feature dimensionality reduction module is used to perform dual-branch processing on the basic feature maps of each modality. The first branch applies lightweight convolution to the basic feature maps of each modality to obtain compressed and refined first-branch features. The second branch performs max pooling to filter the basic feature maps of each modality to obtain second-branch features that highlight the difference regions. The first-branch features and second-branch features of each modality are fused element by element and then convolved with equal-sized features to obtain the dimensionality reduction features of each modality. The feature enhancement module is used to perform global average pooling on the dimensionality reduction features of the three modalities respectively to obtain the global feature vectors of each modality. Based on the cosine similarity between the global feature vectors of the three modalities, a cross-modal affinity matrix is ​​constructed, and the transfer features of each modality are calculated based on the cross-modal affinity matrix. The transfer features of each modality are fused with its dimensionality reduction features element by element to obtain the enhanced feature map of each modality. The liveness detection module is used to perform channel and spatial attention weighting on the augmented feature maps of the three modalities respectively, and to concatenate them according to the channels to obtain a face fusion feature map. The obtained face fusion feature map is then subjected to dimensionality reduction, pooling and full connection processing in sequence to obtain a fusion feature vector. The obtained fusion feature vector is input into a preset binary classifier to output the liveness or non-liveness face discrimination result.

[0014] Compared with the prior art, the beneficial effects of the present invention are: This invention effectively addresses the issues of susceptibility to interference and insufficient robustness of single-modal information by constructing a cross-modal affinity matrix based on cosine similarity and a feature transfer reinforcement mechanism, significantly improving the accuracy and generalization ability of multimodal face liveness detection. Unlike existing technologies that isolate features of each modality, this invention utilizes the cosine similarity between global feature vectors to quantify the "affinity" between RGB, infrared, and depth modalities in a high-level semantic space. Based on this, by setting a similarity threshold and a learnable projection matrix, the system can adaptively transfer discriminative features from highly similar modalities to modalities with relatively weak or interfered information. For example, when a depth image suffers from geometric information loss due to overexposure, semantically highly related RGB or infrared features can be directionally transferred to reinforce the feature map of the depth modality. This intermodal knowledge transfer and complementarity mechanism enables the model to maintain stable feature representation under non-ideal imaging conditions, effectively overcoming the performance degradation of traditional methods when a single modality fails.

[0015] This invention employs a dual-branch parallel processing structure that balances the efficiency of feature compression with the accuracy of key region selection, addressing the challenge of balancing computational overhead and discriminative capability in existing technologies. The first branch utilizes a lightweight convolutional sequence consisting of depthwise and pointwise convolutions to compress the channel dimensions of the basic feature map and deeply extract information at extremely low computational cost, ensuring lightweight model deployment. The second branch introduces multi-scale parallel pooling layers and attention calibration layers to filter and amplify the most decisive local difference regions for distinguishing between live and fake attacks from different spatial granularities. By fusing the compact semantic features extracted from the first branch with the spatially salient features highlighted by the second branch element-wise, the resulting dimensionality-reduced features not only significantly reduce dimensionality but also achieve concise and precise information condensation. This dual-branch collaborative design enables the model to accurately pinpoint attack clues while reducing computational burden, significantly improving the efficiency and accuracy of discriminative attacks against complex presentation attacks.

[0016] This invention employs a progressive channel and spatial hybrid attention weighting mechanism and follows a specific modal order for channel stitching, achieving precise focusing and organic fusion of multimodal features in key discriminative dimensions. After obtaining the enhanced feature map, this invention does not directly perform stitching and fusion; instead, it first independently applies channel and spatial attention weighting to each modality. The channel attention module automatically learns and enhances feature channels rich in discriminative information, suppressing redundant channels irrelevant to the task. The subsequent spatial attention module further locates key spatial regions on the feature map, enabling subsequent processing to focus on the most persuasive image regions. Finally, channel stitching is performed in a fixed order of "RGB-IR-Depth," ensuring the consistency of the feature parsing logic. This progressive strategy of "first filtering channels, then locating space, and finally orderly fusion" greatly improves the purity and representational ability of the fused feature map, ensuring that the final fused feature vector input to the binary classifier is rich in highly condensed multimodal discriminative information, thereby effectively improving the accuracy and confidence of liveness detection. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the overall method flow of the present invention; Figure 2 Comparison of multimodal face images provided in embodiments of the present invention; Figure 3 This is a schematic diagram of a dual-branch structure provided in an embodiment of the present invention; Figure 4 An example diagram of the cross-modal affinity matrix provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of feature transfer provided in an embodiment of the present invention; Figure 6This is a structural block diagram of the overall system of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments.

[0019] It should be noted that, unless otherwise defined, the technical or scientific terms used in this invention should have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0020] Example: Please see Figures 1 to 5 The present invention provides a technical solution: A face liveness detection method based on multimodal feature fusion, comprising the following steps: S1: Simultaneously acquire RGB, infrared, and depth images of the face to be detected. Apply multiple different Gabor filter kernels to the RGB image for texture enhancement. Perform pixel-level registration on the filtered RGB, infrared, and depth images to obtain spatially aligned input data for the three modalities.

[0021] In the field of face liveness detection technology, single-modal image information often has limitations. For example, while ordinary RGB images can capture rich color and texture details, they are easily affected by changes in ambient lighting, moiré patterns on the captured screen, and deception techniques such as high-resolution photo or video playback. Infrared images, although effectively avoiding the influence of lighting and possessing a certain robustness against spoofing attacks, struggle to capture fine texture and color features and have limited ability to distinguish between spoofing attacks using complex materials. Depth images can intuitively reflect the three-dimensional structure of a face and effectively resist deception using planar photographs, but they are susceptible to interference from measurement distance and environmental reflections, and are prone to data loss and noise when the face is occluded or the pose changes significantly. Therefore, for face liveness detection tasks, fusing multimodal information is a key path to improve detection accuracy and robustness. RGB images contain rich color and texture details, infrared images can effectively distinguish the thermal radiation characteristics of live skin from counterfeit materials, while depth images provide information on the three-dimensional geometric structure of a real face. These three modalities characterize the essential attributes of the human face from different physical dimensions. Through synchronous collection and joint analysis, they can effectively resist various presentation attacks such as photos, videos, masks, and 3D head models.

[0022] Please see Figure 2 The accompanying comparison image of multimodal face images illustrates the typical differences between live and non-live faces under multimodal imaging. After Gabor filtering enhancement, the RGB image of a live face clearly shows microscopic structures such as skin pores and fine textures; its infrared image exhibits the unique thermal radiation distribution pattern of live skin, with uniform brightness in the facial area and a clear contrast with the background; the depth image accurately reflects the three-dimensional curved geometry of the face. In contrast, the Gabor-enhanced image of a non-live face (such as a high-resolution printed photograph) has an overly smooth texture or displays regular dots; the infrared image, lacking the thermal characteristics of real skin, shows abnormally bright or dark areas; and the depth image, due to the planar nature of the surface, exhibits a nearly flat depth value distribution. These cross-modal differences provide a reliable basis for the liveness detection method in this embodiment.

[0023] This embodiment first simultaneously acquires the RGB image, infrared image, and depth image of the face to be detected. The core significance of simultaneous acquisition is to ensure that the three images describe the state of the object to be detected at exactly the same spatiotemporal point, eliminating the misalignment of information between modalities caused by object movement or environmental changes, and providing a reliable physical basis for subsequent pixel-level registration and feature fusion.

[0024] After acquiring the original images in three modalities, this embodiment does not directly feed the three images into the subsequent network. Instead, it first uses multiple different Gabor filter kernels to perform texture enhancement processing on the original RGB images individually. This operation is based on the prior knowledge that there are fundamental differences in the microscopic texture between a live face and a fake face. Real live face skin has natural microstructures such as pores and fine lines, while faces presented in photos, masks, or on screens often have problems such as pixel grids, printed dots, or overly smooth surfaces. As a linear filter with good localization characteristics in both the spatial and frequency domains, the Gabor filter's kernel function is essentially a sine plane wave modulated by a Gaussian function. By designing Gabor filter kernels with different directions (such as 0°, 45°, 90°, 135°) and different scales (corresponding to sine waves of different frequencies), a multi-directional, multi-resolution filter bank can be constructed. Applying this filter bank to an RGB image can extract texture response features of different directions and coarseness from the original image.

[0025] The mathematical expression for the Gabor filter kernel is: in: ; ; The new coordinates after coordinate rotation; The wavelength of the sine wave is used to control the spatial frequency response of the filter; The orientation angle of the filter kernel determines the filter's sensitivity to texture orientation; This is the phase offset, typically taken as 0 or... To form a pair of orthogonal filters; The standard deviation of the Gaussian envelope controls the spatial expansion range of the filter. The aspect ratio determines the elliptical shape of the filter's receptive field.

[0026] This embodiment sets multiple sets of Gabor filter kernel parameters to cover multi-scale and multi-directional features of facial texture. In the scale dimension, the wavelength is set. The set of values ​​is Pixels, corresponding to the response ranges of fine, medium, and coarse textures respectively; in the orientation dimension, the orientation angle is set. By sampling uniformly at 30° intervals within a range of 0° to 150°, a filter bank with 6 directions is formed. The Gabor filter kernel bank thus constructed contains a total of 18 filter kernels, forming a multi-scale and multi-directional coverage of the texture features of RGB images.

[0027] The specific process for Gabor texture enhancement of an RGB image is as follows: convert the original RGB image to a grayscale image. The 18 Gabor filter cores are respectively compared with... Two-dimensional convolution operations are performed to obtain 18 filter response maps. For each pixel location, its response value on the 18 filter response maps is extracted to form the texture feature descriptor of that pixel. To reduce the feature dimension and enhance discriminability, local energy integration is performed on the filter response maps, that is, the sum of squares and mean of each filter response map within the local window are calculated to obtain the texture energy map. Finally, the multi-scale and multi-directional texture energy maps are stacked according to the channel dimension to form the texture-enhanced RGB feature representation.

[0028] Gabor filtering significantly enhances the subtle texture structures (such as skin pores, wrinkles, and hair) in RGB images, while suppressing low-frequency interference caused by changes in illumination. This enhances the distinguishability of real human skin from various forgeries in terms of local texture patterns, providing more discriminative input information for subsequent shallow convolutional networks than the original RGB pixels. Consequently, it improves the robustness of the entire model against complex attack methods such as highly realistic masks and high-definition printed photos.

[0029] After completing the texture enhancement processing of the RGB image, this embodiment further performs pixel-level registration on the filtered RGB image, infrared image, and depth image. The core objective of pixel-level registration is to transform the images of the three modalities to the same coordinate reference system, so that for any pixel coordinate, the pixel value at that position in the three images corresponds to the same physical point on the face surface. The RGB image after Gabor texture enhancement, along with the synchronously acquired infrared and depth images, originates from different imaging sensors, resulting in inherent differences in their physical imaging planes, optical parameters, and spatial resolutions. This leads to inconsistent pixel coordinates for facial features at the same spatial location in the three images. To achieve accurate fusion and joint analysis of multimodal features, pixel-level registration of the three modalities is necessary to establish a strict spatial correspondence. This embodiment uses the RGB image as a reference and utilizes pre-calibrated camera intrinsic and extrinsic parameters to transform the infrared and depth images to the RGB coordinate system through geometric projection. Bilinear interpolation resampling is then employed to ensure that the three modalities accurately correspond to the same physical point on the face surface at the same pixel position, thereby forming spatially aligned multimodal input data and providing a consistent foundation for subsequent feature fusion.

[0030] S2: Input data from the three modalities are fed into a shallow convolutional network. Channel expansion, local feature extraction, and channel attention weighting are performed on the input data of each modality in sequence to obtain the basic feature maps of the three modalities.

[0031] After pixel-level registration, the image data of the three modalities have achieved strict spatial consistency. However, the data characteristics and feature representation requirements of each modality differ significantly: RGB images, after Gabor enhancement, contain multi-channel texture features, infrared images have single-channel thermal radiation intensity, and depth images have single-channel distance information. Using a single deep network to extract features from all modalities without differentiation would not only incur huge computational costs but could also lead to the network learning noise couplings unrelated to the modalities. Therefore, this embodiment adopts a "divide and conquer" strategy, configuring shallow convolutional networks with identical structures but independent parameters for each of the three modalities. This design allows the network for each modality to focus on learning the most discriminative shallow feature patterns from its specific modal data, avoiding interference introduced by immature feature mixing between modalities. Furthermore, the "shallow" design principle ensures that a shallow network layer limits the receptive field to a local neighborhood, extracting fine-grained features closely related to the microscopic texture and local geometric undulations of the live object, rather than high-level semantic information. This is crucial for capturing the essential differences in local details between real and fake faces.

[0032] Specifically, for any modality of input data, the processing flow of its shallow convolutional network consists of the following three sequentially connected stages: Channel expansion phase: The input data channels for the three modalities have different dimensions, so they need to be uniformly expanded to the same number of intermediate channels to achieve dimension alignment for subsequent parallel processing. Let the target number of channels be... For the input feature map Where H is the image height and W is the image width. The number of input channels is used to expand the channels of the input feature map I. The convolutional layer is implemented using the following formula: in The kernel weights are the number of output channels, the number of input channels, the kernel height, and the kernel width, respectively. This represents the convolution operation; It is the bias vector; It is the ReLU activation function; This is the expanded feature map. This operation, without changing the spatial resolution, establishes a unified feature representation space for each modality through linear combination and dimensional expansion.

[0033] Local Feature Extraction Stage: After channel expansion, this embodiment employs depthwise separable convolution for local feature extraction to balance computational efficiency and feature extraction capability. This stage includes two sub-steps: depthwise convolution and pointwise convolution.

[0034] Depthwise convolution performs spatial convolution independently on each channel, capturing local spatial patterns within a single channel. The specific formula is as follows: Where c is the channel index; For the c-th channel Depth convolution kernel; Input feature map The c-th channel slice; output This is the feature map after depthwise convolution.

[0035] Pointwise convolution through Convolutional processing fuses information across channels, enhancing feature interaction between channels. The specific formula is as follows: in, For pointwise convolutional kernel weights, their four dimensions represent the number of output channels, the number of input channels, the kernel height, and the kernel width, respectively. The bias vector; output This is the feature map after pointwise convolution. Depthwise separable convolution decomposes standard convolution into two steps: spatial filtering and channel mixing, significantly reducing both the number of parameters and computational cost, thus improving the efficiency of shallow networks.

[0036] Channel attention weighting stage: To adaptively enhance discriminative channels and suppress redundant channels, this embodiment introduces a channel attention mechanism. This mechanism learns the importance weights of each channel to achieve adaptive recalibration of feature channels. Specifically, it adopts a simplified form of the SE module: First, the feature map Global average pooling is performed to compress the spatial information of each channel into a global descriptor. The specific formula is as follows: Where i and j are the spatial row index and column index of the feature map, respectively; Representation of feature map In position The pixel value of the c-th channel; The global average response of the c-th channel; output channel statistical vector. Its c-th element is .

[0037] Subsequently, the non-linear interactions between channels are learned through the fully connected layer to generate channel attention weights, with the specific formula as follows: in, This is the weight matrix of the first fully connected layer. The compression ratio represents the compression ratio of the intermediate hidden layer dimension relative to the number of input channels; This is the first layer bias vector; It is the ReLU activation function; This is the weight matrix for the second fully connected layer; This is the second layer bias vector; Use the Sigmoid activation function; output Let c be the attention weight vector for each channel. This indicates the importance of channel c.

[0038] Finally, the attention weights are multiplied channel by channel by channel of the original feature map to complete the channel weighting. The specific formula is as follows: Output This is the basic feature map of this mode, and its spatial dimension is... With channel dimension All remain unchanged, but the response strength of each channel has been recalibrated according to the importance weights of adaptive learning.

[0039] The three-stage processing described above is performed independently for the RGB, infrared, and depth modalities, generating the corresponding basic feature maps. The network parameters for each modality are not shared, ensuring that the feature extraction process can adaptively match the data characteristics of each modality: the RGB modality focuses on texture details, the infrared modality focuses on thermal radiation distribution, and the depth modality focuses on geometric structure. The basic feature map retains the original spatial resolution, providing a rich feature base for subsequent two-branch dimensionality reduction processing.

[0040] S3: Perform dual-branch processing on the basic feature maps of each modality. The first branch applies lightweight convolution to the basic feature maps of each modality to obtain compressed and refined first-branch features. The second branch performs max pooling on the basic feature maps of each modality to obtain second-branch features that highlight the difference regions. The first-branch features and second-branch features of each modality are fused element by element and then convolved with equal-size features to obtain the dimensionality-reduced features of each modality.

[0041] After obtaining the basic feature maps for each of the three modalities through step S2, these feature maps, while possessing preliminary discriminative capabilities, still present two key issues: First, the feature maps have a high number of channels and spatial dimensionality, which, if directly used for subsequent cross-modal interaction and fusion, would incur significant computational overhead, hindering lightweight model deployment and real-time detection requirements. Second, the basic feature maps represent a uniform extraction of local information, failing to explicitly highlight the highly discriminative local difference regions that are crucial for distinguishing between live and non-live subjects. To address these issues, this embodiment introduces a dual-branch parallel processing structure for the basic feature maps of each modality in step S3. This structure is designed to balance information preservation and key information enhancement during feature dimensionality reduction. Specifically, the dual-branch structure includes a first branch for feature compression and refinement, and a second branch for highlighting and filtering key regions.

[0042] Please see Figure 3 The diagram illustrates the dual-branch structure, showcasing the detailed processing flow of dual-branch feature dimensionality reduction. The first branch employs a lightweight convolution sequence of "depth-wise convolution—stable activation—pointwise convolution—stable activation" to achieve channel compression and feature extraction with low computational cost. The second branch captures multi-granularity salient regions through parallel max pooling at three scales, followed by channel concatenation and attention calibration to highlight key differences. The feature maps output from the two branches are fused element-wise and then locally integrated through equal-size convolutions, ultimately yielding a dimensionality-reduced feature map that combines compact semantic information with spatial saliency. This structure effectively preserves discriminative information while reducing channel dimensionality, achieving a balance between model lightweighting and detection accuracy.

[0043] The first branch is designed to "reduce workload and increase efficiency." It applies a carefully crafted, lightweight convolutional operation to the base feature map, aiming to compress feature channels and re-encode information with extremely low computational cost. This process is not merely a simple dimensionality reduction, but a feature refinement process. It forces the network to learn how to express equally rich, or even more discriminative, information with fewer channels, resulting in a concise and refined first-branch feature set. This lays the foundation for the lightweight operation and fast response of the entire model.

[0044] The second branch is designed to "refine and eliminate the dross." The difference between a real face and a fake face (such as a photo or a mask) often lies in subtle features of certain key local areas, such as the pixel grid of a photo, the edge seams of a mask, and the abnormal flatness of the depth map in fake areas. The second branch performs max pooling on the base feature map, its core purpose being to explicitly filter and amplify the regions with the strongest responses and most significant differences in the spatial dimension. Max pooling retains the strongest feature responses within a local neighborhood, thus filtering out background or non-critical smooth areas, allowing the model to focus more on the most convincing live / non-live cues. The resulting second-branch features are rich in significantly enhanced local differential information.

[0045] Furthermore, the first branch in the dual-branch processing applies lightweight convolution to the basic feature maps of each modality in the following specific way: a lightweight convolution structure is constructed that sequentially includes a depthwise convolutional layer, a first stabilizing layer, a first activation function, a pointwise convolutional layer, a second stabilizing layer, and a second activation function.

[0046] In the dual-branch processing framework described in step S3, the first branch undertakes the core task of efficiently compressing and extracting features from the basic feature maps of each modality. In order to achieve feature dimensionality reduction while controlling the computational complexity and number of parameters of the model to the greatest extent, this embodiment adopts a six-layer serial architecture of "depthwise convolution-stabilization-activation-pointwise convolution-stabilization-activation".

[0047] This structural sequence is not constructed arbitrarily, but rather based on careful consideration of the stability of information flow and nonlinear expressive power during feature transformation. The depthwise convolutional layer efficiently extracts the spatial structural features within each input channel without increasing channel interaction. The following first stabilizing layer and the first activation function together form the first "normalization-activation" unit, designed to accelerate model convergence, stabilize the training process, and introduce necessary nonlinearity into the features. Subsequently, the pointwise convolutional layer is used to perform cross-channel information fusion and dimensionality compression on the previously processed feature map, a crucial step in achieving feature dimensionality reduction and cross-channel information interaction. Finally, the second stabilizing layer and the second activation function form the second "normalization-activation" unit, performing final numerical distribution calibration and nonlinear mapping on the compressed feature map, thereby outputting a feature map that is numerically more stable and expressively richer.

[0048] Furthermore, for any modality's base feature map, the depthwise convolutional layer first performs spatial convolution independently on each input channel of the base feature map to extract the spatial features of all channels. Then, the first stabilizing layer performs batch normalization on the spatial features of all channels and uses the first activation function to perform nonlinear transformation on all normalized spatial features to form a multi-channel feature map.

[0049] As mentioned earlier, the lightweight convolutional structure of the first branch is a sequential processing pipeline. The base feature map obtained after processing any modality (RGB, infrared, or depth) through the S2 step... First, spatial convolution is performed independently on each input channel of the base feature map by depthwise convolutional layers. The core feature of depthwise convolution is channel decoupling, that is, each output channel is generated by convolution of only the corresponding single input channel, rather than a linear combination of all input channels in standard convolution.

[0050] Let the depthwise convolution kernel be... Its three dimensions represent the number of channels, kernel height, and kernel width, respectively. For the c-th channel, Depthwise convolution is represented as: in, For spatial location index, ; The c-th channel of the input feature map is located in the neighborhood. The pixel values ​​are zero-padded at the edges. Output Spatial resolution The number of channels remains unchanged. .

[0051] Subsequently, the first stabilizing layer performs batch normalization on the spatial features of all channels. Batch normalization stabilizes the data distribution during network training and alleviates internal covariate shift issues by standardizing the feature responses of each channel. For the c-th channel, the specific formula for batch normalization is: in, Let be the mean value of the c-th channel in the training batch. Let be the variance of the c-th channel in the training batch. Both are calculated statistically from the current batch during training and replaced by the moving average estimate saved during the training phase during inference. For learnable scaling parameters; The offset parameters are learnable, and both are optimized through backpropagation; This is a numerical stability constant to prevent division by zero errors; output Each channel feature is normalized to a standard distribution with a mean of 0 and a variance of 1, and then the flexibility of expressive power is restored through learnable parameters.

[0052] Subsequently, a nonlinear transformation is performed on all normalized spatial features using the first activation function. In this embodiment, the ReLU activation function is used, and the specific formula is as follows: The ReLU function truncates negative responses to zero while preserving positive responses, introducing sparse nonlinearity to enhance the network's selectivity and discriminative ability regarding features. The output... This refers to the multi-channel feature map, in which each channel has been subjected to depthwise spatial filtering, distribution stabilization, and nonlinear activation to form a mid-level feature representation rich in local spatial patterns, laying the foundation for subsequent cross-channel fusion via pointwise convolution.

[0053] Furthermore, the obtained multi-channel feature maps are input into a pointwise convolutional layer for cross-channel linear combination and channel dimension compression to form compressed feature maps. Then, the second stabilization layer performs batch normalization on the compressed feature maps, and the second activation function is used to perform nonlinear transformation on the normalized compressed feature maps to output the first branch features after compression and refinement.

[0054] After the depthwise convolution, the first stabilizing layer, and the first activation function described above, the first branch has obtained a multi-channel feature map that is spatially refined, numerically stable, and possesses non-linear expressive capabilities. However, each channel of this multi-channel feature map is still obtained by processing the original input channels independently; the information between different channels is isolated and has not yet interacted or fused. Furthermore, its channel dimension remains consistent with the input base feature map, failing to achieve the expected feature compression and dimensionality reduction goals. Therefore, this section will elaborate on the latter half of the lightweight convolutional structure's processing flow, aiming to achieve cross-channel information fusion, feature dimension compression, and final numerical stability and non-linear output.

[0055] The obtained multi-channel feature map The input is processed by a pointwise convolutional layer, which performs cross-channel linear combination and channel dimension compression. The pointwise convolution employs... Convolutional kernels achieve dimensionality transformation through linear projection across channels without altering spatial resolution. Let the target output channel number be... ,and The pointwise convolution kernel is Its four dimensions represent the number of output channels, the number of input channels, the kernel height, and the kernel width, respectively. For the ... One channel, The specific formula for pointwise convolution is: in, To connect the input channel c to the output channel c channel Convolution weights. This operation, for each spatial location (i,j), will... The linear combination of the input channels is as follows: Each output channel enables cross-channel feature fusion and dimensionality compression. Its spatial resolution The number of channels remains unchanged from [previous number] to [previous number]. Compress to Compression ratio is The number of parameters for pointwise convolution is The computational complexity is It achieves channel-level compression and information integration with extremely low computational cost.

[0056] The linear combination and dimensionality compression operations across channels will again alter the numerical distribution of each channel in the feature map. To ensure the effectiveness of subsequent activation functions and the stability of network training, this embodiment introduces a second stabilization layer after the pointwise convolutional layer to compress the feature map. Perform a second batch normalization process.

[0057] For the One channel, The specific formula for batch normalization is: in, and The first Batch mean and variance of the channel; For learnable scaling parameters, These are learnable offset parameters; This is the numerical stability constant. After processing by the second stabilization layer, the compressed feature map is output. The numerical distribution was calibrated again to a stable range with a mean of 0 and a variance of 1. The compressed feature distribution was stabilized, which is beneficial for subsequent activation and deep network training, and creates ideal conditions for the effective operation of subsequent activation functions.

[0058] Finally, a second activation function is used to perform a nonlinear transformation on the normalized compressed feature map, enabling the first branch to learn and express more complex feature mapping relationships. This embodiment uses the ReLU activation function, and the specific transformation formula is as follows: Thus, the first branch successfully transforms the input basic feature map into a compact feature representation with fewer channels, unchanged spatial dimensions, and feature information that has undergone depthwise refinement and cross-channel fusion through a complete, lightweight convolutional structure consisting of depthwise convolutions, two "normalization-activation" units, and a centrally located pointwise convolution. The features from the first branch will be fused with the features from the second branch, originating from the same modality, in subsequent steps to jointly constitute the final, highly efficient dimensionality reduction features for that modality.

[0059] Furthermore, the second branch in the dual-branch processing performs max pooling screening on the basic feature maps of each modality in the following specific way: constructing a max pooling screening structure consisting of a multi-scale parallel pooling layer and an attention calibration layer.

[0060] This embodiment employs a two-stage cascaded architecture of "multi-scale parallel pooling layer - attention calibration layer," aiming to capture salient response regions at different spatial granularities and strengthen key discriminative regions through an attention mechanism. Live faces and forgery attacks exhibit differentiated feature patterns at different spatial scales. Fine-grained scales (small pooling windows) are sensitive to local micro-textures, capturing subtle differences such as skin pores and fine wrinkles; coarse-grained scales (large pooling windows) are sensitive to macro-structures, capturing structural differences such as overall contours and large-area deformations. Therefore, by setting multiple scales of max pooling operations in parallel, a multi-granularity saliency filtering mechanism is formed, enhancing the adaptability of feature representation to various types of attacks. The purpose of introducing the attention calibration layer is that the multiple feature maps generated by multi-scale pooling have different spatial resolutions and response characteristics, requiring adaptive fusion and filtering through an attention mechanism. This layer first concatenates the multi-scale features with the original feature channels to form a high-dimensional feature representation; then, through convolutional compression and channel attention weighting, it learns the importance weights of each channel, suppresses redundant responses, strengthens discriminative regions, and finally outputs filtered features that highlight the differences.

[0061] Furthermore, for any modality's base feature map, the multi-scale parallel pooling layer first performs multi-scale spatial downsampling on the base feature map in parallel, and uses multiple spatial pooling windows of different sizes to select the maximum response value in the local area they cover, generating multiple sets of pooling feature maps with different spatial resolutions.

[0062] In the dual-branch processing framework described in step S3, the primary task of the second branch is to filter and highlight the most discriminative local difference regions for liveness detection from the base feature map. To achieve this goal, the multi-scale parallel pooling layer designed in this embodiment does not employ a single-scale pooling operation, but instead constructs a multi-scale parallel processing architecture, aiming to capture and retain the strongest feature responses from all spatial observation ranges of different granularities.

[0063] For any modality, the fundamental feature map First, a multi-scale parallel pooling layer performs multi-scale spatial downsampling on the basic feature map in parallel. This embodiment sets three pooling scales, with pooling window sizes of [sizes to be filled in]. Corresponding scale index The step size is the same as the window size, that is, the step size of the k-th scale is... .

[0064] For the k-th scale, let the pooling window size be... ,in Within the local area covered by each pooling window, the max pooling operation selects the maximum response value of all pixels within that area as the output. Its mathematical expression is: Where (i,j) is the spatial index of the feature map after pooling. ; The height after pooling at the k-th scale. The width after pooling at the k-th scale; This represents the spatial resolution after pooling at the k-th scale. For channel indexing; The set of input region coordinates covering the output position (i,j) at the k-th scale is specifically as follows: Max pooling is performed independently in each channel, preserving the strongest response of each channel in the local region, suppressing background noise and weak response interference, and highlighting salient feature patterns.

[0065] In summary, the multi-scale parallel pooling layer, by configuring multiple pooling windows of different sizes, performs spatial downsampling and maximum response filtering on the base feature maps in parallel. This successfully transforms the original base feature maps into a set of pooled feature maps with varying spatial resolutions, highly condensed with the strongest local responses. This set of feature maps provides rich, multi-granular, and pre-enhanced high-quality input for subsequent attention calibration layers to perform cross-scale information integration and adaptive importance recalibration.

[0066] Furthermore, the multiple pooling feature maps are concatenated according to the channel dimension through the attention calibration layer, and convolutional compression and attention weighting are performed on the concatenated features to map the multiple pooling features to a unified channel dimension and highlight key spatial regions, outputting the second branch features that highlight the differences in the regions.

[0067] After processing by the multi-scale parallel pooling layers, the second branch has obtained three sets of pooling feature maps with different spatial resolutions: Scale 1 ( window), It retains a relatively refined spatial structure; Scale 2 ( window), Extract medium-grained regional patterns; Scale 3 ( window), This captures coarse-grained global significance.

[0068] These three sets of feature maps capture the strongest local responses from the base feature maps at different spatial scales, but they are currently isolated from each other—their spatial resolutions vary, and their information is scattered across multiple scale branches. Furthermore, features at different scales are not equally important for the final liveness detection task, and different spatial locations within the same scale also exhibit significantly different discriminative values. To address these issues, this embodiment introduces the second core component of the second branch—an attention calibration layer—after the multi-scale parallel pooling layer. This layer integrates the scattered multi-scale pooling features into a compact, unified, and highly focused second-branch feature set on the key discrimination region by sequentially performing four key steps: channel concatenation, spatial alignment, convolutional compression, and attention weighting.

[0069] The first step in the attention calibration layer is to concatenate the three pooling feature maps from the multi-scale parallel pooling layer along the channel dimension. However, since the three feature maps have different spatial resolutions, channel concatenation cannot be performed directly on the original spatial dimensions. Therefore, before concatenation, the spatial resolution of all feature maps must first be unified to a common target size.

[0070] This embodiment first uses bilinear interpolation to upsample the three sets of pooling feature maps to their original spatial resolution. The mathematical expression is: in This represents a bilinear interpolation upsampling operation with a target resolution of . ; k is the scale index, k=1 corresponds to a 2×2 pooling window, k=2 corresponds to a 4×4 pooling window, and k=3 corresponds to an 8×8 pooling window; the output This is the feature map after upsampling at the k-th scale. Bilinear interpolation upsampling operation achieves smooth spatial resolution restoration by calculating the weighted average of the target location's four neighboring pixels in the source feature map, effectively avoiding the checkerboard effect caused by hard interpolation.

[0071] The three sets of upsampled pooled feature maps are compared with the original base feature map. Concatenated according to channel dimensions, the mathematical expression is: in, This indicates a channel-level concatenation operation; The spliced ​​feature map has a total of [number] channels. It contains complete information from pooling features at three scales and the original features, forming a high-dimensional multi-scale feature representation.

[0072] Convolutional compression and attention weighting are performed on the concatenated features. First, through... The convolution kernel performs channel dimension compression; let the kernel weights be... The bias vector is ,in To determine the target number of output channels, Given the number of input channels and the 1×1 kernel size, the specific formula for convolution operation is: Where (i,j) is the spatial location index, i∈[1,H],j∈[1,W]; For output channel index; For input channel index; To connect the input channel c to the output channel c Channel convolution weights; output The number of channels is from Compress to This achieves cross-channel fusion and dimensional unification of multi-scale information.

[0073] Next, channel attention weighting needs to be performed on the compressed feature map to learn the importance weights of each channel. This embodiment first extracts channel statistical features through global average pooling, with the specific formula as follows: in, For the first The global average response of the channel, output channel statistical vector. .

[0074] This embodiment learns channel attention weights through a two-layer fully connected network. Let the weights of the first fully connected layer be... The bias vector is The weights of the second fully connected layer are The bias vector is , To reduce the ratio, the specific formula for the first-layer fully connected operation is: in For the reduced feature representation, It is the ReLU activation function; The specific formula for the second-layer fully connected operation is: in, Here are the attention weight vectors for each channel. The Sigmoid activation function normalizes the weights to the range (0,1).

[0075] After learning the channel attention weights through two layers of fully connected networks, the attention weights are multiplied channel by channel by the compressed feature map to complete the weighting. The mathematical expression is: Output This refers to the second branch feature that highlights the differences in the region.

[0076] In the second branch, a multi-scale parallel pooling layer first processes the base feature map by configuring max pooling windows of three scales: 2×2, 4×4, and 8×8. Parallel spatial downsampling and maximum response filtering are performed. The essence of max pooling is to retain the strongest feature response and suppress background noise and weak response interference within the local area covered by each pooling window. The 2×2 small-scale window is sensitive to fine local micro-textures and can capture salient patterns at the level of details such as skin pores and fine wrinkles; the 4×4 medium-scale window extracts regional structural differences; and the 8×8 large-scale window captures coarse-grained global salient features. The parallel operation of the three scales forms a comprehensive coverage of the strongest response regions at different spatial granularities, ensuring that both local subtle differences and macroscopic structural anomalies can be effectively captured.

[0077] Subsequently, the attention calibration layer upsamples the three pooling feature maps to the original spatial resolution H×W using bilinear interpolation, and then compares them with the original base feature maps. By splicing along the channel dimension, a total of channels are formed. High-dimensional multi-scale feature representation, i.e. This concatenation operation achieves deep fusion of the original features and multi-scale pooling features, enabling subsequent processing to utilize both the original detailed information and the filtered saliency information. A 1×1 convolution kernel is used to... Compress to The channel completes cross-scale information integration and dimensional unification, outputting a compact compressed feature map. .

[0078] Finally, global average pooling is used to extract... Global statistical characteristics of each channel The attention weights for each channel are learned via a two-layer fully connected network. The weight vector is normalized to the (0,1) range by the Sigmoid activation function, representing the importance of each channel to the final discrimination task; each channel is multiplied by its attention weight channel by channel to achieve adaptive enhancement of the discriminative channel and suppression of redundant channels.

[0079] In summary, the second-branch features obtained through the second branch... The reason it can "highlight regions of difference" lies in the following mechanism: multi-scale max pooling operations filter and retain the local regions with the strongest feature responses from multiple spatial granularities, naturally suppressing uniform backgrounds and weak response regions; cross-scale concatenation and convolutional compression achieve the organic fusion of multi-granularity saliency information; and the channel attention mechanism further adaptively amplifies the channel responses rich in discriminative information. The synergistic effect of these three mechanisms allows the second branch features to maintain the same spatial resolution H×W and channel dimension as the first branch. Under this premise, its feature distribution focuses more on those key spatial regions that are most convincing in distinguishing between living and non-living organisms, providing complementary representations rich in significant differences for subsequent fusion with the features of the first branch.

[0080] Thus, the first branch features obtained after dual-branch processing have the same spatial resolution H×W and the same channel dimension as the second branch features. The two have a feature fusion basis with dimension alignment.

[0081] This embodiment uses an element-wise addition method, adding the first branch features and the second branch features according to their corresponding spatial positions and channel indices. The mathematical expression is as follows: This is the intermediate feature map after fusion. The core significance of this fusion strategy lies in achieving information integration of two complementary feature types: the first branch focuses on compressing the channel dimension and refining features through lightweight convolution, preserving the compact semantic information encoded in depth; the second branch focuses on highlighting significantly different regions through multi-scale pooling and attention mechanisms, preserving spatial saliency information rich in discriminative cues. Element-wise addition allows the two types of features to be directly superimposed in the same dimensional space, avoiding the dimensionality expansion problem caused by channel splicing and ensuring that the information from the two feature streams can fully interact and complement each other.

[0082] For the fused feature map Further processing with equal-size convolution, namely using a convolution operation with a kernel size of 3×3, a stride of 1, and padding of 1, performs local spatial integration and channel information reorganization on the fused features without changing the spatial resolution H×W.

[0083] Let the weight parameters of the equal-size convolution be... , bias is Dimensionality reduction features The specific calculation formula is as follows: Where (i,j) is the spatial location index. For the output channel index, boundary pixels are zero-padding. The purpose of this equal-size convolution is twofold: firstly, to smooth and integrate the fused features through convolution operations within the local receptive field, eliminating the discontinuity in numerical distribution that may be introduced by element-wise addition; secondly, to adaptively adjust the contribution ratio of each channel feature through learnable convolution weights, further refining and solidifying the effective information after dual-branch fusion.

[0084] Final output This refers to the dimensionality reduction features of each modality, whose spatial resolution remains consistent with the original base feature map, while the channel dimension has been reduced from... Compress to The compression ratio is 0.5 ( It integrates the compact semantic features of the first branch with the differential significance features of the second branch, providing a compact, information-rich and highly discriminative feature representation for global average pooling and cross-modal affinity matrix construction in the subsequent S4 step.

[0085] S4: Perform global average pooling on the dimensionality reduction features of the three modalities respectively to obtain the global feature vectors of each modality. Based on the cosine similarity between the global feature vectors of the three modalities, construct a cross-modal affinity matrix, and calculate the transfer features of each modality based on the cross-modal affinity matrix. Then, fuse the transfer features of each modality with its dimensionality reduction features element by element to obtain the enhanced feature map of each modality.

[0086] After the dual-branch parallel processing and fusion in step S3, the system has generated efficient dimensionality-reduced feature maps for RGB, infrared, and depth modalities respectively. Within their respective modalities, these dimensionality-reduced feature maps have completed the transformation from basic features to compact representations—preserving both global compressed and refined information and highlighting key local differences. However, at this point, the dimensionality-reduced features of the three modalities are still isolated, each carrying only a single physical dimension (texture, thermal radiation, geometric structure) of discriminative information; no information interaction or complementarity has occurred between the modalities. In real-world face liveness detection scenarios, different modalities exhibit significant complementary sensitivities to different types of forgery attacks. For example, the RGB modality is extremely sensitive to the pixel grid and screen moiré patterns of printed photos, but it may be difficult to distinguish highly realistic 3D silicone masks based solely on texture; the infrared modality can effectively capture the difference in thermal radiation between live skin and counterfeit materials, giving it a natural advantage against attacks that lack the thermal properties of real skin, such as masks and photos, but it lacks texture details; the depth modality can intuitively reflect the three-dimensional geometric structure of a human face, making it highly immune to attacks on planar photos and video playback, but when faced with high-precision 3D head models, even simple structural information can be deceived.

[0087] To enable the three modalities to learn from and enhance each other, this embodiment introduces a cross-modal knowledge transfer and reinforcement mechanism based on feature similarity in step S4: First, the dimensionality-reduced feature maps of each modality are compressed into global feature vectors that represent their "global style" or "high-order semantic attributes"; then, by calculating the similarity between these global feature vectors, the "affinity" of the three modalities in the high-level semantic space is quantified, and a cross-modal affinity matrix is ​​constructed accordingly; finally, based on the affinity matrix, feature transfer is selectively performed between different modalities, and the discriminative information of similar modalities is used to reinforce the feature expression of the current modality.

[0088] Specifically, in this embodiment, a global average pooling operation is first performed on the dimensionality reduction features of each modality output in step S3. This compresses the spatial information of each channel into a single global statistic, resulting in a global feature vector for each modality. The essence of this operation is to transform high-dimensional spatial distribution features into compact semantic representations, eliminate the absolute dependence on spatial location, extract modality-level global semantic fingerprints, and provide a feature space of a unified scale for subsequent cross-modal similarity comparisons.

[0089] Secondly, based on the pairwise cosine similarity of the global feature vectors of the three modalities, a 3×3 cross-modal affinity matrix is ​​constructed. This matrix quantifies the semantic affinity relationship between the RGB, infrared, and depth modalities in a numerical manner. The diagonal elements are always 1, representing complete similarity between modalities; the off-diagonal elements take values ​​in the range [-1, 1], with positive values ​​indicating high semantic consistency between modalities and negative values ​​indicating semantic divergence. The establishment of the affinity matrix realizes a paradigm shift from isolated modality processing to associated modality analysis, providing a quantifiable basis for feature transfer.

[0090] Furthermore, based on the affinity matrix, the transfer features of each modality are calculated. Transferable elements are filtered by setting a similarity threshold. The global feature vectors of highly similar modalities (column modalities) are mapped to the feature space of the target modality (row modalities) via a learnable projection matrix to obtain transfer components. All transfer components of the same target modality are weighted and summed to form the transfer features of that modality. The core value of this mechanism lies in achieving information complementarity between modalities: when a modality suffers information loss or degradation due to factors such as illumination, occlusion, or noise, information can be transferred from other semantically similar modalities to supplement the information, enhancing the robustness and completeness of feature representation.

[0091] Finally, the transfer features of each modality are fused element-wise with their dimensionality-reduced features to obtain augmented feature maps for each modality. This fusion operation achieves the organic integration of local detail information and global semantic information: the dimensionality-reduced features retain rich spatial structural details and local discriminative cues, while the transfer features inject cross-modal global semantic priors and complementary information. The superposition of the two enables the augmented feature maps to possess both fine spatial resolution and complete semantic representation capabilities, laying a high-quality feature foundation for the subsequent attention weighting and modality fusion steps in S5.

[0092] Please see Figure 4 The example diagram of the cross-modal affinity matrix shown illustrates a comparison of cross-modal affinity matrices for live and non-live samples. For live samples, the cosine similarity between the global feature vectors of the RGB, infrared, and depth modalities is generally higher than 0.75, indicating that real faces have inherent semantic consistency in texture, thermal radiation, and geometric structure dimensions, providing a high-confidence association basis for cross-modal feature transfer. For non-live samples, the similarity between modalities significantly decreases to the range of 0.28-0.42, reflecting the essential differences in the representation of forgery attacks across different physical dimensions. This comparison verifies the effectiveness of the similarity threshold-based strategy for selecting transferable elements in this embodiment—feature transfer is triggered when the off-diagonal elements of the affinity matrix exceed a set threshold, ensuring that live samples receive sufficient cross-modal information reinforcement while preventing non-live samples from introducing interfering information due to erroneous transfer.

[0093] Furthermore, the specific method for constructing the cross-modal affinity matrix based on the cosine similarity between the global feature vectors of the three modalities is as follows: For the global feature vectors of the three modalities, calculate the cosine similarity between each pair; construct a third-order matrix with the three modalities as rows and columns respectively, so that the diagonal elements are 1 and the off-diagonal elements are the cosine similarity between the global feature vectors of the corresponding row mode and column mode, and use this third-order matrix as the cross-modal affinity matrix.

[0094] First, for the global feature vectors of the three modalities, calculate the cosine similarity between each pair of them. Let the global feature vectors obtained after global average pooling for RGB, infrared, and depth modalities be respectively... ,in Given the feature vector dimension, then any two modalities and ( The formula for calculating the cosine similarity between () is: in, Representing modes The formula takes the value of the c-th dimension of the global feature vector; the numerator is the dot product of the two vectors, representing the degree of overlap of their projections in a direction; the denominator is the product of the Euclidean norms of the two vectors. By using this product as the denominator, the influence of vector length differences can be eliminated, allowing the similarity to purely reflect the degree of overlap in a direction. This normalization process ensures that global feature vectors of different modalities have a fair and comparable basis when comparing similarity, avoiding misjudgments caused by differences in feature numerical scales.

[0095] Based on the above calculations, a 3×3 third-order matrix is ​​constructed using the three modes as rows and columns respectively. The mathematical expression for the cross-modal affinity matrix is ​​as follows: In this model, all diagonal elements have a value of 1, indicating that any modality is completely similar to itself; this is an inherent property of cosine similarity. Off-diagonal elements... The value represents the cosine similarity between the global feature vectors of the corresponding row mode (row i) and column mode (column j).

[0096] In the cross-modal affinity matrix constructed in this embodiment, the row direction of the matrix represents the "target mode" (the mode that receives migration information) and the column direction represents the "source mode" (the mode that provides migration information). This row and column definition provides a clear directional basis for subsequent calculation of migration features based on the affinity matrix.

[0097] Furthermore, the specific method for calculating the modal transfer features based on the cross-modal affinity matrix is ​​as follows: set a similarity threshold, and select all off-diagonal elements in the cross-modal affinity matrix whose values ​​are greater than the similarity threshold as transferable elements.

[0098] Since global feature vectors are typically processed by non-negative activation functions such as ReLU, their actual value range is [0, +∞). The effective value range for cosine similarity is [0, 1]. Therefore, the similarity threshold must also be set within the range of [0, 1]. Only when Only when the column modality j and row modality i are considered to have sufficient semantic affinity can feature transfer from modality j to modality i be allowed.

[0099] Furthermore, the diagonal elements in the cross-modal affinity matrix all have a value of 1, indicating that each modality is completely similar to itself, and there is no need for transfer features; while the off-diagonal elements truly reflect the degree of semantic association between different modalities. Therefore, the operation of filtering transferable elements only needs to be performed on the 6 off-diagonal elements (actually 3 independent elements considering symmetry) to eliminate the interference of the diagonal elements.

[0100] In this embodiment, the preferred setting range for the similarity threshold is 0.5 to 0.8, for the following reasons: The lower limit of the similarity threshold is set to 0.5. When the similarity is below 0.5, it indicates that the angle between the two modalities in the semantic space exceeds 60°, and their directional consistency is weak. Forcing feature transfer at this point would introduce external information that deviates significantly from the essential attributes of the target modality, leading to feature space chaos and weakening the model's discriminative ability. The threshold of 0.5 ensures that the source modality and the target modality have at least a moderate degree of semantic relevance.

[0101] The upper limit of the similarity threshold is set to 0.8. When the similarity is higher than 0.8, it indicates that the two modalities describe almost the same semantic content, and the information gain brought by the transfer is limited. Moreover, an overly strict threshold will cause a large number of samples to fail to meet the transfer conditions, rendering the cross-modal complementarity mechanism ineffective. The threshold of 0.8 ensures high-quality transfer information while retaining a moderate probability of transfer triggering.

[0102] This embodiment adopts a dynamic adjustment strategy for setting the similarity threshold, which can be fine-tuned according to the characteristics of specific modality combinations. For example, RGB and infrared modalities usually have high semantic consistency (especially in the live skin region), so the threshold for this direction can be set in the higher range of 0.7-0.8; while the depth modality is easily affected by measurement noise, and the similarity with RGB / infrared fluctuates greatly, so it can be appropriately reduced to the range of 0.5-0.6 to ensure that necessary feature supplementation can still be obtained from other modalities when the depth data quality deteriorates.

[0103] Through the threshold screening mechanism described above, the system can adaptively identify and utilize high-confidence associations between modalities, and perform feature transfer only when it is necessary and the information is reliable, thereby effectively improving the intelligence and robustness of multimodal fusion.

[0104] Furthermore, for any transferable element, the global feature vector of the column mode of the element is mapped to the feature space of the global feature vector of the row mode of the element through the learnable projection matrix corresponding to the current transfer direction, so as to obtain the transfer component of the row mode of the element.

[0105] After filtering based on similarity thresholds, the system has identified which off-diagonal elements in the cross-modal affinity matrix are "transferable elements." Each element marked as transferable represents an effective path for feature transfer from the column modality to the row modality. Although the source modality (column modality) and target modality (row modality) selected by cosine similarity have high semantic consistency, the global feature vectors of different modalities originate from their respective independent shallow convolutional networks. Their feature spaces have inherent differences in basis orientation, numerical distribution, and semantic encoding methods, making direct linear superposition impossible. Therefore, adaptive mapping and alignment of the feature space must be achieved through a learnable projection matrix.

[0106] In the cross-modal affinity matrix A, the off-diagonal elements The physical meaning is as follows: column index j represents the source mode, i.e., the feature provider, and row index i represents the target mode, i.e., the feature receiver. The current migration direction is defined as from column mode j to row mode i. Since the migration between the three modes is direction-specific, for example, the migration from RGB to infrared and the migration from infrared to RGB are two different transformation requirements, a learnable projection matrix needs to be configured independently for each possible migration direction.

[0107] Let the set of the three modes be . Then all possible migration directions constitute an ordered set of mode pairs. It contains 6 elements in total; for each migration direction Configure a learnable projection matrix Its dimensions are To ensure that the feature vector dimensions remain consistent before and after mapping, the matrix is ​​generated as follows: The initial weight matrix is ​​obtained by random initialization using a normal distribution with a mean of 0 and a standard deviation of 0.01. The elements in this initial weight matrix are represented as This projection matrix serves as a trainable parameter for the network and is continuously optimized during backpropagation. Each line in Equivalent to the current target mode number A synthetic filter with dimensional features, whose column elements This matrix represents the contribution weight of the c-th dimension feature of the source mode to the synthesis process. The linear transformation achieved through this matrix essentially rotates, scales, and projects the feature vector of the source mode from the original coordinate system to the feature coordinate system of the target mode, eliminating the feature space basis misalignment problem caused by independent initialization of network parameters.

[0108] For any transferable element The global eigenvector of the column mode j of this element. The projection matrix corresponding to the current migration direction (i,j) is used. Perform a linear transformation to map the data into the feature space of row mode i, and obtain the transfer component of row mode i of that element. The specific calculation formula is as follows: The expansion of this matrix-vector multiplication is as follows: in, This represents the first projection matrix. The element in row c; This represents the value of the c-th dimension of the global eigenvector of source mode j. Indicates the first migration component The linear transformation operation results in the transfer component having numerical values ​​in each dimension. It has direct comparability and fusion with the inherent characteristics of the current target mode i.

[0109] In this embodiment, the projection matrix The parameters are not preset and fixed, but are jointly optimized with the entire liveness detection network through an end-to-end backpropagation algorithm. During training, the network automatically learns the optimal projection transformation strategy: when there is a correspondence between a certain type of semantic feature of the source modality and the target modality (e.g., the consistency between RGB texture details and infrared thermal radiation distribution in the live skin region), the projection matrix will strengthen the feature transfer in that direction; when there are inherent differences between the two types of features (e.g., the inconsistency between depth edges and RGB color boundaries), the projection matrix will adaptively suppress interfering mappings. This data-driven learning approach ensures that the feature transfer mechanism can adapt to the complex and ever-changing multimodal association patterns in real-world scenarios.

[0110] Through the construction, initialization, and adaptive learning of the projection matrix, each transferable element is transformed into a transfer component aligned with the target modality feature space, laying a homogeneous feature foundation for subsequent weighted fusion.

[0111] Furthermore, in the cross-modal affinity matrix, all migration components of the same row of modes are weighted and summed to obtain the migration characteristics of each mode.

[0112] In a cross-modal affinity matrix, for a given target modality, there may be multiple source modalities that satisfy the similarity threshold condition, i.e., multiple transferable elements. For example, when the RGB modality has high semantic consistency with both the infrared and depth modalities, both the infrared and depth modalities can serve as source modalities for transfer features to the RGB modality. In this case, the target modality will correspond to multiple transfer components, which originate from different feature spaces and carry complementary semantic information. Therefore, this embodiment integrates different transfer components of the same target modality into a single transfer feature through a weighted fusion strategy. The rationality of this design is based on the following considerations: the similarity in the cross-modal affinity matrix. The semantic consistency between source modality j and target modality i was quantified. The higher the similarity, the more reliable the transfer information provided by the source modality and the better the fit with the target modality. Therefore, it should be given a higher fusion weight. Conversely, although the source modality with low similarity meets the threshold condition, its transfer information may have some deviation. Therefore, its influence weight should be appropriately reduced.

[0113] Let the set of all transferable source modes corresponding to target mode i be formed. Its corresponding migration component is Affinity weight is Then the transfer characteristics of target mode i The calculation formula is: Where j traverses the set All source modes, Similarly, traverse the set. Used to calculate the normalized denominator, then the denominator As a normalization factor, it ensures that the sum of all weights is 1, thus giving the weighted result numerical stability; while the numerator This represents the affinity between source mode j and target mode i. Represents the source mode Affinity with target mode i; Let be the migration component from source mode j to target mode i. The expanded form of the above formula is: in, Indicates the transfer feature number The values ​​of each dimension Represents migration components No. The numerical values ​​of each dimension.

[0114] When target mode i has no source mode that meets the conditions, it indicates that the semantic consistency between this mode and other modes is low. In this case, transfer features are set. It is the zero vector, that is This processing strategy avoids forcibly transferring unreliable information, ensuring that feature enhancement is only performed when credible cross-modal associations exist, thus maintaining the purity of feature representation.

[0115] Through the aforementioned weighted summation mechanism, the transfer features of each modality can achieve the following technical objectives: First, multi-source information integration, fusing complementary semantic information from different modalities into a unified feature representation; second, adaptive quality control, automatically adjusting the contribution ratio of each source modality through affinity weighting to suppress interference from low-quality transfer information; and third, dimensionality consistency maintenance, ensuring that the transfer features have the same dimension as the global feature vector and the dimensionality-reduced features. This provides a basis for dimensional matching for subsequent element-by-element fusion operations.

[0116] Please see Figure 5 The illustrated feature transfer diagram, using RGB mode receiving transferred features as an example, demonstrates a feature transfer mechanism based on a cross-modal affinity matrix. When the texture information of the RGB mode degrades due to overexposure of ambient light, the system detects that the similarity between the infrared mode and the RGB mode is as high as 0.85, exceeding the high similarity threshold (≥0.80), triggering feature transfer from the infrared mode to the RGB mode—mapping the infrared thermal radiation features to the RGB texture feature space via a learnable projection matrix to enhance the discriminative ability of the RGB mode; while the similarity between the depth mode and the RGB mode is only 0.62, falling within the medium transfer range, its geometric structural features, after projection, participate in weighted fusion as auxiliary information. This diagram intuitively presents the adaptive control of the transfer path by the similarity threshold screening mechanism: strong transfer is implemented between high-similarity modes, and weak transfer or suppressed transfer is implemented between low-similarity modes, ensuring the accuracy and reliability of feature enhancement.

[0117] S5: Perform channel and spatial attention weighting on the augmented feature maps of the three modalities respectively, and concatenate them according to the channels to obtain the face fusion feature map. Perform dimensionality reduction, pooling and fully connected processing on the obtained face fusion feature map in sequence to obtain the fusion feature vector. Input the obtained fusion feature vector into the preset binary classifier and output the live or non-live face discrimination result.

[0118] This step is the final decision-making stage of the entire face liveness detection method oriented towards multimodal feature fusion. Its core task is to perform deep fusion and classification of the three modal features enhanced by step S4, and output the result of determining whether the face to be detected is live or not.

[0119] Modal enhancement feature maps obtained in step S4. Although it has integrated local detail information and global transfer semantics, the importance of each channel and spatial location within it for the final liveness detection task is not equal. For example, some channels may encode background noise unrelated to the attack type, and some spatial regions may correspond to non-face image edges. If modal concatenation and classification are performed directly, this redundant information will interfere with the decision-making process. Therefore, this step first applies channel and spatial attention weights to the augmented feature maps of the three modalities, and strengthens the discriminative channels and key spatial regions through an adaptive learning mechanism to suppress irrelevant responses and improve the purity and specificity of feature representation.

[0120] After attention weighting of each modality, the feature maps of the three modalities possess high discriminativeness and complementarity. To achieve the organic integration of cross-modal information, this embodiment employs a channel stitching strategy, stacking the attention-weighted feature maps of RGB, infrared, and depth modalities along the channel dimension to form a face fusion feature map. This fusion method preserves the complete information of each modality, avoiding information loss that may occur in early fusion. However, the channel dimension of the fused feature map expands to three times the original, and directly using it for classification will lead to a surge in parameters and the risk of overfitting. Therefore, it is necessary to sequentially process the high-dimensional feature map through dimensionality reduction convolution, global pooling, and fully connected layers to compress it into a compact fused feature vector. ,in The output dimension of a fully connected layer is typically set to a value between 128 and 512 to balance expressive power and computational efficiency.

[0121] Ultimately, the feature vectors will be fused. The system takes a pre-defined binary classifier as input, which consists of a fully connected network layer and a Softmax activation function. The classifier outputs the probability distribution of the detected face belonging to the "live" and "non-live" categories. By comparing the probability values ​​of the two categories, the one with the larger probability is taken as the final judgment result, thus completing the entire liveness detection process.

[0122] Furthermore, the specific method for weighting the enhanced feature maps of the three modalities with channel and spatial attention is as follows: a hybrid weighted structure containing a channel attention module and a spatial attention module is constructed, wherein the channel attention module consists of a global pooling unit, a fully connected network layer and a first weight function, and the spatial attention module consists of a channel pooling unit, a channel convolutional layer and a second weight function.

[0123] Channel attention and spatial attention focus on different attribute dimensions of feature maps: channel attention learns "which features are more important," i.e., the contribution of each channel's feature map to the discrimination task; spatial attention learns "where is more important," i.e., which spatial regions on the feature map carry key discrimination cues. The two are cascaded to form a hybrid weighted structure, first filtering channels and then locating spatially, forming a progressive attention mechanism of "selecting channels first, then locations." This design aligns with the hierarchical nature of feature cognition—first identifying feature types with discrimination potential, then focusing on the spatial distribution of those features, avoiding mutual interference caused by simultaneous optimization of two dimensions, and improving the stability and interpretability of attention learning.

[0124] The channel attention module consists of three sequentially connected components: First, there's the global pooling unit, which augments the input feature map. Perform global max pooling and global average pooling operations respectively to generate two channel statistical vectors. Global max pooling captures the strongest response value of each channel, reflecting the significance of the feature; global average pooling captures the overall response level of each channel, reflecting the universality of the feature. The parallel use of the two pooling methods allows channel attention to focus on both "peak features" and "average features" simultaneously, enhancing sensitivity to anomalous responses.

[0125] The second layer is the fully connected network. The two pooling results are sequentially input into a two-layer fully connected network with shared parameters. The first layer compresses and interacts the channel dimension, and the second layer restores the original dimension. Output two transformed channel vectors.

[0126] Finally, there is the first weighting function, which in this embodiment uses the Sigmoid activation function. The two channel vectors are added element-wise and then fused to map to the (0,1) range to generate channel attention weights. The weight vector is multiplied channel by channel with the original feature map to achieve adaptive recalibration of the channel dimension.

[0127] The spatial attention module also consists of three sequentially connected components: First, there is the channel pooling unit, which processes the feature map after channel weighting. Perform global max pooling and global average pooling along the channel dimension respectively to generate two spatial description graphs. , representing the peak and average responses of each spatial location in the channel dimension.

[0128] The second step is the channel convolutional layer, which concatenates the two spatial description maps according to the channel dimension. Then, feature mapping and channel compression are performed using convolution kernels to output a single-channel spatial response map. In this convolutional layer, the kernel size used in this embodiment is 7×7. The large kernel size ensures a sufficient receptive field to capture the contextual relationships of the spatial region.

[0129] Finally, the second weighting function, which in this embodiment also uses the Sigmoid activation function, normalizes the spatial response map after convolution into a spatial attention weight map. It is then multiplied with the channel-weighted feature map at each spatial location to achieve adaptive focusing of the spatial dimension.

[0130] Through the above hybrid weighting structure, the enhanced feature maps of the three modalities are all transformed into attention-weighted feature maps with accurate channel selection and clear spatial positioning, providing high-quality feature input for subsequent modality fusion and classification decisions.

[0131] Furthermore, for any modality's augmented feature map, it is input into the channel attention module. The global pooling unit performs global max pooling and global average pooling operations on the augmented feature map of the current modality, respectively. The two pooling results are then sequentially input into the fully connected network layer to obtain two pooling vectors corresponding to the two pooling results. The two pooling vectors are then added and fused element by element, and the channel attention weights are generated on the fused vector through the first weight function. The obtained channel attention weights are then multiplied element by element with the augmented feature map of the current modality to obtain the channel weighted feature map of the current modality.

[0132] For any modality (RGB, infrared, or depth) augmented feature map The global pooling unit performs two complementary aggregation operations along the spatial dimension H×W: Global max pooling extracts the strongest response value of each channel, reflecting the saliency peak of that channel's feature in the entire image. It is sensitive to sparse but strong discriminative cues (such as local anomalies in attack traces). Its mathematical expression is: Global average pooling extracts the mean of the overall response for each channel, reflecting the spatial distribution density and prevalence of the channel's features. It is representative of widely distributed semantic information (such as the overall texture pattern of living skin), and its mathematical expression is: Two pooling results The characteristics of the channel are characterized from two statistical perspectives: "extreme values" and "mean values," forming complementary channel description information.

[0133] The two pooling results were then compared. Two fully connected layers with shared parameters are input separately, and the channel importance weights are learned. Let the weights of the first fully connected layer be... , bias is Let the weights of the second fully connected layer be... , bias is The transformation process is as follows: in, It is the ReLU activation function; This is an intermediate compressed representation; The two pooling vectors corresponding to the two pooling results encode the channel importance information after nonlinear transformation.

[0134] The two pooling vectors are then added element-wise and fused together. The channel evaluation results of the extreme value path and the mean path are combined, and the expression is: This is the intermediate vector after fusion. It is then processed through the first weighting function, namely the Sigmoid activation function. For the fused intermediate vector The channel attention weights are generated using the following formula: income The values ​​of each element are in the range of (0,1). The closer the value is to 1, the higher the importance of the corresponding channel. The closer the value is to 0, the more the response of the channel should be suppressed.

[0135] Finally, the obtained channel attention weights are multiplied element-wise with the augmented feature map of the current modality to achieve adaptive recalibration of the channel dimension. This multiplication operation broadcasts the vectors via a broadcast mechanism. Expand to Executed after dimension, the expression is: Output This is the channel-weighted feature map of the current modality. The responses of each channel have been recalibrated according to the importance weights of adaptive learning. Discriminative channels are enhanced and redundant channels are suppressed, providing a clean feature base after channel filtering for subsequent spatial attention processing.

[0136] Furthermore, the channel-weighted feature map of the current modality is input into the spatial attention module. The channel pooling unit performs global max pooling and global average pooling operations on the channel-weighted feature map along the channel dimension. The two pooling results are concatenated according to the channel. The concatenated result is input into the channel convolutional layer for feature mapping. The spatial attention weights are generated on the convolutional feature vector through the second weight function. The obtained spatial attention weights are multiplied element-wise with the channel-weighted feature map of the current modality to obtain the attention-weighted feature map of the current modality.

[0137] The resulting channel-weighted feature map of the current mode. The channel pooling unit of the input spatial attention module. Unlike the global pooling unit, which operates along the spatial dimension, the channel pooling unit operates along the channel dimension. Perform aggregation to generate statistical descriptions of spatial locations.

[0138] Global max pooling extracts the strongest channel response at each spatial location along the channel dimension, reflecting whether there is significant discriminative feature activation at that location. It is sensitive to local anomalies (such as prosthesis edges and screen moiré patterns). Its mathematical expression is: Global average pooling extracts the average channel response at each spatial location along the channel dimension, reflecting the overall feature activity and semantic importance at that location. Its mathematical expression is: Two pooling results All are single-channel spatial feature maps, which characterize the importance of each spatial location from the perspectives of "peak significance" and "average activity", forming complementary spatial description information.

[0139] The two pooling results are concatenated along the channel dimension to generate a dual-channel spatial feature map, expressed as follows: This concatenation operation integrates the spatial descriptions of max pooling and average pooling into a unified multi-attribute feature representation, enabling subsequent convolutional layers to simultaneously utilize both statistical properties for spatial importance assessment, thus avoiding the one-sided representation that may result from a single pooling method.

[0140] splicing result The input channel convolutional layer performs feature mapping. This embodiment uses a 7×7 convolutional kernel with a stride of 1 and padding of 3 to ensure that the output spatial resolution is consistent with the input, i.e., H×W, while providing sufficient receptive field to capture the contextual relationships of spatial regions. Let the convolutional kernel weights be... , bias is The specific formula for convolution operation is: Output It is a single-channel spatial response map, and the numerical value of each position reflects the discriminative importance of that position and its neighborhood, realizing a nonlinear mapping from the original channel features to the spatial attention score.

[0141] Through the second weighting function, namely the Sigmoid activation function A spatial attention weight map is generated from the convolutional spatial response map, using the following formula: income The values ​​of each element are in the range of (0,1). The closer the value is to 1, the more attention the corresponding spatial location is worth. The closer the value is to 0, the more likely the location is to be background or interference area.

[0142] Finally, the obtained spatial attention weights are multiplied element-wise with the channel-weighted feature map of the current modality to achieve adaptive recalibration of the spatial dimension. The mathematical expression is as follows: This multiplication operation broadcasts the spatial weight map of a single channel via a broadcast mechanism. Expand to Execute after dimensioning, applying a consistent weighting strategy to the same spatial location across all channels.

[0143] Output This is the attention-weighted feature map of the current modality. Its channel dimension has been filtered by the channel attention module, and its spatial dimension has been located by the spatial attention module. Discriminative information is enhanced in both channel and spatial dimensions, providing a high-quality feature representation with precise focus for subsequent cross-modal fusion and classification decisions.

[0144] Furthermore, the specific method for obtaining the face fusion feature map is as follows: perform simple linear transformations on the attention-weighted feature maps of the three modalities respectively to align the channel dimensions of the three modalities, and then stitch the aligned attention-weighted feature maps of each modality according to the order of RGB modality, infrared modality and depth modality, according to the channels to obtain the face fusion feature map.

[0145] After the processing described above, attention-weighted feature maps of the RGB, infrared, and depth modalities are generated. Although they have the same spatial resolution H×W and channel dimensions However, the numerical distribution of its feature space and semantic encoding still exhibit modality-specific differences. To ensure that the subsequently concatenated features have a consistent numerical scale and compatibility, simple linear transformations need to be performed on each of the three. This embodiment uses a 1×1 convolutional layer to implement the linear transformation.

[0146] Let the target alignment channel dimension be... Then for any mode Its transformation formula is: in, Let m be the alignment transformation weight matrix. It is the bias vector; The 1×1 convolutional layer used in this embodiment achieves numerical normalization and dimensional alignment of the feature space through linear combination across channels without changing the spatial resolution, laying a homogeneous foundation for subsequent channel stitching.

[0147] The aligned attention-weighted feature maps of each modality Following a fixed order of RGB mode, infrared mode, and depth mode, the data is stitched along the channel dimension, as expressed in the following expression: in, This represents a tensor concatenation operation along the channel dimension; the output is a face fusion feature map. The channel dimension is expanded to The spatial resolution remains unchanged at H×W.

[0148] In this embodiment, the stitching order is arranged in a fixed sequence: RGB mode, infrared mode, and depth mode. This is based on the following: the RGB mode contains the richest color and texture details, making it the first channel group for subsequent network use of visual information; the infrared mode provides thermal radiation features, serving as the second channel group to inherit the optical information from RGB; and the depth mode provides geometric structure information, supplementing three-dimensional spatial cues as the third channel group. This fixed order ensures consistent feature parsing logic during training and inference, avoiding feature localization errors caused by disordered modal order.

[0149] Through the above processing, the face fusion feature map achieves explicit fusion of information from three modalities, which not only preserves the complete discrimination clues of each modality, but also forms a joint representation of multimodal features through the expansion of the channel dimension.

[0150] Furthermore, the specific method for inputting the fused feature vector into a preset binary classifier and outputting the result of live or non-live face discrimination is as follows: Construct a binary classifier containing a fully connected network layer and a Softmax activation function. Input the fused feature vector obtained from the face fusion features into the binary classifier. After passing through the fully connected network layer, the fused feature vector is mapped into a two-dimensional vector. Then, the Softmax activation function is used to convert the obtained two-dimensional vector into a probability distribution of live and non-live faces. Compare the probability values ​​of the two results and take the one with the larger probability value as the final discrimination result.

[0151] Currently, the obtained face fusion feature maps Although rich in multimodal information, its high-dimensionality would lead to a surge in parameters and computational burden if directly used for classification. Therefore, this embodiment performs dimensionality reduction, pooling, and fully connected processing on the face fusion feature map sequentially to achieve hierarchical compression and semantic extraction of features.

[0152] First, a 1×1 convolutional layer is used... Perform channel dimension compression, assuming the target number of channels for dimensionality reduction is... Output dimensionality-reduced feature map This operation integrates multimodal redundant information through cross-channel linear combination, reducing the dimensionality complexity of subsequent processing.

[0153] Next, global average pooling is performed on the dimensionality-reduced feature map, compressing the spatial dimension H×W into a single global description, generating a pooled feature vector. Its mathematical expression is: This operation eliminates the absolute dependence on spatial location, extracts the overall semantic representation of the feature map, and significantly reduces the feature dimensionality.

[0154] Next, pool the feature vectors. The input to the fully connected network layer is mapped to the feature space required for classification. Let the output dimension of the fully connected layer be . The weight is , bias is Then the feature vectors are fused. The calculation formula is: in, The ReLU activation function is used. This fused feature vector is a compact semantic representation of the entire face fusion feature map after multi-level compression. It has low dimensionality, high information density, and strong discriminative power, and can be directly input into a classifier for final decision-making.

[0155] Construct a binary classifier that includes a fully connected network layer and a Softmax activation function.

[0156] The first layer of this binary classifier is a fully connected mapping layer that fuses feature vectors. Mapped to a two-dimensional vector Corresponding to the two categories of "living" and "non-living", its mathematical expression is: in, This is the classification weight matrix. This is for classification bias.

[0157] The second layer of the classifier uses the Softmax activation function to convert the resulting two-dimensional vector into a probability distribution. Let k=1 represent a live object and k=2 represent a non-live object. Then, for any class k, the formula for calculating its output probability is: The obtained p(1), p(2)∈(0,1), and p(1)+p(2)=1, where p(1) and p(2) represent the posterior probabilities of the face to be detected being a live or non-live person, respectively.

[0158] Compare the probability values ​​of the two outcomes, and take the one with the larger probability value as the final judgment result. If the result is "live", then the output is "live". If the result is not positive, the output will be "not alive".

[0159] Through the above processing, this embodiment realizes a complete mapping from multimodal fusion features to binary classification results, providing interpretable decision-making basis in a probabilistic output manner, while retaining probability values ​​for subsequent risk assessment and threshold optimization. It is the final output link of the entire face liveness detection method oriented towards multimodal feature fusion.

[0160] Please see Figure 6 The present invention also provides a face liveness detection system oriented towards multimodal feature fusion, wherein the face liveness detection system oriented towards multimodal feature fusion is used to execute the above-mentioned face liveness detection method oriented towards multimodal feature fusion, comprising: The preprocessing module is used to simultaneously acquire RGB, infrared, and depth images of the face to be detected. Multiple different Gabor filter kernels are used to enhance the texture of the RGB image. The filtered RGB, infrared, and depth images are then registered at the pixel level to obtain spatially aligned input data for the three modalities. The feature extraction module is used to feed the input data of the three modalities into the shallow convolutional network respectively, and to perform channel expansion, local feature extraction and channel attention weighting on the input data of each modality in sequence to obtain the basic feature maps of the three modalities. The feature dimensionality reduction module is used to perform dual-branch processing on the basic feature maps of each modality. The first branch applies lightweight convolution to the basic feature maps of each modality to obtain compressed and refined first-branch features. The second branch performs max pooling to filter the basic feature maps of each modality to obtain second-branch features that highlight the difference regions. The first-branch features and second-branch features of each modality are fused element by element and then convolved with equal-sized features to obtain the dimensionality reduction features of each modality. The feature enhancement module is used to perform global average pooling on the dimensionality reduction features of the three modalities respectively to obtain the global feature vectors of each modality. Based on the cosine similarity between the global feature vectors of the three modalities, a cross-modal affinity matrix is ​​constructed, and the transfer features of each modality are calculated based on the cross-modal affinity matrix. The transfer features of each modality are fused with its dimensionality reduction features element by element to obtain the enhanced feature map of each modality. The liveness detection module is used to perform channel and spatial attention weighting on the augmented feature maps of the three modalities respectively, and to concatenate them according to the channels to obtain a face fusion feature map. The obtained face fusion feature map is then subjected to dimensionality reduction, pooling and full connection processing in sequence to obtain a fusion feature vector. The obtained fusion feature vector is input into a preset binary classifier to output the liveness or non-liveness face discrimination result.

[0161] The above formulas are all dimensionless calculations. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.

[0162] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented in software, the above embodiments can be implemented, in whole or in part, as a computer program product. Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution.

[0163] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0164] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A face liveness detection method based on multimodal feature fusion, characterized in that, The specific steps include: S1: Simultaneously acquire RGB image, infrared image and depth image of the face to be detected, apply multiple different Gabor filter kernels to the RGB image for texture enhancement, and perform pixel-level registration on the filtered RGB image, infrared image and depth image to obtain spatially aligned input data for the three modalities. S2: Input data from the three modalities are fed into a shallow convolutional network. Channel expansion, local feature extraction, and channel attention weighting are performed on the input data of each modality in sequence to obtain the basic feature maps of the three modalities. S3: Perform dual-branch processing on the basic feature maps of each modality. The first branch applies lightweight convolution to the basic feature maps of each modality to obtain compressed and refined first-branch features. The second branch performs max pooling on the basic feature maps of each modality to obtain second-branch features that highlight the difference regions. The first-branch features and second-branch features of each modality are fused element by element and then convolved with equal-sized features to obtain the dimensionality-reduced features of each modality. S4: Perform global average pooling on the dimensionality reduction features of the three modalities respectively to obtain the global feature vectors of each modality. Based on the cosine similarity between the global feature vectors of the three modalities, construct a cross-modal affinity matrix, and calculate the transfer features of each modality based on the cross-modal affinity matrix. Then, fuse the transfer features of each modality with its dimensionality reduction features element by element to obtain the enhanced feature map of each modality. S5: Perform channel and spatial attention weighting on the augmented feature maps of the three modalities respectively, and concatenate them according to the channels to obtain the face fusion feature map. Perform dimensionality reduction, pooling and fully connected processing on the obtained face fusion feature map in sequence to obtain the fusion feature vector. Input the obtained fusion feature vector into the preset binary classifier and output the live or non-live face discrimination result.

2. The face liveness detection method based on multimodal feature fusion according to claim 1, characterized in that, The first branch in the dual-branch processing applies lightweight convolution to the basic feature maps of each modality in the following way: a lightweight convolution structure is constructed that sequentially includes a depthwise convolutional layer, a first stabilizing layer, a first activation function, a pointwise convolutional layer, a second stabilizing layer, and a second activation function. For any modality's base feature map, firstly, the depthwise convolutional layer independently performs spatial convolution on each input channel of the base feature map to extract the spatial features of all channels. Then, the first stabilization layer performs batch normalization on the spatial features of all channels and uses the first activation function to perform nonlinear transformation on all normalized spatial features to form a multi-channel feature map. The obtained multi-channel feature maps are input into a pointwise convolutional layer for cross-channel linear combination and channel dimension compression to form compressed feature maps. Then, the second stabilizing layer performs batch normalization on the compressed feature maps and uses the second activation function to perform nonlinear transformation on the normalized compressed feature maps, outputting the first branch features after compression and refinement.

3. The face liveness detection method based on multimodal feature fusion according to claim 1, characterized in that, The second branch in the dual-branch processing performs max pooling screening of the basic feature maps of each modality in the following way: a max pooling screening structure consisting of a multi-scale parallel pooling layer and an attention calibration layer is constructed. For any modality's base feature map, the multi-scale parallel pooling layer first performs multi-scale spatial downsampling on the base feature map in parallel. Multiple spatial pooling windows of different sizes are used to select the maximum response value in the local area they cover, generating multiple sets of pooling feature maps with different spatial resolutions. The attention calibration layer concatenates the multiple pooling feature maps according to the channel dimension, and performs convolutional compression and attention weighting on the concatenated features to map the multiple pooling features to a unified channel dimension and highlight key spatial regions, outputting a second branch feature that highlights the differences in the regions.

4. The face liveness detection method based on multimodal feature fusion according to claim 1, characterized in that... The specific method for constructing a cross-modal affinity matrix based on the cosine similarity between the global feature vectors of the three modalities is as follows: For the global feature vectors of the three modalities, calculate the cosine similarity between each pair; construct a third-order matrix with the three modalities as rows and columns respectively, so that the diagonal elements are 1 and the off-diagonal elements are the cosine similarity between the global feature vectors of the corresponding row mode and column mode, and use this third-order matrix as the cross-modal affinity matrix.

5. A face liveness detection method based on multimodal feature fusion according to claim 4, characterized in that, The specific method for calculating the modal transfer features based on the cross-modal affinity matrix is ​​as follows: set a similarity threshold, and select all off-diagonal elements in the cross-modal affinity matrix whose values ​​are greater than the similarity threshold as transferable elements; For any transferable element, the global feature vector of the column mode of the element is mapped to the feature space of the global feature vector of the row mode of the element through the learnable projection matrix corresponding to the current transfer direction, so as to obtain the transfer component of the row mode of the element; in the cross-modal affinity matrix, all transfer components of the same row mode are weighted and summed to obtain the transfer features of each mode.

6. The face liveness detection method based on multimodal feature fusion according to claim 1, characterized in that, The specific method for applying channel and spatial attention weighting to the augmented feature maps of the three modalities is as follows: a hybrid weighting structure containing channel attention modules and spatial attention modules is constructed, wherein the channel attention module consists of a global pooling unit, a fully connected network layer and a first weight function, and the spatial attention module consists of a channel pooling unit, a channel convolutional layer and a second weight function. For any modality's augmented feature map, it is input into the channel attention module. The global pooling unit performs global max pooling and global average pooling operations on the augmented feature map of the current modality, respectively. The two pooling results are then input into the fully connected network layer to obtain two pooling vectors corresponding to the two pooling results. The two pooling vectors are added and fused element by element, and the channel attention weights are generated on the fused vector through the first weight function. The obtained channel attention weights are multiplied element by element with the augmented feature map of the current modality to obtain the channel weighted feature map of the current modality. The channel-weighted feature map of the current modality is input into the spatial attention module. The channel pooling unit performs global max pooling and global average pooling operations on the channel-weighted feature map along the channel dimension. The two pooling results are concatenated according to the channel. The concatenated result is input into the channel convolutional layer for feature mapping. The spatial attention weights are generated on the convolutional feature vector through the second weight function. The obtained spatial attention weights are multiplied element-wise with the channel-weighted feature map of the current modality to obtain the attention-weighted feature map of the current modality.

7. The face liveness detection method for multimodal feature fusion according to claim 6, characterized in that, The specific method for obtaining the face fusion feature map is as follows: perform simple linear transformations on the attention-weighted feature maps of the three modalities respectively to align the channel dimensions of the three modalities, and then stitch the aligned attention-weighted feature maps of each modality according to the order of RGB modality, infrared modality and depth modality, according to the channels to obtain the face fusion feature map.

8. The face liveness detection method based on multimodal feature fusion according to claim 1, characterized in that, The specific method for inputting the fused feature vector into a preset binary classifier and outputting the result of live or non-live face discrimination is as follows: Construct a binary classifier containing a fully connected network layer and a Softmax activation function. Input the fused feature vector obtained from the face fusion features into the binary classifier. After passing through the fully connected network layer, the fused feature vector is mapped into a two-dimensional vector. Then, the Softmax activation function is used to convert the obtained two-dimensional vector into a probability distribution of live and non-live faces. Compare the probability values ​​of the two results and take the one with the larger probability value as the final discrimination result.

9. A face liveness detection system for multimodal feature fusion, characterized in that, The face liveness detection system oriented towards multimodal feature fusion is used to execute the face liveness detection method oriented towards multimodal feature fusion as described in any one of claims 1-8, including: The preprocessing module is used to simultaneously acquire RGB, infrared, and depth images of the face to be detected. Multiple different Gabor filter kernels are used to enhance the texture of the RGB image. The filtered RGB, infrared, and depth images are then registered at the pixel level to obtain spatially aligned input data for the three modalities. The feature extraction module is used to feed the input data of the three modalities into the shallow convolutional network respectively, and to perform channel expansion, local feature extraction and channel attention weighting on the input data of each modality in sequence to obtain the basic feature maps of the three modalities. The feature dimensionality reduction module is used to perform dual-branch processing on the basic feature maps of each modality. The first branch applies lightweight convolution to the basic feature maps of each modality to obtain compressed and refined first-branch features. The second branch performs max pooling to filter the basic feature maps of each modality to obtain second-branch features that highlight the difference regions. The first-branch features and second-branch features of each modality are fused element by element and then convolved with equal-sized features to obtain the dimensionality reduction features of each modality. The feature enhancement module is used to perform global average pooling on the dimensionality reduction features of the three modalities respectively to obtain the global feature vectors of each modality. Based on the cosine similarity between the global feature vectors of the three modalities, a cross-modal affinity matrix is ​​constructed, and the transfer features of each modality are calculated based on the cross-modal affinity matrix. The transfer features of each modality are fused with its dimensionality reduction features element by element to obtain the enhanced feature map of each modality. The liveness detection module is used to perform channel and spatial attention weighting on the augmented feature maps of the three modalities respectively, and to concatenate them according to the channels to obtain a face fusion feature map. The obtained face fusion feature map is then subjected to dimensionality reduction, pooling and full connection processing in sequence to obtain a fusion feature vector. The obtained fusion feature vector is input into a preset binary classifier to output the liveness or non-liveness face discrimination result.

Images