Face detection method and system for fusing visible light and infrared images

Abstract

The application discloses a kind of fusion visible light and infrared image live face detection method and system, it is first through parallel network in the extraction visible light and infrared feature, generate a face semantic mask chart that can accurately segment out eye, nose and other key areas simultaneously.Secondly, using the mask chart as anchor point, guide model to the regionalization feature pooling of visible light and infrared feature map, to obtain each semantic region independent, aligned feature vector.Finally, through cross-modal attention mechanism, these regional features are compared one by one, and through inconsistency amplification module to actively find and highlight the region that texture and temperature distribution do not match.This method forces the model to perform fine-grained logical verification, ensuring that the heat distribution accurately matches the facial geometry, thereby effectively identifying high-level fake attacks with temperature distribution that do not match real physiological features.

Description

Technical Field

[0001] This application relates to the field of live face detection, and more specifically, to a live face detection method and system that integrates visible light and infrared images. Background Technology

[0002] With the widespread application of facial recognition technology, its security issues have become increasingly prominent. To prevent unauthorized access, liveness detection technology has emerged, aiming to distinguish real users from spoofing attacks such as photos, videos, or 3D masks. Traditional liveness detection methods based on single visible light (RGB) images, while able to resist some two-dimensional attacks, fall short against high-precision three-dimensional attacks such as masks. Therefore, the industry has introduced infrared (IR) images, leveraging their ability to capture biological temperature information, adding a new dimension to liveness detection. Integrating the rich texture details of visible light images with the physiological temperature characteristics of infrared images has become an important technical approach to improve the robustness of liveness detection.

[0003] However, existing multimodal fusion schemes still face significant technical bottlenecks when dealing with advanced forgery attacks. Current mainstream fusion methods typically involve simply concatenating or adding feature maps from visible light and infrared images in the later stages of the feature extraction network, then passing this to subsequent network layers for implicit learning. While this global or coarse-grained fusion approach can detect the simultaneous presence of facial texture and heat signals in an image, it struggles to effectively counter advanced attacks such as heated masks that simulate real human temperatures. The deceptive nature of these attacks lies in their ability to not only generate temperature but also potentially heat key areas locally, making it difficult for simple fusion models to distinguish between genuine and fake. In other words, these shortcomings stem from the lack of explicit mechanisms in existing fusion methods for fine-grained spatial consistency verification of cross-modal features. During training, to minimize loss, the network may learn shortcuts, neglecting in-depth verification of the logical consistency of texture-temperature spatial distribution. For example, the model cannot be forced to precisely determine whether the temperature distribution shape in the nose region accurately matches the nose's geometric contour, nor can it meticulously compare the intrinsic relationship between the fine texture of the eye region and its heat distribution.

[0004] Therefore, how to guide the model to perform accurate alignment and comparison of cross-modal features within a specific semantic region, and achieve deep logical verification of the consistency between texture and temperature space, thereby effectively resisting advanced forgery attacks, is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0005] To address the problems of coarse granularity in existing technologies and the inability to verify spatial consistency of texture and temperature, this application provides a method for live face detection that fuses visible light and infrared images, comprising: acquiring a visible light image of a face and an infrared image of a face.

[0006] performing parallel multi-modal feature and semantic mask extraction on the face visible light image and the face infrared image to obtain a face RGB depth feature map, a face IR depth feature map, and a face semantic mask map.

[0007] performing semantic region guided feature pooling on the face RGB depth feature map and the face IR depth feature map based on the face semantic mask map to obtain a face RGB region feature vector set and a face IR region feature vector set.

[0008] performing cross-modal attention and inconsistency amplification on the face RGB region feature vector set and the face IR region feature vector set to obtain a live feature score.

[0009] performing final decision based on the face RGB depth feature map, the face IR depth feature map, and the live feature score to obtain a live face detection result.

[0010] According to another aspect of the present application, a live face detection system fusing visible light and infrared images is provided, which comprises: a face image acquisition module configured to acquire a face visible light image and a face infrared image.

[0011] a face image feature multi-modal analysis module configured to perform parallel multi-modal feature and semantic mask extraction on the face visible light image and the face infrared image to obtain a face RGB depth feature map, a face IR depth feature map, and a face semantic mask map.

[0012] a face image feature pooling module configured to perform semantic region guided feature pooling on the face RGB depth feature map and the face IR depth feature map based on the face semantic mask map to obtain a face RGB region feature vector set and a face IR region feature vector set.

[0013] a live feature score calculation module configured to perform cross-modal attention and inconsistency amplification on the face RGB region feature vector set and the face IR region feature vector set to obtain a live feature score.

[0014] a live face detection module configured to perform final decision based on the face RGB depth feature map, the face IR depth feature map, and the live feature score to obtain a live face detection result.

[0015] Compared with the prior art, the application provides a living face detection method and system fusing visible light and infrared images, which establishes a cross-modal inconsistency checking mechanism based on semantic region guidance to solve the problems of coarse fusion granularity and inability to check the spatial consistency of texture and temperature in the prior art. Instead of global and fuzzy feature fusion, the application first generates a face semantic mask map that can accurately segment key regions such as eyes and nose through a parallel network while extracting visible light and infrared features. Then, the mask map is used as an anchor point to guide the model to perform regional feature pooling on the visible light and infrared feature maps, thereby obtaining independent and aligned feature vectors for each semantic region. Finally, the cross-modal attention mechanism is used to compare these regional features one by one, and the inconsistency amplification module is used to actively find and highlight the regions where the texture and temperature distribution do not match. This method forces the model to perform fine-grained logical checking, ensuring that the heat distribution accurately matches the facial geometric structure, thereby effectively identifying high-level fake attacks with inconsistent temperature distribution and real physiological features. BRIEF DESCRIPTION OF DRAWINGS

[0016] The above and other objects, features and advantages of the present application will become more apparent from the following detailed description of embodiments of the present application when taken in conjunction with the accompanying drawings.

[0017] Figure 1 A flowchart of the living face detection method fusing visible light and infrared images according to the embodiment of the present application.

[0018] Figure 2 A data flow diagram of the living face detection method fusing visible light and infrared images according to the embodiment of the present application.

[0019] Figure 3 A flowchart of step S2 in the living face detection method fusing visible light and infrared images according to the embodiment of the present application.

[0020] Figure 4 A flowchart of step S3 in the living face detection method fusing visible light and infrared images according to the embodiment of the present application.

[0021] Figure 5 A flowchart of step S4 in the living face detection method fusing visible light and infrared images according to the embodiment of the present application.

[0022] Figure 6 A block diagram of the living face detection system fusing visible light and infrared images according to the embodiment of the present application. DETAILED DESCRIPTION

[0023] Embodiments of the present disclosure will be described below in greater detail with reference to the accompanying drawings. It should be understood that the accompanying drawings and embodiments of the present disclosure are merely for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0024] To solve the technical problems in the background art, the present application proposes a living face detection method fusing visible light and infrared images. Figure 1 A flowchart of the living face detection method fusing visible light and infrared images according to an embodiment of the present application. Figure 2 A data flow diagram of the living face detection method fusing visible light and infrared images according to an embodiment of the present application. As shown in Figure 1 and Figure 2 As shown in the living face detection method fusing visible light and infrared images according to an embodiment of the present application, it includes: step S1, acquiring a face visible light image and a face infrared image; step S2, performing parallel multi-modal feature and semantic mask extraction on the face visible light image and the face infrared image to obtain a face RGB depth feature map, a face IR depth feature map, and a face semantic mask map; step S3, based on the face semantic mask map, performing semantic region guided feature pooling on the face RGB depth feature map and the face IR depth feature map to obtain a face RGB region feature vector set and a face IR region feature vector set; step S4, performing cross-modal attention and inconsistency amplification on the face RGB region feature vector set and the face IR region feature vector set to obtain a living feature score; and step S5, based on the face RGB depth feature map, the face IR depth feature map, and the living feature score, performing final decision to obtain a living face detection result.

[0025] In step S1, the face visible light image and the face infrared image are acquired. It should be understood that in the field of living face detection, the technical path of relying solely on visible light images has exposed its vulnerability in resisting high-level attacks such as high-simulation masks. These attack means can easily reproduce the texture and color of the face, making traditional detection methods ineffective. In order to build a more secure defense mechanism, it is inevitable to introduce new information dimensions. Therefore, the first step of the present application is to simultaneously acquire the visible light image and the infrared image of the face, providing the most original and complete multi-modal data basis for subsequent fine-grained analysis, that is, simultaneously capturing visible light information that can reflect the fine texture and contour structure of the face, and infrared thermal imaging information that can reveal physiological activity and skin temperature distribution. These two heterogeneous but complementary data sources are the premise for realizing deep logical verification of texture-temperature spatial consistency, thereby providing a key basis for identifying counterfeit attacks that can simulate the temperature of a real person.

[0026] In one possible technical solution, step S1 performs the following operation: the face visible light image, also known as the RGB image, refers to an image captured by a standard image sensor in the visible light spectrum range (about 400-700 nanometers), which contains rich appearance details of the face such as color, texture, shape, etc., and is usually represented in a three-channel red, green, and blue color model. The face infrared image specifically refers to an image captured by a long-wave infrared (LWIR) thermal imaging sensor (the working waveband is usually 8-14 micrometers), which records the thermal radiation distribution of the face surface due to metabolism and can reflect the temperature difference of different regions, and the pixel value directly or indirectly corresponds to the temperature level.

[0027] To realize the acquisition of images, a binocular camera module integrating two sensors needs to be deployed. The module includes a high-resolution visible light camera and a non-cooled focal plane array infrared thermal imaging camera. In physical installation, the lens optical centers of the two cameras need to be as close as possible and keep the optical axes approximately parallel to reduce the initial parallax. Before acquisition, the binocular module needs to be strictly calibrated for internal and external parameters. The calibration process calculates the internal parameters (such as focal length and principal point coordinates) of each camera and the relative rotation and translation relationship between the two cameras by shooting a specially designed calibration board, and finally obtains an accurate spatial transformation matrix. The matrix is used to align the pixel coordinate system of the infrared image to the pixel coordinate system of the visible light image in subsequent processing, ensuring that the images of the two modalities are strictly corresponding in space, i.e., the same point on the face has convertible coordinates in the two images.

[0028] In a specific acquisition scenario, when the person to be detected enters the effective field of view range of the acquisition device, a trigger mechanism is activated. The mechanism can be started by a face detection algorithm in the visible light video stream or an independent distance sensor. Once triggered, a central synchronous controller sends a hardware trigger signal to the visible light camera and the infrared camera at the same time, ensuring that the capture of the two images is completely synchronized in time, with an error controlled within milliseconds.

[0029] Upon receiving the trigger signal, the visible light camera captures a 24-bit true-color image with a resolution of 1920×1080 pixels and encodes it into a visible light image of the face, denoted as image RGB_001. This image clearly shows the skin texture, facial features, and lighting and shadows. Simultaneously, the infrared camera captures a frame of thermal imaging data with a resolution of 320×240 pixels. This data is processed into a 16-bit single-channel grayscale image, denoted as image IR_001, where brighter pixels represent higher temperature areas (such as the eye sockets and nasal cavity area), and darker pixels represent lower temperature areas (such as the cheek edges). These two images, image RGB_001 and image IR_001, are stored together as a data pair, along with metadata information such as the acquisition timestamp.

[0030] In step S2, parallel multimodal feature and semantic mask extraction is performed on the visible light image and infrared image of the face to obtain the face RGB depth feature map, face IR depth feature map, and face semantic mask map. Correspondingly, given the lack of fine-grained spatial consistency verification in existing technologies, it is impossible to accurately compare the texture and temperature of specific facial regions. To establish the basis for such regional verification, this application uses this parallel processing step to extract feature maps that can characterize the deep information of their respective modalities from the visible light and infrared images; on the other hand, by fusing the features of these two modalities, a high-precision face semantic mask map is generated. This mask map will serve as a key spatial index for region guidance and feature alignment in subsequent steps, thereby laying the foundation for achieving accurate cross-modal logical verification.

[0031] In a feasible technical solution Figure 3 This is a flowchart of step S2 in the live face detection method that fuses visible light and infrared images according to an embodiment of this application. Figure 3 As shown, step S2 involves parallel multimodal feature and semantic mask extraction of the visible light image and infrared image of the face to obtain the face RGB depth feature map, face IR depth feature map, and face semantic mask map. This includes: step S21, performing shared shallow feature extraction on the visible light image and infrared image of the face to obtain the face RGB shallow feature map and face IR shallow feature map; step S22, performing independent deep feature extraction on the face RGB shallow feature map and face IR shallow feature map to obtain the face RGB depth feature map and face IR depth feature map; step S23, performing feature fusion on the face RGB depth feature map and face IR depth feature map to obtain the face multimodal deep fusion feature map; and step S24, inputting the face multimodal deep fusion feature map into a lightweight segmentation head network module to obtain the face semantic mask map.

[0032] In the above technical solution, step S2 performs the following operation: before being sent into the network, first, the acquired pair of images is preprocessed. The facial visible light image with a resolution of 1920x1080 pixels, image RGB_001, is adjusted to a uniform size of 224x224 pixels through a bilinear interpolation algorithm. At the same time, the facial infrared image with a resolution of 320x240 pixels, image IR_001, is also upsampled to 224x224 pixels in the same way to ensure that the inputs of the two modalities have the same spatial dimensions. Subsequently, the pixel values of the two images are normalized, for example, the pixel values are linearly scaled to the interval of floating-point numbers from 0 to 1, which aims to eliminate dimensional differences and adapt to the input requirements of the subsequent neural network model.

[0033] A shared shallow feature extraction network module is adopted in step S21, and the weight parameters of the module remain consistent when processing the two modalities of input. The module is composed of a convolutional layer with a kernel size of 7x7, a step of 2, and an output channel number of 64, followed by a batch normalization layer, a rectified linear unit (ReLU) activation function layer, and a maximum pooling layer with a kernel size of 3x3 and a step of 2. The weight parameters of the shared module are not fixed in advance, but are obtained through continuous iteration and updating by the back propagation algorithm and the optimizer (such as Adam) during the end-to-end training process of the entire model. In this way, the network is guided to autonomously learn and extract the basic structural information common to the two modalities of images, such as the outline of the face, the basic morphology of the facial features, and other low-level features. The preprocessed 224x224x3 facial visible light image is input into the module, and preliminary edge and corner information is extracted through convolution operation. The batch normalization layer is used to stabilize the training process and speed up convergence, the activation function introduces necessary nonlinear expression capability, and finally the maximum pooling layer reduces the spatial dimension of the feature map while retaining the most significant local features. The output is a facial RGB shallow feature map with a size of 56x56 and a channel number of 64. In parallel, the single-channel data of the preprocessed 224x224x1 facial infrared image is replicated three times in the channel dimension to construct a 224x224x3 pseudo-color image, which is then sent into the same shared shallow feature extraction network module. Due to weight sharing, the network is guided to learn the common low-level structural information of the two modalities, such as the outline and basic shape, and finally outputs a facial IR shallow feature map with the same size of 56x56x64.

[0034] In step S22, two deep feature extraction network modules with similar structure but independent weight parameters are set, named as RGB deep feature extraction module and IR deep feature extraction module respectively. Each module is stacked by multiple residual network units, aiming to learn the advanced semantic information specific to the two modalities respectively. A typical residual network unit adopts a bottleneck structure, which is composed of a 1x1 convolutional layer for channel dimension reduction, a 3x3 convolutional layer for spatial feature extraction, and a 1x1 convolutional layer for restoring channel dimension, and contains a shortcut connection that directly adds the input to the output. This effectively alleviates the gradient vanishing problem in deep network training. The face RGB shallow feature map (56x56x64) obtained in the previous step is input into the RGB deep feature extraction module. The module is stacked by several residual network units, and in this process, two down-sampling operations are performed by convolution operation with a step of 2, so that the spatial resolution of the feature map is gradually reduced to 14x14, and the channel dimension is correspondingly increased to 256. The network learns the complex patterns specific to visible light images, such as skin fine texture, light and shadow, color distribution, etc. Finally, the module outputs a face RGB deep feature map with a size of 14x14 and a channel number of 256. Similarly, the face IR shallow feature map (56x56x64) is sent to the IR deep feature extraction module. The module learns the heat distribution patterns, high and low temperature region shapes, and deep features related to human physiological activity in the infrared image through its independent weight parameters, and finally outputs a face IR deep feature map with a size of 14x14 and a channel number of 256. The weight parameters of the two modules are obtained by end-to-end training on a dataset containing a large amount of multi-modal face data and corresponding live labels, and are iteratively updated by the back propagation algorithm and the optimizer.

[0035] In step S23, the deep semantic information of the two modalities is integrated together to prepare for generating a unified semantic mask. In one feasible technical solution, step S23, the face RGB deep feature map and the face IR deep feature map are fused to obtain a face multi-modal deep fusion feature map, including: the face RGB deep feature map and the face IR deep feature map are fused according to the following formula: wherein, is the face RGB deep feature map, is the face IR deep feature map, is a feature concatenation function, The face multi-modal deep fusion feature map is obtained. Specifically, the 14x14x256 face RGB depth feature map and the 14x14x256 face IR depth feature map are taken as inputs, and the above formula is executed. The operation stacks the two tensors in the third dimension, that is, the channel axis, to generate a face multi-modal deep fusion feature map with a size of 14x14x512. The feature vector at each spatial position of the fusion feature map simultaneously encodes the texture semantics under visible light and the temperature semantics under infrared at the point, thereby providing more comprehensive and rich information for subsequent semantic segmentation.

[0036] In step S24, the low-resolution depth feature map is decoded into a pixel-level classification probability map. First, the semantic categories of the face need to be preset, for example, a set of 6 categories is determined: background, skin, left eye, right eye, nose, and mouth. The lightweight segmentation head network module is composed of a plurality of up-sampling layers and convolutional layers, forming a decoder structure. The specific architecture can be an up-sampling unit containing three stages, each unit being composed of a transposed convolutional layer with a kernel size of 2x2 and a step of 2 (used to double the spatial resolution of the feature map), followed by a batch normalization layer and a 3x3 ordinary convolutional layer (with a ReLU activation function, used to refine the features after up-sampling). The 14x14x512 face multi-modal deep fusion feature map is input into the module, and the spatial resolution of the feature map is restored from 14x14 to 112x112, while the number of channels is adjusted step by step through the step-by-step processing of the three up-sampling units. Finally, a bilinear interpolation layer is used to accurately enlarge the resolution to 224x224, and a convolutional layer with a kernel size of 1x1 and an output channel number of 6, i.e., the number of preset semantic categories, is used for the final feature mapping to obtain a 224x224x6 original prediction map. Subsequently, for each pixel position of the original prediction map, the Softmax function is applied to the vector composed of the 6 channels. The function converts the original numerical value into a probability distribution, ensuring that the sum of the probabilities of each pixel in all 6 categories is always 1. After this processing, the final output is a face semantic mask map with a size of 224x224x6. In this map, the value of each pixel point is a 6-dimensional vector representing the probability of the pixel belonging to the background, skin, left eye, etc. In particular, the weight parameters of the segmentation head network module are also obtained by joint training on a multi-modal face dataset with pixel-level semantic annotation, and the training target is to minimize the pixel-level cross-entropy loss between the predicted mask and the real annotation mask.

[0037] In step S3, based on the face semantic mask map, the face RGB depth feature map and the face IR depth feature map are subjected to semantic region guided feature pooling to obtain a face RGB region feature vector set and a face IR region feature vector set. It can be understood that the pre-sequence step has successfully extracted the depth feature map containing rich information and the semantic mask map as a spatial index, however, these features still exist in the form of two-dimensional spatial distribution, lacking the structure for direct cross-modal regional comparison. If the entire feature map is directly fused, it will return to the global and coarse-grained processing mode in the prior art, and cannot solve the problem of checking the texture and temperature consistency of specific semantic regions (such as nose and eyes). Therefore, in the present application, the semantic region guided feature pooling uses the semantic mask map as an accurate spatial template to discretize and structure the continuous depth feature map into a series of feature vectors corresponding to specific facial organs (such as left eye and nose), providing aligned, regionalized and directly comparable inputs for subsequent cross-modal attention analysis and inconsistency checking, thereby building a bridge for fine-grained logical checking.

[0038] In one possible technical solution, Figure 4 The flowchart of step S3 in the method for fusing visible light and infrared images for live face detection according to the embodiment of the present application is shown in FIG. 3. As shown in FIG. 3, Figure 4 Step S3, based on the face semantic mask map, the face RGB depth feature map and the face IR depth feature map are subjected to semantic region guided feature pooling to obtain a face RGB region feature vector set and a face IR region feature vector set, including: step S31, performing size alignment and decomposition of the semantic mask of the face semantic mask map to obtain a single-channel probability mask set; step S32, extracting a first single-channel probability mask from the single-channel probability mask set; step S33, based on the first single-channel probability mask, performing parallel guided weighted pooling and feature vector generation on the face RGB depth feature map and the face IR depth feature map to obtain a first face RGB region feature vector and a first face IR region feature vector.

[0039] In the above technical solution, step S3 performs the following operations: in step S31, the face semantic mask map is size-aligned and decomposed. Since the face semantic mask map (224x224) and the two depth feature maps (14x14) are not matched in spatial resolution, they cannot be directly used to guide feature extraction, so the first task is to align the spatial size. Here, a spatial downsampling module is used, which uses a bilinear interpolation algorithm to downsample the input 224x224x6 face semantic mask map to 14x14x6. Bilinear interpolation calculates the probability value of a pixel point on the target small-size mask map by weighted averaging the probability values of the adjacent four pixel points on the original large-size mask map. This method can smoothly transition the probability distribution and effectively preserve the semantic information of the original mask. After downsampling, a size-aligned semantic mask map with a size of 14x14x6 is obtained, which is spatially aligned with the depth feature map. Next, the size-aligned semantic mask map is decomposed. This operation is performed along the channel dimension, i.e., the 6 semantic class channels, to slice and decompose it into 6 independent single-channel probability mask maps. Taking the aforementioned 6 semantic classes (background, skin, left eye, right eye, nose, and mouth) as an example, this decomposition operation will generate 6 14x14 matrices. For example, one of the matrices is the nose probability mask, and each value in the 14x14 elements represents the probability that the corresponding spatial position of the depth feature map belongs to the nose semantic class. Similarly, left eye probability mask, skin probability mask, etc. are also generated. These 6 single-channel probability mask maps together form a set, i.e., a single-channel probability mask set, which prepares for subsequent independent feature extraction for each semantic region.

[0040] In step S32, a first single-channel probability mask is extracted from the single-channel probability mask set, marking the beginning of the iterative processing of each semantic region. In a specific execution sequence, first, the probability mask representing the nose region, i.e., the nose probability mask, is extracted from the single-channel probability mask set as the first single-channel probability mask. This mask is a 14x14 matrix with element values between 0 and 1.

[0041] In step S33, guided weighted pooling and feature vector generation are performed on the face RGB depth feature map and the face IR depth feature map in parallel based on the first single-channel probability mask, i.e., the nose probability mask. This process aims to calculate a single feature vector that can represent the nose region, and is performed in parallel for both RGB and IR modalities to ensure comparability. In one possible implementation, step S33, guided weighted pooling and feature vector generation are performed on the face RGB depth feature map and the face IR depth feature map in parallel based on the first single-channel probability mask to obtain a first face RGB region feature vector and a first face IR region feature vector, includes: step S331, calculating the sum of all elements in the first single-channel probability mask to obtain a first normalization factor; step S332, element-wise multiplying the face RGB depth feature map with the broadcasted first single-channel probability mask tensor to obtain a first weighted face RGB depth feature map; step S333, summing the first weighted face RGB depth feature map in its spatial dimensions to obtain a first face RGB region initial feature vector; and step S334, dividing the first face RGB region initial feature vector by the first normalization factor to obtain the first face RGB region feature vector.

[0042] Specifically, the first step S331. Sum all the 196 probability values in the first single-channel probability mask, i.e. the 14x14 matrix of nose probability mask, to get a scalar value, denoted as the nose region normalization factor, i.e. the first normalization factor. This factor is essentially the total weight of the nose region in the whole feature map. The second step S332, element-wise multiply the face RGB depth feature map with the broadcasted first single-channel probability mask tensor. Since the face RGB depth feature map has a size of 14x14x256, and the first normalization factor is 14x14, the mask needs to be broadcasted first, i.e. duplicated 256 times in the channel dimension, to form a 14x14x256 weight tensor. Then, element-wise multiply this weight tensor with the face RGB depth feature map. In this way, the feature vectors corresponding to the nose region in the feature map are weighted and preserved by their high probability values, while the feature vectors corresponding to the non-nose region are effectively suppressed by multiplying with probability values close to zero. The output of this operation is a 14x14x256 first weighted face RGB depth feature map. The third step S333, sum the first weighted face RGB depth feature map in its spatial dimensions. This operation sums along the height and width dimensions, i.e. the 14x14 spatial plane. For each of the 256 channels, sum its 14x14 weighted feature values to get a sum value. Finally, these 256 sum values form a 256-dimensional vector, denoted as the first face RGB region initial feature vector. The fourth step S334, divide the first face RGB region initial feature vector by the first normalization factor. Divide each element in the 256-dimensional initial feature vector obtained in the previous step by the nose region normalization factor calculated in step S331. This division operation completes the weighted average calculation, and the final 256-dimensional vector is the first face RGB region feature vector that accurately represents the average feature of the nose region.

[0043] In parallel, perform the exact same guided weighted pooling operation on the face IR depth feature map. The key is that this process reuses the same nose region normalization factor calculated in step S331, and uses the same nose probability mask for guidance. Broadcast the nose probability mask and element-wise multiply it with the 14x14x256 face IR depth feature map to get the first weighted face IR depth feature map. Similarly, sum its spatial dimensions to get the first face IR region initial feature vector. Finally, divide this initial vector by the nose region normalization factor to get the final 256-dimensional first face IR region feature vector.

[0044] After the processing of the nose region is completed, the generated first face RGB region feature vector is added to an initially empty face RGB region feature vector set, and the first face IR region feature vector is added to a face IR region feature vector set. Subsequently, the method returns to step S32 to extract the probability mask of the next semantic region, for example, the left eye probability mask, from the single-channel probability mask set, and repeats all the operations of step S33 to generate the RGB and IR feature vectors representing the left eye region and adds them to the respective sets. This iterative process continues until the probability masks of all 6 semantic regions (background, skin, left eye, right eye, nose, and mouth) are processed.

[0045] When all iterations are completed, the final output is two sets: a face RGB region feature vector set containing 6 256-dimensional feature vectors corresponding to the visible light features of the 6 semantic regions, and a face IR region feature vector set also containing 6 256-dimensional feature vectors corresponding to the infrared features of the 6 semantic regions.

[0046] In step S4, cross-modal attention and inconsistency amplification are performed on the face RGB region feature vector set and the face IR region feature vector set to obtain a live feature score. It should be understood that the previous steps have successfully structured the continuous feature map into a region feature vector set corresponding to the facial organs, providing basic data for fine-grained cross-modal comparison. However, simply calculating and summing the consistency of each region independently cannot effectively distinguish the importance differences of different semantic regions in live judgment, nor can it capture the subtle differences of various inconsistency patterns in fake attacks. For example, a small texture-temperature mismatch in the eye region may be more indicative than a larger deviation in the cheek skin. Therefore, the present application establishes a decision mechanism that can comprehensively evaluate the consistency and inconsistency of all regions and dynamically assign importance weights according to their internal relationships through cross-modal attention and inconsistency amplification. By actively amplifying key inconsistency signals and performing weighted aggregation, a single and robust live feature score that is highly sensitive to advanced fake attack features can be generated.

[0047] In one possible technical solution, Figure 5 The flowchart of step S4 in the method of fusing visible light and infrared images for live face detection according to an embodiment of the present application is shown in FIG. 4. As shown in FIG. 4, the method first performs cross-modal attention and inconsistency amplification on the face RGB region feature vector set and the face IR region feature vector set to obtain a live feature score. Figure 5As shown, in step S4, the cross-modal attention and inconsistency amplification are performed on the face RGB region feature vector set and the face IR region feature vector set to obtain the living feature score, including: in step S41, calculating the region consistency score between each corresponding face RGB region feature vector and face IR region feature vector in the face RGB region feature vector set and the face IR region feature vector set to obtain a region consistency score set; in step S42, calculating the region inconsistency feature vector between each corresponding face RGB region feature vector and face IR region feature vector in the face RGB region feature vector set and the face IR region feature vector set to obtain a region inconsistency feature vector set; in step S43, based on the region inconsistency feature vector set, calculating the importance attention weight set of each region; in step S44, based on the importance attention weight set of each region and the region inconsistency feature vector set, performing inconsistency punishment and weighted aggregation on the region consistency score set to obtain the living feature score.

[0048] In the above technical solution, step S4 performs the following operation: in step S41, a basic quantitative index is provided for the cross-modal feature similarity of each semantic region. In a feasible technical solution, in step S41, the region consistency score between each corresponding face RGB region feature vector and face IR region feature vector in the face RGB region feature vector set and the face IR region feature vector set is calculated to obtain a region consistency score set, including: the region consistency score between each corresponding face RGB region feature vector and face IR region feature vector is calculated according to the following formula: wherein, and are the face RGB region feature vector and the face IR region feature vector, is the region consistency score. Specifically, for each of the 6 semantic regions, its corresponding face RGB region feature vector and face IR region feature vector are extracted, and the region consistency score is calculated according to the above formula. The formula calculates the cosine similarity between the two feature vectors. Taking the nose region as an example, the dot product operation is performed on the 256-dimensional face RGB feature vector and the 256-dimensional face IR feature vector, and then the L2 norm (i.e. the length of the vector) of the two vectors is calculated, respectively. Finally, the dot product result is divided by the product of the two moduli. The result is a scalar value between -1 and 1. The closer the value is to 1, the more consistent the visible light feature and the infrared feature of the nose region in the abstract semantic space, that is, the higher the consistency. Repeat this operation for all 6 semantic regions, and finally obtain a region consistency score set containing 6 scalar values.

[0049] In step S42, unlike the scalar score computed in step S41, this step aims to generate a vector that can characterize the specific difference between the two modalities in the feature level in detail. For each semantic region, for example, the nose region, take its corresponding 256-dimensional face RGB region feature vector and face IR region feature vector, perform element-wise subtraction operation, and then take the absolute value of each element in the resulting vector. The calculation formula is: The output is a 256-dimensional region inconsistency feature vector, and the value in each dimension of the vector quantifies the difference between the RGB and IR features in that particular dimension. The larger the value in a dimension, the more significant the divergence between the two modalities in that feature. Repeat this operation for all 6 semantic regions, and finally obtain a set of region inconsistency feature vectors containing 6 256-dimensional vectors.

[0050] In step S43, a self-attention module is introduced to analyze the internal correlations among all regional inconsistencies and determine the relative importance of each region for the final live body decision. The specific architecture of this self-attention module is a standard transformer encoder layer. The 6 256-dimensional vectors in the set of regional inconsistency feature vectors obtained in the previous step are input into this encoder layer as a sequence. Inside the module, each input inconsistency feature vector is first mapped to a query vector, a key vector and a value vector through three different, learnable linear transformation layers (weight matrices). Then, a scaled dot-product attention computation is performed: for each query vector, the dot product of it with all 6 key vectors is computed, the result is divided by a scaling factor, which is the square root of the key vector dimension to stabilize the gradients, then the scores are converted to a set of weights through a Softmax function, and finally this set of weights is used to weight the sum of all 6 value vectors. This process enables the model to learn which other regions, such as eyes and skin, should be given more attention when a certain region, such as the nose, presents a certain inconsistency pattern. To enhance the expressive power of the model, a multi-head self-attention mechanism is adopted, i.e. multiple independent attention computations are performed in parallel and the results are concatenated and then linearly transformed again. The output of the self-attention computation is passed through a residual connection and layer normalization, and then fed into a position- feedforward network consisting of two linear layers and an activation function such as a ReLU, and then again through a residual connection and layer normalization. It is worth mentioning that all the weight parameters of this transformer encoder layer, including the weights and biases of the linear transformation layers, are learned through the backpropagation algorithm during the end-to-end training of the entire live body detection model. To obtain the final scalar attention weights, the output sequence of the transformer encoder layer, i.e. the set of 6 updated vectors, is fed into a final linear layer that maps each 256-dimensional vector to a scalar, and then the Softmax function is applied to the 6 scalars to ensure that the 6 weight values are all positive and sum to 1. These 6 weight values constitute the importance attention weight set of the regions.

[0051] In step S44, an aggregation formula is used. In one possible technical solution, step S44, based on the importance attention weight set of the regions and the set of regional inconsistency feature vectors, performs inconsistency penalty and weighted aggregation on the set of regional consistency scores to obtain the live body feature score, including: based on the importance attention weight set of the regions and the set of regional inconsistency feature vectors, performing inconsistency penalty and weighted aggregation on the set of regional consistency scores according to the following formula: wherein, is an inconsistency amplifier module, is a learnable scalar parameter, for each regional consistency score in the set of regional consistency scores, for the importance attention weight of each region, for each regional inconsistency feature vector in the set of regional inconsistency feature vectors, is the live feature score. First, we explain the inconsistency amplifier module in the formula. This module is a small multi-layer perceptron, e.g., consisting of two fully connected layers and a nonlinear activation function, which takes a 256-dimensional regional inconsistency feature vector as input and outputs a scalar penalty value. The weight parameters of this module are also learned during model training, and its role is to identify and amplify those inconsistency patterns that are highly relevant to the fake attack. The parameter is a learnable global scalar parameter that adjusts the overall strength of the inconsistency penalty. For each semantic region c, first, its inconsistency feature vector is input into the inconsistency amplifier module to obtain a penalty value, and then this value is multiplied by . Next, this scaled penalty term is subtracted from the original consistency score of this region to obtain the post-penalty consistency score of this region. Finally, this post-penalty score is multiplied by the importance attention weight of this region. Perform the above calculation for all 6 semantic regions, and sum the 6 weighted post-penalty scores to obtain the final single scalar value, which is the live feature score. This score integrates the cross-modal consistency of all regions, the key inconsistency patterns, and their importance in the global context, and can accurately evaluate the live reliability of the input face.

[0052] In particular, when performing inconsistency penalty on the regional consistency score, there is a core technical challenge, i.e., the geometric alignment of the physical meaning of the penalty term and the physical meaning of the consistency measure (cosine similarity). The consistency score By cosine similarity calculation, its metric focuses entirely on the direction (angle) between the face RGB region feature vector and the face IR region feature vector, and is insensitive to the length of the vector itself, that is, even if the lengths of the two vectors differ greatly, as long as the directions are the same, the cosine similarity is 1. However, the above region inconsistency feature vector in inconsistency penalty, for example, based on the absolute value of the element-by-element difference, its L1 distance represents the influence of both vector direction and length difference, and then learns the penalty value from this mixed difference signal via a general inconsistency amplifier module and a single learnable scalar parameter. This mismatch may lead to low learning efficiency or insufficient robustness of the model, for example, an attack may be well simulated in the feature direction (high consistency score), but there is a large difference in the feature length, and the model needs to spend a lot of learning ability to decouple and understand the importance of this length difference. Therefore, in order to solve this geometric mismatch and greatly improve the model's discrimination accuracy and learning efficiency for different attack modes, the present application decouples the ambiguous and mixed inconsistency penalty into two geometrically orthogonal components - direction inconsistency penalty and length inconsistency penalty through decoupled cross-modal region inconsistency orthogonal penalty calculation. By making the penalty mechanism directly correspond to the geometric meaning of the consistency measure, the model can more clearly and efficiently learn different types of inconsistency, thereby specifically punishing advanced fake attacks that only have flaws in distribution patterns (direction) or only in energy intensity (length).

[0053] In a feasible preferred technical solution, step S44, based on the importance attention weight set of each region and the region inconsistency feature vector set, the region consistency score set is punished and aggregated by inconsistency to obtain the live feature score, including: step S441, L2 normalization is performed on each corresponding face RGB region feature vector and face IR region feature vector in the face RGB region feature vector set and the face IR region feature vector set to obtain the face RGB region direction feature vector set and the face IR region direction feature vector set, that is: wherein, and are the face RGB region feature vector and the face IR region feature vector, is the L2 norm of the calculated vector, and are the face RGB region directional feature vector and the face IR region directional feature vector, respectively. First, the face RGB region feature vector and the face IR region feature vector are L2 normalized to decouple and extract their pure directional information, so as to strip the length information of the feature vector and only retain its direction in the multi-dimensional space. Specifically, for each semantic region, for example, the nose region, take its face RGB region feature vector and face IR region feature vector, and process them according to the above formula. In this way, regardless of the strength of the original feature vector, the directional feature vector obtained after processing is a unit vector with a length of 1, which only retains the directional information of the original feature in the multi-dimensional space. This operation is performed on the feature vectors of all semantic regions, and finally the face RGB region directional feature vector set and the face IR region directional feature vector set are obtained.

[0054] Step S442, the directional inconsistency feature vector set is obtained by calculating the directional inconsistency vector difference of each group of face RGB region directional feature vectors and face IR region directional feature vectors in the face RGB region directional feature vector set and the face IR region directional feature vector set, that is: ; wherein, is the absolute value calculation, is the directional inconsistency feature vector. Then, in order to quantify the difference between the two modalities in the feature mode or distribution form. By calculating the absolute value of the element-by-element difference of the corresponding directional feature vector of each group based on the two sets of directional feature vectors obtained in the previous step, the directional inconsistency feature vector is obtained. The calculation result is a pure directional difference measure, and each element in the vector represents the directional deviation size of the two modalities in the corresponding feature dimension. This high-dimensional difference vector provides more detailed information than a single scalar, which can capture the subtle deviations that occur when a fake attack imitates a real physiological pattern. By calculating the directional inconsistency of all regions to obtain the directional inconsistency feature vector set.

[0055] Step S443, the length inconsistency feature difference of each corresponding face RGB region feature vector and face IR region feature vector is calculated to obtain the length inconsistency feature value set, that is: ; wherein, is the modulus inconsistency feature value. Then, the difference in feature strength or energy between the two modalities is quantified. Based on the original, non-normalized face RGB region feature vector and face IR region feature vector, for each pair of corresponding feature vectors, the absolute value of the difference of their L2-norms is computed. This computation completely ignores the direction of the vectors and only focuses on the difference in overall strength between the two, and the result is a non-negative scalar value, which is a pure modulus difference measure. This scalar value directly reflects the mismatch in energy level between the heat of the heat-emitting mask and the physiological heat radiation of the real face, for example. Finally, the above computation is performed for all regions, and the output is a set containing all the modulus inconsistency scalar values, i.e., the modulus inconsistency feature value set.

[0056] Step S444, the direction inconsistency feature vector set and the modulus inconsistency feature value set are input into the direction penalty value calculation model and the modulus penalty value calculation model respectively to obtain the direction penalty value set and the modulus penalty value set. Then, the original difference measure calculated in the previous step is mapped to more discriminative penalty values by a learnable model, and their contributions are balanced by learnable weights. Specifically, each vector in the direction inconsistency feature vector set is input into an independent direction penalty value calculation model, which is a small multi-layer perceptron whose weight parameters are learned in training, for mapping the direction difference vector to a scalar penalty value, thereby obtaining the direction penalty value set. Similarly, each scalar value in the modulus inconsistency feature value set is input into another independent modulus penalty value calculation model to obtain the modulus penalty value set.

[0057] Step S445, based on the direction penalty value set and the modulus penalty value set, learnable scalar penalty contribution calculation is performed on each region consistency score in the region consistency score set to obtain the penalty contribution value set, i.e., ; wherein, and are independent learnable scalar parameters, is the direction penalty value, is the modulus penalty value, is each region consistency score in the region consistency score set, is the penalty contribution value. Then, the two and As a global knob, it is automatically learned and adjusted by backpropagation algorithm and optimizer during the end-to-end training of the whole neural network model, to determine whether more emphasis is placed on the inconsistency in direction or in magnitude. For each semantic region, its final score is calculated by the above formula, which subtracts the independently weighted direction inconsistency penalty and magnitude inconsistency penalty from the original region consistency score. In this way, if an attack can perfectly simulate the pattern of heat distribution (direction consistency), but cannot simulate its true energy intensity (magnitude inconsistency), the magnitude penalty contribution factor will learn to give it a higher weight to punish such attacks. Conversely, the same is true.

[0058] In step S446, the penalty contribution value set is weighted and aggregated to obtain the live feature score, i.e. ; wherein, is each penalty contribution value in the penalty contribution value set, is the importance attention weight of each region, is the live feature score. Finally, the penalty contribution value set is weighted and aggregated, i.e. the penalty contribution value of each region is multiplied by its corresponding importance attention weight, which is calculated by the preceding self-attention module, and then the results of all regions are summed to calculate the final live feature score.

[0059] That is, this scheme splits a unified inconsistency amplifier into and , and splits the unified penalty scalar into and . Among them, and are also small multi-layer perceptrons, which are used as direction penalty value calculation models and magnitude penalty value calculation models, respectively. Each vector in the direction inconsistency feature vector set is input into model to nonlinearly map the high-dimensional direction difference vector to a scalar penalty value, thereby obtaining a set of direction penalty values. Similarly, each scalar value in the magnitude inconsistency feature value set is input into model to obtain a set of magnitude penalty values. In this way, by independently learning and weighing two different types of attacks, the model can achieve higher discrimination accuracy. For example, an attack may perfectly simulate the pattern of heat distribution (direction consistency), but cannot simulate its true energy intensity (magnitude inconsistency), then will learn to give Higher weights to penalize such attacks. Conversely, another attack might be globally correct (consistent in length), but the heat distribution is a chaotic pattern of spots (inconsistent in direction), then The model will penalize this case. And, since the task is decomposed more clearly, each subnetwork And The mapping relationship that needs to be learned is also simpler, which brings faster convergence speed and higher model accuracy. That is, the network no longer has to waste capacity to learn how to decouple the mixed difference signals, but can directly focus on giving the decoupled signals the appropriate penalty. The final live feature score integrates the fine-grained, decoupled inconsistency information of all regions and their importance in the global context, enabling highly accurate and robust judgments of the live state.

[0060] In step S5, a final decision is made based on the face RGB depth feature map, the face IR depth feature map, and the live feature score to obtain a live face detection result. That is, through fine-grained regional comparison and inconsistency amplification, a live feature score highly sensitive to local cross-modal mismatch has been generated. However, relying solely on this local verification score may overlook some macro anomalies presented by certain fake attacks in the overall features, such as global texture distortion or overall heat distribution that does not conform to natural laws. Therefore, the live feature score obtained based on fine-grained spatial consistency verification is fused with the depth features that can represent global information to form a final criterion that takes into account both local details and overall view, ensuring the comprehensiveness and robustness of the decision and avoiding false positives due to misleading local information.

[0061] In one possible technical solution, step S5 performs the following operations: In order to incorporate global information in the final decision, it is necessary to first compress the information of the two depth feature maps and extract their global representations. This process is accomplished by performing global average pooling operations on the face RGB depth feature map and the face IR depth feature map, respectively. For a face RGB depth feature map of 14x14x256, global average pooling calculates the average value of each of its 256 channels. Specifically, for the first channel, the feature values at its 14x14=196 spatial locations are all added together, and then divided by 196 to obtain a scalar value representing the global average response of the channel. This operation is repeated for all 256 channels, and finally the 14x14x256 feature map is compressed into a 256-dimensional vector, which is the face RGB depth feature vector. This vector summarizes the macro texture and structural features of the entire visible light image. In the same way, global average pooling is performed on the face IR depth feature map of 14x14x256 to obtain a 256-dimensional face IR depth feature vector, which summarizes the overall heat distribution pattern of the entire infrared image.

[0062] Next, the vital feature score, global RGB feature and global IR feature obtained in the previous step are integrated to construct an informative feature vector for final classification. The integration is realized by a feature concatenation operation. The scalar vital feature score, 256-dimensional face RGB deep feature vector and 256-dimensional face IR deep feature vector are concatenated to form a final decision feature vector with dimension 1+256+256=513. The first element of the vector is the score based on the local fine-grained check, and the following 512 elements are the compact encoding of the global information of the two modalities, thus realizing the effective combination of local criterion and global view.

[0063] Finally, the 513-dimensional final decision feature vector is input into a fully connected classifier to output the final detection result. The classifier is a simple multi-layer perceptron, whose specific architecture includes an input layer, a hidden layer and an output layer. The number of neurons in the input layer is 513, which completely matches the dimension of the final decision feature vector. The hidden layer can be set to contain 128 neurons, and the rectified linear unit (ReLU) is used as the activation function to introduce non-linear transformation ability and enhance the fitting ability of the model. The output layer contains 2 neurons, corresponding to the two classes of live and attack. All weights and bias parameters of the multi-layer perceptron are learned through the back propagation algorithm and the optimizer (such as Adam) in the end-to-end training process of the entire live detection model, on a dataset containing a large number of multi-modal face samples labeled as live or attack, and the training target is to minimize the cross-entropy loss between the classification prediction and the true label.

[0064] When making a prediction, after the 513-dimensional final decision feature vector is forward propagated through the multi-layer perceptron, the two neurons in the output layer will each produce an original score value. Then, the Softmax function is applied to the two score values to convert them into a probability distribution. For example, if the original score of the output layer is [2.5, -1.0], after the Softmax function processing, it may get [0.97, 0.03]. The two values represent the probability of the input sample being judged as live and the probability of being judged as attack, respectively, and their sum is 1. The final decision is to choose the class with higher probability value. In the above example, since 0.97 corresponding to live is greater than 0.03 corresponding to attack, the final live face detection result is determined to be live. Conversely, if the probability of the attack class is higher, the result is attack. Thus, the entire live face detection process is completed.

[0065] In summary, the living face detection method fusing visible light and infrared images based on the embodiments of the present application is illustrated, which aims at the problem of coarse fusion granularity and the inability to perform spatial consistency verification of texture and temperature in the prior art. The present application establishes a cross-modal inconsistency verification mechanism based on semantic region guidance. Instead of global and fuzzy feature fusion, the present application first generates a face semantic mask map that can accurately segment out key regions such as eyes and nose through a parallel network while extracting visible light and infrared features. Subsequently, the mask map is used as an anchor point to guide the model to perform regional feature pooling on the visible light and infrared feature maps, thereby obtaining independent and aligned feature vectors for each semantic region. Finally, the regional features are compared one by one through a cross-modal attention mechanism, and the inconsistency amplification module is used to actively find and highlight the regions where the texture and temperature distribution do not match. This method forces the model to perform fine-grained logical verification, ensuring that the heat distribution accurately matches the facial geometry, thereby effectively identifying high-level fake attacks with inconsistent temperature distribution and real physiological features.

[0066] Figure 6 A block diagram of a living face detection system fusing visible light and infrared images according to an embodiment of the present application. As shown in Figure 6 The living face detection system fusing visible light and infrared images 100 according to an embodiment of the present application includes: a face image acquisition module 110 for acquiring a face visible light image and a face infrared image; a face image feature multi-modal analysis module 120 for performing parallel multi-modal feature and semantic mask extraction on the face visible light image and the face infrared image to obtain a face RGB depth feature map, a face IR depth feature map, and a face semantic mask map; a face image feature pooling module 130 for performing semantic region guided feature pooling based on the face semantic mask map to obtain a face RGB regional feature vector set and a face IR regional feature vector set; a living feature score calculation module 140 for performing cross-modal attention and inconsistency amplification on the face RGB regional feature vector set and the face IR regional feature vector set to obtain a living feature score; and a living face detection module 150 for performing final decision based on the face RGB depth feature map, the face IR depth feature map, and the living feature score to obtain a living face detection result.

[0067] Here, those skilled in the art can understand that the specific operations of each step in the living face detection system fusing visible light and infrared images described above have been described in detail above with reference to the living face detection method fusing visible light and infrared images described in Figures 1 to 5 Therefore, the repeated description will be omitted.

Claims

1. A method for fusing visible and infrared images for live face detection, characterized in that, The method comprises the following steps: obtaining a face visible light image and a face infrared image; performing parallel multi-modal feature and semantic mask extraction on the face visible light image and the face infrared image to obtain a face RGB depth feature map, a face IR depth feature map and a face semantic mask image, including: performing shared shallow feature extraction on the face visible light image and the face infrared image to obtain a face RGB shallow feature map and a face IR shallow feature map; performing independent deep feature extraction on the face RGB shallow feature map and the face IR shallow feature map to obtain the face RGB depth feature map and the face IR depth feature map; performing feature fusion on the face RGB depth feature map and the face IR depth feature map to obtain a face multi-modal deep fusion feature map; inputting the face multi-modal deep fusion feature map into a lightweight segmentation head network module to obtain the face semantic mask image; based on the face semantic mask image, performing semantic region guided feature pooling on the face RGB depth feature map and the face IR depth feature map to obtain a face RGB region feature vector set and a face IR region feature vector set; performing cross-modal attention and inconsistency amplification on the face RGB region feature vector set and the face IR region feature vector set to obtain a live feature score, including: calculating a region consistency score between each corresponding face RGB region feature vector and face IR region feature vector in the face RGB region feature vector set and the face IR region feature vector set to obtain a region consistency score set; calculating a region inconsistency feature vector between each corresponding face RGB region feature vector and face IR region feature vector in the face RGB region feature vector set and the face IR region feature vector set to obtain a region inconsistency feature vector set; based on the region inconsistency feature vector set, calculating a set of importance attention weights of each region; based on the set of importance attention weights of each region and the region inconsistency feature vector set, performing inconsistency penalty and weighted aggregation on the region consistency score set to obtain the live feature score; based on the face RGB depth feature map, the face IR depth feature map and the live feature score, performing final decision to obtain a live face detection result.

2. The method according to claim 1, wherein, The face RGB depth feature map and the face IR depth feature map are fused to obtain a face multi-modal deep fusion feature map, including: fusing the face RGB depth feature map and the face IR depth feature map according to the following formula: ; wherein, is the face RGB depth feature map, is the face IR depth feature map, is a feature splicing function, is the face multi-modal deep fusion feature map.

3. The method of claim 1, wherein the method further comprises: based on the face semantic mask image, performing semantic region guided feature pooling on the face RGB depth feature map and the face IR depth feature map to obtain a face RGB region feature vector set and a face IR region feature vector set, including: performing size alignment and decomposition of the semantic mask of the face semantic mask image to obtain a single-channel probability mask set; extracting a first single-channel probability mask from the single-channel probability mask set; based on the first single-channel probability mask, performing parallel guided weighted pooling and feature vector generation on the face RGB depth feature map and the face IR depth feature map to obtain a first face RGB region feature vector and a first face IR region feature vector.

4. The method of claim 3, wherein the method further comprises: Based on the first single-channel probability mask, the face RGB depth feature map and the face IR depth feature map are subjected to parallel guided weighted pooling and feature vector generation to obtain a first face RGB region feature vector and a first face IR region feature vector, including: calculating the sum of all elements in the first single-channel probability mask to obtain a first normalization factor; element-wise multiplication of the face RGB depth feature map and the broadcasted first single-channel probability mask tensor to obtain a first weighted face RGB depth feature map; summing the first weighted face RGB depth feature map in its spatial dimension to obtain a first face RGB region initial feature vector; dividing the first face RGB region initial feature vector by the first normalization factor to obtain the first face RGB region feature vector.

5. The method of claim 1, wherein the method further comprises: calculating the region consistency score between each corresponding face RGB region feature vector and face IR region feature vector in the face RGB region feature vector set and the face IR region feature vector set to obtain a region consistency score set, including: The region consistency score between the corresponding face RGB region feature vector and face IR region feature vector of each group is calculated by the following formula: wherein, and are the face RGB region feature vector and face IR region feature vector, respectively, is the region consistency score.

6. The method of claim 1, wherein the method further comprises: The region consistency score set is punished for inconsistency and aggregated by weight based on the region importance attention weight set and the region inconsistency feature vector set to obtain the living body feature score, including: the region consistency score set is punished for inconsistency and aggregated by weight based on the region importance attention weight set and the region inconsistency feature vector set according to the following formula: wherein, is an inconsistency amplifier module, is a learnable scalar parameter is each region consistency score in the region consistency score set, is the region importance attention weight, is each region inconsistency feature vector in the region inconsistency feature vector set, is the living body feature score.

7. A live human face detection system fusing visible and infrared images, characterized in that, including: a face image acquisition module for acquiring a face visible light image and a face infrared image; a face image feature multi-modal analysis module for performing parallel multi-modal feature and semantic mask extraction on the face visible light image and the face infrared image to obtain a face RGB depth feature map, a face IR depth feature map, and a face semantic mask image, including: performing shared shallow feature extraction on the face visible light image and the face infrared image to obtain a face RGB shallow feature map and a face IR shallow feature map; performing independent deep feature extraction on the face RGB shallow feature map and the face IR shallow feature map to obtain the face RGB depth feature map and the face IR depth feature map; performing feature fusion on the face RGB depth feature map and the face IR depth feature map to obtain a face multi-modal deep fusion feature map; inputting the face multi-modal deep fusion feature map into a lightweight segmentation head network module to obtain the face semantic mask image; a face image feature pooling module for performing semantic region guided feature pooling on the face RGB depth feature map and the face IR depth feature map based on the face semantic mask image to obtain a face RGB region feature vector set and a face IR region feature vector set; The live body feature score calculation module is configured to perform cross-modal attention and inconsistency amplification on the face RGB region feature vector set and the face IR region feature vector set to obtain a live body feature score, and includes: calculating a region consistency score between each corresponding face RGB region feature vector and face IR region feature vector in the face RGB region feature vector set and the face IR region feature vector set to obtain a region consistency score set; calculating a region inconsistency feature vector between each corresponding face RGB region feature vector and face IR region feature vector in the face RGB region feature vector set and the face IR region feature vector set to obtain a region inconsistency feature vector set; calculating a set of importance attention weights of each region based on the region inconsistency feature vector set; and performing inconsistency punishment and weighted aggregation on the region consistency score set based on the set of importance attention weights of each region and the region inconsistency feature vector set to obtain the live body feature score. The live body face detection module is configured to perform final decision based on the face RGB depth feature map, the face IR depth feature map and the live body feature score to obtain a live body face detection result.

Images