Three-mode-based binocular face living body detection method, medium and equipment

Abstract

The application provides a three-mode-based binocular face living body detection method, medium and equipment; the method is: acquiring an RGB image and an infrared image of a binocular camera, and generating a depth map; inputting the three mode images into a face living body detection model; the face living body detection model comprises a ViT model and a mutual information module; after the three mode images are processed by a layer normalization module and a multi-head self-attention module, two mode features are input into the mutual information module to form a mode pair; the input features of the mode pair are spliced, convoluted and activated, reweighted, spliced and convoluted with the interaction features; after integration, the model output is obtained, so that the face living body detection result is obtained. The method maximizes the mutual information between different modes to adaptively enhance the beneficial mode and suppress the unfavorable mode, excavates more effective classification features, and improves the classification accuracy and generalization performance of the face fraud detection method.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, more particularly, to a three-modality-based binocular face living body detection method, medium and device. BACKGROUND

[0002] Face living body detection is a very important part of face recognition task, which can ensure the reliability of face recognition in the financial industry or security scene, such as face payment, face access to cash in self-service machines, etc. Since the types of attacks on face recognition are increasing rapidly, researchers believe that single-modality face living body detection has weak ability to resist attacks. Fortunately, multi-modality face images can provide additional and complementary information, which can greatly improve the robustness of face living body detection. Specifically, 2D attacks based on printing are very easy to distinguish in the depth modality, but it is difficult to distinguish in the visible light modality. Under the interference of complex light, the RBG image is not clear, overexposed or overexposed, while the infrared light image is not affected by the complex light.

[0003] However, the current multi-modality face living body detection has the following limitations, which makes it difficult to be widely used in some application scenarios:

[0004] (1) The visible light and infrared light camera hardware design is simple and easy to obtain, but the depth map involves complex light path design, environmental light interference, high cost, etc. It is difficult to obtain;

[0005] (2) The camera with depth map is too large in size and difficult to deploy on mobile terminals;

[0006] (3) Due to the complexity of the application scene, the current multi-modality living body detection algorithm has poor recognition performance, and the main reasons are as follows:

[0007] (1) Previous invention patents believe that multi-modality can discover more internal fraud traces, and most of them simply splice the features of the three modalities; however, the reliability of each modality can fluctuate according to the attack type, and it is not advisable to strictly or uniformly process each modality; therefore, the model must adaptively prioritize processing a particular modality or image feature area according to its reliability, and a simple fusion of each modality feature strategy may not meet the requirements; for example, some modalities (e.g., depth) may have non-defensive properties for specific attacks (e.g., 3D masks);

[0008] (2) During the customer use process, the customer's face is too far or too close, and it is not necessarily in the center of the camera screen, which leads to poor face photos input into the model and reduces the recognition rate.

[0009] (4) On some application terminals, the client is required to cooperate with the camera to ensure that the face is in front of the camera, and the client experience is not good.

[0010] Currently, there are the following documents for face liveness detection using binocular stereo vision technology:

[0011] In 2015, Beijing Haixin Kejin Company proposed an invention patent "Face detection method, device and system based on binocular stereo vision" (publication number: CN104834901A). This method generates a depth map through binocular imaging, and then determines whether the face image constitutes a three-dimensional face structure according to a preset three-dimensional structure classification rule to determine whether the face is a living body or a non-living body. The false positive rate in 3D head model face is 100%, and the effect is very poor. In 2015, the Institute of Semiconductors of the Chinese Academy of Sciences proposed an invention patent "Face liveness detection method and system" (publication number: CN105023010A). This method is basically the same as the invention patent CN104834901A of Beijing Haixin Kejin Company. Both methods generate a depth map through binocular imaging, and then determine whether the face is a living body according to the depth information. For 3D head model face, the false positive rate is high, and the effect is poor. In 2022, Jiangsu Jiquai Intelligent Optoelectronic Company proposed an invention patent "Staff fatigue detection method and device" (publication number: CN115880757A). This method uses binoculars to achieve liveness detection and obtains face width through binocular stereo vision without generating a depth map. The recognition rate of binocular liveness detection is limited.

[0012] There are the following documents for face liveness detection using multi-modal technology:

[0013] The invention patent "Multi-modal face living body detection method and system" (publication number: CN112487922A) proposed by OBI Vision Company in 2021, the method is to detect face in three modalities (color image, infrared image and depth image) respectively, and then input into three neural networks for face feature extraction, and then combine the features to judge whether it is a living body. The invention patent "Single-modal face living body detection method based on multi-modal face training" (publication number: CN113705400A) proposed by Sun Yat-sen University at the end of 2021. The idea of this method is to generate infrared and depth images from visible light images, and then splice the features of the three modalities to achieve it; because the generated images by the generative adversarial network and the images actually collected by the camera are still quite different, the actual test performance is very poor, and the generator is unstable. The invention patent "Multi-modal fusion face living body detection model generation method and device, and electronic equipment" (publication number: CN116311451A) proposed by Hangzhou Qiyuan Vision Company in 2023, the multi-modal of this method only includes two modalities, which are color image and infrared image, and the method is realized by cross comparison, that is, visible light generates infrared light image, infrared light image generates visible light image, and the authenticity is realized by generating image and original image. The biggest defect of this method is the instability of the generation network. The invention patent "Multi-modal face anti-fraud detection method based on AR-MLP" (publication number: CN117011911A) proposed by Zhejiang University of Technology in 2023, the method processes the input face RGB image, depth image and infrared image to realize face multi-modal feature fusion, and the fusion method is to fuse the features of the same face area patch of the three modalities. This method is better than CN112487922A and CN116311451A, but the actual test shows that the recognition performance is limited. The invention patent "Multi-modal face living body detection method based on attention mechanism" (publication number: CN117894082A) proposed by Beijing University of Technology in 2024, the method is to respectively perform image preprocessing and feature extraction on three modalities (color image, infrared image and depth image), and multi-level feature fusion; from the experimental results given by the patent, the BPCER is 1.82% in a single database, which is not bad, but it still deviates from the actual application, because the actual application must encounter multiple domains, that is, cross-domain test, and the cross-domain performance is still poor in actual test. SUMMARY

[0014] In order to overcome the defects and shortcomings in the prior art, the purpose of the present application is to provide a three-modal binocular face living body detection method, medium and equipment; the method proposes a method of maximizing mutual information between different modalities to realize adaptive enhancement of beneficial modalities while suppressing unfavorable modalities, and excavates more rich and effective classification features, improves the classification accuracy and generalization performance of the existing face fraud detection method.

[0015] To achieve the above objectives, the present invention provides a three-modal binocular face liveness detection method, comprising the following steps:

[0016] Step S1: Acquire RGB and infrared images from the binocular camera and generate a depth map;

[0017] Step S2: Combine RGB image, infrared image, and depth image. Figure Three The images of various modalities are input into the face liveness detection model;

[0018] The face liveness detection model includes three ViT models and two mutual information modules; the three ViT models are respectively designed to process three modalities of images; each of the three ViT models includes a layer normalization module and a multi-head self-attention module; the two ViT models that process RGB images and infrared images also include a layer normalization module and a multilayer perceptron.

[0019] After the RGB image, infrared image, and depth map are processed by the layer normalization module and the multi-head self-attention module, respectively, features are obtained. , , ; Features and characteristics Forming a modal pair, features and characteristics This forms another modal pair, which are then input into the two mutual information modules respectively;

[0020] The mutual information module will pair the two modes. m1 and m2 Input features , Interactive features are obtained by concatenation and convolution. f fused ; through interaction features f fused Perform modal m1 and m2 Activation yields two information masks. m m1 , m m2 Use information masking m m1 , m m2 For input features respectively , Features are obtained by reweighting z m1_new , z m2_new Then, respectively with interaction features ffused The output features of the mutual information module are obtained by performing splicing and convolution z out_m1 、 z out_m2 ;

[0021] The features corresponding to the RGB image and the features corresponding to the infrared image After being processed by the layer normalization module two and the multilayer perceptron, the output features of the mutual information module corresponding to the modal are respectively superimposed, and then integrated to obtain the output of the face living body detection model; and the face living body detection result is obtained according to the output of the face living body detection model.

[0022] Preferably, the interaction feature f fused is:

[0023] ;

[0024] Wherein, Conv(·) represents convolution calculation; Cat represents feature connection;

[0025] Two information masks m m1 、 m m2 is:

[0026] ;

[0027] Wherein, sigmoid(·) is an activation function;

[0028] The features z m1_new 、 z m2_new are respectively:

[0029] ;

[0030] ;

[0031] The output features of the mutual information module z out_m1 、 z out_m2 are respectively:

[0032] ;

[0033] .

[0034] Preferably, the RGB image, infrared image, and depth map are processed by a layer normalization module and a multi-head self-attention module, respectively. This means that for each modal image of the RGB image, infrared image, and depth map, each modal image is first processed by a layer normalization module and a multi-head self-attention module. Where H, W, and C represent the height, width, and number of channels of the image, respectively, the image is uniformly divided into b image blocks, each with a width and height of s. ; and represent image patches as sequences through a single layer of two-dimensional convolution. Then in the sequence E I Add a category tag to the header After adding location codes, the data is input into the multi-head self-attention module for processing.

[0035] Preferably, in the ViT model for processing depth maps, the layer normalization module and the multi-head self-attention module are connected sequentially; the multi-head self-attention module is connected to the mutual information module.

[0036] In the two ViT models processing RGB and infrared images, the layer normalization module 1 and the multi-head self-attention module are connected sequentially; the multi-head self-attention module is connected to the mutual information module; the output of the multi-head self-attention module is added to the layer normalization module 1; then it is connected sequentially to the layer normalization module 2 and the multilayer perceptron, and added to the output of the multilayer perceptron and the output of the mutual information module to obtain the output of the ViT model; the outputs of the two ViT models are concatenated and feature fused, and then connected to the Softmax layer to obtain the output of the face liveness detection model.

[0037] Preferably, the binocular camera includes an adjustment module for adjusting the angle;

[0038] In step S1, during the process of acquiring the RGB and infrared images from the binocular camera, the adjustment module adjusts the angle of the binocular camera so that the face is centered in the RGB and infrared images.

[0039] Preferably, in step S1, the method for generating the depth map is as follows: first, based on the RGB and infrared images acquired by the binocular cameras, calculate the distance from each point on the face to the planes of the two camera sensors of the binocular cameras. z :

[0040] ; ;

[0041] in, f This represents the focal length, which is the distance from the lens to the sensor. b This represents the distance between two camera sensors, that is, the physical distance between the two camera sensor chips; u L and uR respectively represent the horizontal coordinates of the pixel points on which the points on the face are imaged on the left camera and the right camera respectively; d representing parallax;

[0042] According to the distance z , a depth map is generated.

[0043] A readable storage medium, wherein the storage medium stores a computer program, the computer program causes the processor to execute the three-modal based binocular face living body detection method when the processor executes the computer program.

[0044] A computer device, comprising a processor and a memory for storing a program executable by the processor, the processor implements the three-modal based binocular face living body detection method when executing the program stored in the memory.

[0045] Compared with the prior art, the present application has the following advantages and beneficial effects:

[0046] 1. In view of the problem that the current face living body detection technology has insufficient generalization ability when coping with unknown attack modes, multiple types of camera sensors and complex environmental changes, the present application proposes a method of maximizing mutual information between different modalities to realize adaptive enhancement of beneficial modalities while suppressing unfavorable modalities, thereby mining more rich and effective classification features and improving the classification accuracy and generalization performance of existing face fraud detection methods.

[0047] 2. The integrated face fraud module is superimposed on the target intelligent tracking technology of the distance measuring and dual-camera adjustment module to realize adaptive face of the system, which not only improves the overall safety and reliability of domestic bank terminal devices, but also greatly enhances the actual operation experience of customers. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure One is a flowchart of the three-modal based binocular face living body detection method of the present application;

[0049] Fig. 2(a) and Fig. 2(b) are respectively a geometric diagram and a calculation principle diagram of a triangle formed by the image plane of two image sensors and the measured object;

[0050] Figure Three is a structural schematic diagram of the face living body detection model of the present application;

[0051] Figure Four is a schematic diagram of the internal implementation of the mutual information module of the present application. DETAILED DESCRIPTION

[0052] The present application will be further described in detail below in combination with the drawings and specific embodiments.

[0053] Embodiment one

[0054] The embodiment is a three-modal binocular face living body detection method, as shown in the following steps: Figure One

[0055] Step S1, the binocular camera hardware is two independent monocular cameras, the camera wavelength is 380nm~1200nm, and the parameters are set to collect visible light and infrared light, that is, to collect RGB images and infrared images respectively. The binocular camera is provided with an adjusting module for adjusting the angle; the adjusting module can adopt a micro motor or the like.

[0056] The binocular camera collects face video data, and the hardware structure ensures that the face is within the preset common field of view of the binocular camera. Based on the RGB image, the distance and the angle of the upper and lower positions of the current face and the camera are obtained; the adjusting module adjusts the angle of the binocular camera to make the face in the center of the RGB image and the infrared image.

[0057] According to the face video data of the binocular camera, the RGB image and the infrared image are obtained, and the depth map is generated. Based on the parallax of the left eye camera and the right eye camera of the binocular camera, the three-dimensional information is obtained by the triangulation principle, that is, a triangle is formed between the image planes of the two image sensors and the measured object, as shown in FIG. 2(a). Knowing the positional relationship between the two image sensors, the three-dimensional coordinates of the object in the common field of view of the two image sensors can be obtained, and the calculation principle is shown in FIG. 2(b); the geometric relationship formula is:

[0058] ;

[0059] Among them, z represents the distance from each point on the face of the photographed object to the plane of the two camera sensor planes of the binocular camera; f represents the focal length, that is, the distance from the lens to the sensor; b represents the distance between the two camera sensors, that is, the physical distance between the two camera sensor chips; u L and u R respectively represent the horizontal coordinates of the pixel points of the points on the face imaged on the left camera and the right camera;

[0060] According to mathematical operation, z is:

[0061] ; ;

[0062] Among them, d represents the parallax; according to the distance z , the depth map is generated. ​

[0063] Step S2, inputting the RGB image, the infrared image and the depth image into the face living body detection model. Figure Three A modality image is input into the face living body detection model.

[0064] The face living body detection model, as shown in the figure, is established on a ViT (Visual Transformer) and is fine-tuned by using frozen pre-training weights. Figure Three The face living body detection model includes three ViT models and two mutual information modules (MI); the three ViT models correspond to processing three kinds of modality images respectively; the three ViT models each include a layer normalization module one (Norm) and a multi-head self-attention module (MHA); the two ViT models processing the RGB image and the infrared image further include a layer normalization module two (Norm) and a multi-layer perceptron (MLP).

[0065] The face living body detection model of the application adopts a mutual information module, which is derived by the inventors of the application through the following research:

[0066] Let random variables A and B represent possible values of the RGB image feature representation, the infrared image feature representation and the depth map feature representation respectively; the mutual information value is calculated by the following formula:

[0067] ;

[0068] Wherein, is the mutual information value, a and b represent possible values of the RGB image feature, the infrared image feature and the depth map feature respectively, is the joint probability distribution of two different modality images, , is a modality marginal probability distribution of the RGB image feature, the infrared image feature and the depth map feature; a threshold is set, and the degree of dependence is judged according to the mutual information value: when , it is determined that the dependence between the modalities is strong; when , it is determined that the dependence between the modalities is weak; by analyzing the unique information provided by each modality and its potential contribution to the fusion task, the complementarity between the modalities is determined.

[0069] Specifically: 1) the depth map feature can well prevent various material attacks in two-dimensional planes, but it is non-defensive to specific attacks (for example, 3D masks), when the attack object is defined as a 3D mask stereoscopic face portrait, the depth map and the RGB image (or the depth map and the infrared image) The value is very small, that is, the mutual information of the depth map and the RGB image or the infrared image is independent of each other; when the attack object is a two-dimensional paper image and an electronic tablet image, The value is large, that is, the mutual information of the depth map and the RGB image or the infrared image is large, and the depth map can be clearly known to be a fake attack;

[0070] 2) The infrared image feature is independent of the ambient light illumination. When the ambient light is overexposed or too dark (such as at night), the infrared image imaging is not affected; the RGB image is just the opposite, related to the ambient light intensity; when the ambient light is overexposed or too dark, the infrared image and the RGB image The value is very large, that is, the mutual information of the infrared image and the RGB image is large, and the current lighting environment can be known through the RGB image and the infrared image. The current model is suitable for using the characteristics of the infrared image rather than the characteristics of the RGB image; when the ambient light is normal brightness, The value is very small, that is, the mutual information of the infrared image and the RGB image is large and independent of each other, and the current model needs to use the characteristics of both to jointly determine the authenticity of the living body.

[0071] They usually maximize the mutual information between the features extracted from different views, modalities or images, which come from data augmentation, aiming to capture high-level factors that affect across different perspectives - for example, there are certain different perspective spoof traces or certain inconsistencies in the data. This ability is particularly valuable in multi-modal FAS, where each modality has unique advantages or weaknesses in countering specific attack types. By maximizing the mutual information between modalities, the model can adaptively emphasize task-related information, thereby enhancing reliable modalities while mitigating the impact of unreliable modalities.

[0072] In general, the reliability of each modality can fluctuate depending on the attack type, and it is not advisable to strictly or uniformly handle each modality. Therefore, the model must adaptively prioritize specific patterns or regions according to their reliability. To achieve this, the present invention proposes a mutual information module (MI) based on the ViT model, which dynamically emphasizes reliable patterns and suppresses unreliable patterns by utilizing mutual information maximization, improving the classification accuracy and generalization performance of existing face fraud detection methods.

[0073] In the ViT model processing the depth map, the layer normalization module one and the multi-head self-attention module are connected in turn; the multi-head self-attention module is connected with the mutual information module;

[0074] In the two ViT models processing RGB and infrared images, the layer normalization module 1 and the multi-head self-attention module are connected sequentially; the multi-head self-attention module is connected to the mutual information module; the output of the multi-head self-attention module is added to the layer normalization module 1; then it is connected sequentially to the layer normalization module 2 and the multilayer perceptron, and added to the output of the multilayer perceptron and the output of the mutual information module to obtain the output of the ViT model; the outputs of the two ViT models are concatenated and feature fused, and then connected to the Softmax layer to obtain the output of the face liveness detection model.

[0075] After the RGB image, infrared image, and depth map are processed by the layer normalization module and the multi-head self-attention module, respectively, features are obtained. , , .

[0076] Specifically, the RGB image, infrared image, and depth map are processed by a layer normalization module and a multi-head self-attention module, respectively. This means that for each modal image of the RGB image, infrared image, and depth map, each modal image is first processed by a layer normalization module and a multi-head self-attention module. Where H, W, and C represent the height, width, and number of channels of the image, respectively, the image is uniformly divided into b image blocks, each with a width and height of s. ; and represent image patches as sequences through a single layer of two-dimensional convolution. Then in the sequence E I Add a category tag to the header After adding location codes, the data is input into the multi-head self-attention module for processing.

[0077] Features and characteristics Forming a modal pair, features and characteristics Another modality pair is formed and input into two mutual information modules respectively; the present invention feeds the output of each multi-head self-attention module to the mutual information module for weight adjustment so that the reliability of each modality can fluctuate according to the attack type.

[0078] In each mutual information module, the two modes of the mode pair are m1 and m2 That is, if m1 When in RGB mode, m2 For infrared mode ir or depth mode d;

[0079] Modality pairing of two modes m1 and m2 Input features , Concatenate and convolve by directly connecting features along the channel dimension and feeding into lightweight cross-attention convolution blocks to obtain cross-attention features f fused :

[0080] ;

[0081] where, Conv(·) denotes convolution calculation; Cat denotes feature connection;

[0082] By activating the cross-attention features f fused modalities m1 and m2 , two information masks m m1 , m m2 :

[0083] ;

[0084] where, sigmoid(·) is an activation function.

[0085] Two information masks m m1 , m m2 reflect the information with the most semantic information as the mutual information (MI) rich region. We believe that the higher the weight of the region (information point), the more reliable the information it involves. On the contrary, the region associated with a lower weight may carry redundant or negative information.

[0086] Then, the information masks m m1 , m m2 are respectively used to reweight the input features 、 to obtain features z m1_new 、 z m2_new :

[0087] ;

[0088] ;

[0089] respectively concatenated and convolved with the cross-attention features f fused to obtain the output features of the mutual information module z out_m1 、 z out_m2 :

[0090] ;

[0091] ;

[0092] Figure Four is an example of interaction between RGB images and infrared images, a schematic diagram of the implementation of the mutual information module (MI).

[0093] features corresponding to the RGB image features corresponding to the infrared image After being processed by the second layer normalization module and the multi-layer perceptron, the output features of the corresponding modal of the mutual information module are superimposed, and then integrated to obtain the output of the face living body detection model; the face living body detection result is obtained according to the output of the face living body detection model.

[0094] The present application has achieved the best HTER and accuracy on four public data sets CASIA-CeFA, PADISI-Face, CASIA-SURF and WMCA.

[0095] The CASIA-SURF data set has a total of 1000 volunteers participating in recording, a total of 21000 multi-modal video segments. The data comes from multiple channels (visible light, depth map and near-infrared), and uses flat printing attacks or curl printing attacks, and randomly deducts the eye, nose and mouth regions. The upgraded version of the data set CASIA-SURF CeFA has been released in 2020, adding cross-racial volunteer identities to test the generalization of the algorithm between races. The multi-channel presentation attack (WMCA) data set contains 1941 short video records from 72 different identities of real people and presentation attack disguises. The data is recorded from several channels, including color, depth, infrared and thermal imaging. The PADISI-Face data set has a total of 360 volunteers participating in recording, a total of 1105 real video, 924 presentation attack videos, and as many as 37 types of attacks.

[0096] In order to verify the beneficial effects of the method of the present application, the method of the present application and the prior art are compared in the experiments of the four public data sets CASIA-CeFA (abbreviated as C), PADISI-Face (abbreviated as P), CASIA-SURF (abbreviated as S) and WMCA (abbreviated as W). The results are shown in Table 1; wherein, CPS->W represents the training set as C, P, S, and the test set as W; CPW->S represents the training set as C, P, W, and the test set as S; CSW->P represents the training set as C, S, W, and the test set as P; PSW->C represents the training set as P, S, W, and the test set as C.

[0097] Table 1 Experimental results in this paper (unit: %)

[0098]

[0099] From Table 1, it can be seen that on the four data sets, the method of the present application based on maximizing mutual information between different modalities to adaptively enhance the beneficial modalities while suppressing the adverse modalities achieves quite competitive results. The average half error rate (HTER, the lower the better) and the accuracy (AUC, the higher the better) are optimal. ViTAF, ViT+AMA and MMDG in Table 1 are three algorithms based on the ViT model, and the method of the present application is obviously superior to them in cross-domain test effect. The above experimental results show that the method of the present application can better process and classify when facing various attack modes that the model has not seen.

[0100] Embodiment Two

[0101] The embodiment is a readable storage medium, wherein the readable storage medium stores a computer program, and the computer program, when executed by a processor, causes the processor to execute the three-modal based binocular face living body detection method of embodiment one.

[0102] Embodiment Three

[0103] The embodiment is a computer device, comprising a processor and a memory for storing a program executable by the processor, and the processor, when executing the program stored in the memory, implements the three-modal based binocular face living body detection method of embodiment one.

[0104] The above embodiments are the preferred embodiments of the present application, but the embodiments of the present application are not limited to the above embodiments, and any changes, modifications, substitutions, combinations and simplifications made without departing from the spirit and principles of the present application shall be equivalent replacement modes and shall be included in the protection scope of the present application.

Claims

1. A three-modal binocular face liveness detection method, characterized in that: Includes the following steps: Step S1: Acquire RGB and infrared images from the binocular camera and generate a depth map; Step S2: Input the RGB image, infrared image, and depth map into the face liveness detection model; The face liveness detection model includes three ViT models and two mutual information modules; the three ViT models are respectively designed to process three modalities of images; each of the three ViT models includes a layer normalization module and a multi-head self-attention module; the two ViT models that process RGB images and infrared images also include a layer normalization module and a multilayer perceptron. After the RGB image, infrared image, and depth map are processed by the layer normalization module and the multi-head self-attention module, respectively, features are obtained. , , ; Features and characteristics Forming a modal pair, features and characteristics This forms another modal pair, which are then input into the two mutual information modules respectively; The mutual information module will pair the two modes. m1 and m2 Input features , Interactive features are obtained by concatenation and convolution. f fused ; through interaction features f fused Perform modal m1 and m2 Activation yields two information masks. m m1 , m m2 Use information masking m m1 , m m2 For input features respectively , Features are obtained by reweighting z m1_new , z m2_new Then, respectively with interaction features f fused By concatenating and convolving, the output features of the mutual information module are obtained. z out_m1 , z out_m2 ; Features corresponding to RGB images Features corresponding to infrared images After processing by the layer normalization module 2 and the multilayer perceptron, the output features of the corresponding modal of the mutual information module are superimposed and then integrated to obtain the output of the face liveness detection model; the face liveness detection result is obtained based on the output of the face liveness detection model.

2. The trimodal binocular face liveness detection method according to claim 1, characterized in that: The interactive features f fused for: ； Where Conv(·) represents convolution computation; Cat represents feature connection; Two information masks m m1 , m m2 for: ； Where sigmoid(·) is the activation function; The features z m1_new , z m2_new They are respectively: ； ； Output characteristics of mutual information module z out_m1 , z out_m2 They are respectively: ； 。 3. The trimodal binocular face liveness detection method according to claim 1, characterized in that: The RGB image, infrared image, and depth map are processed by a layer normalization module and a multi-head self-attention module, respectively. This means that for each modal image of the RGB image, infrared image, and depth map, each modal image is first processed by a layer normalization module and a multi-head self-attention module. Where H, W, and C represent the height, width, and number of channels of the image, respectively, the image is uniformly divided into b image blocks, each with a width and height of s. ; and represent image patches as sequences through a single layer of two-dimensional convolution. Then in the sequence E I Add a category tag to the header After adding location codes, the data is input into the multi-head self-attention module for processing.

4. The trimodal binocular face liveness detection method according to claim 1, characterized in that: In the ViT model for processing depth maps, the layer normalization module 1 and the multi-head self-attention module are connected sequentially; the multi-head self-attention module is connected to the mutual information module. In the two ViT models processing RGB and infrared images, the layer normalization module 1 and the multi-head self-attention module are connected sequentially; the multi-head self-attention module is connected to the mutual information module; the output of the multi-head self-attention module is added to the layer normalization module 1; then it is connected sequentially to the layer normalization module 2 and the multilayer perceptron, and added to the output of the multilayer perceptron and the output of the mutual information module to obtain the output of the ViT model; the outputs of the two ViT models are concatenated and feature fused, and then connected to the Softmax layer to obtain the output of the face liveness detection model.

5. The trimodal binocular face liveness detection method according to claim 1, characterized in that: The binocular camera is equipped with an adjustment module for adjusting the angle; In step S1, during the process of acquiring the RGB and infrared images from the binocular camera, the adjustment module adjusts the angle of the binocular camera so that the face is centered in the RGB and infrared images.

6. The trimodal binocular face liveness detection method according to claim 1, characterized in that: In step S1, the method for generating the depth map is as follows: First, based on the RGB and infrared images acquired by the binocular cameras, calculate the distance from each point on the face to the planes of the two camera sensors of the binocular cameras. z : ； ； in, f Represents focal length; b This represents the distance between the two camera sensors; u L and u R These represent the x-coordinates of the pixels on the left and right cameras, respectively, representing the points on the human face as imaged by the left and right cameras. d Represents parallax; According to distance z Generate a depth map.

7. A readable storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, causes the processor to perform the trimodal binocular face liveness detection method according to any one of claims 1-6.

8. A computer device comprising a processor and a memory for storing a processor-executable program, characterized in that, When the processor executes the program stored in the memory, it implements the trimodal binocular face liveness detection method according to any one of claims 1-6.

Images