Multi-modal face living body detection method and device, electronic equipment and medium

Abstract

The invention provides a multi-modal human face in-vivo detection method, which comprises the following steps of: constructing a modal unified embedding preprocessing module, and carrying out size alignment and image enhancement processing on a received human face visible light image, a depth image and an infrared image to obtain feature vectors of various images; performing deep expression learning on the feature vectors through a cross-layer fusion visual Transform module to obtain a feature matrix of three modals; constructing a dynamic weighted feature fusion module based on the feature matrix, introducing a learnable modal perception weight, calculating a weight feature corresponding to each modal, realizing unified scale adjustment through normalization, and performing fusion to obtain a multi-modal fusion feature; and classifying the faces according to the fused features to obtain real and false face classifications. According to the method, information of different modals can be effectively fused, detail clues in true and false face images can be effectively extracted, and the problems that an existing multi-modal face living body detection scheme is low in accuracy, poor in generalization performance and the like are solved.

Description

Technical Field

[0001] The present invention relates to the field of face recognition technology, and in particular to a multimodal face liveness detection method, device, electronic equipment and medium. Background Art

[0002] With the booming development of smart cities, intelligent devices, and other related industries, facial recognition technology has gradually become part of people's daily lives. It is widely used in scenarios such as face unlocking, digital payments, and access control. However, facial fraud attackers can use forgeries such as photos, videos, or 3D masks to bypass facial recognition systems and gain access as someone else. Ensuring the security and reliability of facial recognition systems has become a pressing task. Therefore, as an indispensable component of facial recognition technology, facial liveness detection plays a crucial role in information security and personal privacy. It also offers implications for biometric authentication and secure identification in other fields. Therefore, improving the resilience of facial recognition systems and preventing fake face attacks from penetrating facial recognition systems is a crucial research topic in the field of machine vision.

[0003] As the most complex biological entity, the human face possesses a variety of features, including appearance, expression, posture, and texture. However, deep neural networks, as black-box models, cannot determine whether specific features extracted from facial images are interference features. Furthermore, due to the diverse types of facial attacks, the distribution of data samples is relatively dispersed. Therefore, the network is likely to extract various facial interference features. For example, the features learned by the network model may represent the differences between faces of different identities rather than the differences between real and fake faces. Furthermore, the sample distribution gap between real face samples can be increased by differences in race and expression. These uncertainties pose significant challenges to the design of facial liveness detection models and algorithms.

[0004] Current face liveness detection methods based on multimodal fusion are easily dominated by certain modalities. When information from certain modalities is missing, the classification accuracy of the constructed face anti-spoofing model will be poor. Furthermore, information differences between images of different modalities can lead to low fusion efficiency.

[0005] Therefore, how to effectively integrate multimodal information and overcome the lack and interference of certain modal information is a technical problem that needs to be urgently solved in the field of multimodal face liveness detection. Summary of the Invention

[0006] In view of this, the embodiments of the present disclosure provide a multimodal face liveness detection method, device, electronic device, and medium, which at least partially solve the problems existing in the prior art.

[0007] In a first aspect, an embodiment of the present disclosure provides a multimodal face liveness detection method, which includes:

[0008] A modality-unified embedding preprocessing module is constructed to perform size normalization and image enhancement on the received visible light image, depth image, and infrared image of the face, obtaining the feature vectors of the visible light image, depth image, and infrared image, respectively.

[0009] Constructing a cross-layer fusion visual Transformer module to perform deep expression learning on the feature vectors of the visible light image, the depth image, and the infrared image, and fusing shallow texture with deep semantic information to obtain feature matrices of the three modalities;

[0010] Based on the feature matrix, a dynamic weighted feature fusion module is constructed, which introduces learnable modal perception weights, calculates the weight features corresponding to each modality, uniformly scales the normalized weight features, and fuses them to obtain multimodal fusion features with modality selectivity.

[0011] The faces are classified according to the multimodal fusion features to obtain real and false face classifications.

[0012] According to a specific implementation of the embodiment of the present disclosure, the constructed modality is uniformly embedded in a preprocessing module, and size normalization and image enhancement processing are performed on the received visible light image, depth image, and infrared image of the face to obtain a feature vector of the visible light image, a feature vector of the depth image, and a feature vector of the infrared image, respectively, including:

[0013] Resizing and scaling the visible light image of the face, the depth image, and the infrared image to preset specifications consistent with a convolutional neural network input layer;

[0014] Perform rotation, flip, and transposition operations on the resized visible light image, depth image, and infrared image of the face;

[0015] Perform block slicing and image block embedding operations on the visible light image, depth image, and infrared image of the face after rotation, flipping, and transposition operations to obtain multiple image blocks with overlapping areas;

[0016] A depth embedding operation is performed on each image block to obtain a feature vector of the visible light image, a feature vector of the depth image, and a feature vector of the infrared image.

[0017] According to a specific implementation of the embodiment of the present disclosure, the cross-layer fusion visual Transformer module is constructed to perform deep expression learning on the feature vectors of the visible light image, the feature vectors of the depth image, and the feature vectors of the infrared image, and to fuse shallow texture and deep semantic information to obtain feature matrices of the three modalities, including:

[0018] Using a multi-head self-attention mechanism to extract features from the feature vector;

[0019] Performing adaptive transformation and cross-layer connection of feature vectors on the attention calculation in the self-attention mechanism to fuse shallow texture features with deep structural semantic features, thereby obtaining self-attention mechanism output results of the visible light image of the face, the depth image, and the infrared image;

[0020] The output results of the self-attention mechanism of the three modalities are normalized and residually connected, and the results of the normalization and residual connection are processed in the feedforward neural network layer to obtain the feature matrix.

[0021] According to a specific implementation of an embodiment of the present disclosure, the adaptive transformation of feature vectors for attention calculation in the self-attention mechanism and cross-layer connection include:

[0022] Performing dimension conversion on the Value vectors of the modality corresponding to the visible light image feature vector, the modality corresponding to the depth image feature vector, and the modality corresponding to the infrared image feature vector, respectively, to obtain a conversion vector having the same dimension as the modality;

[0023] Calculate the dot product of the modality's Key vector and each vector in the modality's Query vector to obtain the attention value of the modality at different spatial positions and hierarchical relationships;

[0024] Normalizing the attention values to obtain a similarity matrix within the modality;

[0025] Adding the similarity matrix and the transformation vector to obtain adaptive attention with inter-layer structure adaptability;

[0026] Calculate the head of each adaptive attention, splice the heads of multiple calculations, and obtain the output result of the cross-layer self-attention mechanism.

[0027] According to a specific implementation of the embodiment of the present disclosure, a dynamic weighted feature fusion module is constructed based on the feature matrix, a learnable modal perception weight is introduced, the weight features corresponding to each modality are calculated, the normalized weight features are uniformly scaled, and fused to obtain a multimodal fusion feature with modality selectivity, including:

[0028] The weight of each modality dynamically learned on a specific sample in the feature matrix is normalized and fused to obtain the multimodal fusion feature with modality selectivity.

[0029] According to a specific implementation of an embodiment of the present disclosure, the weight of each modality dynamically learned by the feature matrix on a specific sample is normalized and fused to obtain the multimodal fusion feature with modality selectivity, including:

[0030] Initializing the weights to obtain learnable parameters of the weights;

[0031] Optimizing the learning parameters using a gradient descent method to obtain optimized weights;

[0032] Normalizing the optimized weights to obtain modal normalized weights with probabilistic semantics;

[0033] Multiply the feature matrix of each mode by the modal normalization weight corresponding to each mode to obtain the weight feature;

[0034] The weighted features of the three modalities are summed to obtain the multimodal fusion feature with modality selectivity.

[0035] According to a specific implementation of the embodiment of the present disclosure, classifying faces based on the multimodal fusion features to obtain real and false face classifications includes:

[0036] Performing model prediction on the multimodal fusion features to obtain a prediction result;

[0037] Compare the predicted results and true labels of the entire sample space to obtain the number of correct predicted results;

[0038] Performing back-propagation update on the network model parameters, and clearing the gradients before the back-propagation update;

[0039] Outputting the loss rate and accuracy of the model training according to the number of prediction results, saving the model parameters updated each time, and selecting the model parameters corresponding to the target accuracy as the parameters of the model training;

[0040] The input visible light image, depth image and infrared image of the face are predicted in the trained network model, and the face liveness detection result is output.

[0041] In a second aspect, an embodiment of the present disclosure provides a multimodal face liveness detection device, comprising:

[0042] The modality unified embedding preprocessing module is used to perform size normalization and image enhancement on the received visible light image, depth image, and infrared image of the face, respectively obtaining the feature vector of the visible light image, the feature vector of the depth image, and the feature vector of the infrared image;

[0043] The cross-layer fusion visual Transformer module is used to learn deep representations of feature vectors and fuse shallow textures with deep semantic information to obtain feature matrices of the three modalities.

[0044] The dynamic weighted feature fusion module is used to introduce learnable modal perception weights based on the feature matrix, calculate the weight features corresponding to each modality, uniformly scale the normalized weight features, and fuse them to obtain multi-modal fusion features with modality selectivity;

[0045] The face classification module is used to classify faces according to the multimodal fusion features to obtain real and false face classifications.

[0046] According to a specific implementation of an embodiment of the present disclosure, the modality unified embedding preprocessing module performs size normalization and image enhancement processing on the received visible light image, depth image, and infrared image of the face to obtain a feature vector of the visible light image, a feature vector of the depth image, and a feature vector of the infrared image, respectively, including:

[0047] Resizing and scaling the visible light image of the face, the depth image, and the infrared image to preset specifications consistent with a convolutional neural network input layer;

[0048] Perform rotation, flip, and transposition operations on the resized visible light image, depth image, and infrared image of the face;

[0049] Perform block slicing and image block embedding operations on the visible light image, depth image, and infrared image of the face after rotation, flipping, and transposition operations to obtain multiple image blocks with overlapping areas;

[0050] A depth embedding operation is performed on each image block to obtain a feature vector of the visible light image, a feature vector of the depth image, and a feature vector of the infrared image.

[0051] According to a specific implementation of the embodiment of the present disclosure, the cross-layer fusion visual Transformer module uses a multi-head self-attention mechanism to extract features from the feature vector;

[0052] Performing adaptive transformation and cross-layer connection of feature vectors on the attention calculation in the self-attention mechanism to fuse shallow texture features with deep structural semantic features, thereby obtaining self-attention mechanism output results of the visible light image of the face, the depth image, and the infrared image;

[0053] The output results of the self-attention mechanism of the three modalities are normalized and residually connected, and the results of the normalization and residual connection are processed in the feedforward neural network layer to obtain the feature matrix.

[0054] According to a specific implementation of the embodiment of the present disclosure, the cross-layer fusion visual Transformer module performs dimensionality conversion on the Value vectors of the modality corresponding to the visible light image feature vector, the modality corresponding to the depth image feature vector, and the modality corresponding to the infrared image feature vector, respectively, to obtain a conversion vector with the same dimension as the modality;

[0055] Calculate the dot product of the modality's Key vector and each vector in the modality's Query vector to obtain the attention value of the modality at different spatial positions and hierarchical relationships;

[0056] Normalizing the attention values to obtain a similarity matrix within the modality;

[0057] Adding the similarity matrix and the transformation vector to obtain adaptive attention with inter-layer structure adaptability;

[0058] Calculate the head of each adaptive attention, splice the heads of multiple calculations, and obtain the output result of the cross-layer self-attention mechanism.

[0059] According to a specific implementation of the embodiment of the present disclosure, a dynamic weighted feature fusion module is constructed based on the feature matrix, a learnable modal perception weight is introduced, the weight features corresponding to each modality are calculated, the normalized weight features are uniformly scaled, and fused to obtain a multimodal fusion feature with modality selectivity, including:

[0060] The weight of each modality dynamically learned on a specific sample in the feature matrix is normalized and fused to obtain the multimodal fusion feature with modality selectivity.

[0061] According to a specific implementation of the embodiment of the present disclosure, the dynamic weighted feature fusion module initializes the weights to obtain learnable parameters of the weights;

[0062] Optimizing the learning parameters using a gradient descent method to obtain optimized weights;

[0063] Normalizing the optimized weights to obtain modal normalized weights with probabilistic semantics;

[0064] Multiply the feature matrix of each mode by the modal normalization weight corresponding to each mode to obtain the weight feature;

[0065] The weighted features of the three modalities are summed to obtain the multimodal fusion feature with modality selectivity.

[0066] According to a specific implementation of the embodiment of the present disclosure, the face classification module performs model prediction on the multimodal fusion features to obtain a prediction result;

[0067] Compare the predicted results and true labels of the entire sample space to obtain the number of correct predicted results;

[0068] Performing back-propagation update on the network model parameters, and clearing the gradients before the back-propagation update;

[0069] Outputting the loss rate and accuracy of the model training according to the number of prediction results, saving the model parameters updated each time, and selecting the model parameters corresponding to the target accuracy as the parameters of the model training;

[0070] The input visible light image, depth image and infrared image of the face are predicted in the trained network model, and the face liveness detection result is output.

[0071] In a third aspect, an embodiment of the present disclosure provides an electronic device, the electronic device comprising:

[0072] At least one processor; and a memory communicatively connected to the at least one processor; wherein,

[0073] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the above-mentioned multimodal face liveness detection method.

[0074] In a fourth aspect, an embodiment of the present disclosure provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute the above-mentioned multimodal face liveness detection method.

[0075] In summary, compared with the existing technology, this embodiment has the following advantages: it captures rich distinguishing information between each modality through a cross-layer fusion visual Transformer structure based on multimodal face liveness detection, and effectively extracts detailed clues in real and fake face images; at the same time, it designs a dynamic weighted feature fusion module that can dynamically learn the weight of each modality on a specific sample, effectively fusing different modal information. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below in conjunction with specific embodiments and accompanying drawings. Here, the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0077] As used herein, the term "including" and its variations represent open inclusion, i.e., "including but not limited to." Unless otherwise stated, the term "or" means "and / or." The term "based on" means "based at least in part on." The terms "an example embodiment" and "an embodiment" mean "at least one example embodiment." The term "another embodiment" means "at least one additional embodiment." The terms "first," "second," etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0078] Figure 1 A schematic flow chart of a multimodal face liveness detection method provided by an embodiment of the present invention;

[0079] Figure 2 A schematic diagram of a framework for a multimodal face liveness detection method based on cross-layer fusion visual Transformer and dynamic weighted feature fusion provided in an embodiment of the present application;

[0080] Figure 3 A schematic diagram of a cross-layer fusion visual Transformer framework provided in an embodiment of the present application;

[0081] Figure 4 A schematic diagram of an adaptive attention mechanism provided in an embodiment of the present application;

[0082] Figure 5 A schematic diagram of a dynamic weighted feature fusion structure provided in an embodiment of the present application;

[0083] Figure 6 A schematic diagram of the structure of a multimodal face liveness detection device provided in an embodiment of the present application;

[0084] Figure 7 A schematic diagram of an exemplary structure of a device capable of implementing the method according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0085] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0086] The following describes the embodiments of the present disclosure through specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present disclosure.

[0087] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on this disclosure, it should be understood by those skilled in the art that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects described herein can be used to implement an apparatus and / or practice a method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this apparatus and / or practice this method.

[0088] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present disclosure. The illustrations only show components related to the present disclosure and are not drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component can be changed at will, and the component layout type may also be more complicated.

[0089] Additionally, in the following description, specific details are provided to provide a thorough understanding of the examples. However, one skilled in the art will appreciate that the aspects described can be practiced without these specific details.

[0090] See also Figure 1 , Figure 1 A flowchart of a multimodal face liveness detection method provided in an embodiment of the present application is provided. The multimodal face liveness detection method can be executed by an electronic device, and in particular, by one or more processors within the electronic device, to implement the following steps:

[0091] S101. Construct a modality unified embedding preprocessing module to perform size normalization and image enhancement on the received visible light image, depth image, and infrared image of the face, and obtain the feature vector of the visible light image, the feature vector of the depth image, and the feature vector of the infrared image, respectively.

[0092] The size normalization and image enhancement processing of the received visible light image, depth image, and infrared image of the face include:

[0093] (1) The received visible light image, depth image, and infrared image of the face are resized and resized to the preset specifications consistent with the input layer of the convolutional neural network, for example, to a size of 128×128.

[0094] (2) Perform rotation, flip and transposition operations on the resized visible light image, depth image and infrared image of the face.

[0095] (3) The visible light image, depth image and infrared image of the face after rotation, flipping and transposition operations are cut and image block embedding operations are performed to obtain multiple image blocks with overlapping areas, so as to retain local texture details while introducing neighborhood context information; then a depth embedding operation is performed on each image block, and the image block is mapped to a feature representation suitable for subsequent processing through the embedding function to obtain the feature vector of the visible light image, the feature vector of the depth image and the feature vector of the infrared image.

[0096] For example, if an image of size W×W×C is to be divided into image blocks of size L×L×C, there will be M = W×W / L×L image blocks. These image blocks are then arranged in a column as input to the convolutional neural network. Here, W is the size of the original face image, C is the number of channels in the input image, and L is the size of an image block. Furthermore, the image block embedding operation converts the M rows of L×L×C vector dimensions into a two-dimensional input of size M×P.

[0097] In order to convert these image patches into feature representations suitable for subsequent processing, the electronic device introduces an embedding function E embed Perform feature mapping on each image block. The embedding function is defined as follows:

[0098]

[0099] in, is the flattened vector of the jth image block. W embed and W block is the learnable weight matrix, b block Is the bias term. Embedding function E embed The flattened vector applied to each image patch To generate its feature representation. These feature representations are then concatenated into a long feature vector and combined with the image category information x class and the position encoding vector E pos Combined, the final feature vector Z0 is formed. The specific operation is to perform a linear transformation on each flattened image block vector. For example, after passing through a fully connected layer E with an input size of L×L×C and an output size of P, the dimension of the vector after dimensionality reduction is P; the specific calculation formula is as follows:

[0100]

[0101] Among them, x class Indicates the category information of the image, represents the output of the i-th image block after the embedding function, Is the flattened vector of the i-th image block. In order to maintain the spatial position information between the input image blocks, it is necessary to add a position encoding vector E to the image block embedding pos After adding the position coding information, the electronic device then uniformly embeds the preprocessing module according to the modality to obtain the feature vectors of the three modalities of visible light image, depth image and infrared image, where the length of the feature vector sequence is M+1.

[0102] For details, please refer to Figure 2 , Figure 2 A schematic diagram of the framework of a multimodal face liveness detection method based on cross-layer fusion visual Transformer and dynamic weighted feature fusion provided in an embodiment of the present application.

[0103] S102. Construct a cross-layer fusion visual Transformer module to perform deep expression learning on each feature vector, and fuse shallow texture and deep semantic information to obtain the feature matrix of the three modalities after learning.

[0104] Preferably, the feature vector can be input into the cross-layer fusion visual Transformer for learning, see Figure 3 , Figure 3 This is a schematic diagram of a cross-layer fusion visual Transformer framework provided in an embodiment of the present application. Figure 3 As shown, each modal feature (such as Figure 3 In the image processing, the visible light modality features, the depth modality features, and the infrared modality features are used for feature extraction (for example, extracting long-distance dependent contextual information from image features); preferably, an adaptive transformation of the feature vector is introduced in the attention calculation to perform cross-layer connection, so as to fuse the shallow texture features with the deep structural semantic features, and finally output the results of the multi-head self-attention mechanism of the visible light image, the depth image, and the infrared image;

[0105] For example, let the modality corresponding to the visible light image feature vector be x, the modality corresponding to the depth image feature vector be y, and the modality corresponding to the infrared image feature vector be z. The Key vector, Value vector, and Query vector of modality x are respectively denoted as K x 、V x and Q x The Key vector, Value vector and Query vector of mode y are respectively denoted as K y 、V y and Q y ; The Key vector, Value vector and Query vector of mode z are respectively denoted as K z 、V z and Q z .

[0106] In the cross-layer fusion visual Transformer, V x V is the feature vector generated after the input face image is divided into local image blocks or feature embedding. x Contains the spatial position information and local texture feature information of the input face image. x Perform dimension conversion to obtain the conversion vector V with the same dimension as the modal x Trans As shown in formula (3):

[0107]

[0108] The Permute operation is used to swap the dimensions of the tensor, swapping the dimensions at positions 0 and 1, and swapping the dimensions at positions 2 and 3. Next, the tensor shape is transformed to (B, N, C×M), where B represents the input batch size, N represents the sequence length, and C represents the dimension of the modality x.

[0109] Furthermore, calculate K x Each vector in Q x The dot product of each vector in is used to obtain the attention value of the visible light image modality at different spatial positions and hierarchical relationships, that is, the query vector Q x With the key vector K x The similarity matrix is obtained by multiplying the transpose of k It is k x The similarity matrix is then normalized by the Softmax function, and the normalized similarity matrix is Transformed into a matrix distributed in the interval [0,1], and finally combined with V Trans The final cross-layer adaptive attention is calculated by addition, as shown in formula (4):

[0110]

[0111] See Figure 4 , Figure 4 A schematic diagram of an adaptive attention mechanism provided in an embodiment of the present application. In the Attention calculation process, V Trans The purpose of this operation is to keep the output tensor dimension of the self-attention mechanism consistent with the input V x The feature dimensions remain consistent, and this design introduces a direct, unweighted connection between each sublayer of the Vision Transformer. By adding an adaptive transformation of the feature vector V, this is equivalent to introducing information about the spatial position and local texture features of the facial image. This operation helps the network capture more features at specific locations and directly injects local spatial information into the attention mechanism, which has a positive impact on the description of key facial parts and local features (such as facial features and texture).

[0112] Then, repeat the above operation and calculate the Head of each operation. It should be noted that the number of repetitions can be preset.

[0113] The specific calculation is as shown in formula (5):

[0114]

[0115] Where W i Q Indicates the calculation of Q when the i-th Head x The weight, W i K Indicates that K is calculated when calculating the i-th head x The weight, W i V Indicates that V is calculated when calculating the i-th head x The weight of i=1,2,3...m, where m is the number of repeated operations.

[0116] By concatenating the n-times repeated results, we can get the result of the modality x after the multi-head self-attention mechanism, as shown in formula (6):

[0117] MultiHead(Q x ,K x ,V x )=concat(Head1,Head2,……,Head m )W O (6)

[0118] Among them, W o is the weight matrix output in the multi-head attention mechanism. After calculating the result of the modality x after multi-head self-attention, Ky 、V y and Q y With K z 、V z and Q z According to the above operations, the results of modality y and modality z after the multi-head self-attention mechanism are calculated.

[0119] After normalizing and residual connection of the self-attention mechanism results of the three modalities after the multi-head self-attention mechanism, the input is input into the feedforward neural network layer (FNN), and the feature matrix F of the three modalities can be obtained. x ,F y ,F z The specific operation is as follows, where W is the weight matrix of each level and b represents the bias vector, as shown in formula (7):

[0120] FNN(x) = max (0,xW1 + b1) W2 + b2 (7)

[0121] S103. Based on the feature matrix, a dynamic weighted feature fusion module is constructed, and learnable modal perception weights are introduced to calculate the weight features corresponding to each modality. The normalized weight features are adjusted to a uniform scale and fused to obtain multimodal fusion features with modality selectivity.

[0122] Among them, after the three modal features are extracted by the multi-feature visual Transformer structure, the three modal features are fused with multiple weights. For details, see Figure 5 , Figure 5 This is a schematic diagram of a dynamic weighted feature fusion structure provided in an embodiment of the present application. Figure 5 As shown in the figure, the weight features corresponding to each modality are calculated, and after Softmax normalization, the weights of each modality are uniformly scaled to achieve modality-selective fusion expression, thereby obtaining multimodal fusion features with stronger semantic consistency and better modality adaptability. The details are as follows:

[0123] Obtain features of three different modalities: modality x, modality y, and modality z. x represents the modality corresponding to the visible light image feature vector, y represents the modality corresponding to the depth image feature vector, and z represents the modality corresponding to the infrared image feature vector. The features of each modality represent the information in the corresponding modality.

[0124] Next, define the corresponding weights W of the three modes x , W y and W z Among them, W x , W y and W zIt is initialized as a learnable parameter and optimized by gradient descent during training. During the training of the model, the network automatically calculates the gradient of these learnable parameters and updates the weights corresponding to the three modalities through the back-propagation algorithm to minimize the defined loss function. The attention weights are normalized using the Softmax function to ensure that the sum of the weights is 1. The Softmax operation converts the feature weights corresponding to each modality into a probability distribution, where each weight value represents the importance of the corresponding modality in the fusion process. Then, the weight corresponding to each modality is multiplied by the modal feature to obtain the weight feature, as shown in formula (8):

[0125]

[0126] Finally, the three modal weight features are fused to obtain a multimodal fusion feature with modality selectivity, which is used for the final classification of live and non-live faces, as shown in formula (9):

[0127]

[0128] S104: Classify faces based on multimodal fusion features to obtain real and false face classifications.

[0129] The electronic device performs model prediction on the fusion features to obtain the prediction results. The prediction results are compared with the true labels, and the number of correct prediction results is calculated by looping through the entire training sample space.

[0130] Backpropagation updates the parameters of the cross-layer fusion visual Transformer network. Before backpropagation, the gradients are cleared. For example, the SGD (Stochastic Gradient Descent) descent method is used to update the network parameters.

[0131] Output the loss rate and accuracy of model training, save the model parameters of each iteration, and select the model parameters with the highest accuracy on the validation set (the model parameters corresponding to the target accuracy).

[0132] After obtaining the optimal model, the visible light image, depth image and infrared image of the face are input into the trained convolutional neural network to obtain a 1×2 vector output. The vector output is mapped to a probability representation in the (0,1) interval through the Softmax function, and the node with the largest probability is selected as the prediction result, thereby realizing the binary classification of face liveness detection.

[0133] In summary, the embodiment of the present application performs network model training and classification through cross-layer fusion of visual Transformer and dynamic weighted feature fusion module, effectively fusing information of different modalities and effectively extracting detail clues in real and fake face images, thereby solving the problems of low accuracy and poor generalization performance of current multimodal face liveness detection solutions.

[0134] The second embodiment of the present invention provides a multimodal face liveness detection device, see Figure 6 , Figure 6 A schematic diagram of the structure of a multimodal face liveness detection device provided in an embodiment of the present application, comprising:

[0135] The modality unified embedding preprocessing module 610 is used to input the visible light image, depth image and infrared image of the face, and perform size normalization and image enhancement on the visible light image, depth image and infrared image of the face to obtain the feature vector of the visible light image, the feature vector of the depth image and the feature vector of the infrared image respectively.

[0136] The cross-layer fusion visual Transformer module 620 is used to perform deep expression learning on each feature vector, fuse shallow texture and deep semantic information, and obtain the feature matrix of the three modalities.

[0137] The dynamic weighted feature fusion module 630 is used to construct a dynamic weighted feature fusion module based on the feature matrix, introduce learnable modal perception weights, calculate the weight features corresponding to each modality, uniformly scale the normalized weight features, and fuse them to obtain multimodal fusion features with modality selectivity.

[0138] The face classification module 640 is used to classify faces based on the multimodal fusion features to obtain real and false face classifications.

[0139] A third embodiment of the present invention further provides an electronic device, comprising:

[0140] At least one processor; and a memory communicatively connected to the at least one processor; wherein,

[0141] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the multimodal face liveness detection method of any of the aforementioned embodiments.

[0142] The fourth embodiment of the present invention further provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the multimodal face liveness detection method described in any of the aforementioned embodiments.

[0143] The fifth embodiment of the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the multimodal face liveness detection method of any of the aforementioned embodiments.

[0144] The sixth embodiment of the present invention further provides a computer program, which includes program instructions. When the program instructions are executed by a computer, the computer is caused to execute the multimodal face liveness detection method of any of the aforementioned embodiments.

[0145] Figure 7 A schematic diagram of a method or device 1000 that can implement an embodiment of the present invention is shown. In some embodiments, the method or device 1000 may include more or fewer devices than shown. In some embodiments, the method or device 1000 may be implemented using a single device or multiple devices. In some embodiments, the method or device 1000 may be implemented using cloud-based or distributed devices.

[0146] like Figure 7 As shown, device 1000 includes a processor 1001, which can perform various appropriate operations and processes according to the programs and / or data stored in a read-only memory (ROM) 1002 or the programs and / or data loaded from a storage portion 1008 into a random access memory (RAM) 1003. Processor 1001 can be a multi-core processor or can include multiple processors. In some embodiments, processor 1001 can include a general-purpose main processor and one or more special coprocessors, such as a central processing unit (CPU), a graphics processing unit (GPU), a neural network processor (NPU), a digital signal processor (DSP), etc. Various programs and data required for the operation of device 1000 are also stored in random access memory 1003. Processor 1001, read-only memory 1002, and random access memory 1003 are connected to each other via a bus 1004. Input / output (I / O) interface 1005 is also connected to bus 1004.

[0147] The processor and memory are used together to execute the program stored in the memory. When the program is executed by the computer, the methods, steps or functions described in the above embodiments can be implemented.

[0148] The following components are connected to the I / O interface 1005: an input section 1006 including a keyboard, a mouse, a touch screen, and the like; an output section 1007 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 1008 including a hard disk and the like; and a communication section 1009 including a network interface card such as a LAN card or a modem. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 1010 as needed, so that computer programs read therefrom can be installed into the storage section 1008 as needed. Figure 7 Only some components are shown schematically, which does not mean that the device 1000 only includes Figure 7 Components shown.

[0149] The systems, devices, modules, or units described in the above embodiments may be implemented by a computer or its associated components. The computer may be, for example, a mobile terminal, a smartphone, a personal computer, a laptop computer, an in-vehicle human-computer interaction device, a personal digital assistant, a media player, a navigation device, a game console, a tablet computer, a wearable device, a smart TV, an Internet of Things system, a smart home, an industrial computer, a server, or a combination thereof.

[0150] Although not shown, in an embodiment of the present invention, a computer-readable storage medium is provided, on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the multimodal face liveness detection method described in Example 1 is implemented.

[0151] Storage media in embodiments of the present invention include permanent and non-permanent, removable and non-removable items that can be used to store information using any method or technology. Examples of storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0152] Although not shown, an embodiment of the present invention further provides a computer program product, including: a computer program / instruction, which, when executed by a processor, implements the multimodal face liveness detection method described in Example 1.

[0153] The methods, programs, systems, and apparatuses of the embodiments of the present invention may be executed or implemented in a single or multiple networked computers, or may be practiced in a distributed computing environment. In the embodiments of this specification, in these distributed computing environments, tasks may be performed by remote processing devices connected via a communication network.

[0154] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, those skilled in the art will appreciate that the functional modules / units or controllers and related method steps described in the above embodiments may be implemented using software, hardware, or a combination of software / hardware.

[0155] Unless explicitly stated, the actions or steps of the methods, procedures, and methods described in accordance with the embodiments of the present invention do not have to be performed in a specific order and can still achieve the desired results. In some implementations, multitasking and parallel processing are also possible or may be advantageous.

[0156] In this document, multiple embodiments of the present invention are described, but for the sake of brevity, the description of each embodiment is not exhaustive, and the same or similar features or parts between the embodiments may be omitted. In this document, "one embodiment", "some embodiments", "example", "specific example", or "some examples" are intended to apply to at least one embodiment or example according to the present invention, but not all embodiments. The above terms do not necessarily mean to refer to the same embodiment or example. Those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are mutually contradictory.

[0157] While the exemplary systems and methods of the present invention have been specifically shown and described with reference to the foregoing embodiments, these are merely examples of the best modes for implementing the present systems and methods. Those skilled in the art will appreciate that various changes may be made to the embodiments of the systems and methods described herein when implementing the present systems and / or methods without departing from the spirit and scope of the present invention as defined in the appended claims.

Claims

1. A multimodal face liveness detection method, characterized in that: include: A modality-unified embedding preprocessing module is constructed to perform size normalization and image enhancement on the received visible light image, depth image, and infrared image of the face, obtaining the feature vectors of the visible light image, depth image, and infrared image, respectively. Constructing a cross-layer fusion visual Transformer module to perform deep expression learning on the feature vectors of the visible light image, the depth image, and the infrared image, and fusing shallow texture with deep semantic information to obtain feature matrices of the three modalities; Based on the feature matrix, a dynamic weighted feature fusion module is constructed, which introduces learnable modal perception weights, calculates the weight features corresponding to each modality, uniformly scales the normalized weight features, and fuses them to obtain multimodal fusion features with modality selectivity. The faces are classified according to the multimodal fusion features to obtain real and false face classifications.

2. The multimodal face liveness detection method according to claim 1, characterized in that: The constructed modality is uniformly embedded in a preprocessing module to perform size normalization and image enhancement on the received visible light image, depth image, and infrared image of the face, respectively obtaining a feature vector of the visible light image, a feature vector of the depth image, and a feature vector of the infrared image, including: Resizing and scaling the visible light image of the face, the depth image, and the infrared image to preset specifications consistent with a convolutional neural network input layer; Perform rotation, flip, and transposition operations on the resized visible light image, depth image, and infrared image of the face; Perform block slicing and image block embedding operations on the visible light image, depth image, and infrared image of the face after rotation, flipping, and transposition operations to obtain multiple image blocks with overlapping areas; A depth embedding operation is performed on each image block to obtain a feature vector of the visible light image, a feature vector of the depth image, and a feature vector of the infrared image.

3. The multimodal face liveness detection method according to claim 1, characterized in that: The cross-layer fusion visual Transformer module is constructed to perform deep expression learning on the feature vectors of the visible light image, the feature vectors of the depth image, and the feature vectors of the infrared image, and to fuse shallow texture and deep semantic information to obtain feature matrices of the three modalities, including: Using a multi-head self-attention mechanism to extract features from the feature vector; Performing adaptive transformation and cross-layer connection of feature vectors on the attention calculation in the self-attention mechanism to fuse shallow texture features with deep structural semantic features, thereby obtaining self-attention mechanism output results of the visible light image of the face, the depth image, and the infrared image; The output results of the self-attention mechanism of the three modalities are normalized and residually connected, and the results of the normalization and residual connection are processed in the feedforward neural network layer to obtain the feature matrix.

4. The multimodal face liveness detection method according to claim 3, characterized in that: The adaptive transformation of the feature vector for the attention calculation in the self-attention mechanism and the cross-layer connection include: Performing dimension conversion on the Value vectors of the modality corresponding to the visible light image feature vector, the modality corresponding to the depth image feature vector, and the modality corresponding to the infrared image feature vector, respectively, to obtain a conversion vector having the same dimension as the modality; Calculate the dot product of the modality's Key vector and each vector in the modality's Query vector to obtain the attention value of the modality at different spatial positions and hierarchical relationships; Normalizing the attention values to obtain a similarity matrix within the modality; Adding the similarity matrix and the transformation vector to obtain adaptive attention with inter-layer structure adaptability; Calculate the head of each adaptive attention, splice the heads of multiple calculations, and obtain the output result of the cross-layer self-attention mechanism.

5. The multimodal face liveness detection method according to claim 1, characterized in that: The method constructs a dynamic weighted feature fusion module based on the feature matrix, introduces learnable modal perception weights, calculates the weight features corresponding to each modality, uniformly scales the normalized weight features, and fuses them to obtain multimodal fusion features with modality selectivity, including: The weight of each modality dynamically learned on a specific sample in the feature matrix is normalized and fused to obtain the multimodal fusion feature with modality selectivity.

6. The multimodal face liveness detection method according to claim 5, characterized in that: Normalizing and fusing the weight of each modality dynamically learned from the feature matrix on a specific sample to obtain the multimodal fusion feature with modality selectivity includes: Initializing the weights to obtain learnable parameters of the weights; Optimizing the learning parameters using a gradient descent method to obtain optimized weights; Normalizing the optimized weights to obtain modal normalized weights with probabilistic semantics; Multiply the feature matrix of each mode by the modal normalization weight corresponding to each mode to obtain the weight feature; The weighted features of the three modalities are summed to obtain the multimodal fusion feature with modality selectivity.

7. The multimodal face liveness detection method according to claim 1, characterized in that: The classifying of faces according to the multimodal fusion features to obtain classification of real and false faces includes: Performing model prediction on the multimodal fusion features to obtain a prediction result; Compare the predicted results and true labels of the entire sample space to obtain the number of correct predicted results; Performing back-propagation update on the network model parameters, and clearing the gradients before the back-propagation update; Outputting the loss rate and accuracy of the model training according to the number of prediction results, saving the model parameters updated each time, and selecting the model parameters corresponding to the target accuracy as the parameters of the model training; The input visible light image, depth image and infrared image of the face are predicted in the trained network model, and the face liveness detection result is output.

8. A multimodal face liveness detection device, characterized in that: include: The modality unified embedding preprocessing module is used to perform size normalization and image enhancement on the received visible light image, depth image, and infrared image of the face, respectively obtaining the feature vector of the visible light image, the feature vector of the depth image, and the feature vector of the infrared image; The cross-layer fusion visual Transformer module is used to learn deep representations of feature vectors and fuse shallow textures with deep semantic information to obtain feature matrices of the three modalities. The dynamic weighted feature fusion module is used to introduce learnable modal perception weights based on the feature matrix, calculate the weight features corresponding to each modality, uniformly scale the normalized weight features, and fuse them to obtain multi-modal fusion features with modality selectivity; The face classification module is used to classify faces according to the multimodal fusion features to obtain real and false face classifications.

9. An electronic device, characterized in that: The electronic device includes: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the multimodal face liveness detection method described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium, characterized in that The non-transitory computer-readable storage medium stores computer instructions, which are used to enable the computer to execute the multimodal face liveness detection method described in any one of claims 1 to 7.

Images