Multimodal face presentation attack detection method based on spatio-temporal decomposition encoder

Abstract

The application discloses a kind of multi-modal face presentation attack detection methods based on space-time decomposition encoder, comprising:1, obtain the sample in multi-modal video dataset and generate feature representation;2, construct feature converter to generate space embedding representation, construct space-time decomposition encoder to extract single mode feature, construct fusion encoder to fuse multi-modal feature, input into classifier to obtain final classification result;3, construct centralized contrast loss and binary cross-entropy loss, train using optimizer and update model parameters;4, input to be tested video tests model, ensure that model can effectively distinguish real face and presentation attack face.The application can improve the accuracy and robustness of face presentation attack detection, so as to ensure the security of face recognition system.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision, and particularly relates to a multi-modal face presentation attack detection method based on a space-time decomposition encoder. BACKGROUND

[0002] Face recognition technology has been widely researched and applied, and attack methods against face recognition systems have become endless. In order to ensure the security of the face recognition system, developing an effective face presentation attack detection technology is crucial to improve the overall quality of the face recognition technology.

[0003] In the face of the anti-counterfeiting needs of the face recognition system, researchers have proposed various face presentation attack detection algorithms, such as texture-based methods, motion-based methods, and fraud trace separation-based methods.

[0004] Most current researches only target a single modality and use convolutional neural network-based methods. However, these methods generally only perform well in 2D attack cases, and their performance is severely reduced in complex 3D attack or local attack cases. Even in 2D attack cases, they cannot effectively generalize to unseen attacks and environments. SUMMARY

[0005] The application overcomes the deficiencies of the prior art and proposes a multi-modal face presentation attack detection method based on a space-time decomposition encoder, so as to improve the accuracy and robustness of face presentation attack detection and ensure the security of the face recognition system.

[0006] The application adopts the following technical solutions to solve the technical problems:

[0007] The multi-modal face presentation attack detection method based on the space-time decomposition encoder has the following steps:

[0008] Step 1, obtaining video samples in a multi-modal video dataset and generating feature representations;

[0009] Step 1.1, denoting any one video sample in the multi-modal video dataset as E={V m |m=1,2,3},wherein V m represents the video of the mth modality, when m=1, V 1 represents a visible light video, when m=2, V 2 represents a depth video, and when m=3, V 3 represents an infrared video; let the true class of the video sample E be denoted as When , it indicates that the video sample E is a real face video, and when When E represents a video of an attacking face, it indicates that the video sample is E.

[0010] Step 1.2, V m Split into U-segment videos, i.e. in, V represents the m-th modal video. m The uth video segment;

[0011] The u-th video segment of the m-th modality Generate feature representations in, Feature representation The k-th characteristic matrix in the matrix is ​​given by H, where H is the number of rows in the characteristic matrix, W is the number of columns in the characteristic matrix, and K is the number of characteristic matrices.

[0012] Step 2: Construct a spatiotemporal decomposition encoder network, including: a feature converter, 3×U spatial encoders, 3 temporal encoders, 1 fusion encoder, and a classifier;

[0013] Step 2.1: The feature converter represents the features. Processing is performed to obtain Corresponding spatial embedding representation

[0014] Step 2.2: The u-th spatial encoder pair corresponding to the m-th modal video. The process is performed to obtain the spatial encoded sequence output by the u-th spatial encoder. This results in the U spatial encoded sequences output by the U spatial encoders, which together form the time vector sequence corresponding to the m-th mode. L1 represents the number of layers in the spatial encoder;

[0015] Step 2.3, the time encoder for Zt m The process is performed to obtain the time-coded sequence of the m-th mode output by the time encoder. m = 1, 2, 3; L2 represents the number of layers in the time encoder;

[0016] Step 2.4, the fusion encoder... The process is performed to obtain the fused encoded sequence output by the fused encoder. Where m = 1, 2, 3, This represents the fused encoder sequence output by the fused encoder corresponding to the m-th mode; when m = 0, The fusion coding sequence represents the fusion bottleneck; and will The fusion classification encoding vector corresponding to the fusion classification vector clsf is denoted as L3 indicates the number of layers in the fusion encoder;

[0017] Step 2.5, the classifier processes clso by using a full connection layer, thereby calculating the probability P of the positive classification and the negative classification corresponding to the video sample E by using formula (9):

[0018] P = sigmoid (clso x ew + b) (9)

[0019] In formula (9), is a classification weight matrix, and b is a bias vector with a dimension of 2;

[0020] The classifier obtains the final classification result G of the video sample E by using formula (10):

[0021]

[0022] In formula (10), pf1 represents the probability of the positive classification in P, G = 1 represents a positive classification result, that is, a real face class; G = 0 represents a negative classification result, that is, a presented attack face class; and σ represents a threshold value, and σ ∈ (0, 1);

[0023] Step 3, constructing a loss function L of the spatial decomposition encoder network total :

[0024] Step 3.1, constructing a binary cross-entropy loss L by using formula (11) BCE :

[0025] L BCE = -(G x log (pf1) + (1-G) x log (1-pf1)) (11)

[0026] Step 3.2, constructing a centralized contrast loss L CCL :

[0027] The centralized contrast loss corresponding to the a th encoder is constructed by formula (12) and formula (13)

[0028]

[0029]

[0030] In formula (12), a = 1, 2,..., B, B represents the total number of encoders, and B = 3 x U + 3 + 1; β a represents the coefficient of the a th encoder, and β a ∈ (0, 1), C a represents the center vector of the current iteration, and the center vector is initialized as a zero vector in the first iteration; D a is the classification encoding vector clsa aand the center vector C of the current iteration a is the Euclidean distance of the center vector C of the current iteration and C a represents the a-th boundary radius;

[0031] The center vector C of the next iteration is updated by using formula (14) on C a a :

[0032]

[0033] In formula (14), alpha represents a coefficient, and alpha is in (0, 1); represents the classification vector of the v-th video sample whose real category is 1 in the a-th encoder output in the current iteration, and V represents the total number of video samples whose real category is 1 in the current iteration;

[0034] Step 3.3, the total loss function L is obtained by using formula (15) total :

[0035]

[0036] In formula (15), gamma and epsilon represent two coefficients, and gamma and epsilon are in (0, 1);

[0037] Step 3.4, the spatio-temporal decomposition encoder network is iteratively trained by using an optimizer, and the total loss function L total is calculated to update the network parameters until the loss function L total converges, so that the spatio-temporal decomposition encoder model with optimal parameters is obtained, which is used for detecting the multi-modal face video and distinguishing the real face and the presented attack face.

[0038] The multi-modal face presented attack detection method based on the spatio-temporal decomposition encoder has the characteristics that the step 2.1 comprises the following steps:

[0039] Step 2.1.1, HxW feature values in the k-th feature matrix are sequentially represented as HxW feature points, that is wherein, represents the i-th feature point in the k-th feature matrix , and the i-th feature point on the transposed matrix of is recorded as

[0040] Let the k-th feature matrix corresponding to the k-th weight matrix be recorded as ​​​​the i-th weight in the i-th row of the j-th column of the matrix W and is randomly initialized;

[0041] the k-th feature matrix is obtained by using formula (1) the k-th normalized relative attention matrix the i-th attention value in the i-th row of the j-th column of the matrix A so that the K normalized relative attention matrices are

[0042]

[0043] Step 2.1.2, using a convolution kernel with a size of Kx1x1 to perform point-by-point convolution calculation on After the point-by-point convolution calculation, the following is obtained the corresponding aggregation matrix

[0044] Step 2.1.3, the feature converter uses n d convolution kernels with a size of 1xdxd to perform convolution calculation with a step size of d on and the convolution result is flattened to obtain a spatial vector sequence with a length of and a dimension of n d wherein, is the g-th vector in the spatial vector sequence with a dimension of n d

[0045] The feature converter uses formula (2) to perform position encoding on the vector sequence to obtain the corresponding spatial embedding representation

[0046]

[0047] In formula (2), is an n d dimensional spatial classification vector to be learned; is a vector sequence with a length of n l +1 and a dimension of n d to be learned, representing the spatial position embedding of the video segment

[0048] Each of the spatial encoders in the step 2.2 is composed of L1 Transformer layers, and each Transformer layer includes a multi-head self-attention sublayer and a spatial perception machine sublayer; and Zt is obtained according to the following steps m :

[0049] ​​​​Step 2.2.1: Define the current Transformer layer as the lth layer. s Layer, and initialize l s =1;

[0050] definition The lth element in the corresponding spatial encoder s The input of the layer is And As the l s -1 layer output, initialization

[0051] Step 2.2.2: The spatial encoder obtains the lth... s Multi-head self-attention sublayer output multi-head self-attention

[0052]

[0053] In equation (3), LN() represents computational layer normalization, and MSA() represents computational multi-head self-attention;

[0054] Step 2.2.3: The spatial encoder obtains the lth... s The output of the spatial perception sublayer in the layer is used as the lth layer. s +1 layer of multi-head self-attention sublayer input Thus, the output of the L1 spatial perception sublayer is obtained. This serves as the u-th spatially encoded sequence corresponding to the m-th modality video of the spatial encoder; the U spatially encoded sequences output by the spatial encoder are then combined to form the time vector sequence corresponding to the m-th modality. Its total length is U×(n) l +1), with dimension n d ;

[0055]

[0056] In equation (4), MLP() is a multilayer perceptron.

[0057] In step 2.3, any one of the time encoders consists of L2 Transformer layers, and is obtained according to the following steps.

[0058] Step 2.3.1, to Zt m Add an nth mode to be learned d 3D time classification vector Where Avg() represents the mean calculation;

[0059] Using equation (5) to analyze Ztm Position encoding is performed to obtain the temporal embedding representation O of the first-level input of the temporal encoder corresponding to the m-th mode. m，1 :

[0060]

[0061] In equation (5), Qt m For a length of U×(n) l +1)+1, with dimension n d The vector sequence representing video V m Timing information;

[0062] Step 2.3.2, the time encoder for O m，1 After performing L2 layer spatial coding, the time-coded sequence output by the time encoder is obtained. m = 1, 2, 3.

[0063] The fusion encoder in step 2.4 is constructed based on multi-head cross-attention and fusion bottleneck. Each layer of the fusion encoder contains a multi-head cross-attention sublayer and a fusion perceptron sublayer, and is obtained according to the following steps.

[0064] Step 2.4.1: The input and output of each layer of the fusion encoder consist of four parts, denoted as the fusion embedding representation. For the lth in the fusion encoder f The input of layer l is also the lth layer. f -1 layer output;

[0065] When m = 1, 2, 3 Representing the lth f The input of the m-th mode of the layer; when m = 0, This indicates a length of U×(n) l +1)+1, with dimension n d Integration bottlenecks;

[0066] Initialize l f =1; When m=1, 2, 3, initialize the input of the m-th mode of the l-th layer of the fusion encoder. When m = 0, initialize the input f of the fusion bottleneck of the first layer of the fusion encoder. 0，1 =Avg({f m，1 |m=1,2,3}), and denote f 0，1 The fusion classification vector clsf = Avg({clst) m |m=1,2,3};

[0067] Step 2.4.2, the fusion encoder utilizes formula (6) and formula (7) to obtain the fusion encoding sequence of the fusion perception machine sublayer output of the lth layer f Multi-head cross attention sublayer output of the layer

[0068]

[0069]

[0070] In formula (7), MCA represents calculating multi-head cross attention, represents the corresponding query matrix, represents the corresponding key matrix, represents the corresponding value matrix, and Attention() represents calculating attention.

[0071] Step 2.4.3, the fusion encoder utilizes formula (8) to calculate the fusion encoding sequence of the fusion perception machine sublayer output of the lth layer f Thus, the fusion encoding sequence of the fusion perception machine sublayer output of the L3 layer is obtained And the fusion classification vector clsf in formula (9) is output as a fusion classification encoding vector by the fusion encoder, and the fusion classification encoding vector is denoted as

[0072]

[0073] The electronic device comprises a memory and a processor, and the memory is configured to store a program supporting the processor to execute any of the multi-modal face presentation attack detection methods, and the processor is configured to execute the program stored in the memory.

[0074] The computer readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the multi-modal face presentation attack detection methods are executed.

[0075] Compared with the prior art, the present application has the following advantages:

[0076] 1. The feature converter and the space-time decomposition encoder are constructed to extract single-modal features, the different parts of the input can be selectively focused according to the correlation, the global receptive field and the adaptive weight are possessed, the short-range and long-range dependencies are established between multiple frames, the face fine-grained features are fully extracted, and thus the features of the real face and the face of the presentation attack are effectively distinguished in a single mode.

[0077] ​​2、The application constructs a fusion encoder based on multi-head cross attention and fusion bottleneck, limits the interaction of information between each modality in the fusion bottleneck, fuses heterogeneous information with semantic correlation and content complementarity of multiple modalities, thereby improving the accuracy and robustness of face presentation attack detection in a complex scene;

[0078] 3、The application introduces a centralized contrast loss, forces real faces to be concentrated towards the center, and false faces to be far away from the center, thereby better distinguishing real faces from false faces. BRIEF DESCRIPTION OF DRAWINGS

[0079] Figure 1 The overall flowchart of the application is shown in the figure;

[0080] Figure 2 The structure diagram of the network in the application is shown in the figure. DETAILED DESCRIPTION

[0081] In this embodiment, a multi-modal face presentation attack detection method based on a space-time decomposition encoder is to acquire samples in a multi-modal video dataset and generate feature representations; then construct a feature converter and a space-time decomposition encoder to extract single-modal features, construct a fusion encoder based on multi-head cross attention and a fusion bottleneck to extract multi-modal features; combine a centralized contrast loss and a binary cross-entropy loss, train and optimize the model parameters using a Lion optimizer; finally, input the video to be tested to test the model, to ensure that the model can effectively distinguish real faces from presentation attack faces; specifically, referring to Figure 1 , the following steps are taken:

[0082] Step 1, acquire video samples in a multi-modal video dataset and generate feature representations;

[0083] Step 1.1, let any one video sample E in the multi-modal video dataset CASIA-SURF be E = {V m |m = 1, 2, 3}, wherein V m represents the video of the mth modality, when m = 1, V 1 represents a visible light video, when m = 2, V 2 represents a depth video, and when m = 3, V 3 represents an infrared video; let the real class of the video sample E be Y When Y = 1, it means that the video sample E is a real face video, and when Y = 0, it means that the video sample E is a false face video. ​The video sample E represents a presentation attack face video; each experimenter in the CASIA-SURF dataset includes 1 real video sample and 6 false video samples under different presentation attack modes, and the six presentation attack modes include removing eye, nose, mouth area and combination of removing area; each modality video is cropped and aligned, and the video resolution is HxW;

[0084] Step 1.2, V m is split into U segments of video, that is wherein, represents the u-th segment of video V m of the m-th modality video;

[0085] Let the video image be the b-th frame image in the video , and c m be the channel number of the image , then the total channel number of the video is Bx c m , from which K channels are uniformly selected, and the intensity values of HxW pixel points on each channel are regarded as the feature values of each feature point on the two-dimensional feature matrix; that is, the u-th segment of video V generates a feature representation wherein, represents the k-th feature matrix in the feature representation , H is the row number of the feature matrix, W is the column number of the feature matrix; K is the number of feature matrices; in this embodiment, U=10, H=640, W=480, K=18;

[0086] Step 2, as shown in Figure 2 , a space-time decomposition encoder network is constructed, including a feature converter, 3xU spatial encoders, 3 temporal encoders, 1 fusion encoder and a classifier; the spatial encoders and temporal encoders in the space-time decomposition encoder network extract single-modality features, and the fusion encoder fuses multi-modality features;

[0087] Step 2.1, the feature converter generates a spatial embedding representation from the feature representation, and the processing of the feature converter is as follows:

[0088] Step 2.1.1, K normalized relative attention matrices are generated from the feature representation, and the HxW feature values in the k-th feature matrix are sequentially represented as HxW feature points, that is wherein, represents the i-th feature point in the k-th feature matrix , and let be the corresponding i-th feature point on the transpose matrix of .

[0089] the kth feature matrix is denoted as the kth weight matrix corresponding to the kth feature matrix is denoted as denoted as the ith weight in the kth weight matrix is denoted as

[0090] the kth feature matrix is obtained by using formula (1) the kth normalized relative attention matrix corresponding to the kth feature matrix is denoted as the ith attention value in the kth normalized relative attention matrix is denoted as the K normalized relative attention matrices are obtained as

[0091]

[0092] Step 2.1.2, generate an aggregation matrix from the K normalized relative attention matrices, use a convolution kernel with a size of Kx1x1 to perform point-by-point convolution calculation on after the point-by-point convolution calculation, obtain the corresponding aggregation matrix

[0093] Step 2.1.3, generate a spatial embedding representation from the aggregation matrix, the feature converter further uses n d convolution kernels with a size of 1xdxd to perform convolution calculation on with a step size of d, and after flattening the convolution result, obtain a spatial vector sequence with a length of and a dimension of n d wherein the gth vector in the spatial vector sequence has a dimension of n d

[0094] The feature converter performs position encoding on the vector sequence using formula (2) to obtain the corresponding spatial embedding representation

[0095]

[0096] In formula (2), is an n d dimensional spatial classification vector to be learned; is a vector sequence to be learned with a length of n l +1 and a dimension of n d , representing the spatial position embedding of the video segment

[0097] ​​​​Step 2.2: Construct a spatial encoder consisting of L1 Transformer layers, each Transformer layer containing a multi-head self-attention sublayer and a spatial perception sublayer;

[0098] Step 2.2.1: Define the current Transformer layer as the lth layer. s Layer, and initialize l s =1;

[0099] definition The lth element in the corresponding spatial encoder s The input of the layer is And As the l s -1 layer output, initialization

[0100] Step 2.2.2: The spatial encoder uses equation (3) to obtain the lth... s Multi-head self-attention sublayer output multi-head self-attention

[0101]

[0102] In equation (3), LN() represents computational layer normalization, and MSA() represents computational multi-head self-attention;

[0103] Step 2.2.3: The spatial encoder uses equation (4) to obtain the lth... s The output of the spatial perception sublayer in the layer is used as the lth layer. s +1 layer of multi-head self-attention sublayer input Thus, the output of the L1 spatial perception sublayer is obtained. This serves as the u-th spatially encoded sequence corresponding to the m-th modality video of the spatial encoder; the U spatially encoded sequences output by the spatial encoder are then combined to form the time vector sequence corresponding to the m-th modality. Its total length is U×(n) l +1), with dimension n d ;

[0104]

[0105] In equation (4), MLP() is a multilayer perceptron; in this embodiment, L1 = 6;

[0106] Step 2.3: Construct a time encoder consisting of L2 Transformer layers;

[0107] Step 2.3.1, to Zt m Add an nth mode to be learned dTemporal classification vector where Avg() denotes the mean calculation;

[0108] The Zt m is positionally encoded to obtain the time embedding representation O m，1 of the first layer input of the time encoder corresponding to the m-th modality

[0109]

[0110] In formula (5), Qt m is a vector sequence with a length of Ux(n l +1)+1 and a dimension of n d , representing the time sequence information of the video V m ;

[0111] Step 2.3.2, after the time encoder performs spatial encoding on O m，1 in the L2 layer, the time encoding sequence output by the time encoder is obtained m=1, 2, 3; in this embodiment, L2=12;

[0112] Step 2.4, a fusion encoder based on multi-head cross attention and fusion bottleneck is constructed, and each layer of the fusion encoder includes a multi-head cross attention sublayer and a fusion perception machine sublayer;

[0113] Step 2.4.1, the input and output of each layer of the fusion encoder are composed of four parts, denoted as fusion embedding representation is the input of the l f -th layer in the fusion encoder, and is also the output of the l f -1-th layer;

[0114] When m=1, 2, 3, represents the input of the m-th modality of the l f -th layer; when m=0, is a fusion bottleneck with a length of Ux(n l +1)+1 and a dimension of n d ; the information interaction between the modalities is limited in the fusion bottleneck f 0，l ;

[0115] Initialize l f =1; when m=1, 2, 3, initialize the input of the m-th modality of the first layer of the fusion encoder When m=0, initialize the input f 0，1 of the fusion bottleneck of the first layer of the fusion encoder m，1| m = 1, 2, 3}), and let f 0，1 the fusion classification vector clsf = Avg({clst m | m = 1, 2, 3}) after encoding by the fusion encoder, for the final classification.

[0116] Step 2.4.2, the fusion encoder obtains the fusion encoding sequence of the lth f layer by using formula (6) and formula (7).

[0117]

[0118]

[0119] In formula (7), MCA represents calculating the multi-head cross attention, represents the corresponding query matrix, represents the corresponding key matrix, represents the corresponding value matrix, and Attention() represents calculating the attention.

[0120] Step 2.4.3, the fusion encoder calculates the fusion encoding sequence of the lth f layer by using formula (8). Thus, the fusion encoding sequence of the fusion perception sublayer output of the L3 layer is obtained and the fusion classification encoding vector corresponding to the fusion classification vector clsf in formula (8) is denoted as

[0121]

[0122] In this embodiment, L3 = 6.

[0123] Step 2.5, the classifier processes clso by using a fully connected layer, so as to calculate the probability P of the positive classification and the negative classification of the video sample E by using formula (9):

[0124] P = sigmoid(clso x ew + b) (9)

[0125] In formula (9), is a classification weight matrix, and b is a bias vector with a dimension of 2.

[0126] The classifier obtains the final classification result G of the video sample E by using formula (10):

[0127] ​

[0128] In formula (10), pf1 represents the probability of the positive classification in P, G = 1 represents the positive classification result, i.e. the real face class; G = 0 represents the negative classification result, i.e. the attack face class; and σ represents a threshold, and σ ∈ (0, 1);

[0129] Step 3, constructing a loss function L of the spatial decomposition encoder network total :

[0130] Step 3.1, constructing a binary cross-entropy loss L by using formula (11) BCE :

[0131] L BCE = -(G x log(pf1) + (1 - G) x log(1 - pf1)) (11)

[0132] Step 3.2, constructing a centralized contrast loss L CCL :

[0133] The centralized contrast loss corresponding to the a-th encoder is constructed by formula (12) and formula (13)

[0134]

[0135]

[0136] In formula (12), a = 1, 2,..., B, B represents the total number of encoders, and B = 3 x U + 3 + 1; β a represents the coefficient of the a-th encoder, and β a ∈ (0, 1), C a represents the center vector of the current iteration, and the center vector is initialized as a zero vector at the first iteration; D a is the Euclidean distance between the classification encoding vector clsa a output by the a-th encoder and the center vector C a of the current iteration, and is calculated by formula (13); bd a represents the a-th boundary radius; if the distance D a exceeds the boundary bd a , the current sample will not be calculated for the loss; the centralized contrast loss makes the real face gather to the center, and the false face is far away from the center;

[0137] The center vector C a is iteratively updated by using formula (14) to obtain the center vector C' a :

[0138]

[0139] In equation (14), α represents a coefficient, and α∈(0,1); Avg() represents the classification vector output by the encoder of the v-th video sample with a true class of 1 in the current iteration, where V represents the total number of video samples with a true class of 1 in the current iteration; Avg() calculates the average difference between the true face video and the center vector of the current iteration.

[0140] Step 3.3: Use equation (15) to obtain the total loss function L. total :

[0141]

[0142] In equation (15), γ and ε represent two coefficients, and γ and ε ∈ (0, 1);

[0143] Step 3.4: Iteratively train the spatiotemporal decomposition encoder network using the Lion optimizer based on program search, and calculate the total loss function L. total To update the network parameters until the loss function L is reached. total The spatiotemporal decomposition encoder model with optimal parameters is obtained by converging until it is found. This model is used to detect multimodal face videos and distinguish between real faces and faces that are presented as attacks.

[0144] Step 4: Input the test video to test the model and ensure that the model can effectively distinguish between real faces and faces that are presented as attacks.

[0145] Step 4.1: Input attack face videos that the model has not used during training for testing to ensure that the model can effectively defend against unknown attack attacks;

[0146] Step 4.2: Input real face videos collected from real-world scenes for testing to ensure that the model's false alarm probability is not too high;

[0147] Step 4.3: If the model test is normal, it can be launched online; otherwise, the model parameters should be readjusted.

[0148] In this embodiment, an electronic device includes a memory and a processor. The memory stores a program that supports the processor in executing the above-described method, and the processor is configured to execute the program stored in the memory.

[0149] In this embodiment, a computer-readable storage medium stores a computer program, which is executed by a processor to perform the steps of the above method.

Claims

1. A method for detecting multi-modal face presentation attack based on spatio-temporal decomposition encoder, characterized in that, The method comprises the following steps: Step 1, obtaining video samples in a multi-modal video dataset and generating feature representations; Step 1.

1. Let any one video sample in the multi-modal video dataset be E = {V m | m = 1, 2, 3}, where V m represents the video of the m-th modality, when m = 1, V 1 represents the visible light video, when m = 2, V 2 represents the depth video, when m = 3, V 3 represents the infrared video; let the real class of the video sample E be denoted as when , it means that the video sample E is the real face video, when , it means that the video sample E is the presentation attack face video; Step 1.2, V m split into u-segment videos, i.e. wherein, represents the u-segment video of the m-th modality video V m m. from the u-th segment of the m-th modality video generating a feature representation wherein, representing the feature representation the k-th feature matrix in the feature representation, H is the number of rows of the feature matrix, W is the number of columns of the feature matrix; K is the number of feature matrices; Step 2, constructing a space-time decomposition encoder network, comprising: a feature converter, 3xU spatial encoders, 3 temporal encoders, 1 fusion encoder and a classifier; Step 2.

1. The feature converter processes the feature representation to obtain a corresponding spatial embedding representation Step 2.2, the u-th spatial encoder pair corresponding to the m-th modality video processing to obtain the spatial encoding sequence output by the u-th spatial encoder Thus, the U spatial encoding sequences output by the U spatial encoders form a time vector sequence corresponding to the m-th modality L1 represents the number of layers of the spatial encoder Step 2.

3. The time encoder processes Zt m to obtain the time encoding sequence of the m-th modality output by the time encoder m = 1, 2, 3; L2 represents the number of layers of the time encoder Step 2.4, the fusion encoder pair is processed to obtain a fusion encoding sequence of the fusion encoder output wherein, when m = 1, 2, 3, represents a fusion encoding sequence of the fusion encoder output corresponding to the mth modality; when m = 0, represents a fusion encoding sequence of the fusion bottleneck; and the fusion classification encoding vector corresponding to the fusion classification vector clsf in the fusion classification encoding vector L3 represents the number of layers of the fusion encoder; Step 2.5, the classifier processes clso by using a full connection layer, thereby calculating the probability P of the positive classification and the negative classification of the video sample E by using formula (9): P = sigmoid(clso x ew + b) (9) In formula (9), is a classification weight matrix, and b is a bias vector with a dimension of 2. The classifier obtains the final classification result G of the video sample E by using formula (10): In formula (10), pf1 represents the probability of the positive classification in P, G = 1 represents the positive classification result, i.e. the real face category; G = 0 represents the negative classification result, i.e. the attack face category; and σ represents a threshold, and σ ∈ (0, 1); Step 3, Constructing the loss function L for the spatial decomposition encoder network total : Step 3.1, Constructing binary cross-entropy loss L with formula (11) BCE : L BCE = -(G x log(pf1) + (1 - G) x log(1 - pf1)) (11) Step 3.2, Building the contrastive loss L CCL : The centralized contrastive loss corresponding to the a-th encoder is constructed by formula (12) and formula (13) as In equation (12), a = 1, 2, ..., A, where A represents the total number of encoders, and A = 3 × U + 3 + 1; β a Let β represent the coefficients of the a-th encoder, and let β a ∈(0,1), C a D represents the center vector of the current iteration, and is initialized to zero during the first iteration; a Let clsa be the classification encoding vector output by E in the a-th encoder. a and the center vector C of the current iteration a The Euclidean distance is calculated using equation (13); bd a This represents the radius of the a-th boundary. The center vector C is updated using equation (14) a to obtain the center vector C' for the next iteration a : In formula (14), a represents a coefficient, and a e (0, 1); represents the classification vector of the vth video sample whose real category is 1 in the current iteration at the a th encoder output, V represents the total number of video samples whose real category is 1 in the current iteration; Step 3.3, the total loss function L is obtained using formula (15) total : In formula (15), γ and ε represent two coefficients, and γ, ε ∈ (0, 1); Step 3.

4. Iteratively train the spatio-temporal decomposition encoder network using the optimizer and compute the total loss function L total to update the network parameters until the loss function L total converges, thereby obtaining the spatio-temporal decomposition encoder model with optimal parameters for detecting multi-modal face videos and distinguishing real faces from presentation attack faces.

2. The method of claim 1, wherein the method is based on a spatio-temporal decomposition encoder. The step 2.1 comprises the following steps: Step 2.1.1, k-th feature matrix The HxW feature values are sequentially represented as HxW feature points, i.e. wherein, denotes the i-th feature point in the k-th feature matrix ; and let denote the i-th feature point on the transpose matrix of ; Let the kth feature matrix be denoted as The corresponding kth weight matrix is denoted as Let be the ith weight in the kth feature matrix and is randomly initialized; be the ith weight in the kth feature matrix and is randomly initialized; The kth feature matrix is obtained using formula (1) The kth normalized relative attention matrix is obtained The ith attention value in the matrix The K normalized relative attention matrices are obtained Step 2.1.2, using a convolution kernel of size K x 1 x 1 After point-wise convolution calculation, we get The corresponding aggregation matrix Step 2.1.3, the feature converter reuses n d convolution kernels of size 1 x d x d, performs convolution calculation with a step size of d on , and obtains a spatial vector sequence of length and dimension n d after flattening the convolution result wherein is the gth vector in the spatial vector sequence of dimension n d . The feature transformer utilizes equation (2) to positionally encode a sequence of vectors to obtain a corresponding spatial embedding representation In equation (2), n to be learned d 3D classification vector; For a length of n l +1, dimension n d The sequence of vectors to be learned, representing video segments. Spatial location embedding.

3. The method of claim 2, wherein the method is based on a spatio-temporal decomposition encoder. Any one of the spatial encoders in step 2.2 is composed of L1 Transformer layers, each of which contains a multi-head self-attention sub-layer and a spatial perception machine sub-layer; and Zt is obtained as follows m : Step 2.2.1, define the current Transformer layer as the l s th layer and initialize l s = 1; definition The lth element in the corresponding spatial encoder s The input of the layer is And As the l s -1 layer output, initialization Step 2.2.

2. The spatial encoder utilizes equation (3) to obtain the 1th s Multi-head self-attention sub-layer output in the layer In formula (3), LN() represents calculating layer normalization, and MSA() represents calculating multi-head self-attention; Step 2.2.3, the spatial encoder utilizes formula (4) to obtain the output of the lth s layer spatial perception machine sublayer and as the input of the lth s layer multi-head self-attention sublayer layer spatial perception machine sublayer and as the spatial encoding sequence of the uth output corresponding to the mth modality video of the spatial encoder; the U spatial encoding sequences output by the spatial encoder form a time vector sequence corresponding to the mth modality , the total length of which is U x (n l +1), and the dimension is n d ; In formula (4), MLP() is a multi-layer perceptron.

4. The method of claim 3, wherein the method is based on a spatio-temporal decomposition encoder. Any one of the time encoders in step 2.3 is composed of L2transformer layers and is obtained by the following steps Step 2.3.1, adding an nth modality to be learned to Zt m n-dimensional time classification vector of the mth modality to be learned d n-dimensional time classification vector of the mth modality to be learned where Avg() denotes the mean calculation; Using equation (5) to analyze Zt m Position encoding is performed to obtain the temporal embedding representation O of the first-level input of the temporal encoder corresponding to the m-th mode. m,1 : In formula (5), Qt m is a vector sequence with length U×(n l +1)+1 and dimension n d , representing the timing information of video V m . Step 2.3.2, the time encoder encodes O m,1 After spatial encoding of the L2 layer, the time encoding sequence output by the time encoder is obtained m = 1, 2, 3.

5. The method of claim 3, wherein the method is based on a spatio-temporal decomposition encoder. The fusion encoder of step 2.4 is constructed based on multi-head cross-attention and fusion bottleneck, each layer of the fusion encoder contains a multi-head cross-attention sublayer and a fusion perception machine sublayer, and is obtained as follows Step 2.4.

1. The input and output of each layer of the fusion encoder consists of 4 parts, denoted as is the input of the l-th layer of the fusion encoder, which is also the output of the l-1-th layer of the fusion encoder. f f is the input of the l-th layer of the fusion encoder, which is also the output of the l-1-th layer of the fusion encoder.​ when m = 1, 2, 3, represents the lth f input of the mth modality of the lth layer; when m = 0, denotes a fusion bottleneck of length U x (n l + 1) + 1 and dimension n d ; Initialize l f = 1; initialize input of the m-th modality of the 1st layer of the fusion encoder when m = 1, 2, 3 Initialize input f of the fusion bottleneck of the 1st layer of the fusion encoder when m = 0 0,1 = Avg({f m,1 |m = 1, 2, 3}) and let f 0,1 = Avg({f m |m = 1, 2, 3}) the fusion classification vector clsf in f Step 2.4.2, the fusion encoder utilizes equation (6) and equation (7) to obtain the lth f Multi-head cross-attention sub-layer output of the lth layer In formula (7), MCA represents calculating multi-head cross attention, representing a corresponding query matrix, representing a corresponding key matrix, representing a corresponding value matrix, Attention() represents calculating attention; Step 2.4.

3. The fusion encoder calculates the fusion encoding sequence of the fusion perceptual machine sub-layer output of the l-th layer using formula (8) f Step 2.4.

3. The fusion encoder calculates the fusion encoding sequence of the fusion perceptual machine sub-layer output of the l-th layer using formula (8) Step 2.4.

3. The fusion encoder calculates the fusion encoding sequence of the fusion perceptual machine sub-layer output of the l-th layer using formula (8) Step 2.4.

3. The fusion encoder calculates the fusion encoding sequence of the fusion perceptual machine sub-layer output of the l-th layer using formula (8) The fusion classification encoding vector output by the fusion encoder in the fusion classification vector clsf is denoted as 6. An electronic device comprising a memory and a processor, characterized in that The memory is used to store a program supporting the processor to execute the multi-modal face presentation attack detection method according to any one of claims 1-5, and the processor is configured to execute the program stored in the memory.

7. A computer-readable storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to execute the steps of the multi-modal face presentation attack detection method according to any one of claims 1-5.

Images