Living body detection method and system based on feature channel representation

Abstract

The application discloses a kind of based on feature channel representation's live body detection method and system, the method includes the following steps: video data set pre-processing, extract face image and set true and false label and domain label;Based on multi-scale feature extraction module, the multi-scale feature of face image is extracted;Based on adaptive feature augmentation module, the representation space of multi-scale feature is expanded to obtain the feature after augmentation;Based on channel level whitening module, the discriminative representation ability of channel feature is improved;Based on channel level rejection module, the channel sensitive to domain bias is inhibited to extract domain invariant information;Based on attention feature fusion module, the feature output by channel level whitening module and channel level rejection module is fused to obtain the final discriminative feature;According to the final discriminative feature, binary classification result is predicted;The weight of network is trained and updated, and the live body detection prediction result is obtained based on the trained network model.The application can improve the discriminative representation ability of channel feature and the domain generalization performance of live body detection model.

Description

Technical Field

[0001] The present invention relates to a face liveness detection technology, and in particular to a liveness detection method and system based on feature channel representation. Background Art

[0002] Liveness detection technology, also known as Face Anti-Spoofing (FAS), is a technique used to authenticate and verify the authenticity of facial images captured by facial recognition systems. Due to its convenience and excellent recognition accuracy, facial recognition technology is widely used in many interactive intelligent applications, such as mobile payments, entry verification, phone unlocking, and attendance systems. However, facial recognition systems can be attacked through methods such as printed paper, video playback, 3D masks, or mannequins, threatening the user's privacy and security. Therefore, research on liveness detection technology is crucial to protect personal facial information from attacks.

[0003] Existing liveness detection methods mainly include traditional manual feature extraction methods and deep learning-based methods. The former usually uses operation trace descriptors (such as local binary patterns, gradient direction histograms, etc.) in various color spaces to extract deception clues. Such methods have low computational complexity and strong interpretability, but because they need to select different feature representations for different objects, they cannot fully exert their performance when facing unknown test scenarios; the latter does not require manual design of features or functions, but uses convolutional neural networks to encode the low-level features (edges, textures, etc.) of the analysis object layer by layer to adaptively obtain high-level semantic information of the image. It has stronger feature representation capabilities and has gradually become the mainstream method in the field of liveness detection.

[0004] In real-world applications, the diversity of camera sensors, environmental changes, and unpredictable attack types can all lead to domain shift, resulting in a difference in the distribution of the actual test set and the training set. Early deep learning methods did not fully account for the distribution differences between different domains. Therefore, such detection models were susceptible to domain shift, resulting in poor generalization. Summary of the Invention

[0005] In order to overcome the defects and deficiencies existing in the prior art, the application provides a living body detection method and system based on feature channel representation, which adopts a double-branch network architecture and takes multi-scale features as input, wherein a first branch utilizes an adaptive feature augmentation module, calculates the statistics of the input features, models the Gaussian distribution of the statistics, introduces perturbation, calculates the uncertainty of the statistics, sets the upper and lower limits to control the degree of perturbation, obtains two affine parameters, and uses the affine parameters to perform augmentation transformation on the features to expand the representation space of the features; based on the augmented features, a channel-level whitening module is used in the branch to decouple the channel features representing the true and false classes, reduce the confusion between the two types of features, and thus improve the discriminative representation ability of the channel features; a channel-level elimination module is introduced in the second branch of the network to remove the channels with domain bias and extract domain-invariant features, thereby improving the domain generalization ability of the detection model.

[0006] In order to achieve the above purpose, the application adopts the following technical scheme:

[0007] The application provides a living body detection method based on feature channel representation, comprising the following steps:

[0008] Video data set is video framed, and the face region of each frame image is extracted to obtain a face image, and the face image is set with corresponding true and false labels and domain labels;

[0009] A multi-scale feature extraction module is constructed to extract multi-scale features of the face image;

[0010] An adaptive feature augmentation module is constructed to expand the representation space of the multi-scale features, the feature statistics of the multi-scale features are calculated, the statistics are modeled as Gaussian distribution and perturbation is introduced, the uncertainty of the statistics is calculated, the upper and lower limits are set to control the degree of perturbation, two affine parameters are obtained, the multi-scale features are augmented and transformed to expand the representation space of the multi-scale features, and the augmented features are obtained;

[0011] A channel-level whitening module is constructed, the augmented features are input into the channel-level whitening module, the multi-channels of the augmented features are mapped into two channels for true and false discrimination by using a mapping layer, in the mapping process, according to the indexes of the mapping layer weights sorted by size, the channels that can best represent the true and false classes are sequentially divided into the same group, the covariance matrix is calculated in each group, and the covariance matrix is constrained by using an identity matrix, and the whitening loss is calculated;

[0012] A channel-level elimination module is constructed, the multi-scale features are input into the channel-level elimination module, the weighted activation values predicted as correct domains are calculated by using the weights of the fully connected layers in the domain discriminator, the domain bias of each channel is measured, the channels with large contribution to the domain discrimination task are eliminated, and domain-invariant features are learned;

[0013] An attention feature fusion module is constructed, and the features optimized by the channel-level whitening module and the domain-invariant features learned by the channel-level pruning module are fused based on a channel attention mechanism to obtain final discriminative features.

[0014] A classifier is constructed, and the final discriminative features are input into the classifier to obtain a binary classification prediction result.

[0015] The output results of the classifier and the output results of the domain discriminator are supervised by using the real and fake labels and the domain labels respectively to obtain a binary classification loss and a domain discrimination loss, and a total loss function is obtained by combining the whitening loss of the channel-level whitening module and weighting, and the network model weight coefficients are updated and trained.

[0016] The live body detection prediction result is obtained based on the trained network model.

[0017] As a preferred technical solution, a multi-scale feature extraction module is constructed to extract multi-scale features of the face image, specifically including:

[0018] The first three layers of ResNet-18 are used as an initial feature extractor for input face images in the multi-scale feature extraction module, and each layer feature of the input face image is extracted, and each level feature obtained is subjected to a scale transformation operation, and multi-scale features are obtained by linear addition.

[0019] As a preferred technical solution, each level feature obtained is subjected to a scale transformation operation, and multi-scale features are obtained by linear addition, specifically including:

[0020] The feature output by the first layer is subjected to a scale transformation operation and linearly added to the output of the second layer, and the processed feature is subjected to the same scale transformation operation and linearly added to the output of the third layer to obtain multi-scale features.

[0021] As a preferred technical solution, an adaptive feature augmentation module is constructed to expand the representation space of the multi-scale features, specifically including:

[0022] The mean and standard deviation of the multi-scale features in the channel dimension are calculated, specifically represented as:

[0023]

[0024]

[0025] Wherein, μ(f) represents the mean, σ(f) represents the standard deviation, H, W and c represent the height, width and channel of the sample feature map respectively, and f c,h,v represents the multi-scale features in the channel dimension;

[0026] The variances of μ(f) and σ(f) are calculated and denoted as and The formula is as follows:

[0027]

[0028]

[0029] Among them E n () indicates finding the mean of feature statistics;

[0030] use and Measure the uncertainty of μ(f) and σ(f) respectively, and use Gaussian distribution to model μ(f) and σ(f), and introduce the standard normal distribution Gaussian perturbation ∈ μ and ∈ σ , based on the uncertainty, the degree of Gaussian perturbation is controlled and two affine parameters are obtained, which are specifically expressed as:

[0031]

[0032]

[0033] Based on the affine parameters β and γ, the features of the simulated domain shift are generated, that is, the augmented features, which are specifically expressed as:

[0034]

[0035] in, represents the features of the simulated domain shift, and f represents the multi-scale features before augmentation.

[0036] As a preferred technical solution, the whitening loss is expressed as:

[0037]

[0038]

[0039]

[0040] Among them, L cw Represents whitening loss, N represents the number of input sample features, and C is the number of channels before the feature passes through the mapping layer. is the number of groups after grouping, Represents the nth sample feature F in a batch input n The gth group, I is the identity matrix, represent The covariance matrix of H and W represents the height and width of the sample feature map, respectively, and F n,i,h,w ,F n,j,h,w(i, j∈[1,2]) represent the channel dimension features representing true and false in each group of the nth sample features, μ n,i ,μ n,j (i,j∈[1,2]) corresponds to the channel mean, 1 and 2 represent the true and false categories respectively, i and j represent the channel index in the group, and the value range is [1,2].

[0041] As a preferred technical solution, the weighted activation value of the correct domain is predicted by using the weights of the fully connected layer in the domain discriminator, which is specifically expressed as:

[0042]

[0043] Among them, s c represents the weighted activation value, c is the channel index, represents the fully connected layer weight corresponding to the channel predicted as the correct domain, F m represents multi-scale features, and GAP represents the global average pooling layer.

[0044] As a preferred technical solution, the domain identification loss of the domain discriminator is expressed as:

[0045]

[0046] Among them, L d represents the domain discrimination loss, P is the number of source domains, n d represents the number of samples with domain label d in a batch input, j is the predicted domain label, Indicates that the value of this formula is 1 when the domain discriminator D predicts correctly, otherwise it is 0, F m Represents multi-scale features.

[0047] As a preferred technical solution, the features optimized by the channel-level whitening module and the domain-invariant features learned by the channel-level culling module are fused based on the channel attention mechanism to obtain the final discriminant features, which are specifically expressed as:

[0048]

[0049] Among them, X and Y represent the features output by the channel-level whitening module and the features output by the channel-level culling module, respectively. represents the addition of corresponding elements, Indicates the multiplication of corresponding elements, A() indicates obtaining the fusion weight based on the attention mechanism. The specific operations are as follows:

[0050]

[0051] Among them, σ is the Sigmoid activation function, L(X) and G(X) represent the channel attention of local features and global features respectively.

[0052] As a preferred technical solution, the final discriminant features are input into the classifier to obtain the binary classification prediction results, which specifically includes:

[0053] The predicted labels and predicted probability values ​​are calculated based on the output features of the classifier, and supervised training is performed based on the true and false labels to construct the binary classification loss, which is specifically expressed as:

[0054] L cls =-(ylog(y′)+(1-y)log(1-y′))

[0055] Among them, L cls represents the binary classification loss, y represents the true and false labels, and y′ represents the predicted labels.

[0056] The present invention also provides a liveness detection system based on feature channel representation, comprising: a data preprocessing module, a multi-scale feature extraction module, an adaptive feature augmentation module, a channel-level whitening module, a channel-level culling module, an attention feature fusion module, a classifier, a network training module, and a testing module;

[0057] The data preprocessing module divides the video data set into frames, extracts the face area of ​​each frame image to obtain a face image, and sets corresponding authenticity labels and domain labels for the face image;

[0058] The multi-scale feature extraction module is used to extract multi-scale features of facial images;

[0059] The adaptive feature augmentation module is used to expand the representation space of multi-scale features by calculating the feature statistics of the multi-scale features, modeling the Gaussian distribution of the features and introducing disturbances, calculating the uncertainty of the statistics, setting upper and lower limits to control the degree of disturbance, and obtaining two affine parameters for performing an augmentation transformation on the multi-scale features to expand their representation space and obtain augmented features;

[0060] The channel-level whitening module uses a mapping layer to map the multi-channels of the augmented features into two channels for true and false discrimination. During the mapping process, the channels that best represent the true and false categories are divided into the same group according to the index of the mapping layer weights after sorting by size. The covariance matrix is ​​calculated within each group and constrained by the identity matrix to calculate the whitening loss.

[0061] The channel-level elimination module is used to calculate the weighted activation value predicted as the correct domain based on the multi-scale features of the input using the weights of the fully connected layer in the domain discriminator, measure the domain tendency of each channel, eliminate channels that contribute greatly to the domain identification task, and learn domain-invariant features;

[0062] The attention feature fusion module is used to fuse the features optimized by the channel-level whitening module and the domain-invariant features learned by the channel-level culling module based on the channel attention mechanism to obtain the final discriminant features;

[0063] The classifier is used to obtain a binary classification prediction result based on the final discriminant features;

[0064] The network training module is used to supervise the output results of the classifier and the output results of the domain discriminator using the true and false labels and domain labels respectively, to obtain the binary classification loss and the domain discrimination loss, and to combine the whitening loss of the channel-level whitening module to obtain the total loss function, and to train and update the network model weight coefficient;

[0065] The testing module is used to obtain a liveness detection prediction result based on the trained network model.

[0066] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0067] (1) The present invention enhances the information transmission between different network layers by fusing the features of multiple layers in the neural network, and uses the multi-scale features of the information of each layer as the input of the subsequent modules. This choice can improve the representation ability of the model and more effectively play the role of each module. Specifically, for the adaptive feature augmentation module of the first branch, using multi-scale features as input can more effectively expand the representation space of features while retaining the powerful representation ability of the original features; for the channel-level elimination module of the second branch, since channels sensitive to domain offset may exist in both shallow and deep layers of the network, if the elimination module is used in multiple layers, too many features may be lost and hinder model learning. Therefore, the multi-scale features covering the information of multiple layers of the network are used as the input of the module, and the channels with domain bias in the features are directly eliminated, thereby effectively reducing the domain bias of the channels while avoiding affecting model training.

[0068] (2) The present invention adopts an adaptive feature augmentation module to calculate the statistics of the input features, model the Gaussian distribution and introduce disturbances, calculate the uncertainty of the statistics, set its upper and lower limits to control the degree of disturbance, and thus obtain two affine parameters for performing augmentation transformation to expand the representation space of the features; previous feature augmentation strategies mostly use linear interpolation, that is, the amplified results are only retained between the lines connecting the corresponding features of the two samples, which cannot guarantee sufficient coverage of the unknown domain; in addition, some methods may introduce excessive disturbances, and in the research field of liveness detection, the offset between different domains is not necessarily very large, so excessive disturbances are likely to be counterproductive. In contrast, the adaptive feature augmentation module proposed in the present invention is more suitable for application in liveness detection technology.

[0069] (3) The present invention simultaneously studies the correlation between discriminative feature channels and the channel's tendency towards domains. The proposed dual-branch network can not only achieve the task of true and false discrimination, but also suppress information that is sensitive to domain offset, thereby improving the domain generalization performance of the liveness detection model. Previous methods mostly conducted research at the domain or instance level, while the present invention studies the domain generalization problem of liveness detection at the channel level, which can more fundamentally analyze the factors that affect domain generalization when using convolutional neural networks for liveness detection.

[0070] (4) Cross-database experimental data prove that the dual-branch network proposed in this invention has good domain generalization performance and is superior to several representative algorithms in the field of liveness detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 Schematic diagram of the flow of the liveness detection method based on feature channel characterization of the present invention;

[0072] Figure 2 Schematic diagram of the network architecture for implementing the liveness detection method based on feature channel representation of the present invention;

[0073] Figure 3 Schematic diagram of the structure of the multi-scale feature extraction module of the present invention;

[0074] Figure 4 Schematic diagram of the implementation process of the adaptive feature augmentation module of the present invention;

[0075] Figure 5 Schematic diagram of the implementation process of the channel-level whitening module of the present invention;

[0076] Figure 6 Schematic diagram of the implementation process of the channel-level rejection module of the present invention;

[0077] Figure 7 Schematic diagram of the network structure of the attention feature fusion module of the present invention. DETAILED DESCRIPTION

[0078] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0079] Example 1

[0080] This embodiment takes the four liveness detection datasets of Oulu_NPU, CASIA-MFSD, MSU_MFSD and Replay-Attack as examples for training and testing, and introduces the implementation process in detail. The Oulu_NPU dataset collects real faces from 55 testers and the corresponding generated deceptive faces. Each person has 90 videos, of which 20 are used for training, 15 for validation, and 20 for testing. The attack methods include two printing attacks and two video replay attacks. The CASIA-MFSD dataset contains 600 videos, and uses three cameras with resolutions of 640×480 pixels, 480×640 pixels, and 1920×1080 pixels to collect real faces from 50 testers and the corresponding generated faces. The dataset consists of 280 videos, which collect real faces from 35 test subjects and the corresponding generated deceptive faces, of which 15 are used for training and 20 for testing. The dataset consists of 1200 videos, which collect real faces from 50 test subjects and the corresponding generated deceptive faces using a MacBook camera with a resolution of 320×240 pixels, and is divided into training, validation and test sets according to the ratio of 3:3:4.

[0081] This example runs on a Linux system and is implemented based on the deep learning framework Pytorch. The graphics card used in the experiment is GTX1080Ti, CUDA version is 10.1.105, and cudnn version is 7.6.4.

[0082] like Figure 1 、 Figure 2 As shown, this embodiment provides a method for liveness detection based on feature channel representation, adopting a dual-branch network architecture, and specifically including the following steps:

[0083] S1. Preprocess the input data: divide the dataset into a training set and a test set, divide the video in the dataset into frames, crop the face area in each frame, extract the face area in each frame to obtain a face image, and set the corresponding authenticity label y and domain label d for the face image;

[0084] In this embodiment, the OpenCV tool library is used to frame all videos in the training set and the test set, the MTCNN face recognition algorithm is used to detect the face area and obtain the face position frame, and after the face area is cropped, the Lanczos interpolation algorithm is used to adjust the resolution to H×W×C, where H, W, and C represent the height, width, and number of channels of the face image, respectively. In this embodiment, H=256, W=256, C=3, and the number of data in each batch N=8.

[0085] S2. Construct a multi-scale feature extraction module: Input the face image I into the initial feature extractor to obtain the features of each network depth. After performing scale transformation operations on the features at each level and adjusting the size of the feature maps at each level, they are linearly added to obtain a multi-scale feature F containing comprehensive information. m ;

[0086] In this embodiment, the multi-scale feature extraction module is used to extract multi-scale features of the input image, such as Figure 3 As shown in Figure 1, the multi-scale feature extraction module uses the first three layers of ResNet-18 as the initial feature extractor of the input face image I to extract the features of each layer of the input face image I. The semantic information of the features extracted at different depths in the network is different. The scale transformation operation is performed on the features at each level, and the multi-scale feature F is obtained by linear addition. m , so that the features at each level can prompt each other and supplement the discriminative information during the learning process.

[0087] The specific implementation method is as follows: for a batch of input data I (the number of data in each batch is recorded as N), the feature F1 output by the first layer is scaled and linearly added to the output F2 of the second layer, and the processed feature is also scaled and linearly added to the output F3 of the third layer, and finally a multi-scale feature is obtained that integrates the information of each layer.

[0088] S3, constructing an adaptive feature augmentation module: This embodiment adopts a dual-branch network architecture, constructing an adaptive feature augmentation module in the first branch, and calculating the multi-scale feature F m The characteristic statistics of , Gaussian distribution model is used to introduce disturbance, the uncertainty of the statistics is calculated, and the upper and lower limits are set to control the degree of disturbance, thereby obtaining two affine parameters for multi-scale feature F m Perform augmentation transformation to expand its representation space and obtain augmented features

[0089] like Figure 4 As shown, the adaptive feature augmentation module is used to expand the representation space of the input features. The module first calculates F m The feature statistics of (denoted as f) in the channel dimension include mean μ(f) and standard deviation σ(f), μ(f) and σ(f) are modeled as Gaussian distributions and Gaussian perturbations ∈ μ and ∈ σ , based on which two affine parameters β and γ are generated for feature augmentation; in addition, in order to avoid the negative impact of excessive perturbation on network learning, the uncertainty of μ(f) and σ(f) is calculated and the upper and lower limits are set for them to control the degree of Gaussian perturbation. Finally, γ and β are used to perform the multi-scale feature F mPerform augmentation transform to generate features of simulated domain shift Thereby expanding the representation space of features;

[0090] Specifically, the multi-scale feature F m The calculation formulas for the mean μ(f) and standard deviation σ(f) of (abbreviated as f) in the channel dimension are as follows:

[0091]

[0092]

[0093] Among them, H, W and c represent the height, width and channel of the sample feature map respectively, f c,h,w It is a multi-scale feature in the channel dimension.

[0094] Calculate the variance of μ(f) and σ(f), which are respectively denoted as and The formula is as follows:

[0095]

[0096]

[0097] Among them E n () indicates finding the mean of feature statistics;

[0098] use and The uncertainty of μ(f) and σ(f) is measured respectively, and the Gaussian distribution is used to model μ(f) and σ(f). In order to perform feature augmentation, the standard normal distribution is introduced. Gaussian perturbation ∈ μ and ∈ σ , using the obtained uncertainty and The degree of Gaussian perturbation is controlled respectively to obtain two affine parameters β and γ, and the formula is as follows:

[0099]

[0100]

[0101] Using the affine parameters β and γ, the features of the simulated domain shift are finally generated That is, the augmented features The formula is as follows:

[0102]

[0103] Therefore, the multi-scale feature F m After the adaptive feature augmentation module, the feature

[0104] S4. Construct channel-level whitening module: Use the mapping layer to transform the augmented features The multi-channel mapping is two channels for true and false discrimination. In the mapping process, according to the index of the mapping layer weight after size sorting, the channels that best represent the true and false categories are divided into the same group in turn. The covariance matrix is ​​calculated in each group and constrained by the unit matrix to calculate the whitening loss L cw ;

[0105] In this embodiment, the channel-level whitening module takes into account that in the mapping layer of the convolutional neural network, the channel features can represent a kind of semantic information, which specifically represents the two categories of true and false in the field of liveness detection. This module uses the weight index of the sorted mapping layer to sequentially divide the channels that can best represent the true and false categories in the mapping process into the same group. Through the decorrelation operation, the covariance between the two channels in each group is reduced, the confusion between the channels representing the true and false categories is reduced, and the discriminative representation ability of the channel features is improved. This module mainly includes three parts: weight sorting, weight index grouping, and whitening loss calculation. Figure 5 The specific process is as follows:

[0106] First, a mapping layer is used to transform the features The multi-channels (assuming the number of channels before mapping is C) are mapped into two channels representing true and false categories, where two sets of weights corresponding to true and false channels can be obtained, with the subscript r representing true and the subscript s representing false, namely:

[0107]

[0108]

[0109] Sort the weights of the two groups of mapping layers above from large to small. Assume that the sorted indexes are as follows:

[0110]

[0111]

[0112] According to the sorted weight index in formula (10-11), the channels that best represent the true and false categories are divided into the same group, that is, channel 3 and channel 16 are grouped together, channel 14 and channel 5 are grouped together, and so on. The covariance matrix is ​​calculated in each group to obtain the whitening loss L cw , and constrain it using the identity matrix. By reducing the covariance between the two channels in each group, the confusion between the channels representing the true and false categories is reduced, thereby improving the discriminative representation ability of channel features. The formula for whitening loss is as follows:

[0113]

[0114] Among them, C is the number of channels before the feature passes through the mapping layer, is the number of groups in the module after grouping, Represents the nth sample feature F in a batch input nn The gth group, I is the identity matrix, represent The covariance matrix of is:

[0115]

[0116] The formula for calculating the covariance matrix is ​​as follows:

[0117]

[0118] Among them, H and W represent the height and width of the sample feature map respectively, and F n,i,h,w ,F n,j,h,w (i, j∈[1,2]) represent the channel dimension features representing true and false in each group of the nth sample features, μ n,i ,μ n,j (i,j∈[1,2]) corresponds to the channel mean, 1 and 2 represent the true and false categories respectively, i and j represent the channel index in the group, and the value range is [1,2].

[0119] S5. Construct a channel-level rejection module: This module is located in the second branch of the two-branch network. It uses the domain discriminator D and measures the domain bias of each channel according to its weight. It removes channels that contribute greatly to the domain identification task to suppress channels that are sensitive to domain shift.

[0120] In this embodiment, the channel-level culling module extracts domain-invariant features by removing channels that are sensitive to domain shift. Figure 6 As shown in Figure 1, the domain discriminator D used in this module consists of a global average pooling layer (GAP) and a fully connected layer. A gradient reversal layer (GRL) is used before D to truncate the return gradient. Its function is to avoid the negative impact of the domain division task on the main network classification task. m Input this module and use the weights of the fully connected layer in the domain discriminator D to calculate the weighted activation value s predicted as the correct domain c , thereby measuring the domain tendency of each channel. The specific formula is as follows:

[0121]

[0122] Among them, c is the channel index, represents the fully connected layer weight corresponding to the channel predicted as the correct domain d, s c The larger the value, the greater the contribution of the channel to the domain identification task, that is, the more sensitive it is to domain shift. Therefore, it is necessary to continuously guide the model to remove such channels that are biased towards the domain during the forward propagation process to improve the domain generalization performance of the detection model. This module uses the idea of ​​adversarial learning. The task of the domain discriminator D is to identify channels that are sensitive to domain shift as much as possible, and the ultimate goal of the module is to learn domain invariant features. The domain identification loss L involved in this module is d The formula is as follows:

[0123]

[0124] Where P is the number of source domains, n d represents the number of samples with domain label d in a batch input, j is the predicted domain label, It means that the value of this formula is 1 when the domain discriminator D predicts correctly, and 0 otherwise.

[0125] S6. Construct an attention feature fusion module: Use the channel attention mechanism to fuse the features processed by the two branches to obtain the final discriminant feature

[0126] In this embodiment, the attention feature fusion module uses the channel attention mechanism to fuse the two parts of features after the two branches in the network are optimized to obtain the final discriminant feature. like Figure 7 As shown in the figure, the channel attention mechanism adopted by this module generates fusion weights based on two branches (global channel attention and local channel attention) with different scales to achieve better feature fusion effect. The specific fusion method is as follows:

[0127]

[0128] Among them, X and Y represent the features output after processing by the channel-level whitening module of the first branch and the features output after processing by the channel-level culling module of the second branch, respectively. represents the addition of corresponding elements, Indicates the multiplication of corresponding elements, A() indicates obtaining the fusion weight based on the attention mechanism. The specific operation of the process is as follows:

[0129]

[0130] Where σ is the Sigmoid activation function, L(X) and G(X) represent the channel attention of local features and global features, respectively. The specific operations of L(X) mainly include 1×1 point-by-point convolution, BN layer, and ReLU activation function. The only difference between G(X) and L(X) is that G(X) first passes through a global average pooling layer before performing several operations in L(X).

[0131] S7. Build a classifier: The final discriminant features Send it to the classifier to get the binary classification prediction result;

[0132] In this embodiment, the classifier includes a fully connected layer with an input dimension of N×8×8 and an output dimension of 4N, a ReLU activation function layer, a Dropout layer, and a fully connected layer with an input dimension of 4N and an output dimension of 2.

[0133] Specifically, Input the classifier to get the output feature of size 2. Calculate based on the output feature to get the predicted label and predicted probability value, and use the true and false label y of the sample to supervise it to get the classification loss function L cls , whose formula is as follows:

[0134] L cls =-(ylog(y′)+(1-y)log(1-y′)) (19)

[0135] Among them, y represents the true or false label of the sample, and y′ represents the predicted label;

[0136] S8. Network training: Use the true or false label y and domain label d of the input sample to supervise the output of the classifier and the output of the domain discriminator D, respectively, to obtain the binary classification loss L cls and domain discrimination loss L d , combined with the whitening loss L of the channel-level whitening module cw , calculate the total loss function L all , used to constrain the model training process, update the network weight coefficient, and save the network model and optimal weight after training;

[0137] Specifically, the RGB face image is input into the network for end-to-end training, and the Adam optimizer is used as the training optimizer, and the learning rate is set to 2×10 -4 The exponential decay rate β1 of the first-order moment estimate is 0.9, and the exponential decay rate β2 of the second-order moment estimate is 0.999. The network weights are updated with the goal of minimizing the total loss function. After training, the network model and optimal weights are saved. The total loss function involved in this process is as follows:

[0138] L all= λ1L cls + λ2L cw + λ3L d (20)

[0139] wherein, L cls is a binary classification loss, L cw is a whitening loss obtained by the channel-level whitening module in the first branch of the dual-stream network, L d is a domain discrimination loss obtained by the channel-level rejection module in the second branch of the network, λ1, λ2, λ3 are proportion weights of the three losses, and in the embodiment, the values thereof are 1, 0.5 and 1 respectively;

[0140] S9, network testing: using the trained model to test, calculating various benchmark indexes according to the prediction results.

[0141] The specific process is as follows: input the test set sample into the trained network to perform feature extraction and prediction, output the prediction result of the test set sample, calculate various benchmark indexes according to the prediction result, and the benchmark indexes used include false positive rate (FPR), false negative rate (FNR), half total error rate (HTER) and area under curve (AUC) of receiver operating characteristic curve (ROC).

[0142] The false positive rate (FPR) refers to the ratio of the number of non-living face judgments to the number of living face to the number of labeled non-living face, and the formula is as follows:

[0143]

[0144] The false negative rate (FNR) refers to the ratio of the number of living face judgments to the number of non-living face to the number of labeled living face, and the formula is as follows:

[0145]

[0146] The half total error rate (HTER) refers to the average of the false positive rate and the false negative rate, and the smaller the HTER value, the better the detection effect of the model, and the formula is as follows:

[0147]

[0148] The ROC curve is drawn by calculating the false positive rate (FPR) and true positive rate (TPR) under different thresholds, with FPR as the horizontal axis and TPR as the vertical axis. The AUC is the area under the ROC curve. The larger the AUC value, the better the detection effect of the model.

[0149] The present invention is based on a dual-branch network and utilizes whitening and culling operations at the feature channel level, so that the model can not only achieve the task of true and false discrimination, but also suppress information that is sensitive to domain shift, thereby improving the domain generalization performance of the liveness detection model.

[0150] This embodiment conducts a three-database-across-one-database experiment (training on three databases and testing on the remaining database) on four databases, namely Oulu_NPU (denoted as O database), CASIA-MFSD (denoted as C database), MSU-MFSD (denoted as M database), and Replay-Attack (denoted as I database) to verify the generalization performance of the algorithm. The three databases are OCI database-across-M database (OCI_M), OMI database-across-C database (OMI_C), OCM database-across-I database (OCM_I), and ICM database-across-O database (ICM_O). The experimental results are shown in the following table:

[0151] Table 1 Cross-library test results (HTER)

[0152] method OCI_M OMI_C OCM_I ICM_O average value CAD 11.64％ 17.51％ 15.08％ 14.27％ 14.63％ SSAN-R 6.67％ 10.00％ 8.88％ 13.72％ 9.82％ The present invention 5.83％ 10.56％ 10.12％ 9.98％ 9.12％

[0153] Table 2 Cross-library test results (AUC)

[0154] method OCI_M OMI_C OCM_I ICM_O average value CAD 95.27％ 89.98％ 91.92％ 93.04％ 92.55％ SSAN-R 98.75％ 96.67％ 96.79％ 93.63％ 96.46％ The present invention 98.52％ 97.15％ 96.53％ 95.76％ 96.99％

[0155] This embodiment uses CAD and SSAN-R as comparison algorithms of the present invention. CAD was published in ICASSP in 2022. This method improves cross-database performance through feature decoupling and cross-adversarial training. SSAN-R was published in CVPR in 2022. This method enhances feature diversity through style recombination and uses adversarial learning to extract domain-invariant features to improve the generalization performance of the detection model. Both are representative algorithms in the field of liveness detection, among which SSAN-R is one of the currently leading algorithms. As can be seen from Tables 1 and 2, the HTERs of the present invention in the four cross-library experiments of OCI library across M library, OMI library across C library, OCM library across I library, and ICM library across O library are 5.83%, 10.56%, 10.12%, and 9.98%, respectively, with an average of 9.12%; the AUCs are 98.52%, 97.15%, 96.53%, and 95.76%, respectively, with an average of 96.99%. It can be seen that the present invention is generally better than the two comparison algorithms, showing good domain generalization performance, which verifies the effectiveness of the method of this embodiment.

[0156] Example 2

[0157] This embodiment provides a liveness detection system based on feature channel representation, which is used to implement the liveness detection method based on feature channel representation in the above embodiment 1. The system includes: a data preprocessing module, a multi-scale feature extraction module, an adaptive feature augmentation module, a channel-level whitening module, a channel-level culling module, an attention feature fusion module, a classifier, a network training module, and a testing module.

[0158] In this embodiment, the data preprocessing module divides the video data set into frames, extracts the face area of ​​each frame image to obtain a face image, and sets corresponding authenticity labels and domain labels for the face image;

[0159] In this embodiment, the multi-scale feature extraction module is used to extract multi-scale features of the face image;

[0160] In this embodiment, the adaptive feature augmentation module is used to expand the representation space of multi-scale features. By calculating the feature statistics of the multi-scale features, Gaussian distribution modeling is performed on them and perturbations are introduced. The uncertainty of the statistics is calculated, and upper and lower limits are set to control the degree of perturbations. Two affine parameters are obtained to perform an augmentation transformation on the multi-scale features to expand their representation space and obtain augmented features.

[0161] In this embodiment, the channel-level whitening module uses a mapping layer to map the multi-channel augmented features into two channels for true and false discrimination. During the mapping process, the channels that best represent the true and false categories are grouped according to the index of the mapping layer weights after sorting by size. The covariance matrix is ​​calculated within each group and constrained using the identity matrix to calculate the whitening loss.

[0162] In this embodiment, the channel-level culling module is used to calculate the weighted activation value predicted as the correct domain based on the multi-scale features of the input, using the weights of the fully connected layer in the domain discriminator, measure the domain bias of each channel, eliminate channels that contribute significantly to the domain identification task, and learn domain-invariant features.

[0163] In this embodiment, the attention feature fusion module is used to fuse the features optimized by the channel-level whitening module and the domain-invariant features learned by the channel-level culling module based on the channel attention mechanism to obtain the final discriminant features;

[0164] In this embodiment, the classifier is used to obtain a binary classification prediction result based on the final discriminant features;

[0165] In this embodiment, the network training module is used to supervise the output results of the classifier and the output results of the domain discriminator using the authenticity label and domain label respectively, to obtain the binary classification loss and domain discrimination loss, and to combine the whitening loss of the channel-level whitening module to obtain the total loss function, and to train and update the network model weight coefficient;

[0166] In this embodiment, the testing module is used to obtain a liveness detection prediction result based on the trained network model.

[0167] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A method for liveness detection based on feature channel representation, characterized in that: The steps include: The video dataset is divided into frames, the face area of ​​each frame is extracted to obtain the face image, and the corresponding authenticity label and domain label are set for the face image; Construct a multi-scale feature extraction module to extract multi-scale features of face images; An adaptive feature augmentation module is constructed to expand the representation space of multi-scale features. By calculating the feature statistics of multi-scale features, Gaussian distribution modeling is performed on the feature statistics and perturbations are introduced. The uncertainty of the statistics is calculated and upper and lower uncertainty limits are set to control the degree of perturbation. Two affine parameters are obtained, which are used to perform an augmentation transformation on the multi-scale features to expand the multi-scale feature representation space and obtain the augmented features. Construct a channel-level whitening module and input the augmented features into the module. Use the mapping layer to map the multi-channels of the augmented features into two channels for true and false discrimination. During the mapping process, the channels that best represent the true and false categories are divided into the same group according to the index of the mapping layer weight after sorting by size. Calculate the covariance matrix within each group and constrain the covariance matrix using the identity matrix to calculate the whitening loss. Construct a channel-level elimination module, input multi-scale features into the channel-level elimination module, use the weight calculation of the fully connected layer in the domain discriminator to predict the weighted activation value of the correct domain, measure the domain tendency of each channel, eliminate channels that contribute greatly to the domain identification task, and learn domain-invariant features; Construct an attention feature fusion module, which fuses the features optimized by the channel-level whitening module and the domain-invariant features learned by the channel-level culling module based on the channel attention mechanism to obtain the final discriminant features; Build a classifier and input the final discriminant features into the classifier to obtain the binary classification prediction results; The output of the classifier and the output of the domain discriminator are supervised by using the authenticity label and domain label respectively, obtaining the binary classification loss and domain discrimination loss. The total loss function is weighted by combining the whitening loss of the channel-level whitening module to train and update the weight coefficient of the network model. The liveness detection prediction results are obtained based on the trained network model.

2. The method for liveness detection based on feature channel representation according to claim 1, characterized in that: Construct a multi-scale feature extraction module to extract multi-scale features of facial images, including: The multi-scale feature extraction module uses the first three layers of ResNet-18 as the initial feature extractor of the input face image, extracts the features of each layer of the input face image, performs scale transformation operations on the obtained features at each level, and obtains multi-scale features through linear addition.

3. The method for liveness detection based on feature channel characterization according to claim 2, characterized in that: The obtained features at all levels are scaled and linearly added to obtain multi-scale features, including: The features output by the first layer are scaled and then linearly added to the output of the second layer. The processed features are also scaled and then linearly added to the output of the third layer to obtain multi-scale features.

4. The method for liveness detection based on feature channel representation according to claim 1, characterized in that: Construct an adaptive feature augmentation module to expand the representation space of multi-scale features, specifically including: Calculate the mean and standard deviation of multi-scale features in the channel dimension, which can be expressed as: Among them, μ(f) represents the mean, σ(f) represents the standard deviation, H, W and c represent the height, width and channel of the sample feature map respectively, and f c,h,w Represent multi-scale features in the channel dimension; Calculate the variance of μ(f) and σ(f), which are respectively denoted as and The formula is as follows: Among them E n () indicates finding the mean of feature statistics; use and Measure the uncertainty of μ(f) and σ(f) respectively, and use Gaussian distribution to model μ(f) and σ(f), and introduce the standard normal distribution Gaussian perturbation ∈ μ and ∈ σ , based on the uncertainty, the degree of Gaussian perturbation is controlled and two affine parameters are obtained, which are specifically expressed as: Based on the affine parameters β and γ, the features of the simulated domain shift are generated, that is, the augmented features, which are specifically expressed as: in, represents the features of the simulated domain shift, and f represents the multi-scale features before augmentation.

5. The method for liveness detection based on feature channel representation according to claim 1, characterized in that: The whitening loss is expressed as: Among them, L cw Represents whitening loss, N represents the number of input sample features, and C is the number of channels before the feature passes through the mapping layer. is the number of groups after grouping, Represents the nth sample feature F in a batch input n The gth group, I is the identity matrix, represent The covariance matrix of H and W represents the height and width of the sample feature map, respectively, and F n,i,h,w ,F n,j,h,w (i, j∈[1,2]) represent the channel dimension features representing true and false in each group of the nth sample features, μ n,i ,μ n,j (i,j∈[1,2]) corresponds to the channel mean, 1 and 2 represent the true and false categories respectively, i and j represent the channel index in the group, and the value range is [1,2].

6. The method for liveness detection based on feature channel representation according to claim 1, characterized in that: The weighted activation value predicted as the correct domain is calculated using the weights of the fully connected layer in the domain discriminator, specifically expressed as: Among them, s c represents the weighted activation value, c is the channel index, represents the fully connected layer weight corresponding to the channel predicted as the correct domain, F m represents multi-scale features, and GAP represents the global average pooling layer.

7. The method for liveness detection based on feature channel representation according to claim 1, characterized in that: The domain discrimination loss of the domain discriminator is expressed as: Among them, L d represents the domain discrimination loss, P is the number of source domains, n d represents the number of samples with domain label d in a batch input, j is the predicted domain label, Indicates that the value of this formula is 1 when the domain discriminator D predicts correctly, otherwise it is 0, F m Represents multi-scale features.

8. The method for liveness detection based on feature channel representation according to claim 1, characterized in that: Based on the channel attention mechanism, the features optimized by the channel-level whitening module and the domain-invariant features learned by the channel-level culling module are fused to obtain the final discriminant features, which are specifically expressed as: Among them, X and Y represent the features output by the channel-level whitening module and the features output by the channel-level culling module, respectively. represents the addition of corresponding elements, Indicates the multiplication of corresponding elements, A() indicates obtaining the fusion weight based on the attention mechanism. The specific operations are as follows: Among them, σ is the Sigmoid activation function, L(X) and G(X) represent the channel attention of local features and global features, respectively.

9. The method for liveness detection based on feature channel representation according to claim 1, characterized in that: The final discriminant features are input into the classifier to obtain the binary classification prediction results, including: The predicted labels and predicted probability values ​​are calculated based on the output features of the classifier, and supervised training is performed based on the true and false labels to construct the binary classification loss, which is specifically expressed as: L cls =-(ylog(y ′ )+(1-y)log(1-y ′ )) Among them, L cls represents the binary classification loss, y represents the true and false labels, y ′ Represents the predicted label.

10. A liveness detection system based on feature channel representation, characterized in that: include: Data preprocessing module, multi-scale feature extraction module, adaptive feature augmentation module, channel-level whitening module, channel-level culling module, attention feature fusion module, classifier, network training module and testing module; The data preprocessing module divides the video data set into frames, extracts the face area of ​​each frame image to obtain a face image, and sets corresponding authenticity labels and domain labels for the face image; The multi-scale feature extraction module is used to extract multi-scale features of facial images; The adaptive feature augmentation module is used to expand the representation space of multi-scale features by calculating the feature statistics of the multi-scale features, modeling the feature statistics with Gaussian distribution and introducing disturbances, calculating the uncertainty of the statistics, setting upper and lower limits of the uncertainty to control the degree of disturbance, and obtaining two affine parameters for performing an augmentation transformation on the multi-scale features to expand the multi-scale feature representation space and obtain augmented features; The channel-level whitening module uses a mapping layer to map the multi-channels of the augmented features into two channels for true and false discrimination. During the mapping process, the channels that best represent the true and false categories are divided into the same group according to the index of the mapping layer weights after size sorting. The covariance matrix is ​​calculated within each group and constrained by the identity matrix to calculate the whitening loss. The channel-level elimination module is used to calculate the weighted activation value predicted as the correct domain based on the multi-scale features of the input using the weights of the fully connected layer in the domain discriminator, measure the domain tendency of each channel, eliminate channels that contribute greatly to the domain identification task, and learn domain-invariant features; The attention feature fusion module is used to fuse the features optimized by the channel-level whitening module and the domain-invariant features learned by the channel-level culling module based on the channel attention mechanism to obtain the final discriminant features; The classifier is used to obtain a binary classification prediction result based on the final discriminant features; The network training module is used to supervise the output results of the classifier and the output results of the domain discriminator using the true and false labels and domain labels respectively, to obtain the binary classification loss and the domain discrimination loss, and to combine the whitening loss of the channel-level whitening module to obtain the total loss function, and to train and update the network model weight coefficient; The testing module is used to obtain a liveness detection prediction result based on the trained network model.

Images