Face anti-spoofing method based on feature consistency and scale pyramid loss

Abstract

The application claims a face anti-spoofing method based on feature consistency and scale pyramid loss, belonging to the technical field of computer vision. The method comprises the following steps: first, in order to solve the problem that the model generalization is reduced due to the insufficient feature consistency under different viewing angles in living body detection, a feature consistency module is proposed to encourage the consistency of feature maps under different viewing angles and ensure that feature maps under different scales can be matched and aligned with each other. Secondly, a module MFFM for multi-scale feature fusion is designed, so that the field migration part can learn more scale information when generating a depth map, which helps the network to better capture the correlation and spatial structure between features. Finally, a scale pyramid loss SP Loss is designed, which can make the model consider the information of the image from multiple scales, including global semantics and local details, to improve the perception ability of the model to different scale information and further improve the performance and robustness of the model.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision, and in particular relates to a face anti-spoofing method. BACKGROUND

[0002] In recent years, with the rapid development of deep learning technology, face recognition technology has been widely applied in various fields, such as mobile phone unlocking, station ticket checking, access control system, and financial payment, etc. These applications have brought great convenience to people's life, but at the same time, they also bring some security risks. With the popularization of the Internet, the leakage of face information becomes easier and easier. Criminals can easily obtain the facial information of others from social platforms and other channels, and present it in the form of video playback, printed photos, etc. in front of the face recognition system to commit fraud by using the identity information of others. This poses a potential threat to the security of the face recognition system.

[0003] To solve this problem, face anti-spoofing technology has emerged. Early researchers designed texture feature extraction algorithms to deal with face presentation attacks, but with the development of deep learning, more and more researchers began to use convolutional neural networks to extract discriminative features from images, which achieved significant performance improvement. However, the current face anti-spoofing technology still has some degree of instability in dealing with factors such as light, image quality, and device type. The influence of these factors will greatly reduce the discriminative performance of the model, making it difficult to deploy in real environments. Therefore, how to improve the adaptability of the model to these uncontrollable factors has become an important problem to be solved in the field of face anti-spoofing.

[0004] To improve the robustness and generalization ability of face anti-spoofing models, some researchers have begun to use auxiliary supervision modalities and domain transfer techniques to improve the discriminative performance of the model. Among them, the most commonly used auxiliary supervision modality is depth map (Depthmap), which is generated by the PRNet network and is intended to represent the 3D geometric structure information of the real face. Most researchers use the DepthNet network to extract the facial contour information on the image for the prediction of the depth map. However, the consistency of the features of this network may not be sufficient when facing different angles, which may lead to a decrease in model performance. The lack of consistency loss constraint may cause the feature representation learned by the model to change greatly under different angles, thereby reducing the stability and accuracy of the model. This will result in inconsistent recognition results of the model for face images under different poses or lighting conditions. During the domain transfer process, insufficient consideration of capturing local details and global semantic information will result in the model only being able to obtain limited domain knowledge related to the target (depth) domain from the source (RGB) domain. To solve the above problems, the present application proposes a face anti-spoofing method based on regularization domain transfer and scale pyramid loss.

[0005] Prior application CN117409453A, a face anti-deception method and device based on multi-scale domain transfer, includes: step S1, obtaining multi-scale face contour feature information; step 2, obtaining GAN network gradient parameters according to multi-scale face contour feature information; step 3, using the updated FEM extraction feature of GAN network gradient parameter, sending the extracted last level feature into the feature aggregation module, and then into the classifier for discrimination, and performing face anti-deception. The technical scheme of the present application improves the accuracy of depth map prediction.

[0006] Defects:

[0007] -Depth of feature fusion: Although multi-scale feature extraction and aggregation are proposed, there may be limitations in the depth fusion of features and the comprehensive utilization of context information.

[0008] -View consistency problem: There is no clear solution to the problem of feature consistency under different views, which may lead to performance degradation when processing faces with different poses.

[0009] -Loss function design: The loss function may not fully consider the integration and optimization of multi-scale information, affecting the learning and utilization of features by the model during training.

[0010] Overcoming method of the present invention:

[0011] Solution:

[0012] -Feature consistency module: by ensuring that the features extracted under different views remain consistent in the feature space, the influence of view and illumination changes on feature consistency is overcome.

[0013] -Scale pyramid loss: The design of SP Loss scale pyramid loss enables the model to consider the differences between generated images and real images during domain transfer from multiple scales, including global semantics and local details, enhancing the model's perception of different scales and improving the quality of the generated depth map.

[0014] -Multi-scale feature fusion: In the preliminary image feature extraction, a multi-scale feature fusion method is used to integrate features of different scales, enhancing the network's ability to perceive multi-scale information and significantly improving the network's performance in processing multi-scale images. Summary of the invention

[0015] The present invention aims to solve the problems of the above prior art. A face anti-deception method based on feature consistency and scale pyramid loss is proposed. The technical scheme of the present invention is as follows:

[0016] A face anti-spoofing method based on feature consistency and scale pyramid loss, comprising the following steps:

[0017] Step 1, first, the input face image is passed through a liveness detection network, and two images under different views (View1 and View2) are generated through a feature enhancement module. Then, the two images and the original face image are extracted through an encoder (Encoder) module to obtain initial features. The features extracted from the low, medium and high layers of the encoder are sent to a multi-scale feature fusion module for fusion. At the same time, the features extracted under the two different views are sent to a feature consistency module to ensure their consistency in the feature space.

[0018] Step 2, the multi-scale feature fusion module fully fuses the low, medium and high layer features and sends them to the Decoder module of the network for domain migration operation of RGB and depth map, and calculates the discriminator loss (GAN Loss) and scale pyramid loss (SP Loss) in the domain migration module.

[0019] Step 3, the features extracted under the two different views are sent to the feature consistency module, and the feature consistency loss of the two different views is calculated through the density similarity loss; finally, the discriminator loss (GAN Loss) and the scale pyramid loss (SP Loss) are combined to form the total loss, and the model parameters are updated through iterative adversarial training.

[0020] Step 4, the high layer features of the real samples extracted by the updated Encoder part of the original image are sent to the classifier for liveness classification.

[0021] Further, the step 1 generates two images under different views (View1 and View2) through the input face image through the feature enhancement module, and then extracts initial features through the encoder (Encoder) module. The features extracted from the low, medium and high layers of the encoder are sent to the multi-scale feature fusion module for fusion. At the same time, the features extracted under the two different views are sent to the feature consistency module, which comprises the following steps:

[0022] 1.1 First, the pictures input into the training network are subjected to sample balancing operation, i.e. the number of true and false samples participating in training each time is kept at a ratio of 1:1, so as to reduce the influence of sample imbalance on the discrimination performance of the model;

[0023] 1.2 Then, the balanced training samples are subjected to feature enhancement to generate two enhanced images under different views. The face image x is subjected to a series of data enhancement methods including random cropping, random shuffling of regions, rotation, scaling, translation and random erasing to generate two enhanced images x1 and x2.

[0024] 1.3 Input the face image x, and the generated enhanced images x1, x2 into the Encoder module to extract the high-level features of the images, to obtain the low, medium and high three-layer feature maps f1, f2, f3 of the original image and the high-level features H3, T3 of the enhanced image; feed f1, f2, f3 to the multi-scale feature fusion module; and feed H3, T3 to the feature consistency module.

[0025] Further, the step 1.3 inputs the balanced training sample and the enhanced image into the Encoder to extract the facial contour feature information, to obtain the low, medium and high three-layer feature maps f1, f2, f3 of the original image and the high-level features H3, T3 of the two enhanced images, which specifically includes:

[0026] 1) Perform convolution preprocessing operation on the image input into the network to obtain a feature map f0 with 64 channels and two enhanced features H0 and T0 with 64 channels;

[0027] 2) Input the feature map f0 and H0 and T0 into the first feature extraction block in the Encoder, which is composed of 3 consecutive convolution layers, and the first 2 convolution layers are followed by a normalization layer and an activation layer, and the last convolution layer is followed by only a normalization layer. After the processing of the first feature extraction block, three feature maps f1, H1, T1 with 128 channels and a size of 128*128 can be obtained;

[0028] 3) Input the feature maps f1, H1, T1 into the second feature extraction block in the Encoder, which has the same structure as the first feature extraction block. After the processing of the second feature extraction block, three feature maps f2, H2, T2 with 128 channels and a size of 64*64 are obtained.

[0029] 4) Input the feature maps f2, H2, T2 into the second feature extraction block in the Encoder, which has the same structure as the first two feature extraction blocks. After the processing of the third feature extraction block, three feature maps f3, H3, T3 with 128 channels and a size of 32*32 are obtained. The above process can be represented by the following formula:

[0030] f0=C0(x)#(1)

[0031]

[0032] f i =P(σ(C i (f i-1 )+B i (f i-1 ))),i∈{1,2,3}#(3)

[0033] where x is the input image, Conv 3*3 is a convolutional layer with a kernel of 3*3, B i is the i-th feature extraction module in the Encoder, σ is a nonlinear activation function, P represents a max-pooling layer, f i is the feature obtained after the activation function and the max-pooling layer in the Encoder.

[0034] Further, the step 2 feeds the extracted f1, f2, f3 to the multi-scale feature fusion module for RGB and depth map multi-scale domain migration operation, and calculates the loss of the generator and the discriminator in the domain migration module; H3, T3 are fed to the feature consistency module to calculate the consistency loss. Specifically, the following steps are included:

[0035] 2.1 Compared with the baseline network DTN of the application, it only considers a single fixed scale of depth map in the domain migration module, which makes the model in the source domain only learn limited domain knowledge related to the target domain. Different from this, the application considers the influence of different scale auxiliary modal information on the discrimination performance of the model, introduces the domain migration module after multi-scale feature fusion, so as to help the model learn rich domain knowledge related to the target domain in the source domain, so as to fully utilize the auxiliary modal depth map information and improve the discrimination of the model;

[0036] 2.2 The multi-scale feature maps obtained in step 1 are spliced, first fed into the multi-scale feature fusion module for feature fusion, and then fed into the generator Decoder for depth map prediction;

[0037] 2.3 The predicted depth map and the corresponding size RGB image are spliced in the channel dimension, and then the spliced image is fed into the GAN network to calculate the loss value and update the gradient parameters in the network module.

[0038] Further, the step 2.2 splices the multi-scale feature maps obtained in step 1, first feeds them into the multi-scale feature fusion module for feature fusion, and then feeds them into the generator Decoder for depth map prediction. Specifically, the following steps are included:

[0039] 1) First, use adaptive average pooling operation to change the size of the feature maps f1, f2 to be consistent with f3, and then feed the three feature maps into the multi-scale feature fusion module to obtain a feature map f con . Specifically, it includes:

[0040] 2) Add f1 and f2 to obtain feature f xa Then fxa The local feature f xl and the global feature f xg are input into the local and global attention mechanism. xl The local feature f xg and the global feature f xlg are added to obtain f .

[0041] 3) The f xlg is mapped to the range [0, 1] by the Sigmoid function to obtain the fusion weight ω. The input feature f1 and the feature f2 are multiplied by the fusion weight ω and (1-ω) respectively, and the results are added to obtain the final fusion feature f xo . At the same time, the same operation is performed on the features f1 and f3 to obtain the fused feature f xo1 , and the fused feature is subjected to a channel-level splicing operation to obtain the final feature f con .

[0042]

[0043] wherein, represents pixel-level summation, σ represents the Sigmoid function, represents channel-level splicing operation, L represents local attention, and G represents global attention.

[0044] 4) f con is input into the generator Decoder composed of three consecutive convolution layers, and a normalization layer and a nonlinear activation layer are followed after each convolution layer, to predict a depth map with a channel number of 1, i.e. to obtain a depth map with a size of 32*32 and a feature dimension of 1 The process is represented in the following formula:

[0045]

[0046] wherein, Decoder(.) represents the depth map generator, Down(.) represents down-sampling operation, represents the predicted depth map with a size of 32*32.

[0047] Further, the step 2.3 splices the predicted multi-scale depth map and the corresponding size RGB image in the channel dimension, and then inputs the spliced image into the GAN network, calculates the loss value and updates the gradient parameters in the network module, which specifically includes the following steps:

[0048] 1) first, the depth map is spliced with the corresponding size RGB image x 32The channel-level splicing operation is performed, and the gradient calculation is not allowed, and then the standard depth map d obtained by the PRNet network is sent into the discriminator Dis respectively 32 The RGB image x 32 The channel-level splicing operation is performed, and the gradient calculation is not allowed, and then the standard depth map d obtained by the PRNet network is sent into the discriminator Dis respectively

[0049]

[0050] wherein D i represents the discriminator for discriminating the image with the size of i*i, x i represents the RGB image with the size of i*i, d i represents the standard depth map with the size of i*i, represents the predicted depth map with the size of i*i.

[0051] 2) The depth map d is respectively spliced with the RGB image x 32 The channel-level splicing operation is performed, and the gradient calculation is not allowed, and then the standard depth map d obtained by the PRNet network is sent into the discriminator Dis respectively 32 In order to enable the generator to generate a depth map with higher quality, the predicted depth map is also supervised by a scale pyramid loss (SP Loss), which is composed of a pyramid loss and a multi-scale loss. The specific steps are as follows: the target depth map d and the predicted depth map d are respectively subjected to multi-scale interpolation operations to adjust them to different scales, and then the square error loss of the interpolation results at each scale is calculated to measure the difference between the predicted value and the target value, and the losses at each scale are accumulated to obtain the pyramid loss Subsequently, the predicted value and the target value are respectively down-sampled to multiple scales, and the L1 loss of the predicted value and the target value at each scale is calculated to measure the absolute difference between them, and the losses at each scale are accumulated and averaged to obtain the multi-scale loss Finally, the pyramid loss and the multi-scale loss are weighted and summed by a weighting factor a to obtain the final scale pyramid loss (SP Loss) The GAN loss of the generator and the scale pyramid loss (SP Loss) are calculated, and the model parameters in the generator Decoder and Encoder are updated. The above process is represented by the following formula:

[0052]

[0053] where N denotes the number of pyramid scales, m i denotes the predicted value denotes the interpolated result at the i-th scale i denotes the target value d 32 denotes the interpolated result at the i-th scale scaledk and Y scaledk respectively represent the predicted value and the target value d 32 denotes the interpolated result at the k-th scale, a represents the weighting factor, MSE denotes the mean square error loss function, and L1 denotes the L1 loss function.

[0054] Further, the step 3 sends the features extracted under two different perspectives into a feature consistency module, and calculates the feature consistency loss under two different perspectives through a density similarity loss, and specifically includes the following steps:

[0055] 1) First, the features H3 and T3 extracted under two perspectives are sent into a predictor (Prediction) to further process and predict the input features. The predictor is a neural network structure with two hidden layers. The first hidden layer includes a sequence of convolution, batch normalization, and ReLU activation function. The sequence first uses a 1x1 convolution kernel to convert the input features from 128 dimensions to 128 dimensions of the hidden layer, then performs batch normalization on the output, and finally uses the ReLU activation function for nonlinear transformation. The second hidden layer is a 1x1 convolution layer that converts the output of the first hidden layer from 128 dimensions to 128 dimensions of the output. Finally, the final two flattened tensors where s 2 represents the spatial resolution of the feature map, and d represents the dimension of each feature vector; the density similarity loss is used to calculate the similarity loss between the feature maps under two different perspectives; the density similarity loss calculates the density similarity between two tensors, considering the relative relationship between each element, rather than only the similarity between the global average vectors, which can better capture the relationship between local features. The above process is represented by the following formula:

[0056]

[0057] where H i denotes the i-th row of the feature map H, and T j denotes the j-th column of the feature map T.

[0058] 3.1 After completing the steps in 3 1), the feature information is extracted again using the Encoder, and the final level feature f3 is obtained.

[0059] 3.2 The final level feature f3 is sent into the classifier to perform classification operation on the sample and calculate the classification loss.

[0060] The step 3.2 sends the aggregation feature f3 into the classifier, performs the classification operation on the sample, and calculates the classification loss, and specifically includes the following steps:

[0061] 1) The feature f3 extracted by the Encoder is sent into the classifier, and the classifier is composed of 3 convolutional layers, 2 maximum pooling layers, 1 global average pooling layer and 1 fully connected layer, and a normalization layer and a nonlinear activation layer are followed after each convolutional layer; the classifier is used for the final discrimination operation on the input image, that is, to judge whether the input face image is real or attack;

[0062] 2) The label value of the real sample is set to 1, and the label value of the attack sample is set to 0, and through this type of label value setting, the discrimination operation of the real and false samples can be regarded as a binary classification operation; the classifier predicts a score value for each sample, and the binary cross-entropy loss function is used to supervise the predicted value, and finally, the cross-entropy loss value is used to update the weight parameters in the classifier and the Encoder. The above process is represented by the following formula:

[0063]

[0064] Wherein, represents the predicted value of the sample by the classifier, and y represents the label value of the sample.

[0065] Further, the loss function in the step 2.3 1) and the loss function in the step 2.3 2) are added to the loss function in the step 3 1) to form the final total loss function, and specifically includes:

[0066]

[0067] Wherein, λ1, λ2, λ3, λ4 represent the weighted hyperparameters of the total loss function.

[0068] The advantages and beneficial effects of the present application are as follows:

[0069] 1. Previous researchers often only use single-scale features when performing domain transfer using image features, which cannot fully extract feature information. This makes the model only learn limited domain knowledge in the source domain, to some extent, hindering the model's learning of multi-scale generalization features and reducing the model's generalization ability. In view of this, the present application designs a multi-scale feature fusion module (Multi-Feature Fusion Model, MFFM) from the perspective of multi-scale feature fusion, aiming to effectively integrate features of different scales. By doing so, the network's ability to perceive multi-scale information is enhanced, significantly improving the network's performance in processing multi-scale images. This integration is achieved by combining local and global attention mechanisms. MFFM more effectively captures the relationships and spatial structures between features, resulting in richer and more expressive feature representations. By utilizing local details and global semantics, MFFM enhances the overall feature representation, thereby improving the model's discriminative ability and robustness against various representation attacks.

[0070] 2. Previous researchers often directly calculate the loss of the generated depth map and the target depth map when using domain transfer networks to solve the face anti-spoofing problem, without fully considering multi-scale information to improve the domain transfer ability. This method limits the model's ability to acquire comprehensive domain knowledge from the source domain, affecting the model's generalization ability at different scales. To solve this problem, the present application designs a scale pyramid loss (SP Loss) mechanism. This mechanism integrates multi-scale depth map information into the domain transfer module, enabling the model to learn more comprehensive and relevant knowledge from the source domain. This multi-scale learning strategy not only enhances the model's ability to capture features at different scales, but also improves the model's generalization and robustness in complex environments. By comprehensively considering image information from macro to micro levels, the model can handle images more meticulously during the learning process. This comprehensive learning process enables the model to more accurately utilize depth map information when mapping source domain images to the target domain, thereby learning more extensive and general feature representations, significantly improving the model's discriminative ability.

[0071] 3.The previous domain generalization based face anti-spoofing method may perform well at a specific view angle, but the performance decreases at other view angles, resulting in unstable recognition accuracy. At the same time, the features extracted from the same thing at different view angles may have significant differences, making it difficult for the model to capture the essential features of the face. Due to the inconsistency of feature representation, the classifier may have difficulty accurately distinguishing between real faces and attack samples. To solve this problem, the present application designs a feature consistency module (FCM). Through the feature consistency module, the model can integrate information from different view angles to provide more comprehensive recognition capabilities. The consistency constraint helps to reduce the overfitting of the model on the training data and improve the generalization ability of the model in actual application. This module ensures the consistency of feature representation, enabling the classifier to more accurately identify and distinguish between real faces and attack samples, thereby significantly improving the discriminant performance of the model. The introduction of FCM not only enhances the stability of the model at different view angles, but also improves the robustness of the model in a diversified environment. BRIEF DESCRIPTION OF DRAWINGS

[0072] Figure 1 is the overall network model structure schematic diagram of the preferred embodiment provided by the present application.

[0073] Figure 2 is the structure schematic diagram of the multi-scale feature fusion module (MFFM) of the present application.

[0074] Figure 3 is the structure schematic diagram of the scale pyramid loss (SP Loss) of the present application. DETAILED DESCRIPTION

[0075] The technical solutions in the embodiments of the present application will be described in detail below with reference to the drawings in the embodiments of the present application. The described embodiments are only a part of the embodiments of the present application.

[0076] The technical solution of the present application to solve the above technical problems is:

[0077] The present application is designed based on domain transfer network, see the literature Wang Y, Song X, Xu T, et al. From RGB to depth: domain transfer network for face anti-spoofing [J]. IEEE Transactions on Information Forensics and Security, 2021, 16: 4280-4290. The literature first uses the DepthNet network to extract the facial contour information on the image, and carries out the prediction of the depth map, and then carries out the domain transfer operation on the predicted depth map, so as to improve the discrimination performance of the model. Different from this method, the present application fully considers the influence of multi-scale features on the discrimination performance of the model, effectively integrates the feature information of different scales, so as to retain more details and global information. Through this fusion, the model can better capture the features of the face under different scales, and enhance its adaptability to complex scenes. In order to fully utilize the multi-scale information of the target domain in the domain transfer process, the present application designs a multi-scale pyramid loss mechanism to evaluate the difference between the predicted depth map and the real depth map at multiple scales, so as to make the model produce high-quality output. In order to solve the problem of performance degradation of the model caused by insufficient consistency of features under different angles when extracting features, we design a feature consistency module. The module ensures the consistency of the features extracted under different angles in the feature space, improves the stability and accuracy of the model recognition, and enhances the stability of the model under different angles. The introduction of the module not only enhances the stability of the model under different angles, but also improves the robustness of the model in diversified environment, so as to significantly improve the discrimination performance of the model.

[0078] Figure 1 The present application is designed based on domain transfer network, see the literature Wang Y, Song X, Xu T, et al. From RGB to depth: domain transfer network for face anti-spoofing [J]. IEEE Transactions on Information Forensics and Security, 2021, 16: 4280-4290. The literature first uses the DepthNet network to extract the facial contour information on the image, and carries out the prediction of the depth map, and then carries out the domain transfer operation on the predicted depth map, so as to improve the discrimination performance of the model. Different from this method, the present application fully considers the influence of multi-scale features on the discrimination performance of the model, effectively integrates the feature information of different scales, so as to retain more details and global information. Through this fusion, the model can better capture the features of the face under different scales, and enhance its adaptability to complex scenes. In order to fully utilize the multi-scale information of the target domain in the domain transfer process, the present application designs a multi-scale pyramid loss mechanism to evaluate the difference between the predicted depth map and the real depth map at multiple scales, so as to make the model produce high-quality output. In order to solve the problem of performance degradation of the model caused by insufficient consistency of features under different angles when extracting features, we design a feature consistency module. The module ensures the consistency of the features extracted under different angles in the feature space, improves the stability and accuracy of the model recognition, and enhances the stability of the model under different angles. The introduction of the module not only enhances the stability of the model under different angles, but also improves the robustness of the model in diversified environment, so as to significantly improve the discrimination performance of the model.

[0079] A face anti-spoofing method based on regularization domain transfer and scale pyramid loss, comprising the following steps:

[0080] Step 1, first, the input face image is passed through the live detection network, and two images under different angles (View1 and View2) are generated through the feature enhancement module. Then, the two images and the original face image are extracted through the encoder (Encoder) module. The features extracted in the low, medium and high layers of the encoder are sent to the multi-scale feature fusion module for fusion. At the same time, the features extracted under two different angles are sent to the feature consistency module to ensure their consistency in the feature space;

[0081] Step 2, the multi-scale feature fusion module fully fuses the low, medium and high three-layer features and then sends them to the Decoder module of the network for RGB and depth map domain migration operation, and calculates the discriminator loss (GAN Loss) and scale pyramid loss (SP Loss) in the domain migration module;

[0082] Step 3, the features extracted under two different perspectives are sent to the feature consistency module, and the feature consistency loss under two different perspectives is calculated through the density similarity loss; finally, the discriminator loss (GAN Loss) and the scale pyramid loss (SP Loss) are combined to form the total loss, and the model parameters are updated through iterative adversarial training.

[0083] Step 4, the high-level features of the real samples extracted by the updated Encoder part of the original image are sent to the classifier for live classification.

[0084] Further, the step 1 generates two images under different perspectives (View1 and View2) through the feature enhancement module based on the input face image, and then extracts the initial features of the two images and the original face image through the Encoder module. The low, medium and high layer features extracted by the Encoder are sent to the multi-scale feature fusion module for fusion. At the same time, the features extracted under two different perspectives are sent to the feature consistency module, which includes the following steps:

[0085] 1.1 First, the sample balancing operation is performed on the pictures input into the training network, that is, the number of true and false samples participating in training each time is kept at a ratio of 1:1, so as to reduce the influence of sample imbalance on the discrimination performance of the model;

[0086] 1.2 Then, the balanced training samples are subjected to feature enhancement to generate two enhanced images under different perspectives. The face image x is subjected to a series of data enhancement methods including random cropping, random shuffling of regions, rotation, scaling, translation and random erasing to generate two enhanced images x1 and x2.

[0087] 1.3 The face image x and the generated enhanced images x1 and x2 are input into the Encoder module to extract the high-level features of the images, obtaining the low, medium and high three-layer feature maps f1, f2 and f3 of the original image and the high-level features H3 and T3 of the enhanced images; f1, f2 and f3 are fed to the multi-scale feature fusion module; and H3 and T3 are fed to the feature consistency module.

[0088] Further, the step 1.3 sends the balanced training sample and the enhanced image into the Encoder to extract the facial contour feature information to obtain the low, medium and high three-layer feature maps f1, f2, f3 of the original image and the high-layer features H3, T3 of the two enhanced images, which specifically includes:

[0089] 1) The image input into the network is first subjected to a convolution preprocessing operation to obtain a feature map f0 with 64 channels and two enhanced features H0 and T0 with 64 channels;

[0090] 2) The feature map f0 and H0 and T0 are input into the first feature extraction block in the Encoder, which is composed of three consecutive convolution layers, the first two convolution layers are followed by a normalization layer and an activation layer, and the last convolution layer is followed by only a normalization layer. After the processing of the first feature extraction block, three feature maps f1, H1 and T1 with 128 channels and a size of 128*128 are obtained;

[0091] 3) The feature maps f1, H1 and T1 are input into the second feature extraction block in the Encoder, which has the same structure as the first feature extraction block. After the processing of the second feature extraction block, three feature maps f2, H2 and T2 with 128 channels and a size of 64*64 are obtained.

[0092] 4) The feature maps f2, H2 and T2 are input into the second feature extraction block in the Encoder, which has the same structure as the first two feature extraction blocks. After the processing of the third feature extraction block, three feature maps f3, H3 and T3 with 128 channels and a size of 32*32 are obtained. The above process can be represented by the following formula:

[0093] f0=C0(x)#(1)

[0094]

[0095] f i =P(σ(C i (f i-1 )+B i (f i-1 ))),i∈{1,2,3}#(3)

[0096] Wherein, x is the input image, Conv 3*3 is a convolution layer with a convolution kernel of 3*3, B i is the i-th feature extraction module in the Encoder, σ is a nonlinear activation function, P represents a max-pooling layer, f i is the feature obtained after the activation function and the max-pooling layer in the Encoder.

[0097] Further, the step 2 feeds the extracted f1, f2, f3 to the multi-scale feature fusion module for RGB and depth map multi-scale domain migration operation, and calculates the loss of the generator and the discriminator in the domain migration module; H3, T3 are fed to the feature consistency module to calculate the consistency loss. Specifically, the following steps are included:

[0098] 2.1 Compared with the benchmark network DTN of the present application, it only considers a single fixed scale of depth map in the domain migration module, which makes the model in the source domain only learn limited domain knowledge related to the target domain. Different from this, the present application considers the influence of different scale auxiliary modal information on the discrimination performance of the model, introduces the domain migration module after multi-scale feature fusion, so as to help the model learn rich domain knowledge related to the target domain in the source domain, so as to achieve the purpose of making full use of auxiliary modal depth map information and improving the discrimination of the model;

[0099] 2.2 The multi-scale feature maps obtained in step 1 are spliced, first fed into the multi-scale feature fusion module for feature fusion, and then fed into the generator Decoder for depth map prediction;

[0100] 2.3 The predicted depth map and the corresponding size RGB image are spliced in the channel dimension, and then the spliced image is fed into the GAN network to calculate the loss value and update the gradient parameters in the network module.

[0101] Further, the step 2.2 splices the multi-scale feature maps obtained in step 1, first feeds them into the multi-scale feature fusion module for feature fusion, and then feeds them into the generator Decoder for depth map prediction. Specifically, the following steps are included:

[0102] 1) First, use adaptive average pooling operation to change the size of the feature maps f1, f2 to be consistent with f3, and then feed the three feature maps into the multi-scale feature fusion module to obtain a feature map f with a feature dimension of 128*3 and a size of 32*32. con Specifically, it includes:

[0103] 1) Add f1 and f2 to obtain feature f xa Then f xa is fed into the local and global attention mechanisms respectively to obtain local feature f xl and global feature f xg . Add local feature f xl and global feature f xg to obtain f xlg .

[0104] 2) f xlgThe mapping to [0, 1] range gets the fusion weight ω. The input feature f1 and the feature f2 are multiplied by the fusion weight ω and (1-ω) respectively, and the results are added to get the final fusion feature f xo Meanwhile, the same operation is performed on the features f1 and f3 to get the fused feature f xo1 The fused feature is subjected to a channel-level splicing operation to get the final feature f con .

[0105]

[0106] wherein, represents pixel-level summation, and σ represents a Sigmoid function, represents a channel-level splicing operation, L represents local attention, and G represents global attention.

[0107] 3) f con is sent into a generator Decoder composed of three consecutive convolutional layers, each followed by a normalization layer and a nonlinear activation layer, to predict a depth map with a channel number of 1, i.e., to obtain a depth map with a size of 32*32 and a feature dimension of 1 The process is represented in the form of a formula as follows:

[0108]

[0109] wherein, Decoder(.) represents a depth map generator, Down(.) represents a downsampling operation, represents a predicted depth map with a size of 32*32.

[0110] Further, the step 2.3 performs a channel-level splicing operation on the predicted multi-scale depth map and the corresponding size RGB image, and then sends the spliced image into the GAN network to calculate the loss value and update the gradient parameters in the network module, which specifically includes the following steps:

[0111] 2) 1) first perform a channel-level splicing operation on the depth map and the corresponding size RGB image x 32 , so that they do not participate in the gradient calculation, and then send them into the discriminator Dis, respectively, and perform a channel-level splicing operation on the standard depth map d 32 obtained through the PRNet network and the RGB image x 32 , so that they participate in the gradient calculation, and then send them into the discriminator Dis. The GAN loss of the discriminator is calculated, and the above process is represented in the form of a formula as follows:

[0112]

[0113] wherein, Di represents a discriminator that discriminates images of size i*i, x i represents an RGB image of size i*i, d i represents a standard depth map of size i*i, represents a predicted depth map of size i*i.

[0114] 3) Perform a channel-level splicing operation on the depth map and the RGB image x 32 respectively, so that they participate in gradient calculation, and then use the updated discriminator with weight parameters to discriminate the spliced image, and in order to enable the generator to generate a depth map of higher quality, we also supervise the predicted depth map with a scale pyramid loss (SP Loss), which is composed of a pyramid loss and a multi-scale loss, and the specific steps are as follows: first, perform a multi-scale interpolation operation on the target depth map d 32 obtained by the PRNet network and the predicted depth map to adjust them to different scales, and then calculate the square error loss of the interpolation results at each scale to measure the difference between the predicted value and the target value, and accumulate the loss at each scale to obtain the pyramid loss Subsequently, downsample the predicted value and the target value to multiple scales, calculate the L1 loss of the predicted value and the target value at each scale to measure the absolute difference between them, accumulate the loss at each scale, and average to obtain the multi-scale loss Finally, perform weighted summation of the pyramid loss and the multi-scale loss by a weighting factor α to obtain the final scale pyramid loss (SP Loss) Calculate the GAN loss of the generator and the scale pyramid loss (SP Loss), and update the model parameters in the generator Decoder and Encoder. The above process can be represented by the following formula:

[0115]

[0116] where N represents the number of pyramid scales, m i represents the interpolation result of the predicted value at the i-th scale, Y i represents the interpolation result of the target value d 32 at the i-th scale, m scaledk and Y scaledk represent the predicted value and the target value d 32 at the k-th scale, respectively, α represents the weighting factor, MSE represents the mean square error loss function, and L1 represents the L1 loss function.

[0117] Further, the step 3 sends the features extracted under two different perspectives into a feature consistency module, and calculates the feature consistency loss under two different perspectives through a dense similarity loss, specifically including the following steps:

[0118] 2) First, the features H3 and T3 extracted under two perspectives are sent into a predictor (Prediction) to further process and predict the input features. The predictor is a neural network structure with two hidden layers. The first hidden layer includes a sequence of convolution, batch normalization and ReLU activation function. The sequence first uses a 1x1 convolution kernel to convert the input features from 128 dimensions to 128 dimensions of the hidden layer, then performs batch normalization on the output, and finally uses the ReLU activation function for nonlinear transformation. The second hidden layer is a 1x1 convolution layer that converts the output of the first hidden layer from 128 dimensions to 128 dimensions of the output. Finally, the final two flattened tensors where s 2 represents the spatial resolution of the feature map, and d represents the dimension of each feature vector; the similarity loss between the feature maps under two different perspectives is calculated through a dense similarity loss; the dense similarity loss calculates the dense similarity between two tensors, considering the relative relationship between each element, rather than just the similarity between the global average vectors, which can better capture the relationship between local features. The above process is represented by the following formula:

[0119]

[0120] where H i represents the i-th row of the feature map H, and T j represents the j-th column of the feature map T.

[0121] 3.1 After completing the steps in 3 1), the feature information is extracted again using the Encoder, and the last level feature f3 is obtained;

[0122] 3.2 The last level feature f3 is sent into the classifier to perform classification operation on the sample and calculate the classification loss;

[0123] The step 3.2 sends the aggregated feature f3 into the classifier to perform classification operation on the sample and calculate the classification loss, specifically including the following steps:

[0124] 1) The feature f3 extracted by the Encoder is sent to the classifier, which is composed of 3 convolutional layers, 2 max-pooling layers, 1 global average pooling layer and 1 fully connected layer, and each convolutional layer is followed by a normalization layer and a nonlinear activation layer; the classifier is used for the final discrimination operation of the input image, that is, to judge whether the input face image is real or attack;

[0125] 2) The label value of the real sample is set to 1, and the label value of the attack sample is set to 0; through this type of label value setting, the discrimination operation of real and false samples can be regarded as a binary classification operation; the classifier predicts a score value for each sample, and the predicted value is supervised by a binary cross-entropy loss function, and finally the cross-entropy loss value is used to update the weight parameters in the classifier and the Encoder. The above process can be expressed as follows:

[0126]

[0127] wherein, represents the predicted value of the classifier for the sample, and y represents the label value of the sample.

[0128] Further, the loss function in step 2.3 1) and the loss function in step 2.3 2) are added to the loss function in step 3 1) to form the final total loss function, which specifically includes:

[0129]

[0130] wherein, λ1, λ2, λ3, λ4 represent the weighted hyperparameters of the total loss function.

[0131] The application designs a new domain migration module by introducing multi-scale feature fusion and scale pyramid loss and feature consistency module into the domain migration module, so as to enable the model to learn rich domain knowledge related to the target domain in the source domain, achieve the purpose of fully utilizing the auxiliary modal information of the depth map, and enable the model to learn more generalization features, which to some extent enhances the robustness and generalization ability of the model. The application designs a new MFFM, which can extract multi-scale features by integrating different scale feature information, and enhances the understanding and learning of the model for multi-scale depth maps. Compared with the traditional single-scale feature extraction, MFFM can capture more detailed and comprehensive features, including global semantic information and local details. Through this fusion, the model can generate more accurate and information-rich depth maps, thereby showing higher robustness and accuracy in the face anti-spoofing task. In addition, the FCM model designed by the application can maintain the consistency of features between different angles, ensuring the stability and reliability of feature representation. This consistency is crucial for improving the accuracy of the model in practical applications, especially when facing different pose changes and light conditions. FCM improves the discriminability of the model for real faces and spoof attacks by encouraging consistency in the feature space. At the same time, the SP Loss loss designed by the application evaluates the model's prediction at multiple scales, ensuring the model's comprehensive perception ability for different scale information. The design of SP Loss not only improves the model's sensitivity to details, but also strengthens the understanding of the overall image structure. The introduction of this loss function enables the model to pay more attention to generating high-quality, high-resolution depth maps during the training process, thereby being more accurate and robust in actual anti-spoofing tasks.

[0132] The systems, devices, modules or units illustrated in the above embodiments can be specifically implemented by computer chips or entities, or by products with certain functions.

[0133] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can implement information storage by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0134] It should also be noted that the terms "comprising", "comprising" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or apparatus including a series of elements includes not only those elements, but also other elements not explicitly listed or inherent to such a process, method, article or apparatus. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus including the element.

[0135] The above embodiments should be understood as only for illustrating the present application and not for limiting the protection scope of the present application. After reading the content of the present application, the skilled in the art can make various changes or modifications to the present application, and these equivalent changes and modifications also fall within the scope defined by the claims of the present application.

Claims

1. A face anti-spoofing method based on feature consistency and scale pyramid loss, characterized in that, Includes the following steps: Step 1: First, the input face image is passed through a liveness detection network and then through a feature enhancement module to generate images from two different perspectives. Then, these two images and the original face image are passed through an encoder module to extract initial features. The features extracted from the low, medium and high levels of the encoder are then fed into a multi-scale feature fusion module for fusion. At the same time, the features extracted from the two different perspectives are fed into a feature consistency module to ensure their consistency in the feature space. Step 2: The multi-scale feature fusion module fully fuses the low, medium and high-level features and then feeds them into the network's Decoder module to perform domain transfer operations between the RGB and depth maps, and calculates the discriminator loss and scale pyramid loss in the domain transfer module. Step 3: Input the features extracted from the two different perspectives into the feature consistency module, and calculate the feature consistency loss between the two different perspectives through density similarity loss; Finally, the discriminator loss and the scale pyramid loss are combined to form the total loss, and the model parameters are updated through iterative adversarial training. Step 4: Input the high-level features of the real samples extracted from the Encoder part of the original image after parameter update into the classifier for liveness classification.

2. The face anti-spoofing method based on feature consistency and scale pyramid loss according to claim 1, characterized in that, Step 1 specifically includes the following steps: 1.1 First, perform a sample balancing operation on the images input into the training network, that is, keep the ratio of real to fake samples in each training session at 1:

1. 1.2 Next, the balanced training samples are augmented to generate two enhanced images from different perspectives. The face image x is augmented using a series of data augmentation methods, including random cropping, random shuffling of regions, rotation, scaling, translation, and random erasure, to generate two enhanced images x1 and x2. 1.3 Input the face image x and the generated enhanced images x1 and x2 into the Encoder module to extract the high-level features of the image, and obtain the low, medium and high-level feature maps f1, f2 and f3 of the original image and the high-level features H3 and T3 of the enhanced image; feed f1, f2 and f3 to the multi-scale feature fusion module; feed H3 and T3 to the feature consistency module.

3. The face anti-spoofing method based on feature consistency and scale pyramid loss according to claim 2, characterized in that, Step 1.3 involves feeding the balanced training samples and the enhanced images into the Encoder to extract facial contour feature information, thereby obtaining the low, medium, and high-level feature maps f1, f2, and f3 of the initial image and the high-level features H3 and T3 of the two enhanced images. Specifically, this includes: 1) First, perform convolution preprocessing on the image input into the network to obtain a feature map f0 with 64 channels and two enhanced features H0 and T0 with 64 channels each; 2) Input feature maps f0, H0, and T0 into the first feature extraction block in the Encoder. The first feature extraction block consists of three consecutive convolutional layers. The first two convolutional layers are followed by a normalization layer and an activation layer, while the last convolutional layer is followed by only a normalization layer. After processing by the first feature extraction block, a feature map f1, H1, and T1 with three channels of 128 each and a size of 128*128 is obtained. 3) The feature maps f1, H1, and T1 are fed into the second feature extraction block in the Encoder. The construction of this block is the same as that of the first feature extraction block. After processing by the second feature extraction block, three feature maps f2, H2, and T2 with 128 channels and a size of 64*64 are obtained. 4) The feature maps f2, H2, and T2 are fed into the second feature extraction block in the Encoder. This block is constructed in the same way as the first two feature extraction blocks. After processing by the third feature extraction block, three feature maps f3, H3, and T3 with 128 channels and a size of 32*32 are obtained. The above process can be expressed as the following formula: f0=C0(x)#(1) f i =P(σ(C i (f i-1 )+B i (f i-1 ))),i∈{1,2,3}#(3) Where x is the input image, Conv 3*3 It is a convolutional layer with a 3x3 kernel, B i This is the i-th feature extraction module in the Encoder, where σ is a non-linear activation function, P represents the max pooling layer, and f i These are features obtained after processing by activation functions and max pooling layers in the Encoder.

4. The face anti-spoofing method based on feature consistency and scale pyramid loss according to claim 1, characterized in that, Step 2 involves feeding the extracted f1, f2, and f3 to the multi-scale feature fusion module for multi-scale domain transfer operations between the RGB and depth maps, and calculating the losses of the generator and discriminator in the domain transfer module; H3 and T3 are fed to the feature consistency module to calculate the consistency loss, specifically including the following steps: 2.1 By fusing multi-scale features and introducing them into the domain transfer module, the model can learn rich domain knowledge related to the target domain in the source domain; 2.2 The multi-scale feature maps are stitched together and first fed into the multi-scale feature fusion module for feature fusion, and then fed into the Decoder generator for depth map prediction; 2.3 The predicted depth map and the corresponding RGB image are stitched together along the channel dimension. The stitched image is then fed into the GAN network to calculate the loss value and update the gradient parameters in the network module.

5. The face anti-spoofing method based on feature consistency and scale pyramid loss according to claim 4, characterized in that, Step 2.2 involves stitching together the obtained multi-scale feature maps, first feeding them into the multi-scale feature fusion module for feature fusion, and then feeding them into the Decoder generator for depth map prediction. Specifically... Includes the following steps: 1) First, adaptive average pooling is used to make the dimensions of feature maps f1 and f2 the same as those of f3. Then, these three feature maps are fed into the multi-scale feature fusion module to obtain a feature map f with a feature dimension of 128*3 and a size of 32*32. con ; 2) Add f1 and f2 together to obtain the feature f. xa Then f respectively xa The data is fed into local and global attention mechanisms to obtain local features f. xl and global features f xg , local features f xl and global features f xg Adding them together gives f xlg ; 3) Use the Sigmoid function to convert f xlg Mapping to the range [0,1] yields the fusion weight ω; multiplying the input features f1 and f2 by the fusion weight ω and (1-ω) respectively, and summing the results, yields the final fusion feature f. xo Simultaneously, the same operation is performed on features f1 and f3 to obtain the fused feature f. xo1 The fused features are then subjected to channel-level concatenation to obtain the final feature f. con ; in, This represents pixel-level summation, and σ represents the Sigmoid function. L represents channel-level splicing operation, and G represents local attention. 4) f con The data is fed into a generator (Decoder) consisting of three consecutive convolutional layers. Each convolutional layer is followed by a normalization layer and a non-linear activation layer to predict a depth map with one channel, resulting in a 32x32 depth map with a feature dimension of 1. The process can be expressed as a formula as follows: Here, Decoder(.) represents the depth map generator, and Down(.) represents the downsampling operation. This represents a predicted depth map with dimensions of 32*32.

6. The face anti-spoofing method based on feature consistency and scale pyramid loss according to claim 4, characterized in that, Step 2.3 involves stitching the predicted multi-scale depth map with the corresponding RGB image in the channel dimension, then feeding the stitched image into the GAN network to calculate the loss value and update the gradient parameters in the network module. Specifically, this includes the following steps: 1) First, create a depth map. RGB image x of corresponding size 32 Perform channel-level stitching to prevent it from participating in gradient calculation, and then feed them separately into the discriminator Dis. This will then combine the standard depth map d obtained through the PRNet network. 32 With RGB image x 32 Perform channel-level concatenation to participate in gradient calculation, and then feed the results into the discriminator Dis; calculate the GAN loss of the discriminator, and express the above process as the following formula: Among them, D i x represents a discriminator that performs discrimination on an image of size i*i. i Represents an RGB image of size i*i, d i A standard depth map of size i*i. This represents the predicted depth map of size i*i; 2) Depth map Compared with RGB image x respectively 32 Channel-level stitching is performed to participate in gradient calculation. The discriminator, updated with weight parameters, is used to distinguish the stitched image. Scale pyramid loss (SP Loss) is also used to supervise the predicted depth map. The scale pyramid loss is composed of pyramid loss and multi-scale loss.

7. The face anti-spoofing method based on feature consistency and scale pyramid loss according to claim 6, characterized in that, The predicted depth map is supervised using Scale Pyramid Loss (SP Loss), which consists of pyramid loss and multi-scale loss. The specific steps are as follows: First, the target depth map d obtained by the PRNet network 32 and the predicted depth map Multi-scale interpolation operations are performed to adjust the values ​​to different scales. Then, the variance loss is calculated for the interpolation results at each scale to measure the difference between the predicted and target values. The losses at each scale are summed to obtain the pyramid loss. Subsequently, the predicted and target values ​​are downsampled to multiple scales. For each scale, the L1 loss is calculated to measure the absolute difference between the predicted and target values. The losses from each scale are then summed and averaged to obtain the multi-scale loss. Finally, the pyramids were lost. and multi-scale loss The final scale pyramid loss is obtained by weighting and summing using the weighting factor α. The GAN loss and scale pyramid loss of the generator are calculated, and the model parameters in the Decoder and Encoder of the generator are updated.

8. The face anti-spoofing method based on feature consistency and scale pyramid loss according to claim 1, characterized in that, Step 3 involves feeding the features extracted from two different perspectives into the feature consistency module, and calculating the feature consistency loss between the two different perspectives using density similarity loss. This specifically includes the following steps: 1) First, the features H3 and T3 extracted from the two perspectives are fed into the predictor for further processing and prediction. The predictor is a neural network structure with two hidden layers. The first hidden layer contains a sequence of convolution, batch normalization, and ReLU activation functions. This sequence first uses a 1×1 convolution kernel to transform the input features from 128 dimensions to 128 dimensions of the hidden layer, then performs batch normalization on the output, and finally uses the ReLU activation function for non-linear transformation. The second hidden layer is a 1×1 convolution layer that transforms the output of the first hidden layer from 128 dimensions to 128 dimensions of the output. Finally, the final two flattened tensors H3 and T3 are obtained. Where s 2 d represents the spatial resolution of the feature map, while d represents the dimension of each feature vector. The similarity loss between feature maps from two different perspectives is calculated using dense similarity loss. Dense similarity loss calculates the dense similarity between two tensors, considering the relative relationships between each element, rather than just the global average similarity between vectors. This better captures the relationships between local features. The above process can be expressed as the following formula: Among them, H i T represents the feature of the i-th row of feature map H. j This represents the j-th column of the feature map T; The feature information can be extracted again using the Encoder to obtain the last level feature f3; The last-level feature f3 is fed into the classifier to classify the samples and calculate the classification loss.

9. The face anti-spoofing method based on feature consistency and scale pyramid loss according to claim 8, characterized in that, The process of feeding the last-level feature f3 into the classifier to classify the samples and calculate the classification loss includes the following steps: 1) The feature f3 extracted by the Encoder is fed into the classifier, which consists of 3 convolutional layers, 2 max pooling layers, 1 global average pooling layer and 1 fully connected layer. Each convolutional layer is followed by a normalization layer and a non-linear activation layer. The classifier is used to perform the final discrimination operation on the input image, that is, to determine whether the input face image is real or malicious. 2) Set the label value of the real sample to 1 and the label value of the attack sample to 0. By setting the label value in this way, the discrimination between real and fake samples can be regarded as a binary classification operation. Let the classifier predict a score value for each sample, and use the binary cross-entropy loss function to supervise the predicted value. Finally, update the weight parameters in the classifier and encoder with the cross-entropy loss value. The above process can be expressed by the following formula: in, y represents the classifier's predicted value for the sample, and y represents the sample's label value.

10. The face anti-spoofing method based on feature consistency and scale pyramid loss according to claim 9, characterized in that, The final total loss function specifically includes: Where λ1, λ2, λ3, and λ4 represent the weighted hyperparameters of the total loss function.

Images