Liveness detection method and device based on patch amplitude and entropy guidance strategy

Abstract

The present invention discloses a liveness detection method and device based on patch amplitude and entropy guidance strategy. The SE module is used to screen out k significant patches in a picture. For each significant patch, its amplitude is randomly mixed with the amplitude of patches of other pictures of the same category and combined with the phase information of the significant patch to generate diversified data. With the continuous generation of diversified patches, the liveness detection model is indirectly encouraged to use information in other areas of the face to judge true and false faces. In addition, the present invention uses an entropy-guided difficult example mining strategy to mine difficult examples, and can dynamically adjust the importance weight of the sample image according to the total amount of information contained in the token, so that the liveness detection model can still combine the remaining feature information to make a robust judgment when the key area of ​​the face is blocked. The present invention can alleviate the overfitting problem of the liveness detection model and improve the generalization ability of the liveness detection model.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision, and in particular to a method and device for detecting a living body based on a patch amplitude and entropy guidance strategy. Background Art

[0002] Liveness detection is a method used to verify identity, which aims to determine the true physiological characteristics of the subject. Liveness detection plays an important role in face recognition and can effectively resist various common attack methods, such as fraud using photos, videos, face-changing technology, masks, occlusion, 3D animation, and screen reshoots. By performing liveness detection on users, their authenticity can be ensured, thereby protecting the interests and safety of users.

[0003] With the widespread application of transformers in the field of computer vision, the performance of liveness detection methods based on vision transformers (ViT) has been greatly improved. Such methods can learn highly discriminative features from large-scale face datasets. Although ViT-based liveness detection technology has made significant progress, there are still some defects, such as:

[0004] Since the ViT-based model lacks an inductive bias similar to the convolutional layer, the model is difficult to train and prone to overfitting. To solve this problem, existing methods have tried different data augmentation methods (such as random erasing, cutout, color jitter, gamma contrast adjustment, etc.) to construct diverse training sample images, but these data augmentation methods inevitably destroy the key structural information of the face;

[0005] During the training process, the ViT-based liveness detection model tends to overfit certain areas (such as eyes and forehead). Once the key areas are blocked (such as wearing glasses or hats), the model will make incorrect classifications, resulting in poor generalization performance of the model.

[0006] The hard example mining strategies used by previous ViT models are designed for convolutional neural networks. These strategies usually use instance-level metrics of sample images (such as sample image prediction probability) to mine hard examples, while ViT's predictions are determined by the dominant tokens. Therefore, directly using biased metrics to mine hard examples is suboptimal.

[0007] In view of this, the present invention is deeply conceived to address the many deficiencies and inconveniences caused by the imperfection of the existing ViT-based liveness detection technology, and actively researches, improves and tests to develop the present invention. Summary of the invention

[0008] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a liveness detection method based on patch amplitude and entropy guidance strategy, which can alleviate the overfitting problem of the liveness detection model and improve the generalization ability of the liveness detection model.

[0009] In order to achieve the above object, the solution of the present invention is:

[0010] A method for liveness detection based on patch amplitude and entropy guidance strategy comprises the following steps:

[0011] Obtain a facial image of the user to be detected, pre-process the facial image, and then input it into a liveness detection model deployed in the face recognition system;

[0012] The liveness detection model performs reasoning on the input face image;

[0013] The liveness detection model outputs a liveness score in the shape of [M, 2]. The category corresponding to the maximum score is taken as the predicted category of the liveness detection model. According to the predicted category, it is judged whether the input face image is alive and output to the face recognition system; where:

[0014] The liveness detection model uses the transformer architecture. During the learning of the liveness detection model, the significant patch amplitude perturbation strategy and the entropy-guided difficult example mining strategy are introduced to accurately distinguish between real and fake faces in the image. The learning of the liveness detection model includes the following steps:

[0015] Step 1: The input sample image is passed through the convolution module and then added to the position embedding information; then, it is further extracted through the transformer encoder;

[0016] Step 2: The salient patch amplitude perturbation strategy uses the SE module to further mine the prior knowledge provided by the liveness detection model to obtain salient patch information; then, randomly select patch information of the same category of sample images, and the salient patch information and the randomly selected patch information are transformed through Fourier transform and inverse Fourier transform to obtain the reconstructed image;

[0017] Step 3: After obtaining the reconstructed image, use the reconstructed image as the input of the living body detection model, perform the processing of step 1 again, and divide the output into two paths, one of which enters step 4 and the other enters step 5;

[0018] Step 4: The entropy-guided hard example mining strategy extracts global information from the output of step 3 and dynamically adjusts the weight of the sample image according to the total amount of information;

[0019] Step 5. After the output of step 3 passes through the fully connected layer, calculate the cross entropy loss;

[0020] Step 6: Multiply the cross entropy loss in step 5 by the weight of the sample image obtained in step 4 to get the final loss.

[0021] Furthermore, in step 2, the steps of combining random patches with significant patches are as follows:

[0022] Step A1, performing Fourier transform on the significant patch to obtain amplitude and phase information;

[0023] Step A2, randomly mixing the amplitude of the salient patch with the amplitude of randomly selected patches of the same category of the image;

[0024] Step A3, combining the mixed amplitude with the phase information of the original significant patch, performing inverse Fourier transform, and generating a new patch;

[0025] Step A4: re-embed the salient patches after the amplitude perturbation processing into the original image to generate a new diversified image, that is, generate a reconstructed image.

[0026] Furthermore, the specific process of the significant patch amplitude perturbation strategy includes the following steps:

[0027] Step B1, for the input feature map of shape [N, Embed_size] ( ) to reshape the feature shape, and the reshaped feature is , the shape is [N× Embed_size], where N is the number of patches and Embed_size is the length of the embedding feature vector;

[0028] Step B2: Features After the SE module, ReLU activation layer, fully connected layer, and Sigmoid activation layer, the output features are obtained. , the shape is [N],

[0029]

[0030] Step B3: Filter out the first k significant patches from the N patches, and perform the following processing on the k significant patches respectively:

[0031] Step C1: Map the index of the salient patch back to the coordinate system of the original image to determine the position of the salient patch in the original image. According to the mapped coordinates, extract the image information of these areas from the original image. Assume Indicates the regional location information of the significant patch k mapped back to the original image, is the original image, is the image information corresponding to the salient patch,

[0032]

[0033] Step C2: Assumptions The width and height of Perform Fourier transform and get ,

[0034]

[0035] in, is an imaginary unit; is the spatial domain coordinate of the image; is the frequency domain coordinate after Fourier transformation, e is the natural base, is the ratio of pi;

[0036] Step C3: randomly select an image from the same category of images and randomly select a patch from the randomly selected image. Assume that the image information corresponding to the patch is , similarly, Perform Fourier transform and get ;

[0037] Step C4: Obtain and The magnitude and phase of

[0038]

[0039]

[0040]

[0041]

[0042] in, is the amplitude of the Fourier transform result; is the phase of the Fourier transform result; is the real part of the Fourier transform result; is the imaginary part of the Fourier transform result; : Inverse tangent function, express The imaginary part of express The real part of

[0043] Step C5: Random Mixing and The amplitude information of the mixed patch is combined with the phase corresponding to the significant patch to obtain a new complex number ,

[0044]

[0045]

[0046] in, is the amplitude after mixing; is a random number between 0 and 1; represents the phase, i is the imaginary unit, and e is the natural base;

[0047] Step C6: Perform inverse Fourier transform, Convert the image back to the spatial domain ,

[0048]

[0049] Step C7: Embedded into the corresponding position of the original image, we get the reconstructed image.

[0050] Furthermore, the specific process of the entropy-guided hard example mining strategy includes the following steps:

[0051] Step D1: Use the output of the tansformer encoder as the input of the entropy-guided hard example mining strategy ( ), the shape is [N×Embed_size], where N represents the number of patches, Embed_size represents the length of the embedding feature vector, and reshapes the input feature. The reshaped feature is , shape is [N,Embed_size];

[0052] Step D2: Along the feature The last dimension of , find the standard deviation of the feature, and get the feature , shape is [N], along the last dimension, take features The mean of (This feature as information entropy), with shape [1],

[0053]

[0054]

[0055] in, Indicates the standard deviation of the calculated feature; Indicates the mean of the calculated feature; Indicates related operations along the last dimension;

[0056] Step D3: The entropy value is transformed by a negative exponential function to obtain the weight of the sample image.

[0057]

[0058] The simpler the sample image, the greater the entropy value and the more information it contains. The more difficult the sample image, the smaller the entropy value and the less information it contains. Through the above formula, it is ensured that the weight of sample images with higher entropy values ​​is lower, while the weight of sample images with lower entropy values ​​is higher.

[0059] Furthermore, during the training of the liveness detection model, before the sample image is input into the liveness detection model to be trained, it is randomly horizontally flipped, and then the input image is normalized and standardized to limit the pixel value of the image to between 0 and 1.

[0060] Furthermore, the training cycle of the liveness detection model is 100 epochs, the batch size is 256, the learning rate is 0.1, the optimizer is SGD, the momentum is 0.9, and the learning rate adjustment strategy is StepLR. At the 20th, 40th, 60th, and 80th epochs, the learning rate is reduced to half of the original.

[0061] Furthermore, the forward propagation of the liveness detection model is divided into two parts. The first part does not need to calculate the gradient information. The liveness detection model is used to obtain the location information of the significant patches, that is, the prior knowledge described in step 2, and reconstruct the input image. In the second part, the input is the reconstructed image. The fully connected layer module is connected after the transformer encoder. The fully connected layer module contains two fully connected layers and two batch normalization layers. Through the two fully connected layers, the input features are gradually reduced in dimension and transformed to meet the requirements of the classification task. The two batch normalization layers normalize the features. After forward propagation, the features are mapped to the final classification dimension.

[0062] Furthermore, in step 5, the cross entropy loss The calculation formula is as follows,

[0063]

[0064] in, is the number of sample images; A is the sample image label; Prediction value of the liveness detection model.

[0065] The final loss is obtained by multiplying the calculated cross entropy loss with the weight of the sample image obtained from the entropy-guided hard example mining strategy. , the calculation formula is as follows,

[0066] .

[0067] Furthermore, the loss function The optimization is divided into two parts: (1) minimizing , which can enable the liveness detection model to better learn discriminative features from diverse sample images; (2) minimize , that is, maximizing the entropy value, encouraging the liveness detection model to fully explore the information contained in each token and significantly enhancing the feature expression ability of the token.

[0068] Another object of the present invention is to provide a liveness detection device based on patch amplitude and entropy guidance strategy, which can alleviate the overfitting problem of the liveness detection model and improve the generalization ability of the liveness detection model.

[0069] In order to achieve the above object, the solution of the present invention is:

[0070] A liveness detection device based on patch amplitude and entropy guidance strategy, including an image acquisition module and a liveness detection model;

[0071] The image acquisition module acquires a face image of a user to be detected, pre-processes the face image, and then inputs it into a liveness detection model deployed in the face recognition system;

[0072] The liveness detection model uses the transformer architecture, including an image preprocessing module, a significant patch amplitude perturbation module, a hard example mining module, a cross entropy loss calculation module, an inference module, and an output judgment module.

[0073] The liveness detection model uses the transformer architecture. During the learning of the liveness detection model, the significant patch amplitude perturbation strategy and the entropy-guided difficult example mining strategy are introduced to accurately distinguish between real and fake faces in the image. The learning of the liveness detection model includes the following steps:

[0074] The image preprocessing module preprocesses the input image by first performing a convolution module and then adding the position embedding information; then, the information is further extracted through the transformer encoder;

[0075] The salient patch amplitude perturbation module uses the SE module to further mine the prior knowledge provided by the liveness detection model to obtain salient patch information; randomly select patch information of sample images of the same category, and obtain a reconstructed image after Fourier transform and inverse Fourier transform of the salient patch information and the randomly selected patch information. The reconstructed image is used as the input of the liveness detection model, and is preprocessed again by the image preprocessing module, and the output is divided into two paths, one of which enters the difficult example mining module, and the other enters the cross entropy loss calculation module through the fully connected layer;

[0076] The hard example mining module extracts global information and dynamically adjusts the weight of sample images according to the total amount of information;

[0077] The cross entropy loss calculation module is multiplied by the weight of the sample image of the hard example mining module to obtain the final loss;

[0078] The inference module performs inference on the input face image;

[0079] The output judgment module outputs a liveness score in the shape of [M, 2]. The category corresponding to the maximum score is taken as the predicted category. Based on the predicted category, it is judged whether the input face image is live and output to the face recognition system.

[0080] After adopting the above scheme, compared with the existing technology, the liveness detection method and device based on patch amplitude and entropy guidance strategy of the present invention uses SE module (squeeze-and-excitation module, a kind of attention mechanism) to screen out k significant patches in the picture. For each significant patch, its amplitude will be randomly mixed with the amplitude of patches of other pictures of the same category and combined with the phase information of the significant patch to generate diversified data. With the continuous generation of diversified patches, the liveness detection model is indirectly encouraged to use other regional information of the face to judge true and false faces. In addition, the present invention uses the entropy-guided difficult example mining strategy to mine difficult examples, which can dynamically adjust the importance weight of the sample image according to the total amount of information contained in the token, so that the liveness detection model can still combine the remaining feature information to make a robust judgment when the key area of ​​the face is blocked.

[0081] The present invention uses an amplitude perturbation strategy for salient patches to replace commonly used instance-level data augmentation strategies (such as random erasing, cutout, etc.), obtains the location of salient patches through the prior knowledge provided by the liveness detection model and performs amplitude perturbations on them, thereby increasing data diversity, alleviating the overfitting problem of the liveness detection model, and improving the generalization ability of the liveness detection model.

[0082] The liveness detection method and device based on patch amplitude and entropy guidance strategy of the present invention use an entropy-guided difficult example mining strategy to mine difficult examples, avoid being dominated by local tokens, and enable the liveness detection model to dynamically adjust the weight of the sample image according to the global information, thereby enhancing the generalization performance of the liveness detection model.

[0083] The two strategies of the liveness detection method and device based on the patch amplitude and entropy guidance strategy of the present invention can be seamlessly and conveniently introduced into the existing liveness detection model framework, effectively improving the generalization ability of the liveness detection model. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] Figure 1 It is a schematic diagram of the overall framework of the present invention.

[0085] Figure 2 It is a schematic diagram of patch information fusion of the present invention.

[0086] Figure 3 Flow chart of the significant patch value perturbation strategy of the present invention.

[0087] Figure 4 Flowchart of the strategy for guiding difficult example mining in the present invention.

[0088] Figure 5 Flowchart of the fully connected layer of the present invention. DETAILED DESCRIPTION

[0089] In order to further explain the technical solution of the present invention, the present invention is described in detail below through specific embodiments.

[0090] like Figure 1 As shown, the present invention discloses a liveness detection method based on patch amplitude and entropy guidance strategy, which uses a transformer structure, introduces a significant patch amplitude perturbation strategy and an entropy-guided difficult example mining strategy to accurately distinguish between true and false faces in an image, and specifically includes the following steps:

[0091] Step 1: The input image is passed through the convolution module and then added to the position embedding information; then, it is further extracted through the transformer encoder;

[0092] Step 2: The salient patch amplitude perturbation strategy uses the SE module to further mine the prior knowledge (output of the transformer encoder) provided by the liveness detection model to obtain salient patch information; then, randomly select patch information of the same category of images, and obtain the reconstructed image after Fourier transform and inverse Fourier transform of the salient patch information and the randomly selected patch information to increase the diversity of the sample image, which can effectively alleviate the overfitting problem of the liveness detection model. The fusion of patch information is as follows: Figure 2 shown.

[0093] Step 3: After obtaining the reconstructed image, use the reconstructed image as the input of the living body detection model, perform the processing of step 1 again, and divide the output into two paths, one of which enters step 4 and the other enters step 5;

[0094] Step 4: The entropy-guided hard example mining strategy extracts global information from the output of step 3 and dynamically adjusts the weight of the sample image according to the total amount of information. In this way, the liveness detection model is encouraged to focus on tokens with less information, avoiding the risk of the liveness detection model being dominated by key tokens.

[0095] Step 5: After the output of step 3 passes through the fully connected layer, the cross entropy loss can be calculated. The obtained loss is multiplied by the weight of the sample image obtained in step 4 to obtain the final loss.

[0096] By introducing the significant patch amplitude perturbation data enhancement strategy and the entropy-guided difficult example mining strategy, the liveness detection method of the present invention can diversify sample images and make full use of the information in tokens to improve the ability to distinguish between real and fake faces, ultimately improving the user experience in the face recognition system.

[0097] Next, the salient patch amplitude perturbation strategy, entropy-guided hard example mining strategy, and the liveness detection model training and reasoning process will be introduced in detail.

[0098] The present invention uses a significant patch amplitude perturbation strategy to increase data diversity by perturbing the significant areas of the image, alleviate the overfitting problem of the liveness detection model, and improve the generalization ability of the liveness detection model. Specifically, it uses the SE module to filter out significant patches in the image from the output of the transformer encoder. The significant patches represent important areas in the image, which have an important impact on the classification decision of the liveness detection model. Subsequently, patch information of images of the same category is randomly selected and combined with the selected significant patches. Figure 2 As shown in Figure 2, the steps of combining random patches with significant patches are as follows:

[0099] Step A1, performing Fourier transform on the significant patch to obtain amplitude and phase information;

[0100] Step A2, randomly mixing the amplitude of the salient patch with the amplitude of randomly selected patches of the same category of the image;

[0101] Step A3, combining the mixed amplitude with the phase information of the original significant patch, performing inverse Fourier transform, and generating a new patch;

[0102] Step A4: Re-embed the significant patches after amplitude perturbation into the original image to generate a new diversified image, that is, generate a reconstructed image. The reconstructed image retains the overall structure of the original image, while introducing new feature information, thereby increasing the diversity of the data.

[0103] like Figure 3 As shown in FIG. 1 , the specific process of the significant patch amplitude perturbation strategy includes the following steps:

[0104] Step B1, for the input feature map of shape [N, Embed_size] ( ) to reshape the feature shape, and the reshaped feature is , the shape is [N× Embed_size], where N represents the number of patches and Embed_size represents the length of the embedding feature vector.

[0105] Step B2: Features After the SE module (the SE module includes fully connected layer 1 (FC1), ReLU activation layer, fully connected layer 2 (FC2), Sigmoid activation layer), the output features are obtained , the shape is [N], FC1 and FC2 are fully connected layers,

[0106]

[0107] Step B3: Filter out the first k significant patches from the N patches. Perform the following processing on the k significant patches:

[0108] Step C1: Map the index of the salient patch back to the coordinate system of the original image to determine the position of the salient patch in the original image. According to the mapped coordinates, extract the image information of these areas from the original image. Assume Indicates the regional location information of the significant patch k mapped back to the original image, is the original image, is the image information corresponding to the salient patch,

[0109]

[0110] Step C2: Assumptions The width and height of Perform Fourier transform and get ,

[0111]

[0112] in, is an imaginary unit; is the spatial domain coordinate of the image; is the frequency domain coordinate after Fourier transformation, e is the natural base, is the ratio of pi.

[0113] Step C3: randomly select an image from the same category of images and randomly select a patch from the randomly selected image. Assume that the image information corresponding to the patch is , similarly, Perform Fourier transform and get .

[0114] Step C4: Obtain and The magnitude and phase of

[0115]

[0116]

[0117]

[0118]

[0119] in, is the amplitude of the Fourier transform result; is the phase of the Fourier transform result; is the real part of the Fourier transform result; is the imaginary part of the Fourier transform result; : Inverse tangent function, express The imaginary part of express The real part of .

[0120] Step C5: Random Mixing and The amplitude information of the mixed patch is combined with the phase corresponding to the significant patch to obtain a new complex number ,

[0121]

[0122]

[0123] in, is the amplitude after mixing; is a random number between 0 and 1, Represents the phase, i is the imaginary unit, and e is the natural base.

[0124] Step C6: Perform inverse Fourier transform, Convert the image back to the spatial domain ,

[0125]

[0126] Step C7: Embedded into the corresponding position of the original image, we get the reconstructed image.

[0127] The entropy-guided hard case mining strategy is an important method used by the present invention to enhance the generalization ability of the liveness detection model based on ViT. The entropy-guided hard case mining strategy dynamically adjusts the importance weights of sample images, so that the liveness detection model can more effectively utilize global information during the training process, avoid being dominated by local areas, and thus improve the robustness of the liveness detection model. Figure 4 As shown in Figure 2, the specific process of the entropy-guided difficult example mining strategy is as follows:

[0128] Step D1: First, the output of the tansformer encoder is used as the input of the entropy-guided hard example mining strategy ( ), the shape is [N×Embed_size], where N is the number of patches and Embed_size is the length of the embedding feature vector. Reshape the input feature, and the reshaped feature is , with a shape of [N,Embed_size].

[0129] Step D2: Follow The last dimension of , find the standard deviation of the feature, and get the feature , shape is [N]. Along the last dimension, take the features The mean of (This feature as information entropy), with shape [1],

[0130]

[0131]

[0132] in, Indicates the standard deviation of the calculated feature; Indicates the mean of the calculated feature; Indicates a correlation operation along the last dimension.

[0133] Step D3: The entropy value is transformed by a negative exponential function to obtain the weight of the sample image.

[0134]

[0135] The simpler the sample image, the greater the entropy value (more information), and the more difficult the sample image, the smaller the entropy value (less information). Through the above formula, it is ensured that the weight of samples with higher entropy values ​​is lower, while the weight of sample images with lower entropy values ​​is higher, which can encourage the liveness detection model to pay more attention to sample images with less information during training, thereby improving the robustness of the liveness detection model.

[0136] The purpose of adding 1 in the above formula is to ensure that the weight is always greater than 1, so that all sample images can have sufficient influence on the loss function during training.

[0137] During the training of the liveness detection model, the sample image only needs to be randomly horizontally flipped before being input into the liveness detection model to be trained to improve the richness of the training samples. Then, the input image is normalized and standardized to limit the pixel values ​​of the image to between 0 and 1.

[0138] The training cycle of the liveness detection model is 100 epochs (when the liveness detection model is trained to 100 epochs, the loss of the liveness detection model has converged), the batch size is 256, the learning rate is 0.1, the optimizer is SGD, the momentum is 0.9, and the learning rate adjustment strategy is StepLR. At the 20th, 40th, 60th, and 80th epochs, the learning rate is reduced to half of the original.

[0139] The forward propagation of the liveness detection model is divided into two parts. The first part does not need to calculate the gradient information. It aims to use the liveness detection model to obtain the location information of the salient patches (i.e. the prior knowledge described in step 2) and reconstruct the input image. The second part, such as Figure 5 As shown, the input is a reconstructed image, and a fully connected layer module is connected after the transformer encoder. The fully connected layer module includes two fully connected layers and two batch normalization layers. Through the two fully connected layers, the input features are gradually reduced in dimension and transformed to meet the requirements of the classification task (in the present invention, the number of classification categories is 2 - real people and attacks). The two batch normalization layers normalize the features to accelerate the convergence of the liveness detection model and improve the training stability.

[0140] After the above processing, the features are mapped to the final classification dimension, ready for the subsequent classification task.

[0141] The loss function of the liveness detection model is cross entropy loss (cross-entropy loss), the calculation formula is as follows,

[0142]

[0143] in, is the number of sample images; A is the sample image label; Prediction value of the liveness detection model.

[0144] The final loss is obtained by multiplying the calculated cross entropy loss with the weight of the sample image obtained from the entropy-guided hard example mining strategy. , the calculation formula is as follows,

[0145]

[0146] For the loss function The optimization is divided into two parts: (1) minimizing , which can enable the liveness detection model to better learn discriminative features from diverse sample images; (2) minimize , that is, maximizing the entropy value, encouraging the liveness detection model to fully explore the information contained in each token and significantly enhancing the feature expression ability of the token.

[0147] During the inference of the liveness detection model, the image is normalized and then input into the liveness detection model. The output shape of the liveness detection model is a liveness score of [M,2], and the category corresponding to the maximum score is taken as the category predicted by the liveness detection model. There is no need to use reconstructed images at this stage.

[0148] The present invention introduces a significant patch amplitude perturbation strategy, which can diversify sample images without destroying facial structure information and enhance the generalization of the living body detection model.

[0149] The present invention uses the SE module to screen out significant patches in the image, performs amplitude perturbation on the significant patches through Fourier transform and inverse Fourier transform, combines them with patch information of images of the same category, increases data diversity, and alleviates the overfitting problem of the living body detection model.

[0150] The present invention uses an entropy-guided difficult example mining strategy to dynamically assign different weights to sample images of different difficulty levels, so that the liveness detection model can more effectively utilize global information during the training process and avoid being dominated by local areas, thereby improving the robustness of the liveness detection model.

[0151] During the training process, the present invention dynamically adjusts the weights of sample images through an entropy-guided difficult example mining strategy, so that the liveness detection model can more effectively learn and adapt to sample images of different difficulty levels, thereby improving the overall detection performance.

[0152] The present invention also discloses a living body detection device, including a saliency detection module, an amplitude disturbance module, an entropy calculation module, a weight distribution module and a feature extraction module, which can implement the above method and improve the security and user experience of the face recognition system.

[0153] The above embodiments and drawings do not limit the product form and style of the present invention. Any appropriate changes or modifications made by ordinary technicians in the relevant technical field should be deemed to be within the patent scope of the present invention.

Claims

1. A method for liveness detection based on patch amplitude and entropy guidance strategy, comprising the following steps: Obtain a facial image of the user to be detected, pre-process the facial image, and then input it into a liveness detection model deployed in the face recognition system; The liveness detection model performs reasoning on the input face image; The liveness detection model outputs a liveness score with a shape of [M,2], is the number of sample images, the category corresponding to the maximum score is taken as the predicted category of the liveness detection model, and whether the input face image is alive is judged according to the predicted category and output to the face recognition system; it is characterized by: The liveness detection model uses the transformer architecture. During the learning of the liveness detection model, the significant patch amplitude perturbation strategy and the entropy-guided difficult example mining strategy are introduced to accurately distinguish between real and fake faces in the image. The learning of the liveness detection model includes the following steps: Step 1: The input sample image is passed through the convolution module and then added to the position embedding information; then, it is further extracted through the transformer encoder; Step 2: The salient patch amplitude perturbation strategy uses the SE module to further mine the prior knowledge provided by the liveness detection model to obtain salient patch information; then, randomly select patch information of the same category of sample images, and obtain the reconstructed image after Fourier transform and inverse Fourier transform of the salient patch information and the randomly selected patch information; The specific process of the significant patch amplitude perturbation strategy includes the following steps: Step B1: Input feature map of shape [N, Embed_size] Reshape the feature shape, and the reshaped feature is , the shape is [N×Embed_size], where N is the number of patches and Embed_size is the length of the embedded feature vector; Step B2: Features The output features are obtained through the SE module , the shape is [N], the SE module contains ReLU activation layer, fully connected layer, Sigmoid activation layer, Step B3: Filter out the first k significant patches from the N patches, and perform the following processing on the k significant patches respectively: Step C1: Map the index of the salient patch back to the coordinate system of the original image to determine the position of the salient patch in the original image. According to the mapped coordinates, extract the image information of these areas from the original image. Assume Indicates the regional location information of the significant patch k mapped back to the original image, is the original image, is the image information corresponding to the salient patch, Step C2: The width and height of Perform Fourier transform to get the features , in, is the spatial domain coordinate of the image; is the frequency domain coordinate after Fourier transformation, e is the natural base, is pi, i represents the imaginary unit; Step C3: randomly select an image from the same category of images and randomly select a patch from the randomly selected image. Assume that the image information corresponding to the patch is , similarly, Perform Fourier transform and get ; Step C4: Obtain and The magnitude and phase of in, is the amplitude of the Fourier transform result; is the phase of the Fourier transform result; is the real part of the Fourier transform result; is the imaginary part of the Fourier transform result; : inverse tangent function; express The imaginary part of express The real part of Step C5: Random Mixing and The amplitude information of the mixed patch is combined with the phase corresponding to the significant patch to obtain a new complex number , in, is the amplitude after mixing; is a random number between 0 and 1; represents the phase, i is the imaginary unit, and e is the natural base; Step C6: Perform inverse Fourier transform, Convert the image back to the spatial domain , Step C7: Embedded into the corresponding position of the original image, the reconstructed image is obtained; Step 3: After obtaining the reconstructed image, use the reconstructed image as the input of the living body detection model, perform the processing of step 1 again, and divide the output into two paths, one of which enters step 4 and the other enters step 5; Step 4: The entropy-guided hard example mining strategy extracts global information from the output of step 3 and dynamically adjusts the weight of the sample image according to the total amount of information; Step 5. After the output of step 3 passes through the fully connected layer, calculate the cross entropy loss; Step 6: Multiply the cross entropy loss in step 5 by the weight of the sample image obtained in step 4 to get the final loss.

2. The method for liveness detection based on patch amplitude and entropy guidance strategy as claimed in claim 1, characterized in that: In step 2, the combination of random patches and salient patches is as follows: Step A1, performing Fourier transform on the salient patch and the randomly selected image patch of the same category respectively to obtain their respective amplitude and phase information; Step A2, randomly mixing the amplitude of the salient patch with the amplitude of randomly selected patches of the same category of the image; Step A3, combining the mixed amplitude with the phase information of the original significant patch, performing inverse Fourier transform, and generating a new patch; Step A4: re-embed the salient patches after the amplitude perturbation processing into the original image to generate a new diversified image, that is, generate a reconstructed image.

3. The method for liveness detection based on patch amplitude and entropy guidance strategy as claimed in claim 1, characterized in that: The specific process of the entropy-guided hard example mining strategy includes the following steps: Step D1: Use the output of the tansformer encoder as the input feature map of the entropy-guided hard example mining strategy , shape is [N×Embed_size], where N is the number of patches and Embed_size is the length of the embedding feature vector; reshape the input feature map The shape of the reshaped feature is , shape is [N, Embed_size]; Step D2: Along the feature The last dimension of The standard deviation of , shape is [N], along the last dimension, take features The mean of , this feature As information entropy, the shape is [1], in, Indicates the standard deviation of the calculated feature; Indicates the mean of the calculated feature; Indicates related operations along the last dimension; Step D3: The entropy value is transformed as follows by a negative exponential function to obtain the weight of the sample image: The simpler the sample image, the greater the entropy value and the more information it contains. The more difficult the sample image, the smaller the entropy value and the less information it contains. Through the above formula, it is ensured that the weight of sample images with higher entropy values ​​is lower, while the weight of sample images with lower entropy values ​​is higher.

4. The method for liveness detection based on patch amplitude and entropy guidance strategy as claimed in claim 1, characterized in that: During the training of the liveness detection model, the sample image is randomly horizontally flipped before being input into the liveness detection model to be trained. Subsequently, the input image is normalized and standardized to limit the pixel value of the image to between 0 and 1.

5. The method for liveness detection based on patch amplitude and entropy guidance strategy according to claim 1, characterized in that: The training cycle of the liveness detection model is 100 epochs, the batch size is 256, the learning rate is 0.1, the optimizer is SGD, the momentum is 0.9, and the learning rate adjustment strategy is StepLR. At the 20th, 40th, 60th, and 80th epochs, the learning rate is reduced to half of the original.

6. The method for liveness detection based on patch amplitude and entropy guidance strategy as claimed in claim 1, characterized in that: The forward propagation of the liveness detection model is divided into two parts. The first part does not need to calculate the gradient information. The liveness detection model is used to obtain the location information of the significant patch, that is, the prior knowledge described in step 2, and the input image is reconstructed. In the second part, the input is the reconstructed image. The fully connected layer module is connected after the transformer encoder. The fully connected layer module contains two fully connected layers and two batch normalization layers. Through the two fully connected layers, the input features are gradually reduced in dimension and transformed to meet the requirements of the classification task. The two batch normalization layers normalize the features. After forward propagation, the features are mapped to the final classification dimension.

7. The method for liveness detection based on patch amplitude and entropy guidance strategy as claimed in claim 3, characterized in that: In step 5, the cross entropy loss The calculation formula is as follows, in, is the number of sample images; is the sample image label; Predicted value for the liveness detection model; The calculated cross entropy loss is multiplied by the weight of the sample image obtained from the entropy-guided hard example mining strategy to obtain the final loss , the calculation formula is as follows, 。 8. The method for liveness detection based on patch amplitude and entropy guidance strategy as claimed in claim 7, characterized in that: For the loss function The optimization is divided into two parts, one is to minimize , which can enable the liveness detection model to better learn discriminative features from diverse training sample images; the other part is to minimize , that is, maximizing the entropy value, encouraging the liveness detection model to fully explore the information contained in each token and significantly enhancing the feature expression ability of the token.

9. A liveness detection device based on patch amplitude and entropy guidance strategy, characterized in that: Including image acquisition module and living body detection model; The image acquisition module acquires a face image of a user to be detected, pre-processes the face image, and then inputs it into a liveness detection model deployed in the face recognition system; The liveness detection model uses the transformer architecture, including an image preprocessing module, a significant patch amplitude perturbation module, a hard example mining module, a cross entropy loss calculation module, an inference module, and an output judgment module. The liveness detection model uses the transformer architecture. During the learning of the liveness detection model, the significant patch amplitude perturbation strategy and the entropy-guided difficult example mining strategy are introduced to accurately distinguish between real and fake faces in the image. The learning of the liveness detection model includes the following steps: The image preprocessing module preprocesses the input image by first performing a convolution module and then adding the position embedding information; then, the information is further extracted through the transformer encoder; The salient patch amplitude perturbation module uses the SE module to further mine the prior knowledge provided by the liveness detection model to obtain salient patch information; randomly select patch information of sample images of the same category, and obtain a reconstructed image after Fourier transform and inverse Fourier transform of the salient patch information and the randomly selected patch information. The reconstructed image is used as the input of the liveness detection model, and is preprocessed again by the image preprocessing module, and the output is divided into two paths, one of which enters the difficult example mining module, and the other enters the cross entropy loss calculation module through the fully connected layer; The specific process of the significant patch amplitude perturbation strategy includes the following steps: Step B1: Input feature map of shape [N, Embed_size] Reshape the feature shape, and the reshaped feature is , the shape is [N×Embed_size], where N is the number of patches and Embed_size is the length of the embedded feature vector; Step B2: Features The output features are obtained through the SE module , the shape is [N], the SE module contains ReLU activation layer, fully connected layer, Sigmoid activation layer, Step B3: Filter out the first k significant patches from the N patches, and perform the following processing on the k significant patches respectively: Step C1: Map the index of the salient patch back to the coordinate system of the original image to determine the position of the salient patch in the original image. According to the mapped coordinates, extract the image information of these areas from the original image. Assume Indicates the regional location information of the significant patch k mapped back to the original image, is the original image, is the image information corresponding to the salient patch, Step C2: The width and height of Perform Fourier transform to get the features , in, is the spatial domain coordinate of the image; is the frequency domain coordinate after Fourier transformation, e is the natural base, is pi, i represents the imaginary unit; Step C3: randomly select an image from the same category of images and randomly select a patch from the randomly selected image. Assume that the image information corresponding to the patch is , similarly, Perform Fourier transform and get ; Step C4: Obtain and The magnitude and phase of in, is the amplitude of the Fourier transform result; is the phase of the Fourier transform result; is the real part of the Fourier transform result; is the imaginary part of the Fourier transform result; : inverse tangent function; express The imaginary part of express The real part of Step C5: Random Mixing and The amplitude information of the mixed patch is combined with the phase corresponding to the significant patch to obtain a new complex number , in, is the amplitude after mixing; is a random number between 0 and 1; represents the phase, i is the imaginary unit, and e is the natural base; Step C6: Perform inverse Fourier transform, Convert the image back to the spatial domain , Step C7: Embedded into the corresponding position of the original image, the reconstructed image is obtained; The hard example mining module extracts global information and dynamically adjusts the weight of sample images according to the total amount of information; The cross entropy loss calculation module is multiplied by the weight of the sample image of the hard example mining module to obtain the final loss; The inference module performs inference on the input face image; The output judgment module outputs the liveness score in the shape of [M,2]. is the number of sample images, the category corresponding to the maximum score is taken as the predicted category, and whether the input face image is alive is judged according to the predicted category, and the result is output to the face recognition system.

Images