A multi-modal fusion face living body detection model generation method and device and electronic equipment

Abstract

The application discloses a multi-modal fusion face living body detection model generation scheme, which comprises the following steps: constructing a first training set composed of visible light face images and a second training set composed of infrared face images; training a first stage deep neural network according to the first training set and the second training set to obtain a first stage network model; extracting a forward generator in the first stage network model; inputting a third training set composed of visible light face images into the forward generator to obtain infrared face images corresponding to the visible light face images; iteratively training a second stage deep neural network according to the third training set and the infrared face images to obtain network models generated after each iteration; and screening models meeting preset conditions from the network models generated after each iteration as multi-modal fusion face living body detection models. The face living body detection model generated by the scheme has high face recognition result accuracy and strong generalization ability.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of face recognition, and in particular to a multi-modal fusion face living body detection model generation method and device and electronic equipment. BACKGROUND

[0002] In order to ensure the security of the face recognition system, the face living body detection as a kind of biological authentication method has achieved very significant achievements. However, due to the diversity of attack means in the real environment, the face recognition system still faces serious threats, therefore, it is of great significance to study the face living body detection method for ensuring the security of face recognition. The current face living body detection method can be mainly divided into two categories:

[0003] The traditional face living body detection method adopts the manually made features such as local binary pattern, histogram of oriented gradient and scale invariant feature transform to extract the texture information of the face image, and uses the support vector machine and other classifiers to determine the classification boundary, however, these features are usually cross-distinguishable and the performance of the classifier is limited, so the recognition accuracy and universality are poor.

[0004] The face living body detection method based on deep learning can learn the deeper representation of the face image, the early method takes the face living body detection as an ordinary binary classification task, usually only uses single modal data, the single modal data cannot fully represent the fraud clues contained in the face image, and is prone to overfitting, so the accuracy of the obtained face living body detection result is low. SUMMARY

[0005] The purpose of the embodiment of the present application is to provide a multi-modal fusion face living body detection model generation method and device and electronic equipment, which can solve the problem of low detection result accuracy of the existing face living body detection model.

[0006] To solve the above technical problems, the present application provides the following technical scheme:

[0007] The embodiment of the present application provides a multi-modal fusion face living body detection model generation method, comprising:

[0008] A first training set composed of visible light face images and a second training set composed of infrared face images are constructed;

[0009] According to the first training set and the second training set, a first stage deep neural network is trained to obtain a first stage network model, wherein the first stage deep neural network comprises a forward generator, a backward generator, a visible light face image discriminator and an infrared face image discriminator;

[0010] The forward generator in the first stage network model is extracted;

[0011] inputting a third training set composed of visible light face images into the forward generator to obtain infrared face images corresponding to the visible light face images;

[0012] iteratively training a second-stage deep neural network according to the third training set and the infrared face images to obtain network models generated after each iteration;

[0013] screening, from the network models generated after each iteration, a model meeting a preset condition as a multi-modal fusion face living body detection model.

[0014] Optionally, the step of training a first-stage deep neural network according to the first training set and the second training set to obtain a first-stage network model comprises:

[0015] scaling images in the first training set and the second training set to a preset size and then performing normalization processing;

[0016] extracting one image from the first training set and the second training set respectively and inputting the image into the first-stage deep neural network;

[0017] calculating a first adversarial loss of the forward generator of the first-stage deep neural network and the infrared face image discriminator;

[0018] calculating a second adversarial loss of the backward generator of the first-stage deep neural network and the visible light face image discriminator;

[0019] calculating a cycle-consistency loss and a total variation loss of the first-stage deep neural network;

[0020] calculating a total loss of the first-stage deep neural network according to the first adversarial loss, the second adversarial loss, the cycle-consistency loss and the total variation loss;

[0021] optimizing parameters of the first-stage deep neural network according to the total loss by using an Adam optimizer.

[0022] Optionally, after the step of optimizing the parameters of the first-stage deep neural network according to the total loss by using the Adam optimizer, the method further comprises:

[0023] comparing the total loss obtained by the first-stage deep neural network in this training with a total loss value obtained by the first-stage deep neural network in a previous training;

[0024] if the two are the same, the process of iteratively training the first-stage deep neural network based on the first training set and the second training set ends;

[0025] If the total loss obtained in the current training is less than the total loss obtained in the previous training, the step of inputting an image extracted from the first training set and the second training set into the first-stage deep neural network is returned, and the next iteration training of the first-stage deep neural network is started.

[0026] Optionally, the step of iteratively training the second-stage deep neural network according to the third training set and the infrared face image to obtain a network model generated after each iteration includes:

[0027] An image is extracted from the third training set and the second training set respectively and input into the second-stage deep neural network to obtain a prediction result.

[0028] The cross-entropy loss of the second-stage deep neural network is calculated according to the prediction result.

[0029] The second-stage deep neural network is updated by back propagation, the Adam optimizer is used, the parameters of the second-stage deep neural network are optimized according to the cross-entropy loss, and a network model generated after iteration training is obtained.

[0030] Optionally, the step of screening a model meeting a preset condition from the network models generated after each iteration as a multi-modal fusion face living body detection model includes:

[0031] The accuracy of each network model generated after each iteration is tested according to a test set to obtain the accuracy corresponding to each network model generated after each iteration.

[0032] The network model with the highest accuracy is used as a multi-modal fusion face living body detection model.

[0033] Optionally, after the step of screening a model meeting a preset condition from the network models generated after each iteration as a multi-modal fusion face living body detection model, the method further includes:

[0034] The visible light face image to be detected is input into the first-stage network model to obtain a generated infrared face image.

[0035] The visible light face image to be detected and the infrared face image are input into the second-stage network model to obtain a face living body detection result.

[0036] The embodiment of the application provides a multi-modal fusion face living body detection model generation device, which comprises:

[0037] A construction module is configured to construct a first training set composed of visible light face images and a second training set composed of infrared face images.

[0038] a first training module, configured to train a first-stage deep neural network according to the first training set and the second training set to obtain a first-stage network model, wherein the first-stage deep neural network comprises a forward generator, a backward generator, a visible light face image discriminator, and an infrared face image discriminator;

[0039] a extracting module, configured to extract the forward generator in the first-stage network model;

[0040] an inputting module, configured to input a third training set composed of visible light face images into the forward generator to obtain infrared face images corresponding to the visible light face images;

[0041] a second training module, configured to iteratively train a second-stage deep neural network according to the third training set and the infrared face images to obtain network models generated after each iteration;

[0042] a screening module, configured to screen a model meeting a preset condition from the network models generated after each iteration as a multi-modal fusion face living body detection model.

[0043] Optionally, the first training module comprises:

[0044] a first sub-module, configured to normalize images in the first training set and the second training set after scaling the images to a preset size;

[0045] a second sub-module, configured to input one image in the first training set and the second training set into the first-stage deep neural network respectively;

[0046] a third sub-module, configured to calculate a first adversarial loss of the forward generator of the first-stage deep neural network and the infrared face image discriminator;

[0047] a fourth sub-module, configured to calculate a second adversarial loss of the backward generator of the first-stage deep neural network and the visible light face image discriminator;

[0048] a fifth sub-module, configured to calculate a cycle-consistency loss and a total variation loss of the first-stage deep neural network;

[0049] a sixth sub-module, configured to calculate a total loss of the first-stage deep neural network according to the first adversarial loss, the second adversarial loss, the cycle-consistency loss, and the total variation loss;

[0050] a seventh sub-module, configured to optimize parameters of the first-stage deep neural network according to the total loss by using an Adam optimizer.

[0051] Optionally, the apparatus further comprises:

[0052] a comparison module configured to, after the seventh submodule adopts the Adam optimizer to optimize the parameters of the first-stage deep neural network according to the total loss, compare the total loss obtained by the first-stage deep neural network in the current training with the total loss value obtained in the previous training;

[0053] an iterative training module configured to, if the two are the same, end the process of iteratively training the first-stage deep neural network based on the first training set and the second training set;

[0054] a reverse adjustment module configured to, if the total loss obtained in the current training is less than the total loss obtained in the previous training, return to the step of inputting an image extracted from the first training set and the second training set into the first-stage deep neural network, and start the next iterative training of the first-stage deep neural network.

[0055] Optionally, the second training module is specifically configured to:

[0056] extract an image from the third training set and the second training set into the second-stage deep neural network respectively to obtain a prediction result;

[0057] calculate the cross-entropy loss of the second-stage deep neural network according to the prediction result;

[0058] update the second-stage deep neural network by back propagation, adopt the Adam optimizer, optimize the parameters of the second-stage deep neural network according to the cross-entropy loss, and obtain a network model generated after iterative training.

[0059] Optionally, the screening module is specifically configured to:

[0060] test the accuracy of each network model generated after iterative training according to a test set to obtain the accuracy corresponding to each network model generated after iterative training;

[0061] use the network model with the highest accuracy as the multi-modal fusion face liveness detection model.

[0062] Optionally, the device further comprises:

[0063] a prediction module configured to, after the screening module screens the model meeting the preset condition from the network models generated after iterative training as the multi-modal fusion face liveness detection model, input a visible light face image to be detected into the first-stage network model to obtain a generated infrared face image, and input the visible light face image to be detected and the infrared face image into the second-stage network model to obtain a face liveness detection result.

[0064] An electronic device is provided, which includes a processor, a memory, and a program or instructions stored on the memory and executable on the processor, and the program or instructions implement the steps of any one of the face living body detection model generation methods described above when executed by the processor.

[0065] A readable storage medium is provided, which stores a program or instructions, and the program or instructions implement the steps of any one of the multi-modal fusion face living body detection model generation methods described above when executed by a processor.

[0066] The multi-modal fusion face living body detection model generation scheme provided by the embodiments of the present application constructs a first training set composed of visible light face images and a second training set composed of infrared face images; the first stage deep neural network is trained according to the first training set and the second training set, and a first stage network model is obtained; a forward generator in the first stage network model is extracted; a third training set composed of visible light face images is input into the forward generator, and infrared face images corresponding to the visible light face images are obtained; the second stage deep neural network is iteratively trained according to the third training set and the infrared face images, and a network model generated after each iteration is obtained; from the network models generated after each iteration, a model meeting a preset condition is selected as a multi-modal fusion face living body detection model. In the embodiments of the present application, the multi-modal data of the face images are generated by the generative adversarial network, and the multi-modal fusion face living body detection network is used for training and classification, which effectively avoids the interference of features irrelevant to the living body in the face images, and can improve the accuracy and generalization of the face living body detection method. BRIEF DESCRIPTION OF DRAWINGS

[0067] Figure 1 is a step flow chart representing a multi-modal fusion face living body detection model generation method of the embodiments of the present application;

[0068] Figure 2 is a structural schematic diagram of a first stage deep neural network of the embodiments of the present application;

[0069] Figure 3 is a structural schematic diagram of a forward generator and a backward generator of the embodiments of the present application;

[0070] Figure 4 is a structural schematic diagram of a visible light face image discriminator and an infrared face image discriminator of the embodiments of the present application;

[0071] Figure 5 is a structural schematic diagram of a second stage deep neural network of the embodiments of the present application;

[0072] Figure 6is a structural schematic diagram of an adaptive fusion unit of an embodiment of the present application.

[0073] Figure 7 is a structural block diagram of a multi-modal fusion face living body detection model generation device of an embodiment of the present application. DETAILED DESCRIPTION

[0074] To make the technical problems, technical solutions and advantages of the present application clearer, the following will be described in detail with reference to the drawings and specific embodiments.

[0075] The face living body detection model generation scheme provided by the embodiments of the present application will be described in detail below in combination with the drawings, specific embodiments and application scenarios.

[0076] As shown in the accompanying Figure 1 , the multi-modal fusion face living body detection model generation method of the embodiments of the present application includes the following steps:

[0077] Step 101: constructing a first training set composed of visible light face images and a second training set composed of infrared face images.

[0078] Prepare the first-stage deep neural network training data, taking the visible light face images as the first training set R and the infrared face images as the second training set I.

[0079] In the present application, after the first-stage deep neural network, the second-stage deep neural network are created and the first training set, the second training set, the third training set are prepared, the multi-modal fusion face living body detection model is generated after the two-stage deep neural networks are trained based on the three training sets.

[0080] The first training set and the second training set each contain multiple training samples. When constructing the training set, the corresponding type of face image can be preprocessed. The preprocessing of the face image can include but is not limited to scaling the size of the face image to a preset size, for example, 256x256 pixels, and normalizing the pixels thereof to between 0 and 1, x Norm = x / 255.

[0081] Wherein, x is the original face image, x Norm is the normalized face image, multiple normalized visible light face images constitute the first training set, and multiple normalized infrared face images constitute the second training set.

[0082] Step 102: training the first-stage deep neural network according to the first training set and the second training set to obtain a first-stage network model.

[0083] The first-stage deep neural network can be pre-built or built after the two training sets are prepared.

[0084] Figure 2 is a structural schematic diagram representing the first-stage deep neural network of the embodiment of the present application; as shown in Figure 2 the first-stage deep neural network includes a forward generator, a backward generator, a visible light face image discriminator, and an infrared face image discriminator.

[0085] The first-stage deep neural network is built by imitating a cycle generative adversarial neural network, and its structure includes a forward generator (for generating an infrared face image from a visible light face image), a backward generator (for generating a visible light face image from an infrared face image), a visible light face image discriminator, and an infrared face image discriminator. The forward generator is responsible for converting an input visible light face image into an infrared face image, the backward generator is responsible for converting an input infrared face image into a visible light face image, the visible light face image discriminator is responsible for identifying whether an input image is a real visible light face image or a synthetic visible light face image, and the infrared face image discriminator is responsible for identifying whether an input image is a real infrared face image or a synthetic infrared face image.

[0086] Figure 3 is a structural schematic diagram representing the forward generator and the backward generator of the embodiment of the present application; as shown in Figure 3 the forward generator and the backward generator have the same structure, and both include an input layer, a first convolutional layer, a second convolutional layer, a third convolutional layer, a first transposed convolutional layer, a second transposed convolutional layer, a first residual unit, a second residual unit, a third residual unit, a fourth residual unit, a fifth residual unit, a sixth residual unit, a seventh residual unit, an eighth residual unit, a ninth residual unit, a ReLU activation layer, and an output layer. The step lengths of the first convolutional layer, the second convolutional layer, the first transposed convolutional layer, and the second transposed convolutional layer are all 2, and they are responsible for down-sampling and up-sampling features in the network. The input layer is connected to the first convolutional layer, the first convolutional layer is connected to the first residual unit, the first residual unit is connected to the second residual unit, the second residual unit is connected to the second convolutional layer, the second convolutional layer is connected to the third residual unit, the third residual unit is connected to the fourth residual unit, the fourth residual unit is connected to the fifth residual unit, the fifth residual unit is connected to the sixth residual unit, the sixth residual unit is connected to the seventh residual unit, the seventh residual unit is connected to the first transposed convolutional layer, the first transposed convolutional layer is connected to the eighth residual unit, the eighth residual unit is connected to the ninth residual unit, the ninth residual unit is connected to the second transposed convolutional layer, the second transposed convolutional layer is connected to the third convolutional layer, the third convolutional layer is connected to the ReLU activation layer, and the ReLU activation layer is connected to the output layer.

[0087] The first residual unit, the second residual unit, the third residual unit, the fourth residual unit, the fifth residual unit, the sixth residual unit, the seventh residual unit, the eighth residual unit and the ninth residual unit each comprise a first convolutional layer, a second convolutional layer, a first ReLU activation layer and a second ReLU activation layer. The first convolutional layer is connected to the first ReLU activation layer, and the first ReLU activation layer is connected to the second convolutional layer. The output of the second convolutional layer is added to the input of the residual unit and then input to the second ReLU activation layer to obtain the output of the residual unit. The processing method is as follows:

[0088] Outputs = ReLU (Inputs + Conv2 (ReLU (Conv1 (Inputs))))

[0089] Figure 4 is a structural schematic diagram of a visible light face image discriminator and an infrared face image discriminator according to an embodiment of the present application. The visible light face image discriminator and the infrared face image discriminator have the same structure, and the discriminator structure in PatchGAN is adopted, which specifically comprises an input layer, a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer, a first LeakyReLU activation layer, a second LeakyReLU activation layer, a third LeakyReLU activation layer, a fourth LeakyReLU activation layer and an output layer. The input layer is connected to the first convolutional layer, the first convolutional layer is connected to the first LeakyReLU activation layer, the first LeakyReLU activation layer is connected to the second convolutional layer, the second convolutional layer is connected to the second LeakyReLU activation layer, the second LeakyReLU activation layer is connected to the third convolutional layer, the third convolutional layer is connected to the third LeakyReLU activation layer, the third LeakyReLU activation layer is connected to the fourth convolutional layer, the fourth convolutional layer is connected to the fourth LeakyReLU activation layer, the fourth LeakyReLU activation layer is connected to the fifth convolutional layer, and the fifth convolutional layer is connected to the output layer.

[0090] Figure 5 is a structural schematic diagram of a second-stage deep neural network according to an embodiment of the present application. As shown in Figure 5As shown, the second stage deep neural network is a double-branch fusion network, and the structure thereof includes an input layer, a first convolutional layer, a second convolutional layer, a third convolutional layer, a first global average pooling layer, a second global average pooling layer, a first ReLU activation layer, a second ReLU activation layer, a first full connection layer, a second full connection layer, a Sigmoid activation layer, a first residual unit, a second residual unit, a third residual unit, a fourth residual unit, a fifth residual unit, a sixth residual unit, a seventh residual unit, an eighth residual unit, a first adaptive fusion unit, a second adaptive fusion unit, and a third adaptive fusion unit. As can be seen, the visible light face image and the generated infrared face image are respectively input to the first convolutional layer and the second convolutional layer for down-sampling, and are respectively input to the first residual unit and the second residual unit after down-sampling. The outputs of the first residual unit and the second residual unit are input to the first adaptive fusion unit together to generate low-level fusion features. The outputs of the first residual unit and the second residual unit are input to the third residual unit and the fourth residual unit, and the outputs of the third residual unit and the fourth residual unit are input to the second adaptive fusion unit together to generate medium-level fusion features. The outputs of the third residual unit and the fourth residual unit are input to the fifth residual unit and the sixth residual unit, and the outputs of the fifth residual unit and the sixth residual unit are input to the third adaptive fusion unit together to generate high-level fusion features. The low-level fusion features, the medium-level fusion features, and the high-level fusion features are spliced and sequentially passed through the third convolutional layer, the first ReLU activation layer, the global average pooling layer, the first full connection layer, the second ReLU activation layer, the second full connection layer, and the Sigmoid activation layer to obtain a classification result.

[0091] The adaptive fusion unit includes a convolutional layer, a global average pooling layer, a first full connection layer, a second full connection layer, a third full connection layer, a ReLU activation layer, a first Sigmoid activation layer, and a second Sigmoid activation layer. Assuming that the input features are F1 and F2, and the sizes of F1 and F2 are both CxHxW, the features are added and sequentially passed through the convolutional layer, the global average pooling layer, the first full connection layer, and the ReLU activation layer to obtain a compact feature Z. The processing method is as follows:

[0092] Z = ReLU (Fc1 (GAP (Conv (F1+F2))))

[0093] wherein, Conv, GAP, and Fc respectively represent the convolutional layer, the global average pooling layer, and the full connection layer. The size of the feature Z is The compact feature Z is input to the second full connection layer, the second Sigmoid activation layer, the third full connection layer, and the third Sigmoid activation layer to obtain soft attentions of different branches. The processing method is as follows:

[0094] α1 = Sigmoid1 (Fc2 (Z))

[0095] a2=Sigmoid2(Fc3(z))

[0096] wherein, Fc2, Fc3, Sigmoid2, Sigmoid3 are the second full connection layer, the third full connection layer, the first Sigmoid activation layer, the second Sigmoid activation layer respectively, the channel number of a1 and a2 is consistent with the input channel number of the input and adaptive fusion unit, the weight factor is calculated according to a1 and a2, and the processing method can be:

[0097]

[0098] wherein, c is a channel. The output of the adaptive fusion unit is:

[0099] F out,c =β c ·F 1,c +(1-β c )·F 2,c

[0100] wherein, F out is the output of the adaptive fusion unit.

[0101] It should be noted that the above only exemplarily introduces the structure of the first stage deep neural network and each module in the network, and in the actual implementation process, the specific structure of each module can be flexibly set according to actual needs, and the present application embodiment does not make specific limitation thereto.

[0102] An optional way of training the first stage deep neural network according to the first training set and the second training set to obtain the first stage network model can include the following sub-steps:

[0103] Sub-step one: after scaling the images in the first training set and the second training set to a preset size, performing normalization processing;

[0104] Sub-step two: respectively extracting an image from the first training set and the second training set and inputting the image into the first stage deep neural network;

[0105] Sub-step three: calculating the first adversarial loss of the forward generator and the infrared face image discriminator of the first stage deep neural network;

[0106] Sub-step four: calculating the second adversarial loss of the backward generator and the visible light face image discriminator of the first stage deep neural network;

[0107] The processing method of calculating the adversarial loss of the generator and the discriminator can be:

[0108] L G-GAN =E i∈I[logD I (i)]+E r∈R [log(1-D I (G(r)))]+E r∈R [log(1-D I (G(r)))]

[0109] L F-GAN =E r∈R [logD R (r)]+E i∈I [log(1-D R (F(i)))]+E i∈I [log(1-D R (F(i)))]

[0110] Wherein, G represents a forward generator, D I represents an infrared face image discriminator, L G-GAN is the adversarial loss of the forward generator and the infrared face image discriminator, F represents a backward generator, D R represents a visible light face image discriminator, and L F-GAN is the adversarial loss of the backward generator and the visible light face image discriminator.

[0111] Substep five: calculate the cycle consistency loss and the total variation loss of the first stage deep neural network;

[0112] In the actual implementation process, the cycle consistency loss of the first stage network can be calculated based on the following formula:

[0113]

[0114] Wherein, L cyc is the cycle consistency loss of the network.

[0115] In order to make the generated face image more smooth and clear, the total variation loss is added in the generator:

[0116]

[0117]

[0118] Wherein, i∈I, r∈R, i, j, k are the coordinates of the pixels in the generated image, and L TV is the total variation loss of the network.

[0119] Substep six: calculate the total loss of the first stage deep neural network according to the first adversarial loss, the second adversarial loss, the cycle consistency loss and the total variation loss.

[0120] In the actual implementation process, the total loss of the first-stage deep neural network can be calculated according to the following formula:

[0121] L1 = λ1 · (L G-GAN + L F-GAN ) + λ2 · L cyc + λ3 · L TV

[0122] wherein λ1, λ2, and λ3 are hyperparameters for controlling the loss weight, and are divided into 1, 10, and 1e-5.

[0123] Sub-step seven: using the Adam optimizer to optimize the parameters of the first-stage deep neural network according to the total loss.

[0124] Sub-steps one to seven are the process of training the first-stage deep neural network for three times. In the actual implementation process, the first-stage deep neural network needs to be iteratively trained for multiple times until the preset condition is met to terminate the iterative training. It should be noted that the model obtained from the previous iterative training will be used as the basis for the next iterative training model.

[0125] The preset condition can be set as stopping the iterative training when the loss of the discriminator no longer decreases.

[0126] In an alternative embodiment, after using the Adam optimizer to optimize the parameters of the first-stage deep neural network according to the total loss, the method further comprises: comparing the total loss obtained from the current training of the first-stage deep neural network with the total loss value obtained from the previous training; if they are the same, the process of iteratively training the first-stage deep neural network based on the first training set and the second training set is ended; if the total loss obtained from the current training is less than the total loss obtained from the previous training, returning to the step of extracting one image from the first training set and the second training set and inputting it into the first-stage deep neural network, and starting the next iterative training of the first-stage deep neural network.

[0127] This alternative iterative training method for the first-stage deep neural network can ensure that the first-stage network model meets the requirements while occupying the smallest possible computing resources.

[0128] After completing the training of the first-stage neural network to obtain the first-stage network model, the second-stage deep neural network needs to be trained to obtain the second-stage network model.

[0129] Step 103: extracting the forward generator in the first-stage network model.

[0130] The forward generator in the obtained first-stage network model is extracted and its parameters are fixed.

[0131] Step 104: inputting a third training set composed of visible light face images into the forward generator to obtain infrared face images corresponding to the visible light face images.

[0132] The third training set composed of visible light face images can be the same as or different from the first training set. The visible light face images in the third training set also need to undergo the same preprocessing operation as the face images in the first training set.

[0133] Step 105: iteratively training the second-stage deep neural network according to the third training set and the infrared face images to obtain a network model generated after each iteration.

[0134] An optional way of iteratively training the second-stage deep neural network according to the third training set and the infrared face images to obtain a network model generated after each iteration can be as follows:

[0135] First, one image is extracted from the third training set and the second training set respectively and input into the second-stage deep neural network to obtain a prediction result.

[0136] Second, the cross-entropy loss of the second-stage deep neural network is calculated according to the prediction result.

[0137] In this step, the data of the two modalities are input into the second-stage deep neural network, and the cross-entropy loss of the second-stage deep neural network is calculated according to the prediction result of the second-stage deep neural network:

[0138]

[0139] where p is the probability of network prediction as a real face, y is the label of the input sample, y is 0 indicating a fake face image, and its value is 1 indicating a real face image, and N is the number of input network samples. i i i

[0140] Finally, the second-stage deep neural network is updated by back propagation, the Adam optimizer is adopted, the parameters of the second-stage deep neural network are optimized according to the cross-entropy loss, and a network model generated after iteration is obtained.

[0141] It should be noted that steps 102 to 105 are the process of single parameter training of the first-stage deep neural network and the second-stage deep neural network. In actual implementation, the training process needs to be repeated until the upper limit of the preset iteration number is reached.

[0142] Step 106: selecting a model meeting a preset condition from the network models generated after each iteration as a multi-modal fusion face liveness detection model.

[0143] ​​​The manner of screening the model meeting the preset condition from the network model generated after each iteration as the multi-modal fusion face living body detection model can be as follows: the accuracy of each network model generated after each iteration is tested according to the test set, and the accuracy corresponding to each network model generated after each iteration is obtained; and the network model with the highest accuracy is taken as the multi-modal fusion face living body detection model.

[0144] The network model of each iteration is saved after multiple iterations, and then the recognition accuracy of each iteration of the network model is tested through the preset verification set, and the accuracy corresponding to each network model is recorded. The parameters of the model with the highest accuracy on the verification set are selected as the parameters of the multi-modal fusion face living body detection model.

[0145] The face living body detection model generation method provided by the embodiment of the application comprises the following steps: a first training set composed of visible light face images and a second training set composed of infrared face images are constructed; the first stage deep neural network is trained according to the first training set and the second training set, and a first stage network model is obtained; a forward generator in the first stage network model is extracted; a third training set composed of visible light face images is input into the forward generator, and infrared face images corresponding to the visible light face images are obtained; the second stage deep neural network is iteratively trained according to the third training set and the infrared face images, and a network model generated after each iteration is obtained; and a model meeting a preset condition is screened from the network model generated after each iteration as a multi-modal fusion face living body detection model. In the embodiment of the application, the multi-modal data of the face image is generated by the generative adversarial network, and the multi-modal fusion face living body detection network is used for training and classification, which effectively avoids the interference of the features irrelevant to the living body in the face image, and improves the accuracy and generalization of the face living body detection method.

[0146] In an optional embodiment, after the model meeting the preset condition is screened from the network model generated after each iteration as the multi-modal fusion face living body detection model, the following face image prediction process can be further included:

[0147] The visible light face image to be detected is input into the first stage network model, and a generated infrared face image is obtained;

[0148] The visible light face image to be detected and the infrared face image are input into the second stage network model, and a face living body detection result is obtained.

[0149] The optional manner of identifying the face image can be completed by only the first branch of the multi-modal fusion face living body detection model, and the calculation amount is small and the processing efficiency is high.

[0150] The face living body detection model generation method provided by the present application will be described below in combination with a specific embodiment.

[0151] The face living body detection model generation method in the present application mainly includes the following steps:

[0152] As shown in Figure 1 , the present application provides a two-stage disentangled face living body detection method of infrared image global attention, and the specific steps are as follows:

[0153] Step one: collect visible light face images and infrared face images as training set R and I, scale the images in the data set to 256x256 pixels and normalize them to facilitate the calculation of the deep neural network.

[0154] Step two: build a first-stage deep neural network.

[0155] As shown in Figure 2 , the first-stage deep neural network is built by imitating the cyclic generative adversarial neural network, and its structure includes a forward generator (generating infrared face images from visible light face images), a reverse generator (generating visible light face images from infrared face images), a visible light face image discriminator, and an infrared face image discriminator. The forward generator is responsible for converting the input visible light face image into an infrared face image, the reverse generator is responsible for converting the input infrared face image into a visible light face image, the visible light face image discriminator is responsible for identifying whether the input image is a real visible light face image or a synthetic visible light face image, and the infrared face image discriminator is responsible for identifying whether the input image is a real infrared face image or a synthetic infrared face image.

[0156] Step three: use data set R and data set I to train the first-stage deep neural network.

[0157] The method for training the first-stage deep neural network can include the following sub-steps:

[0158] S3.1: input the face images in data set R and data set I into the built deep neural network;

[0159] S3.2: calculate the adversarial loss of the generator and the discriminator, and the processing method is:

[0160]

[0161]

[0162] Wherein, G represents the forward generator, D I represents the infrared face image discriminator, and L G-GANL is the adversarial loss of the forward generator and the infrared face image discriminator, F represents the backward generator, D R L represents the visible light face image discriminator, L F-GAN is the adversarial loss of the backward generator and the visible light face image discriminator.

[0163] S3.3: Calculate the cycle consistency loss of the first stage deep neural network:

[0164]

[0165] L is the cycle consistency loss of the first stage deep neural network. cyc

[0166] S3.4: In order to make the generated face image more smooth and clear, add total variation loss in the generator:

[0167]

[0168]

[0169] L is the total variation loss of the network. TV

[0170] S3.5: According to the above calculation, the total loss of the first stage deep neural network can be obtained:

[0171] L1 = λ1·(L G-GAN +L F-GAN )+λ2·L cyc +λ3·L TV

[0172] L1 = λ1·(L G-GAN +L F-GAN )+λ2·L cyc +λ3·L TV

[0173] According to the loss, the parameters of the first stage deep neural network are optimized by using Adam optimizer.

[0174] S3.6: After the loss of the discriminator no longer decreases, stop training and obtain the first stage deep neural network parameters.

[0175] Step four: collect visible light face images as dataset X, scale the images in the dataset to 256x256 and normalize them to facilitate deep neural network calculation;

[0176] Step five: build the second stage deep neural network.

[0177] The specific structure of the second-stage deep neural network and each module included in the second-stage deep neural network can refer to the related description in the foregoing embodiments, and will not be described here again.

[0178] Step six: training the second-stage deep neural network using the dataset X and the forward generator in the first-stage network to obtain a second-stage network model.

[0179] The training method of the second-stage network model can include the following sub-steps:

[0180] S6.1: scale the visible light face image training set X and its test set to 256x256 and normalize them to facilitate deep neural network calculation;

[0181] S6.2: extract the obtained forward generator in the first-stage network, fix its parameters, input the visible light face image, obtain the infrared face image from the forward generator, input the data of the two modalities into the second-stage network, and calculate the cross-entropy loss of the second-stage network according to the prediction result of the second-stage network:

[0182]

[0183] Wherein, p i is the probability that the network predicts a real face, y i is the label of the input sample, y i is 0 indicating a false face image, and its value is 1 indicating a real face image, and N is the number of samples input into the network.

[0184] S6.3: update the parameters of the second-stage network by back propagation, and update the network parameters using the Adam optimizer.

[0185] S6.4: output the accuracy of the model training, save the network model of each iteration, and select the model parameters with the highest accuracy on the validation set.

[0186] Step seven: inputting the visible light face image to be detected into the first-stage network to obtain the generated infrared face image, and inputting the visible light face image to be detected and its generated infrared face image into the second-stage network to obtain the face liveness detection result.

[0187] The multi-modal fusion face liveness detection model generation method provided in the specific embodiment generates multi-modal data of face images through a generative adversarial network, trains and classifies using a multi-modal fusion face liveness detection network, effectively avoids features irrelevant to the living body in the face image from interfering with it, and can improve the low accuracy and generalization of the face liveness detection method.

[0188] Figure 7A structural block diagram of a multi-modal fusion face living body detection model generation device according to an embodiment of the present application.

[0189] The multi-modal fusion face living body detection model generation device provided by the embodiment of the present application comprises the following functional modules:

[0190] The construction module 701 is configured to construct a first training set composed of visible light face images and a second training set composed of infrared face images;

[0191] The first training module 702 is configured to train a first-stage deep neural network according to the first training set and the second training set to obtain a first-stage network model, wherein the first-stage deep neural network comprises a forward generator, a backward generator, a visible light face image discriminator and an infrared face image discriminator;

[0192] The extraction module 703 is configured to extract the forward generator in the first-stage network model;

[0193] The input module 704 is configured to input a third training set composed of visible light face images into the forward generator to obtain infrared face images corresponding to the visible light face images;

[0194] The second training module 705 is configured to iteratively train a second-stage deep neural network according to the third training set and the infrared face images to obtain network models generated after each iteration;

[0195] The screening module 706 is configured to screen a model meeting a preset condition from the network models generated after each iteration as a multi-modal fusion face living body detection model.

[0196] Optionally, the first training module comprises:

[0197] The first sub-module is configured to normalize the images in the first training set and the second training set after scaling the images to a preset size;

[0198] The second sub-module is configured to input one image from the first training set and the second training set into the first-stage deep neural network, respectively;

[0199] The third sub-module is configured to calculate a first adversarial loss of the forward generator of the first-stage deep neural network and the infrared face image discriminator;

[0200] The fourth sub-module is configured to calculate a second adversarial loss of the backward generator of the first-stage deep neural network and the visible light face image discriminator;

[0201] a fifth sub-module configured to calculate a cycle-consistency loss and a total variation loss of the first-stage deep neural network;

[0202] a sixth sub-module configured to calculate a total loss of the first-stage deep neural network according to the first adversarial loss, the second adversarial loss, the cycle-consistency loss and the total variation loss;

[0203] a seventh sub-module configured to optimize parameters of the first-stage deep neural network according to the total loss by using an Adam optimizer.

[0204] Optionally, the device further comprises:

[0205] a comparison module configured to compare the total loss obtained by the first-stage deep neural network in the present training with a total loss value obtained by the first-stage deep neural network in the previous training after the seventh sub-module optimizes the parameters of the first-stage deep neural network according to the total loss by using the Adam optimizer;

[0206] an iterative training module configured to end the process of iteratively training the first-stage deep neural network based on the first training set and the second training set if the total loss obtained in the present training is the same as the total loss value obtained in the previous training.

[0207] a reverse adjustment module configured to return to the step of inputting an image extracted from the first training set and the second training set into the first-stage deep neural network and start the next iterative training of the first-stage deep neural network if the total loss obtained in the present training is less than the total loss value obtained in the previous training.

[0208] Optionally, the second training module is specifically configured to:

[0209] input an image extracted from the third training set and the second training set into the second-stage deep neural network respectively to obtain a prediction result;

[0210] calculate a cross-entropy loss of the second-stage deep neural network according to the prediction result;

[0211] update the second-stage deep neural network by back propagation, optimize parameters of the second-stage deep neural network according to the cross-entropy loss by using an Adam optimizer, and obtain a network model generated after iterative training.

[0212] Optionally, the screening module is specifically configured to:

[0213] test the accuracy of the network model generated after each iteration according to a test set to obtain an accuracy corresponding to the network model generated after each iteration;

[0214] The network model with the highest accuracy is taken as the multi-modal fusion face liveness detection model.

[0215] Optionally, the apparatus further comprises:

[0216] The prediction module is configured to, after the step of screening the model meeting the preset condition from the network models generated after each iteration by the screening module as the multi-modal fusion face liveness detection model, input the visible light face image to be detected into the first-stage network model to obtain a generated infrared face image; and input the visible light face image to be detected and the infrared face image into the second-stage network model to obtain a face liveness detection result.

[0217] The face liveness detection model generation apparatus provided in the embodiments of the present application generates multi-modal data of face images through a generative adversarial network, and uses a multi-modal fusion face liveness detection network for training and classification, thereby effectively avoiding interference of features irrelevant to a living body in a face image, and improving the accuracy of a face liveness detection method and the generalization.

[0218] The multi-modal fusion face liveness detection model generation apparatus shown in the embodiments of the present application can be an apparatus, or a component, an integrated circuit, or a chip in a three-dimensional scanner. Figure 7 The multi-modal fusion face liveness detection model generation apparatus shown in the embodiments of the present application can be an apparatus with an operating system. Figure 7 The multi-modal fusion face liveness detection model generation apparatus shown in the embodiments of the present application can be an apparatus with an operating system.

[0219] The multi-modal fusion face liveness detection model generation apparatus provided in the embodiments of the present application can implement the processes of the method embodiments, and thus repeated details are not described herein. Figure 7 The multi-modal fusion face liveness detection model generation apparatus provided in the embodiments of the present application can implement the processes of the method embodiments, and thus repeated details are not described herein. Figure 1 The method embodiments implement various processes, and thus repeated details are not described herein.

[0220] Optionally, the embodiments of the present application further provide an electronic device, which includes a processor, a memory, a program or instructions stored on the memory and executable on the processor, the program or instructions being executed by the processor to implement various processes of the multi-modal fusion face liveness detection model generation method embodiments, and achieve the same technical effects. It should be noted that the electronic device in the embodiments of the present application includes the server described above.

[0221] The embodiments of the present application further provide a readable storage medium having a program or instructions stored thereon, the program or instructions being executed by a processor to implement various processes of the multi-modal fusion face liveness detection model generation method embodiments, and achieve the same technical effects. Repeated details are not described herein.

[0222] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0223] The embodiment of the present application further provides a chip, which comprises a processor and a communication interface, the communication interface is coupled with the processor, the processor is used for running programs or instructions to realize each process of the above-mentioned multi-modal fusion face living body detection model generation method embodiment, and the same technical effects can be achieved. To avoid repetition, details are not described herein.

[0224] It should be understood that the chip mentioned in the embodiment of the present application can also be referred to as a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0225] It should be noted that in this document, the term "comprising" or "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0226] The above is the preferred embodiment of the present application. It should be noted that for those skilled in the art, without departing from the principles of the present application, several improvements and refinements can be made, which should also be considered within the scope of protection of the present application.

Claims

1. A method for generating a multi-modal fusion face liveness detection model, comprising: constructing a first training set composed of visible light face images and a second training set composed of infrared face images; training a first-stage deep neural network according to the first training set and the second training set to obtain a first-stage network model, wherein the first-stage deep neural network comprises a forward generator, a backward generator, a visible light face image discriminator, and an infrared face image discriminator; the forward generator is configured to convert an input visible light face image into an infrared face image, the backward generator is configured to convert an input infrared face image into a visible light face image, the visible light face image discriminator is configured to identify whether an input image is a real visible light face image or a synthetic visible light face image, and the infrared face image discriminator is configured to identify whether an input image is a real infrared face image or a synthetic infrared face image; extracting the forward generator in the first-stage network model; inputting a third training set composed of visible light face images into the forward generator to obtain infrared face images corresponding to the visible light face images; iteratively training a second-stage deep neural network according to the third training set and the infrared face images output by the forward generator to obtain network models generated after each iteration; selecting a model meeting a preset condition from the network models generated after each iteration as the multi-modal fusion face liveness detection model; wherein the step of iteratively training the second-stage deep neural network according to the third training set and the infrared face images output by the forward generator to obtain network models generated after each iteration comprises: inputting one image respectively extracted from the third training set and the infrared face images output by the forward generator into the second-stage deep neural network to obtain a prediction result; calculating a cross-entropy loss of the second-stage deep neural network according to the prediction result; updating the second-stage deep neural network through back propagation, optimizing parameters of the second-stage deep neural network according to the cross-entropy loss by using an Adam optimizer, and obtaining a network model generated after iteration training; the step of selecting a model meeting a preset condition from the network models generated after each iteration as the multi-modal fusion face liveness detection model comprises: testing the accuracy of each network model generated after each iteration according to a test set to obtain an accuracy corresponding to each network model generated after each iteration; and selecting a network model with the highest accuracy as the multi-modal fusion face liveness detection model.

2. The method of claim 1, wherein, the step of training the first-stage deep neural network according to the first training set and the second training set to obtain the first-stage network model comprises: scaling images in the first training set and the second training set to a preset size and then performing normalization processing; and inputting one image respectively extracted from the first training set and the second training set into the first-stage deep neural network. calculating a first adversarial loss of the forward generator of the first-stage deep neural network and the infrared face image discriminator; calculating a second adversarial loss of the backward generator of the first-stage deep neural network and the visible light face image discriminator; calculating a cycle-consistency loss and a total variation loss of the first-stage deep neural network; calculating a total loss of the first-stage deep neural network according to the first adversarial loss, the second adversarial loss, the cycle-consistency loss and the total variation loss; optimizing parameters of the first-stage deep neural network according to the total loss by using an Adam optimizer.

3. The method of claim 2, wherein, After the step of optimizing the parameters of the first-stage deep neural network according to the total loss by using the Adam optimizer, the method further comprises: comparing the total loss of the first-stage deep neural network obtained in this training with a total loss value obtained in the last training; if the two are the same, the process of iteratively training the first-stage deep neural network based on the first training set and the second training set is ended; if the total loss obtained in this training is less than the total loss obtained in the last training, returning to the step of extracting an image from the first training set and the second training set respectively and inputting the image into the first-stage deep neural network to start the next iterative training of the first-stage deep neural network.

4. A multi-modal fusion face living body detection model generation device, comprising: a construction module configured to construct a first training set composed of visible light face images and a second training set composed of infrared face images; a first training module configured to train a first-stage deep neural network based on the first training set and the second training set to obtain a first-stage network model, wherein the first-stage deep neural network comprises a forward generator, a backward generator, a visible light face image discriminator and an infrared face image discriminator; the forward generator is configured to convert an input visible light face image into an infrared face image, the backward generator is configured to convert an input infrared face image into a visible light face image, the visible light face image discriminator is configured to identify whether an input image is a real visible light face image or a synthetic visible light face image, and the infrared face image discriminator is configured to identify whether an input image is a real infrared face image or a synthetic infrared face image; an extraction module configured to extract the forward generator in the first-stage network model; an input module configured to input a third training set composed of visible light face images into the forward generator to obtain infrared face images corresponding to the visible light face images; a second training module configured to iteratively train a second-stage deep neural network based on the third training set and the infrared face images output by the forward generator to obtain network models generated after each iteration; a screening module configured to screen a model meeting a preset condition from the network models generated after each iteration as a multi-modal fusion face living body detection model; the second training module is specifically configured to: extract one image from the infrared face image output by the third training set and the positive generator respectively and input the image into the second stage deep neural network to obtain a prediction result; calculate a cross-entropy loss of the second stage deep neural network according to the prediction result; update the second stage deep neural network by back propagation, use an Adam optimizer to optimize parameters of the second stage deep neural network according to the cross-entropy loss, and obtain a network model generated after iterative training; the screening module is specifically used for: testing the accuracy of the network model generated after each iteration according to a test set to obtain an accuracy corresponding to the network model generated after each iteration; using the network model with the highest accuracy as a multi-modal fusion face liveness detection model.

5. The apparatus of claim 4, wherein, the first training module comprises: a first sub-module configured to normalize images in the first training set and the second training set after scaling the images to a preset size; a second sub-module configured to extract one image from the first training set and the second training set respectively and input the image into the first stage deep neural network; a third sub-module configured to calculate a first adversarial loss of the positive generator of the first stage deep neural network and the infrared face image discriminator; a fourth sub-module configured to calculate a second adversarial loss of the reverse generator of the first stage deep neural network and the visible light face image discriminator; a fifth sub-module configured to calculate a cycle consistency loss and a total variation loss of the first stage deep neural network; a sixth sub-module configured to calculate a total loss of the first stage deep neural network according to the first adversarial loss, the second adversarial loss, the cycle consistency loss, and the total variation loss; a seventh sub-module configured to use an Adam optimizer to optimize parameters of the first stage deep neural network according to the total loss.

6. The apparatus of claim 5, wherein, the device further comprises: a comparison module configured to compare a total loss of the first stage deep neural network obtained in the present training with a total loss value obtained in the previous training after the seventh sub-module uses the Adam optimizer to optimize the parameters of the first stage deep neural network according to the total loss; an iterative training module configured to end the process of iterative training of the first stage deep neural network based on the first training set and the second training set if the two total losses are the same; a reverse adjustment module configured to return to the step of extracting one image from the first training set and the second training set respectively and inputting the image into the first stage deep neural network if the total loss obtained in the present training is less than the total loss obtained in the previous training, and start the next iterative training of the first stage deep neural network. 7.An electronic device comprising a processor, a memory, and a program or instructions stored on the memory and executable on the processor, wherein the program or instructions are executed by the processor to implement the steps of the multi-modal fusion face liveness detection model generation method according to any one of claims 1-3.

Images