A face reshooting audit method and system based on deep learning

Abstract

The present invention discloses a face duplication audit method and system based on deep learning, relates to the field of face recognition technology, and solves the technical problem of low efficiency of traditional image processing technology. The method comprises: collecting face image data related to application scenarios of face audit, and dividing the collected face image data into positive samples and negative samples; performing enhancement processing on the positive samples to obtain a positive sample set, and performing enhancement processing on the negative samples to obtain a negative sample set, and dividing the positive sample set and the negative sample set into a training set and a test set respectively in a ratio of 8:2, the training set is used for model learning and training, and the test set is used for model performance evaluation; constructing a neural network model, and training the neural network model according to the training set, and evaluating the trained neural network model according to the test set to obtain a face duplication detection model; collecting face images on site, and inputting them into the face duplication detection model to obtain face duplication detection results.

Description

Technical Field

[0001] The present invention relates to the field of face recognition technology, and more specifically, to a face re-photographing auditing method and system based on deep learning. Background Art

[0002] Currently, facial recognition technology has been widely used in various audit scenarios, such as the operator's business ID verification scenario. Operators need to audit the consistency of on-site photos and ID photos of natural persons handling business. However, facial photos taken on-site can be easily forged using technologies such as copying, so copying detection is required.

[0003] In related technologies, traditional image processing technology is used to detect the copying of facial photos, such as reconstructing the denoised image through wavelet transform, subtracting the original image from the denoised image to obtain the noise distribution, and then using relevant indicators as features to characterize the noise histograms of the two types of images. These feature indicators are classified by classifiers to obtain the copying results.

[0004] The detection process of the above-mentioned method based on traditional image processing technology is relatively cumbersome and requires more preprocessing operations before the classification task, resulting in low detection efficiency. In addition, this method is difficult to integrate into the application end of the detection system, which causes great inconvenience for large-scale audit tasks. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to address the above-mentioned shortcomings of the prior art. One purpose of the present invention is to provide a face reshooting audit method based on deep learning to improve the audit efficiency and accuracy.

[0006] The second purpose of the present invention is to provide a face re-photographing audit system based on deep learning to improve audit efficiency and accuracy.

[0007] In order to achieve the above-mentioned objective 1, the present invention provides a face re-photographing auditing method based on deep learning, comprising:

[0008] Collect as much facial image data as possible that is relevant to the application scenario of face auditing, and divide the collected facial image data into positive samples and negative samples;

[0009] The positive samples are enhanced to obtain a positive sample set, and the negative samples are enhanced to obtain a negative sample set. The positive sample set and the negative sample set are respectively divided into a training set and a test set in a ratio of 8:2. The training set is used for model learning and training, and the test set is used for model performance evaluation;

[0010] Constructing a neural network model, training the neural network model according to the training set, and evaluating the trained neural network model according to the test set to obtain a face remake detection model;

[0011] A face image is collected on site and input into the face duplication detection model to obtain a face duplication detection result.

[0012] As a further improvement, building a neural network model specifically includes:

[0013] Construct the input layer, which is the starting point of the network and is responsible for receiving input images from the outside;

[0014] Construct the first convolutional layer to perform convolution operations on the input layer. The convolution operation can extract the features of the input image. The convolution kernel size is 3×3, the number is 16, the convolution step is set to 1, and the padding is set to same so that the image size does not change after convolution.

[0015] Construct the first ReLU layer to perform nonlinear activation on the output of the first convolutional layer, which increases the nonlinear mapping capability of the network.

[0016] Construct the first BN layer to normalize the output of the first ReLU layer;

[0017] Construct the first Dropout layer to perform dropout operation on the output of the first BN layer. Regularization is achieved by randomly discarding input neurons. The dropout rate is set to 0.5.

[0018] Construct the second convolutional layer to perform convolution on the output of the first Dropout layer. The convolution kernel size is 3×3, the number is 32, the convolution step is set to 1, and the padding is set to the same;

[0019] Construct the second ReLU layer to perform nonlinear activation on the output of the second convolutional layer;

[0020] Construct the second BN layer to normalize the output of the second ReLU layer;

[0021] Construct the second Dropout layer to perform dropout operation on the output of the second BN layer, and set the dropout rate to 0.5;

[0022] Construct the first attention mechanism module, which consists of the first C module and the first S module. The first C module is the first half of the first attention mechanism module, and the first S module is the second half of the first attention mechanism module, which is used to further process the output of the second Dropout layer;

[0023] Construct the third convolutional layer to perform convolution on the output layer of the first attention mechanism module. The convolution kernel size is 3×3, the number is 64, the convolution step is set to 1, and the padding is set to the same;

[0024] Construct the third ReLU layer to perform nonlinear activation on the output of the third convolutional layer;

[0025] Construct the third BN layer to normalize the output of the third ReLU layer;

[0026] Construct the first MaxPooling layer to perform the maximum pooling operation on the output of the third BN layer. The pooling window size is set to 2×2, the pooling step size is set to 2, and the padding is set to the same;

[0027] Construct the third Dropout layer to perform dropout operation on the output of the first MaxPooling layer, and set the dropout rate to 0.5;

[0028] Construct a second attention mechanism module, which consists of a second C module and a second S module. The second C module is the first half of the second attention mechanism module, and the second S module is the second half of the second attention mechanism module, which is used to further process the output of the third Dropout layer.

[0029] Construct the fourth convolutional layer to perform convolution on the output layer of the second attention mechanism module. The convolution kernel size is 3×3, the number is 128, the convolution step is set to 1, and the padding is set to the same;

[0030] Construct the fourth ReLU layer to perform nonlinear activation on the output of the fourth convolutional layer;

[0031] Construct the fourth BN layer to normalize the output of the fourth ReLU layer;

[0032] Construct the second MaxPooling layer to perform the maximum pooling operation on the output of the fourth BN layer. The pooling window size is set to 2×2, the pooling step size is set to 2, and the padding is set to the same;

[0033] Construct the first Flatten layer to perform dimensionality reduction and flattening on the output of the second MaxPooling layer, that is, transforming the two-dimensional matrix into a one-dimensional vector;

[0034] Construct the first fully connected layer with 256 neurons, which is fully connected with the first Flatten layer.

[0035] Construct the fifth ReLU layer to perform nonlinear activation on the output of the first fully connected layer;

[0036] Construct the second fully connected layer with the number of neurons set to 128, which is used to fully connect with the fifth ReLU layer;

[0037] Construct the sixth ReLU layer to perform nonlinear activation on the output of the second fully connected layer;

[0038] Construct the output layer. The output layer is the end point of the network and is used to fully connect with the sixth ReLU layer to output the calculation results of the network.

[0039] Furthermore, the construction process of the first C module is as follows:

[0040] Construct the AvgPool layer of the first C module to perform average pooling on the output of the second Dropout layer;

[0041] Construct the MaxPool layer of the first C module to perform the maximum pooling operation on the output of the second Dropout layer;

[0042] Construct the first fully connected layer of the first C module, with the number of neurons set to 16, for full connection with the AvgPool layer and MaxPool layer of the first C module;

[0043] Construct the second fully connected layer of the first C module, with the number of neurons set to 32, for full connection with the two outputs of the first fully connected layer of the first C module;

[0044] Construct the Add layer of the first C module to add the two outputs of the second fully connected layer of the first C module;

[0045] Construct the Sigmoid layer of the first C module to perform nonlinear activation operations on the output of the Add layer;

[0046] Construct the Reshape layer of the first C module to reshape the output of the Sigmoid layer and change the dimension information of the Reshape layer;

[0047] Construct the Multiply layer of the first C module to multiply the output of the Reshape layer and the output of the second Dropout layer.

[0048] Furthermore, the construction process of the first S module is as follows:

[0049] Construct the AvgPool layer of the first S module to perform average pooling on the output of the Multiply layer of the first C module;

[0050] Construct the MaxPool layer of the first S module to perform the maximum pooling operation on the output of the Multiply layer of the first C module;

[0051] Construct the Concatenate layer of the first S module to perform a concatenation operation on the AvgPool layer and the MaxPool layer of the first S module;

[0052] Construct the convolution layer of the first S-module to perform convolution on the output of the Concatenate layer of the first S-module. The convolution kernel size is 7×7, the number is 1, the convolution step is set to 1, and the padding is set to the same;

[0053] Construct the Sigmoid layer of the first S module to perform nonlinear activation operations on the output of the convolutional layer of the first S module;

[0054] Construct the Multiply layer of the first S module to multiply the output of the Multiply layer of the first C module and the output of the Sigmoid layer of the S module. This layer is the output layer of the first attention mechanism module.

[0055] Furthermore, the construction process of the second C module is as follows:

[0056] Construct the AvgPool layer of the second C module to perform average pooling on the output of the third Dropout layer;

[0057] Construct the MaxPool layer of the second C module to perform the maximum pooling operation on the output of the third Dropout layer;

[0058] Construct the first fully connected layer of the second C module, with the number of neurons set to 16, for full connection with the AvgPool layer and MaxPool layer of the second C module;

[0059] Construct the second fully connected layer of the second C module, with the number of neurons set to 64, for full connection with the two outputs of the first fully connected layer of the second C module;

[0060] Construct the Add layer of the second C module to add the two outputs of the second fully connected layer of the second C module;

[0061] Construct the Sigmoid layer of the second C module to perform nonlinear activation operations on the output of the Add layer;

[0062] Construct the Reshape layer of the second C module to reshape the output of the Sigmoid layer and change the dimension information of the Reshape layer;

[0063] Construct the Multiply layer of the second C module to multiply the output of the Reshape layer and the output of the second Dropout layer.

[0064] Furthermore, the construction process of the second S module is as follows:

[0065] Construct the AvgPool layer of the second S module to perform average pooling on the output of the Multiply layer of the second C module;

[0066] Construct the MaxPool layer of the second S module to perform the maximum pooling operation on the output of the Multiply layer of the second C module;

[0067] Construct the Concatenate layer of the second S module to perform the concatenation operation on the AvgPool layer and the MaxPool layer of the second S module;

[0068] Construct the convolution layer of the second S-module to perform convolution on the output of the Concatenate layer of the second S-module. The convolution kernel size is 7×7, the number is 1, the convolution step is set to 1, and the padding is set to the same;

[0069] Construct the Sigmoid layer of the second S module to perform nonlinear activation operations on the output of the convolutional layer of the second S module;

[0070] Construct the Multiply layer of the second S module to multiply the output of the Multiply layer of the second C module and the output of the Sigmoid layer of the second S module as the output layer of the second attention mechanism module.

[0071] Furthermore, training the neural network model specifically includes:

[0072] Step S1. Initialize all parameters of the neural network model so that all parameters of the model satisfy the Gaussian distribution with a mean of 0 and a variance of 1;

[0073] Step S2. Set the initial learning rate to 10 -4 , the number of learning iterations is set to 50000 epochs, and the batch size batch_size is set to 32;

[0074] Step S3. Randomly select batch_size face images from the training set and input them into the neural network. After forward propagation through the network, the network prediction score is output.

[0075] Step S4. Use the following softmax cross entropy loss function as the loss function to calculate the loss between the network output and the true value:

[0076]

[0077] Among them, n is the number of categories predicted by the model, and the value of n here is 2, that is, binary classification; zi is the probability of each category predicted by the model; zy is the label of each sample, that is, the true value of the sample;

[0078] Step S5. Use the Adam adaptive gradient descent algorithm to update the weight parameters of the network, and the number of iterations increases by 1;

[0079] Step S6: Determine whether the set number of iterations has been reached. If so, the training ends and the network weight parameters are saved. If not, repeat step S3.

[0080] Furthermore, evaluating the trained neural network model specifically includes:

[0081] Step S11: performing pixel normalization on the images in the test set;

[0082] Step S12: Input all images in the test set into the network model to obtain the classification probability of the network for all images;

[0083] Step S13. Obtain the confusion matrix by comparing the classification results and the true values ​​of the test set samples:

[0084]

[0085] Among them, TP represents the number of positive examples predicted as positive, FP represents the number of negative examples predicted as positive, FN represents the number of positive examples predicted as negative, and TN represents the number of negative examples predicted as negative.

[0086] Step S14. Calculate the value of the evaluation parameter through the confusion matrix:

[0087]

[0088]

[0089]

[0090]

[0091] Among them, Accuracy is the accuracy, Precision is the precision, Recall is the recall rate, and F1 score is the harmonic mean of F1 precision and recall rate.

[0092] Step S15: Change the model weight parameters and continue with step 12 until a set number of test results are obtained;

[0093] Step S16: Compare different test results and select the model with the best evaluation parameters as the face remake detection model.

[0094] In order to achieve the above-mentioned second objective, the present invention provides a face reshooting audit system based on deep learning, including an underlying capability library and an audit application layer;

[0095] The underlying capability library includes an image information processing module for acquiring facial images and obtaining face retake detection results based on the above-mentioned deep learning-based face retake audit method; the underlying capability library can be encapsulated into SDK, dll or so format files according to the motion environment for invocation by the audit application layer;

[0096] The audit application layer includes:

[0097] The personnel structure module is responsible for managing personnel and organizational structures. It associates permissions through the scope of responsibilities and is divided into two roles: administrator and auditor. The administrator is responsible for managing personnel and organizational structures, and assigning audit permissions to auditors. Auditors are responsible for conducting secondary audits of the results of face re-photographing detection.

[0098] The audit task management module is used to package the problematic face re-photographing detection results into task orders based on personnel responsibilities and permissions, and distribute them to the corresponding auditors. The auditors can review the problematic data, and the audit task management module automatically tracks and reviews the processing results until the rectification is completed and the audit process ends;

[0099] Statistical query module, used to view or compile statistics of problem information based on regional dimensions;

[0100] The database and nginx modules are responsible for storing work order information, image processing results, and ensuring high system availability.

[0101] Beneficial effects

[0102] Compared with the prior art, the present invention has the following advantages:

[0103] The method of the present invention is based on the deep learning technology in artificial intelligence. Compared with the underlying algorithms of other existing face reshooting and auditing technologies, it has the advantages of fast computing speed, high accuracy, and easy integration. In addition, the image information processing module of the system of the present invention can be developed separately as a bottom-level library, and other modules can be developed separately as an application layer. The coupling degree between the two is low, which makes it easy to expand the system and meet the personalized needs of users. BRIEF DESCRIPTION OF THE DRAWINGS

[0104] Figure 1 This is an architecture diagram of the system in the present invention;

[0105] Figure 2It is the workflow diagram of the system in the present invention. DETAILED DESCRIPTION

[0106] The present invention will be further described below with reference to the specific embodiments in the accompanying drawings.

[0107] See Figure 1 、 2 , a face remake audit method based on deep learning, including:

[0108] Collect as much facial image data as possible related to the application scenarios of face auditing, and divide the collected facial image data into positive samples and negative samples. Specifically, the collected facial image data are divided into positive samples and negative samples according to the classification standard of re-photographed images. Positive samples are non-re-photographed images, that is, natural images of faces obtained through normal photography. Negative samples are re-photographed images, which are characterized by blurry images, moiré patterns, reflections, and the presence of mobile phone or computer frames. Label the positive and negative samples, with the label of the positive sample set to 1 and the label of the negative sample set to 0.

[0109] The positive samples are enhanced to obtain the positive sample set, and the negative samples are enhanced to obtain the negative sample set. The enhancement process involves performing operations such as image flipping, image cropping, distorting image features, and enhancing image noise on the image to increase the number of samples and make the model have better generalization ability. Even if the image data quality is relatively poor, the model can learn useful information from a large amount of image data, thereby achieving better results. After the enhancement process, the sample labels in the positive sample set are all 1, and the sample labels in the negative sample set are all 0. The positive sample set and the negative sample set are respectively divided into a training set and a test set in a ratio of 8:2. The training set is used for model learning and training, and the test set is used for model performance evaluation.

[0110] Construct a neural network model, train the neural network model based on the training set, and evaluate the trained neural network model based on the test set to obtain a face remake detection model;

[0111] Face images are collected on-site and fed into the face duplication detection model to obtain face duplication detection results. Face images collected on-site are often not usable directly due to various restrictions and random interference. They must undergo face image preprocessing in the early stages of image processing, such as light compensation, grayscale conversion, histogram equalization, normalization, geometric correction, filtering, and sharpening.

[0112] Building a neural network model specifically includes:

[0113] 1. Construct the input layer. The input layer is the starting point of the network and is responsible for receiving input images from the outside.

[0114] Construct the first convolutional layer to perform convolution operations on the input layer. The convolution operation can extract the features of the input image. The convolution kernel size is 3×3, the number is 16, the convolution step is set to 1, and the padding (padding pixels) is set to same so that the image size does not change after convolution.

[0115] Construct the first ReLU layer to perform nonlinear activation on the output of the first convolutional layer, which increases the nonlinear mapping capability of the network.

[0116] Construct the first BN layer to normalize the output of the first ReLU layer;

[0117] Construct the first Dropout layer to perform dropout operation on the output of the first BN layer. Regularization is achieved by randomly discarding input neurons. The dropout rate is set to 0.5.

[0118] Construct the second convolutional layer to perform convolution on the output of the first Dropout layer. The convolution kernel size is 3×3, the number is 32, the convolution step is set to 1, and the padding is set to the same;

[0119] Construct the second ReLU layer to perform nonlinear activation on the output of the second convolutional layer;

[0120] Construct the second BN layer to normalize the output of the second ReLU layer;

[0121] Construct the second Dropout layer to perform dropout operation on the output of the second BN layer, and set the dropout rate to 0.5;

[0122] Construct the first attention mechanism module, which consists of the first C module and the first S module. The first C module is the first half of the first attention mechanism module, and the first S module is the second half of the first attention mechanism module, which is used to further process the output of the second Dropout layer;

[0123] Construct the third convolutional layer to perform convolution on the output layer of the first attention mechanism module. The convolution kernel size is 3×3, the number is 64, the convolution step is set to 1, and the padding is set to the same;

[0124] Construct the third ReLU layer to perform nonlinear activation on the output of the third convolutional layer;

[0125] Construct the third BN layer to normalize the output of the third ReLU layer;

[0126] Construct the first MaxPooling layer to perform the maximum pooling operation on the output of the third BN layer. The pooling window size is set to 2×2, the pooling step size is set to 2, and the padding is set to the same;

[0127] Construct the third Dropout layer to perform dropout operation on the output of the first MaxPooling layer, and set the dropout rate to 0.5;

[0128] Construct a second attention mechanism module, which consists of a second C module and a second S module. The second C module is the first half of the second attention mechanism module, and the second S module is the second half of the second attention mechanism module, which is used to further process the output of the third Dropout layer.

[0129] Construct the fourth convolutional layer to perform convolution on the output layer of the second attention mechanism module. The convolution kernel size is 3×3, the number is 128, the convolution step is set to 1, and the padding is set to the same;

[0130] Construct the fourth ReLU layer to perform nonlinear activation on the output of the fourth convolutional layer;

[0131] Construct the fourth BN layer to normalize the output of the fourth ReLU layer;

[0132] Construct the second MaxPooling layer to perform the maximum pooling operation on the output of the fourth BN layer. The pooling window size is set to 2×2, the pooling step size is set to 2, and the padding is set to the same;

[0133] Construct the first Flatten layer to perform dimensionality reduction and flattening on the output of the second MaxPooling layer, that is, transforming the two-dimensional matrix into a one-dimensional vector;

[0134] Construct the first fully connected layer with 256 neurons, which is fully connected with the first Flatten layer.

[0135] Construct the fifth ReLU layer to perform nonlinear activation on the output of the first fully connected layer;

[0136] Construct the second fully connected layer with the number of neurons set to 128, which is used to fully connect with the fifth ReLU layer;

[0137] Construct the sixth ReLU layer to perform nonlinear activation on the output of the second fully connected layer;

[0138] Construct the output layer. The output layer is the end point of the network and is used to fully connect with the sixth ReLU layer to output the calculation results of the network.

[0139] The construction process of the first C module is as follows:

[0140] Construct the AvgPool layer of the first C module to perform average pooling on the output of the second Dropout layer;

[0141] Construct the MaxPool layer of the first C module to perform the maximum pooling operation on the output of the second Dropout layer;

[0142] Construct the first fully connected layer of the first C module, with the number of neurons set to 16, for full connection with the AvgPool layer and MaxPool layer of the first C module;

[0143] Construct the second fully connected layer of the first C module, with the number of neurons set to 32, for full connection with the two outputs of the first fully connected layer of the first C module;

[0144] Construct the Add layer of the first C module to add the two outputs of the second fully connected layer of the first C module;

[0145] Construct the Sigmoid layer of the first C module to perform nonlinear activation operations on the output of the Add layer;

[0146] Construct the Reshape layer of the first C module to reshape the output of the Sigmoid layer and change the dimension information of the Reshape layer;

[0147] Construct the Multiply layer of the first C module to multiply the output of the Reshape layer and the output of the second Dropout layer.

[0148] The construction process of the first S module is as follows:

[0149] Construct the AvgPool layer of the first S module to perform average pooling on the output of the Multiply layer of the first C module;

[0150] Construct the MaxPool layer of the first S module to perform the maximum pooling operation on the output of the Multiply layer of the first C module;

[0151] Construct the Concatenate layer of the first S module to perform a concatenation operation on the AvgPool layer and the MaxPool layer of the first S module;

[0152] Construct the convolution layer of the first S-module to perform convolution on the output of the Concatenate layer of the first S-module. The convolution kernel size is 7×7, the number is 1, the convolution step is set to 1, and the padding is set to the same;

[0153] Construct the Sigmoid layer of the first S module to perform nonlinear activation operations on the output of the convolutional layer of the first S module;

[0154] Construct the Multiply layer of the first S module to multiply the output of the Multiply layer of the first C module and the output of the Sigmoid layer of the S module. This layer is the output layer of the first attention mechanism module.

[0155] The construction process of the second C module is as follows:

[0156] Construct the AvgPool layer of the second C module to perform average pooling on the output of the third Dropout layer;

[0157] Construct the MaxPool layer of the second C module to perform the maximum pooling operation on the output of the third Dropout layer;

[0158] Construct the first fully connected layer of the second C module, with the number of neurons set to 16, for full connection with the AvgPool layer and MaxPool layer of the second C module;

[0159] Construct the second fully connected layer of the second C module, with the number of neurons set to 64, for full connection with the two outputs of the first fully connected layer of the second C module;

[0160] Construct the Add layer of the second C module to add the two outputs of the second fully connected layer of the second C module;

[0161] Construct the Sigmoid layer of the second C module to perform nonlinear activation operations on the output of the Add layer;

[0162] Construct the Reshape layer of the second C module to reshape the output of the Sigmoid layer and change the dimension information of the Reshape layer;

[0163] Construct the Multiply layer of the second C module to multiply the output of the Reshape layer and the output of the second Dropout layer.

[0164] The construction process of the second S module is as follows:

[0165] Construct the AvgPool layer of the second S module to perform average pooling on the output of the Multiply layer of the second C module;

[0166] Construct the MaxPool layer of the second S module to perform the maximum pooling operation on the output of the Multiply layer of the second C module;

[0167] Construct the Concatenate layer of the second S module to perform the concatenation operation on the AvgPool layer and the MaxPool layer of the second S module;

[0168] Construct the convolution layer of the second S-module to perform convolution on the output of the Concatenate layer of the second S-module. The convolution kernel size is 7×7, the number is 1, the convolution step is set to 1, and the padding is set to the same;

[0169] Construct the Sigmoid layer of the second S module to perform nonlinear activation operations on the output of the convolutional layer of the second S module;

[0170] Construct the Multiply layer of the second S module to multiply the output of the Multiply layer of the second C module and the output of the Sigmoid layer of the second S module as the output layer of the second attention mechanism module.

[0171] Training a neural network model specifically involves:

[0172] Step S1. Initialize all parameters of the neural network model so that all parameters of the model satisfy the Gaussian distribution with a mean of 0 and a variance of 1;

[0173] Step S2. Set the initial learning rate to 10 -4 , the number of learning iterations is set to 50000 epochs, and the batch size batch_size is set to 32;

[0174] Step S3. Randomly select batch_size face images from the training set and input them into the neural network. After forward propagation through the network, the network prediction score is output.

[0175] Step S4. Use the following softmax cross entropy loss function as the loss function to calculate the loss between the network output and the true value:

[0176]

[0177] Where n is the number of categories predicted by the model. Here, the value of n is 2, which means two categories, namely remake and non-remake; zi is the probability of each category predicted by the model; zy is the label of each sample, that is, the true value of the sample;

[0178] Step S5. Use the Adam adaptive gradient descent algorithm to update the weight parameters of the network, and the number of iterations increases by 1;

[0179] Step S6: Determine whether the set number of iterations has been reached. If so, the training ends and the network weight parameters are saved. If not, repeat step S3.

[0180] Evaluating a trained neural network model involves:

[0181] Step S11: performing pixel normalization on the images in the test set;

[0182] Step S12: Input all images in the test set into the network model to obtain the classification probability of the network for all images;

[0183] Step S13. Obtain the confusion matrix by comparing the classification results and the true values ​​of the test set samples:

[0184]

[0185]

[0186] Among them, TP represents the number of positive examples predicted as positive, FP represents the number of negative examples predicted as positive, FN represents the number of positive examples predicted as negative, and TN represents the number of negative examples predicted as negative.

[0187] Step S14. Calculate the value of the evaluation parameter through the confusion matrix:

[0188]

[0189]

[0190]

[0191]

[0192] Among them, Accuracy is the accuracy, Precision is the precision, Recall is the recall rate, and F1 score is the harmonic mean of F1 precision and recall rate.

[0193] Step S15: Change the model weight parameters and continue with step 12 until a set number of test results are obtained;

[0194] Step S16: Compare different test results and select the model with the best evaluation parameters as the face remake detection model.

[0195] A deep learning-based face re-photography audit system, including an underlying capability library and an audit application layer;

[0196] The underlying capability library includes an image information processing module for acquiring facial images and obtaining face remake detection results based on the aforementioned deep learning-based face remake audit method. The underlying capability library can be packaged into SDK, dll, or so format files based on the motion environment for invocation by the audit application layer.

[0197] The audit application layer includes:

[0198] The personnel structure module is used to manage personnel and organizational structure. It is divided into two roles, administrator and auditor, through the scope of responsibilities and associated permissions. Administrators are responsible for the management of personnel and organizational structure, and assigning audit permissions to auditors. Auditors are responsible for secondary auditing of the results of face re-photographing detection. Personnel management can create, modify, and delete user information. The manageable information includes user name and contact information, login username and password, associated areas, and batch import of information. Organizational structure management can create, modify, and delete organizations, as well as associate and cancel administrative areas and adjust levels. Auditors have the following permissions: view their own user information; modify and retrieve their own account passwords; have the permissions to view, export, count, and task status of audit information of user-bound areas and their subordinates. System administrators, compared with auditors, have more permissions including user management permissions, authorization permissions, and the permission to view all audit information. User authorization permissions mean that administrators can grant administrator and auditor identities to other users.

[0199] The audit task management module is used to package the result information of problematic face re-photographing detection into task orders according to personnel responsibilities and authorities, and distribute them to corresponding auditors. Auditors can view the problematic data, and the audit task management module automatically tracks and reviews the processing results until the rectification is correct and the audit process is terminated. Specifically, a timed task can be set to automatically create an audit task at a specified time point and execute the audit task periodically. When the audit task starts to execute, the facial image information to be detected will be pulled from the database interface, and these images will be re-photographed and detected. The detection results will be written to the database or recorded in text. Based on the detection results, the problematic work orders will be packaged into audit tasks according to the scope of responsibility of the auditors, and then distributed to the auditors. Only one user can sign for an audit task. After signing, only the signatory can provide feedback on the task order, and other personnel can only view it. In the audit task, facial images that fail the re-photographing detection can be browsed. Auditors can confirm whether it is a system audit error by viewing the image and submit feedback on the results.

[0200] Statistical query module, used to view or compile statistics of problem information based on regional dimensions;

[0201] The database and nginx modules are responsible for storing work order information, image processing results, and ensuring high system availability.

[0202] The present invention performs the following process for detecting a specific face image remake:

[0203] 1. Obtain a scene image and preprocess it;

[0204] 2. Resize the preprocessed image to 224×224 size;

[0205] The network’s input layer receives the preprocessed image and converts it into a 224×224×3 tensor.

[0206] 4. The 16 3×3 convolution kernels of the first convolutional layer perform convolution operations on the input layer tensor to obtain 16 224×224 feature maps, which can be regarded as a 224×224×16 tensor;

[0207] 5. The first ReLU layer performs a nonlinear activation operation on the tensor of the first convolutional layer to obtain a 224×224×16 tensor;

[0208] 6. The first BN layer normalizes the tensor of the first ReLU layer to obtain a 224×224×16 tensor;

[0209] 7. The first Dropout layer randomly drops neurons from the tensor of the first BN layer to obtain a 224×224×16 tensor;

[0210] 8. The 32 3×3 convolution kernels of the second convolutional layer perform convolution operations on the first Dropout layer tensor, obtaining 32 224×224 feature maps, which can be regarded as a 224×224×32 tensor;

[0211] 9. The second ReLU layer performs a nonlinear activation operation on the tensor of the second convolutional layer to obtain a 224×224×32 tensor;

[0212] 10. The second BN layer normalizes the tensor of the second ReLU layer to obtain a 224×224×32 tensor;

[0213] 11. The second Dropout layer randomly drops neurons from the tensor of the second BN layer to obtain a 224×224×32 tensor;

[0214] 12. The first attention mechanism module processes the second Dropout layer tensor as follows: (1) The AvgPool layer of the C module performs an average pooling operation on the second Dropout layer tensor to obtain a one-dimensional tensor with a length of 32; (2) The MaxPool layer of the C module performs a maximum pooling operation on the second Dropout layer tensor to obtain a one-dimensional tensor with a length of 32; (3) The first fully connected layer of the C module is fully connected with the tensors of the MaxPool layer and the AvgPool layer respectively to obtain two one-dimensional tensors with a length of 16; (4) The second fully connected layer of the C module is fully connected with the tensors of the MaxPool layer and the AvgPool layer The two tensors of the first fully connected layer are fully connected to obtain two one-dimensional tensors of length 32; (5) The Add layer of the C module adds the two tensors of the second fully connected layer to obtain a one-dimensional tensor of length 32; (6) The Activation layer of the C module performs a nonlinear activation operation on the Add layer to obtain a one-dimensional tensor of length 32; (7) The Reshape layer of the C module adjusts the one-dimensional tensor of the Activation layer to a 1×1×32 tensor; (8) The Multiply layer of the C module performs a nonlinear activation operation on the tensors of the Reshape layer and the second Dropout layer. The multiplication operation is performed on the tensors of the Multiply layer of the S module to obtain a 224×224×32 tensor; (9) The AvgPool layer of the S module performs an average pooling operation on the third channel of the Multiply layer tensor of the C module to obtain a 224×224×1 tensor; (10) The MaxPool layer of the S module performs an average pooling operation on the third channel of the Multiply layer tensor of the C module to obtain a 224×224×1 tensor; (11) The Concatenate layer of the S module continues to splice the tensors of the MaxPool and AvgPool layers of the S module to obtain a 224 ×224×2 tensor; (12) A 7×7 convolution kernel of the convolution layer of the S module performs a convolution operation on the tensor of the Concatenate layer to obtain a 224×224×1 tensor; (13) The Activation layer of the S module performs a nonlinear activation operation on the tensor of the Convolution layer to obtain a 224×224×1 tensor; (14) The Multiply layer of the S module performs a multiplication operation on the Activation layer and the Multiply layer of the C module to obtain a 224×224×32 tensor, which is also the tensor output by the first attention mechanism module;

[0215] 13. The 64 3×3 convolution kernels of the third convolutional layer perform convolution operations on the tensor output by the first attention mechanism module, resulting in 64 224×224 feature maps, that is, a tensor of size 224×224×64;

[0216] 14. The third ReLU layer performs a nonlinear activation operation on the tensor of the third convolutional layer to obtain a 224×224×64 tensor;

[0217] 15. The third BN layer normalizes the tensor of the third ReLU layer to obtain a 224×224×64 tensor;

[0218] 16. The first MaxPooling layer performs a maximum pooling operation on the tensor of the third BN layer with a window size of 2×2, obtaining a 112×112×64 tensor;

[0219] 17. The third Dropout layer randomly drops neurons from the tensor of the first MaxPooling layer to obtain a 112×112×64 tensor;

[0220] 18. The second attention mechanism module processes the third Dropout layer tensor as follows: (1) The AvgPool layer of the C module performs an average pooling operation on the third Dropout layer tensor to obtain a one-dimensional tensor with a length of 64; (2) The MaxPool layer of the C module performs a maximum pooling operation on the third Dropout layer tensor to obtain a one-dimensional tensor with a length of 64; (3) The first fully connected layer of the C module is fully connected with the tensors of the MaxPool layer and the AvgPool layer respectively to obtain two one-dimensional tensors with a length of 16; (4) The second fully connected layer of the C module is fully connected with the tensors of the MaxPool layer and the AvgPool layer The two tensors of the first fully connected layer are fully connected to obtain two one-dimensional tensors of length 64; (5) The Add layer of the C module adds the two tensors of the second fully connected layer to obtain a one-dimensional tensor of length 64; (6) The Activation layer of the C module performs a nonlinear activation operation on the Add layer to obtain a one-dimensional tensor of length 64; (7) The Reshape layer of the C module adjusts the one-dimensional tensor of the Activation layer to a 1×1×64 tensor; (8) The Multiply layer of the C module performs a nonlinear activation operation on the tensors of the Reshape layer and the third Dropout layer. The multiplication operation is performed on the tensors of the Multiply layer of the S module to obtain a 112×112×64 tensor; (9) The AvgPool layer of the S module performs an average pooling operation on the third channel of the Multiply layer tensor of the C module to obtain a 112×112×1 tensor; (10) The MaxPool layer of the S module performs an average pooling operation on the third channel of the Multiply layer tensor of the C module to obtain a 112×112×1 tensor; (11) The Concatenate layer of the S module continues to splice the tensors of the MaxPool and AvgPool layers of the S module to obtain a 112 ×112×2 tensor; (12) A 7×7 convolution kernel of the convolution layer of the S module performs a convolution operation on the tensor of the Concatenate layer to obtain a 112×112×1 tensor; (13) The Activation layer of the S module performs a nonlinear activation operation on the tensor of the Convolution layer to obtain a 112×112×1 tensor; (14) The Multiply layer of the S module performs a multiplication operation on the Activation layer and the Multiply layer of the C module to obtain a 112×112×64 tensor, which is also the tensor output by the second attention mechanism module;

[0221] 19. The 128 3×3 convolution kernels of the fourth convolutional layer perform convolution operations on the tensor output by the second attention mechanism module, resulting in 128 112×112 feature maps, that is, a tensor of size 112×112×128;

[0222] 20. The fourth ReLU layer performs a nonlinear activation operation on the tensor of the fourth convolutional layer to obtain a 112×112×128 tensor;

[0223] 21. The fourth BN layer normalizes the tensor of the fourth ReLU layer to obtain a 112×112×128 tensor;

[0224] 22. The second MaxPooling layer performs a maximum pooling operation on the tensor of the fourth BN layer with a window size of 2×2, obtaining a 64×64×128 tensor;

[0225] 23. The first Flatten layer flattens the second MaxPooling tensor to obtain a one-dimensional tensor of length 401408;

[0226] 24. The first fully connected layer is fully connected to the first Flatten layer to obtain a one-dimensional tensor of length 256;

[0227] 25. The second fully connected layer is fully connected to the first fully connected layer to obtain a one-dimensional tensor of length 128;

[0228] 26. The output layer is connected to the second fully connected layer to obtain a one-dimensional tensor of length 2;

[0229] 27. Compare the two probabilities output by the output layer and select the larger one as the output result of the network. This result is the probability that the face photo is a fake.

[0230] The above is only a preferred embodiment of the present invention. It should be pointed out that for those skilled in the art, several modifications and improvements can be made without departing from the structure of the present invention. These will not affect the effect of the implementation of the present invention and the practicality of the patent.

Claims

1. A face re-photographing audit method based on deep learning, characterized in that: include: Collect as much facial image data as possible that is relevant to the application scenario of face auditing, and divide the collected facial image data into positive samples and negative samples; The positive samples are enhanced to obtain a positive sample set, and the negative samples are enhanced to obtain a negative sample set. The positive sample set and the negative sample set are respectively divided into a training set and a test set in a ratio of 8:

2. The training set is used for model learning and training, and the test set is used for model performance evaluation; Constructing a neural network model, training the neural network model according to the training set, and evaluating the trained neural network model according to the test set to obtain a face remake detection model; Collecting facial images on site and inputting them into the face remake detection model to obtain face remake detection results; Building a neural network model specifically includes: Construct the input layer, which is the starting point of the network and is responsible for receiving input images from the outside; Construct the first convolutional layer to perform convolution operations on the input layer. The convolution operation can extract the features of the input image. The convolution kernel size is 3×3, the number is 16, the convolution step is set to 1, and the padding is set to same so that the image size does not change after convolution. Construct the first ReLU layer to perform nonlinear activation on the output of the first convolutional layer, which increases the nonlinear mapping capability of the network. Construct the first BN layer to normalize the output of the first ReLU layer; Construct the first Dropout layer to perform dropout operation on the output of the first BN layer. Regularization is achieved by randomly discarding input neurons. The dropout rate is set to 0.

5. Construct the second convolutional layer to perform convolution on the output of the first Dropout layer. The convolution kernel size is 3×3, the number is 32, the convolution step is set to 1, and the padding is set to the same; Construct the second ReLU layer to perform nonlinear activation on the output of the second convolutional layer; Construct the second BN layer to normalize the output of the second ReLU layer; Construct the second Dropout layer to perform dropout operation on the output of the second BN layer, and set the dropout rate to 0.5; Construct the first attention mechanism module, which consists of the first C module and the first S module. The first C module is the first half of the first attention mechanism module, and the first S module is the second half of the first attention mechanism module, which is used to further process the output of the second Dropout layer; Construct the third convolutional layer to perform convolution operations on the output layer of the first attention mechanism module. The convolution kernel size is 3×3, the number is 64, the convolution step is set to 1, and the padding is set to the same; Construct the third ReLU layer to perform nonlinear activation on the output of the third convolutional layer; Construct the third BN layer to normalize the output of the third ReLU layer; Construct the first MaxPooling layer to perform the maximum pooling operation on the output of the third BN layer. The pooling window size is set to 2×2, the pooling step size is set to 2, and the padding is set to the same; Construct the third Dropout layer to perform dropout operation on the output of the first MaxPooling layer, and set the dropout rate to 0.5; Construct a second attention mechanism module, which consists of a second C module and a second S module. The second C module is the first half of the second attention mechanism module, and the second S module is the second half of the second attention mechanism module, which is used to further process the output of the third Dropout layer. Construct the fourth convolutional layer to perform convolution on the output layer of the second attention mechanism module. The convolution kernel size is 3×3, the number is 128, the convolution step is set to 1, and the padding is set to the same; Construct the fourth ReLU layer to perform nonlinear activation on the output of the fourth convolutional layer; Construct the fourth BN layer to normalize the output of the fourth ReLU layer; Construct the second MaxPooling layer to perform the maximum pooling operation on the output of the fourth BN layer. The pooling window size is set to 2×2, the pooling step size is set to 2, and the padding is set to the same; Construct the first Flatten layer to perform dimensionality reduction and flattening on the output of the second MaxPooling layer, that is, transforming the two-dimensional matrix into a one-dimensional vector; Construct the first fully connected layer with 256 neurons, which is fully connected with the first Flatten layer. Construct the fifth ReLU layer to perform nonlinear activation on the output of the first fully connected layer; Construct the second fully connected layer with the number of neurons set to 128, which is used to fully connect with the fifth ReLU layer; Construct the sixth ReLU layer to perform nonlinear activation on the output of the second fully connected layer; Construct the output layer, which is the end point of the network and is used to fully connect with the sixth ReLU layer to output the network calculation results; Training the neural network model specifically includes: Step S1. Initialize all parameters of the neural network model so that all parameters of the model satisfy the Gaussian distribution with a mean of 0 and a variance of 1; Step S2. Set the initial learning rate to 10 -4 , the number of learning iterations is set to 50000 epochs, and the batch size batch_size is set to 32; Step S3. Randomly select batch_size face images from the training set and input them into the neural network. After forward propagation through the network, the network prediction score is output. Step S4. Use the following softmax cross entropy loss function as the loss function to calculate the loss between the network output and the true value: Among them, n is the number of categories predicted by the model, and the value of n here is 2, that is, binary classification; zi is the probability of each category predicted by the model; zy is the label of each sample, that is, the true value of the sample; Step S5. Use the Adam adaptive gradient descent algorithm to update the weight parameters of the network, and the number of iterations increases by 1; Step S6: Determine whether the set number of iterations has been reached. If so, the training ends and the network weight parameters are saved. If not, repeat step S3.

2. The method for verifying face reproduction based on deep learning according to claim 1, characterized in that: The construction process of the first C module is as follows: Construct the AvgPool layer of the first C module to perform average pooling on the output of the second Dropout layer; Construct the MaxPool layer of the first C module to perform the maximum pooling operation on the output of the second Dropout layer; Construct the first fully connected layer of the first C module, with the number of neurons set to 16, for full connection with the AvgPool layer and MaxPool layer of the first C module; Construct the second fully connected layer of the first C module, with the number of neurons set to 32, for full connection with the two outputs of the first fully connected layer of the first C module; Construct the Add layer of the first C module to add the two outputs of the second fully connected layer of the first C module; Construct the Sigmoid layer of the first C module to perform nonlinear activation operations on the output of the Add layer; Construct the Reshape layer of the first C module to reshape the output of the Sigmoid layer and change the dimension information of the Reshape layer; Construct the Multiply layer of the first C module to multiply the output of the Reshape layer and the output of the second Dropout layer.

3. The method for verifying face reproduction based on deep learning according to claim 1, characterized in that: The construction process of the first S module is as follows: Construct the AvgPool layer of the first S module to perform average pooling on the output of the Multiply layer of the first C module; Construct the MaxPool layer of the first S module to perform the maximum pooling operation on the output of the Multiply layer of the first C module; Construct the Concatenate layer of the first S module to perform a concatenation operation on the AvgPool layer and the MaxPool layer of the first S module; Construct the convolution layer of the first S-module to perform convolution on the output of the Concatenate layer of the first S-module. The convolution kernel size is 7×7, the number is 1, the convolution step is set to 1, and the padding is set to the same; Construct the Sigmoid layer of the first S module to perform nonlinear activation operations on the output of the convolutional layer of the first S module; Construct the Multiply layer of the first S module to multiply the output of the Multiply layer of the first C module and the output of the Sigmoid layer of the S module. This layer is the output layer of the first attention mechanism module.

4. The method for verifying face reproduction based on deep learning according to claim 1, characterized in that: The construction process of the second C module is as follows: Construct the AvgPool layer of the second C module to perform average pooling on the output of the third Dropout layer; Construct the MaxPool layer of the second C module to perform the maximum pooling operation on the output of the third Dropout layer; Construct the first fully connected layer of the second C module, with the number of neurons set to 16, for full connection with the AvgPool layer and MaxPool layer of the second C module; Construct the second fully connected layer of the second C module, with the number of neurons set to 64, for full connection with the two outputs of the first fully connected layer of the second C module; Construct the Add layer of the second C module to add the two outputs of the second fully connected layer of the second C module; Construct the Sigmoid layer of the second C module to perform nonlinear activation operations on the output of the Add layer; Construct the Reshape layer of the second C module to reshape the output of the Sigmoid layer and change the dimension information of the Reshape layer; Construct the Multiply layer of the second C module to multiply the output of the Reshape layer and the output of the second Dropout layer.

5. The method for verifying face reproduction based on deep learning according to claim 1, characterized in that: The construction process of the second S module is as follows: Construct the AvgPool layer of the second S module to perform average pooling on the output of the Multiply layer of the second C module; Construct the MaxPool layer of the second S module to perform the maximum pooling operation on the output of the Multiply layer of the second C module; Construct the Concatenate layer of the second S module to perform the concatenation operation on the AvgPool layer and the MaxPool layer of the second S module; Construct the convolution layer of the second S-module to perform convolution on the output of the Concatenate layer of the second S-module. The convolution kernel size is 7×7, the number is 1, the convolution step is set to 1, and the padding is set to the same; Construct the Sigmoid layer of the second S module to perform nonlinear activation operations on the output of the convolutional layer of the second S module; Construct the Multiply layer of the second S module to multiply the output of the Multiply layer of the second C module and the output of the Sigmoid layer of the second S module as the output layer of the second attention mechanism module.

6. The method for verifying face reproduction based on deep learning according to claim 1, characterized in that: Evaluating a trained neural network model involves: Step S11: performing pixel normalization on the images in the test set; Step S12: Input all images in the test set into the network model to obtain the classification probability of the network for all images; Step S13. Obtain the confusion matrix by comparing the classification results and the true values ​​of the test set samples: Among them, TP represents the number of positive examples predicted as positive, FP represents the number of negative examples predicted as positive, FN represents the number of positive examples predicted as negative, and TN represents the number of negative examples predicted as negative. Step S14. Calculate the value of the evaluation parameter through the confusion matrix: Among them, Accuracy is the accuracy, Precision is the precision, Recall is the recall rate, and F1 score is the harmonic mean of F1 precision and recall rate; Step S15: Change the model weight parameters and continue with step 12 until a set number of test results are obtained; Step S16: Compare different test results and select the model with the best evaluation parameters as the face remake detection model.

7. A face remake audit system based on deep learning, characterized by: Including the underlying capability library and audit application layer; The underlying capability library image information processing module is used to obtain a facial image and obtain a result of face remake detection according to a face remake audit method based on deep learning according to any one of claims 1 to 6; The underlying capability library can be encapsulated into SDK, dll or so format files according to the motion environment for the audit application layer to call; The audit application layer includes: The personnel structure module is responsible for managing personnel and organizational structures. It associates permissions through the scope of responsibilities and is divided into two roles: administrator and auditor. The administrator is responsible for managing personnel and organizational structures, and assigning audit permissions to auditors. Auditors are responsible for conducting secondary audits of the results of face re-photographing detection. The audit task management module is used to package the problematic face re-photographing detection results into task orders based on personnel responsibilities and permissions, and distribute them to the corresponding auditors. The auditors can review the problematic data, and the audit task management module automatically tracks and reviews the processing results until the rectification is completed and the audit process ends; Statistical query module, used to view or compile statistics of problem information based on regional dimensions; The database and nginx modules are responsible for storing work order information, image processing results, and ensuring high system availability.

Images