A face recognition privacy protection method that can achieve forced scrambling

Abstract

The present invention relates to a face recognition privacy protection method that can achieve forced scrambling. After the present invention is jointly trained end-to-end by two deep neural networks, namely E1 and G1, and then separated, a private face protection module E and a private face recognition module G are obtained. E1 is a new network obtained by adding convolutional layers and modifying the output layer to the deep neural network UNet, and G1 is a Resnet residual network. Module E can directly modify face data to generate face images with high blurriness that cannot be recognized by human vision; module G recognizes face images that have undergone face privacy protection processing. The present invention enhances the recognition rate of faces after privacy protection. During the recognition process, there is no need for a decryption process, but instead, the privacy-protected face images in the face database are directly used to compare and recognize the incoming face images, thereby achieving face privacy protection. The face images cannot be restored inversely, effectively protecting the security of the face database on the server side.

Description

Technical Field

[0001] The present invention belongs to the technical fields of face privacy protection and face recognition, and relates to a face recognition privacy protection method capable of realizing strong scrambling. Background Art

[0002] In today's big data era, with the rapid development of technology, the field of face recognition has been gradually improved, and various face recognition technologies have been used in various fields of life. However, the widespread use of face recognition systems has also brought a series of problems. Currently, the most widely mentioned is the issue of face privacy protection. The data of face recognition is crucial, which is directly related to personal property safety. However, in order to rapidly promote face recognition technology, face databases have been widely collected, which will bring many security risks that damage personal privacy, such as over-collection, improper use of databases, and theft of databases. In this environment, face privacy protection becomes extremely important.

[0003] In a face recognition system, a face image needs to be transmitted to a server through a collection device and compared with the face database in the server for recognition. In this process, some attempt to protect face privacy through encryption and decryption methods, that is, the face database in the server is encrypted, and the recognizable face image transmitted to the server is also encrypted. However, in the specific recognition process, it is still necessary to decrypt the transmitted face and the face in the face database for recognition. In this case, there is a risk of privacy leakage in the face database of the server.

[0004] Existing face privacy protection mostly deletes soft biometric information such as gender, age, and identity in a face image or a face image feature vector through a deep neural network. Although these methods protect face image privacy to a certain extent, the strength of face privacy protection is limited and can only play an auxiliary role in the field of face recognition privacy protection.

[0005] The present invention proposes a face recognition privacy protection method capable of realizing strong scrambling, which can better combine face privacy protection and face recognition technology. By processing a face image through a deep neural network, the face image cannot be recognized by human eye vision, but a specially designed neural network model can normally recognize the corresponding face, solving the problem of face privacy leakage. Summary of the Invention

[0006] The purpose of the present invention is to provide a face recognition privacy protection method capable of realizing strong scrambling.

[0007] The present invention includes a privacy protection face generation module E and a privacy protection face recognition module G, and both E and G are deep neural networks.

[0008] Modules E and G are two networks separated after end-to-end training by two deep neural networks, E1 and G1. E1 is a new network obtained by adding convolutional layers and modifying the output layer to the deep neural network UNet. G1 is a Resnet residual network.

[0009] Module E is used to directly modify face data to generate face images with high blurriness that are unrecognizable to human vision.

[0010] Module G is used to recognize face images processed by face privacy protection.

[0011] Specifically, it includes the following steps:

[0012] In the first step, select face image data from the face image database and perform face localization preprocessing operations on the face image data.

[0013] In the second step, pre-train the face image data processed in the first step in G1.

[0014] In the third step, send the face database into the large network of E1+G1 for end-to-end training. After the training is completed, separate the entire large network. The specific operation is as follows: Cut off the last part of the convolutional layer of E1 and retain the other layers of the deep neural network as the privacy protection face generation module E, and add the last convolutional layer of E1 to G1 as the privacy protection face recognition module G.

[0015] In the fourth step, send the face image into the privacy protection face generation module E obtained in the third step, output the privacy-protected face image and store it in the server as the privacy-protected face template database. During the recognition process, the privacy-protected face image is compared with the privacy-protected face template database through the privacy protection recognition module G for face recognition.

[0016] The end-to-end training loss function of E1+G1 described in the third step In the backpropagation of the loss function, only modify the parameters in G1, and E1 only adjusts the parameters of the BN layer (Batch Normal, batch normalization, a common structure in the field of deep neural network learning) through self-learning. That is, the learning purpose of G1 is to recognize the face image after E1, and E1 mainly obtains the mean and variance data statistically calculated during the training stage in the BatchNormal layer. The remaining neuron parameters of the E1 network are all initial value parameters, which can be regarded as random numbers and are not modified.

[0017] When the face image only passes through E1, although the face biometric information is protected, in order to enhance the blurriness of the face image and achieve a better face privacy protection effect. The intermediate-layer face is extracted from E1. Although the size of the picture is enlarged, the protection effect of face privacy is better. The face image blurriness F = 1 - histogram(x, x′), where histogram(x, x’) refers to the histogram algorithm, which is a general algorithm in the field of image processing. The process of extracting the intermediate-layer face is to set the face image blurriness F of the extracted intermediate-layer face ≥ ω2, and ω2 is artificially set by the user according to the actual situation. In this embodiment, the value is 85%; if the blurriness of the extracted face image does not meet the requirements, then extract the face image output by the previous layer before the intermediate layer, and check whether the blurriness meets the requirements. If it still does not meet the requirements, then extract the face image of the previous layer one more time and check whether it meets the blurriness requirements. If it still does not meet the requirements, then reset the initial value of the network and repeat the third step.

[0018] Verify the effect of E. After the face image is processed by E, the human vision cannot normally recognize its biometric information, and the face image blurriness F ≥ ω2.

[0019] Set the recognition accuracy ≥ ω1 when performing face recognition at G1, and ω1 is artificially set by the user according to the actual situation; verify the effect of G. The privacy-protected face image after passing through E is recognized on G, and the recognition accuracy ≥ ω1.

[0020] The present invention recognizes the face after privacy protection. While ensuring the recognition accuracy, compared with the existing face privacy protection and face recognition technologies, it has the following advantages:

[0021] 1. It combines face privacy protection and face recognition technologies better. The traditional face recognition privacy protection method is to save the encrypted or weakly scrambled face image in the face database, but during recognition, it still needs to be decrypted for recognition, that is, the face privacy protection process and the face recognition process are carried out separately. The present invention jointly trains two deep neural networks, namely the privacy protection network and the face recognition network, and adjusts the module composition of the two networks after training, which not only makes the face privacy protection effect better, but also enhances the recognition rate of the face after privacy protection.

[0022] 2. The face image is processed by the private face generation module and then transmitted to the server for recognition. The face images in the face database of the server have all been processed for privacy protection, with good blurriness and cannot be normally recognized by human eyes. During the recognition process, there is no need for a decryption process, but directly compare and recognize the privacy-protected face image in the face database with the incoming face image, which better achieves the task of face privacy protection.

[0023] 3. The face image after face privacy protection cannot be restored inversely, effectively protecting the security of the face database on the server side. Even if it is stolen by a third party, it is not easy to cause privacy damage such as the leakage of the original face image. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 FIG. is a comparison diagram of a traditional face recognition system and a face recognition privacy protection system based on forced scrambling;

[0025] Figure 2 FIG. is the network architecture diagram of the deep neural network E1 of the present invention;

[0026] FIG. 3(a) is the network architecture diagram of the privacy protection face generation module E of the present invention;

[0027] FIG. 3(b) is the network architecture diagram of the privacy protection face recognition module G of the present invention;

[0028] Figure 4 FIG. is the face recognition privacy protection flow chart;

[0029] FIG. 5(a) is Figure 4 the privacy protection face training flow chart in;

[0030] FIG. 5(b) is Figure 4 the privacy protection face recognition flow chart with front-end and back-end separation in;

[0031] FIG. 6(a) is the original face image in the embodiment;

[0032] FIG. 6(b) is the face image after the original face is operated by E1;

[0033] FIG. 6(c) is the face image after the original face is operated by E. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0034] To better understand the present invention, the following will combine the drawings to make some detailed embodiments and specific operation processes of the present invention:

[0035] As Figure 1 shown, a face recognition privacy protection method capable of realizing forced scrambling includes two deep neural networks E1 and G1 in the training stage. After training, it includes a private face generation module E and a private face recognition module G. The network architecture diagram of E1 is as Figure 2 shown, and the network architecture diagrams of E and G are as Figure 3(a) and 3(b) shown.

[0036] As Figure 4 shown, this embodiment includes face privacy protection recognition training and privacy protection face recognition with front-end and back-end separation;

[0037] Face privacy protection recognition training specifically includes the following steps:

[0038] As shown in Figure 5(a), E1 uses the UNet network of the deep neural network, changes the output of UNet to a color image of the same size as the original image, and adds a convolutional layer conv1 before the last layer conv of UNet. UNetc is the deep neural network of UNet without the last convolutional layer. That is: Unet(x) = conv(Unetc(x)), x1 = E1(x) = conv(conv1(Unetc(x)));

[0039] As Figure 6(a) and 6(b) shown, for the face image x, the face image obtained after training by E1 is x1.

[0040] G1 uses the Resnet network with the residual network structure of the deep neural network to perform face recognition on normal faces (unscrambled faces) after pre-training. Specifically, it includes the following steps:

[0041] S1. Preprocessing of face images. In this embodiment, for the publicly available face dataset, such as Labled Faces in the Wild (LFW), face cropping work is carried out, and a general face alignment method in the field is used, such as using the MTCNN neural network. MTCNN is a general face alignment neural network in the field.

[0042] S2. The preprocessed face images are sent into the face recognition module G for pre-training.

[0043] S3. The face dataset is sent into the large network of E1 + G1 for end-to-end training. The initial parameters of the E1 network are random parameters.

[0044] S4. After end-to-end training, E1 generates a face image that cannot be recognized by human eyes, as shown in Figure 6(b).

[0045] S5. G1 can complete the face recognition task of Figure 6(b).

[0046] Enhance the image blur of the face image, appropriately enlarge the size of the face image, separate the last part of the convolutional layer of E1 and incorporate it into G1 to form two new modules, the privacy protection face generation module E and the privacy protection face recognition module G. The size of the privacy protection face image is enlarged 3 times horizontally and vertically. The result is shown in Figure 6(c), and it can be clearly seen that the scrambling effect is better than that of Figure 6(b).

[0047] E is a new deep neural network obtained by adding a new convolutional layer conv1 after Unet finishes the last layer of convolution, aiming to generate a face image x2 with better protection of face privacy effects. x2 is shown in Fig. 6(c). The blurriness of the face image with privacy protection by E, F(x2, x) ≥ ω2, where ω2 = 85% in this embodiment. E = conv1(Unetc), x1 = E1(x) = conv(E(x)), x2 = E(x);

[0048] G is a new deep neural network obtained by adding the last convolutional layer conv of Unet to Resnet. The G module can accurately recognize the face in Fig. 6(c), and the recognition accuracy rate ≥ ω1, where ω1 = 98% in this embodiment. G = conv(G1), G1(x1) = G1(E1(x)) = G1(conv(E(x))) = G(E(x)) = G(x2).

[0049] Separate the two modules. E is the core content of the front end, only responsible for the function of protecting face privacy for face images. G is the core content of the back end, receiving the face images with privacy protection transmitted from the front end and pairing them with the face images with privacy protection already existing in the face database for face recognition.

[0050] As shown in Fig. 5(b), the front-end system includes a face image preprocessing module and a privacy-protected face generation module E. The core of the front-end system is the privacy-protected face generation module E. The face image preprocessing module is used for preprocessing work such as cropping and positioning of the face, and E is used to generate privacy-protected faces.

[0051] The back-end system includes a face database and a privacy-protected face recognition G. The core of the back-end system is the privacy-protected face recognition module G. The faces in the face database are all privacy-protected faces after passing through E. G is used to recognize the faces after privacy protection.

[0052] The specific steps of privacy-protected face recognition with front-end and back-end separation are as follows:

[0053] 1. Obtain the face image to be recognized through the data acquisition camera.

[0054] 2. Send the face image to the front end.

[0055] 3. The front-end system first preprocesses the face image through the face image preprocessing module, and then sends the preprocessed face to the privacy-protected face generation module E to generate a privacy-protected face.

[0056] 4. Transmit the privacy-protected face to the back-end system.

[0057] 5. The backend sends the incoming privacy-protected face into the privacy-preserving face recognition module G and compares it with the existing privacy-protected face database.

[0058] 6. After obtaining the result, if the face image in the face database is successfully matched, the backend returns success and the name of the successfully matched face to the frontend. If the match fails, it returns failure.

[0059] 7. Finally, a backup process is performed on the matched face, and the matching time and name are stored in the backend database as the title of the picture.

[0060] The beneficial effects of the present invention are as follows: Starting directly from the face image database, the image is directly modified to make its biometric features unable to be normally recognized by human vision, and then face recognition is performed. This effectively prevents third parties from misusing face data and other behaviors that damage normal interests after obtaining the database. At the same time, by using relevant technologies of deep neural networks rather than encryption and decryption technologies, attackers are unable to restore the original appearance of the face image from the existing images in the face database, effectively realizing the relevant content of face privacy protection. Finally, on the basis of face privacy protection, the relevant content of face recognition is also completed. Through experimental verification, it has a relatively high recognition accuracy.

[0061] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments. The above embodiments and the descriptions in the specification only illustrate the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention will have various changes and improvements, and these changes and improvements all fall within the scope of the present invention claimed. The scope of protection claimed by the present invention is defined by the appended claims and their equivalents.

Claims

1. A face recognition privacy protection method capable of realizing forced scrambling, characterized in that: It includes a privacy - protected face generation module E and a private face recognition module G. Both E and G are deep neural networks; Modules E and G are two networks separated after being jointly trained end - to - end by two deep neural networks E1 and G1. E1 is a new network obtained by adding convolutional layers and modifying the output layer to the deep neural network UNet, and G1 is a Resnet residual network; Module E is used to directly modify face data to generate face images with high blur that cannot be recognized by human vision; Module G is used to recognize face images processed by face privacy protection; Specifically, it includes the following steps: First step, select face image data from the face image database and perform face localization pre - processing operations on the face image data; Second step, pre - train the face image data processed in the first step in G1; Third step, send the face database into the large network of E1 + G1 for end - to - end training. After the training is completed, separate the entire large network. The specific operation is: cut off the last part of the convolutional layer of E1 and keep the other layers of the deep neural network as the privacy - protected face generation module E, and add the last convolutional layer of E1 to G1 as the privacy - protected face recognition module G; The end - to - end training loss function of E1+G1; in the backpropagation of the loss function, only modify the parameters in G1. The initial parameters of E1 are random numbers, and E1 only adjusts the parameters of the batch normalization layer through self - learning. Extract the intermediate - layer face from E1. If the extracted face does not meet the image blur F≥ω2, where ω2 is a set threshold, then extract the face from a more previous layer until the requirement is met. The face image blur is obtained based on the histogram algorithm; Fourth step, send the face image into the privacy - protected face generation module E obtained in the third step, output the privacy - protected face image and store it in the server as the privacy - protected face template database. During the recognition process, the privacy - protected face image is compared with the privacy - protected face template database through the privacy - protected recognition module G for face recognition.

Images