Non-reversible Facial Information Encryption and Face Recognition Method Based on Generative Adversarial Network

Abstract

The present invention discloses an irreversible face information encryption and face recognition method based on a generative adversarial network. This method extracts features from face information based on face attribute editing, uses irreversible encryption of face attribute mapping to obtain encrypted data values, and edits a corresponding new face information. By comparing the encrypted face information, the recognition result is obtained. The present invention can be used in application scenario platforms that require face recognition, such as security inspection, payment, unlocking, ticket purchasing, comparison, identity verification, etc. It can make the face data stored in each application platform become irreversibly encrypted face data, protecting the privacy of personal facial information. At the same time, by applying different encryption parameters in different application platforms, the independence and security of face storage information in different application platforms can be achieved, reflecting the wide applicability of the present invention. The implementation of the present invention will help to strengthen the protection of face privacy information.

Description

Technical Field

[0001] The present invention belongs to the fields of artificial intelligence and computer vision, and particularly relates to an irreversible face information encryption and face recognition method based on a generative adversarial network. Background Art

[0002] As an important application of the development of artificial intelligence technology, computer vision technology has become common in our daily lives. In industries such as finance, mobile, and security, face recognition, as one of the mainstream technologies, is widely used in many scenarios such as account identity authentication, mobile phone face unlock, automatic pedestrian flow statistics, and specific person identification. Compared with technologies such as iris recognition, fingerprint scanning, and palmprint scanning, face recognition has unique advantages in application: it is convenient to use and has a high user acceptance rate. The face recognition technology uses a general camera as the recognition information acquisition device and completes the recognition process in a non-contact manner without the awareness of the recognition object. Compared with other biometric technologies, the recognition accuracy of the face recognition technology is at a relatively high level, with a low false recognition rate and rejection rate.

[0003] However, while the face recognition technology creates a more convenient and secure environment for life, considering the special sensitivity of the human face, all sectors of society are increasingly concerned about the potential technical defects, discrimination, and unpredictability of the face recognition technology, and the threats and challenges brought to the privacy and equal protection of natural persons.

[0004] The current face recognition technology directly stores the scanned face information in the system for recognition, and the face information is directly stored in the database. Many collections of face information have not obtained the consent of the person being collected, and some even involve crimes. A large amount of multi-dimensional data can constitute a three-dimensional identity information database. Since the current security technical standards and usage specifications for face recognition technology are not perfect, there are no relevant regulations on the responsibilities and obligations of face data controllers, the rights of face data subjects, and the security measures that should be taken in each link of face data collection, storage, and processing. Therefore, relying solely on the commitment of the manager for the security of the use of face information is unreliable. If mismanagement leads to large-scale leakage, it may infringe on personal privacy rights. In addition, the network security ecological environment continues to deteriorate, and the black industrial chain of buying and selling personal information such as face data continues to be prohibited. System security vulnerabilities are almost inevitable. Especially in some important industries and fields related to the national economy and people's livelihood, many have not established a data security management system for important data, and the enterprise's data security risk monitoring, early warning, and incident handling capabilities are still relatively weak. Therefore, face database leakage incidents are not uncommon. Most of the security measures taken by face recognition technology development enterprises and application service providers may also be difficult to cope with the security threats faced by face recognition technology, and security incidents such as face data leakage are likely to occur.

[0005] The specific existing problems of face recognition are as follows:

[0006] From the perspective of personal information security, both fingerprints and human faces can directly establish a connection with an individual's identity. Similar to fingerprints and the irises of human eyes, human faces are also unique. A person's facial structure can be photographed and then digitally edited, and it can definitely be used by others to establish a three-dimensional identity information database. At the same time, through face recognition technology, it can further track an individual's identity information, daily whereabouts, the matching of people and vehicles, the matching of kinship, and the matching of frequently contacted people, etc. At present, the legislative comprehensiveness of face recognition is still insufficient. Due to the lack of corresponding theoretical reserves, China has not yet issued a separate law to regulate it, and it is only sporadically involved in other department laws, with extremely limited regulatory intensity. In the absence of a regulatory mechanism, the face recognition industry is difficult to be effectively regulated.

[0007] From the perspective of the data collection link, face recognition is unconscious and non-contact, can function at a distance, and can accumulate data on a large scale for a long time without being noticed by users, with strong invasiveness. From the current situation, many data collection links of face recognition are also suspected of serious illegal or even criminal acts. Because the acquisition of personal facial data in many occasions has not obtained the consent of the person being collected at all, it is difficult to be considered legally obtained.

[0008] From the perspective of the data usage link, since there are no restrictions, with the rampant expansion of the application scenarios of face recognition technology, the phenomena of abuse and discrimination will be inevitable. The development of the face recognition technology industry in China at the present stage lacks a unified access threshold. R & D personnel only need hardware facilities, supplemented by algorithms and databases, and they can own and apply this technology. As a result, the scope of the technology owners expands, greatly increasing the risk of abuse of face recognition technology.

[0009] How to protect user privacy, prevent the leakage of face information, and achieve the anonymization of face data is an urgent problem to be solved. Summary of the Invention

[0010] The purpose of the present invention is to provide an irreversible face information encryption and face recognition method based on a generative adversarial network in view of the deficiencies such as the leakage of face data and other security threats in the prior art. The present invention has high accuracy and security.

[0011] The object of the present invention is achieved by the following technical solutions: A non-reversible face information encryption and face recognition method based on a generative adversarial network, including non-reversible face attribute encryption and face attribute editing. Using a convolutional neural network to recognize face attributes, non-reversibly encrypting the face attribute features, mapping and encrypting the face attributes to generate encrypted face attributes, and adding an attention transformation unit to the cross-layer connection between the encoder and decoder of the adversarial generative network to perform face attribute editing and generate encrypted face information for face recognition.

[0012] Further, the non-reversible face attribute encryption includes: encrypting the face attribute features using a non-reversible encryption face attribute mapping. By performing face attribute recognition on a face image, a feature vector of length n is generated, and each element in the vector corresponds to a feature of the face; and this face feature vector as a whole is non-reversibly encrypted to output an N-bit key; intercepting a key vector of length n starting from a fixed starting point, performing a hash calculation on each element of the key vector to complete the attribute mapping, and respectively generating new face attribute vectors of length n.

[0013] Further, the face attribute editing includes: adding a selective transformation unit to the cross-layer connection between the encoder and decoder. Using a face attribute editing algorithm, first continuously downsample through the encoder to obtain features, and then upsample through the decoder to obtain an image. At the same time, add a cross-layer connection between the encoder and decoder, and add an attention mechanism selective transformation unit between the cross-layer connections to adaptively select and modify the encoder features.

[0014] Further, the face recognition includes: constructing a face recognition test module and dividing face recognition into two steps of detection and recognition. First, use a network model similar to the discriminator of the generative adversarial network for face detection, and then input all detected faces into the face detection network model to compare with the faces in the database to identify the identity.

[0015] Further, the face attribute editing is mainly divided into two parts: a generator and a discriminator. The generator consists of an encoder G enc and a decoder G dec . G enc is composed of 5 convolutional layers and is used to extract abstract image features. G dec is composed of 5 deconvolutional layers and is used to generate the target image. The discriminator has two branches D att and D adv . D adv is composed of 5 convolutional layers and two fully connected layers and is used to judge the authenticity of the image. D att and D adv share convolutional parameters, and use two other fully connected layers to predict the attributes of the generated image. In addition, Genc The first 4 layers are connected to G through the selective conversion unit dec with the layers corresponding to the respective depths, adaptively converting the encoder features guided by the attributes to be changed, and connecting them with the decoder features to enhance the image quality and the attribute manipulation ability.

[0016] Furthermore, the input of the generator G(x, att diff ) is the image x and the differential attribute att diff . For a given input image x, the encoder features are obtained through f = G enc (x); where respectively represent the features of each layer in the encoding stage, with a total of 5 layers. Then, under the guidance of att diff , the selective conversion unit is applied to convert the encoder features of each layer. The selective conversion units deployed in different layers do not share parameters. Let then the editing result of G dec is obtained from ; where respectively represent the features of each layer in the decoding stage, with a total of 4 layers.

[0017] Furthermore, when the target features are exactly the same as the source features, i.e., att diff = 0, at this time, the reconstruction loss L rec is defined as:

[0018] L rec = ‖x - G(x, 0)‖ 1

[0019] When the target features are different from the source features, i.e., att diff ≠0, at this time, an adversarial loss is introduced to constrain the difference between the editing result and the real image. The adversarial generation network model follows WGAN - GP, and the training losses of the real - fake discriminator D adv and its corresponding generator G adv are defined as the adversarial loss function in WGAN - GP, denoted as and

[0020]

[0021]

[0022] where is the upsampling between the paired real image and the generated image, represents the gradient of the discriminator D adv (x) in the x direction; represents the generated result of the attribute editing; || || 2Denote as L 2 Regularization; λ is the trade-off parameter of the algorithm model; E represents expectation.

[0023] In the case where the source image does not exist, introduce the feature control loss. Use the feature control loss to train the attribute classifier D att and the attribute generator G att , D att and G att The attribute classification loss function of D and G uses binary cross-entropy loss, denoted as and

[0024]

[0025]

[0026] where att (i) represents the i-th source attribute label, represents the i-th discriminator, and c represents the number of discriminators; is expressed as the i-th eigenvalue of att(D att (x)).

[0027] The objective function for training the discriminator D of the adversarial generation network model is denoted as L D :

[0028]

[0029] where λ 1 is the trade-off parameter of the algorithm model.

[0030] The objective function of the generator G is:

[0031]

[0032] where λ 2 and λ 3 are the trade-off parameters of the algorithm model, and L rec represents the loss function that constrains the distance between x and .

[0033] The beneficial effects of the present invention are as follows: The present invention is mainly applied to face recognition systems in various industries, such as account identity authentication, mobile phone face unlocking, security inspection and identity recognition in various places, face payment for shopping, automatic population statistics, and identification of specific persons. While ensuring the security of sensitive data and achieving the goals of legality and compliance, it also protects data availability and the accuracy of face recognition as much as possible. Due to the existence of the encryption algorithm, the face data information is desensitized, preventing the abuse of privacy data within the organization and also preventing the outflow of privacy data from the organization without desensitization, thus protecting the privacy data while maintaining compliance and availability. Different platforms can adopt different encryption algorithms, and the face information of different platforms cannot be shared, which also ensures the security of face information. If the present invention is applied to each face system recognition module, it will greatly protect user security and prevent the abuse of face information. It is of great significance for improving the level of data security protection in our country and will be able to improve data security problems from the source. Moreover, under the current development of artificial intelligence technology represented by face recognition technology, through the promotion of the present invention, it helps all stakeholders jointly maintain data security, drives the improvement of the overall level of personal biometric information protection, standardizes ethical responsibilities, enhances transparency, and establishes a privacy protection system, which is conducive to the healthy and sustainable development of the face recognition industry. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a flowchart of the non-reversible face information encryption and face recognition method based on the generative adversarial network of the present invention;

[0035] Figure 2 It is the basic structure diagram of the adversarial generative network described in the present invention;

[0036] Figure 3 It is a flowchart of model application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0037] A non-reversible face information encryption and face recognition method based on the generative adversarial network of the present invention first encrypts a real face image through non-reversible face attribute mapping to generate a re-encoded false face image, and then stores it in the database; during face identity recognition, similarly, the real face is first converted into a false face and then face recognition is performed. Among them, the present invention adopts a face information encryption technology that combines a non-reversible encryption algorithm with face attribute editing, introduces data encryption technology into face recognition, encrypts the real face through a non-reversible encrypted face attribute mapping module, and then generates an encrypted face through a face attribute editing module. The desensitized face information is stored in the database of the face recognition system, and the encrypted face is used for matching. Since the asymmetric encryption technology algorithm is irreversible, it is impossible to restore it to the real face information, thus protecting the privacy of users.

[0038] (1) Non-reversible Encryption Facial Attribute Mapping Module

[0039] As Figure 1 shown, the present invention uses non-reversible encryption facial attribute mapping to encrypt the facial attribute features of each face to protect the privacy of users.

[0040] 1. Non-reversible Encryption Facial Attribute Mapping Function

[0041] (1) Input information of any length, and after processing, output information of N bits;

[0042] (2) Different inputs result in different outputs (uniqueness);

[0043] (3) It is impossible to reverse-infer the input information based on the N-bit output result (irreversibility).

[0044] 2. Algorithm Process

[0045] The overall process is as follows:

[0046] The facial image undergoes downsampling for facial attribute recognition to generate a facial attribute feature vector of length n, and each element of the vector is a feature of the face;

[0047] The facial feature vector as a whole undergoes non-reversible encryption to output an N-bit password;

[0048] Intercept a segment of length n starting from a fixed starting point of the password, and use a hash table to perform attribute mapping on each element respectively to generate new attributes. All the new attributes are connected to generate a new feature vector of length n, thus completing the entire process of non-reversible encryption facial attribute mapping.

[0049] Specifically, the non-reversible encryption algorithm can calculate messages with a length not exceeding 2^64 bits. The input is processed in units of 512-bit data blocks, and a 160-bit message digest is generated as the output. The processing flow of the non-reversible encryption algorithm is roughly divided into 5 steps:

[0050] (1) Append padding bits. Pad the input data so that the remainder of the data bit length divided by 512 is 448. The highest bit of the padding bit string is filled with a 1, and the remaining bits are filled with 0s. Padding is always performed, even if the length of the message meets the required length.

[0051] (2) Append the length value. Append 64 bits after the message to represent the original length of the message, making the message length a multiple of 512 bits.

[0052] (3) Initialize the MD cache. A 160-bit MD buffer is used to save the results of intermediate and final hash functions.

[0053] It can be represented as five 32-bit registers (A, B, C, D, E). Initialized as:

[0054] A = 67452301 B = EFCDAB89 C = 98BADCFE

[0055] D = 10325476 E = C3D2E1F0

[0056] (4) Process the message in 512-bit (16-word) blocks. The core of this step is a module called the compression function, which consists of four rounds, each round containing 20 processing steps. The four rounds have a similar structure, but each round uses a different basic logic function, called f1, f2, f3, f4.

[0057] (5) After all L 512-bit data blocks are processed, output a 160-bit message digest.

[0058] (II) Face Attribute Editing Module

[0059] The generator in the image-to-image conversion task generally adopts an encoder-decoder structure, that is, first continuously downsample through the encoder to obtain features, and then upsample through the decoder to obtain the image. This structure is likely to result in low-quality generated images. The usual solution is to add cross-layer connections between the encoder and the decoder, which can enhance the image quality but reduce the image conversion ability.

[0060] As Figure 2 shown, the present invention adaptively selects and modifies the encoder features by adding an attention mechanism-based selective conversion unit during cross-layer connection to enhance the image conversion ability. At the same time, considering that one image-to-image conversion is only related to the domain to be converted, instead of directly inputting the target domain label, the difference between the target domain label and the source domain label is used as the input of the model. Figure 2 The difference attribute in t represents the difference vector of the domain label. Taking face attribute editing as an example, different attributes are regarded as different domains, and the difference between the target attribute label att s and the source attribute label att diff is denoted as att

[0061] The face attribute editing module of the present invention is mainly divided into two parts: a generator (generator, G) and a discriminator (discriminator, D). The generator consists of an encoder G enc and a decoder G dec . G enc is composed of five convolutional layers and is used to extract abstract image features. G dec is composed of five transposed convolutional layers and is used to generate the target image. The discriminator has D att and Dadv Two branches, D adv Consists of 5 convolutional layers and two fully connected layers, used to judge the authenticity of images, D att And D adv Share convolutional parameters, and use another two fully connected layers to predict the attributes of the generated image. In addition, G enc The first 4 layers of are connected to the layers of G dec With corresponding depths, adaptively transform the encoder features guided by the attributes to be changed, and connect them with the decoder features to enhance the image quality and attribute manipulation ability.

[0062] The generator G(x, att diff ) takes the image x and the differential attribute att diff as input. For a given input image x, the encoder features can be obtained through f = G enc (x); where, represent the features of each layer in the encoding stage respectively, with a total of 5 layers. Then, under the guidance of att diff , the selective transformation unit is applied to transform the encoder features of each layer. Since the dimensions are different and the features of the inner layers are more abstract than those of the outer layers, the selective transformation units (STU) deployed in different layers do not share parameters. Let then the editing result of G dec can be obtained by ; where, represent the features of each layer in the decoding stage respectively, with a total of 4 layers.

[0063] When the target features are exactly the same as the source features, that is, att diff = 0, at this time, the editing result is required to be approximately the same as the source image. Therefore, the reconstruction loss L rec is defined as:

[0064] L rec = ‖x - G(x, 0)‖ 1

[0065] where, in order to maintain the clarity of the reconstruction result, L 1 regularization ‖.‖ 1 is adopted.

[0066] When the target features are different from the source features, that is, att diff ≠ 0, the authenticity of the source image editing result is unknown. Therefore, an adversarial loss is introduced to constrain the difference between the editing result and the real image.

[0067] The adversarial generative network model follows WGAN-GP, with the real and fake discriminator D adv and its corresponding generator Gadv The training loss is defined as the adversarial loss function in WGAN-GP, denoted as and

[0068]

[0069]

[0070] where is the upsampling between paired real images and generated images, represents the discriminator D adv (x) is the gradient in the x direction; represents the generated result of attribute editing; || || 2 represents L 2 regularization; λ is the trade-off parameter of the algorithm model; E represents expectation.

[0071] In the absence of the source image, in order to make the editing result still have the required target features, a feature control loss is introduced.

[0072] The feature control loss is used to train the attribute classifier D att and the attribute generator G att , D att and G att The attribute classification loss function of uses binary cross-entropy loss, denoted as and

[0073]

[0074]

[0075] where att (i) represents the i-th source attribute label, represents the i-th discriminator, c represents the number of discriminators; is denoted as the i-th eigenvalue of att(D att (x)).

[0076] Taking the above losses into comprehensive consideration, the objective function for training the discriminator D of the adversarial generation network model is denoted as L D :

[0077]

[0078] where λ 1 is the trade-off parameter of the algorithm model.

[0079] The objective function of the generator G is:

[0080]

[0081] Among them, λ 2 and λ 3 are the trade-off parameters of the algorithm model, and L rec represents the loss function for the distance between the constraints x and each other.

[0082] (III) Face recognition test module

[0083] The fake faces generated by the irreversible encrypted face attribute mapping module and the face attribute editing module will be stored in the database. In order to verify that the encryption technology does not affect the accuracy of face recognition while protecting privacy, the present invention constructs a face recognition test module.

[0084] Face recognition is divided into two steps: detection and recognition. First, face detection is performed in the figure and feature extraction is carried out, and then compared with the faces in the database to identify the identity. If the system can still accurately identify the identity of the subject after the real face is converted into a fake face, it is sufficient to prove the effectiveness of the method described in the present invention.

[0085] Referring to Figure 3 , an irreversible face information encryption and face recognition method based on a generative adversarial network includes the following steps:

[0086] Model training part:

[0087] Step 1: Preprocess the data. Collect real face images and make them into a standard data set format. The preprocessing of the data is to calibrate the images to obtain image labels with target categories and target positions.

[0088] Step 2: Train the data set to achieve face feature detection. Use a generative adversarial network. After the data preprocessing, obtain the labels of the images. Here, these labels and the data set are used as the input of the generative adversarial network, and the final detection model is obtained after network training.

[0089] Face registration part:

[0090] Step 1: Perform face detection. Input all the detected faces for feature extraction to generate a feature vector of length n, that is, face attributes. Each element of the vector is a feature of the face;

[0091] Step 3: Use the irreversible encrypted face attribute mapping module to irreversibly encrypt the face feature vector to generate a new face feature vector;

[0092] Step 4: Use the encrypted feature vector to perform upsampling in the generative adversarial network of the face attribute editing module to generate an encrypted fake face as the output of the network;

[0093] Step Five, store the fake face output by the face attribute editing module into the database.

[0094] Face recognition part:

[0095] The first four steps are the same as those in the face registration part, and the last step is to compare the similarity between the new face (processed by the irreversible encrypted face attribute mapping module and the face attribute editing module) and the existing face data in the database. If the similarity is greater than the set threshold, the face is considered to be successfully matched.

Claims

1. A non-reversible face information encryption and face recognition method based on a generative adversarial network, characterized in that, it includes non-reversible face attribute encryption and face attribute editing; uses a convolutional neural network to recognize face attributes, performs non-reversible encryption on each face attribute feature, maps the encrypted face attributes, generates encrypted face attributes, and adds an attention transformation unit to the cross-layer connection between the encoder and decoder of the adversarial generative network to perform face attribute editing and generate encrypted face information for face recognition; The face attribute editing mainly consists of two parts: a generator and a discriminator; the generator consists of an encoder G enc and a decoder G dec . G enc is composed of 5 convolutional layers and is used to extract abstract image features. G dec is composed of 5 transposed convolutional layers and is used to generate the target image; the discriminator has two branches, D att and D adv . D adv is composed of 5 convolutional layers and two fully connected layers and is used to judge the authenticity of the image. D att and D adv share convolutional parameters, and another two fully connected layers are used to predict the attributes of the generated image; in addition, the first 4 layers of G enc are connected to the corresponding depth layers of G dec through a selective transformation unit, adaptively transform the encoder features guided by the attributes to be changed, and connect them with the decoder features to enhance the image quality and attribute manipulation ability; The input of the generator G(x, att diff ) is the image x and the differential attribute att diff ; for the given input image x, the encoder features are obtained through f = G enc (x); where respectively represent the features of each layer in the encoding stage, with a total of 5 layers; then under the guidance of att diff , the selective transformation unit is applied to transform the encoder features of each layer; the selective transformation units deployed in different layers do not share parameters; let f t = {f t 1 ,... f t 4}, then the editing result of G dec is obtained from ; where f t 1 ,... f t 4 respectively represent the features of each layer in the decoding stage, with a total of 4 layers; When the target feature is exactly the same as the source feature, i.e., att diff = 0. At this time, the reconstruction loss L rec is defined as: L rec = ||x - G(x,0)|| 1 When the target feature is different from the source feature, i.e., att diff ≠ 0, at this time, an adversarial loss is introduced to constrain the difference between the edited result and the real image; according to WGAN-GP, the adversarial generation network model defines the training losses of the real and fake discriminators D adv and its corresponding generator G adv as the adversarial loss function in WGAN-GP, denoted as and Among them, is the upsampling between paired real images and generated images, represents the discriminator D adv (x) the gradient of (x) in the x direction; represents the generated result of attribute editing; || || 2 represents L 2 regularization; λ is the trade-off parameter of the algorithm model; E represents expectation; In the case where the source image does not exist, introduce a feature control loss; use the feature control loss to train the attribute classifier D att and the attribute generator G att , D att and G att The attribute classification loss functions of D and Among them, att (i) represents the i-th source attribute label, represents the i-th discriminator, and c represents the number of discriminators; is expressed as the i-th eigenvalue of att(D att (x)); The objective function for training the discriminator D of the adversarial generative network model is denoted as L D : Among them, λ 1 is the trade-off parameter of the algorithm model; The objective function of the generator G is: Among them, λ 2 and λ 3 are trade-off parameters of the algorithm model, and L rec represents the loss function of the distance between x and .

2. The non-reversible face information encryption and face recognition method based on a generative adversarial network according to claim 1, characterized in that, the non-reversible face attribute encryption includes: using non-reversible encrypted face attribute mapping to encrypt face attribute features; generating a feature vector of length n by performing face attribute recognition on a face image, and each element in the vector corresponds to a feature of the face; and taking this face feature vector as a whole to perform non-reversible encryption, outputting an N-bit key; intercepting a key vector of length n starting from a fixed starting point of the key, performing a hash calculation on each element of the key vector to complete attribute mapping, and respectively generating new face attribute vectors of length n.

3. The non-reversible face information encryption and face recognition method based on a generative adversarial network according to claim 1, characterized in that, the face attribute editing includes: adding a selective conversion unit to the cross-layer connection between the encoder and decoder; using a face attribute editing algorithm, first continuously downsampling through the encoder to obtain features, and then upsampling through the decoder to obtain an image; at the same time, adding a cross-layer connection between the encoder and decoder, and adding an attention mechanism selective conversion unit between the cross-layer connections to adaptively select and modify encoder features.

4. The non-reversible face information encryption and face recognition method based on a generative adversarial network according to claim 1, characterized in that, the face recognition includes: constructing a face recognition test module and dividing face recognition into two steps of detection and recognition; first using a network model similar to the discriminator of the generative adversarial network for face detection, and then inputting all detected faces into the face detection network model to compare with the faces in the database to identify the identity.

Images