Face soft biometric privacy enhancement method and apparatus based on information bottleneck theory

Abstract

The application discloses a face soft biometric privacy enhancement method and device based on information bottleneck theory, and the device comprises: a face image data preprocessing module, which is used for performing alignment, cropping, standardization and other preprocessing operations; an information complexity encoder calculation module, which is used for generating face representation data; an utility decoder calculation module, which is used for predicting identity information corresponding to face image data in a training set; a face soft biometric privacy enhancement calculation module based on information bottleneck theory, which is used for training a face information bottleneck network, deploying the trained information complexity encoder on a channel between a user and an untrusted third party, so that extraction of face representation data of the user cannot infer sensitive information, that is, soft biometric information such as gender, age or race, but the discrimination ability of a face authentication service is still ensured; and the application can be widely applied in the field of face biometric privacy protection.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of privacy protection of face recognition systems, and in particular to a face soft biometric privacy enhancement method and device based on information bottleneck theory. BACKGROUND

[0002] Face recognition applications based on deep neural networks have been widely deployed in intelligent devices to provide people with biometric application services such as face recognition or identity authentication. However, while face recognition applications make people's daily life more convenient, privacy leakage problems are particularly prominent. The main reason is that the face representation data extracted by the face recognition model not only encodes the identity information of the user, but also encodes a large amount of soft biometric attribute information into the face representation data, such as gender, age, or race information. This causes the face representation data disclosed by the user to an untrusted service provider to be inferred to be such sensitive information. After such information is used by illegal malicious entities, it can cause portrait reconstruction, targeted advertising recommendation, fraud, and other malicious behaviors against the user. Therefore, how to implement a soft biometric privacy enhancement computing method and device for face representation data is a key problem.

[0003] Currently, many researchers have proposed soft biometric privacy enhancement techniques for face representation data to inhibit attackers from inferring soft biometric attribute information from disclosed face representation data. The mainstream method mainly protects the privacy of a single specific soft biometric attribute information, such as gender attribute. However, related research shows that face representation data not only encodes a single soft biometric attribute information, which makes it impossible to prevent attackers from inferring other soft biometric attribute information, such as race, age, or health status, based on a single specific privacy protection technology. Therefore, how to comprehensively protect the privacy of soft biometric attribute information of face representation data has become a problem to be solved in the current privacy computing and face recognition field. SUMMARY

[0004] The purpose of the present application is to provide a face soft biometric privacy enhancement method and device based on information bottleneck theory.

[0005] The technical solution adopted by the present application is:

[0006] The face soft biometric privacy enhancement device based on information bottleneck theory comprises:

[0007] A face image data preprocessing module is used to sample data from a face image data set , align, crop, and image data standardization preprocessing operations are performed on the face image data x n , wherein n is the identity label u n corresponding to the face image data xn The index of the data entry (x n , u n ) is divided into a training data set and a test data set according to a data set ratio of 7:3, and the preprocessed face image data is taken as the input of the information complexity encoder calculation module;

[0008] The information complexity encoder calculation module is used for receiving the preprocessed face image data x n , and extracting adult face representation data z n The information complexity encoder calculation module P φ (Z|X) is a conditional probability density function, wherein X and Z represent random variables of face image data and face representation data respectively, and the parameter φ is a neural network parameter for constructing the information complexity encoder calculation module P φ (Z|X);

[0009] The utility decoder calculation module is used for receiving the face representation data z n , and predicting the corresponding identity label u n The utility decoder calculation module P θ (U|Z) is a conditional probability density function, wherein Z and U represent random variables of face representation data and identity label respectively, and the parameter θ is a neural network parameter for constructing the utility decoder module P θ (U|Z);

[0010] The face soft biometric privacy enhancement calculation module based on the information bottleneck theory is used for training the face information bottleneck network composed of the information complexity encoder P φ (Z|X) and the utility decoder P θ (U|Z), and taking the trained information complexity encoder as a channel connecting between a user and an untrusted third party.

[0011] Further, the information complexity encoder calculation module comprises a convolutional neural network calculation unit, a feature vector calculation unit and a face representation vector generation unit;

[0012] The convolutional neural network calculation unit is used for receiving the face image data x n , extracting a 512-dimensional feature vector through a ResNet-50 convolutional neural network, and taking the feature vector as the input of the feature vector calculation unit;

[0013] The feature vector calculation unit is used for receiving the feature vector, further processing the feature vector through a BatchNorm and a LeakyReLU activation function, and taking the corresponding output feature vector as the input of the face representation vector generation unit;

[0014] The face representation vector generation unit is configured to receive the feature vector and pass it through two fully connected layers to obtain face representation data z n the logarithm of the variance of z the standard deviation σ φ (x n ) and the mean μ φ (x n ), while being equipped with a random noise ε ~ N(0, I) that follows a standard normal distribution, so that the generation process of the face representation data z n can be calculated by z = μ φ (x n ) + ε ⊙ σ φ (x n ), where I denotes the identity matrix , and ⊙ represents the Hadamard product of two vectors, and the calculation result generates the face representation data z n that follows a normal distribution The randomness of generating the face representation data is given by a deterministic function f φ composed of a neural network and a noise ε that follows a standard normal distribution.

[0015] Further, the face representation vector generation unit is configured to receive the feature vector and pass it through two fully connected layers to obtain face representation data z n the logarithm of the variance of z the standard deviation σ φ (x n ) and the mean μ φ (x n ), specifically:

[0016] The received representation data will pass through a 512x512 fully connected layer and generate 512-dimensional face representation data z n the logarithm of the variance of z After one step of operation, the standard deviation σ n (x φ ) of the face representation data z n is obtained; in addition, the received representation data will pass through another 512x512 fully connected layer and generate the mean μ φ (x n ) of the 512-dimensional face representation data z n .

[0017] Further, the utility decoder calculation module includes a normalized hypersphere and angular margin calculation unit and a logic probabilistic calculation.

[0018] The normalized hypersphere and angular margin calculation unit is configured to receive the face representation data z n, further standardizing the face representation data z n / ||z n ||, for the built-in dimension Each row vector in the fully connected layer Also perform a standardization operation θ j / ||θ j ||, by calculating the inner product between the standardized face representation data z n / ||z n || and the row vector θ j / ||θ j ||, Each component in the generated logic represents the cosine similarity cosω j between the face representation and the identity label in the training set; where ω j represents the angle between the face representation data and the row vector θ j in the hypersphere prevalence; for the row vector index j in the fully connected layer corresponds to the true identity label u n , that is, j = u n , an additive angular margin penalty parameter m is added to the angle ω j ;

[0019] The logic probability calculation unit further utilizes the SoftMax function to probabilize the logic by multiplying each element in the logic by a feature scaling factor ρ, and the index value of the logic vector corresponding to the maximum probability value obtained by calculation is the predicted identity label u n corresponding to the face representation data z n .

[0020] Further, the face soft biometric privacy enhancement calculation module based on the information bottleneck theory comprises a face information bottleneck network training unit and a trained information complexity encoder unit; specifically:

[0021] The face information bottleneck network training unit is used to combine the complexity encoder P φ (Z|X) and the utility decoder P θ (U|Z), thereby constructing a face information bottleneck network, which establishes a functional relationship between the face representation data z n extracted from the face image data x n and the predicted identity label u n , and trains the face information bottleneck network according to the target loss function and the trade-off parameter β;

[0022] The trained information complexity encoder unit is used to deploy the complexity encoder calculation module in the trained face information bottleneck network to the channel between the user and the untrusted third-party service platform, so that the face representation data extracted from the face image data obtained from the client has identity discrimination ability and inhibits the attacker from inferring soft biometric attribute information including gender and race from the face representation data.

[0023] Further, the present application also discloses a face soft biometric privacy enhancement method based on the information bottleneck theory, which comprises the following steps:

[0024] S1, a face information bottleneck network training unit combines an information complexity encoder P φ (Z|X) and a utility decoder P θ (U|Z) to construct a face information bottleneck network, the face information bottleneck network establishes a function relationship between face representation data z n extracted from face image data x n and predicted identity label u n .

[0025] S2, the face information bottleneck network training unit trains the face information bottleneck network according to a target loss function and a trade-off parameter β,

[0026] S3, the trained information complexity encoder unit deploys the information complexity encoder calculation module in the trained face information bottleneck network to the channel between the user and the untrusted third-party service platform, so that the face representation data extracted from the face image data obtained from the client has identity discrimination ability and inhibits the attacker from inferring soft biometric attribute information including gender and race from the face representation data.

[0027] Further, S1 comprises the following steps:

[0028] S1-1, receiving preprocessed face image data x n from face image data collected from a training set; specifically comprising the following steps:

[0029] S1-1-1, extracting a feature vector through a convolutional neural network calculation unit;

[0030] S1-1-2, processing the feature vector through a feature vector calculation unit;

[0031] S1-1-3, generating face representation data z n through a face representation generation unit;

[0032] S1-2, receiving face representation data z n ; specifically comprising the following steps:

[0033] S1-2-1, normalized hypersphere and angular margin calculation unit;

[0034] S1-2-2, after the logical probabilistic computation unit, predict the training set identity information u corresponding to this representation data. n .

[0035] Furthermore, in S2, the target loss function of the face information bottleneck network is trained based on the provided tradeoff parameter β. Expressed using the following formula:

[0036]

[0037] Where φ and θ are respectively the information complexity encoder calculation modules P φ (Z|X) and utility decoder calculation module P θ The neural network parameters (U|Z) are such that β is a trade-off parameter that controls the privacy protection performance of soft biometrics and the utility performance of face recognition.

[0038] From face image datasets Mid-sample a face image data x n As input data to the information complexity encoder computation module, the face representation data z is calculated. n The corresponding calculation process is expressed as z n =f φ (x n The generated face representation data z follows a normal distribution.

[0039] Based on Q z =N(0,I) d Under the prior assumption that the distribution is standard normal, the target loss function is... D in KL (P φ (Z|x n )||Q z The item has a parsed form:

[0040]

[0041] in, and These represent the computational modules P from the information complexity encoder. φ (Z|X) is the i-th index component in the d-dimensional feature vector learned by Z;

[0042] For the utility decoder calculation module P θ (U|Z) From the information complexity encoder computation module P φ A face representation data z is sampled from (Z|X).n The probability value expressed as an identity tag is expressed as P θ (u n | f φ (x n , ε)) The specific analytical form is:

[0043]

[0044] Wherein, denotes the logical sum of all elements except the specific index j≠u n ;

[0045] Using the gradient descent strategy, the face information bottleneck network parameters are updated, and when the target function is less than the convergence termination threshold, the training is stopped. In the training iteration process on the face image training set , the absolute value of the difference between the target loss function value calculated in this round and the target loss function value calculated in the last round is less than the convergence termination threshold, then the training is stopped.

[0046] Further, S2 includes the following steps:

[0047] S2-1, select the trade-off parameter value β;

[0048] S2-2, retrieve a batch of face images and their corresponding identity tags from the face training set;

[0049] S2-3, pre-process the face image data;

[0050] S2-4, input the pre-processed face image to the face bottleneck network;

[0051] S2-5, calculate the target loss function according to the input result of the face bottleneck network and the identity tag;

[0052] S2-6, whether the current data set cycle is completed; if yes, jump to S2-7; otherwise, jump to S2-9;

[0053] S2-7, calculate the average loss function value in this round, and calculate the absolute value of the difference with the result of the last round;

[0054] S2-8, whether the absolute value result is greater than the threshold value; if yes, jump to S2-9; otherwise, jump to S2-10;

[0055] S2-9, update the face information bottleneck network parameters using the neural network back propagation algorithm, and jump to S2-2;

[0056] S2-10, stop training.​

[0057] Further, the specific steps of S3 are as follows:

[0058] S3-1, accept the user request;

[0059] S3-2, obtain the user face image data;

[0060] S3-3, pre-process the face image data;

[0061] S3-4, extract the face representation data by using the trained information complexity encoder;

[0062] S3-5, whether the user's biometric template has been registered; otherwise, jump to S3-5-1; if yes, jump to S3-5-2;

[0063] S3-5-1, whether the data needs to be stored in the database as the user's biometric template; if yes, jump to S3-6; otherwise, jump to S3-9;

[0064] S3-5-2, whether the user needs to be authenticated; if yes, jump to S3-7; otherwise, the request is completed;

[0065] S3-6, store the face representation data as the user's biometric template in the database, and jump to S3-9;

[0066] S3-7, call the user's biometric template and calculate the cosine similarity with the face representation data;

[0067] S3-8, whether the cosine similarity is greater than the threshold; if yes, the authentication is successful; otherwise, the authentication fails;

[0068] S3-9, the request is completed.

[0069] The above technical scheme is specially used for face image data, can extract face representation data with privacy enhancement ability and identity discrimination ability, prevent attackers from inferring soft biometric attribute information from face representation data, and meet the identity authentication and face recognition ability. The present application can be widely applied in the field of privacy protection of face biometric features. BRIEF DESCRIPTION OF DRAWINGS

[0070] The present application will be further described in detail below in combination with the drawings and specific embodiments;

[0071] Fig. 1 Structure diagram of the face soft biometric feature privacy enhancement computing device based on the information bottleneck theory;

[0072] Fig. 2A schematic diagram of the bottleneck network structure for face information, which combines the information complexity encoder computation module and the utility decoder computation module.

[0073] Fig. 3 A flowchart illustrating the training method for a bottleneck network for facial information.

[0074] Fig. 4 A flowchart of a service request for a trained information complexity encoder device. Implementation

[0075] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0076] like Figs. 1 to 4 As shown in one example, this invention discloses a facial soft biometric privacy enhancement device based on the information bottleneck theory, the device comprising:

[0077] The preprocessing module 110 for face image data is used to preprocess face image data from a face image dataset. Medium-sampled data, for face image data x n Perform alignment, cropping, and image data normalization preprocessing operations, where n is the face image data x n Corresponding identity tag u n Constituting data entries (x) n ,u n The dataset is indexed and the number of samples in the face image dataset is divided into training and testing datasets in a 7:3 ratio for analysis of the face image data x. n Preprocessing operations such as alignment, cropping, and image data standardization are performed, and the preprocessed face image data is used as input to the information complexity encoder calculation module; the face image dataset adopts the CelebA dataset, and the image size is cropped to 224×224;

[0078] Information complexity encoder calculation module 120 is used to receive preprocessed face image data x n And extract adult face representation data z n Information complexity encoder calculation module P φ (Z|X) is a conditional probability density function, where X and Z represent random variables of face image data and face representation data, respectively, and the parameter φ is the computational module P for constructing the information complexity encoder. φ (Z|X) neural network;

[0079] The utility decoder calculation module 130 is used to receive face representation data z. nAnd predict the corresponding identity tag u n Utility decoder calculation module P θ (U|Z) is a conditional probability density function, where Z and U represent random variables of face representation data and identity labels, respectively, and the parameter θ is the parameter used to construct the utility decoder module P. θ The neural network parameters of (U|Z);

[0080] A face soft biometric privacy enhancement computing module 140 based on the information bottleneck theory is used to train the information complexity encoder P. φ (Z|X) and utility decoder P θ A face information bottleneck network consisting of (U|Z) is used, and a trained information complexity encoder is used as a data processing channel connecting the user and an untrusted third party.

[0081] In a specific embodiment, the information complexity encoder calculation module 120 includes a convolutional neural network calculation unit 111, a feature vector calculation unit 112, and a face representation vector generation unit 113.

[0082] Convolutional neural network computing unit 111 is used to receive face image data x of size 224×224. n The feature vectors are extracted into 512 dimensions by the ResNet-50 convolutional neural network and used as the input of the feature vector calculation unit.

[0083] The feature vector calculation unit 112 is used to receive feature vectors, which are further processed by BatchNorm and LeakyReLU activation functions, and its output feature vector is used as the input of the face representation vector generation unit.

[0084] The face representation vector generation unit 113 receives the feature vector and passes it through two fully connected layers to obtain face representation data z. n logarithm of variance Standard deviation σ φ (x n ) and mean μ φ (x n The statistics of the face representation data z are given, along with a random noise ε ~ N(0,I) that follows a standard normal distribution, so that the face representation data z n The generation process can be described by z = μ φ (x n )+ε⊙σ φ (x n The calculation yields face representation data z, where I represents the identity matrix and ⊙ represents the Hadamard product of two vectors. n It will follow a normal distribution. The randomness in generating facial representation data is determined by a deterministic function f constructed by a neural network. φ And the noise ε that follows a standard normal distribution.

[0085] The face representation vector generation unit 113 receives the feature vector and obtains the face representation data z by passing it through two fully connected layers. n logarithm of variance Standard deviation σ φ (x n ) and mean μ φ (x n The statistics for ) are as follows:

[0086] The received representation data will be processed through a 512×512 fully connected layer to generate 512-dimensional face representation data z. n logarithm of variance After one step The operation yields the facial representation data z. n Standard deviation σ φ (x n Furthermore, the received representation data will pass through another 512×512 fully connected layer to generate 512-dimensional face representation data z. n mean μ φ (x n ).

[0087] In a specific embodiment, the utility decoder calculation module 120 includes a normalized hypersphere and angular margin calculation unit 121 and a logical probabilistic calculation unit 122.

[0088] Normalized hypersphere and angular margin calculation unit 121 is used to receive face representation data z n Further standardize facial representation data z n / ||z n ||, for the built-in dimensions Each row vector θ in the fully connected layer j , Standardized operations are also performed. j / ||θ j ||, by calculating standardized face representation data z n / ||z n || with row vector θ j / ||θ j ||, The inner product between them ensures that each component in the generated logic represents the cosine similarity (cosω) between the face representation and the identity labels in the training set. j , where ω j This represents the relationship between face representation data and row vector θ in hypersphere popularization.j angle between the two, for row vector index j in the fully connected layer corresponds to the real identity label u n , that is, j = u n , then an additional angle ω j Add an additive angle margin penalty parameter m; where The number of all body numbers in the CelebA dataset, the additive angle margin penalty parameter m = 0.5;

[0089] The logic probability calculation unit 122 further utilizes the SoftMax function to calculate the probability of the logic by multiplying each element in the logic by the feature scaling factor ρ, and the index value of the logic vector corresponding to the maximum probability value obtained by the calculation is the predicted face representation data z n Corresponding identity label u n ; Wherein the feature scaling factor ρ = 64.

[0090] In specific embodiments, the face soft biometric privacy enhancement calculation module 140 based on the information bottleneck theory includes a face information bottleneck network training unit 141 and a trained information complexity encoder unit 142.

[0091] The face information bottleneck network training unit 141 is used to combine the information complexity encoder P φ (Z|X) and the utility decoder P θ (U|Z), thereby constructing a face information bottleneck network that establishes a function relationship between extracting face representation data z n From face image data x n And predicting identity label u n According to the target loss function And the trade-off parameter β to train the face information bottleneck network; Wherein the range of β is [0, 1];

[0092] The trained information complexity encoder unit 142 is used to deploy the information complexity encoder calculation module in the trained face information bottleneck network to the channel between the user and the untrusted third-party service platform, so that the face representation data extracted from the face image data obtained from the client has identity discrimination ability and suppresses the attacker from inferring soft biometric attribute information, including gender and race, from the face representation data.

[0093] The device can be used as a face recognition system designer. According to privacy protection level requirements or legal regulations, the device sets trade-off parameters to train a face information bottleneck network, and discloses the trained information complexity encoder to a client or an untrusted third-party service platform. A user can use the model to extract his / her face representation data and send the data to the untrusted third-party service platform. The untrusted third-party service platform can obtain the model parameters to develop an attribute inference classifier for the face representation data. However, the face representation data generated by the model can prevent an attacker from inferring soft biometric attributes, thereby effectively solving the privacy leakage problem.

[0094] Further, the application also discloses a face soft biometric privacy enhancement method based on the information bottleneck theory, which comprises the following steps:

[0095] S1, a face information bottleneck network training unit combines an information complexity encoder P φ (Z|X) and a utility decoder P θ (U|Z) to construct a face information bottleneck network. The face information bottleneck network establishes a function relationship between face image data x n , extracts face representation data z n , and predicts identity labels u n .

[0096] S2, the face information bottleneck network training unit trains the face information bottleneck network according to a target loss function and a trade-off parameter β,

[0097] S3, the trained information complexity encoder unit deploys the information complexity encoder calculation module in the trained face information bottleneck network to a channel between a user and an untrusted third-party service platform, so that the face representation data extracted from the face image data obtained from the client has identity discrimination ability and can inhibit an attacker from inferring soft biometric attribute information, including gender and race, from the face representation data.

[0098] Further, S1 comprises the following steps:

[0099] S1-1, the face information bottleneck network training unit receives preprocessed face image data x n from face image data collected from a training set;

[0100] S1-1-1, the convolutional neural network calculation unit extracts a feature vector;

[0101] S1-1-2, the feature vector calculation unit processes the feature vector;

[0102] S1-1-3, the face representation generation unit generates face representation data z n ;

[0103] S1-2, receiving face representation data z n ;

[0104] S1-2-1, passing through a normalized hyper-sphere and an angle margin calculation unit;

[0105] S1-2-2, passing through a logic probabilistic calculation unit to predict the identity information u of the training set corresponding to the representation data n .

[0106] Further, the S2 specific method is:

[0107] According to the provided trade-off parameter β to train the face information bottleneck network target loss function Use the following formula:

[0108]

[0109] Where, φ and θ are neural network parameters of the information complexity encoder calculation module P φ (Z|X) and the utility decoder calculation module P θ (U|Z), respectively, and β is a trade-off parameter that controls the soft biometric privacy protection performance and the face recognition utility performance; A face image data x is sampled from a face image data set n , which can be used as input data of the information complexity encoder calculation module, and the face representation data z n is calculated, and the calculation process is expressed as z n =f φ (x n ,ε), and the generated face representation data z follows a normal distribution Based on the prior assumption that Q z =N(0,I d ) is a standard normal distribution, then the D KL (P φ (Z|x n )||Q z ) term in the target loss function has an analytical form:

[0110]

[0111] Where, and respectively represent the i-th index component in the d-dimensional feature vector learned from the information complexity encoder calculation module P φ (Z|X); for the utility decoder calculation module P θ (U|Z), the information complexity encoder calculation module P φ (Z|X)Sampling a face representation data z in (Z|X) n The probability value of predicting an identity label can be expressed as P θ (u n |f φ (x n ,ε)) which has a specific analytical form:

[0112]

[0113] wherein, denotes the logical sum of all elements except the specific index j≠u n , and the gradient descent strategy is used to update and optimize the parameters of the face information bottleneck network. When the target function In the training iteration process on the face image training set , the absolute value of the difference between the target loss function value calculated in this round and the target loss function value calculated in the last round is less than the convergence termination threshold, the training is stopped.

[0114] Embodiment 2 provides a face information bottleneck network structure and a target loss function for training network parameters. The network architecture can effectively and stably train network parameters. The generated face representation data can enhance the discrimination ability while suppressing the encoding of soft biometric information into the face representation data.

[0115] Embodiment 3: Referring to Fig. 3 , the face information bottleneck network training unit 141 trains the face information bottleneck network according to the target loss function and the trade-off parameter β. Specifically, S2 includes the following steps:

[0116] S2-1, select the trade-off parameter value β; in a specific implementation, β∈{0.0001, 0.001, 0.01, 0.1, 1};

[0117] S2-2, retrieve a batch of face images and their corresponding identity labels from the face training set; in a specific implementation, the batch size is 64;

[0118] S2-3, pre-process the face image data; in a specific implementation, the face image is cropped to 224×224;

[0119] S2-4, input the pre-processed face image to the face bottleneck network;

[0120] S2-5, calculate the target loss function according to the input result of the face bottleneck network and the identity label;

[0121] ​​S2-6, whether the current round of data set cycle is completed; if yes, jump to S2-7; otherwise, jump to S2-9;

[0122] S2-7, calculate the average loss function value of the current round, and calculate the absolute value result of the difference with the result of the last round;

[0123] S2-8, whether the absolute value result is greater than the threshold value; if yes, jump to S2-9; otherwise, jump to S2-10; in a specific implementation, the threshold value is set to 0.01.

[0124] S2-9, update the face information bottleneck network parameters by using the neural network back propagation algorithm, and jump to S2-2;

[0125] S2-10, stop training.

[0126] Inventive Example 3 provides a training process of the face information bottleneck network, uses a public face data set to train the network parameters, a target loss function, and a neural network back propagation algorithm to update the face information bottleneck network parameters, and gives the condition for stopping training.

[0127] Example 4: see Fig. 4 , the trained information complexity encoder unit 142 is used to deploy the information complexity encoder calculation module in the trained face information bottleneck network to the channel between the user and the untrusted third-party service platform, so that the face representation data extracted from the face image data obtained from the client has identity discrimination ability and suppresses the attacker from inferring soft biometric attribute information including gender and race from the face representation data; specifically, S3 includes the following steps:

[0128] S3-1, accept user request;

[0129] S3-2. Obtain user face image data;

[0130] S3-3, pre-process the face image data; in a specific implementation, the face image is cropped to 224x224;

[0131] S3-4, extract face representation data by using the trained information complexity encoder; in a specific implementation, the dimension of the face representation data is 512.

[0132] S3-5, whether the user's biometric template has been registered; otherwise, jump to S3-5-1; if yes, jump to S3-5-2;

[0133] S3-5-1, whether the data needs to be stored as the user's biometric template in the database; if yes, jump to S3-6; otherwise, jump to S3-9;

[0134] S3-5-2, whether the user needs to be authenticated; if yes, jump to S3-7; otherwise, the request is completed;

[0135] S3-6, the face representation data is taken as the user's biometric template and stored in the database, and jump to S3-9;

[0136] S3-7, the user's biometric template is called and the cosine similarity is calculated with the face representation data;

[0137] S3-8, whether the cosine similarity is greater than a threshold; if yes, the authentication is successful; otherwise, the authentication fails; in a specific implementation, the threshold is set to 0.9;

[0138] S3-9, the request is completed.

[0139] Inventive Example 4 provides a service process of the deployed trained information complex encoder, extracts face representation data with privacy enhancement function from user-provided face image data, and sends it to an untrusted third-party server, meeting the basic service of the biometric system.

[0140] The above technical scheme is adopted, and the face representation data with privacy enhancement ability and identity discrimination ability can be extracted, the soft biometric attribute information is prevented from being inferred from the face representation data by the attacker, the identity authentication and face recognition ability are met, and the application can be widely applied in the privacy protection field of face biometrics.

[0141] Obviously, the described embodiments are part of the embodiments of the present application, not all. The embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The components of the embodiments of the present application described and shown in the drawings can be arranged and designed in various different configurations. Therefore, the detailed description of the embodiments of the present application is not intended to limit the scope of the claimed application, but only represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

Claims

1. A facial soft biometric privacy enhancement device based on the information bottleneck theory, characterized in that: The device includes: A preprocessing module for face image data is used to process face image datasets. Medium-sampled data, for face image data x n Perform alignment, cropping, and image data normalization preprocessing operations, where n is the face image data x n Corresponding identity tag u n Constituting data entries (x) n ,u n The index of the face image dataset is used to divide the number of samples in the dataset into training and testing datasets in a 7:3 ratio. The information complexity encoder calculation module is used to receive preprocessed face image data x. n And extract adult face representation data z n Information complexity encoder calculation module P φ (Z|X) is a conditional probability density function, where X and Z represent random variables of face image data and face representation data, respectively, and the parameter φ is the computational module P for constructing the information complexity encoder. φ The neural network parameters of (Z|X); The utility decoder calculation module is used to receive face representation data z. n And predict the corresponding identity tag u n Utility decoder calculation module P θ (U|Z) is a conditional probability density function, where Z and U represent random variables of face representation data and identity labels, respectively, and the parameter θ is the parameter used to construct the utility decoder module P. θ The neural network parameters of (U|Z); A privacy-enhancing computational module for facial soft biometrics based on the information bottleneck theory is used to train an information complexity encoder P. φ (Z|X) and utility decoder P θ A face information bottleneck network consisting of (U|Z) is used, and a trained information complexity encoder is used as a channel connecting the user and an untrusted third party.

2. The face soft biometric privacy enhancement device based on information bottleneck theory according to claim 1, characterized in that: The information complexity encoder calculation module includes a convolutional neural network calculation unit, a feature vector calculation unit, and a face representation vector generation unit. A convolutional neural network computation unit is used to receive face image data x. n The feature vector is extracted into a 512-dimensional feature vector by the ResNet-50 convolutional neural network; The feature vector calculation unit is used to receive feature vectors and process them through the BatchNorm and LeakyReLU activation functions to obtain the corresponding output feature vectors. The face representation vector generation unit receives the output feature vector and obtains the face representation data z through two fully connected layers. n logarithm of variance Standard deviation σ φ (x n ) and mean μ φ (x n The statistics of the face representation data z are provided; simultaneously, a random noise ε ~ N(0,I) following a standard normal distribution is provided, making the face representation data z n The generation process is based on z = μ φ (x n )+ε⊙σ φ (x n The calculation yields the following: where I represents the identity matrix, and ⊙ represents the Hadamard product of two vectors; the calculated face representation data z is generated from the result. n It will follow a normal distribution.

3. The face soft biometric privacy enhancement device based on the information bottleneck theory according to claim 2, characterized in that: The specific implementation method of the face representation vector generation unit is as follows: The received representation data will be passed through a 512×512 fully connected layer to generate 512-dimensional face representation data z. n logarithm of variance After one step The operation yields the facial representation data z. n Standard deviation σ φ (x n Furthermore, the accepted representation data will be passed through another 512×512 fully connected layer to generate 512-dimensional face representation data z. n mean μ φ (x n ).

4. The face soft biometric privacy enhancement device based on the information bottleneck theory according to claim 1, characterized in that: The utility decoder calculation module includes a normalized hypersphere and angular margin calculation unit and a logical probabilistic calculation unit; the normalized hypersphere and angular margin calculation unit is used to receive face representation data z. n Standardized face representation data z n / ||z n ||;For the built-in dimensions Each row vector θ in the fully connected layer j , Perform standardized operations θ j / ||θ j ||;By calculating standardized facial representation data z n / ||z n || with row vector θ j / ||θ j ||, The inner product between them ensures that each component in the generated logic represents the cosine similarity (cosω) between the face representation and the identity labels in the training set. j ; where ω j This represents the relationship between face representation data and row vector θ in hypersphere popularization. j The angle between them; for the row vector index j in the fully connected layer, it corresponds to the real identity label u. n That is, j = u n Then, additionally to the angle ω j Add an additive angle margin penalty parameter m; The logic probabilistic computation unit multiplies each element in the logic by a feature scaling factor ρ, uses the SoftMax function to probabilize the logic, and calculates the logic vector index value corresponding to the maximum probability value, which is the predicted face representation data z. n Corresponding identity tag u n .

5. The face soft biometric privacy enhancement device based on the information bottleneck theory according to claim 1, characterized in that: The facial soft biometric privacy enhancement computing module based on information bottleneck theory includes a facial information bottleneck network training unit and a trained information complex encoder unit; specifically: The face information bottleneck network training unit is used to combine the information complexity encoder P. φ (Z|X) and utility decoder P θ (U|Z), thereby constructing a face information bottleneck network. The face information bottleneck network is established from face image data x n Extracting face representation data z n And predict identity tags u n The functional relationship between them, and based on the target loss function The face information bottleneck network is trained by weighing the parameter β. The trained information complexity encoder unit is used to deploy the information complexity encoder computation module in the trained face information bottleneck network to the channel between the user and the untrusted third-party service platform, so that the face representation data has the ability to identify the identity and to prevent attackers from inferring soft biometric attribute information from the face representation data. Soft biometric attribute information includes gender and race.

6. A method for enhancing the privacy of soft biometric features of faces based on the information bottleneck theory, employing the soft biometric privacy enhancement device for faces based on the information bottleneck theory as described in any one of claims 1 to 5, characterized in that: The method includes the following steps: S1, the face information bottleneck network training unit combined with the information complexity encoder P φ (Z|X) and utility decoder P θ (U|Z) Construct a face information bottleneck network. The face information bottleneck network is established from face image data x n Extracting face representation data z n And predict identity tags u n The functional relationship between them; S2, the face information bottleneck network training unit is based on the target loss function. The face information bottleneck network is trained using a trade-off parameter β. S3, the trained information complexity encoder unit deploys the information complexity encoder computation module in the trained face information bottleneck network to the channel between the user and the untrusted third-party service platform, so that the face representation data has the ability to identify the identity and inhibits attackers from inferring soft biometric attribute information from the face representation data. Soft biometric attribute information includes gender and race.

7. The method for enhancing facial soft biometric privacy based on the information bottleneck theory according to claim 6, characterized in that: S1 includes the following steps: S1-1, Receive face image data collected from the training set, and receive preprocessed face image data x. n Specifically, it includes the following steps: S1-1-1, after passing through the convolutional neural network computation unit, extracts the feature vector; S1-1-2, after passing through the feature vector calculation unit, the feature vector is processed; S1-1-3, after passing through the face representation generation unit, generates face representation data z. n ; S1-2, Receive face representation data z n Specifically, it includes the following steps: S1-2-1, normalized hypersphere and angular margin calculation unit; S1-2-2, after the logical probabilistic computation unit, predict the training set identity information u corresponding to this representation data. n .

8. The method for enhancing facial soft biometric privacy based on the information bottleneck theory according to claim 6, characterized in that: In S2, the objective loss function of the face information bottleneck network is trained based on the provided tradeoff parameter β. Expressed using the following formula: Where φ and θ are respectively the information complexity encoder calculation modules P φ (Z|X) and utility decoder calculation module P θ The neural network parameters (U|Z) are such that β is a trade-off parameter that controls the privacy protection performance of soft biometrics and the utility performance of face recognition. From face image datasets Mid-sample a face image data x n As input data to the information complexity encoder computation module, the face representation data z is calculated. n The corresponding calculation process is expressed as z n =f φ (x n The generated face representation data z follows a normal distribution. Based on Q z =N(0,I) d Under the prior assumption that the distribution is standard normal, the target loss function is... D in KL (P φ (Z|x n )||Q z The item has a parsed form: in, and These represent the computational modules P from the information complexity encoder. φ (Z|X) is the i-th index component in the d-dimensional feature vector learned by Z; For the utility decoder calculation module P θ (U|Z) From the information complexity encoder computation module P φ A face representation data z is sampled from (Z|X). n The probability value of predicting an identity label is expressed as P. θ (u n |f φ (x n ,ε)), its specific analytical form is: in, This indicates that, except for a specific index j≠u, the logic is correct. n In addition, other elements are summed through calculations; using a gradient descent strategy, the parameters of the bottleneck network for face information are optimized and updated, when the objective function... In the face image training set During the previous training iteration, the target loss function calculated in this round The value is the target loss function calculated in the previous round. Training stops when the absolute value of the difference between the values ​​is less than the convergence termination threshold.

9. The method for enhancing facial soft biometric privacy based on the information bottleneck theory according to claim 6, characterized in that: S2 includes the following steps: S2-1, Select the value of the trade-off parameter β; S2-2, retrieve a batch of face images and their corresponding identity labels from the face training set; S2-3, Preprocess the face image data; S2-4, input the preprocessed face image into the face bottleneck network; S2-5, Calculate the target loss function based on the input results of the face bottleneck network and the identity label; S2-6: Has the current round of dataset looping been completed? If yes, jump to S2-7; otherwise, jump to S2-9. S2-7, calculate the average loss function value for this round, and calculate the absolute value of the difference between the result and the result of the previous round; S2-8: Is the absolute value result greater than the threshold? If so, proceed to S2-9. Otherwise, proceed to S2-10; S2-9 uses the backpropagation algorithm of the neural network to update the parameters of the bottleneck network of face information, and then jumps to S2-2; S2-10, Stop training.

10. The method for enhancing the privacy of facial soft biometric features based on the information bottleneck theory according to claim 6, characterized in that: The specific steps for S3 are as follows: S3-1, Accept user requests; S3-2, Obtain user's facial image data; S3-3, Preprocessing of facial image data; S3-4, use the trained information complexity encoder to extract face representation data; S3-5: Has the user's biometric template been registered? If not, proceed to S3-5-1; if yes, proceed to S3-5-2. S3-5-1, Should this data be stored in the database as a user's biometric template? If yes, proceed to S3-6; otherwise, proceed to S3-9. S3-5-2, Do you need to authenticate the user? If yes, proceed to S3-7; otherwise, the request is complete. S3-6, use the facial representation data as the user's biometric template and store it in the database, then jump to S3-9; S3-7: Retrieve the user's biometric template and calculate the cosine similarity with the facial representation data; S3-8, Is the cosine similarity greater than the threshold? If yes, authentication is successful; otherwise, authentication fails. S3-9, Request complete.

Images