A method and apparatus for identity privacy protection of a face image

Abstract

The application discloses a face image identity privacy protection method and device, the method comprises the following steps: constructing a feature learning model, extracting features of an acquired face image by the feature learning model, and decoupling statistical information and background information in the face image by a face identity feature decoupling subnetwork model; integrating the features extracted by the feature learning model and the statistical information to obtain integrated information; constructing an identity disturbance model, adding random disturbance to the integrated information based on a privacy level of a face image user; and constructing an image reconstruction model, reconstructing the face image based on the integrated information after adding random disturbance. The method of the application can effectively resist background knowledge attacks, reconstruction attacks and combination attacks, realize provable and controllable face identity privacy protection, and ensure the optimization trade-off between the privacy and usability of the published image.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data security, in particular to a face image identity privacy protection method and device. BACKGROUND

[0002] Early face image privacy protection technologies mainly use methods such as masking, pixelization, and filtering to protect the privacy of the main facial regions such as the eyes and mouth. However, these methods have a simple implementation mechanism, and since all facial information details are covered, the utility of the protected data is low, limiting their application range. Inspired by the classic anonymization mechanism k-anonymity, researchers proposed the k-same anonymization face image privacy protection mechanism, which uses distance measurement to select the most similar image from the data set to the original face and replaces the original face with the average pixel value of the most similar image, providing formal anonymity for face image privacy protection and preserving some visual information of the original face. However, the k-same mechanism, which replaces the original image with the average pixel value, is susceptible to inaccurate alignment of facial features, leading to misplacement and superposition of features, resulting in poor visual effects of the final published image. Subsequent research work combines the k-same mechanism with a generative learning model to address the issues with the k-same mechanism, encodes the image using the generative learning model, and then uses the k-same mechanism to protect identity-related attributes, adjusting the trade-off between privacy protection level and facial attribute preservation to improve the accuracy of the published image.

[0003] In the research and analysis of existing technologies, the following problems were found in the existing technologies:

[0004] 1) Lack of quantifiable and provable privacy guarantees

[0005] Since the k-same mechanism is similar to the classic anonymization mechanism k-anonymity, both mechanisms provide privacy protection for sensitive attributes based on the attacker's background knowledge assumption by designing a privacy parameter k. Therefore, the face image privacy protection method based on k-same also has the problem of being unable to resist background knowledge attacks, reconstruction attacks, and combination attacks, and lacks quantifiable and provable privacy guarantees.

[0006] 2) Unable to achieve controllable and fine-grained attribute protection

[0007] Facial features such as contours, poses, and expressions are important for improving the utility of privacy-protected face images, but most existing methods cannot preserve various facial poses, expressions, and other geometric attributes. SUMMARY

[0008] Therefore, the present application provides a face image identity privacy protection method and device, which can solve the technical problems of the prior art that there is a lack of quantifiable and provable privacy guarantee for face images and that controllable and fine-grained attribute protection cannot be achieved.

[0009] To solve the above technical problems, the present application is implemented as follows.

[0010] A face image identity privacy protection method, comprising the following steps:

[0011] Step S1: constructing a feature learning model, performing feature extraction on an acquired face image by the feature learning model, and decoupling feature information and background information in the face image by a face identity feature decoupling sub-network model; integrating the features extracted by the feature learning model and the feature information to obtain integrated information;

[0012] Step S2: constructing an identity perturbation model, adding random perturbation to the integrated information based on the privacy level of a face image user;

[0013] Step S3: constructing an image reconstruction model, reconstructing a face image based on the integrated information after adding random perturbation.

[0014] Preferably, the feature learning model comprises a sub-network based on face visual attributes and a decoupling sub-network; the sub-network based on face visual attributes is composed of a lightweight CNN model, a channel attention model and a graph convolution model connected with each other; the acquired face image is input into the lightweight CNN model, the lightweight CNN model acquires multi-scale face features of different feature channels; the channel attention model fuses the multi-scale face features of different feature channels extracted by the lightweight CNN model to obtain a spatial semantic attribute face feature map; the spatial semantic attribute face feature map is input into a GCN network, and the GCN network learns the spatial semantic attribute face feature map to obtain geometric features of structural relationships between local regions of a face.

[0015] Preferably, the feature information refers to features of gender and age attributes; the face identity feature decoupling sub-network model is composed of a U-Net model and a CNN-based decoupling feature model; attribute operations of feature information are performed on the multi-scale face features of different feature channels to obtain feature information I' n , the U-Net model takes the feature information I' n as input to extract a face feature map; the CNN-based decoupling feature model receives the face feature map and decouples face visual attribute features and feature information in the face feature map.

[0016] Preferably, the geometric features of the structural relationship between the local regions of the face, the decoupled face visual attribute features and the decoupled feature information are obtained, and the geometric features of the structural relationship between the local regions of the face, the decoupled face visual attribute features and the decoupled feature information are fused through a linear weighting layer to obtain integrated information.

[0017] Preferably, in the step S1, the feature extraction of the obtained face image by the representation learning model comprises: the input of the representation learning model is a single image, represented as I n is input into the lightweight CNN network for feature learning to obtain multi-scale face features of different feature channels, represented as wherein i=1, 2; n=1, 2, …, N represents the number of layers of the feature channels; the multi-scale visual feature fusion operation and the attribute operation of the feature information are respectively performed on the multi-scale face features of the different feature channels to respectively obtain multi-scale visual fusion features and feature information I′ n ; the is input into a channel attention model, and the is fused to obtain a spatial semantic attribute face feature map containing spatial semantic attributes, wherein the spatial semantic attributes refer to semantic features containing the potential relationship of different spatial sub-regions of the face image; the spatial semantic attribute face feature map is input into the GCN network, and the spatial semantic attribute face feature map is extracted to represent the sub-feature blocks of the left eye right eye nose mouth , and each sub-feature block is defined as a node in a graph network G={V, E}, wherein G is a graph network, V is each graph node, E represents the boundary between nodes, and the connection relationship between nodes is represented as a boundary e∈E, wherein the boundary relationship between all graph nodes is represented as an adjacency matrix M, and M represents the relationship between the features of each node, wherein the graph G learns the mutual relationship between nodes by using GCNs to obtain the geometric features of the structural relationship between the local regions of the face.

[0018] In the step S1, the feature information and the background information in the face image are decoupled by the face identity representation decoupling sub-network model, comprising:

[0019] I′ nThe input U-Net model outputs an intermediate feature vector through an encoder model of the U-net model, and then inputs the intermediate feature vector into a decoder model of the U-Net to obtain a face feature map; the face feature map is input into a CNN-based decoupling representation model to decouple the face visual attribute feature and the feature information, and the decoupled face visual attribute feature is denoted as h back , and the decoupled feature information is denoted as h d1 .

[0020] Preferably, the step S3 comprises that the image reconstruction model is a DenseNet model, the DenseNet network reconstructs the integrated information based on the feature information after adding the random disturbance, and the DenseNet network further reconstructs the background information of the face image, so as to fuse the reconstructed integrated information and the reconstructed background information of the face image.

[0021] The device for identity privacy protection of a face image provided by the application comprises:

[0022] The feature extraction module is configured to construct a representation learning model, extract features of an acquired face image by the representation learning model, and decouple feature information and background information in the face image by a face identity representation decoupling sub-network model.

[0023] The disturbance module is configured to construct an identity disturbance model, and add random disturbance to the integrated information based on a privacy level of a face image user.

[0024] The reconstruction module is configured to construct an image reconstruction model, and reconstruct a face image based on the integrated information after adding random disturbance.

[0025] The computer readable storage medium provided by the application stores a plurality of instructions; the plurality of instructions are used for loading and executing the method as described above by a processor.

[0026] The electronic device provided by the application comprises:

[0027] The processor is used for executing a plurality of instructions.

[0028] The memory is used for storing a plurality of instructions.

[0029] The plurality of instructions are used for storing in the memory and loading and executing the method as described above by the processor.

[0030] Advantages:

[0031] The application designs a representation learning model based on an attention mechanism and a graph convolution network, preserves face geometry attributes, protects demographic attributes by using a differential privacy mechanism, and reconstructs an image by a generative learning network to replace an original image for publishing for a computer vision analysis task.

[0032] The application has the following technical effects:

[0033] (1) The application can improve the precision and effectiveness of existing face image privacy protection methods.

[0034] (2) The application can effectively resist background knowledge attacks, reconstruction attacks and combination attacks, and achieve provable and controllable face identity privacy protection, ensuring the optimization trade-off between privacy and usability of the published image.

[0035] (3) The application is simple to implement.

[0036] (4) The application improves the precision of the published face image.

[0037] (5) The application combines differential privacy technology and generative learning network, and proposes a face identity de-identification method with provable and controllable privacy protection degree, which can effectively resist background knowledge attacks, reconstruction attacks and combination attacks, and achieve controllable and fine-grained face identity privacy protection. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 A flowchart of a face image identity privacy protection method provided by the application is shown.

[0039] Figure 2 An architecture diagram of the face image identity privacy protection method provided by the application is shown.

[0040] Figure 3 A comparison diagram of an original face image and a generated face image provided by the application is shown.

[0041] Figure 4 A structure diagram of a face image identity privacy protection device provided by the application is shown. DETAILED DESCRIPTION

[0042] The application will be described in detail below in combination with the drawings and examples.

[0043] As shown in the drawings, Figures 1-2 the application proposes a face image identity privacy protection method, which includes the following steps:

[0044] Step S1: constructing a feature learning model, performing feature extraction on an acquired face image by the feature learning model, decoupling feature information and background information in the face image by a face identity feature decoupling sub-network model, and integrating the feature extracted by the feature learning model and the feature information to obtain integrated information;

[0045] Step S2: constructing an identity perturbation model, adding random perturbation to the integrated information based on a privacy level of a face image user;

[0046] Step S3: constructing an image reconstruction model, reconstructing a face image based on the integrated information after adding random perturbation.

[0047] The present application considers that the geometric feature deformation change process such as face rotation and expression change actually belongs to nonlinear transformation, and these features are mainly represented by parts such as eyes, lips and nose, therefore, the feature learning model is designed by using attention mechanism and graph convolutional network (GCNs), the main geometric features are extracted from the original face image data by the attention mechanism, the relationship between the geometric features is measured by using GCNs, the geometric information related to the identity in the face image is preserved, the feature information and the background information are decoupled by using U-Net network, and the preserved geometric features and the feature information are integrated. Finally, the perturbed information is reconstructed by using DenseNet network, the reconstructed face is naturally fused with the background of the original image, and the face image publishing with differential privacy protection is realized.

[0048] Further, the representation learning model comprises a sub-network based on facial visual attributes and a decoupling sub-network; the sub-network based on facial visual attributes is composed of a light CNN model (lightCNN), a channel attention model (Channel Attention Model, CAM), and a graph convolutional model (Graph Convolutional Network, GCN) connected with each other; the obtained facial image is input into the light CNN model, and the light CNN model obtains multi-scale facial features of different feature channels. For example, deep semantic features are extracted through convolution operation, and the deep semantic features include facial posture, face shape, and expression. The channel attention model fuses the multi-scale facial features of different feature channels extracted by the light CNN model through multi-scale aggregation of different convolution layers to obtain a spatial semantic attribute facial feature map. For example, the visual semantic features such as facial posture and expression extracted by the light CNN model and the spatial features of each part of the face are fused to obtain a spatial semantic attribute facial feature map. The spatial semantic attribute facial feature map is input into the GCN network, and the GCN network learns the spatial semantic attribute facial feature map to obtain geometric features of structural relationships between local regions of the face, which are geometric features describing structural relationships between local regions of the face such as eyes, nose, and mouth.

[0049] The feature information refers to demographic attribute features such as gender and age. The facial identity representation decoupling sub-network model is composed of a U-Net model and a CNN-based decoupling representation model; the multi-scale facial features of different feature channels are subjected to feature information attribute operation to obtain feature information I′ n The U-Net model takes the feature information I′ n as input to extract a facial feature map; the CNN-based decoupling representation model receives the facial feature map, decouples facial visual attribute features and feature information in the facial feature map, and realizes decoupling of feature information and background information.

[0050] The geometric features of structural relationships between local regions of the face, decoupled facial visual attribute features, and decoupled feature information are fused through a linear weighting layer.

[0051] The feature map capable of editing facial identity attributes extracted by the representation learning model is integrated with the feature information to obtain integrated information.

[0052] In the step S1, the feature extraction of the obtained facial image by the representation learning model comprises:

[0053] The input of the characteristic learning model is a single image, represented as To obtain complete facial features, first, I n is input into the lightweight CNN network for feature learning to obtain multi-scale facial features in different feature channels, represented as where i = 1, 2; n = 1, 2, …, N represents the number of layers of the feature channel. On this basis, multi-scale visual feature fusion operations and attribute operations of feature information are performed on the multi-scale facial features in different feature channels, respectively, to obtain multi-scale visual fusion features and feature information I′ n ; the input channel attention model is input to fuse to obtain a spatial semantic attribute facial feature map containing spatial semantic attributes, which refers to semantic features containing the potential relationship of different spatial sub-regions of the facial image; the spatial semantic attribute facial feature map is input into the GCN network to extract sub-feature blocks representing the left eye right eye nose lips of the spatial semantic attribute facial feature map, and each sub-feature block is defined as a node in a graph network G = {V, G}, where G is a graph network, V is each graph node, E represents the boundary between nodes, and the connection relationship between nodes is represented as a boundary e E, where the boundary relationship between all graph nodes is represented as an adjacency matrix M, and M represents the relationship between the features of each node. The graph G learns the mutual relationship between nodes using GCNs to obtain geometric features of the structural relationship between facial local regions:

[0054] h g (I) = LReLU(MVW) = {h1(I), h2(I), …, h i2 (I)} i2 = 1, 2, …, m, where LReLU represents the activation function LeakyReLU.

[0055] Further, the adjacency matrix M and the state update weight value W are randomly initialized in the GCNs training process.

[0056] The step S1, the feature information and background information in the facial image are decoupled by the facial identity representation decoupling sub-network model, comprising:

[0057] I′ nThe input U-Net model outputs an intermediate feature vector through an encoder model of the U-net model, and then inputs the intermediate feature vector into a decoder model of the U-Net to obtain a face feature map; the face feature map is input into a CNN-based decoupling representation model to decouple the face visual attribute feature and the feature information, and the decoupled face visual attribute feature is denoted as h back , and the decoupled feature information is denoted as h d1 .

[0058] In this embodiment, the background representation information is represented by multi-level feature mapping The k-th level feature mapping of the U-Net decoder, k represents the number of feature layers, and G(·) represents the U-Net decoding function, so as to retain the detail information of different levels of the face image.

[0059] The step S2: constructing an identity perturbation model, adding random perturbation to the integrated information based on the privacy level of the face image user, wherein:

[0060] The identity perturbation model is a perturbation model using an ε-differential privacy mechanism, and the ε-differential privacy mechanism is calculated according to the decoupled feature information to determine the privacy level of the face image user; the differential privacy algorithm is used to add Laplace random perturbation to the corresponding part of the integrated information corresponding to the decoupled feature information h d1 to obtain the perturbed feature information.

[0061] The purpose of the identity perturbation model is to set the privacy level of the used face data, and the privacy level is set by using the identity perturbation model, wherein ε represents the privacy budget value, and the smaller the value is, the stronger the privacy protection degree is.

[0062] In this embodiment, the privacy level is determined according to the decoupled feature information obtained from the face image. The definition of ε-differential privacy is: given any two images D and D', if there exists an algorithm that outputs any result satisfying , then satisfies ε-differential privacy, , which represents the value range of . After obtaining the privacy level of the face image user, the privacy budget parameter ε is obtained, and then the perturbation is performed by using the Laplace noise generator, and the perturbation is performed according to the Laplace mechanism, wherein I and I' represent any two images, Δf is the global sensitivity of the mapping function f, , which represents the maximum L1 norm of the demographic representation information of any I and I'. The differential privacy algorithm The part of the integrated information corresponding to the decoupled visual attribute feature of the face is disturbed by adding a Laplace mechanism, d1 The corresponding part is disturbed by adding a Laplace mechanism, After the above operation, the final face publishing image with differential privacy protection is obtained.

[0063] In this embodiment, according to the definition of differential privacy, Satisfy the e-differential privacy:

[0064]

[0065] The step S3: constructing an image reconstruction model, the image reconstruction model reconstructs the face image based on the integrated information after adding random disturbance, comprising:

[0066] The image reconstruction model is a DenseNet model, the DenseNet network reconstructs the integrated information based on the feature information after adding random disturbance, and the DenseNet network further reconstructs the background information of the face image, so that the reconstructed integrated information is fused with the background of the face image.

[0067] First, the part of the integrated information corresponding to the decoupled visual attribute feature of the face and the disturbed feature information are integrated to obtain first integrated feature; then the first integrated feature is input into the image reconstruction model, the image reconstruction model realizes the extraction of the first face feature through dense convolution operation, realizes the reconstruction of the integrated information; the image reconstruction model further reconstructs the background information of the face image, so that the reconstructed integrated information is fused with the background of the face image, to obtain the fused face image; the fused face image is published.

[0068] The present application is experimentally verified by using specific examples. Figure 3 is the contrast chart of the original face image and the generated face image in this embodiment. As shown in Figure 3 The generated face image effectively retains the geometric information such as the outline, posture and expression of the user in the original image, realizes controllable and fine-grained attribute protection, and the loss degree of its precision compared with the original face image is also within the application acceptable range.

[0069] The present application also provides an identity privacy protection device for face image, as shown in Figure 4 The device comprises:

[0070] The feature extraction module is configured to construct a feature learning model, perform feature extraction on the acquired face image by the feature learning model, and decouple feature information and background information in the face image by a face identity feature decoupling sub-network model; and integrate the feature extracted by the feature learning model and the feature information to obtain integrated information.

[0071] The disturbance module is configured to construct an identity disturbance model, and add random disturbance to the integrated information based on a privacy level of a user of the face image.

[0072] The reconstruction module is configured to construct an image reconstruction model, and reconstruct the face image based on the integrated information after adding the random disturbance.

[0073] The specific embodiments above only describe the design principles of the present application, and the shapes and names of the components in the description can be different and are not limited. Therefore, the person skilled in the art of the present application can modify or equivalently replace the technical solutions described in the foregoing embodiments; and these modifications and replacements do not deviate from the purpose and technical solutions of the present application, and should all belong to the protection scope of the present application.

Claims

1. A method for identity privacy protection of a face image, characterized in that, The method comprises the following steps: Step S1: constructing a feature learning model, extracting features of an acquired face image by the feature learning model, and decoupling feature information and background information in the face image by a face identity feature decoupling sub-network model; integrating the features extracted by the feature learning model and the feature information to obtain integrated information; Step S2: constructing an identity disturbance model, and adding random disturbance to the integrated information based on a privacy level of a user of the face image; Step S3: constructing an image reconstruction model, and reconstructing the face image based on the integrated information after adding random disturbance; The feature learning model comprises a sub-network based on face visual attributes and a decoupling sub-network; the sub-network based on face visual attributes is composed of a lightweight CNN model, a channel attention model and a graph convolution model connected with each other; the acquired face image is input into the lightweight CNN model, and the lightweight CNN model acquires multi-scale face features of different feature channels; the channel attention model fuses the multi-scale face features of different feature channels extracted by the lightweight CNN model to obtain a spatial semantic attribute face feature map; the spatial semantic attribute face feature map is input into a GCN network, and the GCN network learns the spatial semantic attribute face feature map to obtain geometric features of structural relationships between face local regions; The human face identity representation decoupling sub-network model is connected by a U-Net model and a CNN-based decoupling representation model; the multi-scale human face features of different feature channels are subjected to feature information attribute operation to obtain feature information I' n , the U-Net model takes the feature information I' n as input to extract a human face feature map; the CNN-based decoupling representation model receives the human face feature map to decouple the human face visual attribute features and the feature information in the human face feature map; The image reconstruction model is a DenseNet model, the DenseNet model reconstructs the integrated information based on the feature information after adding random disturbance, and the DenseNet model further reconstructs background information of the face image, so that the reconstructed integrated information is fused with the reconstructed background information of the face image.

2. The method of claim 1, wherein, The feature information refers to features of gender and age attributes.

3. The method of claim 2, wherein, The geometric features of structural relationships between face local regions, decoupled face visual attribute features and decoupled feature information are fused by a linear weighting layer to obtain integrated information.

4. The method of claim 1, wherein, The step S1, wherein the feature extraction of the acquired face image by the representation learning model comprises: the input of the representation learning model is a single image, represented as I n is input into the lightweight CNN model for feature learning to acquire multi-scale face features of different feature channels, represented as wherein, i=1, 2; n=1, 2, …, N represents the number of layers of the feature channels; the multi-scale visual feature fusion operation and the attribute operation of the feature information are respectively performed on the multi-scale face features of the different feature channels to respectively obtain multi-scale visual fusion features and feature information I′ n ; the is input into a channel attention model, and the is fused to obtain a spatial semantic attribute face feature map containing spatial semantic attributes, the spatial semantic attributes refer to semantic features containing potential relationships of different spatial sub-regions of the face image; the spatial semantic attribute face feature map is input into the GCN network, and sub-feature blocks representing left eye parts right eye parts nose parts mouth parts are extracted from the spatial semantic attribute face feature map, and each sub-feature block is defined as a node in a graph network G={V, E}, wherein G is the graph network, V is each graph node, E represents the boundary between the nodes, and the connection relationship between the nodes is represented as the boundary e∈E, wherein the boundary relationship between all graph nodes is represented as an adjacency matrix M, M represents the relationship between the features of each node, wherein the graph G learns the mutual relationship between the nodes by using the GCNs to obtain geometric features of the structural relationship between the local regions of the face. In the step S1, the face identity feature decoupling sub-network model decouples the feature information and the background information in the face image, comprising: I' n inputting the intermediate feature vector into an encoder model of the U-Net model, outputting an intermediate feature vector, and inputting the intermediate feature vector into a decoder model of the U-Net to obtain a face feature map; inputting the face feature map into a CNN-based decoupling representation model to decouple the face visual attribute feature and the feature information, wherein the decoupled face visual attribute feature is denoted as h back , and the decoupled feature information is denoted as h d1 .

5. An apparatus for identity privacy protection of a face image, characterized by, The device comprises: A feature extraction module configured to construct a feature learning model, extract features of an acquired face image by the feature learning model, decouple feature information and background information in the face image by a face identity feature decoupling sub-network model, and integrate the features extracted by the feature learning model and the feature information to obtain integrated information; A disturbance module configured to construct an identity disturbance model and add random disturbance to the integrated information based on a privacy level of a user of the face image; A reconstruction module configured to construct an image reconstruction model, and reconstruct the face image based on the integrated information after adding random disturbance. The characteristic learning model comprises a sub-network based on facial visual attributes and a decoupling sub-network; the sub-network based on facial visual attributes is composed of a lightweight CNN model, a channel attention model and a graph convolution model connected with each other; the obtained facial image is input into the lightweight CNN model, and the lightweight CNN model obtains multi-scale facial features of different feature channels; the channel attention model fuses the multi-scale facial features of different feature channels extracted by the lightweight CNN model to obtain a spatial semantic attribute facial feature map; the spatial semantic attribute facial feature map is input into a GCN network, and the GCN network learns the spatial semantic attribute facial feature map to obtain geometric features of structural relationships between local regions of a face; The human face identity representation decoupling sub-network model is connected by a U-Net model and a CNN-based decoupling representation model; the multi-scale human face features of different feature channels are subjected to feature information attribute operation to obtain feature information I' n The U-Net model takes the feature information I' n as input to extract a human face feature map; the CNN-based decoupling representation model receives the human face feature map to decouple the human face visual attribute features and the feature information in the human face feature map. The image reconstruction model is a DenseNet model, the DenseNet model reconstructs the integrated information based on the feature information added with random disturbance, and the DenseNet model further reconstructs the background information of the facial image, so that the reconstructed integrated information is fused with the reconstructed background information of the facial image. 6.A computer readable storage medium, the storage medium storing a plurality of instructions; the plurality of instructions are used to load and execute the method of any one of claims 1-4 by a processor.

7. An electronic device, comprising: The electronic device comprises: a processor configured to execute a plurality of instructions; a memory configured to store the plurality of instructions; wherein the plurality of instructions are stored in the memory and executed by the processor to perform the method of any one of claims 1-4.

Images