Face privacy protection method, device and equipment and computer readable storage medium

Abstract

The embodiment of the present application relates to the technical field of computer vision, and discloses a face privacy protection method, device and equipment and a computer readable storage medium, the method comprising: performing average processing on original face data to obtain average features; performing privacy protection processing on the average features to obtain protected features; determining to-be-edited features according to the average features and the protected features; inputting the to-be-edited features and the original face data into a preset face editing model to obtain processed face data; wherein the face editing model is used for selectively performing feature editing on the original face data according to the to-be-edited features. Through the above manner, the embodiment of the present application can ensure that the identity information of the face is hidden while the modification of the face is small, thereby achieving a balance between face privacy protection and face image utility.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, specifically to a method, apparatus, device, and computer-readable storage medium for protecting facial privacy. Background Technology

[0002] The widespread use of modern social media has provided convenience and broader access to images, but it has also significantly threatened personal privacy. Therefore, facial image privacy protection has gained increasing attention with the development and popularization of computer vision technology. The goal is to remove sensitive private information, such as facial features and identity information, before uploading and sharing facial images with unknown third parties.

[0003] The inventors of this application discovered during the implementation of the embodiments of the present invention that existing methods for protecting facial privacy are difficult to balance between controlling privacy and image utility. Summary of the Invention

[0004] In view of the above problems, embodiments of the present invention provide a face privacy protection method to solve the problem of difficulty in achieving a balance between controlling privacy and image utility in the prior art.

[0005] According to one aspect of the present invention, a method for protecting facial privacy is provided, the method comprising:

[0006] The original face data is averaged to obtain the averaged features;

[0007] The averaged features are then subjected to privacy protection processing to obtain the protected features;

[0008] The features to be edited are determined based on the averaged features and the protected features;

[0009] The features to be edited and the original face data are input into a preset face editing model to obtain processed face data; wherein, the face editing model is used to selectively edit the original face data according to the features to be edited.

[0010] By determining the features to be edited based on the averaged features and the protected features, this approach differs from related technologies that directly protect facial privacy based on the protected features. It enables a smaller degree of modification to facial images, thereby maximizing the preservation of the original information of facial images and achieving a balance between facial privacy protection and the utility of facial images.

[0011] In an alternative approach, the method further includes:

[0012] The averaged features are anonymized to obtain anonymized features;

[0013] Differential privacy protection is applied to the anonymized features to obtain the protected features.

[0014] By combining anonymization with differential privacy protection to protect the privacy of averaged features, the accuracy of privacy protection can be improved.

[0015] In one alternative approach, the anonymization process includes k-anonymization; the differential privacy protection includes differential privacy protection based on an exponential mechanism.

[0016] By using k-anonymization and differential privacy protection based on an exponential mechanism, the degree of face privacy protection can be controlled and measured by setting parameters in k-anonymization.

[0017] In an alternative approach, the method further includes:

[0018] The original face data is used to predict features to obtain the protected features;

[0019] Random downsampling is performed on a pre-defined face dataset to obtain a subset feature space;

[0020] Select the k nearest neighbors corresponding to the protected feature from the feature space of the subset;

[0021] The averaged post-feature is determined based on the k nearest neighbors.

[0022] In this embodiment, the steps of downsampling and selecting the k nearest neighbors can be understood as cluster analysis. By predicting and clustering the original face data to obtain averaged features, this embodiment can achieve feature obfuscation of the original face data, thereby protecting the privacy of the original face.

[0023] In an alternative approach, the method further includes:

[0024] The feature to be edited is determined based on the difference between the averaged feature and the protected feature.

[0025] By using the difference between the averaged features and the protected features as the features to be edited, instead of directly identifying the protected features as the features to be edited, it is possible to protect facial privacy while making less editing to the face, thus ensuring a balance between the utility of the protected image and facial privacy.

[0026] In one alternative approach, the face editing model includes an encoder, a selection transmission unit, and a decoder; wherein the encoder is used to extract features from the original face data to obtain multiple original encoded features, and outputs the original encoded features to the selection transmission unit;

[0027] The selection transmission unit is used to select original encoded features from the plurality of original encoded features according to the feature to be edited, transform them to obtain transformed features, and input the transformed features and the untransformed original encoded features into the decoder;

[0028] The decoder is used to decode the transformed features and the original encoded features that have not been transformed, to obtain the processed face data.

[0029] By obtaining encoded features at different scales at different resolution layers through the encoder, and by using a selective transmission unit to adaptively transform the encoder features guided by the features to be edited and combining them with the encoder-decoder structure, the quality of the generated image and the feature manipulation capability are better maintained, achieving both precision and flexibility in face privacy protection.

[0030] In an alternative approach, the method further includes:

[0031] The feature loss of the processed face data is determined based on the processed face data and the features to be edited;

[0032] The authenticity of the processed face data is determined to obtain the adversarial loss of the processed face data;

[0033] The processed face data is optimized based on the feature loss and / or the adversarial loss.

[0034] By optimizing the processed facial data using defined feature loss and adversarial loss, the processed facial data can achieve effective privacy protection on the one hand, and avoid excessive distortion on the other, thus meeting the utility requirements of facial images.

[0035] According to another aspect of the present invention, a face privacy protection device is provided, comprising:

[0036] The first processing module is used to average the original face data to obtain the averaged features;

[0037] The protection module is used to perform privacy protection processing on the averaged features to obtain protected features;

[0038] The determination module is used to determine the feature to be edited based on the averaged features and the protected features;

[0039] The second processing module is used to input the features to be edited and the original face data into a preset face editing model to obtain processed face data; wherein, the face editing model is used to selectively edit the original face data according to the features to be edited.

[0040] According to another aspect of the present invention, a face privacy protection device is provided, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other through the communication bus;

[0041] The memory is used to store at least one executable instruction that causes the processor to perform operations as described in the face privacy protection method.

[0042] According to another aspect of the present invention, a computer-readable storage medium is provided, the storage medium storing at least one executable instruction that causes a face privacy protection device to perform the face privacy protection method as described above.

[0043] This invention embodiment involves averaging the original facial data to obtain averaged features; performing privacy protection processing on the averaged features to obtain protected features; determining the features to be edited based on the averaged features and the protected features; and inputting the features to be edited and the original facial data into a preset facial editing model to obtain processed facial data. The facial editing model is used to selectively edit the original facial data based on the features to be edited, enabling less modification to the facial image and preserving more of the original information of the facial image. This protects the privacy of the facial image while maximizing its utility, thereby achieving a balance between privacy protection and image utility during the facial image de-identification process.

[0044] The above description is merely an overview of the technical solutions of the embodiments of the present invention. In order to better understand the technical means of the embodiments of the present invention and to implement them in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the embodiments of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0045] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0046] Figure 1 A flowchart illustrating the face privacy protection method provided in an embodiment of the present invention is shown;

[0047] Figure 2 This diagram illustrates the structure of the face editing model in the face privacy protection method provided by an embodiment of the present invention.

[0048] Figure 3A flowchart illustrating a face privacy protection method according to another embodiment of the present invention is shown;

[0049] Figure 4 This diagram illustrates a comparison between the original facial data and the processed facial data in the facial privacy protection method provided by an embodiment of the present invention.

[0050] Figure 5 A schematic diagram comparing the face privacy protection method provided by the embodiments of the present invention with the face privacy protection effect of the prior art is shown;

[0051] Figure 6 A schematic diagram of the structure of the face privacy protection device provided in an embodiment of the present invention is shown;

[0052] Figure 7 A schematic diagram of the structure of the face privacy protection device provided in an embodiment of the present invention is shown. Detailed Implementation

[0053] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0054] Before describing the embodiments of the present invention, the prior art and its problems will be further explained:

[0055] Currently, there are three main types of face privacy protection algorithms in the field of computer vision: traditional algorithms, those based on generative adversarial networks (GANs), and those based on adversarial interference techniques. Traditional algorithms, such as blurring, pixelation, and adding Gaussian noise to pixels, are widely used due to their simplicity and ease of operation. These algorithms aim to directly obfuscate sensitive information in face images; however, existing research has shown that these methods have limited effectiveness in protecting face privacy, and encrypted face images can still be recognized with high accuracy by some deep learning models. Advanced algorithms based on GANs significantly improve the image quality and perception of protected faces. Face protection using adversarial interference techniques involves adding adversarial noise to the face to be protected, which can effectively perturb the performance of neural networks without being perceptible to the human eye.

[0056] The challenges of facial image privacy protection lie first in how to balance privacy and image utility to generate high-quality facial recognition images; and second in the measurement and interpretability of privacy protection. Although existing generative adversarial network-based facial recognition methods have been widely used to hide sensitive facial image privacy information, they have failed to control the trade-off between privacy and utility both qualitatively and quantitatively.

[0057] In summary, existing facial privacy protection methods fail to achieve a balance between privacy and utility in both qualitative and quantitative aspects.

[0058] Before describing the embodiments of the present invention, the relevant terms will be explained:

[0059] KNN (K nearest neighbor) is based on the idea that if most of the K most similar (i.e., the nearest neighbors in the feature space) samples of a sample belong to a certain category, then the sample also belongs to that category.

[0060] The KNN algorithm can be used not only for classification but also for regression.

[0061] KNN may include the following steps:

[0062] Calculate distance: Given a test object, calculate its distance to every object in the training set.

[0063] Find neighbors: Select the K nearest training objects as the nearest neighbors of the test object.

[0064] Classification: Classify the test object based on the primary category to which the K nearest neighbors belong.

[0065] KNN class determination can use a general voting method: the minority obeys the majority, and the class with the most points in the nearest neighbors is assigned to that class.

[0066] K-anonymity requires that the published data contain a certain number (at least k) of records that are indistinguishable in terms of quasi-identifiers, making it impossible for attackers to determine the specific individual to whom the private information belongs, thus protecting personal privacy. K-anonymity specifies the maximum information disclosure risk a user can tolerate through the parameter k. Here, "identifier" generally refers to a unique identifier for an individual, such as name, address, or phone number; this information needs to be removed when publishing the data. "Quasi-identifier" refers to identifiers that are not unique, such as zip codes, age, or birthdays, but help researchers manage relevant data.

[0067] There are two main types of K-anonymity methods:

[0068] (1) One method is to delete the corresponding data column and replace it with an asterisk;

[0069] (2) Another method is to use generalization to make them indistinguishable, for example, to generalize the number of ages into an age range.

[0070] Differential privacy: Two datasets D and D′ are called adjacent datasets if they differ in one and only one record x. For adjacent datasets, the probability of querying them and obtaining the same value is very close.

[0071] Definition: Given two neighboring datasets D and D', if an algorithm K, with a range of Range(m), outputs a result O (O∈Range(m)) on D and D' that satisfies the following inequality, then the algorithm K is said to satisfy ε-differential privacy:

[0072] Pr[K(D)=O]≤exp(ε)*Pr[K(D')=O]

[0073] Where Pr[.] represents the probability that the output of algorithm K is O, and the privacy budget ε controls the degree of privacy protection. The value of ε is inversely proportional to the degree of privacy protection; the smaller ε is, the more noise needs to be added, and the higher the degree of privacy protection.

[0074] The main mechanisms for implementing ε-differential privacy include: the Laplace mechanism, which adds noise that follows a Laplace distribution to the query results, suitable for numerical outputs; and the exponential mechanism, which uses an exponential distribution to adjust probabilities in the query results, suitable for non-numerical outputs.

[0075] The exponential mechanism is specifically defined as follows: Given a random algorithm M, input is a dataset D, the range is Range, and the output is the entity object r∈Range(m).

[0076] q(D,r) is the scoring function, and Δq is the sensitivity of the scoring function q.

[0077] If the algorithm is proportional to If the probability is selected from Range and r is output, then algorithm m satisfies ε-differential privacy.

[0078] The Selective Transmission Unit (STU) is used to selectively transform encoder features to make them compatible and complementary to decoder features, rather than directly connecting encoder and decoder features via skip connections. It is implemented on the structure of a GRU (Gate Recurrent Unit), selecting encoded features under differential tag control. This allows for arbitrary image feature editing, rather than considering the entire target feature vector, only the features to be changed to preserve more information about the source image.

[0079] Figure 1 A flowchart of a face privacy protection method provided by an embodiment of the present invention is shown. This method is executed by a computer processing device. The computer processing device may include a mobile phone, a laptop computer, etc. Figure 1 As shown, the method includes the following steps:

[0080] Step 10: Average the original face data to obtain the averaged features.

[0081] In one embodiment of the present invention, the original face data may be a face image that requires privacy protection processing. Averaging is used to extract the average features of the original face, and these average features are used as the basis for privacy protection processing. This preserves the feature information of the original face data while eliminating the need for direct privacy protection processing, thus further improving the security of face privacy.

[0082] Specifically, feature prediction can be performed on the original facial data, and the predicted features can be used as the averaged features. This prediction can be achieved by clustering the original facial data based on a large dataset of facial samples, averaging the features of multiple other facial samples within the same cluster to obtain the averaged features of the original facial data. Editing the face to be protected towards the averaged features ensures that identity information is hidden while making only minor modifications. That is, while the averaged features can represent the characteristic information of the original facial data and thus serve as the basis for privacy protection processing, the original features included in the original facial data are not directly exposed, thereby ensuring the security and accuracy of privacy protection.

[0083] Therefore, in one embodiment of the present invention, step 10 further includes:

[0084] Step 101: Perform feature prediction on the original face data to obtain the protected features.

[0085] In one embodiment of the present invention, face image feature prediction can be regarded as a multi-label classification problem. A pre-trained face classification model can be used to predict the protected features corresponding to the original face data. The face classification model outputs the prediction results of multiple preset key features for the input original face data. The number of key features can be 40. The face classification model can be trained using the MultiLabelSoftMarginLoss loss function based on cross-entropy loss.

[0086] Step 102: Perform random downsampling based on the preset face dataset to obtain a subset feature space.

[0087] In one embodiment of the present invention, the preset probability distribution can be a Poisson distribution. Downsampling of the face dataset reduces the dimensionality of features while retaining effective information, avoiding overfitting of the subset feature space. The face dataset represents generalized, mass-market face data, and may include multiple reference face sample data, which can be obtained by preprocessing a face database using big data techniques. By downsampling the face dataset representing mass-market faces, noisy features that obfuscate the protected features are obtained, making the protected features closer to average face features and more representative of the general population, thereby achieving the goal of face privacy protection.

[0088] Step 103: Select the k nearest neighbors corresponding to the protected feature from the subset feature space.

[0089] In one embodiment of the present invention, k nearest neighbors are selected based on the distance between the protected feature and each feature in the subset feature space, wherein the distance may include Euclidean distance.

[0090] Step 104: Determine the averaged feature based on the k nearest neighbors.

[0091] In one embodiment of the present invention, in order to limit the global sensitivity of multi-label classification problems and make the subsequent addition of random noise more efficient, the averaged features can be obtained by voting from the k nearest neighbors under the τ approximation mechanism. Optionally, the average attribute of the given k nearest neighbors can also be calculated as the averaged features.

[0092] Step 20: Perform privacy protection processing on the averaged features to obtain protected features.

[0093] In one embodiment of the invention, privacy protection is used to introduce random noise into the averaged features to interfere with and obfuscate them, thereby preventing the original facial features from being directly obtained from the protected features, thus protecting facial privacy. To ensure the controllability and measurability of the degree of privacy protection during the privacy protection process, differential privacy computation can be used, and the degree of privacy protection can be controlled by adjusting the computational parameters in the differential privacy computation. Furthermore, anonymization can be performed before differential privacy protection to improve the accuracy of privacy protection. The degree of privacy protection can be further controlled by setting the parameters of the anonymization process.

[0094] Therefore, in one embodiment of the present invention, step 20 further includes:

[0095] Step 201: Anonymize the averaged features to obtain anonymized features.

[0096] In one embodiment of the present invention, the anonymization process includes k-anonymization. By setting the parameter k in the k-anonymization process, the degree of face protection can be controlled and measured. The k-anonymization theory protects the original data information by publishing data with lower precision through a two-step operation of summarizing and concealing the dataset. This theory is based on the quasi-identifier assumption, that is, the data holder can identify features that may appear in external data, requiring that each record is indistinguishable from at least k-1 other records in the original dataset. A "quasi-identifier" refers to a set of features that can be joined with an external table to identify an individual. Since facial features can be used to identify and associate identity information in facial data, in this algorithm, "quasi-identifiers" are defined as facial features, and their semantic features are defined as facial facial features and associated identity features.

[0097] Step 202: Perform differential privacy protection on the anonymized features to obtain the protected features.

[0098] In one embodiment of the present invention, considering that k-anonymization can provide a certain degree of privacy guarantee, it also has certain limitations, such as being unable to resist homogenization attacks, i.e., the theory fails if the sensitive feature values ​​within a group are exactly the same; at the same time, k-anonymization theory yields relatively deterministic results, allowing attackers to obtain private information with a high probability based on existing background knowledge. To address these shortcomings in privacy protection processing of anonymization, one embodiment of the present invention further performs differential privacy calculation processing on the anonymized features. Differential privacy protects user privacy against differential attacks, aiming to provide a method that maximizes data query accuracy while minimizing the possibility of identifying records when querying from a statistical database. By combining anonymization processing with differential privacy protection to perform privacy protection processing on the averaged features, a more protected set of features is obtained.

[0099] Differential privacy protection can be further divided into Laplacian, exponential, and Gaussian mechanisms, depending on the noise distribution. Since we use Boolean data to represent the presence or absence of certain attributes in face images, differential privacy protection based on the exponential mechanism can further guarantee privacy on top of anonymized features.

[0100] Step 30: Determine the feature to be edited based on the averaged features and the protected features.

[0101] In one embodiment of the present invention, the feature to be edited is used to characterize the features in the original face data that need to be edited, such as facial features in the dimensions of hairstyle, skin color, and facial features. By comparing the averaged feature and the protected feature, the difference between the two is obtained, and this difference is used as the feature to be edited. Based on the feature to be edited, specific features are selected from the original face data for targeted editing, making the original face data closer to the protected feature, thus achieving the purpose of privacy protection. At the same time, it eliminates the need for direct face protection processing based on the protected feature, enhancing the flexibility of the features and simplifying the subsequent training process of the face editing model.

[0102] Therefore, in one embodiment of the present invention, step 30 further includes:

[0103] Step 301: Determine the feature to be edited based on the difference between the averaged feature and the protected feature.

[0104] In one embodiment of the present invention, the difference between the averaged feature and the protected feature can be calculated, and the difference feature can be identified as the feature to be edited.

[0105] Step 40: Input the features to be edited and the original face data into a preset face editing model to obtain processed face data; wherein, the face editing model is used to selectively edit the original face data according to the features to be edited.

[0106] In one embodiment of the present invention, privacy protection can be achieved simply by editing the corresponding features in the original face data based on the editable features that characterize the difference between the ideal edited face and the original face data. This avoids the problem of a large workload when directly processing the original face data based on the protected features, and improves the targeting and flexibility of face editing. Therefore, the editable features and the original face data can be input together into the face editing model, allowing the face editing model to perform targeted editing on the features corresponding to the editable features in the original face data. This makes the feature values ​​of the processed face data closer to the protected features, thus protecting the privacy of the face features in the processed face data. At the same time, the original face data is not directly processed based on the protected features to preserve the original image information to the greatest extent and enhance the operational capabilities of face features.

[0107] To achieve selective feature processing of raw facial data in the face editing model, an input layer, an output layer, and multiple intermediate layers for selectively connecting the input and output layers can be set up in the face editing model. Through selective connections in the intermediate layers, certain features are selectively transmitted to the output layer, achieving the goal of selective feature processing. Specifically, the selection transmission unit selectively transforms encoder features, making them compatible and complementary to decoder features, rather than directly connecting encoder and decoder features through skip connections, thus improving feature editing capabilities and image quality. The decoder decodes image features and combines them with fused features from various resolutions from the multi-scale feature fusion module, ultimately synthesizing processed facial image data with both realism and the features to be edited.

[0108] Therefore, in another embodiment of the present invention, the face editing model includes an encoder, a selection transmission unit, and a decoder; wherein, the encoder is used to extract features from the original face data to obtain multiple original encoded features, and output the original encoded features to the selection transmission unit;

[0109] The selection transmission unit is used to select original encoded features from the plurality of original encoded features according to the feature to be edited, transform them to obtain transformed features, and input the transformed features and the untransformed original encoded features into the decoder;

[0110] The decoder is used to decode the transformed features and the original encoded features that have not been transformed, to obtain the processed face data.

[0111] In one embodiment of the present invention, an encoder extracts features from the original face data to reduce data dimensionality, facilitating subsequent selective editing of features. A selective transmission unit selectively connects the encoder and decoder, enabling the selection and transformation of original encoded features from a plurality of original encoded features based on the features to be edited, resulting in transformed features. These transformed features, along with the untransformed original encoded features, are input to the decoder. Finally, the decoder decodes the received features to obtain the processed face features. Specifically, the encoder extracts features from the input original face data to obtain original encoded features at different resolutions. i represents the layer number of the encoder, i∈n. Subsequently, a transmission unit is selected instead of a skip connection to directly concatenate the encoder and decoder in the first n-1 layers. The encoder function is selectively transformed according to the features to be edited to make it compatible and complementary with the decoder function. Thus, the face editing model can obtain encoded features of different scales through the encoder at different resolution layers. At the same time, the transmission unit is used to adaptively transform the encoder features guided by the features to be edited and combine them with the encoder-decoder structure, thereby better maintaining the quality of the generated image and the feature manipulation capability.

[0112] Specifically, such as Figure 2 As shown, the working process of the face editing model can be as follows: First, the encoder extracts features from the input raw face data to obtain encoded features at different resolutions. i represents the layer number of the encoder, i∈n. Subsequently, by using a selected transmission unit instead of skipped connections, the encoder and decoder are directly concatenated in the first n-1 layers. The encoder function is selectively converted according to the features to be edited, making it compatible and complementary to the decoder function. j+1 This represents the hidden state of layer j+1. The hidden state s is then processed using a transposed convolution. j+1 Upsampling is performed, and then the mathematical model of the GRU is used to update the hidden state and the transform encoder function. j represents the layer number of the encoder, j∈n-1. Finally, the features encoded in the nth layer are... and the transformed first n-1 layer coding features The data is input to the decoder, which ultimately decodes and synthesizes the processed face data that matches the features to be edited. In the face editing model, layer n refers to the last layer of the encoder, which is related to the network structure of the encoder used. For example, layer n might be the fifth layer, and the corresponding layers n-1 would be layers one through four.

[0113] In one embodiment of the present invention, feature loss and realism assessment can be performed on the processed face data for classification. Based on at least one of the above two assessment results, the processed face data can be further optimized to make it closer to the ideal direction represented by the edited features and / or more realistic and credible. Therefore, after step 40, the following may be included:

[0114] Step 401: Determine the feature loss of the processed face data based on the processed face data and the features to be edited.

[0115] In one embodiment of the present invention, firstly, feature prediction is performed on the processed face data to obtain processed features. Optionally, to maintain consistency in feature prediction standards and improve the accuracy of feature loss calculation, feature prediction can be performed directly using the face classification model described in step 101. The face classification model predicts the processed face features corresponding to the processed face data, calculates the feature loss for classification based on the processed face, and thus optimizes the face features edited by the face editing model to better match the features to be edited. Specifically, the face classification model is a discriminator based on the degree of condition satisfaction. The cross-entropy between the processed features and the features to be edited is determined, and the cross-entropy is used to characterize the feature loss of the processed face.

[0116] Step 402: Perform an authenticity assessment on the processed face data to obtain the adversarial loss of the processed face data.

[0117] In one embodiment of the present invention, a face discrimination model can be used to distinguish and score real and fake faces in the processed face data, and calculate adversarial loss, thereby optimizing the quality of the processed face data synthesized by the face editing model to more closely resemble real face images. The face discrimination model can use Wasserstein distance as the adversarial loss. A gradient penalty-based stable training strategy can be used to stabilize the training process of the face discrimination model. Optionally, the face discrimination model can share convolutional layer weights with the aforementioned face classification model.

[0118] Step 403: Optimize the processed face data based on the feature loss and / or the adversarial loss.

[0119] In one embodiment of the present invention, the processed face is iteratively optimized with the goal of minimizing feature loss and / or adversarial loss.

[0120] In yet another embodiment of the invention, such as Figure 3As shown, firstly, the average post-features of the original face data are predicted using a face classification model. Then, privacy protection processing is applied to the protected post-features using a differential privacy and k-anonymization algorithm, resulting in protected post-features. Finally, the difference between the average post-features and the protected post-features is calculated as the features to be edited. Next, a face editing model performs end-to-end feature transformation on the original face data based on the original face data to be edited and the protected post-features to be edited, completing the face feature editing and synthesizing processed face data that conforms to the features to be edited. This embodiment of the invention combines a face editing model and a face classification model to realize the training and inference process. By combining differential privacy and k-anonymization, the face classification model performs privacy protection processing on the average post-features output from the original face data to obtain protected post-features. These protected post-features are used as guidance and control conditions for feature editing, and together with the original face data, the face editing model is trained to learn the multi-domain mapping from the original face data to the features to be edited, thus synthesizing processed face data. This achieves targeted editing and protection of sensitive face features.

[0121] The face editing model takes the difference between preserved and averaged features, along with the original face data to be edited, as input to synthesize an image that matches the target face features, thereby enhancing the flexibility of feature transformation and simplifying the training process. Specifically, the face editing model includes an encoder, a selection and transmission unit, and a decoder; wherein the encoder extracts features from the input original face data to be edited, obtaining encoded features at different resolutions. i represents the layer number of the encoder, i∈n; a transmission unit is selected instead of a skip connection, and the encoder and decoder are directly concatenated in the first n-1 layers. The encoder function is selectively converted to be compatible and complementary to the decoder function based on the facial features to be edited. Using the encoder, features are extracted from the input raw facial data to obtain encoded features f at different resolutions. i middle.

[0122] Finally, the generated processed face data is optimized based on the classification loss and adversarial loss. Specifically, the face classification model is used to predict the face features of the processed face data, and the classification loss is calculated based on the features to be edited, thereby optimizing the face features edited by the face editing model to better match the features to be edited.

[0123] The face classification model is a discriminator based on the degree of condition satisfaction. It shares convolutional layer weights with the face discrimination model and calculates the matching loss between the image and the conditions by generating the cross-entropy between the image-protected features and the features to be edited, which serves as the feature loss. The face discrimination model is used to score the processed face data as real or fake, and calculates the adversarial loss to optimize the quality of the processed face data synthesized by the face editing model, making it closer to real face images. The face discrimination model uses Wasserstein distance as the adversarial loss. A stable training strategy of gradient penalty is used to stabilize the training process of the face discrimination model.

[0124] Specifically, for training the models mentioned above, a ResNet-50 network can be used to train the face classification model, which will then be used as the feature extractor network in predicting average features. The Adam optimizer will be employed, with a batch size of 128 samples. MultiLabelSoftMarginLoss will be used as the loss function, which is a multi-label loss optimized based on maximum entropy. A base learning rate of 4×10⁻⁶ can be set. -4 It decays to 5×10 through polynomial decay. -4 A 2048-dimensional vector is extracted from the fully connected layer as deep features. The learning rate is initialized to 2×10⁻⁶ when training the face editing and face discrimination models. -4 After 100 rounds, the learning rate is reduced to 2×10. -5 Make minor adjustments.

[0125] like Figure 3 As shown, this embodiment of the invention predicts the averaged post-features of the original face data using a face classification model, and combines differential privacy protection and k-anonymization to perform privacy protection processing on the averaged post-features, resulting in a privacy-protected face. Then, based on the difference between the averaged post-features and the protected post-features, the features to be edited are determined. Finally, a face editing model performs end-to-end feature transformation on the original face data according to the features to be edited, completing the face feature editing and obtaining the processed face data. On the one hand, by performing targeted feature editing on the original face data based on the difference between the averaged post-features representing the average attributes of the original face data and the privacy-protected post-features, a smaller degree of modification to the face image can be achieved, thus ensuring the utility of the modified face image. On the other hand, in terms of privacy protection, by combining traditional privacy protection k-anonymization and differential privacy theory with deep learning face editing, metric privacy is achieved on the basis of face de-identification, and identity information is de-identified based on the indistinguishability of face features, realizing controllable and measurable face privacy protection.

[0126] The processing effect of the original face data in the embodiments of the present invention can be used as a reference. Figure 4 as well as Figure 5 ,in, Figure 4 The images in the diagram show the protection results generated in this embodiment, with the left image being the original image and the right image being the protected image. It can be seen that the method provided by this embodiment can preserve the original image and the protected image. Figure 1 It achieves a certain similarity and generates natural and realistic results.

[0127] Figure 5 The same source image was selected as input to observe the generation results of various methods. From left to right, the results are traditional algorithms (blurring, Gaussian noise, pixelation) and deep learning-based face image privacy protection algorithms (AnonymousNet, DeepPrivacy, CIAGAN). The rightmost result is from this embodiment. It can be seen that the method in this embodiment can successfully transform the source image according to the features to be edited, with fewer artifacts and editing traces.

[0128] This invention embodiment involves averaging the original facial data to obtain averaged features; performing privacy protection processing on the averaged features to obtain protected features; determining the features to be edited based on the averaged features and the protected features; and inputting the features to be edited and the original facial data into a preset facial editing model to obtain processed facial data. The facial editing model is used to selectively edit the original facial data based on the features to be edited, enabling less modification to the facial image and preserving more of the original information of the facial image. This protects the privacy of the facial image while maximizing its utility, thereby achieving a balance between privacy protection and image utility during the facial image de-identification process.

[0129] Figure 6 A schematic diagram of the structure of a face privacy protection device provided in an embodiment of the present invention is shown. Figure 6 As shown, the device 50 includes: a first processing module 501, a protection module 502, a determination module 503, and a second processing module 504.

[0130] The first processing module 501 is used to perform average processing on the original face data to obtain the averaged features;

[0131] Protection module 502 is used to perform privacy protection processing on the averaged features to obtain protected features;

[0132] The determination module 503 is used to determine the feature to be edited based on the averaged features and the protected features;

[0133] The second processing module 504 is used to input the features to be edited and the original face data into a preset face editing model to obtain processed face data; wherein, the face editing model is used to selectively edit the original face data according to the features to be edited.

[0134] The operation process of the face privacy protection device in this embodiment of the invention is largely the same as that in the foregoing embodiments of the invention, and will not be described again.

[0135] The facial privacy protection device of this invention performs averaging processing on the original facial data to obtain averaged features; performs privacy protection processing on the averaged features to obtain protected features; determines the features to be edited based on the averaged features and the protected features; and inputs the features to be edited and the original facial data into a preset facial editing model to obtain processed facial data. The facial editing model is used to selectively edit the original facial data based on the features to be edited, enabling less modification to the facial image and preserving more of the original information of the facial image. This protects facial image privacy while maximizing the utility of the facial image, thereby achieving a balance between privacy protection and image utility during the facial image de-identification process.

[0136] Figure 7 The diagram shows a structural schematic of a face privacy protection device provided in an embodiment of the present invention. The specific embodiments of the present invention do not limit the specific implementation of the face privacy protection device.

[0137] like Figure 7 As shown, the face privacy protection device may include: a processor 602, a communications interface 604, a memory 606, and a communication bus 608.

[0138] The processor 602, communication interface 604, and memory 606 communicate with each other via communication bus 608. Communication interface 604 is used to communicate with other network elements such as clients or other servers. The processor 602 executes program 610, specifically performing the relevant steps described above in the embodiment of the face privacy protection method.

[0139] Specifically, program 610 may include program code, which includes computer-executable instructions.

[0140] Processor 602 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. The face privacy protection device includes one or more processors, which may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0141] Memory 606 is used to store program 610. Memory 606 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0142] Specifically, program 610 can be called by processor 602 to cause the face privacy protection device to perform the following operations:

[0143] The original face data is averaged to obtain the averaged features;

[0144] The averaged features are then subjected to privacy protection processing to obtain the protected features;

[0145] The features to be edited are determined based on the averaged features and the protected features;

[0146] The features to be edited and the original face data are input into a preset face editing model to obtain processed face data; wherein, the face editing model is used to selectively edit the original face data according to the features to be edited.

[0147] The operation process of the face privacy protection device in this embodiment of the invention is largely the same as that in the foregoing embodiments of the invention, and will not be described again.

[0148] The facial privacy protection device of this invention performs averaging processing on the original facial data to obtain averaged features; performs privacy protection processing on the averaged features to obtain protected features; determines the features to be edited based on the averaged features and the protected features; and inputs the features to be edited and the original facial data into a preset facial editing model to obtain processed facial data. The facial editing model is used to selectively edit the original facial data based on the features to be edited, enabling less modification to the facial image and preserving more of the original information of the facial image. This protects facial image privacy while maximizing the utility of the facial image, thereby achieving a balance between privacy protection and image utility during the facial image de-identification process.

[0149] This invention provides a computer-readable storage medium storing at least one executable instruction. When the executable instruction is executed on a face privacy protection device, the face privacy protection device performs the face privacy protection method described in any of the above method embodiments.

[0150] Specifically, the executable instructions can be used to cause the face privacy protection device to perform the following operations:

[0151] The original face data is averaged to obtain the averaged features;

[0152] The averaged features are then subjected to privacy protection processing to obtain the protected features;

[0153] The features to be edited are determined based on the averaged features and the protected features;

[0154] The features to be edited and the original face data are input into a preset face editing model to obtain processed face data; wherein, the face editing model is used to selectively edit the original face data according to the features to be edited.

[0155] The operation process of the executable instructions stored in the computer-readable storage medium of this invention embodiment is largely the same as that of the foregoing invention embodiments, and will not be repeated here.

[0156] The executable instructions stored in the computer-readable storage medium of this invention perform the following steps: averaging the original face data to obtain averaged features; performing privacy protection processing on the averaged features to obtain protected features; determining the features to be edited based on the averaged features and the protected features; and inputting the features to be edited and the original face data into a preset face editing model to obtain processed face data. The face editing model is used to selectively edit the original face data based on the features to be edited, enabling less modification to the face image and preserving more of the original information of the face image. This protects the privacy of the face image while maximizing its utility, thereby achieving a balance between privacy protection and image utility during the face image de-identification process.

[0157] This invention provides a face privacy protection device for performing the above-described face privacy protection method.

[0158] This invention provides a computer program that can be invoked by a processor to cause a face privacy protection device to execute the face privacy protection method in any of the above method embodiments.

[0159] This invention provides a computer program product, which includes a computer program stored on a computer-readable storage medium. The computer program includes program instructions, which, when executed on a computer, cause the computer to perform the face privacy protection method described in any of the above method embodiments.

[0160] The algorithms or displays provided herein are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used in conjunction with the teachings herein. The required structure for constructing such systems is apparent from the above description. Furthermore, the embodiments of the present invention are not directed to any particular programming language. It should be understood that the content of the invention described herein can be implemented using various programming languages, and the above description of specific languages ​​is for the purpose of disclosing the best mode of implementation of the invention.

[0161] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0162] Similarly, it should be understood that, in order to streamline the invention and aid in understanding one or more of the various aspects of the invention, features of the embodiments of the invention are sometimes grouped together in a single embodiment, figure, or description thereof in the above description of exemplary embodiments of the invention. However, this disclosure should not be construed as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim.

[0163] Those skilled in the art will understand that modules in the device of the embodiments can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or device so disclosed. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0164] It should be noted that the above embodiments are illustrative of the invention and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be construed as limiting the order of execution.

Claims

1. A method for protecting facial privacy, characterized in that, The method includes: Feature prediction is performed on the original face data to obtain the protected features; random downsampling is performed on the preset face dataset to obtain a subset feature space; k nearest neighbors corresponding to the protected features are selected from the subset feature space; the averaged features are determined based on the k nearest neighbors; The averaged features are anonymized to obtain anonymized features; the anonymized features are then subjected to differential privacy protection processing to obtain protected features with a higher degree of protection. The feature to be edited is determined based on the difference between the averaged feature and the more protected feature; The features to be edited and the original face data are input into a preset face editing model to obtain processed face data; wherein, the face editing model is used to selectively edit the original face data according to the features to be edited.

2. The method according to claim 1, characterized in that, The anonymization process includes k-anonymization; the differential privacy protection includes differential privacy protection based on an exponential mechanism.

3. The method according to claim 1, characterized in that, The face editing model includes an encoder, a selection transmission unit, and a decoder; wherein, the encoder is used to extract features from the original face data to obtain multiple original encoded features, and outputs the original encoded features to the selection transmission unit; The selection transmission unit is used to select original encoded features from the plurality of original encoded features according to the feature to be edited, transform them to obtain transformed features, and input the transformed features and the untransformed original encoded features into the decoder; The decoder is used to decode the transformed features and the original encoded features that have not been transformed, to obtain the processed face data.

4. The method according to claim 1, characterized in that, After inputting the features to be edited and the original face data into a preset face editing model to obtain processed face data, the process further includes: The feature loss of the processed face data is determined based on the processed face data and the features to be edited; The authenticity of the processed face data is determined to obtain the adversarial loss of the processed face data; The processed face data is optimized based on the feature loss and / or the adversarial loss.

5. A facial privacy protection device, characterized in that, The device includes: The first processing module is used to predict features from the original face data to obtain protected features; to perform random downsampling on a preset face dataset to obtain a subset feature space; to select the k nearest neighbors corresponding to the protected features from the subset feature space; and to determine the averaged features based on the k nearest neighbors. The protection module is used to anonymize the averaged features to obtain anonymized features; and to perform differential privacy protection processing on the anonymized features to obtain protected features with a higher degree of protection. The determination module is used to determine the feature to be edited based on the difference between the averaged feature and the protected feature with a higher degree of protection; The second processing module is used to input the features to be edited and the original face data into a preset face editing model to obtain processed face data; wherein, the face editing model is used to selectively edit the original face data according to the features to be edited.

6. A facial privacy protection device, characterized in that, include: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction that causes the processor to perform the operation of the face privacy protection method as described in any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The storage medium stores at least one executable instruction, which, when executed on the face privacy protection device, causes the face privacy protection device to perform the operation of the face privacy protection method as described in any one of claims 1-4.

Images