Face edge intelligent application part fine-tuning privacy protection model and method

Abstract

The present application relates to a kind of human face edge intelligent application partial fine-tuning privacy protection model and method.Currently, edge computing provides many benefits for many intelligent applications, but at the same time, when performing end-edge distributed computing during edge computing offloading, privacy leakage problems may occur.In order to deal with the challenge of privacy leakage, the present application proposes a distributed privacy protection strategy that guarantees the extraction of human face attribute features and the concealment of private features, which is a two-way task of privacy feature concealment method.This method is implemented by a privacy concealment feature extraction module that includes an identity feature concealment and gender feature extraction function, an attribute feature analysis reasoning module, and an attribute classification module.Different modules run on different entities, and the feature reasoning network with gradient flip layer is adjusted with the feature concealment layer and the multi-step privacy feature concealment of adversarial training.The final experimental results show that this network framework algorithm can effectively achieve privacy concealment.

Description

Technical Field

[0001] This invention belongs to the field of edge computing and relates to a method for protecting facial privacy, specifically a model and method for fine-tuning privacy protection in a facial edge intelligent application. Background Technology

[0002] Mobile Edge Computing (MEC) [1] offloads the computing tasks of user intelligent applications to nearby edge servers, which can reduce the overhead of local resources, alleviate the communication delay and congestion caused by cloud computing communication, improve the efficiency of task computing, and meet the user application service experience quality requirements. Edge computing has received increasing attention [2][3]. However, since edge intelligence needs to offload tasks to edge servers or other service nodes for distributed computing, and the owners of these service nodes or servers may exist in various forms, it has raised more serious privacy protection issues. For example, the intermediate processing results of the server may include private data, and attackers can infringe on privacy by attacking the task data offloaded to the edge server [4][5]. These issues are called new privacy protection issues of edge computing. If the private data is further transmitted to the remote server, the privacy issues will also be leaked during the transmission process. The privacy attribute protection provided by the existing edge computing is not perfect and cannot guarantee that the security of the data uploaded to the remote server will not be violated.

[0003] To date, many privacy protection methods have been proposed. However, existing privacy methods are not specifically designed for edge computing scenarios. They mainly consider one-time transformation, scrambling, and encryption [6][7][8], followed by detransformation, descrambling, and decryption on the cloud server, which introduces a large amount of computation. Therefore, the privacy protection provided by existing edge computing is not perfect and cannot guarantee that the data uploaded to the edge will not be violated. Currently, many edge intelligent applications related to faces are extracting various facial attribute features. Since identity features and these attributes are closely related, there is a risk that identity attribute features may also be extracted while recognizing a certain attribute.

[0004] 1.1 Current Status of Research on Domain Privacy Protection Algorithms

[0005] In general, existing methods for privacy protection algorithms can be roughly divided into the following three categories:

[0006] (1) Secure Multi-Party Privacy Computation Method

[0007] In the absence of a third party, multi-party secure computation adopts a protocol standard solution to allow multiple participants to cooperate and complete the computation together. The data required for the computation is always stored in the local database of each participant throughout the computation process, which ensures the privacy of the input data. Each participant performs the computation collaboratively and returns their respective computation results after the task is completed, ensuring the correctness of the computation

[14]

[15] .

[0008] Secure multi-party computation is also often used to protect the privacy of sensitive data in biometric identification systems. Face recognition privacy protection schemes using secure multi-party computation can both perform face recognition and ensure the privacy of face data. This process usually requires the simultaneous use of multiple cloud servers

[16] . Cai et al.

[17] proposed a secure face recognition scheme in a joint outsourced cloud environment, which can effectively protect the privacy of users' sensitive data for semi-trusted models. The eigenface algorithm is used as a face recognition scheme and runs in a privacy-preserving manner on two semi-trusted and non-colluding cloud servers. The Paillier cryptosystem is used for the design of the security model to ensure that the recognition result of the scheme is exactly the same as the matching result of the standard eigenface algorithm

[18] . Ma et al. proposed a secure face authentication scheme

[19] , which uses a convolutional neural network to extract the feature vector of the face and uses two non-colluding servers. One server is used to store the encrypted face features of the user, and the other is used to complete the face verification. All data is transmitted in encrypted state, and no other device except the face authentication server can decrypt the data. Haghighat et al.

[20] used secure circuits and homomorphic encryption to realize face recognition in the encrypted domain. The scheme adopts a KD tree structure, which is not only conducive to realizing face recognition in the encrypted domain, but also greatly improves the recognition efficiency of the system.

[0009] (2) Privacy protection for image retrieval

[0010] Image retrieval methods, including text retrieval and content retrieval, may lead to the theft and leakage of private data of images uploaded to the server in an unencrypted state

[21]

[22] . Content retrieval can steal the visual features (color, texture, shape) of images, while text retrieval can select images from a large number of images based on semantic attribute annotations, resulting in privacy leakage.

[0011] Creating an encrypted index for the retrieved image is a typical solution that can balance the problems of excessive computational complexity and excessive ciphertext storage space brought about by encryption technology

[23]

[24] . For example, Ferreira et al.

[25] proposed a novel encryption scheme for content-based image retrieval with privacy protection (PCBIR), which processes the color and texture information of an image separately, encrypts the texture information with a random encryption scheme to protect the image content, and processes the color information with a deterministic encryption scheme to support retrieval under ciphertext. Xia et al.

[26] proposed a secure K-nearest neighbor (KNN) algorithm to create an encrypted index for the retrieved image, which enables the server to effectively arrange the search results without increasing the communication burden. Locality sensitive hashing (LSH) technology is used to cluster similar images, thereby improving retrieval efficiency. Xu et al.

[27] first proposed a PCBIR scheme based on orthogonal decomposition, in which the image is decomposed into two parts through orthogonal decomposition, and encryption and feature extraction are performed separately. Unlike other schemes, this scheme does not have special requirements for the encryption algorithm.

[0012] (3) Key privacy protection technologies in machine learning

[0013] Machine learning methods can accurately infer personal information or predict personal behavior, and if such data is used inappropriately, it will have very serious consequences

[28] .

[0014] Sweeney's anonymization method is a privacy protection technique at the data level. Anonymization technology protects data privacy by obfuscating the key parts of confidential data. However, data accessors have many ways, such as using big data technology and other feature information to deduce the corresponding data of a person in the data table, which seriously affects the accuracy of the data and can analyze privacy information

[29] . Machine learning algorithms operate on encrypted data through encryption technology. They have the advantages of good privacy protection and data accuracy. They can achieve the purpose of making the original data inaccessible and protect data privacy. However, their encryption technology has long computation time and high energy consumption

[30]

[31] . The encryption technology algorithm needs to be further improved. In addition, the general machine learning strategy PATE meets the differential privacy standard. It scrambles data and labels at the same time, and then uses training data and integrated algorithms to train an external model that can recognize scrambled data. This algorithm requires integrated training of scrambled data, which is very complicated

[32]

[33] . Deep learning-based image privacy perception methods

[34] accurately distinguish privacy images and locate privacy regions in images through attention mechanisms and other means, and selectively protect them. However, it is not applicable to applications where privacy information and primary task information overlap. Summary of the Invention

[0015] The purpose of this invention is to provide a face image privacy protection method for edge intelligence that protects task privacy from being stolen by edge servers, and distributes tasks across various nodes without increasing computational load. This invention proposes a privacy protection strategy based on edge-distributed computing and feature analysis with privacy veil removal at the device end; it studies the model training and execution method of edge-layered privacy veil removal, performing veil removal near the device end to effectively retain privacy information as close to the device as possible, ensuring the effectiveness of privacy protection in edge computing scenarios without increasing computational load and reducing the amount of transmitted data. Using a feature extractor, the gender information and sensitive identity information of individuals, which are intertwined (overlapping) in the original data domain, are separated. The processing of both sensitive information is considered, and the extracted intermediate layer features need to retain necessary information about the gender of the main task while protecting the sensitive identity information.

[0016] Based on the purpose of privacy veil removal, the network is divided into three modules: a privacy veil removal feature extraction module (with identity feature veil removal and gender feature extraction capabilities), an attribute feature analysis and inference module, and an attribute classification module. These modules operate on different entities. The privacy veil removal feature extraction module, attribute feature analysis and inference module, attribute classification module, and performance evaluation and loss function module are deployed on different devices and edge servers. To protect privacy and offload edge computing, the privacy veil removal feature extraction module is placed closer to the user device, on mobile devices, user devices, and processors near the user; the inference module is placed farther from the user device and on edge servers.

[0017] like Figure 1 As shown, when image information is input from the user end to the privacy culling feature extraction module, after feature extraction, it is input into the two branches of attribute feature analysis—the identity reasoning module and the gender reasoning module, and then input into the identity and gender classification module. After processing, it is confirmed whether the purpose of privacy culling

[35]

[36] and gender recognition has been achieved, and then it is decided whether to output to the destination end. If not, it returns to the privacy culling feature extraction module to repeat the process.

[0018] A privacy-preserving fine-tuning model for face edge intelligence applications is characterized by comprising the following modules: a privacy-eliminating feature extraction module, an attribute feature analysis and inference module, an attribute classification module, and a performance evaluation and loss function model module. The privacy-eliminating feature extraction module performs feature extraction before branch training of the network, playing a crucial role in feature analysis for the attribute feature analysis and inference module. Within the attribute feature analysis and inference module, the gender recognition network module and the identity recognition network module are two branches. After the feature extraction layer module outputs features during network training, fine-tuning and freezing adjustments are performed during training according to task requirements to achieve the inference objective of this branch. The attribute classification module performs the final classification work, accurately classifying gender and obfuscating identity according to the target.

[0019] The higher the network layer, the more explicitly the network targets the main task, while also striving to eliminate sensitive information. The privacy-preserving feature extraction module consists of Batch Normalization (BN) layers, convolutional layers, and pooling layers. Assume the model network's input is a dataset with image gender and image identity classification labels. Image input x∈X, where X represents the image input space, and image classification label y∈Y, where Y (Y={1,2,3,...,k}) represents the image classification label space. When performing gender classification... s When classifying, k=2, and when performing identity y d During classification, the value of k depends on the number of specific identities; here, gender and identity share the same dataset X(x, y). During the training phase, the preprocessed square face image x is processed by f = G. f (x;θ f After the mapping transformation, a D-dimensional feature vector is formed, completing the initial feature extraction and generating a feature map, i.e., f∈R. D .

[0020] In order to achieve the purpose of privacy culling, the gradient flip layer GRL is used in the privacy culling feature extraction module. What the gradient flip layer GRL does is to multiply the error passed to this layer by a negative number (), so that the training objectives of the network before and after GRL are opposite, thus achieving the effect of adversarial [7].

[0021] This is not a constant, but rather a change from 0 to 1, as shown in equation (1).

[0022]

[0023] Here, p is as shown in equation (2).

[0024]

[0025] In equation (2), epoch ranges from 0 to epoch. maxThe values ​​of `i` increase sequentially, where `batch_size` is the batch size, and `i` increases from 0 as training progresses. Here, `γ` is a hyperparameter set to 10; as training continues, `p` changes from 0 to 1, representing the current training step count divided by the total training steps.

[0026] The output of the privacy-preserving feature extraction network in a research method for privacy protection in face edge intelligence applications is divided into two branches: a gender attribute feature analysis and a classification and prediction network G. s (x;θ s ) and identity attribute feature analysis classification prediction network G d (x;θ d ).

[0027] The gender inference module consists of multiple convolutional neural networks and fully connected layers. The main task features output from the privacy-preserving feature extraction module are fed into this network module for feature extraction. After passing through convolutional and fully connected layers, the goal is to accurately identify gender. If gender attributes cannot be accurately classified, fine-tuning is used to ensure that the loss function in the classification network module is passed for adjustment.

[0028] The identity feature reasoning module consists of a multi-layer convolutional neural network and fully connected layers. The privacy-eliminating features output by the privacy-eliminating feature extraction module are fed into this network, and then processed through G... d (x;θ d Feature extraction is performed to achieve the goal of making the identity unidentifiable. If privacy culling is not achieved, the process returns to the privacy culling feature extraction module and uses the gradient flipping layer (GRL) to continue privacy culling.

[0029] A performance evaluation and loss function model for attribute classification based on privacy protection in face edge intelligence applications is characterized by progressively increasing or decreasing the accuracy of the output category prediction based on the target. Two metrics are used: recall and precision.

[0030] Recall rate represents the probability that a certain attribute is classified into that attribute class, as shown in equation (3).

[0031]

[0032] Where TP represents the number of times this attribute class is predicted to be this attribute. TN represents the number of times this attribute class is predicted to be another attribute class.

[0033] Precision rate is the proportion of the number of items classified as belonging to that attribute class that actually belongs to that attribute class. As shown in equation (4).

[0034]

[0035] FP represents the number of other attribute classes predicted as this attribute class.

[0036] The required loss function is calculated based on the number of classifications and the objective of the trained model. There are two tasks: first, to achieve accurate gender classification; and second, to calculate the loss function for gender. The goal is to minimize the image classification error, while for the gender classification loss Ld, the second task is to remove hidden identity privacy information to maximize the classification error. Figure 1 As shown.

[0037] The attribute classification performance evaluation and loss function model module is applied to the gender model to gradually improve the category prediction accuracy of the input data samples. The cross-entropy loss function is used to calculate the loss function value of gender. The attribute classification performance evaluation and loss function model module is applied to the identity model to gradually reduce the category prediction accuracy of the input sample data samples. The multi-class cross-entropy loss function is used to calculate the loss function value of identity.

[0038] like Figure 2 As shown, the training process of the aforementioned method for fine-tuning the privacy protection portion of a face edge-based intelligent application includes the following steps:

[0039] Step 1: Input the image. First, for the gender recognition function, use the data to train network A and network B. Train for gender. According to the accuracy (4) and loss formula (6), train network A and network B until the accuracy can reach 90%. Then proceed to step 2.

[0040] Step 2: The image is processed through network A and network C, where A is fixed. C is trained according to the accuracy (4) and loss formula (8) to see if the accuracy can be lower than 10% and if the loss function graph increases to a certain fixed value. If the accuracy is higher than a certain threshold of 10%, proceed to step 3. If it cannot be recognized, the experiment ends.

[0041] Step 3: Fix network B and network C, use gradient flipping layer, adjust network A according to formulas (1)(2)(7), and proceed to step 4.

[0042] Step 4: The image passes through Network A and Network C. Similar to Step 2, if the accuracy is higher than a certain threshold of 10%, proceed to Step 3. If it cannot be recognized, proceed to Step 5.

[0043] Step 5: Now, pass the image through network A and network B, fix network A, and train network B according to the accuracy (4) and loss formula (6) to see if the accuracy can reach 90% and if the loss function graph decreases to a certain fixed value. If it can be recognized, end the training; if it cannot be recognized, execute step 1.

[0044] This process continues until gender can be identified, but identity cannot.

[0045] The advantages of the privacy protection fine-tuning model and method for intelligent face edge applications based on this invention are as follows:

[0046] (1) A branch network framework is proposed, which features a shared privacy-preserving feature extraction layer and implements gender and identity attribute reasoning separately. A gradient flipping layer is added to the privacy-preserving feature extraction layer to facilitate the separate processing of the main gender recognition task and the privacy-preserving identity task. The feature extraction layer is configured to run on a secure device close to the user as much as possible to achieve the function of secure privacy-preserving feature extraction.

[0047] (2) A multi-step privacy feature culling method based on adversarial training was designed. By combining the feature inference network to cyclically adjust the feature culling layer and verify the multi-step operation, the feature output of the feature culling network is kept to retain only gender features and not identity features.

[0048] (3) The proposed network architecture contains many training modules, each with a different loss function. Therefore, an independent modular iterative training method is proposed, which uses a specific loss function to update the parameters of a specific module. Compared with the traditional holistic training method, it is less prone to getting trapped in local optima and converges faster. Experiments were conducted on the LFW dataset for evaluation. Experiments demonstrate that the proposed network framework algorithm can effectively achieve privacy occlusion. Attached Figure Description

[0049] Figure 1 This invention relates to a network model and corresponding server location for a fine-tuning model of privacy protection in intelligent face edge applications.

[0050] Figure 2 This is a flowchart of the network training process for a method for fine-tuning the privacy protection portion of a face edge-based intelligent application, as described in this invention.

[0051] Figure 3 This invention relates to network model parameters for a fine-tuning model of the privacy protection part of a face edge intelligent application.

[0052] Figure 4 This is a gender recognition loss curve of a method for fine-tuning the privacy protection part of a face edge intelligent application based on the present invention;

[0053] Figure 5 This is a curve showing the accuracy of gender recognition based on a method for fine-tuning the privacy protection aspect of a smart application based on facial edge recognition, as described in this invention.

[0054] Figure 6 This invention relates to the loss of identity recognition in a method for fine-tuning the privacy protection portion of a face edge-based intelligent application;

[0055] Figure 7 This invention relates to a method for fine-tuning the privacy protection portion of a face edge-based intelligent application to improve the accuracy of identity recognition.

[0056] Figure 8 This invention relates to an identity recognition confusion matrix diagram of a method for fine-tuning the privacy protection portion of intelligent applications based on facial edge detection.

[0057] Figure 9 This invention relates to an identity loss function after a gradient flipping layer, which is a method for fine-tuning the privacy protection part of a face edge-based intelligent application.

[0058] Figure 10 This invention relates to a method for fine-tuning the privacy protection portion of a face edge-based intelligent application, and the accuracy of identity recognition after a gradient flipping layer.

[0059] Figure 11 This invention relates to a method for fine-tuning the privacy protection portion of a face edge-based intelligent application, specifically the gender loss function after a gradient flipping layer.

[0060] Figure 12 This invention relates to a method for fine-tuning the privacy protection aspect of intelligent face edge applications after applying a gradient flipping layer to improve gender accuracy.

[0061] Figure 13 This invention relates to the loss of identity recognition in a method for fine-tuning the privacy protection portion of a face edge-based intelligent application;

[0062] Figure 14 This invention relates to a method for fine-tuning the privacy protection portion of a face edge-based intelligent application to improve the accuracy of identity recognition.

[0063] Figure 15 This is the final identity recognition confusion matrix diagram of the privacy protection fine-tuning method based on face edge intelligent application of the present invention; Detailed Implementation

[0064] The present invention will now be described in further detail with reference to the accompanying drawings.

[0065] A privacy-preserving fine-tuning model for face edge intelligence applications is characterized by comprising the following modules: a privacy-eliminating feature extraction module, an attribute feature analysis and inference module, an attribute classification module, and a performance evaluation and loss function model module. The privacy-eliminating feature extraction module performs feature extraction before branch training of the network, playing a crucial role in feature analysis for the attribute feature analysis and inference module. Within the attribute feature analysis and inference module, the gender recognition network module and the identity recognition network module are two branches. After the feature extraction layer module outputs features during network training, fine-tuning and freezing adjustments are performed during training according to task requirements to achieve the inference objective of this branch. The attribute classification module performs the final classification work, accurately classifying gender and obfuscating identity according to the target.

[0066] The higher the network layer, the more explicitly the network targets the main task, while also striving to eliminate sensitive information. The privacy-preserving feature extraction module consists of Batch Normalization (BN) layers, convolutional layers, and pooling layers. Assume the model network's input is a dataset with image gender and image identity classification labels. Image input x∈X, where X represents the image input space, and image classification label y∈Y, where Y (Y={1,2,3,...,k}) represents the image classification label space. When performing gender classification... s When classifying, k=2, and when performing identity y d During classification, the value of k depends on the number of specific identities; here, gender and identity share the same dataset X(x, y). During the training phase, the preprocessed square face image x is processed by f = G. f (x;θ f After the mapping transformation, a D-dimensional feature vector is formed, completing the initial feature extraction and generating several feature maps, i.e., f∈R. D .

[0067] In order to achieve the purpose of privacy culling, the gradient flip layer GRL is used in the privacy culling feature extraction module. What the gradient flip layer GRL does is to multiply the error passed to this layer by a negative number (), so that the training objectives of the network before and after GRL are opposite, thus achieving the effect of adversarial [7].

[0068] This is not a constant, but rather a change from 0 to 1, as shown in equation (1).

[0069]

[0070] Here, p is as shown in equation (2).

[0071]

[0072] In equation (2), epoch ranges from 0 to epoch. maxThe values ​​of `i` increase sequentially, where `batch_size` is the batch size, and `i` increases from 0 as training progresses. Here, `Y` is a hyperparameter set to 10; as training continues, `p` changes from 0 to 1, representing the current training step count divided by the total training steps.

[0073] The output of the privacy-preserving feature extraction network in a research method for privacy protection in face edge intelligence applications is divided into two branches: a gender attribute feature analysis and a classification and prediction network G. s (x;θ s ) and identity attribute feature analysis classification prediction network G d (x;θ d ).

[0074] The gender inference module consists of multiple convolutional neural networks and fully connected layers. The main task features output from the privacy-preserving feature extraction module are fed into this network module for feature extraction. After passing through convolutional and fully connected layers, the goal is to accurately identify gender. If gender attributes cannot be accurately classified, fine-tuning is used to ensure that the loss function in the classification network module is passed for adjustment.

[0075] The identity feature reasoning module consists of a multi-layer convolutional neural network and fully connected layers. The privacy-eliminating features output by the privacy-eliminating feature extraction module are fed into this network, and then processed through G... d (x;θ d Feature extraction is performed to achieve the goal of making the identity unidentifiable. If privacy culling is not achieved, the process returns to the privacy culling feature extraction module and uses the gradient flipping layer (GRL) to continue privacy culling.

[0076] A performance evaluation and loss function model for attribute classification based on privacy protection in face edge intelligence applications is characterized by progressively increasing or decreasing the accuracy of the output category prediction based on the target. Two metrics are used: recall and precision.

[0077] Recall rate represents the probability that a certain attribute is classified into that attribute class, as shown in equation (3).

[0078]

[0079] Where TP represents the number of times this attribute class is predicted to be this attribute. TN represents the number of times this attribute class is predicted to be another attribute class.

[0080] Precision rate is the proportion of the number of items classified as belonging to that attribute class that actually belongs to that attribute class. As shown in equation (4).

[0081]

[0082] FP represents the number of other attribute classes predicted as this attribute class.

[0083] The required loss function is calculated based on the number of classifications and the objective of the trained model. There are two tasks: first, to achieve accurate gender classification, with a gender loss L... s The goal is to minimize the image classification error, while the loss L for gender classification is... d The second task is to eliminate the concealment of identity privacy information to maximize the classification error, such as... Figure 1 As shown.

[0084] The attribute classification performance evaluation and loss function model module is applied to the gender model to gradually improve the category prediction accuracy of the input data samples. The cross-entropy loss function is used to calculate the loss function value for gender. The attribute classification performance evaluation and loss function model module is applied to the identity model to gradually reduce the category prediction accuracy of the input sample data samples. The multi-class cross-entropy loss function is used to calculate the loss function value for identity.

[0085] The aforementioned method for fine-tuning the privacy protection portion of intelligent face edge applications includes the following training steps:

[0086] Network flowchart as follows Figure 2 As shown.

[0087] Step 1: Input the image. First, for the gender recognition function, use the data to train network A and network B. Train for gender. According to the accuracy (4) and loss formula (6), train network A and network B until the accuracy can reach 90%. Then proceed to step 2.

[0088] Step 2: The image is processed through network A and network C, where A is fixed. C is trained according to the accuracy (4) and loss formula (8) to see if the accuracy can be lower than 10% and if the loss function graph increases to a certain fixed value. If the accuracy is higher than a certain threshold of 10%, proceed to step 3. If it cannot be recognized, the experiment ends.

[0089] Step 3: Fix network B and network C, use gradient flipping layer, adjust network A according to formulas (1)(2)(7), and proceed to step 4.

[0090] Step 4: The image passes through Network A and Network C. Similar to Step 2, if the accuracy is higher than a certain threshold of 10%, proceed to Step 3. If it cannot be recognized, proceed to Step 5.

[0091] Step 5: Now, pass the image through network A and network B, fix network A, and train network B according to the accuracy (4) and loss formula (6) to see if the accuracy can reach 90% and if the loss function graph decreases to a certain fixed value. If it can be recognized, end the training; if it cannot be recognized, execute step 1.

[0092] Simulation Experiment

[0093] (1) Dataset

[0094] The training and testing datasets for the gender and privacy identity models were obtained from the LFW face recognition dataset. The LFW dataset is an unconstrained natural scene face recognition dataset, consisting of over 13,000 facial images of well-known individuals from around the world in natural scenes, with different orientations, expressions, and lighting conditions. It includes over 5,000 individuals, of whom 1,680 have two or more facial images. Each facial image has a unique identity ID. Identity labels range from 0 to 23. In addition to privacy masking, this paper also focuses on gender recognition, so we need to label the data. For this experiment, 24 individuals were selected from the LFW database for training. Gender is represented by label 0 for males and label 1 for females. The data was then read based on the image location indicated by the label. The dataset selection table is shown in Table 1 below.

[0095] Table 1 LFW Experimental Dataset

[0096]

[0097]

[0098] 80% of the images were randomly selected for training, and the remaining 20% ​​were used for testing. For gender recognition, the output gender comparison was used for verification; for identity privacy masking, the confusion matrix was used for verification.

[0099] (2) Network structure settings

[0100] A privacy-preserving feature extraction network with identity feature removal and gender feature extraction capabilities consists of convolutional layers and pooling layers. The network input image size is 3×224×224, which is then fed into the convolutional network for feature map extraction. In this network, a 5×5 convolutional kernel is first used for image feature extraction, followed by a 2×2 pooling layer to reduce the feature map size. The main purpose of this is to accommodate the initial identity feature extraction while ensuring that the gender feature map is extracted on a smaller scale.

[0101] The gender attribute feature analysis and inference module first extracts image features using a 5×5 convolutional kernel, then reduces the feature map using a 2×2 pooling layer. Next, the extracted features are unfolded in one dimension, followed by three fully connected layer operations with the number of fully connected layers being 120, 84, and 2 respectively. Finally, gender classification is performed. The identity attribute feature analysis and inference module first uses two 3×3 convolutional kernels. This module has a relatively large number of network layers compared to the gender attribute feature analysis and inference module. This is necessary for traditional face recognition to achieve a certain depth of network extraction, followed by privacy culling. Next, the extracted features are unfolded in one dimension, followed by three fully connected layer operations with the number of fully connected layers being 120, 84, and 24 respectively. Finally, gender classification is performed.

[0102] Figure 3 These are the network model parameters, where A is a privacy-preserving feature extraction network (containing a gradient flip layer) with identity feature culling and gender feature extraction capabilities, B is a gender attribute feature inference network, and C is an identity attribute feature inference network. A is used in the gradient flip layer to adjust the features, ultimately enabling both gender and identity privacy feature culling to be achieved simultaneously. B and C can each achieve their respective functions through fine-tuning.

[0103] The parameters for the gender training network, the identity recognition network, and the identity gradient flipping layer are shown in Table 2.

[0104] Table 2 Training Parameter Settings

[0105]

[0106] (3) Results and Analysis

[0107] In the overall training iterations, according to Figure 2 First, in step ①, several training samples are randomly selected from the LFW dataset. These training samples are then randomly shuffled, and gender identification is performed in step ②, achieving the following result: Figure 4 and Figure 5 As shown in the diagram. In the results of process ②, the gender loss value eventually approaches 4, and the gender accuracy approaches 100%. It is clear that the accuracy of gender recognition can reach 90%, and then process ④ can be executed. After 100 iterations of identity recognition ④, the results are as follows. Figure 6 and Figure 7 As shown, the identity recognition accuracy cannot reach below 10%, but rather approaches 60%, indicating that A still contains a large number of identity privacy features. This can be addressed through a confusion function. Figure 8(The confusion matrix was derived from 24 individuals; the vertical axis represents the real identity, and the horizontal axis represents the predicted identity.) As shown, the darker the color along the diagonal lines of the confusion matrix, the higher the accuracy of the data. Figure 8 It can be seen that a slanted trend is still faintly discernible, indicating a leak of identity privacy. Therefore, a gradient flipping layer is needed to adjust network A to eliminate identity privacy features. However, it's also necessary to ensure that the features extracted from network A still include the main task features. Therefore, process ⑤ is required: after passing through the gradient flipping layer, network A performs identity recognition again, as shown... Figure 9 As shown. After specifically applying gradient flipping layers to network A, retraining for identity recognition yields the following results. Figure 9 and Figure 10 As shown in the figure, the training loss is approximately 3.2, and the identity accuracy remains around 3%. Next, we need to further verify whether the main task of gender recognition can be achieved, i.e., process ②. If it can be achieved, the main framework is complete. If it is found that the gender recognition features are weakened due to the gradient flipping layer, further training of gender recognition is required, proceeding to process ⑥, which involves further training networks A and B. After several rounds of training, the final gender recognition image is shown below. Figure 11 and Figure 12 As shown. Verify the loss function graph and accuracy of identity recognition, as follows. Figure 13 The final test results are shown below: For gender recognition verification, 16 images were selected. The true gender and predicted gender are shown in Table 3, and the results are correct. Identity recognition verification, such as the confusion matrix... Figure 15 As shown, for all identities, after privacy culling, the color distribution of the confusion matrix is ​​not a slanted distribution, indicating that privacy culling has been successfully achieved. Figure 11 and Figure 12 As can be seen, while privacy was eliminated, key gender identification features were retained. The specific identities of the identified faces are presented in tabular form, as shown in Table 4.

[0108] Table 3 Comparison of Actual and Predicted Gender Values ​​for Tested Gender

[0109]

[0110] Table 4. Recall and Precision Rates of Face Recognition Tests

[0111]

[0112]

[0113] [1]H.Li,G.Shou,Y.Hu and Z.Guo,″Mobile Edge Computing:Progress andChallenges,″2016 4th IEEE International Conference 0n Mobile Cloud Computing,Services,and Engineering(MobileCloud),2016,pp.83-84,doi:10.1109 / MobileCloud.2016.16.

[0114] [2]J.Ren,G.Yu,Y.He and G.Y.Li,″Collaborative Cloud and Edge Computingfor Latency Minimization,″in IEEE Transactions on Vehicular Technology,vol.68,no.5,pp.5031-5044,May 2019,doi:10.1109 / TVT.2019.2904244.

[0115] [3]S.Joo,H.Kang and J.Kang,″CoSMoS:Cooperative Sky-Ground Mobile EdgeComputing System,″in IEEE Transactions on Vehicular Technology,vol.70,no.8,pp.8373-8377,Aug.2021,doi:10.1109 / TVT.2021.3094584.

[0116] [4]L.Ma,Q.Pei,H.Xiao,H.Li,Z.Li and K.Fan,″Edge Computing EnhancedPrivacy Preserving for Location Based Services,″IEEE INFOCOM 2019-IEEEConference on Computer Communications Workshops(INFOCOM WKSHPS),2019,pp.1-6.

[0117] [5]Xu,Xiaolong,et al.″Joint optimization of offloading utility andprivacy for edge computing enabled IoT.″IEEE Internet of Things Journal 7.4(2019):2622-2629.

[0118] [6]S.-c.S.Cheung,M.U.Rafique and W.-t.Tan,″Privacy-PreservingDistributed Deep Learning with Privacy Transformations,″2018 IEEEInternational Workshop on Information Forensics and Security(WIFS),2018,pp.1-7,doi:10.1109 / WIFS.2018.8630790.

[0119] [7]Z.Chen,L.Li,H.Peng,Y.Liu,H.Zhu and Y.Yang,″An evaluation method ofimage scrambling degree based on pixel distribution,″2018 InternationalSymposium on Security and Privacy in Social Networks and Big Data(SocialSec),2018,pp.206-211,doi:10.1109 / SocialSec.2018.8760371.

[0120] [8]W.Zhang,H.Wang,D.Hou and N.Yu,″Reversible Data Hiding in EncryptedImages by Reversible Image Transformation,″in IEEE Transactions onMultimedia,vol.18,no.8,pp.1469-1479,Aug.2016,doi:10.1109 / TMM.2016.2569497.

[0121] [9]Ganin,Yaroslav,and Victor Lempitsky.″Unsupervised domainadaptation by backpr opagation.″International conference on machinelearning.PMLR,2015.

[0122]

[10] L.Cui,Z.Chen,A.Wang,J.Hu and B.B.Park,″Development of a RobustCooperative Adaptive Cruise Control With Dynamic Topology,″in IEEETransactions on Intelligent Transportation Systems,doi:10.1109 / TITS.2020.3043194.

[0123]

[11] Z.Chen and B.B.Park,″Cooperative Adaptive Cruise Control WithUnconnected Vehicle in the Loop,″in IEEE Transactions on IntelligentTransportation Systems,doi:10.1109 / TITS.2020.3041840.

[0124]

[12] X.CHEN and N.ZHANG,″Layer-wise Adversarial Training Approach toImprove Adversarial Robustness,″2020 International Joint Conference on NeuralNetworks(IJCNN),2020,pp.1-8,doi:10.1109 / IJCNN48605.2020.9206760.

[0125]

[13] M.Javanmardi and T.Tasdizen, "Domain adaptation for biomedical image segmentation using adversarial training," 2018 IEEE 15th InternationalSymposium on Biomedical Imaging (ISBI 2018), 2018, pp.554-558, doi: 10.1109 / ISBI.2018.8363637.

[0126]

[14] H. Akbari-Nodehi and MAMaddah-Ali, "Secure Coded Multi-PartyComputation for Massive Matrix Operations," in IEEE Transactions on Information Theory, vol.67, no.4, pp.2379-2398, April 2021, doi: 10.1109 / TIT.2021.3050853.

[0127]

[15] P.Ah-Fat and M.Huth, "Optimal Accuracy-Privacy Trade-Off for Secure Computations," in IEEE Transactions on Information Theory, vol.65, no.5, pp.3165-3182, May 2019, doi: 10.1109 / TIT.2018.2886458.

[0128]

[16] Yang Qian. A new privacy protection scheme for facial recognition [D]. Central China Normal University, 2020.

[0129]

[17] Cai, Yunlu, and Chunming Tang. "Securely outsourced face recognition under federated cloud environment." 2016 15th International Symposium on Parallel and Distributed Computing (ISPDC). IEEE, 2016.

[0130]

[18] Pearson, Siani, and Andrew Charlesworth. "Accountability as a wayforward for privacy protection in the cloud." IEEE international conference on cloud computing. Springer, Berlin, Heidelberg, 2009.

[0131]

[19] Ma Y, Wu L, Gu X, et al. A secure face-verification scheme based on homomorphic encryption and deep neural networks [J]. IEEE Access, 2017, 5: 16532-16538.

[0132]

[20] Haghighat M, Zonouz S, Abdel-Mottaleb M.CloudID: Trustworthy cloud-based and cross-enterprise biometric identification[J].Expert Systems withApplications, 2015, 42(21):7905-7916.

[0133]

[21] Peng Yuanfan. Research on key technologies for image retrieval with privacy protection [D]. Beijing University of Technology, 2015.

[0134]

[22] H. Akbari-Nodehi and MAMaddah-Ali, "Secure Coded Multi-PartyComputation for Massive Matrix Operations," in IEEE Transactions on Information Theory, vol.67, no.4, pp.2379-2398, April 2021, doi: 10.1109 / TIT.2021.3050853.

[0135]

[23] Z.Huang,M.Zhang and Y.Zhang,″Toward Efficient Encrypted ImageRetriev al in Cloud Environment,″in IEEE Access,vol.7,pp.174541-174550,2019,doi:10.1109 / ACCESS.2019.2957497.

[0136]

[24] Z.Xia,L.Jiang,D.Liu,L.Lu and B.Jeon,″BOEW:A Content-based ImageRetrieval Scheme using Bag-of-Encrypted-Words in Cloud Computing″in IEEETransactions on Services Computing,vol.,no.01,pp.1-1,5555.doi:10.1109 / TSC.2019.2927215

[0137]

[25] Ferreira,Bernardo,et al.″Towards an image encryption scheme withcontent-based image retrieval properties.″Data Privacy Management,AutonomousSpontaneous Security,and Security Assurance.Springer,Cham,2014.311-318.

[0138]

[26] Xia,Zhihua,et al.″EPCBIR:An efficient and privacy-preservingcontent-based image retrieval scheme in cloud computing.″Information Sciences387(2017):195-204.

[0139]

[27] Xu Y, Gong J, Xiong L, et al.A privacy-preserving content-basedimage retrieval method in cloud environment[J].Journal of VisualCommunication and Image Representation, 2017, 43: 164-172.

[0140]

[28] Tang Peng, Huang Zheng, Qiu Weidong. A review of privacy protection technologies in deep learning [J]. Information Security and Communication Confidentiality, 2019, (6): 55-62.

[0141]

[29] S.Abd Razak, NHMohd Nazari and A.Al-Dhaqm.Data AnonymizationUsing Pseudonym System to Preserve Data Privacy[J].IEEE Access, 2020, 8: 43256-43264.

[0142]

[30] Phong LT, Aono Y, Hayashi T, et al.Privacy-Preserving Deep Learning via Additively Homomorphic Encryption[J]. IEEE Transactions on Information Forensics&Security, 2018, 13(5): 1333-1345.

[0143]

[31] GSUehara, A.Spanias and W.Clark, "Quantum Information ProcessingAlgorith ms with Emphasis on Machine Learning," 2021 12th International Conference on Information, Intelligence, Systems & Applications (IISA), 2021, pp.1-11, doi: 10.1109 / IISA52424.2021.9555570.

[0144]

[32] Li Xiaodong, Han Qing, Jin Xin. A secure and efficient face recognition method based on deep learning and homomorphic encryption [P]. CN201810973325.6. 2019-01-04. (School of Electronic Science and Technology, General Office of the CPC Central Committee)

[0145]

[33] Papernot N, Song S, Mironov I, et al.Scalable Private Learning withPATE[C].ICLR2018.

[0146]

[34] Wang H, Zhang Y, You L, et al.Image privacy perception method based on deep learning: USPatent Application 16 / 099, 836[P].2021-7-22.

[0147]

[35] Sun P J.Security and privacy protection in cloud com putting: Discussions and challenges[J]. Journal of Network and Computer Applications, 2020, 160: 102642.

[0148]

[36] Shah M. Reversible information hiding based on Paillier cryptosystem and its extended application in ciphertext domain at the University of Science and Technology of China [D]. University of Science and Technology of China, 2019.

Claims

1. A privacy protection device for intelligent applications based on facial edge detection, characterized in that, It includes the following modules: privacy-preserving feature extraction module, attribute feature analysis and inference module, attribute classification module, and performance evaluation and loss function model module; The privacy-reducing feature extraction module is used to reduce the hidden identity features and extract the gender features. This module is composed of a convolutional neural network. The input to this module is a dataset with image gender classification labels and image identity classification labels. The input x∈X, where X represents the image input space, and the image classification label y∈Y, where Y (Y={1, 2, 3, ..., k}) represents the image classification label space. When performing gender y... s When classifying, k=2, and when performing identity y d During classification, the value of k depends on the number of specific identities; here, gender and identity share the same dataset X(x, y). The image input x, after preprocessing, forms a square face image, which is then processed by f = G. f (x;θ f After the mapping transformation, a D-dimensional feature vector is formed, completing the initial feature extraction and generating several feature maps, i.e., f∈R. D ; To achieve privacy culling, a gradient flip layer (GRL) is used in the privacy culling feature extraction module. The gradient flip layer (GRL) multiplies the error passed to this layer by a coefficient, so that the training objectives of the networks before and after the GRL are opposite, thus achieving an adversarial effect. The coefficient here is not a constant, but changes from 0 to 1, as shown in equation (1). Where p is as shown in equation (2), In equation (2), epoch ranges from 0 to epoch. max The values ​​of i increase sequentially from 0 as training progresses; γ is a hyperparameter set to 10; as training continues, p changes from 0 to 1, representing the current number of training steps / the total number of training steps. The attribute feature analysis and reasoning module includes two branches: a gender feature reasoning module and an identity feature reasoning module. The gender feature reasoning module consists of a multi-layer convolutional neural network and a fully connected layer. It extracts the main task features output by the privacy-eliminating feature extraction module and passes them through convolutional and fully connected layers to achieve accurate gender identification. If the gender attribute cannot be accurately classified, fine-tuning is used to ensure that the identification is adjusted through a loss function. The identity feature reasoning module consists of a multi-layer convolutional neural network and a fully connected layer. It extracts the privacy-eliminating features output by the privacy-eliminating feature extraction module to achieve the goal of not being able to identify the identity. If privacy elimination is not achieved, it will return to the privacy-eliminating feature extraction module and use the gradient flipping layer (GRL) to continue privacy elimination. The attribute classification module includes a gender classification module and an identity classification module to perform the final classification work. According to the goal, gender needs to be accurately classified and identity needs to be confused. The performance evaluation and loss function model module will gradually increase or decrease the category prediction accuracy of the input data sample output according to the goal. Two indicators are used: one is recall and the other is precision. Recall represents the probability that a certain attribute is classified into that attribute class, as shown in equation (3). Where TP represents the number of times the attribute class is predicted to be this attribute; TN represents the number of times the attribute class is predicted to be another attribute class. Precision rate is expressed as the proportion of the number of items classified as belonging to that attribute class to the number of items classified as belonging to that attribute class, as shown in equation (4). Where FP represents the number of other attribute classes predicted as this attribute class; Based on the target of the training model and the number of classifications, the required loss function is calculated. There are two tasks: the first is to achieve accurate gender classification; the second is to eliminate identity privacy information, making it as inaccurate as possible. The performance evaluation and loss function model module is applied to the gender classification module to gradually improve the category prediction accuracy of the input data samples. The cross-entropy loss function is used to calculate the loss function value for gender. The performance evaluation and loss function model module is applied to the identity classification module to gradually reduce the category prediction accuracy of the input sample data samples. The multi-class cross-entropy loss function is used to calculate the loss function value for identity.

2. The privacy protection device for intelligent applications based on facial edge detection according to claim 1, characterized in that, The privacy-reducing feature extraction module, attribute feature analysis and inference module, attribute classification module, and performance evaluation and loss function module are deployed on different devices and edge servers. The privacy-reducing feature extraction module is deployed close to the user device, on mobile devices, user devices, and processors near the user; the inference module is placed far away from the user device and on edge servers.

Images