Model privacy protection method, device and equipment

Abstract

The embodiments of this specification disclose a model privacy protection method, apparatus, and device. The method includes: locating sensitive areas in an original sample; converting the original sample into a desensitized sample by desensitizing the sensitive area; determining an adversarial network model that includes the model to be protected and its corresponding attack model; and performing adversarial training on the adversarial network model using the original sample and the desensitized sample to transform the model to be protected into a privacy-preserving model.

Description

Technical Field

[0001] This specification relates to the field of machine learning technology, and in particular to a model privacy protection method, apparatus, and device. Background Art

[0002] In recent years, significant progress has been made in the development of artificial intelligence (AI), driving the widespread integration of various applications into people's daily lives and production. Examples include facial recognition for clocking in, facial payment, intelligent facial enhancement, and various forms of assisted driving. As the core of AI systems, deep learning models, while performing core operations, also face malicious attacks. Attackers typically attempt to directly access model files to decipher the model's structure and weights, thereby disrupting the system's operational logic and ultimately compromising it. Therefore, protecting model privacy is urgent.

[0003] Currently, model privacy protection can be achieved through post-training encryption and obfuscation methods. These schemes adjust the model structure (obfuscate it) after training and then encrypt the model weights. This prevents attackers from deciphering the model structure and weights, even if they obtain the model file. However, black-box attacks have emerged as a breakthrough against these schemes. These attacks can be performed by deriving an alternative model without explicitly obtaining the model file.

[0004] Based on this, a more effective model privacy protection scheme is needed. Summary of the Invention

[0005] One or more embodiments of this specification provide a model privacy protection method, apparatus, device, and storage medium to solve the following technical problem: a more effective model privacy protection solution is needed.

[0006] To solve the above technical problems, one or more embodiments of this specification are implemented as follows:

[0007] One or more embodiments of this specification provide a model privacy protection method, including:

[0008] Locate sensitive areas in the original sample;

[0009] Converting the original sample into a desensitized sample by desensitizing the sensitive area;

[0010] Determine an adversarial network model that includes the model to be protected and its corresponding attack model;

[0011] The adversarial network model is trained adversarially using the original sample and the desensitized sample, so that the model to be protected is transformed into a model with privacy protection.

[0012] One or more embodiments of this specification provide a model privacy protection device, including:

[0013] Sensitive area positioning module, locates sensitive areas in the original sample;

[0014] A sensitive area desensitization module, which converts the original sample into a desensitized sample by desensitizing the sensitive area;

[0015] An adversarial network determination module determines an adversarial network model including a model to be protected and its corresponding attack model;

[0016] The sensitive adversarial training module uses the original samples and the desensitized samples to perform adversarial training on the adversarial network model, so as to transform the model to be protected into a model with privacy protection.

[0017] One or more embodiments of this specification provide a model privacy protection device, including:

[0018] at least one processor; and,

[0019] a memory communicatively connected to the at least one processor; wherein,

[0020] The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to:

[0021] Locate sensitive areas in the original sample;

[0022] Converting the original sample into a desensitized sample by desensitizing the sensitive area;

[0023] Determine an adversarial network model that includes the model to be protected and its corresponding attack model;

[0024] The adversarial network model is trained adversarially using the original sample and the desensitized sample, so that the model to be protected is transformed into a model with privacy protection.

[0025] One or more embodiments of this specification provide a non-volatile computer storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured to:

[0026] Locate sensitive areas in the original sample;

[0027] Converting the original sample into a desensitized sample by desensitizing the sensitive area;

[0028] Determine an adversarial network model that includes the model to be protected and its corresponding attack model;

[0029] The adversarial network model is trained adversarially using the original sample and the desensitized sample, so that the model to be protected is transformed into a model with privacy protection.

[0030] At least one of the above-mentioned technical solutions adopted in one or more embodiments of this specification can achieve the following beneficial effects: sensitive areas, such as high-response areas with relatively large gradients, can be located more accurately through this solution, and input desensitization can be performed on such areas. The output of the corresponding area can be fine-tuned through adversarial training, so that the model itself can still perform inference more accurately under desensitization conditions, while making it difficult for black box attacks to detect and determine the remapping effect of desensitization on the mapping relationship between the original input and the model output, thereby reducing the value of the apparent direct mapping relationship between the original input and the model output, thereby effectively reducing the effect of black box attacks on the model and helping to better protect the privacy of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0032] Figure 1 A conceptual diagram of the model privacy protection solution provided for one or more embodiments of this specification;

[0033] Figure 2 A flowchart of a model privacy protection method provided for one or more embodiments of this specification;

[0034] Figure 3 In an application scenario provided by one or more embodiments of this specification, Figure 2 A detailed flow chart of the method;

[0035] Figure 4 A schematic structural diagram of a model privacy protection device provided in one or more embodiments of this specification;

[0036] Figure 5 A schematic diagram of the structure of a model privacy protection device provided in one or more embodiments of this specification. DETAILED DESCRIPTION

[0037] The embodiments of this specification provide a model privacy protection method, apparatus, device, and storage medium.

[0038] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0039] In practical applications, machine learning models (hereinafter referred to as models, deep learning models are a typical type of model) can be used for reasoning in various business fields, such as predicting whether there are anomalies in user transactions, predicting the type of objects in images, segmenting images, predicting time series, and so on.

[0040] As mentioned in the background, attackers attempt to obtain model files to crack the model's structure and weights. This approach is a white-box attack. Although the model has been trained, it cannot correctly infer all samples. Moreover, there is a special type of sample that is particularly noteworthy. Compared with samples that can be correctly inferred, it may not appear significantly different on the surface (for example, when observed with the naked eye), but the model will infer incorrect results, which is a weakness of the model. When attackers obtain the model's structure and weights, it is easier to grasp this weakness of the model, so that they can deliberately construct these special samples to deceive the model and achieve the purpose of the attack.

[0041] Furthermore, in a black-box attack model, there's no need to understand the structure and weights of the target model (i.e., the target of the attack). Instead, methods such as membership attacks are used to determine whether a sample belongs to the training data of the target model. Once the training data is effectively identified through such attacks, model distillation can be used to quickly exploit the model's performance in a black-box environment. For example, a new model can be trained on a small sample using the same training data to attempt to obtain a model with inputs and outputs similar to the target model, potentially exploiting the target model's weaknesses. Currently, there is a lack of solutions to counter black-box attacks.

[0042] The following embodiments provide corresponding solutions to the above problems, which are referred to as the present solution. In order to better illustrate the concept of the present solution, the following analysis is conducted.

[0043] This solution recognizes that in real applications, not only the special samples mentioned above, but also local regions within many samples may exhibit high responses. In other words, even slight changes to such local regions can significantly alter the output, potentially influencing the final inference (usually for classification or regression purposes). These local regions are referred to as sensitive regions (to emphasize their characteristics, they can also be referred to as highly sensitive regions or highly responsive regions). For the model, the knowledge learned from these sensitive regions is particularly valuable as part of the model's privacy. Of course, other data, such as weights and structure, also contribute to the model's privacy. This solution focuses on the privacy of sensitive regions, striving to protect this type of privacy and effectively defend against attacks against the model, especially black-box attacks. The main idea is to locate sensitive areas in the original sample, then desensitize it to obtain desensitized samples, while ensuring that the inference correctness of the desensitized samples can reach a high level. Afterwards, the attack model is simulated and adversarial training is performed to try to eliminate the above-mentioned weaknesses. In this way, the feature mapping relationship that can be learned by black-box attacks is generalized, making it more difficult for black-box attacks to steal the model's capabilities and effectively increasing the difficulty for subsequent attackers to construct targeted attack samples.

[0044] Figure 1 A simplified conceptual diagram of the model privacy protection solution provided for one or more embodiments of this specification. The concept mainly includes the following four parts, which are briefly introduced as examples.

[0045] The first part is high-sensitivity area positioning. Based on the input gradient back propagation algorithm, high-sensitivity area positioning is performed. After obtaining the training data, the high-sensitivity area positioning model can be trained.

[0046] The second part is desensitizing highly sensitive areas. We locate highly sensitive areas in the dataset and perform input desensitization on these areas to reduce the model's reliance on them.

[0047] The third part is adversarial training for highly sensitive areas. Targeting black-box attacks, adversarial training is conducted not only at the sample level but also at the regional level.

[0048] The fourth part is model privacy protection. The trained model has the ability to protect its own privacy and can be deployed online for inference work in the corresponding business field. It is also able to resist black-box attacks.

[0049] Based on this concept, the following is further described in detail.

[0050] Figure 2This is a flowchart of a model privacy protection method provided for one or more embodiments of this specification. This method can be applied to model training and use in various business fields, such as image processing, finance, e-commerce, instant messaging, gaming, and government affairs, to improve model privacy and better defend against attacks. The process can be executed by computing devices in the corresponding fields, and certain input parameters or intermediate results in the process can be manually adjusted to help improve accuracy.

[0051] Figure 2 The process in may include the following steps:

[0052] S202: Locate sensitive areas in the original sample.

[0053] In one or more embodiments of this specification, a model to be protected is pre-determined, hereinafter referred to as the protected model or models (this solution also involves other models, which will be distinguished by other names below). This model needs to be protected against potential attacks, particularly black-box attacks. Raw samples can be used for inference within the protected model, and some of these raw samples can also be used to train other models related to the protected model.

[0054] In one or more embodiments of this specification, the sample is an image sample, and the sensitive area is a portion of the pixels in the image. The following embodiments are mainly illustrated by this typical case. For non-image samples, this solution can also be considered. For example, if the sample is represented as a vector, the sensitive area is, for example, a portion of the dimensions in the vector. The original sample is referred to here to distinguish it from the desensitized sample that follows. The state before desensitization can be referred to as the original state.

[0055] The meaning of sensitive areas has been explained above and will not be repeated here. It should be noted that the sensitive areas mentioned here are usually difficult to determine by direct observation of the image with the naked eye, but rather require the model's response. In facial images, the areas that best represent the user's identity, such as the eyes, mouth, and nose, are often called key areas (this is the user's privacy and does not reflect the privacy of the model. This solution focuses on the privacy of the model). These key areas can be determined by direct observation with the naked eye. Therefore, it is important to note that these key areas should not be confused with the sensitive areas in this solution. The definitions and corresponding performances of the two are different.

[0056] In one or more embodiments of the present specification, based on the definition of sensitive areas, the results of inferring the original sample in the model and the results of fine-tuning the local area of ​​the original sample and then inferring in the model can be observed, and by comparing the degree of difference between the two, it can be determined whether the local area is a sensitive area.

[0057] To improve efficiency, we also provide a gradient representation based on the original samples to determine whether the model is highly responsive to a certain area, and thus whether the area is sensitive. Furthermore, we can build and train another machine learning model based on this idea to locate sensitive areas more efficiently. This is called the Sensitive Area Localization Model and will be explained in detail later.

[0058] S204: Convert the original sample into a desensitized sample by desensitizing the sensitive area.

[0059] In one or more embodiments of this specification, desensitization is performed on sensitive areas to reduce the responsiveness of the area. Of course, the purpose is not limited to this. It is also necessary to make the subsequent output of the desensitized sample in the model less variable than the output of the original sample in the model (i.e., fine-tune the output), so that the model can still correctly infer the desensitized sample.

[0060] Similar to the concept of the sensitive area positioning model, another machine learning model can be built and trained to desensitize sensitive areas more efficiently. This is called the desensitization model and will be explained in detail later.

[0061] S206: Determine an adversarial network model including the model to be protected and its corresponding attack model.

[0062] In one or more embodiments of this specification, the attack model simulates attacks in real applications, thereby training the protected model's ability to resist attacks. The attack model's desired goal is to input original samples and obtain output consistent with the protected model, thereby achieving a black-box attack. Of course, this solution does not intend to achieve the desired outcome for the attack model. Therefore, adversarial training is used to counter this desired goal while simultaneously striving to achieve the protected model's original goal: to correctly infer the original samples and their desensitized counterparts.

[0063] S208: Using the original sample and the desensitized sample, perform adversarial training on the adversarial network model to transform the model to be protected into a model with privacy protection.

[0064] In one or more embodiments of this specification, when samples are desensitized, the knowledge actually learned by the protected model is the mapping relationship between the desensitized samples and the correct output corresponding to the original samples. For the resulting privacy-protected model, the original sample undergoes a dual mapping relationship of mapping to the desensitized sample and then remapping from the desensitized sample to the correct output, and the vulnerability of high response in sensitive areas is protected. For the attack model, it attempts to map directly from the original sample to the correct output (called direct mapping), which makes it difficult to learn the features corresponding to the high response, and the adversarial training itself also reduces the possibility that "direct mapping" can correctly replace "dual mapping", thereby helping to achieve model privacy protection.

[0065] In one or more embodiments of this specification, in actual applications, it is also possible to consider not being specific to the regional granularity (for example, not distinguishing different areas in the sample, not considering sensitive areas), but only conducting adversarial training at the sample granularity. In this way, the ability of black box attacks to steal models can be prevented to a certain extent. However, this method is more suitable for classification models with more training samples, and for non-classification models (mainly regression models), such as image segmentation models, face detection models, etc., the effect is poor. The above-mentioned finer-grained solution based on sensitive areas can achieve better results for non-classification models.

[0066] In one or more embodiments of this specification, after training a privacy-preserving model, the model can be used in actual business operations. For example, the model can be deployed on a device to perform a predetermined task, such as image segmentation, face detection, or object classification, depending on the business objectives of the model to be protected.

[0067] The actual usage process is similar to the training process, except that there is no label guidance. For example, after the protected model is converted to a privacy-preserving model, the model can be used to process the object to be inferred, locate sensitive areas in the current object to be inferred (the positioning process can be performed by the sensitive area positioning model mentioned above), desensitize the sensitive areas of the object to be inferred, and convert the object to be inferred into a desensitized object (the desensitization process can be performed by the desensitization model mentioned above). Then, the privacy-preserving model is used to infer the desensitized object to obtain the inference result. In this way, the accuracy of inference can be improved while taking into account the security of the model.

[0068] pass Figure 2Method, sensitive areas, such as high-response areas with relatively large gradients, can be located more accurately through this solution, and input desensitization can be performed on such areas. Through adversarial training, the output of the corresponding area can be fine-tuned, so that the model itself can still perform inference more accurately under desensitization conditions, while making it difficult for black-box attacks to detect and determine the remapping effect of desensitization on the mapping relationship between the original input and the model output, thereby reducing the value of the apparent direct mapping relationship between the original input and the model output, thereby effectively reducing the effect of black-box attacks on the model and helping to better protect the privacy of the model.

[0069] based on Figure 2 This specification also provides some specific implementation plans and extension plans of the method, which will be described below.

[0070] The optional sensitive area positioning model and desensitization model were mentioned above. The following is an exemplary explanation. These two models can be implemented separately or integrated with the model to be protected for end-to-end training.

[0071] As mentioned earlier, sensitive areas are difficult to observe directly with the naked eye. Therefore, a slightly more complex solution can be used to obtain a small batch of training samples with appropriate quantity and high accuracy to assist in effective training. This helps to balance the overall implementation efficiency of the solution.

[0072] A sensitive area positioning model can be pre-built. First training data can be generated based on the gradient representation of a first original sample (which can be a sample other than the original sample in S202) (e.g., the gradient representation in the model to be protected or a model with a similar structure). The first training data can then be used to train the sensitive area positioning model. Subsequently, when S202 is executed, the trained sensitive area positioning model can be used to locate sensitive areas in the second original sample (i.e., the original sample in S202).

[0073] The gradient is usually positively correlated with the degree of response. Therefore, the gradient size corresponding to each area in the first original sample can be analyzed to determine which areas have relatively higher responses and are more sensitive. Take the image sample as an example. Specifically, the image sample as the first original sample is input into the protected model for processing. Through gradient backpropagation, the gradient corresponding to each pixel of the image sample is obtained. The sensitivity of the corresponding pixel is determined according to the gradient. Then, according to the sensitivity of multiple pixels in the same area, the sensitive area is determined in multiple areas of the image sample. The image sample (which can be used as the model input) and the sensitive area determined for it (which can be used as a label) are used as the first training data. The size of a single area can be specified in advance, and the sensitivity of the pixels in each area can be comprehensively considered (for example, by averaging or taking the maximum value). If the sensitivity of a certain area is relatively the highest as a whole, the area is determined as a sensitive area.

[0074] In one or more embodiments of this specification, desensitization presents conflicting objectives: on the one hand, it is desirable to maximize the difference between samples before and after desensitization, while on the other hand, it is desirable for both samples before and after desensitization to produce as consistent output as possible within the protected model. For such situations, this solution considers adversarial training to more specifically balance these objectives.

[0075] For example, a desensitization model is constructed in advance, and adversarial training is performed on the desensitization model based on the desensitization loss of the second training data in the desensitization model and the corresponding original target loss. After that, when executing S204, the trained desensitization model can be used to desensitize the sensitive areas. The desensitization loss reflects the first goal in the previous paragraph, and it is expected that the desensitization loss is as large as possible. The corresponding original target loss reflects the second goal in the previous paragraph, and it is expected that the corresponding original target loss is as small as possible. Based on this, the second training data can be input into the model to be protected and processed and output to obtain the first output result. After the second training data is desensitized by the desensitization model, it is input into the model to be protected and processed and output to obtain the second output result. The corresponding original target loss is determined based on the difference between the first output result and the second output result. During adversarial training, an overall goal can be constructed based on these two goals. The overall goal is used to control the convergence of the overall training state, and then training is performed for these three goals at the same time.

[0076] In one or more embodiments of this specification, when performing adversarial training on an adversarial network model, the approach is similar to the training of the desensitized model described above, but the specific goals of the confrontation are different. Compared to the differences before and after desensitization, the focus here is more on the effectiveness of resisting attacks. On the one hand, it is expected that samples before and after desensitization can run as consistent outputs as possible in the protected model. On the other hand, it is expected that the original samples will run erroneous outputs in the attack model. The attack model is intended to simulate the black box capability of stealing the protected model. Therefore, these two goals are somewhat contradictory.

[0077] Based on this approach, desensitized samples are fed into the protected model contained within the adversarial network model for processing, while original samples are fed into the attack model contained within the adversarial network model for processing, attempting to simulate the output of the protected model. Based on the outputs of the protected and attack models, adversarial training is performed on the adversarial network model. Specifically, appropriate losses are constructed to represent (or even enhance the representation through additional losses) the objectives described in the previous paragraph. For example, the original target loss for the protected model is determined based on the difference between the protected model's output for the desensitized samples and its output for the corresponding labels or original samples. The black-box attack loss for the attack model is determined based on the difference between the protected model's output for the desensitized samples and its output for the attack model. Then, a total loss is constructed based on these two losses, which represent a certain degree of mutual adversarial relationship within this total loss. Furthermore, adversarial training is performed on the adversarial network model with the goal of minimizing the original target loss and maximizing the black-box attack loss (essentially, the training objectives for the local components of the adversarial network model, namely, the training objectives for the protected and attack models, respectively). To ensure convergence of the adversarial network model, the overall adversarial training objective is to minimize the total loss. For example, desensitization loss can be added to the total loss to help the black box attack loss fight against the original target loss.

[0078] It should be noted that in addition to being effective against black-box attacks, the adversarial training method in the previous paragraph may also be effective against white-box attacks, especially when the desensitized model is relatively secure and its structure and weights are difficult to steal. Based on this, a white-box attack loss can also be constructed from the total loss in the previous paragraph to compete with the original target loss. In this case, the structure and weights of the model to be protected and the attack model can be forced to be at least partially the same without fully exposing the desensitized model, thereby simulating a white-box attack scenario in which the attack model attempts to crack or even partially cracks the model file to be protected. Under this condition, the white-box attack loss is determined, for example, based on the difference between the output of the model to be protected for the desensitized sample and the output of the attack model.

[0079] The above describes the present solution in detail. Based on these descriptions, more intuitively, one or more embodiments of this specification further provide an application scenario. Figure 2 A detailed process of the method in order to help better understand this solution, the detailed process is as follows Figure 3 shown.

[0080] exist Figure 3 In the application scenario, the above-mentioned model to be protected is a non-classification model. Specifically, taking the image segmentation model as an example, the above-mentioned original target loss is, for example, the segmentation loss.

[0081] Figure 3 The process in may include the following steps:

[0082] S302: Train the segmentation model through non-adversarial training.

[0083] The input of the segmentation model is an image, and the output is a segmentation map of the image.

[0084] S304: Calculate the corresponding gradient of each input pixel as sensitivity to obtain a sensitivity map.

[0085] S304 can be implemented using previous training samples or new training samples. For example, the following formula is used for calculation:

[0086]

[0087] Among them, sen represents sensitivity, This represents the partial derivative of the loss with respect to the input, i.e., the gradient corresponding to that input. This formula is illustrative only, and there are many other implementations, such as adding an additional bias constant term to the formula, increasing the multiplication coefficient of the constant, etc. The key is to ensure a positive correlation between sensitivity and gradient.

[0088] 306: Calculate the designated area (for example, each area is pre-designated as a 32*32 area) with the largest sum of sensitivities in the sensitivity map as the sensitive area.

[0089] 308: Using the image with the sensitive area located as training data, a sensitive area location model is trained.

[0090] For example, a sensitive area positioning model can be built in advance based on the detection model framework, and then the model can be trained using the obtained training data.

[0091] 310: Define a loss function for the desensitization model, including desensitization loss and segmentation loss, and use some samples that can locate sensitive areas to train the desensitization model and obtain some desensitized samples.

[0092] The structure of the desensitization model is, for example, UNET, whose input is the original image and output is the original image with the sensitive area desensitized.

[0093] Its exemplary loss function is as follows:

[0094]

[0095] Among them, the subscript ori represents the original, the subscript tm represents the desensitization, img represents the image, pred represents the output of the segmentation model, and Loss tm Represents the total loss of the desensitization model. The first part of the total loss (i.e. ) represents the desensitization loss, that is, the difference between the images before and after desensitization. The second part of the total loss (i.e., pred ori -pred tm ) represents the segmentation loss, which is the difference between the output of the segmentation model for the image before and after desensitization.

[0096] S312: Construct an adversarial training model including a segmentation model and an attack model, define segmentation loss and black-box attack loss for the adversarial training model, and use desensitized samples and their corresponding original samples to perform adversarial training on the adversarial training model.

[0097] The adversarial training model consists of a segmentation model and an attack model. The attack model uses the input and output of the segmentation model to try to steal the capabilities of the segmentation model.

[0098] The input of the segmentation model is the original image (and the desensitized original image), and the output is the corresponding segmentation map; the input of the attack model is the original image, and the output is the output of the simulated segmentation model.

[0099] Its exemplary loss function is as follows:

[0100]

[0101] Among them, the subscript gt represents the label, Loss seg Represents the total loss of the adversarial training model. The first part of the total loss (i.e. ) represents the segmentation loss (this segmentation loss is different from the segmentation loss in the previous paragraph), for example, it is the difference between the output of the segmentation model for the desensitized image and the corresponding label. The second part of the total loss (i.e. ) represents the black box attack loss, that is, the difference between the output of the segmentation model for the image before desensitization (original image) or the image after desensitization and the output of the attack model for the original image.

[0102] Based on the above model structure and loss function, training is performed until the model converges.

[0103] S314: Deploy the segmentation model included in the trained adversarial training model on the device side.

[0104] S316: After deployment, for a new image to be segmented, the sensitive area positioning model is used to locate the sensitive area in the image to be segmented.

[0105] S318: Using the desensitization model, annotate the sensitive areas of the image to be segmented to obtain a desensitized image;

[0106] S320: Using the segmentation model, segmentation reasoning is performed on the desensitized image to obtain a segmentation map for guiding the segmentation of the image to be segmented.

[0107] Based on the same idea, one or more embodiments of this specification also provide devices and apparatuses corresponding to the above methods, such as Figure 4 、 Figure 5 shown.

[0108] Figure 4 A schematic diagram of the structure of a model privacy protection device provided in one or more embodiments of this specification, the device comprising:

[0109] Sensitive area positioning module 402, locating sensitive areas in the original sample;

[0110] The sensitive area desensitization module 404 converts the original sample into a desensitized sample by desensitizing the sensitive area;

[0111] The adversarial network determination module 406 determines an adversarial network model including a model to be protected and its corresponding attack model;

[0112] The sensitive adversarial training module 408 uses the original samples and the desensitized samples to perform adversarial training on the adversarial network model, so as to transform the model to be protected into a model with privacy protection.

[0113] Optionally, it also includes:

[0114] A first training module 410 generates first training data based on the gradient representation of the first original sample before locating the sensitive area in the original sample, and trains a sensitive area positioning model using the first training data;

[0115] The sensitive area positioning module 402 locates the sensitive area in the second original sample using the sensitive area positioning model.

[0116] Optionally, the first original sample is an image sample;

[0117] The first training module 410 inputs the image sample into the to-be-protected model for processing, and obtains the gradient corresponding to each pixel of the image sample through gradient back propagation;

[0118] Determining a sensitivity corresponding to the pixel according to the gradient;

[0119] According to the sensitivities of the plurality of pixels in the same area, a sensitive area is determined in the plurality of areas in the image sample, and the image sample and the sensitive area determined therefor are used as first training data.

[0120] Optionally, it also includes:

[0121] A second training module 412 performs adversarial training on the desensitization model based on the desensitization loss of the second training data in the desensitization model and the corresponding original target loss before desensitizing the sensitive area;

[0122] The sensitive area desensitization module 404 performs desensitization on the sensitive area using the desensitization model.

[0123] Optionally, the second training module 412 inputs the second training data into the model to be protected, processes the data, and then outputs the data to obtain a first output result before performing adversarial training on the desensitization model.

[0124] After desensitizing the second training data through the desensitization model, the second training data is input into the model to be protected for processing and then output to obtain a second output result;

[0125] The corresponding original target loss is determined according to a difference between the first output result and the second output result.

[0126] Optionally, the sensitive adversarial training module 408 inputs the desensitized sample into the model to be protected included in the adversarial network model for processing;

[0127] Inputting the original sample into the attack model included in the adversarial network model for processing to attempt to simulate the output of the model to be protected;

[0128] According to the outputs of the protected model and the attack model, adversarial training is performed on the adversarial network model.

[0129] Optionally, the sensitive adversarial training module 408 determines the original target loss corresponding to the to-be-protected model and the black-box attack loss corresponding to the attack model according to the outputs of the to-be-protected model and the attack model respectively;

[0130] The adversarial network model is subjected to adversarial training according to the original target loss and the black-box attack loss.

[0131] Optionally, the sensitive adversarial training module 408 determines the original target loss corresponding to the model to be protected based on the difference between the output of the model to be protected for the desensitized sample and the corresponding label or the output for the original sample;

[0132] The black box attack loss corresponding to the attack model is determined based on the difference between the output of the to-be-protected model for the desensitized sample and the output of the attack model.

[0133] Optionally, the sensitive adversarial training module 408 performs adversarial training on the adversarial network model with minimizing the original target loss and maximizing the black-box attack loss as adversarial training objectives.

[0134] Optionally, a privacy protection reasoning module 414 is also included;

[0135] The sensitive area positioning module 402 locates the sensitive area in the current object to be inferred after the model to be protected is converted into a model with privacy protection;

[0136] The sensitive area desensitization module 404 desensitizes the sensitive area of ​​the object to be inferred, thereby converting the object to be inferred into a desensitized object;

[0137] The privacy protection reasoning module 414 performs reasoning on the desensitized object through the privacy protection model to obtain a reasoning result.

[0138] Optionally, the model to be protected is a non-classified model.

[0139] Optionally, the non-classification model includes an image segmentation model, which, after being converted into the privacy-preserving model, is used to predict image segmentation results and resist black box attacks against the image segmentation model itself.

[0140] Figure 5 A schematic diagram of the structure of a model privacy protection device provided in one or more embodiments of this specification, the device comprising:

[0141] at least one processor; and,

[0142] a memory communicatively connected to the at least one processor; wherein,

[0143] The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to:

[0144] Locate sensitive areas in the original sample;

[0145] Converting the original sample into a desensitized sample by desensitizing the sensitive area;

[0146] Determine an adversarial network model that includes the model to be protected and its corresponding attack model;

[0147] The adversarial network model is trained adversarially using the original sample and the desensitized sample, so that the model to be protected is transformed into a model with privacy protection.

[0148] The processor and the memory may communicate with each other via a bus, and the device may further include an input / output interface for communicating with other devices.

[0149] Based on the same idea, one or more embodiments of this specification further provide a non-volatile computer storage medium corresponding to the above method, storing computer-executable instructions, wherein the computer-executable instructions are configured to:

[0150] Locate sensitive areas in the original sample;

[0151] Converting the original sample into a desensitized sample by desensitizing the sensitive area;

[0152] Determine an adversarial network model that includes the model to be protected and its corresponding attack model;

[0153] The adversarial network model is trained adversarially using the original sample and the desensitized sample, so that the model to be protected is transformed into a model with privacy protection.

[0154] In the 1990s, technological improvements could be clearly distinguished as either hardware improvements (for example, improvements to circuit structures like diodes, transistors, and switches) or software improvements (improvements to process flows). However, with the advancement of technology, many process flow improvements today can now be considered direct improvements to hardware circuit structures. Designers almost always create the corresponding hardware circuit structure by programming the improved process flow into the hardware circuit. Therefore, it cannot be said that a process flow improvement cannot be implemented using hardware modules. For example, a programmable logic device (PLD), such as a field programmable gate array (FPGA), is an integrated circuit whose logical function is determined by user programming. Designers can "integrate" a digital system on a PLD through their own programming, without having to hire a chip manufacturer to design and manufacture a dedicated integrated circuit chip. Moreover, nowadays, instead of manually fabricating integrated circuit chips, this programming is mostly done using "logic compiler" software. This is similar to the software compiler used when developing programs. Before compilation, the original code must also be written in a specific programming language, called a hardware description language (HDL). There is not just one HDL, but many, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art will also understand that by simply programming the method flow in one of these hardware description languages ​​and then programming it into an integrated circuit, a hardware circuit that implements the logic method flow can be easily obtained.

[0155] The controller can be implemented in any suitable manner. For example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that in addition to implementing the controller in a purely computer-readable program code format, the controller can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software modules that implement the method and structures within the hardware component.

[0156] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0157] For the convenience of description, the above devices are described as being divided into various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0158] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, the embodiments of this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0159] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0160] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0161] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0162] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0163] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0164] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0165] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0166] This specification may be described in the general context of computer-executable instructions, such as program modules, executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media, including storage devices.

[0167] The various embodiments in this specification are described in a progressive manner. Similar portions between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from the other embodiments. In particular, the device, apparatus, and non-volatile computer storage medium embodiments are generally similar to the method embodiments, so their descriptions are relatively simplified. For relevant details, refer to the descriptions of the method embodiments.

[0168] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0169] The foregoing description is merely one or more embodiments of this specification and is not intended to limit this specification. It will be apparent to those skilled in the art that various modifications and variations may be made to one or more embodiments of this specification. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of one or more embodiments of this specification are intended to be within the scope of the claims of this specification.

Claims

1. A model privacy protection method, comprising: Locate sensitive areas in the original sample; Converting the original sample into a desensitized sample by desensitizing the sensitive area; Determine an adversarial network model that includes the model to be protected and its corresponding attack model; The adversarial network model is trained adversarially using the original sample and the desensitized sample to transform the model to be protected into a model with privacy protection, specifically including: inputting the desensitized sample into the model to be protected contained in the adversarial network model for processing; inputting the original sample into the attack model contained in the adversarial network model for processing to attempt to simulate the output of the model to be protected; determining the original target loss corresponding to the model to be protected based on the difference between the output of the model to be protected for the desensitized sample and the corresponding label or the output for the original sample; determining the black box attack loss corresponding to the attack model based on the difference between the output of the model to be protected for the desensitized sample and the output of the attack model; and performing adversarial training on the adversarial network model based on the original target loss and the black box attack loss.

2. The method according to claim 1, before locating the sensitive area in the original sample, the method further comprises: Generating first training data according to the gradient performance of the first original sample, and training a sensitive area positioning model using the first training data; The step of locating the sensitive area in the original sample specifically includes: The sensitive area positioning model is used to locate the sensitive area in the second original sample.

3. The method according to claim 2, wherein the first original sample is an image sample; Generating first training data according to the gradient performance of the first original sample specifically includes: Inputting the image sample into the model to be protected for processing, and obtaining the gradient corresponding to each pixel of the image sample through gradient back propagation; Determining a sensitivity corresponding to the pixel according to the gradient; According to the sensitivities of the plurality of pixels in the same area, a sensitive area is determined in the plurality of areas in the image sample, and the image sample and the sensitive area determined therefor are used as first training data.

4. The method according to claim 1, before desensitizing the sensitive area, the method further comprises: Performing adversarial training on the desensitization model according to the desensitization loss of the second training data in the desensitization model and the corresponding original target loss; The desensitization of the sensitive area specifically includes: The desensitization model is used to perform desensitization on the sensitive area.

5. The method according to claim 4, before performing adversarial training on the desensitization model, the method further comprises: Inputting the second training data into the model to be protected, processing it, and then outputting it to obtain a first output result; After desensitizing the second training data through the desensitization model, the second training data is input into the model to be protected for processing and then output to obtain a second output result; The corresponding original target loss is determined according to a difference between the first output result and the second output result.

6. The method according to claim 1, wherein the adversarial training of the adversarial network model based on the original target loss and the black-box attack loss specifically comprises: The adversarial network model is subjected to adversarial training with minimizing the original target loss and maximizing the black box attack loss as adversarial training objectives.

7. The method according to claim 1, wherein after the model to be protected is converted into a model with privacy protection, the method further comprises: Locate sensitive areas in the current object to be inferred; Converting the object to be inferred into a desensitized object by desensitizing the sensitive area of ​​the object to be inferred; The desensitized object is inferred using the privacy-preserving model to obtain an inference result.

8. The method according to any one of claims 1 to 7, wherein the model to be protected is a non-classified model.

9. The method as claimed in claim 8, wherein the non-classification model includes an image segmentation model, and after being converted into the privacy-preserving model, the image segmentation model is used to predict image segmentation results and resist black box attacks against the image segmentation model itself.

10. A model privacy protection device, comprising: Sensitive area positioning module, locates sensitive areas in the original sample; A sensitive area desensitization module, which converts the original sample into a desensitized sample by desensitizing the sensitive area; An adversarial network determination module determines an adversarial network model including a model to be protected and its corresponding attack model; The sensitive adversarial training module uses the original samples and the desensitized samples to perform adversarial training on the adversarial network model to transform the model to be protected into a model with privacy protection, specifically including: inputting the desensitized samples into the protected model contained in the adversarial network model for processing; inputting the original samples into the attack model contained in the adversarial network model for processing to attempt to simulate the output of the model to be protected; determining the original target loss corresponding to the model to be protected based on the difference between the output of the model to be protected for the desensitized samples and the corresponding labels or the output for the original samples; determining the black box attack loss corresponding to the attack model based on the difference between the output of the model to be protected for the desensitized samples and the output of the attack model; and performing adversarial training on the adversarial network model based on the original target loss and the black box attack loss.

11. The apparatus of claim 10, further comprising: A first training module generates first training data based on the gradient performance of the first original sample before locating the sensitive area in the original sample, and trains the sensitive area positioning model using the first training data; The sensitive area positioning module locates the sensitive area in the second original sample using the sensitive area positioning model.

12. The apparatus according to claim 11, wherein the first original sample is an image sample; The first training module inputs the image sample into the to-be-protected model for processing, and obtains the gradient corresponding to each pixel of the image sample through gradient back propagation; Determining a sensitivity corresponding to the pixel according to the gradient; According to the sensitivities of the plurality of pixels in the same area, a sensitive area is determined in the plurality of areas in the image sample, and the image sample and the sensitive area determined therefor are used as first training data.

13. The apparatus of claim 10, further comprising: A second training module, before desensitizing the sensitive area, performs adversarial training on the desensitization model based on the desensitization loss of the second training data in the desensitization model and the corresponding original target loss; The sensitive area desensitization module uses the desensitization model to desensitize the sensitive area.

14. The apparatus of claim 13, wherein the second training module, before performing adversarial training on the desensitization model, inputs the second training data into the model to be protected, processes the data, and then outputs the data to obtain a first output result; After desensitizing the second training data through the desensitization model, the second training data is input into the model to be protected for processing and then output to obtain a second output result; The corresponding original target loss is determined according to a difference between the first output result and the second output result.

15. The apparatus according to claim 10, wherein the sensitive adversarial training module performs adversarial training on the adversarial network model with minimizing the original target loss and maximizing the black-box attack loss as adversarial training objectives.

16. The apparatus of claim 10, further comprising a privacy-preserving reasoning module; The sensitive area positioning module locates the sensitive area in the current object to be inferred after the model to be protected is converted into a model with privacy protection; The sensitive area desensitization module converts the object to be inferred into a desensitized object by desensitizing the sensitive area of ​​the object to be inferred; The privacy protection reasoning module reasons on the desensitized object through the privacy protection model to obtain a reasoning result.

17. The device according to any one of claims 10 to 16, wherein the model to be protected is a non-classified model.

18. The apparatus of claim 17, wherein the non-classification model comprises an image segmentation model, and after being converted into the privacy-preserving model, the image segmentation model is used to predict image segmentation results and resist black box attacks against the image segmentation model itself.

19. A model privacy protection device, comprising: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to: Locate sensitive areas in the original sample; Converting the original sample into a desensitized sample by desensitizing the sensitive area; Determine an adversarial network model that includes the model to be protected and its corresponding attack model; The adversarial network model is trained adversarially using the original sample and the desensitized sample to transform the model to be protected into a model with privacy protection, specifically including: inputting the desensitized sample into the model to be protected contained in the adversarial network model for processing; inputting the original sample into the attack model contained in the adversarial network model for processing to attempt to simulate the output of the model to be protected; determining the original target loss corresponding to the model to be protected based on the difference between the output of the model to be protected for the desensitized sample and the corresponding label or the output for the original sample; determining the black box attack loss corresponding to the attack model based on the difference between the output of the model to be protected for the desensitized sample and the output of the attack model; and performing adversarial training on the adversarial network model based on the original target loss and the black box attack loss.

Images