Methods, devices, apparatuses, and media for protecting sensitive data

Abstract

According to implementations of the present disclosure, methods, devices, apparatuses, and media for protecting sensitive data are provided. In one method, data to be processed is received from a server device. A processing result of the data to be processed by a user is received, the processing result including sensitive data of a processing procedure of the data to be processed by the user. A gradient for training a server model at the server device is determined based on a comparison of the processing result and a predicted result for the data to be processed. The gradient is updated in a direction of change associated with the gradient to generate an updated gradient for sending to the server device. In this way, noise is added only in the direction of change associated with the gradient. On one hand, a corresponding overhead of processing noise in multiple directions can be reduced, and on the other hand, it can also be ensured that the updated gradient is not introduced with excessive noisy data that interferes with the training.

Description

TECHNICAL FIELD

[0001] Exemplary implementations of the present disclosure generally relate to the field of computers, and in particular, to methods, devices, apparatuses, and computer-readable storage media for protecting sensitive data during training of a machine learning model. BACKGROUND

[0002] With the development of machine learning technology, machine learning technology has been used in various industries. A processing model for completing a specific function can be trained based on pre-collected training data. However, certain training data can involve user privacy and / or other sensitive data. At this time, how to collect training data in a more secure manner and use it to train the processing model has become a research hotspot. Currently, a technical solution of adding noise to training data including sensitive information has been proposed. However, the performance of the existing technical solution is not ideal, and thus cannot effectively prevent sensitive information leakage. SUMMARY

[0003] According to exemplary implementations of the present disclosure, a scheme for protecting sensitive data is provided.

[0004] In a first aspect of the present disclosure, a method for protecting sensitive data is provided. In the method, data to be processed is received from a server device. A processing result of a user on the data to be processed is received, the processing result including sensitive data of a processing process of the user on the data to be processed. Based on a comparison of the processing result and a predicted result for the data to be processed, a gradient for training a server model at the server device is determined. The gradient is updated in a change direction associated with the gradient to generate an updated gradient for sending to the server device.

[0005] In a second aspect of the present disclosure, an electronic device is provided, comprising: at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions, when executed by the at least one processing unit, causing the device to perform actions. The actions include: receiving data to be processed from a server device; receiving a processing result of a user on the data to be processed, the processing result including sensitive data of a processing process of the user on the data to be processed; based on a comparison of the processing result and a predicted result for the data to be processed, determining a gradient for training a server model at the server device; and updating the gradient in a change direction associated with the gradient to generate an updated gradient for sending to the server device.

[0006] In a third aspect of the disclosure, an apparatus for protecting sensitive data is provided. The apparatus includes a data receiving module configured to receive data to be processed from a server device; a result receiving module configured to receive a processing result of the data to be processed by a user, the processing result including sensitive data of a processing process of the data to be processed by the user; a gradient determining module configured to determine a gradient for training a server model at the server device based on a comparison of the processing result and a predicted result for the data to be processed; and an update generating module configured to update the gradient in a change direction associated with the gradient to generate an updated gradient for sending to the server device.

[0007] In a fourth aspect of the disclosure, a computer readable storage medium is provided. The medium has stored thereon a computer program which, when executed by a processor, implements the method of the first aspect.

[0008] It should be understood that all statements herein made regarding the present disclosure are intended to be illustrative and not restrictive. The following detailed description of the application is presented to enable any person skilled in the art to make and use the application. The detailed description of the application includes specific details for the purpose of providing a thorough understanding of the application. However, it will be apparent to those skilled in the art that the application can be practiced without these specific details. In some instances, well-known structures and components are shown in block diagram form in order to avoid obscuring the concepts of the application. BRIEF DESCRIPTION OF DRAWINGS

[0009] The above-mentioned and other features and advantages of various implementations of the present disclosure will become more apparent from the following description referring to the accompanying drawings. In the drawings, like reference numerals refer to like elements, wherein:

[0010] Figure 1 A block diagram illustrating an example environment in which implementations of the present disclosure can be implemented is shown;

[0011] Figure 2 A block diagram illustrating a training process performed based on a client / server architecture according to some implementations of the present disclosure is shown;

[0012] Figure 3 A block diagram illustrating a process for determining a gradient change direction according to some implementations of the present disclosure is shown;

[0013] Figure 4A A block diagram illustrating a process for determining an updated gradient based on a Boolean random variable according to some implementations of the present disclosure is shown;

[0014] Figure 4B A block diagram illustrating a process for determining an updated gradient based on a Boolean random variable according to some implementations of the present disclosure is shown;

[0015] Figure 5 A block diagram illustrating a process for determining an updated gradient based on a Gaussian random variable according to some implementations of the present disclosure is shown;

[0016] Figure 6A block diagram illustrating statistical data according to some implementations of the disclosure is shown;

[0017] Figure 7 A block diagram illustrating an interaction process for protecting sensitive data according to some implementations of the disclosure is shown;

[0018] Figure 8 A flow diagram illustrating a method for protecting sensitive data according to some implementations of the disclosure is shown;

[0019] Figure 9 A block diagram illustrating an apparatus for protecting sensitive data according to some implementations of the disclosure is shown; and

[0020] Figure 10 A block diagram of an apparatus that can implement various implementations of the disclosure is shown. DETAILED DESCRIPTION

[0021] Implementations of the present disclosure will be described below with reference to drawings. While certain implementations of the present disclosure are shown in the drawings, it is understood that the present disclosure can be embodied in various forms and should not be interpreted as being limited to the implementations set forth herein; rather, these implementations are provided for more thorough and complete comprehension of the present disclosure. It is understood that the drawings and implementations of the present disclosure are for exemplary purposes only and are not intended to limit the scope of protection of the present disclosure.

[0022] In the description of implementations of the present disclosure, the term "including" and its analogs are to be interpreted broadly to mean "including, but not limited to." The term "based on" is to be interpreted as "based, at least in part, on." The term "one implementation" or "an implementation" is to be interpreted to mean "at least one implementation." The term "some implementations" is to be interpreted to mean "at least some implementations." Other explicit or implicit definitions can also be included below.

[0023] As used herein, the term "model" can learn an association between respective inputs and outputs from training data, such that a corresponding output can be generated for a given input after training is completed. The generation of a model can be based on a machine learning technique. Deep learning is a machine learning algorithm that processes an input and provides a corresponding output by using multiple layers of processing units. A neural network model is one example of a model based on deep learning. In this document, a "model" can also be referred to as a "machine learning model," a "learning model," a "machine learning network," or a "learning network," which are used interchangeably herein.

[0024] A“neural network” is a machine learning network based on deep learning. A neural network is capable of processing inputs and providing corresponding outputs, which generally includes an input layer and an output layer and one or more hidden layers between the input layer and the output layer. Neural networks used in deep learning applications generally include many hidden layers, increasing the depth of the network. The layers of a neural network are connected in sequence, such that the output of a previous layer is provided as input to a subsequent layer, with the input layer receiving the input to the neural network and the output of the output layer as the final output of the neural network. Each layer of the neural network includes one or more nodes (also referred to as processing nodes or neurons), each of which processes input from the previous layer.

[0025] Generally, machine learning can include three stages, namely a training stage, a testing stage, and an application stage (also referred to as an inference stage). In the training stage, a given model can be trained using a large amount of training data, iteratively updating parameter values until the model is able to derive consistent inferences from the training data that satisfy an expected goal. Through training, the model can be considered to have learned an association (also referred to as a mapping) from input to output from the training data. The parameter values of the trained model are determined. In the testing stage, test inputs are applied to the trained model to test whether the model is able to provide correct outputs, thereby determining the performance of the model. In the application stage, the model can be used to process actual inputs based on the trained parameter values to determine corresponding outputs.

[0026] In the field of machine learning, a prediction model can be trained using a training dataset including a large amount of training data, such that the prediction model is able to perform a desired function. For example, in the field of information pushing, information of interest can be pushed to a user. If the user is interested in the pushed information, the user can click and open the pushed information, otherwise the user can ignore the pushed information. A prediction model can be trained using the training data, such that the prediction model is able to push corresponding information of interest to different users. However, during obtaining the training data, user sensitive information can be involved. In the following, first reference will be made to Figure 1 An example environment according to one example implementation of the present disclosure is described.

[0027] Example Environment

[0028] Figure 1 A block diagram of an example environment 100 in which implementations of the present disclosure can be implemented is shown. In Figure 1 In the environment 100, it is desirable to train and use a model (i.e., a prediction model 130) that is configured to predict which data to push to a user. As Figure 1 shown, the environment 100 includes a model training system 150 and a model application system 152. Figure 1The upper part shows the process of the model training stage, and the lower part shows the process of the model application stage. Before training, the parameter values of the prediction model 130 can have initial values, or can have pre-trained parameter values obtained through a pre-training process. Through the training process, the parameter values of the prediction model 130 can be updated and adjusted. After training is completed, the prediction model 130' can be obtained. At this time, the parameter values of the prediction model 130' have been updated, and based on the updated parameter values, the prediction model 130 in the application stage can be used to implement the prediction task.

[0029] In the model training stage, the prediction model 130 can be trained based on a training data set 110 including a plurality of training data 112, and by using a model training system 150. Here, each training data 112 can involve a two-tuple format, and include to-be-processed data 120 (e.g., push data about weather and / or other content) and processing result 122 (i.e., whether the user clicks the push data). In the context of the present disclosure, various details according to one exemplary implementation of the present disclosure will be described by taking only a prediction model about a data push function as an example. According to one exemplary implementation of the present disclosure, the prediction model can also be used to implement different functions.

[0030] The prediction model 130 can be trained by using the training data 112 including the to-be-processed data 120 and the processing result 122. Specifically, the training process can be iteratively performed by using a large amount of training data. After training is completed, the prediction model 130 can recommend corresponding interested data to different users. In the model application stage, the prediction model 130' (at this time, the prediction model 130' has the trained parameter values) can be invoked by using a model application system 152. For example, input data 140 (including to-be-processed data 142) can be received, and a prediction result 144 of whether the user will click the to-be-processed data 142 is output. The prediction model 130' can be used to recommend only the data whose prediction result 144 is "click" to the user.

[0031] In Figure 1 The model training system 150 and the model application system 152 can include any computing system having computing capability, such as various computing devices / systems, terminal devices, servers, and the like. The terminal device can involve any type of mobile terminal, fixed terminal, or portable terminal including a mobile phone, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a media computer, a multimedia tablet, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. The server includes, but is not limited to, a mainframe, an edge computing node, a computing device in a cloud environment, and the like.

[0032] It should be understood that Figure 1The components and arrangements shown in environment 100 are for example only and a computing system suitable for implementing the example implementations described in this disclosure can include one or more different components, other components, and / or different arrangements of components. For example, while shown as separate, model training system 150 and model application system 152 can be integrated in the same system or device. Implementations of this disclosure are not limited in this regard. Example implementations of model training and model application will be described below with reference to the accompanying drawings, respectively.

[0033] It will be appreciated that the training process requires a large amount of training data, and the training data can include sensitive information such as user preferences. Distributed machine learning techniques have been proposed to address this issue, for example, a server device and a client device can have respective machine learning models, and the server device can send model parameters and push data to the client device. The user can process the push data at his / her own client device (e.g., the user can “click” or “not click” the push data). At this time, the processing result including the user’s personal sensitive information is stored at the client device only, and is not sent to other devices.

[0034] In distributed machine learning, the client device can include the complete training data, i.e., the data to be processed 120 (or an encoding thereof) and the processing result 122. The client device can determine gradient information for updating the prediction model, and send the gradient information to the server device. At this time, the sensitive information does not leave the user’s client device, but only the gradient information is sent to the server device. However, an attacker can use techniques such as white-box attack to recover the original sensitive information from the gradient information. To further protect the sensitive information, it has been proposed to add noise to the gradient information to prevent sensitive information leakage. However, the noise data requires additional resource overhead, and too much noise can affect the accuracy of the training data and thus reduce the performance of the training process. Thus, it is desirable to be able to protect sensitive information in a more efficient manner.

[0035] Model Training Architecture

[0036] To at least partially address the deficiencies of the above technical solutions, according to implementations of this disclosure, a method for protecting sensitive data during a training process is proposed. In the following, first refer to Figure 2 A summary of one example implementation according to this disclosure is described. Figure 2 A block diagram 200 of performing a training process based on a client / server architecture according to some implementations of this disclosure is shown. Here, the prediction model is not trained at a single computing device, but the prediction model can include a client model 214 and a server model 224, and the training can be performed in a distributed manner using both a client device 210 and a server device 220.

[0037] Figure 2 The left side illustrates the process at the client device 210 side, and Figure 2 The right side illustrates the process at the server device 220 side. The server device can include a dataset 222 storing data to be processed and a server model 224. The server device 220 can select data to be processed from the dataset 222 and send the data to be processed 120 to the client device 210. The data to be processed 120 herein can include various types of data including but not limited to weather forecast, news, traffic condition, etc. information that can be of interest to the user. It will be appreciated that the data to be processed 120 herein can include push data and / or its encoding (e.g. embedding) and the encoding can be determined based on various encoders that are currently known and / or will be developed in the future.

[0038] Further, the server device 220 can send the client model parameters 212 to the client device 210 so that the client device 210 configures the local client model 214 with the parameters. Further, the client device 210 can process the data to be processed 120 with the configured client model 214 and obtain a prediction result 230. The user 270 can click or not click the data to be processed 120 according to his / her preference, at which time the real processing result 122 (i.e. user label) can be obtained at the client device 210. The client device 210 can determine the loss gradient 240 for updating the prediction model according to the comparison between the processing result 122 from the user 270 and the prediction result 230.

[0039] In this model training process, since the processing result 122 includes sensitive information of the user 270, the processing result 122 is only stored at the client device 210 and will not be sent to other computing devices. In order to strengthen the protection of the gradient 240 and avoid attackers from deducing the sensitive information in the processing result 122 from the gradient 240, the present disclosure proposes to introduce noise 250 only in the direction of the gradient change so as to generate an updated gradient 260. At this time, the noise 250 in the direction of the gradient change can on one hand increase the difficulty for the attacker to obtain the sensitive information and on the other hand can reduce unnecessary interference of excessive noise to the training process and ensure that the prediction model can be trained along the direction of the loss gradient change.

[0040] In the following, reference will be made to Figure 3 More details of one example implementation according to the present disclosure will be described. Compared with the prior art solution of introducing noise in all directions, according to one example implementation of the present disclosure, noise is only introduced in the direction of the gradient change. First, reference will be made to Figure 3 How to determine the direction of the gradient change will be described, which Figure 3A block diagram 300 for determining the gradient change direction is shown according to some implementations of the present disclosure. Specifically, the change direction associated with the gradient can be determined based on a set of candidate results associated with the data to be processed. In the following, how to determine the gradient change direction is described with the example of the binary processing results (e.g., "click" and "no click"). As Figure 3 shown, in the example of sending the data to be processed (e.g., denoted as x) to the user, a set of candidate results (e.g., denoted as y) can be obtained: "no click (y = 0)" 310 and "click (y = 1)" 320. At this time, the training data can be represented as (x, y).

[0041] Further, the gradients associated with the above two candidate results can be determined respectively. According to one example implementation of the present disclosure, the client model parameters 212 for setting the client model 214 can be received from the server device 220. Further, the client model 214 can be configured with the received parameters so as to process the data to be processed 120 and obtain the predicted result 230.

[0042] Further, the gradient for training the server model 224 at the server device can be determined based on the comparison of the real processing result 122 from the user 270 and the predicted result 230. According to one example implementation of the present disclosure, a set of gradients associated with a set of candidate results respectively can be determined. That is, the "no click" 310 and the "click" 320 can be compared with the predicted result 230 respectively. Assuming that the user 270 does not click the push data, the real gradient 312 can be determined based on the "no click" 310, and the opposite gradient 322 opposite to the real gradient can be determined based on the "click" 320.

[0043] Here, the real gradient 312 and the opposite gradient 322 represent that the prediction model will be trained towards two different directions when the user 270 performs different operations. The difference between the two obtained gradients can be determined, and then the change direction associated with the gradient can be determined. As Figure 3 shown, assuming that the real gradient 312 and the opposite gradient 322 are denoted as vectors g0and g1respectively, g1-g0represents the gradient change direction. As Figure 3 shown, the arrow between the real gradient 312 and the opposite gradient 322 shows the direction 330 of the gradient change.

[0044] According to one example implementation of the present disclosure, the gradient change direction can be determined based on the following formula:

[0045] D = g 1-y -g y Formula 1

[0046] where D represents the gradient change direction, gy g, determined based on the real processing result of the user 1-y g, determined based on another candidate processing result in the case of binary classification

[0047] Noise data can be introduced in the gradient change direction 330 so as to increase the difficulty for an attacker to obtain sensitive information and reduce the success rate of the attacker. According to one exemplary implementation of the present disclosure, an updated gradient can be generated based on the gradient and the change direction. Specifically, the updated gradient can be generated based on the following formula 2:

[0048] g = f1(g y , D) Formula 2

[0049] where g represents the updated gradient, g y represents the gradient determined based on the real processing result of the user, D represents the gradient change direction, and f1 represents a function associated with g y and D, the specific representation of which can be determined based on various manners, which will be further described in detail below. The formula 1 above can be brought into the formula 2, and at this time, the updated gradient can be determined based on the following formula 3.

[0050] g = f2(g y , (g 1-y -g y )) Formula 3

[0051] where g represents the updated gradient, g y represents the gradient determined based on the real processing result of the user, g 1-y represents the gradient determined based on another candidate processing result in the case of binary classification, and f2 represents a function associated with g y and g 1-y , which will be further described in detail below.

[0052] According to one exemplary implementation of the present disclosure, a noise factor for updating the gradient can be determined. Here, the noise factor can represent the strength of the introduced noise, and the greater the noise factor, the greater the strength of the introduced noise. The updated gradient can be generated based on the gradient, the noise factor, and the change direction. Specifically, the size of the noise can be determined based on the product of the noise factor and the offset of the gradient determined based on a set of candidate results in the change direction. Further, the updated gradient can be determined based on the real gradient determined based on the real processing result of the user and the noise.

[0053] According to one exemplary implementation of the present disclosure, the updated gradient can be determined based on the following formula 4:

[0054] g = g y + u · (g 1-y -gy ) Formula 4

[0055] The meanings of the various symbols in Formula 4 are the same as the meanings of the symbols in the above-described formula, and u represents a noise factor.

[0056] In Formula 4, g 1-y -g y The vector can not only represent the gradient change direction determined based on different candidate results, but also represent the size of the gradient change. That is, the two endpoints of the vector respectively represent the gradient when the user performs the "click" or "not click" operation. Each point in the vector can represent the gradient change in the process of the user performing the "click" operation and the "not click" operation. Thus, different updated gradients will be generated when the noise factor in Formula 4 takes different values. Specifically, the noise factor can be determined based on a Boolean random variable or a Gaussian random variable.

[0057] According to one exemplary implementation of the present disclosure, the noise factor can be determined based on a Boolean random variable. Here, the Boolean random variable can represent one of the values of 0 and 1. Specifically, the value of the Boolean random variable can be defined based on the following Formula 5.

[0058] Pr[u = 1] = ∈, and Pr[u = 0] = 1 - ∈ Formula 5

[0059] where Pr represents the probability formula, u represents the noise factor, and ∈ represents a real number between 0 and 1. In other words, Formula 5 indicates that the value of the noise factor u can be taken from 1 or 0, where the probability of u = 1 is ∈, and the probability of u = 0 is 1 - ∈.

[0060] In the following, specific examples of determining the updated gradient based on the Boolean random variable will be described with reference to Figure 4A and Figure 4B respectively. Figure 4A A block diagram 400A for determining the updated gradient based on the Boolean random variable according to some implementations of the present disclosure is shown. As Figure 4A shown, assuming that the true processing result of the user is "not click" 310, when u = 0, g = g y + (g

[0061] Figure 4B A block diagram 400B for determining the updated gradient based on the Boolean random variable according to some implementations of the present disclosure is shown. As Figure 4B shown, assuming that the true processing result of the user is "not click" 310, when u = 1, g = g y + (g1-y -g y ). At this time, the update gradient 410B is g 1-y , and is opposite to the real gradient based on the real processing result, i.e., the end point of the gradient change direction. With the exemplary implementation of the present disclosure, the real gradient 312 or the opposite gradient 322 can be provided to the server device 220. In this way, the noise data can be added to the update gradient in a simple and effective manner, thereby reducing the probability of the attacker successfully obtaining the sensitive information.

[0062] In the above example, the update gradient 410A when u = 0 is the same as the real gradient, and the update gradient 420B when u = 1 is opposite to the real gradient. It will be understood that although the opposite gradient can interfere with the judgment of the attacker, however, too much interference will result in the gradient for actual training not being accurate enough, thus it is difficult to obtain an accurate prediction model with limited time and computing resources. Therefore, the case of the opposite gradient should be properly limited.

[0063] According to one exemplary implementation of the present disclosure, in order to ensure that the update gradient can represent the correct gradient determined based on the real processing result in most cases, the range of values of ∈ should be further limited. For example, ∈ can be defined as < 1 / 2. At this time, the success probability of the attacker is 1 - ∈, and the attack AUC (Area Under Curve) parameter is (1 - ∈) 2 + ∈ (1 - ∈) = 1 - ∈.

[0064] It will be understood that the attack AUC is a parameter defined in the context of the present disclosure to represent the success rate of the attacker. In the scenario of the attacker cracking the real label data, the attacker can set a scoring function, and for each sample, a corresponding score t can be generated. A threshold t0 can be selected, and if t > t0, the real label is considered to be 1, and vice versa. For the known t0, the true positive rate or the false positive rate of the attacker can be considered. However, when the threshold t0 is unknown, all thresholds can be traversed and a curve of the true positive rate versus the false positive rate can be generated. For a specific protection method used to generate the update gradient, the larger the AUC of the generated curve is, the more accurate the scoring function is, and the area of 1 represents complete correctness.

[0065] With the exemplary implementation of the present disclosure, by limiting the range of values of ∈, the update gradient can reflect the real label of the user in most cases, thus enabling the prediction to be performed in the direction of making the prediction model consistent with the real training data.

[0066] According to one example implementation of the present disclosure, the noise factor can be determined based on a Gaussian random variable. Specifically, the noise factor can be determined based on the following Equation 6:

[0067]

[0068] where u represents the noise factor, is a Gaussian distribution function with a distribution amplitude of e. In the following, we will refer to Figure 5 More details about determining the noise factor based on a Gaussian random variable are described.

[0069] Figure 5 A block diagram 500 for determining an update gradient based on a Gaussian random variable according to some implementations of the present disclosure is shown. As Figure 5 indicated, a specific value of the noise factor u can be determined based on a Gaussian distribution function 510. Here, the noise factor u can be represented in high-dimensional data and involves multiple directions. Since only the gradient in the gradient change direction 330 contributes to obtaining better training results, the noise factor u can be mapped to the gradient change direction 330 without considering the components in other directions. In this way, a value with a different gradient can be selected from the gradient change direction 330 as the update gradient. For example, a gradient located at a position with a distance ratio of 3:7 (or other positions closer to the real gradient 312) to the real gradient 312 and the opposite gradient 322 in the gradient change direction 330 can be selected. In other words, in most cases, the update gradient should be located at a position closer to the real gradient 312 in the gradient change direction 330.

[0070] With the example implementations of the present disclosure, instead of considering other directions except the gradient change direction 330, noise is only added in the gradient change direction 330. In this way, on the one hand, the related calculation overhead of the noise components in other directions can be reduced, and on the other hand, the interference of the noise components in other directions on the training results can be reduced. The update gradient closer to the real gradient 312 in the gradient change direction 330 can provide a beneficial contribution to improving the accuracy of the prediction model. The update gradient closer to the opposite gradient 322 can increase the difficulty of the attacker’s cracking process, thereby reducing the success rate of the attacker.

[0071] According to one exemplary implementation of the present disclosure, the range of values of ∈ can be further limited so that the update gradient determined based on the Gaussian random variable is closer to the true gradient 312. The noise factor should satisfy the following condition: the update gradient calculated based on the noise factor is closer to the true gradient 312 than to the opposite gradient 322. In other words, the probability that the distance between the update gradient and the true gradient is less than the distance between the update gradient and the opposite gradient (i.e., other candidate gradients in the set of candidate gradients different from the gradient) should not be lower than 1 / 2. ∈ that makes u < 1 / 2 can be determined based on currently known and / or to be developed in the future methods. At this time, the failure probability of the attacker is:

[0072]

[0073] The meanings of various symbols in Equation 7 are the same as those in the above-described equations, and at this time, the attack AUC parameter is Pr[u < 1 / 2].

[0074] Compared with the existing technical solutions of introducing noise in multiple directions of the gradient, introducing noise only in the gradient change direction can be more helpful to prevent the attacker from being successful. Figure 6 A block diagram 600 of statistical data according to some implementations of the present disclosure is shown, in which the vertical coordinate represents the attack AUC described above, and the horizontal coordinate represents the test AUC. Here, the test AUC is an index for measuring the prediction model, and can be used to determine whether the trained prediction model is accurate based on the test samples in the test data set. As shown in Figure 6 The legend 610 represents the statistical data of the comparison between the attack AUC and the test AUC determined according to the technical solutions of the present disclosure, and the legend 620 represents the statistical data of the comparison between the attack AUC and the test AUC determined according to the existing technical solutions. As shown in Figure 6 The curve corresponding to the legend 610 is above the curve corresponding to the legend 620. Thus, compared with the existing technical solutions, the technical solutions of the present disclosure can better protect sensitive data while ensuring the performance of the prediction model.

[0075] According to one example implementation of the present disclosure, after the update gradient has been obtained, the update gradient can be sent to the server device 220 so that the server device 220 determines the client model parameters for setting the client model in the next iteration process according to the update gradient. According to one example implementation of the present disclosure, the above-described process can be iteratively performed until the prediction model meets the predetermined convergence condition. At this time, the server device 220 can send the new parameters to the client device 210 in each iteration process so that the client device determines the gradient and the corresponding update gradient based on the new parameters. The above-described process can be constantly repeated until the predetermined convergence condition is met. At this time, since the update gradient includes noise data, it is difficult for an attacker to perform a cracking process and thus the sensitive data in the original processing result can be protected.

[0076] The above has described that the server device 220 sends the single to-be-processed data 120 to the client device 210 and obtains the corresponding update gradient. According to one example implementation of the present disclosure, the above-described process can be performed in a batch processing manner. For example, the client device 210 can receive a training data packet including multiple to-be-processed data from the server device 220. Further, the client device 210 can process each to-be-processed data in the training data packet. Specifically, the client device 210 can display each to-be-processed data and receive the processing result of each to-be-processed data from the user. The gradient change direction can be determined in a similar process described above, and the gradient is updated in the determined gradient change direction. At this time, multiple update gradients can be generated.

[0077] According to one example implementation of the present disclosure, the above-described process can be iteratively performed according to a predetermined batch size. Assuming that the batch size is n, the server device can send n to-be-processed data to the client device at a time. After processing, the client device can return n update gradients to the server device at a time. Then, the server device can perform the training process using the received update gradients. According to one example implementation of the present disclosure, the server device can update the server model parameters according to the direction of the update gradient. Further, the client device parameters sent to the client device can be determined based on the updated parameters.

[0078] According to one example implementation of the present disclosure, the above-described method can be implemented at the client device connected to the server, and the method can be iteratively performed until the predetermined convergence condition is reached.

[0079] In the following, reference is made to Figure 7 The interaction process between various entities is described. Figure 7A block diagram 700 illustrates an interaction process for protecting sensitive data according to some implementations of this disclosure. For example... Figure 7 As shown, server device 220 can send 710 pieces of data to be processed to client device 210. For example, it can send only one piece of data at a time, or it can send multiple pieces of data at once in batch processing. Client device 210 can display the data to be processed to user 270, and user 270 can send the processing result 712 to client device 210. After receiving the processing result, client device 210 can determine the relevant loss gradient 714 based on the comparison between the true value and the predicted value of the processing result. Further, client device 210 can determine the corresponding gradient change direction and update the gradient 716 only in the gradient change direction. Further, client device 210 can send 718 updated gradient to server device 220. After receiving the updated gradient, server device 220 can use the updated gradient to perform training 720.

[0080] It will be understood that the above description of a process for protecting sensitive data uses only labels including binary classifications as an example. According to one exemplary implementation of this disclosure, user-generated labels can also include more classifications. For example, a user can rate the data being processed (e.g., a rating of 1-100), and so on. Multiple gradient directions can be determined in a similar manner. Assuming the user-generated labels involve k+1 candidate classifications, the gradients g0, g1, ... g associated with each candidate classification can be determined. k Then, it can be based on g1-g0, g2-g0, ..., g k -g0 is used to determine the directions of multiple gradient changes.

[0081] Furthermore, the corresponding noise factors can be determined using either Boolean or Gaussian random variables as described above. Then, in a similar manner, the update gradients corresponding to the real user labels can be determined in each gradient change direction. It will be understood that the number of candidate classifications should be less than the dimension of the vector representation of the training data; otherwise, too many gradient change directions will cause the method disclosed herein to degenerate into adding noise in each direction.

[0082] Using the exemplary implementation of this disclosure, the processing result of sensitive information of user 270 does not leave the client device 210, but is processed locally on the client device 210 to obtain updated gradients. Here, the updated gradients only include noisy data in the gradient change directions that help improve training efficiency. Compared with existing technical solutions that add noisy data in all directions, this approach reduces the overhead of processing noise in multiple directions and ensures that the updated gradients are not introduced with excessive noise data.

[0083] Model Application Architecture

[0084] The training of prediction model 130, as described above, can be provided to, for example, Figure 1 The model is used in the illustrated application system 152 to process the input data 140. It will be understood that although the update gradients used during training include noisy data, the noise only exists in the direction of gradient change. Furthermore, by using a noise factor to control the intensity of the noise data, the update gradient is aligned with the direction of the true gradient, thus allowing the prediction model to be trained in a direction that aligns the prediction model with the real user labels. In this way, the trained prediction model 130' can have high accuracy, thereby providing an accurate basis for subsequent model applications.

[0085] According to an exemplary implementation of this disclosure, after the model training phase is completed, the trained prediction model 130' with trained parameter values ​​can be used to process the received input data. For example, data 142 to be processed can be input into the prediction model 130'. At this time, the prediction model 130' can output a prediction result 144 of whether the user will click on the data to be processed. Specifically, only data of interest to the user can be pushed. For example, assuming that the prediction model 130' indicates that user 270 always clicks on push data related to weather, weather data can be pushed to user 270 preferentially. As another example, assuming that the prediction model 130' indicates that user 270 rarely opens push data related to sports news, the frequency of pushing sports news can be reduced. In this way, the trained prediction model 130' can have high accuracy, thereby providing a basis for judgment in subsequent push operations.

[0086] Example Process

[0087] Figure 8 A flowchart of a method 800 for protecting sensitive data according to some implementations of this disclosure is shown. Specifically, at block 810, data to be processed can be received from a server device.

[0088] At box 820, the user's processing result for the data to be processed can be received. The processing result includes sensitive data about the user's processing of the data to be processed.

[0089] At block 830, a gradient for training a server model at the server device can be determined based on a comparison of the processing result and a predicted result for the to-be-processed data. According to one example implementation of the present disclosure, to determine the gradient, the client model parameters for setting the client model can be received from the server device. Further, the to-be-processed data can be processed based on the client model parameters and the client model to obtain the predicted result for the to-be-processed data.

[0090] According to one example implementation of the present disclosure, the updated gradient can be sent to the server device to cause the server device to determine the client model parameters for setting the client model according to the updated gradient.

[0091] At block 840, the gradient can be updated in a change direction associated with the gradient to generate an updated gradient for sending to the server device. According to one example implementation of the present disclosure, to generate the updated gradient, a change direction associated with the gradient can be determined based on a set of candidate results associated with the to-be-processed data. Then, the updated gradient can be generated based on the gradient and the change direction.

[0092] According to one example implementation of the present disclosure, to determine the change direction associated with the gradient, a set of gradients respectively associated with the set of candidate results can be determined, and the change direction associated with the gradient can be determined based on a difference between the set of gradients.

[0093] According to one example implementation of the present disclosure, to determine the updated gradient, a noise factor for updating the gradient can be determined, the noise factor representing an intensity of noise data introduced to the gradient. Then, the updated gradient can be generated based on the gradient, the noise factor and the change direction.

[0094] According to one example implementation of the present disclosure, to determine the noise factor, the noise factor can be determined based on a Boolean random variable, where the noise factor makes a probability that the updated gradient is equal to the gradient no less than 1 / 2.

[0095] According to one example implementation of the present disclosure, to determine the noise factor, the noise factor can be determined based on a Gaussian random variable. Here, the noise factor makes a probability that a distance between the updated gradient and the gradient is less than a distance between the updated gradient and other candidate gradients in the set of candidate gradients different from the gradient no less than 1 / 2.

[0096] According to one example implementation of the present disclosure, the to-be-processed data is included in a training data packet from the server device, and further comprising: processing other to-be-processed data in the training data packet by the method 800.

[0097] According to one example implementation of the present disclosure, the method 800 is performed at a client device connected to a server, and can be performed iteratively.

[0098] Example Devices and Equipment

[0099] Figure 9 A block diagram of an apparatus 900 for protecting sensitive data according to some implementations of the present disclosure is shown. As shown, the apparatus 900 includes a data receiving module 910, a result receiving module 920, a gradient determining module 930, and an update generating module. Figure 9

[0100] According to one example implementation of the present disclosure, the data receiving module 910 is configured to receive the to-be-processed data from the server device, the result receiving module 920 is configured to receive a processing result of the to-be-processed data by the user, the processing result including sensitive data of a processing process of the to-be-processed data by the user, the gradient determining module 930 is configured to determine a gradient for training a server model at the server device based on a comparison between the processing result and a predicted result for the to-be-processed data, and the update generating module 940 is configured to update the gradient in a change direction associated with the gradient to generate an updated gradient for sending to the server device.

[0101] According to one example implementation of the present disclosure, the gradient determining module 930 is further configured to receive a client model parameter for setting a client model from the server device, and process the to-be-processed data based on the client model parameter and the client model to obtain the predicted result for the to-be-processed data.

[0102] According to one example implementation of the present disclosure, the apparatus 900 further includes a sending module configured to send the updated gradient to the server device, so that the server device determines the client model parameter for setting the client model according to the updated gradient.

[0103] According to one example implementation of the present disclosure, the update generating module 940 is further configured to determine the change direction associated with the gradient based on a set of candidate results associated with the to-be-processed data, and generate the updated gradient based on the gradient and the change direction.

[0104] According to one example implementation of the present disclosure, the update generating module 940 is further configured to determine a set of gradients respectively associated with a set of candidate results, and determine the change direction associated with the gradient based on a difference between the set of gradients.

[0105] ​According to one example implementation of the present disclosure, the update generation module 940 is further configured to determine a noise factor for the update gradient, the noise factor representing an intensity of noise data introduced to the gradient; and generate the update gradient based on the gradient, the noise factor, and the change direction.

[0106] According to one example implementation of the present disclosure, the update generation module 940 is further configured to determine the noise factor based on a Boolean random variable, the noise factor making a probability that the update gradient equals to the gradient no less than 1 / 2.

[0107] According to one example implementation of the present disclosure, the update generation module 940 is further configured to determine the noise factor based on a Gaussian random variable, the noise factor making a probability that a distance between the update gradient and the gradient is smaller than distances between the update gradient and other candidate gradients in the set of candidate gradients different from the gradient no less than 1 / 2.

[0108] According to one example implementation of the present disclosure, the to-be-processed data is included in a training data package from a server device, and the apparatus 900 further comprises a calling module configured to process other to-be-processed data in the training data package by utilizing the above-described various modules.

[0109] According to one example implementation of the present disclosure, the apparatus 900 is implemented at a client device connected to a server, and the apparatus 900 is iteratively called at the client device.

[0110] Figure 10 A block diagram of an apparatus 1000 capable of implementing the implementations of the present disclosure is shown. It should be understood that Figure 10 The illustrated computing device 1000 is merely an example and should not be construed as limiting the functionality and scope of the implementations described herein. Figure 10 The illustrated computing device 1000 can be used to implement, for example, Figure 1 The illustrated model training system 150, as well as the illustrated model application system 152. Figure 1 The illustrated model training system 150, as well as the illustrated model application system 152.

[0111] The illustrated model training system 150, as well as the illustrated model application system 152. Figure 10As shown, the computing device 1000 is in the form of a general-purpose computing device. The components of the computing device 1000 can include, but are not limited to, one or more processors or processing units 1010, a memory 1020, a storage device 1030, one or more communication units 1040, one or more input devices 1050, and one or more output devices 1060. The processing unit 1010 can be a real or virtual processor and is capable of executing various processing in accordance with programs stored in the memory 1020. In a multi-processing system, multiple processing units execute computer-executable instructions in parallel, enabling high-throughput processing by the computing device 1000.

[0112] The computing device 1000 typically includes a plurality of computer storage media. Such media can be removable and / or non-removable, volatile and / or non-volatile. The memory 1020 can be volatile (such as, for example, registers, cache, RAM), non-volatile (such as, for example, ROM, EEPROM, flash memory), or some combination thereof. The storage device 1030 can be removable or non-removable, and can include, but is not limited to, magnetic disks, optical disks, or tape. Specifically, a machine readable medium can include a machine readable storage medium encoded with one or more programs that, when executed, implement the methods or

[0113] The computing device 1000 can further include additional removable / non-removable, volatile / non-volatile storage media. Although not specifically shown in the computing device 1000, these can include, but are not limited to, magnetic disks, optical disks, tape, flash memory, and / or other memory. Figure 10 In some embodiments, a disk drive or other computer readable media storage device can be provided, which can be removable or non-removable, and which can include, but is not limited to, magnetic disks, optical disks, or tape. Such computer readable media storage devices can be connected to the bus by one or more data media interfaces. The memory 1020 can include a computer program product 1025 having one or more program modules configured to carry out the various methods or actions of the present disclosure.

[0114] The communication unit 1040 enables communications with other computing devices over a communication media. Additionally, the functionality of the components of the computing device 1000 can be implemented in a single computing cluster or multiple computer machines that are capable of communicating over a communication connection. Thus, the computing device 1000 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network nodes in a networking environment.

[0115] Input device 1050 can be one or more input devices, such as a mouse, a keyboard, a trackball, etc. Output device 1060 can be one or more output devices, such as a display, a speaker, a printer, etc. Computing device 1000 can also communicate with one or more external devices (not shown) such as a storage device, a display device, etc. through communication unit 1040, and with one or more devices that enable a user to interact with computing device 1000, and / or any devices (e.g., a network card, a modem, etc.) that enable computing device 1000 to communicate with one or more other computing devices. Such communication can occur via Input / Output (I / O) interface (not shown).

[0116] According to an example implementation of the present disclosure, a computer readable storage medium is provided having computer executable instructions stored thereon, where the computer executable instructions are executed by a processor to implement the method described above. According to an example implementation of the present disclosure, a computer program product is also provided that is tangibly stored on a non-transitory computer readable medium and includes computer executable instructions, where the computer executable instructions are executed by a processor to implement the method described above. According to an example implementation of the present disclosure, a computer program product is provided having a computer program stored thereon, which when executed by a processor implements the method described above.

[0117] Various aspects of the disclosure can be described in the context of flow diagrams and / or block diagrams that illustrate the functions and / or acts performed by the methods, apparatus, devices, and computer program products according to the present disclosure. It is to be understood that each block of the flow diagrams and / or block diagrams, and combinations of blocks in the flow diagrams and / or block diagrams, can be implemented by computer readable program instructions.

[0118] These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flow diagrams and / or block diagrams. These computer readable program instructions can also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and / or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including instructions which implement aspects of the function / act specified in the flow diagrams and / or block diagrams.

[0119] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0120] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0121] The above-described implementations of this disclosure are illustrative and not exhaustive, and are not limited to the disclosed implementations. Numerous modifications and adaptations will be apparent to those skilled in the art without departing from the scope and spirit of the disclosed implementations. The choice of words in this document is intended to best explain the principles of the implementations, practical application, or improvement over the technology in the market, or to enable other ordinary skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for protecting sensitive data, comprising: Receive pending data from the server device; Receive the user's processing result for the data to be processed, the processing result including sensitive data of the user's processing process for the data to be processed; Based on the comparison between the processing result and the prediction result for the data to be processed, the gradient of the loss used to train the server model at the server device is determined. as well as Updating the gradient in the direction of change associated with the gradient to generate an updated gradient for sending to the server device includes: Determining the direction of change associated with the gradient based on a set of candidate results associated with the data to be processed includes: determining a set of gradients associated with each of the set of candidate results; and determining the direction of change associated with the gradient based on the differences between the set of gradients; and The updated gradient is generated based on the gradient and the direction of change.

2. The method according to claim 1, further comprising: Receive client model parameters from the server device for setting the client model; as well as Based on the client model parameters and the client model processing the data to be processed, the prediction result for the data to be processed is obtained.

3. The method according to claim 2, further comprising: The update gradient is sent to the server device so that the server device determines the client model parameters for setting the client model based on the update gradient.

4. The method of claim 1, wherein generating the updated gradient based on the gradient and the direction of change comprises: Determine a noise factor for updating the gradient, the noise factor representing the intensity of noise data introduced into the gradient; as well as The updated gradient is generated based on the gradient, the noise factor, and the direction of change.

5. The method of claim 4, wherein determining the noise factor for updating the gradient comprises: The noise factor is determined based on a Boolean random variable, and the noise factor makes the probability that the updated gradient is equal to the gradient not less than 1 / 2.

6. The method of claim 4, wherein determining the noise factor for updating the gradient comprises: The noise factor is determined based on a Gaussian random variable, and the noise factor makes the probability that the distance between the updated gradient and the gradient is less than the distance between the updated gradient and other candidate gradients in the set of gradients that are different from the gradient not less than 1 / 2.

7. The method of claim 1, wherein the data to be processed is included in a training data packet from the server device, and the method further comprises: The method described above is used to process other unprocessed data in the training data packet.

8. The method of claim 1, wherein the method is performed at a client device connected to the server, and the method is performed iteratively at the client device.

9. An electronic device, comprising: At least one processing unit; as well as At least one memory, coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the device to perform the following actions when executed by the at least one processing unit: Receive pending data from the server device; Receive the user's processing result for the data to be processed, the processing result including sensitive data of the user's processing process for the data to be processed; Based on the comparison between the processing result and the prediction result for the data to be processed, the gradient of the loss used to train the server model at the server device is determined. as well as Updating the gradient in the direction of change associated with the gradient to generate an updated gradient for sending to the server device includes: Determining the direction of change associated with the gradient based on a set of candidate results associated with the data to be processed includes: determining a set of gradients associated with each of the set of candidate results; and determining the direction of change associated with the gradient based on the differences between the set of gradients; and The updated gradient is generated based on the gradient and the direction of change.

10. The device according to claim 9, further comprising: Receive client model parameters from the server device for setting the client model; as well as Based on the client model parameters and the client model processing the data to be processed, the prediction result for the data to be processed is obtained.

11. The device according to claim 10, further comprising: The update gradient is sent to the server device so that the server device determines the client model parameters for setting the client model based on the update gradient.

12. The device of claim 9, wherein generating the updated gradient based on the gradient and the direction of change comprises: Determine a noise factor for updating the gradient, the noise factor representing the intensity of noise data introduced into the gradient; as well as The updated gradient is generated based on the gradient, the noise factor, and the direction of change.

13. The apparatus of claim 12, wherein determining the noise factor for updating the gradient comprises: The noise factor is determined based on a Boolean random variable, and the noise factor makes the probability that the updated gradient is equal to the gradient not less than 1 / 2.

14. The apparatus of claim 12, wherein determining the noise factor for updating the gradient comprises: The noise factor is determined based on a Gaussian random variable, and the noise factor makes the probability that the distance between the updated gradient and the gradient is less than the distance between the updated gradient and other candidate gradients in the set of gradients that are different from the gradient not less than 1 / 2.

15. The apparatus of claim 9, wherein the data to be processed is included in a training data packet from the server device, and the action further includes: The action is used to process other unprocessed data in the training data packet.

16. The apparatus of claim 9, wherein the apparatus is executed at a client device connected to the server, and the action is executed iteratively at the client device.

17. An apparatus for protecting sensitive data, comprising: The data receiving module is configured to receive unprocessed data from the server device. The result receiving module is configured to receive the user's processing result for the data to be processed, the processing result including sensitive data of the user's processing process of the data to be processed; A gradient determination module is configured to determine the gradient of the loss used to train the server model at the server device based on a comparison between the processing result and the prediction result for the data to be processed. as well as The update generation module is configured to update the gradient in the direction of change associated with the gradient to generate an updated gradient for sending to the server device, including: Determining the direction of change associated with the gradient based on a set of candidate results associated with the data to be processed includes: determining a set of gradients associated with each of the set of candidate results; and determining the direction of change associated with the gradient based on the differences between the set of gradients; and The updated gradient is generated based on the gradient and the direction of change.

18. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method according to any one of claims 1 to 8.

19. A computer program product having a computer program stored thereon, which, when executed by a processor, implements the method according to any one of claims 1 to 8.

Images