Model training method and device for protecting privacy data

Abstract

The embodiment of the specification provides a model training method and device for protecting privacy data. The method comprises the following steps: a first party receives ciphertext feature data of each training sample in a training sample set from a second party, wherein the ciphertext feature data is obtained by compressing plaintext feature data; the ciphertext feature data of any training sample is input into a deep learning model to obtain an embedding feature vector of the training sample; the embedding feature vector of any training sample held by the first party, a task label of the training sample provided by the second party and a task model provided by a third party are used to jointly train the task model in a secure multi-party computation manner, and the trained task model and the deep learning model jointly realize prediction of a target task. Security can be ensured, and the efficiency of model training can be improved.

Description

Technical Field

[0001] This specification relates to one or more embodiments in the field of computers, and more particularly to model training methods and apparatus for protecting privacy data. Background Technology

[0002] Currently, data held by different data holders may contain users' private information, and data sharing between data holders may infringe on users' privacy. In order to facilitate the flow of data among multiple parties, secure multi-party computation can be used to support joint computation among multiple parties, extract the value of data, and ensure that plaintext information of each party's private data is not leaked during multi-party interactions.

[0003] In a typical scenario, a pre-trained deep learning model is used to further train the model, enabling it to perform better on downstream tasks. The training samples and the deep learning model itself are typically held by different parties and constitute their private data, requiring protection against leakage. While existing technologies such as secure multi-party computation based on secret sharing offer robust security, they incur significant communication overhead, resulting in low training efficiency. Summary of the Invention

[0004] This specification describes one or more embodiments of a model training method and apparatus that protects privacy data, providing both security and improved model training efficiency.

[0005] In a first aspect, a model training method for protecting privacy data is provided. A first party has a pre-trained deep learning model, a second party has plaintext feature data and task labels for each training sample in a training sample set, and a third party has a task model downstream of the deep learning model. The method is executed by the first party and includes:

[0006] The second party receives the ciphertext feature data of each training sample in the training sample set, wherein the ciphertext feature data is obtained by compressing the plaintext feature data of each training sample in the training sample set.

[0007] Input the encrypted feature data of any training sample into the deep learning model to obtain the embedded feature vector of the training sample;

[0008] By utilizing the embedded feature vector of any training sample held by this party, the task label of the training sample provided by the second party, and the task model provided by the third party, the task model is jointly trained through secure multi-party computation. The trained task model and the deep learning model together achieve the prediction of the target task.

[0009] In one possible implementation, the deep learning model is based on a multi-layered transformer.

[0010] In one possible implementation, the embedded feature vector is split into three slices, with the first party, the second party, and the third party each holding two slices; the task label is split into three slices, with the first party, the second party, and the third party each holding two slices.

[0011] Furthermore, the joint training of the task model through secure multi-party computation includes:

[0012] The task results of the training samples are divided into three parts by a first joint computation; the task results are obtained by using the embedded feature vector as the input of the task model; the task results are divided into three parts, and each party holds two parts of the task results.

[0013] Using the task result shards and the task label shards, the shards of the total prediction loss are determined through a second joint calculation; the total prediction loss is split into three shards, with each party holding two shards of the total prediction loss; the total prediction loss is used by the third party to adjust the model parameters of the task model.

[0014] Furthermore, the task model is a multilayer perceptron; the first joint computation includes secure multiplication and secure comparison operations; the second joint computation includes an approximate secure operation of exponentiation.

[0015] Secondly, a model training method for protecting privacy data is provided. A first party has a pre-trained deep learning model, a second party has plaintext feature data and task labels for each training sample in a training sample set, and a third party has a task model downstream of the deep learning model. The method is executed by the second party and includes:

[0016] The plaintext feature data of each training sample in the training sample set is compressed to obtain the ciphertext feature data of each training sample in the training sample set.

[0017] Each encrypted feature data is sent to the first party, so that the first party can input the encrypted feature data of any training sample into the deep learning model to obtain the embedded feature vector of the training sample.

[0018] Using the task label of any training sample held by this party, the embedded feature vector of the training sample provided by the first party, and the task model provided by the third party, the task model is jointly trained through secure multi-party computation. The trained task model and the deep learning model together achieve the prediction of the target task.

[0019] In one possible implementation, the data compression is achieved by matching the feature distributions of ciphertext feature data and plaintext feature data in different sampling embedding spaces to compress a first dataset composed of plaintext feature data into a second dataset composed of ciphertext feature data.

[0020] Furthermore, the data compression includes:

[0021] Randomly generate initial synthetic data;

[0022] Using the maximum average difference between the second distribution of the synthetic data and the first distribution of the plaintext feature data as a loss function, with the goal of minimizing the value of the loss function, the adjusted synthetic data is determined as the ciphertext feature data through multiple rounds of iteration.

[0023] Thirdly, a model training method for protecting privacy data is provided. A first party has a pre-trained deep learning model, a second party has plaintext feature data and task labels for each training sample in a training sample set, and a third party has a task model downstream of the deep learning model. The method is executed by the third party and includes:

[0024] Using the task model held by this party, the embedded feature vector of any training sample provided by the first party, and the task label of the training sample provided by the second party, the task model is jointly trained through secure multi-party computation.

[0025] The target task prediction is jointly performed using the trained task model held by this party and the deep learning model provided by the first party.

[0026] Fourthly, a model training apparatus for protecting privacy data is provided. A first party has a pre-trained deep learning model, a second party has plaintext feature data and task labels for each training sample in a training sample set, and a third party has a task model downstream of the deep learning model. The apparatus is located at the first party and includes:

[0027] The receiving unit is configured to receive from the second party the ciphertext feature data of each training sample in the training sample set, wherein the ciphertext feature data is obtained by compressing the plaintext feature data of each training sample in the training sample set.

[0028] An embedding processing unit is used to input the encrypted feature data of any training sample received by the receiving unit into the deep learning model to obtain the embedding feature vector of the training sample.

[0029] The joint training unit is used to jointly train the task model by using the embedding feature vector of any training sample obtained by the embedding processing unit held by this party, the task label of the training sample provided by the second party, and the task model provided by the third party, through secure multi-party computation. The trained task model and the deep learning model jointly achieve the prediction of the target task.

[0030] Fifthly, a model training apparatus for protecting privacy data is provided. A first party has a pre-trained deep learning model, a second party has plaintext feature data and task labels for each training sample in a training sample set, and a third party has a task model downstream of the deep learning model. The apparatus is disposed in the second party and includes:

[0031] The compression processing unit is used to compress the plaintext feature data of each training sample in the training sample set to obtain the ciphertext feature data of each training sample in the training sample set.

[0032] The sending unit is used to send the encrypted feature data obtained by the compression processing unit to the first party, so that the first party can input the encrypted feature data of any training sample into the deep learning model to obtain the embedded feature vector of the training sample.

[0033] The joint training unit is used to jointly train the task model by using the task label of any training sample held by the party, the embedded feature vector of the training sample provided by the first party, and the task model provided by the third party through secure multi-party computation. The trained task model and the deep learning model jointly achieve the prediction of the target task.

[0034] Sixthly, a model training apparatus for protecting privacy data is provided. A first party has a pre-trained deep learning model, a second party has plaintext feature data and task labels for each training sample in a training sample set, and a third party has a task model downstream of the deep learning model. The apparatus is disposed within the third party and includes:

[0035] The joint training unit is used to jointly train the task model by utilizing the task model held by this party, the embedded feature vector of any training sample provided by the first party, and the task label of the training sample provided by the second party through secure multi-party computation.

[0036] The joint prediction unit is used to jointly perform the prediction of the target task using the task model trained by the joint training unit held by the party and the deep learning model provided by the first party.

[0037] In a seventh aspect, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the methods of the first aspect, the second aspect, or the third aspect.

[0038] Eighthly, a computing device is provided, including a memory and a processor, wherein executable code is stored in the memory, and the processor, when executing the executable code, implements the methods of the first aspect, the second aspect, or the third aspect.

[0039] The method and apparatus provided in the embodiments of this specification involve a first party having a pre-trained deep learning model, a second party having plaintext feature data and task labels for each training sample in a training sample set, and a third party having a task model downstream of the deep learning model. The first party first receives encrypted feature data for each training sample in the training sample set from the second party; the encrypted feature data is obtained by compressing the plaintext feature data of each training sample in the training sample set. Then, the encrypted feature data of any training sample is input into the deep learning model to obtain the embedded feature vector of that training sample. Finally, using the embedded feature vector of any training sample held by the first party, the task label of the training sample provided by the second party, and the task model provided by the third party, the task model is jointly trained through secure multi-party computation. The trained task model and the deep learning model jointly achieve the prediction of the target task. As can be seen from the above, the embodiments of this specification protect privacy data during model training based on data compression and secure multi-party computation. Deep learning models are typically large in scale, while task models are typically small in scale. Joint training of the smaller task model requires only a small amount of communication overhead, thereby improving the efficiency of model training. The second party compresses the plaintext feature data of each training sample to obtain ciphertext feature data, and then sends this ciphertext feature data to the first party. This avoids the leakage of plaintext feature data and reduces the amount of communication between the two parties. Furthermore, the first party performs deep learning model computation locally based on the received ciphertext feature data, without needing to communicate with other parties. Additionally, the task model is jointly trained through secure multi-party computation, preventing the privacy leakage of the embedded feature vectors obtained by the first party and the task labels possessed by the second party. In summary, this approach provides both security and improves model training efficiency. Attached Figure Description

[0040] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is a schematic diagram illustrating an implementation scenario of one embodiment disclosed in this specification;

[0042] Figure 2 This is a schematic diagram illustrating an implementation scenario of another embodiment disclosed in this specification;

[0043] Figure 3 This diagram illustrates a flowchart of a model training method for protecting privacy data according to one embodiment.

[0044] Figure 4 A schematic diagram illustrating the model training process for protecting privacy data according to one embodiment is shown.

[0045] Figure 5 A schematic block diagram of a model training apparatus for protecting privacy data according to one embodiment is shown;

[0046] Figure 6 A schematic block diagram of a model training apparatus for protecting privacy data according to another embodiment is shown;

[0047] Figure 7 A schematic block diagram of a model training apparatus for protecting privacy data according to another embodiment is shown. Detailed Implementation

[0048] The solution provided in this specification will now be described with reference to the accompanying drawings.

[0049] Figure 1 This is a schematic diagram illustrating an implementation scenario of one embodiment disclosed in this specification. This implementation scenario involves model training to protect privacy data. Prior to this model training, a pre-trained deep learning model already exists. Fine-tuning of the deep learning model is achieved by continuing to train a task model. In this embodiment, the deep learning model is relatively large, often referred to as a large model, and has achieved excellent results in various fields. To leverage the pre-trained deep learning model for better performance on different downstream tasks, further model training is required; this model training is also called model fine-tuning. Model fine-tuning refers to fine-tuning a pre-trained deep learning model using training data for a specific task. The hidden layer representation output by the deep learning model is used as input to train a task model, enabling it to better adapt to the specific task. It is understood that the deep learning model and the task model work together to achieve a specific task. The feature data of a sample is input into the deep learning model to obtain the embedded feature vector of the sample. This embedded feature vector is then input into the task model to obtain the task result.

[0050] The specific task mentioned above can be a classification task. For example, the task model mentioned above can be a classifier used to perform classification tasks, such as determining whether a sample user is at risk.

[0051] In the embodiments described in this specification, deep learning models, task-specific training data, and task models are typically held by different parties and constitute their private data. During model training, it is necessary to protect this private data from leakage. While secure multi-party computation based on secret sharing provides strong security guarantees, it requires significant communication overhead, resulting in low model training efficiency.

[0052] Secure multi-party computation: a cryptographic technique that enables multiple untrusted parties to jointly compute a target function without exposing any information other than the output, protecting the inputs of the participants and the intermediate results of the computation.

[0053] Secret sharing is a method of distributing a secret among different participants, with each party receiving a portion of the secret, called a shard. The secret can only be recovered by holding a sufficient number of shards; a single shard cannot be used to recover the secret.

[0054] The embodiments in this specification propose solutions to improve the efficiency of model training, so as to provide both security and efficiency.

[0055] Figure 2 This is a schematic diagram illustrating an implementation scenario of another embodiment disclosed in this specification. This implementation scenario involves model training to protect privacy data. A first party has a pre-trained deep learning model, a second party has plaintext feature data and task labels for each training sample in the training sample set, and a third party has a task model downstream of the deep learning model. Secure model training is achieved through data compression combined with secret sharing, providing provable security, significantly reducing overall communication overhead, and improving the efficiency of model training.

[0056] Data compression refers to compressing a large dataset into a smaller one while preserving important information and features of the original dataset. Data compression, based on existing data compression methods, operates as a black box, transforming the original data. It can be formally expressed as: Given original data T = {(x...} i y i )}, where x i Represents plaintext feature data, y i This represents the corresponding task label; data compression can yield S = {(s i y i )}, where s i y represents the transformed ciphertext feature data. iThis still represents the corresponding task label. Research has found that the training effects based on T and S are similar, meaning that the output of the model trained using S is close to the output of the model trained using T.

[0057] Secret sharing is typically defined on rings or domains, each with its own advantages and disadvantages. Computation on rings is more efficient because its modulo operation can be implicitly handled by hardware, compared to computation on domains which requires manual modulo operations.

[0058] In the embodiments described in this specification, fixed-point numbers are mapped to rings for computation. The mapping construction process is as follows: Assume To be a rational number, we need to... Mapping to the integer field, let Where f is the precision in decimal places, i.e., the number of decimal places, and Int(·) rounds to the nearest integer. Next, for... Perform modulo operation to make Where k represents the number of bits in x, mapping the input to... On the ring.

[0059] In the embodiments of this specification, the arithmetic sharing of fixed-point numbers is in the ring. Above. In the embodiments of this specification, there are three computation methods P0, P1, and P2. The secret input x is split into three parts (x0, x1, x2), satisfying P... i Holding (x) i x i+1 ), and x = (x0 + x1 + x2) mod 2 k .use Represents the arithmetic share of x.

[0060] The following is a list of the common computational primitives used:

[0061] Addition and constant multiplication can be performed directly using the original addition secret sharing protocol. Each participant only needs to perform the calculation locally.

[0062] Multiplication, a fixed-point multiplication operation involves two key calculations. First, the participants perform integer multiplication using a standard multiplication protocol. After the multiplication is complete, due to the limited precision of fixed-point data, the result needs to be truncated. The truncation operation is used to truncate the data after doubling the precision: the last f digits are removed, i.e., divided by 2. f ;

[0063] The comparison operation compares the two arithmetic share inputs. If the input on the left is less than the input on the right, it returns a boolean share of 1; otherwise, it returns a boolean share of 0.

[0064] The embodiments in this specification protect privacy and security through data compression and secret sharing. Furthermore, the combination of these two technologies helps reduce overall communication overhead, thereby providing both security and improving the efficiency of model training.

[0065] The first party, second party, and third party can be any device, platform, server, or device cluster with computing and processing capabilities.

[0066] Figure 3 This diagram illustrates a flowchart of a model training method for protecting privacy data according to one embodiment. A first party has a pre-trained deep learning model, a second party has plaintext feature data and task labels for each training sample in a training sample set, and a third party has a task model downstream of the deep learning model. This method can be based on... Figure 1 and Figure 2 The implementation scenario shown is executed through interaction between three parties. For example... Figure 3 As shown, the model training method for protecting privacy data in this embodiment includes the following steps: Step 31, the second party compresses the plaintext feature data of each training sample in the training sample set to obtain the ciphertext feature data of each training sample in the training sample set; Step 32, the second party sends the ciphertext feature data of each training sample in the training sample set to the first party; Step 33, the first party inputs the ciphertext feature data of any training sample into the deep learning model to obtain the embedded feature vector of the training sample; Step 34, the first party uses the embedded feature vector of any training sample held by itself, the task label of the training sample provided by the second party, and the task model provided by the third party to jointly train the task model through secure multi-party computation; Step 35, the third party uses the trained task model held by itself and the deep learning model provided by the first party to jointly perform the prediction of the target task. The specific execution method of each of the above steps is described below.

[0067] First, in step 31, the second party compresses the plaintext feature data of each training sample in the training sample set to obtain the ciphertext feature data of each training sample in the training sample set. It can be understood that data compression transforms the plaintext feature data, reducing the size of the dataset composed of the plaintext feature data, or in other words, reducing the size of the training sample set.

[0068] For example, a second party has plaintext feature data X and task label y. The plaintext feature data is compressed to obtain ciphertext feature data X' and task label y. This data compression achieves differential privacy, meaning it is difficult to deduce the original feature data X from the transformed X'.

[0069] In one example, the data compression is performed by matching the feature distributions of ciphertext feature data and plaintext feature data in different sampling embedding spaces to compress a first dataset composed of plaintext feature data into a second dataset composed of ciphertext feature data.

[0070] Furthermore, the data compression includes:

[0071] Randomly generate initial synthetic data;

[0072] Using the maximum average difference between the second distribution of the synthetic data and the first distribution of the plaintext feature data as a loss function, with the goal of minimizing the value of the loss function, the adjusted synthetic data is determined as the ciphertext feature data through multiple rounds of iteration.

[0073] Then, in step 32, the second party sends the encrypted feature data of each training sample in the training sample set to the first party. It is understood that there is a difference between the encrypted feature data and the plaintext feature data, ensuring that the plaintext feature data will not be leaked to the first party.

[0074] For example, the second party sends the encrypted feature data X' obtained after data compression to the first party, while keeping the task label y locally.

[0075] Next, in step 33, the first party inputs the encrypted feature data of any training sample into the deep learning model to obtain the embedded feature vector of that training sample. It is understood that this step can be completed locally by the first party without communication with other parties.

[0076] For example, the first party has a pre-trained deep learning model M. The first party takes the encrypted feature data X′ as input and calculates the embedded feature vector Emb = M(X′) locally through its own deep learning model M.

[0077] In one example, the deep learning model is implemented based on a multi-layered transformer.

[0078] In this example, the deep learning model is relatively large and can be referred to as a large model.

[0079] In step 34, the first party uses the embedded feature vector of any training sample held by itself, the task label of the training sample provided by the second party, and the task model provided by the third party to jointly train the task model through secure multi-party computation. It is understood that the third party can be a secure multi-party computation system, which jointly trains the task model with the first and second parties.

[0080] In one example, the embedded feature vector is split into three slices, with the first party, the second party, and the third party each holding two slices; the task label is split into three slices, with the first party, the second party, and the third party each holding two slices.

[0081] Furthermore, the joint training of the task model through secure multi-party computation includes:

[0082] The task results of the training samples are divided into three parts by a first joint computation; the task results are obtained by using the embedded feature vector as the input of the task model; the task results are divided into three parts, and each party holds two parts of the task results.

[0083] Using the task result shards and the task label shards, the shards of the total prediction loss are determined through a second joint calculation; the total prediction loss is split into three shards, with each party holding two shards of the total prediction loss; the total prediction loss is used by the third party to adjust the model parameters of the task model.

[0084] Furthermore, the task model is a multilayer perceptron; the first joint computation includes secure multiplication and secure comparison operations; the second joint computation includes an approximate secure operation of exponentiation.

[0085] In the embodiments of this specification, the first joint computation is used to implement the corresponding operations of the activation function, including safe multiplication and safe comparison operations; the second joint computation is used to implement the corresponding operations of cross-entropy, including an approximate safe operation of exponentiation, wherein the exponentiation operation can be approximated using Taylor expansion.

[0086] Finally, in step 35, the third party jointly performs the prediction of the target task using the trained task model held by itself and the deep learning model provided by the first party. It is understood that performing the prediction of the target task requires both the deep learning model and the task model.

[0087] In the embodiments of this specification, when performing the prediction of the target task, a method similar to training the task model can be adopted. For example, the first party has a pre-trained deep learning model, the second party has plaintext feature data of the target sample, and the third party has the trained task model. First, the second party compresses the plaintext feature data of the target sample to obtain the ciphertext feature data of the target sample. Then, the second party sends the ciphertext feature data of the target sample to the first party. Next, the first party inputs the ciphertext feature data of the target sample into the deep learning model to obtain the embedded feature vector of the target sample. Then, the first party uses the embedded feature vector of the target sample held by itself and the task model provided by the third party to jointly perform the prediction of the target task through secure multi-party computation, for example, predicting the category of the target sample.

[0088] Figure 4 This diagram illustrates a model training process for protecting privacy data according to one embodiment. (Refer to...) Figure 4 A multi-party secure computation system has a task model to be trained. The model holder has a pre-trained large model M, and the data provider has plaintext feature data X and task labels y for each training sample in the training sample set. First, the data holder locally transforms the original data using data compression to obtain ciphertext feature data X' and task labels y for each training sample. Then, the data holder sends the compressed ciphertext feature data X' to the model holder. Next, the model holder uses X' as input and, through its own large model M, locally calculates the embedding feature vector Emb = M(X') for the training samples. Finally, the model holder provides the embedding feature vector Emb as input, and the data holder provides the task labels y as input, jointly training a downstream task-specific model with the multi-party secure computation system. This task model can be designed as a simple multilayer perceptron, involving only multiplication and comparison operations, and can be implemented using a typical secret-sharing scheme. Through multiple iterations, a fine-tuned task-specific model is obtained, completing the fine-tuning of the large model. In this process, the data holder sends the compressed encrypted feature data X' to the model holder, where data compression provides privacy protection; and the downstream task-specific model is jointly trained, where multi-party secure computation provides privacy protection.

[0089] The method provided in the embodiments of this specification involves a first party having a pre-trained deep learning model, a second party having plaintext feature data and task labels for each training sample in a training sample set, and a third party having a task model downstream of the deep learning model. The first party first receives encrypted feature data for each training sample in the training sample set from the second party; the encrypted feature data is obtained by compressing the plaintext feature data of each training sample in the training sample set. Then, the encrypted feature data of any training sample is input into the deep learning model to obtain the embedded feature vector of that training sample. Finally, using the embedded feature vector of any training sample held by the first party, the task label of the training sample provided by the second party, and the task model provided by the third party, the task model is jointly trained through secure multi-party computation. The trained task model and the deep learning model jointly achieve the prediction of the target task. As can be seen from the above, the embodiments of this specification protect privacy data during model training based on data compression and secure multi-party computation. Deep learning models are typically large in scale, while task models are typically small in scale. Joint training of the smaller task model requires only a small amount of communication overhead, thereby improving the efficiency of model training. The second party compresses the plaintext feature data of each training sample to obtain ciphertext feature data, and then sends this ciphertext feature data to the first party. This avoids the leakage of plaintext feature data and reduces the amount of communication between the two parties. Furthermore, the first party performs deep learning model computation locally based on the received ciphertext feature data, without needing to communicate with other parties. Additionally, the task model is jointly trained through secure multi-party computation, preventing the privacy leakage of the embedded feature vectors obtained by the first party and the task labels possessed by the second party. In summary, this approach provides both security and improves model training efficiency.

[0090] According to another embodiment, a model training apparatus for protecting privacy data is also provided. A first party has a pre-trained deep learning model, a second party has plaintext feature data and task labels for each training sample in a training sample set, and a third party has a task model downstream of the deep learning model. The apparatus is disposed within the first party and is used to execute the model provided in the embodiments of this specification. Figure 3 The action of the first party in the method shown. Figure 5 A schematic block diagram of a model training apparatus for protecting privacy data according to one embodiment is shown. Figure 5 As shown, the device 500 includes:

[0091] The receiving unit 51 is used to receive the ciphertext feature data of each training sample in the training sample set from the second party. The ciphertext feature data is obtained by compressing the plaintext feature data of each training sample in the training sample set.

[0092] The embedding processing unit 52 is used to input the encrypted feature data of any training sample received by the receiving unit 51 into the deep learning model to obtain the embedding feature vector of the training sample.

[0093] The joint training unit 53 is used to jointly train the task model by using the embedding feature vector of any training sample obtained by the embedding processing unit 52 held by the party, the task label of the training sample provided by the second party, and the task model provided by the third party, through secure multi-party computation. The trained task model and the deep learning model jointly achieve the prediction of the target task.

[0094] According to another embodiment, a model training apparatus for protecting privacy data is also provided. A first party has a pre-trained deep learning model, a second party has plaintext feature data and task labels for each training sample in a training sample set, and a third party has a task model downstream of the deep learning model. The apparatus is disposed in the second party and is used to execute the model provided in the embodiments of this specification. Figure 3 The action of the second party in the method shown. Figure 6 A schematic block diagram of a model training apparatus for protecting privacy data according to another embodiment is shown. Figure 6 As shown, the device 600 includes:

[0095] Compression processing unit 61 is used to compress the plaintext feature data of each training sample in the training sample set to obtain the ciphertext feature data of each training sample in the training sample set.

[0096] The sending unit 62 is used to send the encrypted feature data obtained by the compression processing unit 61 to the first party, so that the first party can input the encrypted feature data of any training sample into the deep learning model to obtain the embedded feature vector of the training sample.

[0097] The joint training unit 63 is used to jointly train the task model by using the task label of any training sample held by the party, the embedded feature vector of the training sample provided by the first party, and the task model provided by the third party through secure multi-party computation. The trained task model and the deep learning model jointly achieve the prediction of the target task.

[0098] According to another embodiment, a model training apparatus for protecting privacy data is also provided. A first party has a pre-trained deep learning model, a second party has plaintext feature data and task labels for each training sample in a training sample set, and a third party has a task model downstream of the deep learning model. The apparatus is disposed at the third party and is used to execute the model provided in the embodiments of this specification. Figure 3 The actions of a third party in the method shown. Figure 7 A schematic block diagram of a model training apparatus for protecting privacy data according to another embodiment is shown. Figure 7 As shown, the device 700 includes:

[0099] The joint training unit 71 is used to jointly train the task model by using the task model held by this party, the embedded feature vector of any training sample provided by the first party, and the task label of the training sample provided by the second party through secure multi-party computation.

[0100] The joint prediction unit 72 is used to jointly perform the prediction of the target task using the task model trained by the joint training unit held by the party and the deep learning model provided by the first party.

[0101] Using the apparatus provided in the embodiments of this specification, the receiving unit 51 of the first party first receives the encrypted feature data of each training sample in the training sample set from the second party. The encrypted feature data is obtained by compressing the plaintext feature data of each training sample in the training sample set. Then, the embedding processing unit 52 of the first party inputs the encrypted feature data of any training sample into the deep learning model to obtain the embedding feature vector of the training sample. Finally, the joint training unit 53 of the first party uses the embedding feature vector of any training sample held by itself, the task label of the training sample provided by the second party, and the task model provided by the third party to jointly train the task model through secure multi-party computation. The trained task model and the deep learning model jointly achieve the prediction of the target task. As can be seen from the above, the embodiments of this specification protect privacy data in model training based on data compression and secure multi-party computation. The deep learning model is usually large in scale, while the task model is usually small in scale. Joint training of the small task model requires only a small amount of communication overhead, which improves the efficiency of model training. The second party compresses the plaintext feature data of each training sample to obtain ciphertext feature data, and then sends this ciphertext feature data to the first party. This avoids the leakage of plaintext feature data and reduces the amount of communication between the two parties. Furthermore, the first party performs deep learning model computation locally based on the received ciphertext feature data, without needing to communicate with other parties. Additionally, the task model is jointly trained through secure multi-party computation, preventing the privacy leakage of the embedded feature vectors obtained by the first party and the task labels possessed by the second party. In summary, this approach provides both security and improves model training efficiency.

[0102] According to another embodiment, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed in a computer, causes the computer to perform a combination Figure 3 The method described.

[0103] According to another embodiment, a computing device is also provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements a combination... Figure 3 The method described.

[0104] Those skilled in the art will recognize that, in one or more of the examples above, the functions described in this invention can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.

[0105] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.

Claims

1. A model training method for protecting privacy data, wherein a first party has a pre-trained deep learning model, a second party has plaintext feature data and task labels for each training sample in a training sample set, and a third party has a task model downstream of the deep learning model, the method being executed by the first party, comprising: The second party receives the ciphertext feature data of each training sample in the training sample set, wherein the ciphertext feature data is obtained by compressing the plaintext feature data of each training sample in the training sample set. The encrypted feature data of any training sample is input into the deep learning model to obtain the embedded feature vector of the training sample, wherein the calculation using the deep learning model is completed locally by the first party based on the received encrypted feature data. By utilizing the embedded feature vector of any training sample held by this party, the task label of the training sample provided by the second party, and the task model provided by the third party, the task model is jointly trained through secure multi-party computation. The trained task model and the deep learning model together achieve the prediction of the target task.

2. The method as described in claim 1, wherein, The deep learning model is implemented based on a multi-layered transformer.

3. The method as described in claim 1, wherein, The embedded feature vector is split into three pieces, with the first party, the second party, and the third party each holding two pieces; the task label is split into three pieces, with the first party, the second party, and the third party each holding two pieces.

4. The method of claim 3, wherein, The method of jointly training the task model through secure multi-party computation includes: The task results of the training samples are divided into three parts by a first joint computation; the task results are obtained by using the embedded feature vector as the input of the task model; the task results are divided into three parts, and each party holds two parts of the task results. Using the task result shards and the task label shards, the shards of the total prediction loss are determined through a second joint calculation; the total prediction loss is split into three shards, with each party holding two shards of the total prediction loss; the total prediction loss is used by the third party to adjust the model parameters of the task model.

5. The method of claim 4, wherein, The task model is a multilayer perceptron; the first joint computation includes secure multiplication and secure comparison operations; the second joint computation includes an approximate secure operation of exponentiation.

6. A model training method for protecting privacy data, wherein a first party has a pre-trained deep learning model, a second party has plaintext feature data and task labels for each training sample in a training sample set, and a third party has a task model downstream of the deep learning model, the method being executed by the second party, comprising: The plaintext feature data of each training sample in the training sample set is compressed to obtain the ciphertext feature data of each training sample in the training sample set. Each encrypted feature data is sent to the first party, so that the first party can input the encrypted feature data of any training sample into the deep learning model to obtain the embedded feature vector of the training sample. The calculation using the deep learning model is completed locally by the first party based on the received encrypted feature data. Using the task label of any training sample held by this party, the embedded feature vector of the training sample provided by the first party, and the task model provided by the third party, the task model is jointly trained through secure multi-party computation. The trained task model and the deep learning model together achieve the prediction of the target task.

7. The method of claim 6, wherein, The data compression is achieved by matching the feature distributions of ciphertext feature data and plaintext feature data in different sampling embedding spaces, thereby compressing a first dataset composed of plaintext feature data into a second dataset composed of ciphertext feature data.

8. The method of claim 7, wherein, The data compression includes: Randomly generate initial synthetic data; Using the maximum average difference between the second distribution of the synthetic data and the first distribution of the plaintext feature data as a loss function, with the goal of minimizing the value of the loss function, the adjusted synthetic data is determined as the ciphertext feature data through multiple rounds of iteration.

9. A model training method for protecting privacy data, wherein a first party has a pre-trained deep learning model, a second party has plaintext feature data and task labels of each training sample in a training sample set, and a third party has a task model downstream of the deep learning model, the method being executed by the third party, comprising: Using the task model held by this party, the embedded feature vector of any training sample provided by the first party, and the task label of the training sample provided by the second party, the task model is jointly trained through secure multi-party computation. The embedded feature vector is obtained by the first party by inputting the ciphertext feature data of any training sample into the deep learning model. The ciphertext feature data is obtained by compressing the plaintext feature data of each training sample in the training sample set. The computation of the deep learning model is performed locally by the first party based on the received ciphertext feature data. The target task prediction is jointly performed using the trained task model held by this party and the deep learning model provided by the first party.

10. A model training apparatus for protecting privacy data, wherein a first party has a pre-trained deep learning model, a second party has plaintext feature data and task labels of each training sample in a training sample set, and a third party has a task model downstream of the deep learning model, the apparatus being disposed in the first party, comprising: The receiving unit is configured to receive from the second party the ciphertext feature data of each training sample in the training sample set, wherein the ciphertext feature data is obtained by compressing the plaintext feature data of each training sample in the training sample set. An embedding processing unit is used to input the encrypted feature data of any training sample received by the receiving unit into the deep learning model to obtain the embedding feature vector of the training sample, wherein the calculation using the deep learning model is completed locally by the first party based on the received encrypted feature data. The joint training unit is used to jointly train the task model by using the embedding feature vector of any training sample obtained by the embedding processing unit held by this party, the task label of the training sample provided by the second party, and the task model provided by the third party, through secure multi-party computation. The trained task model and the deep learning model jointly achieve the prediction of the target task.

11. A model training apparatus for protecting privacy data, comprising: a first party having a pre-trained deep learning model; a second party having plaintext feature data and task labels for each training sample in a training sample set; and a third party having a task model downstream of the deep learning model; the apparatus being disposed in the second party, comprising: The compression processing unit is used to compress the plaintext feature data of each training sample in the training sample set to obtain the ciphertext feature data of each training sample in the training sample set. The sending unit is used to send the encrypted feature data obtained by the compression processing unit to the first party, so that the first party can input the encrypted feature data of any training sample into the deep learning model to obtain the embedded feature vector of the training sample, wherein the calculation using the deep learning model is completed locally by the first party based on the received encrypted feature data. The joint training unit is used to jointly train the task model by using the task label of any training sample held by the party, the embedded feature vector of the training sample provided by the first party, and the task model provided by the third party through secure multi-party computation. The trained task model and the deep learning model jointly achieve the prediction of the target task.

12. A model training apparatus for protecting privacy data, comprising: a first party having a pre-trained deep learning model; a second party having plaintext feature data and task labels for each training sample in a training sample set; and a third party having a task model downstream of the deep learning model; the apparatus being disposed in the third party, comprising: The joint training unit is used to jointly train the task model using the task model held by this party, the embedded feature vector of any training sample provided by the first party, and the task label of the training sample provided by the second party through secure multi-party computation. The embedded feature vector is obtained by the first party by inputting the ciphertext feature data of any training sample into the deep learning model. The ciphertext feature data is obtained by compressing the plaintext feature data of each training sample in the training sample set. The computation of the deep learning model is performed locally by the first party based on the received ciphertext feature data. The joint prediction unit is used to jointly perform the prediction of the target task using the task model trained by the joint training unit held by the party and the deep learning model provided by the first party.

13. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method of any one of claims 1-9.

14. A computing device comprising a memory and a processor, wherein the memory stores executable code, and the processor, when executing the executable code, implements the method of any one of claims 1-9.

Images