Method and apparatus for two-party joint training of a prediction model protecting data privacy

Abstract

Embodiments of the present specification provide a method and device for two-party joint training of a prediction model for protecting data privacy. Each of the two parties holds different feature parts of a plurality of users, and a representation layer for different feature parts is deployed, wherein the label party further deploys a prediction layer. The method comprises: each of the two parties locally processes feature parts of the same batch of user samples to obtain corresponding in-order representations, and the non-label party further generates a disordering matrix; then, the label party performs secure multi-party computation based on the in-order representation thereof, the in-order representation in the non-label party and the transformation matrix to obtain a disordering fusion representation, thereby determining a disordering gradient propagated to the two representation layers based on a disordering prediction result obtained by processing the disordering fusion representation using the prediction layer, and a disordering prediction label obtained by performing secure matrix multiplication based on the in-order user label and the disordering matrix in the non-label party, and then the non-label party determines an in-order gradient based on the disordering matrix and the disordering gradient for updating the local representation layer of each of the two parties.

Description

TECHNICAL FIELD

[0001] One or more embodiments of the present specification relate to the technical field of data security, and in particular to a method and device for two-party joint training of a prediction model for protecting data privacy. BACKGROUND

[0002] With the development of artificial intelligence technology (AI), deep neural networks (DNN) have gradually been applied to many fields such as risk assessment, speech recognition, facial recognition, and natural language processing. The DNN network structure in different application scenarios is relatively fixed, and more training data is needed to achieve better model performance.

[0003] In many fields such as medical care and finance, different enterprises or institutions have different data samples. Once these data are jointly trained, the model accuracy will be greatly improved, thereby effectively improving the business quality and user satisfaction. However, the original training data contains a large amount of user privacy and business secrets, and once the information is leaked, it will cause irreparable negative effects. Therefore, while solving the data island problem through multi-party joint training, protecting data privacy and data value has become the focus of research in recent years.

[0004] If the training members participating in joint training are in different industries and have the same user group, for example, between a payment platform and a social platform, the payment platform holds more payment data, and the social platform holds more entertainment and social data. The feature spaces of the data collected by them are different, and the sample spaces are the same. This can be referred to as a vertically divided data scenario.

[0005] However, the current joint training methods applied in the vertically divided data scenario are limited, and it is difficult to meet the higher demands in actual applications. Therefore, a joint training scheme is needed that can better meet the actual application requirements, such as improving the model prediction accuracy while effectively protecting data privacy and data value, and reducing communication and computation. SUMMARY

[0006] The embodiments of the present specification describe a method for two-party joint training of a prediction model for protecting data privacy, which performs matrix row transformation, secret sharing, and other operations to effectively protect the privacy and data value of the training data during the training process, while ensuring the model prediction accuracy and effectively reducing the communication and computation.

[0007] According to a first aspect, a method for two parties to jointly train a prediction model for protecting data privacy is provided, wherein the prediction model includes a first representation layer and a prediction layer deployed in the first party, and a second representation layer deployed in the second party; the method is applied to the first party, including: using the first representation layer to process the first feature part of n sample users to obtain a first positive order representation; based on the first positive order representation, performing secure multi-party computation with the second positive order representation held by the second party and an n-order scrambled matrix to determine a scrambled fusion representation for the two positive order representations; the second positive order representation is obtained by using the second representation layer to process the second feature part of the n sample users; based on the scrambled prediction result and the scrambled user label, determining the scrambled gradient propagated to the two representation layers; the scrambled prediction result is obtained by using the prediction layer to process the scrambled fusion representation; the scrambled user label is obtained by performing secure matrix multiplication with the scrambled matrix held by the second party based on the positive order user label; and receiving a target gradient from the second party, which is determined based on the scrambled gradient and the scrambled matrix.

[0008] In one embodiment, determining the out-of-order fusion representation includes: performing security matrix multiplication on the first positive-order representation and the random matrix held by the second party to obtain a first slice of the first random representation, where the first random representation corresponds to the product of the first positive-order representation and the random matrix; receiving an intermediate value from the second party, which is determined based on a second slice of the first random representation and a second random representation, where the second random representation is obtained by multiplying the second positive-order representation using the random matrix; and determining the out-of-order fusion representation based on the first slice and the intermediate value.

[0009] In one embodiment, the determination of the out-of-order user tag includes: performing a security matrix multiplication on the positive-order user tag and the out-of-order square matrix held by the second party to obtain the first tag fragment for the out-of-order user tag; receiving the second tag fragment for the out-of-order user tag from the second party; and constructing the out-of-order user tag based on the first tag fragment and the second tag fragment.

[0010] In a specific embodiment, after receiving the target gradient from the second party, the method further includes: using the target gradient to update the model parameters in the first representation layer.

[0011] In one embodiment, the method further includes: determining a training gradient propagated to the prediction layer based on the out-of-order prediction results and the out-of-order user labels; and updating model parameters in the prediction layer using the training gradient.

[0012] According to a second aspect, a method for training a prediction model in a two-party joint manner to protect data privacy is provided, the prediction model comprising a first representation layer and a prediction layer deployed in a first party, and a second representation layer deployed in a second party; the method is applied to the second party and comprises: processing a second feature part of n sample users by using the second representation layer to obtain a second in-order representation; performing secure multi-party computation with a first in-order representation held by the first party based on the second in-order representation and an n-order disordered matrix, so that the first party obtains a disordered fusion representation for the two in-order representations; receiving a disordered gradient propagated to the two representation layers from the first party, the disordered gradient being determined based on a disordered prediction result and a disordered user label, the disordered prediction result being obtained by the first party by processing the disordered fusion representation by using the prediction layer, and the disordered user label being obtained by the first party based on an in-order user label and performing secure matrix multiplication with the disordered matrix held by the second party; and determining a target gradient based on the disordered gradient and the disordered matrix and sending the target gradient to the first party.

[0013] In one embodiment, the determination of the disordered matrix comprises: the second party performing random disordered processing on an n-order unit matrix in a column direction or a row direction to obtain the disordered matrix.

[0014] In one embodiment, the secure multi-party computation with the first in-order representation held by the first party based on the second in-order representation and the n-order disordered matrix, so that the first party obtains a disordered fusion representation for the two in-order representations, comprises: performing secure matrix multiplication with the first in-order representation held by the first party based on the disordered matrix to obtain a second shard of a first disordered representation corresponding to a product between the first in-order representation and the disordered matrix; performing product processing on the second in-order representation by using the disordered matrix to obtain a second disordered representation; and determining the intermediate value based on the second disordered representation and the second shard and sending the intermediate value to the first party, so that the first party determines the disordered fusion representation based on a first shard of the first disordered representation and the intermediate value.

[0015] In one specific embodiment, the determination of the intermediate value based on the second disordered representation and the second shard comprises: performing summation processing on the second disordered representation and the second shard to obtain the intermediate value.

[0016] In one embodiment, before receiving the disordered gradient propagated to the two representation layers from the first party, the method comprises: performing secure matrix multiplication with the in-order user label held by the first party based on the disordered matrix to obtain a second label shard for the disordered user label; and sending the second label shard to the first party, so that the first party constructs the disordered user label based on the second label shard and a first label shard for the disordered user label.

[0017] In one embodiment, determining the target gradient based on the disordered gradient and the disordered matrix comprises: multiplying the disordered gradient by an inverse matrix of the disordered matrix to obtain an in-order gradient; and determining the target gradient based on the in-order gradient.

[0018] In one specific embodiment, determining the target gradient based on the in-order gradient comprises: sparsifying the in-order gradient to obtain a sparse gradient; and determining the target gradient based on the sparse gradient.

[0019] In a more specific embodiment, sparsifying the in-order gradient to obtain a sparse gradient comprises: randomly selecting a predetermined proportion of elements from the in-order gradient and replacing them with 0 values.

[0020] In another more specific embodiment, determining the target gradient based on the sparse gradient comprises: performing pooling processing on the sparse gradient while retaining the original signs to obtain the target gradient.

[0021] In another specific embodiment, determining the target gradient based on the in-order gradient comprises: performing pooling processing on the in-order gradient while retaining the original signs to obtain the target gradient.

[0022] According to a third aspect, there is provided an apparatus for two-party joint training of a prediction model for protecting data privacy, the prediction model comprising a first representation layer and a prediction layer deployed in a first party, and a second representation layer deployed in a second party; the apparatus is integrated into the first party and comprises: a representation unit configured to process first feature parts of n sample users using the first representation layer to obtain first in-order representations; a secure computation unit configured to perform secure multi-party computation based on the first in-order representations, a second in-order representation held by the second party, and an n-order disordered matrix to determine a disordered fusion representation for the two in-order representations; the second in-order representation is obtained by processing second feature parts of the n sample users using the second representation layer; a disordered gradient determination unit configured to determine a disordered gradient propagated to the two representation layers based on a disordered prediction result and a disordered user label; the disordered prediction result is obtained by processing the disordered fusion representation using the prediction layer; the disordered user label is obtained by performing secure matrix multiplication based on an in-order user label and the disordered matrix held by the second party; and a target gradient receiving unit configured to receive a target gradient from the second party, the target gradient being determined based on the disordered gradient and the disordered matrix.

[0023] According to a fourth aspect, there is provided another apparatus for two-party joint training of a prediction model for protecting data privacy, the prediction model comprising a first representation layer and a prediction layer deployed in a first party, and a second representation layer deployed in a second party; the apparatus is integrated into the second party, and comprises: a representation unit configured to process a second feature part of n sample users by using the second representation layer to obtain a second in-order representation; a secure calculation unit configured to perform secure multi-party computation with the first party on the basis of the second in-order representation and an n-order disordered matrix to enable the first party to obtain a disordered fusion representation for two in-order representations; a disordered gradient receiving unit configured to receive a disordered gradient propagated to two representation layers from the first party, the disordered gradient being determined on the basis of a disordered prediction result and a disordered user label, the disordered prediction result being obtained by the first party by processing the disordered fusion representation by using the prediction layer, and the disordered user label being obtained by the first party by performing secure matrix multiplication on the basis of an in-order user label and the disordered matrix held by the second party; a target gradient determining unit configured to determine a target gradient on the basis of the disordered gradient and the disordered matrix; and a target gradient sending unit configured to send the target gradient to the first party.

[0024] According to a fifth aspect, there is provided a computer-readable storage medium having stored thereon a computer program which, when executed in a computer, causes the computer to perform the method of the first aspect or the second aspect.

[0025] According to a sixth aspect, there is provided a computing device comprising a memory and a processor, the memory having stored therein executable code which, when executed by the processor, implements the method of the first aspect or the second aspect.

[0026] In the above method and apparatus provided by the embodiments of the present disclosure, the matrix row transformation is combined with secure multi-party computation to process the representation data and the label data, thereby protecting the value of the representation data of the label-free training member and the privacy of the label data of the label training member; the label training member performs clear-text calculation on the loss function by using the clear-text representation (disordered representation), thereby greatly saving the calculation time; the label-free training member encrypts the gradient propagated in the reverse direction by using the sparse or pooling method, thereby protecting the disordered matrix and effectively reducing the communication amount and the calculation amount. BRIEF DESCRIPTION OF DRAWINGS

[0027] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings needed in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present disclosure, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0028] Figure 1 Fig. 2 shows a schematic diagram of a scenario of user prediction model decentralized deployment according to one embodiment;

[0029] Figure 2 Fig. 3 shows a schematic diagram of two-party interaction for jointly training a prediction model according to one embodiment;

[0030] Figure 3 Fig. 4 shows a schematic diagram of two-party interaction for determining out-of-order fusion representation according to one embodiment;

[0031] Figure 4 Fig. 5 shows an apparatus for two-party jointly training a prediction model to protect data privacy according to one embodiment;

[0032] Figure 5 Fig. 6 shows an apparatus for two-party jointly training a prediction model to protect data privacy according to another embodiment. DETAILED DESCRIPTION

[0033] The schemes provided in the present specification will be described below with reference to the accompanying drawings.

[0034] The present specification discloses a scheme for jointly training a prediction model, which can better protect the value and privacy safety of user data in training members, while ensuring model accuracy and effectively reducing communication and computation.

[0035] For ease of understanding, the data distribution in two training members (or two parties) participating in joint training will be introduced first. The two parties each hold different feature parts of the same sample users, and one of them also holds user labels. For brevity of description, the label-holding party in the two parties is referred to as the first party, denoted as P A , and the other party is referred to as the second party, denoted as P B .

[0036] In one embodiment, P A is a payment platform, which holds transaction features or payment features of users, such as payment time, payment frequency, payment amount, payment channel, transaction object, etc.; P B is a social platform, which holds social features of users, such as a list of friends with social frequency higher than a predetermined threshold, forwarding and liking social dynamic and content information, social activity, etc.; in addition, P A holds user labels, which can be user risk labels (such as high-risk users or low-risk users), or consumer group labels (such as high-consumption groups or low-consumption groups), etc.

[0037] In another embodiment, P A is a bank, which holds bank flow (such as salary, expenditure, etc.) of users; P BThe credit reporting structure holds the user's credit records (such as loan records, credit card activation records, etc.); In addition, P A The user label held by the party may be a user risk label (such as a high-risk user or a low-risk user), etc.

[0038] The above describes the distribution of user data between the two parties. Next, we will describe the deployment of the prediction model between the two parties. Figure 1 A schematic diagram of a scenario in which a prediction model is distributed according to an embodiment is shown. Figure 1 As shown, the prediction model consists of two representation layers, namely, deployed on P A The first representation layer 110 and the first representation layer 110 deployed in P B The second representation layer 120 in the embodiment further includes a layer deployed on the P A The prediction layer 130 in the embodiment of the present invention. It should be noted that the two representation layers can be defined by each party according to the dimensions of the user features held by the two parties, and the prediction layer 130 can be defined by P A The method is defined based on the output dimensions of the two representation layers and the dimensions of the user tags. In addition, both the representation layer and the prediction layer can be implemented using neural network layers. For example, the first representation layer 110 or the second representation layer 120 is implemented as multiple sets of alternating fully connected layers and activation layers, and the prediction layer 130 can be implemented as several (several refers to one or more) fully connected layers, as well as sigmoid layers or softmax layers, etc. In addition, the prediction model used to process user data can also be called a user prediction model, and the representation layer and prediction layer contained therein can be called a user representation layer and a user prediction layer respectively.

[0039] Thus, the deployment of the prediction model in the two training members is introduced. The following describes the implementation steps of the above training scheme in conjunction with an embodiment. Figure 2 The diagram of the interaction between two parties of the joint training prediction model according to one embodiment is shown. It can be understood that both parties can be implemented as any device, platform, server or equipment cluster with computing and processing capabilities. Figure 2 As shown, the interaction process includes the following steps:

[0040] In step S201, P A The first representation layer 110 (where the model parameters are denoted as θ A ) Process the first feature part F of n sample users A , we get the first positive sequence representation H A .

[0041] It is understood that the training of the prediction model involves multiple iterations. In the current round of training, in one embodiment, P AFang can randomly select multiple (denoted as n) sample users from the sample user set as the sample users used in this round of training.

[0042] Furthermore, if the current round is the first round, this step may also include obtaining the above-mentioned sample user set. In one embodiment, P A Fang and P B Each party executes the Private Set Intersection (PSI) protocol based on the user identification set it holds, and obtains the identification intersection as the sample user set. In one example, the user identification can be the user's ID number, passport number, or mobile phone number. In one example, the execution process of the PSI protocol is as follows: each party calculates the hash value of the user identification locally and uploads it to the server, and then receives the matching hash value intersection from the server. Then, each party determines the identification intersection based on the mapping relationship between the user identification and the hash value, which is used as the above-mentioned sample user set.

[0043] In addition, if the current round is the first round, this step can also include P A The model parameters θ in the first representation layer 110 A Perform random initialization.

[0044] In another embodiment, P A Fang can be obtained from P B The party receives n user identifications corresponding to n sample users, and these n user identifications are generated by P B The sample user set obtained by the party is randomly sampled.

[0045] From the above, P A The n sample users to be used in this round of training can be determined, and then the first user feature f of each sample user can be processed separately using the first representation layer 110. A , get the corresponding first user representation h A In one embodiment, the first user feature f is represented by several neural network layers in the first representation layer 110. A Perform linear transformation and / or nonlinear transformation to obtain the first user representation h A .

[0046] It can be understood that the n first user features f corresponding to n sample users A The first characteristic part F A The dimension of the first user representation may be recorded as m, thus, n m-dimensional first user representation vectors h A Splice them into a representation matrix by row or column as the first positive sequence representation H A , where positive order refers to the correct order, which is also the original order.

[0047] From the above, P A may obtain the first positive sequence representation H A by processing the local user features with the local representation layer. B Similarly, in step S202, P B may obtain the second positive sequence representation H B by processing the second feature part F B of the n sample users with the second representation layer 120 (whose model parameters are denoted as θ B It can be understood that the sorting of the n second user representations in the second positive sequence representation H A is consistent with the sorting of the n first user representations in the first positive sequence representation H

[0048] It should be noted that, in a preferred mode, the number of neurons in the last neural network layer of the second representation layer 120 is designed to be the same as the number of neurons in the last neural network layer of the first representation layer 110, both being m. At this time, the dimension of the second positive sequence representation H B is the same as the dimension of the first positive sequence representation H A , both being n*m or m*n. By making such a design, the subsequent two positive sequence representations can be fused by addition, so that the training gradient propagated to the two representation layers is the same, so that the party that obtains the gradient plaintext will not obtain additional information.

[0049] In addition, the description of the above step S202 can refer to the description of step S201, which will not be repeated here.

[0050] In step S203, P B generates an n-order disordered square matrix S (or transformation square matrix S). It should be understood that the disordered square matrix S acts on another matrix with n rows, which can realize the disordering of the row order in the matrix, for example, the original first row becomes the third row. Similarly, if it acts on another matrix with n columns, it can realize the disordering of the column order in the matrix, for example, the original fifth column becomes the seventh column.

[0051] In one embodiment, P B first generates an n-order unit matrix, and then randomly disorderedly processes the unit matrix by row or by column to obtain the disordered square matrix S. In another embodiment, P B randomly generates an n-order disordered square matrix, which satisfies that there is only one non-zero value in each row and each column. At this time, when the disordered square matrix acts on other matrices, it may also play a role of scaling or taking the opposite number, etc.

[0052] After performing steps S201, S202 and S203, the tag holding party P AFang holds the first positive order representation H A , unlabeled square P B Fang holds the second positive order representation H B And the random matrix S.

[0053] Based on this, step S204 can be executed. A Based on the first positive sequence representation H A , and P B The second positive order representation H held by Fang B Perform secure multi-party computation with the disordered matrix S to determine the disordered fusion representation H for the two positive order representations S . It can be interpreted literally that the disordered fusion representation is the result of fusing and disordering two positive-order representations, and the order of fusion and disorder does not affect the final result. In the case of plain text calculation (this will not be done to protect data privacy, and the description here is to assist understanding), the fusion process can be implemented based on addition, such as direct addition or averaging, and the disorder process can be implemented using a disordered square matrix S. For example, the two positive-order representations are first fused to obtain a fused representation, and then the disordered square matrix is ​​applied to the fused representation to obtain a disordered fusion representation, which can be recorded as: H S =S*(H A +H B ), where H A 、H B and H S Both are matrices of n*m dimensions.

[0054] It can be understood that secure multi-party computation (MPC) can enable multiple computing parties to securely calculate an agreed function without leaking their private data. In this step, the input of the agreed function between the two parties includes the two positive sequence representations H A 、H B And the disordered matrix S, the output is the above disordered fusion representation H S .

[0055] In one embodiment, the MPC calculation in this step can be implemented using globally designed garbled circuits (GC) or globally homomorphic encryption (HE). However, considering the high communication and computational overhead of encryption and decryption operations in this approach, the inventors have further proposed an optimized implementation that can effectively reduce the computational and communication overhead.

[0056] In another embodiment, low communication and low computation overhead are achieved by introducing secure matrix multiplication implemented with secret sharing technique. In this embodiment, the present step includes Figure 3 The interactive steps S301 to S305 are shown as follows:

[0057] Step S301, P A , based on the first normal order representation H A , performs a first secure matrix multiplication with the disordered square matrix S held by P B , and thus P A obtains one of the two secret share pieces of the product result between H A and S, denoted as the first piece <S*H A >1. B P B obtains the other secret share piece, denoted as the second piece <S*H A >2. It should be understood that the implementation of the first secure matrix multiplication is based on the secret sharing (SS) technique, and the specific calculation manner can be implemented in an existing manner, which will not be described herein. In addition, the product result S*H A between H A and S is essentially the first disordered representation obtained by applying the disordered square matrix S to the first normal order representation H B .

[0058] Step S302, P B performs a second secure matrix multiplication with the second normal order representation H B and the disordered square matrix S.

[0059] Step S303, P B , based on the second piece <S*H A >1 of the first disordered representation and the second disordered representation S*H B , determines an intermediate value <h>It is to be understood that determining the intermediate value <h>In this way, the way of slicing in the SS technique used in the first secure matrix multiplication described above is adapted.

[0060] In one example, the way of slicing in the SS technique for any data x is additive slicing, i.e., x = x0+ x1+ x2+ x3. <x> 1+ <x>2, at this time, the second slice <S*H A >2 and the second out-of-order representation S*H B determined in a manner that sums up the intermediate values, i.e.:

[0061] <h>= S * H B + <S * H A > 2 (1)

[0062] In another example, the way to slice any data x in the SS technique is subtraction slicing, e.g., x = 2 <x> 1- <x>2, at this time, the second slice <S*H A >1 and the second out-of-order representation H B The intermediate value is determined in the way of difference:

[0063] <h>= S * H B - <S * H A > 2 (2)

[0064] After that, in step S304, the P A party receives the intermediate value B from the P <h>and at step S305, based on the first fragment <S*H A >1 and the intermediate value <h>, determine the out-of-order fusion representation H S It can be understood that the determination manner of the out-of-order fusion representation H S The determination manner is also adapted to the fragmentation manner in the SS.

[0065] Exemplarily, the fragmentation manner in the SS is additive fragmentation, at this time, the first fragment <S*H A >1 and the intermediate value <h>The manner in which the summation is performed determines the out-of-order fusion representation H S That is:

[0066] H S = <S * H A >1+ <h>(3)

[0067] At this time, substituting equation (1) into the expression on the right side of equation (3), we obtain: H S = <S*H A >1+ <h>= <S*H A >1+S*H B + <S*H A >2=S*(H A +H B ).

[0068] Therefore, in this embodiment, by designing a secure matrix multiplication method using SS technology to implement the above MPC technology, the return P can be calculated efficiently and quickly. A Fang holds the out-of-order fusion representation H S .

[0069] Back to Figure 2 The process steps shown above, P A Through the P B By performing MPC calculation, we can obtain the disordered fusion representation H S It can be understood that the disordered fusion characterization H S The current order of the fusion representation vectors corresponding to n sample users in P is disrupted compared to the original order. A It is difficult to represent H by random fusion S Infer P B The plaintext representation of the party, so that P B Fangzhong’s data privacy is protected.

[0070] Then in step S205, P A The prediction layer 130 is used to process the out-of-order fusion representation H S , get the out-of-order prediction results It can be understood that the out-of-order fusion representation H S The n fusion representation vectors corresponding to the n sample users arranged in random order are included, and the corresponding random prediction results are In addition, the prediction layer 130 may output a prediction result as a regression value (ie, a continuous value, such as a credit score) or a classification value (ie, a discrete value, such as a risk level) for a sample user.

[0071] Then, in step S206, P A Based on the positive sequence user label Y, and P B The random matrix S held by the party performs the second security matrix multiplication to obtain the random user label Y S .

[0072] In one embodiment, the implementation of this step includes: A First, based on the positive sequence user label Y, and P B The second security matrix multiplication is performed on the random matrix S held by the user, and one of the two secret sharing fragments of the product result between the positive user label Y and the random matrix S is obtained, which is called the first label fragment and is recorded as <Y S >1, wherein Y S =S*Y; and P B receives a second label shard of Y S >2 obtained by performing the second secure matrix multiplication; and then P A constructs a shuffled user label Y S based on the first label shard <Y S >1 and the second label shard <Y S >2. For example, the first label shard <Y S >1 and the second label shard <Y S >2 are summed to obtain the shuffled user label Y S , which can be denoted as:

[0073] Y S =<Y S >1+<Y S >2 (4)

[0074] Accordingly, the shuffled user label Y S is obtained by performing the second secure matrix multiplication based on the SS technique. It should be noted that, although P A has the in-order user label and the shuffled user label Y S , it is still difficult to deduce the shuffled matrix S, because the in-order user label is usually a one-dimensional vector with element values of 0 or 1, that is, a non-full rank square matrix, so there is no inverse matrix, and thus the shuffled matrix S cannot be deduced.

[0075] In other embodiments, the second secure matrix multiplication can also be implemented based on the GC circuit or the HE encryption technique, which will not be described herein.

[0076] In the above, P A obtains the shuffled prediction result and the shuffled user label Y S in steps S205 and S206, respectively, and thus determines the shuffled gradient G S propagated to the two representation layers based on and Y S in step S207.

[0077] Specifically, the shuffled prediction result and the shuffled user label Y S are used to determine a training loss, and then the back propagation method is used to determine the training gradient propagated to the prediction layer 130 and the shuffled gradient G S propagated to the two representation layers based on the training loss. In one embodiment, the present step can further include that P A The first gradient is used to update the model parameters β in the prediction layer 130. It should be understood that the gradient propagated to the first representation layer 110 and the second representation layer 120 is the same, which is the disorder gradient G S , which includes n gradient vectors of n sample users corresponding to the disorder.

[0078] Then, in step S208, the P B party updates the model parameters β in the prediction layer 130 based on the disorder gradient G A . S Thus, in step S209, the P B party determines the target gradient G S based on the disorder gradient G t and the disorder matrix S.

[0079] Specifically, the disorder gradient G -1 is multiplied by the inverse matrix S S of the disorder matrix S to obtain the correctly ordered gradient matrix, i.e., the in-order gradient G.

[0080] Further, in an embodiment, the in-order gradient G t may be used as the target gradient G .

[0081] In another embodiment, in order to protect the disorder matrix S from being leaked, the in-order gradient G is encrypted. It should be understood that the target gradient G t obtained through the encryption process is different from the in-order gradient G, thereby preventing the P A party from deducing the disorder matrix S therefrom, but having a sufficiently high usability.

[0082] In a specific embodiment, the in-order gradient G can be sparsified to obtain the target gradient G t . It can be understood that sparsification refers to setting some elements in the in-order gradient G to 0. In one example, a predetermined proportion of elements in the in-order gradient G can be randomly selected and replaced with 0 values, where the predetermined proportion can be pre-set by the staff, such as 5%. In another example, elements with absolute values in the top predetermined number of the in-order gradient G can be determined and replaced with 0 values. The target gradient G t thus obtained can effectively reduce the subsequent communication and calculation amount.

[0083] In another specific embodiment, the in-order gradient G can be pooled and the original signs are preserved to obtain the target gradient G t . Exemplarily, the pooling can be average pooling, etc. Thus, the target gradient G t with sufficient usability and confusion can be obtained.

[0084] In yet another specific embodiment, the forward gradient G can be simultaneously subjected to sparsification and pooling, so as to further strengthen the protection of the disorder matrix S, and further reduce the subsequent communication and calculation. In one example, the forward gradient G can be subjected to sparsification first to obtain a sparse gradient, and then the sparse gradient can be subjected to pooling and the original signs can be retained, so as to obtain the target gradient G t .

[0085] From the above, the target gradient G t is obtained. B Then, in step S210, the P t party sends the target gradient G A to the P

[0086] Thus, in step S211, the P A party updates the model parameters θ t in the first representation layer 110 based on the target gradient G A , and in step S212, the P B party updates the model parameters θ t in the second representation layer 120 based on the target gradient G B . It should be noted that the way of updating the model parameters using the gradient can be implemented by using existing technologies, and will not be described here.

[0087] In this way, the two parties can jointly train the prediction model based on the training data they hold.

[0088] For the above steps, it should be noted that the relative execution order between the above steps is not unique, as long as the data flow is logical.

[0089] In summary, the method for two-party joint training of a prediction model for protecting data privacy disclosed in the embodiments of the present specification uses a method of combining matrix row transformation and secure multi-party computation to process representation data and label data, protecting the value of the representation data of the unlabeled training member and the privacy of the label data of the labeled training member; the labeled training member uses the representation plaintext (disordered representation) to perform plaintext calculation on the loss function, greatly saving the calculation time; the unlabeled training member uses sparsification or pooling to encrypt the gradient propagated in the reverse direction, protecting the disorder matrix and effectively reducing the communication and calculation.

[0090] The training method described above can be applied not only to the scenario of training a user prediction model, but also to the scenario of training a prediction model for other business objects. In an embodiment, the other business object can be a commodity, and the two parties participating in the joint training can be a manufacturer and an e-commerce platform. The commodity feature part held by the manufacturer can include the production cost, origin, ingredients, and shelf life of the commodity, and the commodity feature part held by the e-commerce platform can include the sales volume, procurement cost, and sales price of the commodity. The commodity label held by the e-commerce platform can be the commodity popularity level (e.g., unpopular or popular).

[0091] In another embodiment, the other business object can also be a business event, such as a login event or an access event.

[0092] Corresponding to the training method described above, the embodiments of the present specification also disclose a training device. Figure 4 The device for two-party joint training of a prediction model for protecting data privacy according to an embodiment is shown. The prediction model includes a first representation layer and a prediction layer deployed in a first party, and a second representation layer deployed in a second party. The device is integrated in the first party. As shown in the figure, Figure 4 The device 400 includes the following constituent units:

[0093] The representation unit 410 is configured to process the first feature part of the n sample users by using the first representation layer to obtain a first order representation. The secure computing unit 420 is configured to perform secure multi-party computation based on the first order representation, a second order representation held by the second party, and an n-order disordered matrix to determine a disordered fusion representation for the two order representations. The second order representation is obtained by processing the second feature part of the n sample users by using the second representation layer. The disordered gradient determination unit 430 is configured to determine a disordered gradient propagated to the two representation layers based on a disordered prediction result and a disordered user label. The disordered prediction result is obtained by processing the disordered fusion representation by using the prediction layer. The disordered user label is obtained by performing secure matrix multiplication based on an order user label and the disordered matrix held by the second party. The target gradient receiving unit 440 is configured to receive a target gradient from the second party, which is determined based on the disordered gradient and the disordered matrix.

[0094] In an embodiment, the secure computing unit 420 is specifically configured to: based on the first in-order representation, perform secure matrix multiplication with the out-of-order matrix held by the second party to obtain a first fragment of a first out-of-order representation, the first out-of-order representation corresponding to a product between the first in-order representation and the out-of-order matrix; receive an intermediate value from the second party, the intermediate value being determined based on a second fragment of the first out-of-order representation and a second out-of-order representation, the second out-of-order representation being obtained by performing product processing on the second in-order representation using the out-of-order matrix; and based on the first fragment and the intermediate value, determine the out-of-order fused representation.

[0095] In an embodiment, the apparatus 400 further includes an out-of-order label determination unit 450, configured to: based on the in-order user label, perform secure matrix multiplication with the out-of-order matrix held by the second party to obtain a first label fragment for the out-of-order user label; receive a second label fragment for the out-of-order user label from the second party; and based on the first label fragment and the second label fragment, construct the out-of-order user label.

[0096] In a specific embodiment, the apparatus 400 further includes a representation layer training unit 460, configured to update the model parameters in the first representation layer using the target gradient.

[0097] In a specific embodiment, the apparatus 400 further includes: a prediction layer training unit 470, configured to determine a training gradient propagated to the prediction layer based on the out-of-order prediction result and the out-of-order user label; and update the model parameters in the prediction layer using the training gradient.

[0098] Figure 5 An apparatus for two-party joint training of a prediction model for protecting data privacy is shown, according to another embodiment, the prediction model including a first representation layer and a prediction layer deployed in a first party, and a second representation layer deployed in a second party; the apparatus is integrated in the second party. As shown in Figure 5 The apparatus 500 includes the following constituent units:

[0099] The characterization unit 510 is configured to process the second feature part of the n sample users by using the second characterization layer to obtain a second normal order characterization; the secure calculation unit 520 is configured to perform secure multi-party computation with the first normal order characterization held by the first party based on the second normal order characterization and the n-order disordered square matrix, so that the first party obtains a disordered fusion characterization for the two normal order characterizations; the disordered gradient receiving unit 530 is configured to receive the disordered gradient propagated to the two characterization layers from the first party, the disordered gradient being determined based on a disordered prediction result and a disordered user label, the disordered prediction result being obtained by the first party by processing the disordered fusion characterization by using the prediction layer, and the disordered user label being obtained by the first party by performing secure matrix multiplication on the normal order user label and the disordered square matrix held by the second party; the target gradient determining unit 540 is configured to determine a target gradient based on the disordered gradient and the disordered square matrix; and the target gradient sending unit 550 is configured to send the target gradient to the first party.

[0100] In one embodiment, the apparatus 500 further includes a disordered square matrix determining unit 560 configured to perform random disordered processing on an n-order unit matrix in a column direction or a row direction to obtain the disordered square matrix.

[0101] In one embodiment, the secure calculation unit 520 is specifically configured to perform secure matrix multiplication on the first normal order characterization held by the first party based on the disordered square matrix to obtain a second fragment of a first disordered characterization corresponding to the product between the first normal order characterization and the disordered square matrix, perform product processing on the second normal order characterization by using the disordered square matrix to obtain a second disordered characterization, determine the intermediate value based on the second disordered characterization and the second fragment, and send the intermediate value to the first party, so that the first party determines a disordered fusion characterization based on a first fragment of the first disordered characterization and the intermediate value.

[0102] In a more specific embodiment, the secure calculation unit 520 is configured to determine the intermediate value based on the second disordered characterization and the second fragment, including performing summation processing on the second disordered characterization and the second fragment to obtain the intermediate value.

[0103] In one embodiment, the apparatus 500 further includes a label fragment processing unit 570 configured to perform secure matrix multiplication on the normal order user label held by the first party based on the disordered square matrix to obtain a second label fragment for the disordered user label, and send the second label fragment to the first party, so that the first party constructs the disordered user label based on the second label fragment and a first label fragment for the disordered user label.

[0104] In one embodiment, the target gradient determination unit 540 comprises: a forward gradient determination subunit 541, configured to multiply the out-of-order gradient by an inverse matrix of the out-of-order matrix to obtain a forward gradient; and a target gradient determination subunit 542, configured to determine the target gradient based on the forward gradient.

[0105] In one specific embodiment, the target gradient determination subunit 542 is specifically configured to: perform sparsification processing on the forward gradient to obtain a sparse gradient; and determine the target gradient based on the sparse gradient.

[0106] In one more specific embodiment, the target gradient determination subunit 542 is configured to perform the sparsification processing to obtain the sparse gradient, including: randomly selecting a predetermined proportion of elements from the forward gradient and replacing them with 0 values.

[0107] In one more specific embodiment, the target gradient determination subunit 542 is configured to determine the target gradient based on the sparse gradient, including: performing pooling processing on the sparse gradient while retaining the original signs to obtain the target gradient.

[0108] In one embodiment, the target gradient determination subunit 542 is specifically configured to: perform pooling processing on the forward gradient while retaining the original signs to obtain the target gradient.

[0109] According to another aspect, embodiments further provide a computer readable storage medium having stored thereon a computer program which, when executed in a computer, cause the computer to perform the method described in conjunction with Figure 2 or 3.

[0110] According to another aspect, embodiments further provide a computing device comprising a memory and a processor, the memory having stored thereon executable code that, when executed by the processor, facilitates performing the methods described in conjunction with Figure 2 or 3. It should be appreciated by persons skilled in the art that, in the above one or more examples, functions described by the present application can be implemented in hardware, software, firmware, or any combination thereof. When implemented in software, the functions can be stored in a computer readable medium or transmitted as one or more instructions or code on a computer readable medium.

[0111] The above detailed description merely describes specific embodiments of the application, and the purpose thereof is to explain the principles and implementations of the present application. It should be understood by those skilled in the art that the above-described embodiments are merely used to explain the principles and implementations of the present application, and are not used to limit the scope of the present application. Any modification, equivalent replacement, improvement, etc. made on the basis of the technical solutions of the present application should be included in the scope of the present application.< / h> < / h> < / h> < / h> < / h> < / h> < / x> < / x> < / h> < / x> < / x> < / h> < / h>

Claims

1. A method for jointly training a prediction model between two parties to protect data privacy, wherein the prediction model includes a first representation layer and a prediction layer deployed in a first party, and a second representation layer deployed in a second party; The method is applied to the first party, comprising: Processing the first feature portion of the n sample users using the first representation layer to obtain a first positive sequence representation; performing secure multi-party computation based on the first positive-sequence representation, a second positive-sequence representation held by the second party, and an n-order scrambled matrix to determine a scrambled fusion representation of the two positive-sequence representations; the second positive-sequence representation is obtained by processing the second feature portions of the n sample users using the second representation layer; Determining, based on an out-of-order prediction result and an out-of-order user label, an out-of-order gradient propagated to two representation layers; the out-of-order prediction result is obtained by processing the out-of-order fusion representation using the prediction layer; the out-of-order user label is obtained by performing a security matrix multiplication on the out-of-order square matrix held by the second party based on the positive-order user label; A target gradient is received from the second party, which is determined based on the shuffled gradient and the shuffled square matrix.

2. The method according to claim 1, wherein The determination of the out-of-order fusion representation includes: Performing secure matrix multiplication on the first positive-order representation and the scrambled square matrix held by the second party to obtain a first shard of the first scrambled representation, where the first scrambled representation corresponds to the product of the first positive-order representation and the scrambled square matrix; receiving an intermediate value from the second party, the intermediate value being determined based on a second slice of the first out-of-order representation and a second out-of-order representation, the second out-of-order representation being obtained by multiplying the second normal-order representation by the out-of-order square matrix; The out-of-order fusion representation is determined based on the first fragment and the intermediate value.

3. The method according to claim 1, wherein The determination of the out-of-order user tags includes: Performing a security matrix multiplication on the positive-order user tag and the random-order square matrix held by the second party to obtain the first tag fragment for the random-order user tag; receiving a second tag fragment for the out-of-order user tag from the second party; The out-of-order user label is constructed based on the first label fragment and the second label fragment.

4. The method according to claim 3, wherein: After receiving the target gradient from the second party, further comprising: The target gradient is used to update the model parameters in the first representation layer.

5. The method according to claim 1, wherein Also includes: Determining a training gradient propagated to the prediction layer based on the out-of-order prediction results and the out-of-order user labels; The model parameters in the prediction layer are updated using the training gradient.

6. A method for jointly training a prediction model between two parties to protect data privacy, wherein the prediction model includes a first representation layer and a prediction layer deployed in a first party, and a second representation layer deployed in a second party; The method is applied to the second party, comprising: Processing the second feature parts of the n sample users using the second representation layer to obtain a second positive sequence representation; Performing secure multi-party computation based on the second positive-order representation and the n-th order disordered matrix with the first positive-order representation held by the first party, so that the first party obtains a disordered fusion representation of the two positive-order representations; Receiving, from the first party, a scrambled gradient propagated to two representation layers, the scrambled gradient being determined based on a scrambled prediction result and a scrambled user label, the scrambled prediction result being obtained by the first party processing the scrambled fusion representation using the prediction layer, and the scrambled user label being obtained by the first party performing a secure matrix multiplication based on the positive-order user label and the scrambled square matrix held by the second party; A target gradient is determined based on the random gradient and the random matrix, and is sent to the first party.

7. The method according to claim 6, wherein: The determination of the disordered matrix includes: The second party performs random shuffling of the columns or rows of the n-order unit matrix to obtain the shuffled square matrix.

8. The method according to claim 6, wherein: Performing secure multi-party computation based on the second positive-order representation and the n-th order disordered matrix with the first positive-order representation held by the first party, so that the first party obtains a disordered fusion representation of the two positive-order representations, including: Performing security matrix multiplication on the scrambled square matrix and a first positive-order representation held by the first party to obtain a second shard of the first scrambled representation, where the first scrambled representation corresponds to the product of the first positive-order representation and the scrambled square matrix; Performing a product process on the second positive sequence representation using the random matrix to obtain a second random sequence representation; An intermediate value is determined based on the second out-of-order representation and the second fragment, and is sent to the first party, so that the first party determines an out-of-order fusion representation based on the first fragment of the first out-of-order representation and the intermediate value.

9. The method according to claim 8, wherein Determining an intermediate value based on the second out-of-order representation and the second fragment includes: The second out-of-order representation and the second fragment are summed to obtain the intermediate value.

10. The method according to claim 6, wherein: Before receiving the out-of-order gradients propagated to two representation layers from the first party, the method includes: Performing security matrix multiplication on the random matrix and the positive-order user tag held by the first party to obtain a second tag fragment for the random user tag; The second label fragment is sent to the first party, so that the first party constructs the out-of-order user label based on the second label fragment and the first label fragment for the out-of-order user label.

11. The method according to claim 6, wherein: Determining a target gradient based on the random gradient and the random matrix includes: Multiplying the random gradient by the inverse matrix of the random matrix to obtain a positive gradient; The target gradient is determined based on the positive sequence gradient.

12. The method according to claim 11, wherein Determining the target gradient based on the positive sequence gradient includes: Performing a sparse processing on the positive sequence gradient to obtain a sparse gradient; The target gradient is determined based on the sparse gradient.

13. The method according to claim 12, wherein: The positive sequence gradient is subjected to sparse processing to obtain a sparse gradient, comprising: A predetermined number of elements are randomly selected from the positive sequence gradient and replaced with 0 values.

14. The method according to claim 12, wherein: Determining the target gradient based on the sparse gradient includes: The sparse gradient is pooled and the original positive and negative signs are retained to obtain the target gradient.

15. The method according to claim 11, wherein Determining the target gradient based on the positive sequence gradient includes: Pooling is performed on the positive order gradient, and the original positive and negative signs are retained to obtain the target gradient.

16. A device for jointly training a prediction model between two parties to protect data privacy, wherein the prediction model includes a first representation layer and a prediction layer deployed in a first party, and a second representation layer deployed in a second party; The device is integrated into the first party and includes: a representation unit configured to process the first feature portion of the n sample users using the first representation layer to obtain a first positive sequence representation; a secure computing unit configured to perform secure multi-party computing based on the first positive-order representation, a second positive-order representation held by the second party, and an n-order scrambled matrix to determine a scrambled fusion representation of the two positive-order representations; the second positive-order representation is obtained by processing the second feature portions of the n sample users using the second representation layer; a disordered gradient determination unit configured to determine disordered gradients propagated to two representation layers based on disordered prediction results and disordered user labels; the disordered prediction results are obtained by processing the disordered fusion representation using the prediction layer; the disordered user labels are obtained by performing security matrix multiplication of the positive-order user labels with the disordered square matrix held by the second party; The target gradient receiving unit is configured to receive a target gradient from the second party, which is determined based on the random gradient and the random matrix.

17. A device for jointly training a prediction model between two parties to protect data privacy, wherein the prediction model includes a first representation layer and a prediction layer deployed in a first party, and a second representation layer deployed in a second party; The device is integrated into the second party and includes: a representation unit configured to process the second feature portion of the n sample users using the second representation layer to obtain a second positive sequence representation; a secure computing unit configured to perform secure multi-party computing based on the second positive-order representation and the n-th order disordered matrix with the first positive-order representation held by the first party, so that the first party obtains a disordered fusion representation of the two positive-order representations; a random gradient receiving unit configured to receive random gradients propagated to two representation layers from the first party, the random gradients being determined based on random prediction results and random user labels, the random prediction results being obtained by the first party processing the random fusion representation using the prediction layer, and the random user labels being obtained by the first party performing secure matrix multiplication based on the positive-order user labels and the random square matrix held by the second party; a target gradient determining unit, configured to determine a target gradient based on the random gradient and the random matrix; A target gradient sending unit is configured to send the target gradient to the first party.

18. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed in a computer, the computer is caused to execute the method according to any one of claims 1 to 15.

19. A computing device comprising a memory and a processor, wherein: The memory stores executable code, and when the processor executes the executable code, the method according to any one of claims 1 to 15 is implemented.

Images