Method and apparatus for two-party joint training of a business prediction model that protects data privacy

Abstract

The embodiment of the specification provides a method and device for two-party joint training of a business prediction model for protecting data privacy. The method comprises: each party uses a locally deployed representation layer to process a feature part of a training sample of the party to obtain a corresponding representation; a second party performs fusion processing on the representations of the two parties, and processes the fused representation using a randomly generated reversible function to obtain a confused representation, so that a first party uses a prediction layer to process the confused representation to obtain a confused prediction result; the second party processes the confused prediction result using an inverse function of the reversible function to obtain a real prediction result, so that the first party uses the real prediction result and a sample label to calculate a gradient of the prediction layer and a target gradient propagated to the representation layer; the second party processes the target gradient according to the inverse function and a corresponding mode of the fusion processing, thereby determining a first gradient propagated to a first representation layer and a second gradient of a second representation layer, and then each party completes local model parameter updating according to the gradients.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a method and apparatus for a two-party joint training business prediction model that protects data privacy, and a method and apparatus for a multi-party joint training business prediction model that protects data privacy. Background Technology

[0002] With the development of Artificial Intelligence (AI) technology, Deep Neural Networks (DNNs) have been gradually applied to various fields such as risk assessment, speech recognition, facial recognition, and natural language processing. The network structure of DNNs is relatively fixed across different application scenarios, and more training data is needed to achieve better model performance.

[0003] In many fields such as healthcare and finance, different companies or institutions possess diverse data samples. Jointly training these data can significantly improve model accuracy, thereby effectively enhancing business quality and user satisfaction. However, this raw training data contains substantial amounts of user privacy and trade secrets; any leakage could cause irreparable negative consequences. Therefore, while addressing the data silo problem through multi-party collaborative training, protecting data privacy and data value has become a key research focus in recent years.

[0004] If the training members participating in joint training are from different industries but have the same user group, such as payment platforms and social platforms, where payment platforms mainly hold payment data and social platforms mainly hold entertainment and social data, then the feature spaces of the data they collect are different, but the sample spaces are the same. This can be called a vertically segmented data scenario.

[0005] However, current joint training methods applied to vertical data partitioning scenarios are limited and cannot meet the higher demands of practical applications. Therefore, a joint training scheme is needed to better meet practical application requirements, such as better protecting data privacy and data value. Summary of the Invention

[0006] This specification describes one or more embodiments of a method for jointly training a business prediction model by two parties to protect data privacy by using an invertible function to shuffle the sample space of the hidden layer, thereby protecting data value.

[0007] According to a first aspect, a method for jointly training a business prediction model by two parties to protect data privacy is provided. The business prediction model includes a first representation layer and a prediction layer deployed in a first party, and a second representation layer deployed in a second party. The method is applied to the second party and includes: processing a second feature portion of training samples held by the party using the second representation layer to obtain a second representation, and fusing the second representation and the first representation to obtain a fused representation; wherein the first representation is obtained by the first party using the first representation layer to process the first feature portion of the training samples held by it; processing the fused representation using a randomly generated invertible function to obtain a confused representation; processing the confused prediction result using the inverse function of the invertible function to obtain a true prediction result, wherein the confused prediction result is obtained by the first party using the prediction layer to process the confused representation; processing a target gradient according to the inverse function and the corresponding method of the fusion processing, thereby determining a first gradient propagated to the first representation layer and a second gradient of the second representation layer, and sending the first gradient to the first party, wherein the target gradient is obtained by the first party based on the true prediction result and sample labels.

[0008] In one embodiment, processing the target gradient according to the inverse function and the corresponding method of the fusion process to determine the first gradient propagating to the first representation layer and the second gradient of the second representation layer includes: processing the target gradient propagating to the representation layer corresponding to the confused representation using the inverse function to obtain the restored gradient; and determining the first gradient and the second gradient based on the restored gradient according to the corresponding method of the fusion process.

[0009] In one embodiment, a Trusted Execution Environment (TEE) is further deployed in the second party; wherein, before fusing the second representation and the first representation to obtain a fused representation, the method further includes: receiving a first encrypted representation from the first party, which is obtained by encrypting the first representation using a public key provided by the TEE environment; wherein fusing the second representation and the first representation to obtain a fused representation includes: in the TEE environment, decrypting the first encrypted representation using a private key corresponding to the public key to obtain the first representation, and performing the fusing process on the first representation and the second representation loaded into the TEE environment to obtain the fused representation.

[0010] In one specific embodiment, obtaining the obfuscated characterization, obtaining the true prediction result, and obtaining the restoration gradient are all performed in the TEE environment.

[0011] In one specific embodiment, the fusion process of the second representation and the first representation to obtain the fused representation includes: averaging the second representation and the first representation to obtain the fused representation; wherein, the determination of the first gradient and the second gradient based on the restored gradient according to the corresponding method of the fusion process includes: determining half of the restored gradient as the first gradient and the second gradient respectively.

[0012] In a specific embodiment, the second representation and the first representation are fused to obtain a fused representation, which includes: summing the second representation and the first representation to obtain the fused representation; wherein, according to the corresponding method of the fusion processing, the first gradient and the second gradient are determined based on the restored gradient, which includes: determining the restored gradient as the first gradient and the second gradient respectively.

[0013] In one specific embodiment, the fused representation is processed using a randomly generated invertible function to obtain a confused representation, including: randomly generating a full-rank matrix; multiplying the fused representation using the randomly generated full-rank matrix to obtain the confused representation; wherein, the confused prediction result is processed using the inverse function of the invertible function to obtain the true prediction result, including: multiplying the confused prediction result using the inverse matrix of the full-rank matrix to obtain the true prediction result; wherein, the target gradient corresponding to the confused representation propagated to the representation layer is processed using the inverse function to obtain the restored gradient, including: multiplying the target gradient using the inverse matrix to obtain the restored gradient.

[0014] In a more specific embodiment, the fusion representation is an n*m dimensional matrix; wherein, randomly generating a full-rank matrix includes: concatenating an n-order identity matrix and a randomly generated (Nn)*n dimensional matrix to obtain an N*n dimensional concatenated matrix; and shuffling and reorganizing the concatenated matrix according to the row direction to obtain the full-rank matrix.

[0015] In one embodiment, after determining a first gradient propagating to the first representation layer and a second gradient propagating to the second representation layer, the method further includes: updating model parameters in the second representation layer using the second gradient.

[0016] In one embodiment, processing the confusing prediction result using the inverse function of the invertible function to obtain the true prediction result includes: processing the confusing prediction result using the inverse function to obtain the true original prediction result; and adding noise conforming to the differential privacy mechanism to the original prediction result to obtain the true prediction result.

[0017] In one specific embodiment, sending the first gradient to the first party includes: updating the first gradient to a gradient with noise added to it in accordance with the differential privacy mechanism; and sending the updated first gradient to the first party.

[0018] According to a second aspect, a method for jointly training a business prediction model by two parties to protect data privacy is provided. The business prediction model includes a first representation layer and a prediction layer deployed in a first party, and a second representation layer deployed in a second party. The method is applied to the first party and includes: processing a first feature portion of training samples held by the first representation layer to obtain a first representation; processing a confused representation received from the second party using the prediction layer to obtain a confused prediction result; wherein the confused representation is obtained by the second party processing a fused representation using a randomly generated invertible function, the fused representation being obtained by fusing the first representation and the second representation, the second representation being obtained by processing a second feature portion of the training samples using the second representation layer; calculating a target gradient propagated to the representation layer using the true prediction result and the sample labels of the training samples; the true prediction result is obtained by the second party processing the confused prediction result using the inverse function of the invertible function; updating the model parameters in the first representation layer using the first gradient received from the second party; the first gradient is obtained by the second party processing the target gradient according to the inverse function and the corresponding method of the fusion processing.

[0019] In one embodiment, the method further includes: calculating the training gradient of the prediction layer using the true prediction result and the sample label; and updating the model parameters in the prediction layer using the training gradient.

[0020] According to a third aspect, a method for jointly training a business prediction model by multiple parties to protect data privacy is provided. The business prediction model includes a first feature representation layer and a prediction layer deployed in a first party, and a second feature representation layer deployed in a second party. The method is applied to a trusted third party among the multiple parties and includes: receiving a first representation obtained from the first party by processing a first feature portion of a training sample using the first representation layer, and receiving a second representation obtained from the second party by processing a second feature portion of a training sample using the second representation layer; fusing the first representation and the second representation to obtain a fused representation, and processing the fused representation using a randomly generated invertible function to obtain a confused representation; processing the confused prediction result using the inverse function of the invertible function to obtain a true prediction result, wherein the confused prediction result is obtained by the first party by processing the confused representation using the prediction layer; processing a target gradient according to the inverse function and the corresponding method of the fusion processing, thereby determining a first gradient propagated to the first representation layer and a second gradient propagated to the second representation layer, and sending the first gradient and the second gradient to the first party and the second party respectively; wherein the target gradient is obtained by the first party based on the true prediction result and sample labels.

[0021] In one embodiment, a Trusted Execution Environment (TEE) is deployed in the trusted third party, wherein obtaining the obfuscated representation and obtaining the true prediction result are both performed in the TEE environment.

[0022] According to a fourth aspect, an apparatus is provided for jointly training a business prediction model by two parties to protect data privacy. The business prediction model includes a first representation layer and a prediction layer deployed in a first party, and a second representation layer deployed in a second party. The apparatus is integrated into the second party and includes: a representation unit configured to process a second feature portion of training samples held by the party using the second representation layer to obtain a second representation; a fusion unit configured to fuse the second representation and the first representation to obtain a fused representation; wherein the first representation is obtained by the first party processing the first feature portion of the training samples held by it using the first representation layer; an encryption unit configured to process the fused representation using a randomly generated invertible function to obtain an obfuscated representation; a decryption unit configured to process the obfuscated prediction result using the inverse function of the invertible function to obtain a true prediction result, wherein the obfuscated prediction result is obtained by the first party processing the obfuscated representation using the prediction layer; a gradient processing unit configured to process a target gradient according to the inverse function and the corresponding method of the fusion processing, thereby determining a first gradient propagated to the first representation layer and a second gradient of the second representation layer; and a gradient sending unit configured to send the first gradient to the first party, wherein the target gradient is obtained by the first party based on the true prediction result and sample labels.

[0023] According to a fifth aspect, an apparatus for jointly training a business prediction model by two parties to protect data privacy is provided. The business prediction model includes a first representation layer and a prediction layer deployed in a first party, and a second representation layer deployed in a second party. The apparatus is integrated into the first party and includes: a representation unit configured to process a first feature portion of training samples held by the first representation layer to obtain a first representation; a prediction unit configured to process a confused representation received from the second party using the prediction layer to obtain a confused prediction result; wherein the confused representation is obtained by the second party processing a fused representation using a randomly generated invertible function, the fused representation being obtained by fusing the first representation and the second representation, the second representation being obtained by processing a second feature portion of the training samples using the second representation layer; a gradient calculation unit configured to calculate a target gradient propagated to the representation layer using the true prediction result and the sample labels of the training samples; the true prediction result is obtained by the second party processing the confused prediction result using the inverse function of the invertible function; and an update unit configured to update the model parameters in the first representation layer using the first gradient received from the second party; the first gradient is obtained by the second party processing the target gradient according to the inverse function and the corresponding method of the fusion processing.

[0024] According to a sixth aspect, an apparatus is provided for a multi-party joint training business prediction model that protects data privacy. The business prediction model includes a first feature representation layer and a prediction layer deployed in a first party, and a second feature representation layer deployed in a second party. The apparatus is applied to a trusted third party among the multiple parties and includes: a representation receiving unit configured to receive from the first party a first representation obtained by processing a first feature portion of a training sample using the first representation layer, and a second representation obtained from the second party by processing a second feature portion of a training sample using the second representation layer; a fusion unit configured to fuse the first representation and the second representation to obtain a fused representation; and an encryption unit configured to... To process the fused representation using a randomly generated invertible function to obtain a confused representation; a decryption unit is configured to process the confused prediction result using the inverse function of the invertible function to obtain a true prediction result, wherein the confused prediction result is obtained by the first party using the prediction layer to process the confused representation; a gradient processing unit is configured to process the target gradient according to the inverse function and the corresponding method of the fusion processing, thereby determining the first gradient propagated to the first representation layer and the second gradient of the second representation layer; a gradient sending unit is configured to send the first gradient and the second gradient to the first party and the second party respectively; wherein the target gradient is obtained by the first party based on the true prediction result and the sample label.

[0025] According to a seventh aspect, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the methods of the first aspect, the second aspect, or the third aspect.

[0026] According to an eighth aspect, a computing device is provided, including a memory and a processor, wherein the memory stores executable code, and the processor, when executing the executable code, implements the methods of the first aspect, the second aspect, or the third aspect.

[0027] By employing the methods and apparatus provided in the embodiments of this specification, and the method of jointly training the business prediction model disclosed in the embodiments of this specification, the sample space of the hidden layer is obfuscated and rearranged using invertible functions such as invertible matrix transformation, thereby achieving the protection of data privacy and data value. Attached Figure Description

[0028] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 A schematic diagram illustrating a deployment scenario of a business forecasting model according to one embodiment is shown.

[0030] Figure 2 This diagram illustrates the interaction process of a two-party joint training business prediction model according to one embodiment.

[0031] Figure 3 This diagram illustrates the interaction process of a multi-party joint training business prediction model according to one embodiment.

[0032] Figure 4 A schematic diagram of the device structure for jointly training a business prediction model according to one embodiment is shown.

[0033] Figure 5 A schematic diagram of the apparatus structure for jointly training a business prediction model according to another embodiment is shown.

[0034] Figure 6 A schematic diagram of an apparatus for jointly training a business prediction model according to one embodiment is shown. Detailed Implementation

[0035] The solution provided in this specification will now be described with reference to the accompanying drawings.

[0036] This specification discloses a joint training scheme that can better protect the value of private data among training members. While ensuring that the data sent to other participants is available, it prevents data from being reused, thus avoiding additional loss of data value.

[0037] Figure 1 This diagram illustrates a deployment scenario of a business prediction model according to one embodiment. The model is deployed between two parties participating in joint training. For simplicity, the party holding the label is referred to as the first party, and the other as the second party. It should be understood that both parties can be implemented as any device, platform, or cluster of devices with computing and processing capabilities. Figure 1 As shown, a first representation layer and a prediction layer of the business prediction model are deployed in the first party, and a second representation layer of the business prediction model is deployed in the second party. It should be noted that the first representation layer, the second representation layer, and the prediction layer can all be implemented as neural network layers. The network structure, number of layers, etc., of the neural network layers can be set according to actual needs. For example, the first or second representation layer can be implemented as multiple sets of alternating fully connected layers-activation layers, and the prediction layer can be implemented as several (several refers to one or more) fully connected layers, as well as sigmoid layers or softmax layers, etc.

[0038] To facilitate understanding, the data distribution of the two parties is described below. The first party and the second party each hold different feature portions of the same training samples (or business objects), where the first party holds the sample labels of the training samples (or the business labels of the business objects). For clarity, the private features of the training samples held by the first party are referred to as the first feature portion, and the private features of the training samples held by the second party are referred to as the second feature portion.

[0039] In one embodiment, the aforementioned business object is a user. Further, in a specific embodiment, the first party is a payment platform and the second party is a social platform. In this case, the first feature portion may include user transaction characteristics or payment characteristics, such as payment time, payment frequency, payment amount, and payment channel. The second feature portion may include user social characteristics, such as the number of user friends and social activity level. The user category tag held by the first party may be a consumer group category tag (high-spending group, low-spending group, etc.), a personality category tag (such as extroverted or introverted), or a risk category tag (such as high-risk user or low-risk user). In another specific embodiment, the first party is a bank and the second party is a credit reporting agency. In this case, the first feature portion may include the user's bank statements (such as salary, expenses, etc.), and the second feature portion may include the user's credit records (such as loan records, credit card activation records, etc.). In this case, the user category tag held by the first party may be a risk category tag (such as risky or risk-free).

[0040] In another embodiment, the above-mentioned business object is a commodity. Further, in a specific embodiment, the first party is the manufacturer of the commodity and the second party is an e-commerce platform. In this case, the first feature part may include the production cost, place of origin, ingredients and shelf life of the commodity, and the second feature part may include the sales volume, procurement cost, sales price, etc. of the commodity. The commodity label held by the first party may be the popularity level of the commodity (such as unpopular or popular).

[0041] The distribution of features, labels, and other data from both parties has been described above. The method for jointly training the aforementioned business model by both parties will be introduced below. Figure 2 This diagram illustrates the interaction process of jointly training a business prediction model by two parties according to one embodiment. Figure 2 As shown, the interactive process includes the following steps:

[0042] Step S201: The first party processes the first feature part using the first representation layer to obtain the first representation. It should be understood that one or more neural networks in the first representation layer sequentially perform linear and / or nonlinear transformations on the first feature part to obtain the first representation. Alternatively, if this training iteration (or current iteration) is the first time, this step can also be implemented as follows: the first party initializes the model parameters in the first representation layer, and then processes the first feature part using the initialized first representation layer to obtain the first representation.

[0043] In a single training iteration, a batch of training samples is typically used. Let m be the number of training samples in the current batch, where m = 1, or more typically, m is a positive integer greater than 1. Let n be the output dimension of the last network layer in the first representation layer. Then, the first party inputs the feature parts it holds for each of the m training samples into the first representation layer, which yields m n-dimensional first representation vectors, forming an n*m dimensional matrix, which serves as the aforementioned first representation Z1.

[0044] Thus, the first party can obtain a first representation Z1 by processing the local private features of a batch of training samples. Therefore, in step S202, the first party sends the first representation to the second party. In one embodiment, the second party also deploys a Trusted Execution Environment (TEE). In this case, this step may further include: the first party encrypting the first representation using the public key provided by the TEE environment to obtain a first encrypted representation, and then sending the first encrypted representation to the second party.

[0045] On the other hand, in step S203, the second party processes the second feature part using the second representation layer to obtain the second representation. It should be understood that one or more neural network layers in the second representation layer sequentially perform linear and / or nonlinear transformations on the second feature part to obtain the second representation. Additionally, if this training is the first time, this step can also be implemented as follows: the second party initializes the model parameters in the second representation layer, and then uses the initialized second representation layer to process the second feature part to obtain the second representation.

[0046] In a single training iteration, the feature data processed by the second party corresponds to the same training samples as the feature data processed by the first party, for example, the same batch of users. Furthermore, the output dimension of the last network layer in the second representation layer can be designed to be the same as the output dimension of the last network layer in the first representation layer, allowing for a more secure approach during subsequent feature fusion. Thus, the second party can input its feature portions for each of the aforementioned m training samples into the second representation layer, obtaining m n-dimensional second representation vectors, forming an n*m dimensional matrix, which serves as the aforementioned second representation Z2.

[0047] Thus, the second party can obtain the second representation Z2 by processing the local private features of the training samples.

[0048] As described above, the second party holds the first representation Z1 sent by the first party and the second representation Z2 obtained through local processing. Therefore, in step S204, the second party performs a fusion process on the first and second representations to obtain a fused representation, and then uses a randomly generated invertible function to process the fused representation to obtain a confused representation.

[0049] To enhance security and increase the difficulty of cracking the aforementioned fusion process, a summation-based fusion method is used. In one embodiment, the sum of the first representation Z1 and the second representation Z2 is used as the fusion representation. In another embodiment, half of the summation result is used as the fusion representation, or the average of the first representation Z1 and the second representation Z2 is used as the fusion representation. In yet another embodiment, the product of other multiples of the summation result can also be used as the fusion representation. Thus, a highly secure fusion representation can be obtained.

[0050] Regarding the random generation of the aforementioned invertible function, in one embodiment, the functional form of the invertible function is predetermined. In this case, only the parameters need to be randomly generated. In another embodiment, a set of functional forms of invertible functions can be predefined. Then, a functional form is randomly selected from this set, and the function parameters are randomly generated. In a specific embodiment, the functional form of the invertible function may include: f(x) = Sx, where S is a full-rank matrix and the matrix elements are to be randomly generated; it may also include: f(x) = a / x, where the value of a can be randomly generated.

[0051] According to a specific embodiment, the random generation of an invertible function includes: randomly generating a full-rank matrix S, whose dimension can be denoted as N*n. It should be understood that if N > n, this full-rank matrix can also be called a column-rank matrix. Further, in one example, a (Nn)*n-dimensional random matrix can be generated first, then this random matrix and an n-order identity matrix can be concatenated row-wise to obtain an N*n-dimensional concatenated matrix. This concatenated matrix can then be shuffled and rearranged according to the row direction to obtain the full-rank matrix S. In this way, the generation of full-rank matrices can be achieved more simply and efficiently.

[0052] Furthermore, a randomly generated invertible function f(x) is used to process the fused representation Z. r Obtain the confusion characterization Z s Specifically, the fusion representation Z is utilized. r Assign values ​​to the independent variable in the invertible function f(x) to obtain the value of the dependent variable, which serves as the confusion characterization Z. f According to a specific embodiment, a randomly generated invertible function is denoted as f(x) = Sx, where S is a randomly generated full-rank matrix. Therefore, Z can be calculated. s =S*Z r It should be noted that the full-rank matrix S is used to represent the fusion characterization Z. r Transformation can achieve the effect of shuffling m n-dimensional vectors. Thus, by using matrix transformation to shuffle the correspondence between sample IDs and model hidden layers, the smooth progress of model training is ensured, and the first party with labels cannot reuse the representation data of the second party, thereby protecting the value of the data. According to another specific embodiment, the randomly generated invertible function is f(x) = 1 / x. In this case, the fused representation Z can be... r Each matrix element in the algorithm is used as input to an invertible function to obtain the corresponding transformed value, thus forming the confusion representation Z. s .

[0053] On the other hand, in one embodiment, the second party also deploys a TEE environment. Accordingly, this step can be implemented as follows: loading the second representation and the first encrypted representation received from the first party into the TEE environment; in the TEE environment, decrypting the first encrypted representation using the private key corresponding to the aforementioned public key to obtain the first representation; then fusing the first and second representations; and further processing the fused representation using a randomly generated reversible function to obtain an obfuscated representation. This obfuscated representation can then be output to the second party's non-TEE environment. This makes it difficult for the second party to obtain the plaintext of the first representation, thereby strengthening data privacy protection for the first party.

[0054] Based on the first and second representations, the second party can obtain the confusion representation Z of m training samples. s Therefore, in step S205, the second party sends the obfuscation representation to the first party.

[0055] Next, in step S206, the first party processes the ambiguity representation using the prediction layer to obtain ambiguity prediction results. Specifically, the m representation vectors included in the ambiguity representation, for example, m N-dimensional representation vectors, can be input into the prediction layer to obtain m prediction results, forming the aforementioned ambiguity prediction results. It should be understood that each prediction result in the m prediction results can include the prediction probability corresponding to each category. In addition, if this training is the first time, this step can also be implemented as follows: the first party initializes the model parameters in the prediction layer, and then uses the parameter-initialized prediction layer to process the ambiguity representation to obtain ambiguity prediction results.

[0056] Thus, the first party can obtain the confusion prediction result for the current batch of training samples. Then, in step S207, the first party sends the confusion prediction result to the second party.

[0057] Then, in step S208, the second party uses the inverse function of the invertible function to process the obfuscated prediction results and obtain the true prediction results. It should be understood that the inverse function of an invertible function is unique; once the invertible function f(x) is determined, its inverse function f(x) is also unique. -1 (x) is then determined. In a typical embodiment, the above invertible function is: f(x) = Sx, and its corresponding inverse function is f -1 (x)=S -1 x, where S -1Let S be the inverse matrix of a full-rank matrix. Multiplying the inverse matrix with the scrambled prediction result yields the true prediction result. It should be noted that in this embodiment, the difference between the scrambled prediction result and the true prediction result lies primarily in the fact that the m predictions in the true prediction result correctly correspond to the IDs of the m training samples, while the m predictions in the scrambled prediction result are out of order. On the other hand, in one embodiment, this step can be implemented as follows: loading the scrambled prediction result into a TEE environment deployed in the second party, and then processing the scrambled prediction result using the aforementioned inverse function in the TEE environment to obtain the true prediction result. Furthermore, in another embodiment, this step can be implemented as follows: processing the scrambled prediction result using the inverse function to obtain the true original prediction result; then adding noise conforming to the differential privacy mechanism to the original prediction result to obtain the true prediction result. This further enhances data privacy protection.

[0058] Therefore, the second party can restore the confused prediction result to the true prediction result. Thus, in step S209, the second party sends the true prediction result to the first party.

[0059] Next, in step S210, the first party calculates the target gradient using the true prediction results and sample labels. It should be understood that the sample labels refer to the labels of the current batch of training samples, and the m prediction results included in the true prediction results have a one-to-one correspondence with the batch of sample labels held by the first party.

[0060] Specifically, based on the actual prediction results, sample labels, and the selected loss function, the backpropagation method is used to calculate the training gradient of each neural network layer in the prediction layer from back to front. Accordingly, the calculated gradients can be used to update the weight parameters in the corresponding network layers.

[0061] Therefore, the target gradient corresponding to the confusing representation can be calculated and propagated to the representation layer (or the first representation layer and the second representation layer). Thus, in step S211, the first direction sends the target gradient to the second direction.

[0062] Then, in step S212, the second party uses the inverse function of the invertible function and the corresponding method of the above fusion process to process the target gradient, thereby determining the first gradient propagated to the first representation layer and the second gradient of the second representation layer.

[0063] Specifically, the target gradient is first processed using the aforementioned inverse function to obtain the restored gradient. In a typical embodiment, the inverse function is f. -1 (x)=S -1 x, at this point, the inverse matrix S -1 By multiplying the gradient with the target gradient, the restored gradient can be obtained.

[0064] Furthermore, according to the corresponding method of the fusion processing, a first gradient propagating to the first representation layer and a second gradient to the second representation layer are determined based on the restored gradient. It should be understood that the corresponding method refers to the way the restored gradient is decomposed into a first gradient and a second gradient, which is adapted to the processing method of the fusion processing. In one embodiment, the above fusion processing method is averaging; accordingly, half of the restored gradient can be determined as the first gradient and the second gradient, respectively, meaning the first gradient and the second gradient are equal and equal to half of the restored gradient. In another embodiment, the above fusion processing method is summation; accordingly, the restored gradient can be directly determined as the first gradient and the second gradient, respectively.

[0065] Thus, the first gradient of the first representation layer and the second gradient of the second representation layer can be determined. Furthermore, the second party can use the second gradient to adjust the model parameters in the second representation layer. Additionally, in step S213, the second party sends the first gradient to the first party so that the first party can use the first gradient to update the model parameters in the first representation layer. In one embodiment, the second party can first update the first gradient to a gradient with noise added to it in accordance with the differential privacy mechanism; then send the updated first gradient to the first party. This further enhances data privacy protection.

[0066] According to another embodiment, the terms "first representation layer," "second representation layer," "prediction layer," "first representation," "second representation," "fusion representation," and "obfuscated representation," all corresponding to business objects, can also be replaced with "first object representation layer," "second object representation layer," "first object representation," "second object representation," "object prediction layer," "first object representation," "second object representation," "fusion object representation," and "obfuscated object representation." Furthermore, the "object" in these terms can be replaced with a specific object category, such as a user or a product.

[0067] In summary, the method for jointly training a business prediction model to protect data privacy, as disclosed in the embodiments of this specification, utilizes invertible functions such as invertible matrix transformations to obfuscate and rearrange the sample space of the hidden layer, thereby achieving the protection of data privacy and data value.

[0068] According to another embodiment, the method for jointly training the business prediction model by the two parties described above can also be extended to scenarios with more participants. In one implementation, other data parties participate in the joint training, possessing other feature data of the same training samples as the first and second parties. In this case, the business prediction model may further include a representation layer deployed in the other data parties. Accordingly, the other data parties can use the locally deployed representation layer to process their local feature data, obtain local representations, and then send them to the second party so that the second party can fuse the representations of all participants to obtain a fused representation. Afterward, the other data parties can update the model parameters in their local representation layers based on the gradients received from the second party.

[0069] In another implementation, a trusted third party can be introduced. This trusted third party can be a computing platform or device cluster belonging to an authoritative institution, or it can be a neutral server. Figure 3 This diagram illustrates the interaction process of a multi-party joint training business prediction model according to one embodiment, where the multi-party includes a first party, a second party, and a trusted third party, such as... Figure 3 As shown, the interaction process includes the following steps:

[0070] Step S301: The first party processes the first feature part of the training sample using the first representation layer to obtain the first representation.

[0071] In step S302, the first party sends the first representation to the trusted third party.

[0072] In step S303, the second party uses the second representation layer to process the second feature part of the training sample to obtain the second representation.

[0073] In step S304, the second party sends the second representation to the trusted third party.

[0074] In step S305, a trusted third party performs a fusion process on the first representation and the second representation to obtain a fused representation, and then processes the fused representation using a randomly generated invertible function to obtain a confused representation.

[0075] In step S306, the trusted third party sends the obfuscated representation to the first party.

[0076] Step S307: The first party uses the prediction layer to process the confusing representation and obtains the confusing prediction result.

[0077] In step S308, the first party sends the obfuscation prediction result to a trusted third party.

[0078] In step S309, a trusted third party uses the inverse function of the invertible function to process the obfuscated prediction results and obtain the true prediction results.

[0079] In step S310, the trusted third party sends the actual prediction results to the first party.

[0080] In step S311, the first party calculates the training gradient of the prediction layer and the target gradient corresponding to the confused representation and transmitted to the representation layer based on the true prediction results and sample labels, and updates the model parameters in the prediction layer using the training gradient.

[0081] In step S312, the first party sends the target gradient to a trusted third party.

[0082] In step S313, a trusted third party processes the target gradient according to the inverse function and the corresponding method of the fusion processing, thereby determining the first gradient propagated to the first representation layer and the second gradient of the second representation layer.

[0083] In step S314, the trusted third party sends the first gradient to the first party.

[0084] Step S315: The first party updates the model parameters in the first representation layer using the first gradient.

[0085] In step S316, the trusted third party sends the second gradient to the second party.

[0086] In step S317, the second party updates the model parameters in the second representation layer using the second gradient.

[0087] In one embodiment, a TEE environment is deployed in the trusted third party. In this case, step S302 can also be implemented as follows: the first party sends the first encrypted representation obtained by encrypting the first representation using the public key provided by the TEE environment; step S304 can also be implemented as follows: the second party sends the second encrypted representation obtained by encrypting the second representation using the public key; step S305 can also include: in the TEE environment, decrypting the first encrypted representation and the second encrypted representation respectively using the private key to obtain the first representation and the second representation, and outputting the obtained obfuscated representation outside the TEE environment in the trusted third party; steps S309 and S313 can also be executed in the TEE environment.

[0088] It should be noted that, for Figure 3 For a description of the intermediate steps, please refer to the section on... Figure 2 Description of the steps.

[0089] In summary, the method for multi-party joint training of business prediction models to protect data privacy disclosed in the embodiments of this specification utilizes invertible functions such as invertible matrix transformation to confuse and rearrange the sample space of the hidden layer, and introduces a trusted third party for data processing, thereby achieving effective protection of data privacy and data value.

[0090] Corresponding to the training methods described above, the embodiments of this specification also disclose training devices. Figure 4 A schematic diagram of an apparatus for jointly training a business prediction model according to one embodiment is shown. The business prediction model includes a first representation layer and a prediction layer deployed in the first party, and a second representation layer deployed in the second party; the apparatus is integrated into the second party. Figure 4 As shown, the device 400 includes the following units:

[0091] The representation unit 410 is configured to process the second feature portion of the training samples held by the first party using the second representation layer to obtain a second representation. The fusion unit 420 is configured to fuse the second representation and the first representation to obtain a fused representation; wherein the first representation is obtained by the first party processing the first feature portion of the training samples held by it using the first representation layer. The encryption unit 430 is configured to process the fused representation using a randomly generated invertible function to obtain an obfuscated representation. The decryption unit 440 is configured to process the obfuscated prediction result using the inverse function of the invertible function to obtain a true prediction result, wherein the obfuscated prediction result is obtained by the first party processing the obfuscated representation using the prediction layer. The gradient processing unit 450 is configured to process the target gradient according to the inverse function and the corresponding method of the fusion processing, thereby determining the first gradient propagated to the first representation layer and the second gradient of the second representation layer. The gradient sending unit 460 is configured to send the first gradient to the first party, wherein the target gradient is obtained by the first party based on the true prediction result and the sample label.

[0092] Figure 5 A schematic diagram of an apparatus for jointly training a business prediction model according to another embodiment is shown. The business prediction model includes a first representation layer and a prediction layer deployed in the first party, and a second representation layer deployed in the second party; the apparatus is integrated into the first party. Figure 5 As shown, the device 500 includes the following units:

[0093] The representation unit 510 is configured to process a first feature portion of the training samples held by the first representation layer to obtain a first representation; the prediction unit 520 is configured to process the obfuscated representation received from the second party using the prediction layer to obtain an obfuscated prediction result; wherein the obfuscated representation is obtained by the second party processing the fused representation using a randomly generated invertible function, the fused representation being obtained by fusing the first representation and the second representation, the second representation being obtained by processing the second feature portion of the training samples using the second representation layer; the gradient calculation unit 530 is configured to calculate the target gradient propagated to the representation layer using the true prediction result and the sample labels of the training samples; the true prediction result is obtained by the second party processing the obfuscated prediction result using the inverse function of the invertible function; the update unit 540 is configured to update the model parameters in the first representation layer using the first gradient received from the second party; the first gradient is obtained by the second party processing the target gradient according to the inverse function and the corresponding method of the fusion processing.

[0094] Figure 6 The diagram illustrates a device structure for a multi-party jointly trained business prediction model according to one embodiment. The business prediction model includes a first feature representation layer and a prediction layer deployed in a first party, and a second feature representation layer deployed in a second party; the device is applied to a trusted third party among the multiple parties. Figure 6 As shown, the device 600 includes the following units:

[0095] The representation receiving unit 610 is configured to receive from the first party a first representation obtained by processing a first feature portion of a training sample using a first representation layer, and a second representation obtained from the second party by processing a second feature portion of a training sample using a second representation layer; the fusion unit 620 is configured to fuse the first representation and the second representation to obtain a fused representation; the encryption unit 630 is configured to process the fused representation using a randomly generated invertible function to obtain an obfuscated representation; the decryption unit 640 is configured to process the obfuscated prediction result using the inverse function of the invertible function to obtain a true prediction result, wherein the obfuscated prediction result is obtained by the first party by processing the obfuscated representation using the prediction layer; the gradient processing unit 650 is configured to process the target gradient according to the inverse function and the corresponding method of the fusion processing, thereby determining the first gradient propagated to the first representation layer and the second gradient of the second representation layer; the gradient sending unit 660 is configured to send the first gradient and the second gradient to the first party and the second party respectively; wherein the target gradient is obtained by the first party based on the true prediction result and the sample label.

[0096] It should be noted that the description of the training device can be found in the preceding description of the training method.

[0097] According to another embodiment, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed in a computer, causes the computer to perform a combination Figure 2 or Figure 3 The method described.

[0098] According to another embodiment, a computing device is also provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements a combination... Figure 2 or Figure 3 The methods described herein. Those skilled in the art will recognize that, in one or more of the examples above, the functions described in this invention can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.

[0099] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for jointly training a business prediction model by two parties to protect data privacy, the business prediction model comprising a first representation layer and a prediction layer deployed in the first party, and a second representation layer deployed in the second party; The method is applied to the second party, including: The second feature portion of the training samples held by the first party is processed by the second representation layer to obtain a second representation, and the second representation and the first representation are fused to obtain a fused representation; wherein the first representation is obtained by the first party processing the first feature portion of the training samples held by the first party using the first representation layer. The fused representation is processed using a randomly generated invertible function to obtain a confused representation; The confusing prediction result is processed using the inverse function of the invertible function to obtain the true prediction result, wherein the confusing prediction result is obtained by the first party using the prediction layer to process the confusing representation; According to the inverse function and the corresponding method of the fusion processing, the target gradient is processed to determine the first gradient propagated to the first representation layer and the second gradient of the second representation layer, and the first gradient is sent to the first party, wherein the target gradient is obtained by the first party based on the true prediction result and the sample label.

2. The method according to claim 1, wherein, Based on the inverse function and the corresponding method of the fusion process, the target gradient is processed to determine the first gradient propagating to the first representation layer and the second gradient propagating to the second representation layer, including: The inverse function is used to process the target gradient corresponding to the distorted representation propagated to the representation layer, resulting in the restored gradient. Based on the corresponding method of the fusion processing, the first gradient and the second gradient are determined based on the restoration gradient.

3. The method according to claim 1 or 2, wherein, The second party also deploys a Trusted Execution Environment (TEE); The method further includes, before fusing the second representation and the first representation to obtain the fused representation, receiving a first encrypted representation from the first party, which is obtained by encrypting the first representation using a public key provided by the TEE environment. The process of fusing the second representation and the first representation to obtain a fused representation includes: in the TEE environment, decrypting the first encrypted representation using the private key corresponding to the public key to obtain the first representation, and performing the fusion process on the first representation and the second representation loaded into the TEE environment to obtain the fused representation.

4. The method according to claim 3, wherein, The process of obtaining the obfuscated representation and the true prediction result is performed in the TEE environment.

5. The method according to claim 2, in, The process of fusing the second representation and the first representation to obtain a fused representation includes: averaging the second representation and the first representation to obtain the fused representation; Specifically, determining the first gradient and the second gradient based on the restoration gradient, according to the corresponding method of the fusion processing, includes: Half of the reduction gradient is determined as the first gradient and the second gradient, respectively.

6. The method according to claim 2, in, The process of fusing the second representation and the first representation to obtain a fused representation includes: summing the second representation and the first representation to obtain the fused representation; Specifically, determining the first gradient and the second gradient based on the restoration gradient, according to the corresponding method of the fusion processing, includes: The reduction gradients are respectively determined as the first gradient and the second gradient.

7. The method according to claim 2, in, The fused representation is processed using a randomly generated invertible function to obtain a confused representation, including: randomly generating a full-rank matrix; and multiplying the fused representation using the randomly generated full-rank matrix to obtain the confused representation. The process of processing the confusing prediction results using the inverse function of the invertible function to obtain the true prediction results includes: multiplying the confusing prediction results using the inverse matrix of the full-rank matrix to obtain the true prediction results. Specifically, processing the target gradient corresponding to the confused representation propagated to the representation layer using the inverse function to obtain the restored gradient includes: multiplying the target gradient using the inverse matrix to obtain the restored gradient.

8. The method according to claim 7, wherein, The fusion representation is an n*m dimensional matrix; wherein, a randomly generated full-rank matrix includes: By concatenating an n-order identity matrix and a randomly generated (Nn)*n-dimensional matrix, an N*n-dimensional concatenated matrix is ​​obtained. The spliced ​​matrix is ​​shuffled and rearranged according to the row direction to obtain the full-rank matrix.

9. The method according to claim 1, wherein, After determining the first gradient propagating to the first representation layer and the second gradient propagating to the second representation layer, the method further includes: The model parameters in the second representation layer are updated using the second gradient.

10. The method according to claim 1, wherein, The inverse function of the invertible function is used to process the confusing prediction results to obtain the true prediction results, including: The inverse function is used to process the confused prediction results to obtain the true original prediction results; Noise conforming to the differential privacy mechanism is added to the original prediction result to obtain the true prediction result.

11. The method according to claim 10, wherein, Sending the first gradient to the first party includes: The first gradient is updated to a gradient with noise added to it in accordance with the differential privacy mechanism; The updated first gradient is generated for the first party.

12. A method for jointly training a business prediction model by two parties to protect data privacy, the business prediction model comprising a first representation layer and a prediction layer deployed in a first party, and a second representation layer deployed in a second party; The method is applied to the first party, including: The first feature portion of the training samples held by the party is processed using the first representation layer to obtain the first representation; The prediction layer processes the obfuscated representation received from the second party to obtain an obfuscated prediction result; wherein the obfuscated representation is obtained by the second party processing the fused representation using a randomly generated invertible function, the fused representation being obtained by fusing the first representation and the second representation, the second representation being obtained by processing the second feature part of the training sample using the second representation layer; The target gradient propagated to the representation layer is calculated using the true prediction result and the sample labels of the training samples; the true prediction result is obtained by the second party by processing the confused prediction result using the inverse function of the invertible function; The model parameters in the first representation layer are updated using the first gradient received from the second party; the first gradient is obtained by the second party processing the target gradient according to the inverse function and the corresponding method of the fusion process.

13. The method according to claim 11, wherein, Also includes: The training gradient of the prediction layer is calculated using the actual prediction results and the sample labels; The training gradients are used to update the model parameters in the prediction layer.

14. A method for jointly training a business prediction model by multiple parties to protect data privacy, the business prediction model comprising a first feature representation layer and a prediction layer deployed in a first party, and a second feature representation layer deployed in a second party; The method is applied to a trusted third party among the multiple parties, including: The first representation is obtained by processing a first feature portion of a training sample using a first representation layer from the first party, and the second representation is obtained by processing a second feature portion of a training sample using a second representation layer from the second party. The first and second representations are fused to obtain a fused representation, and the fused representation is then processed using a randomly generated invertible function to obtain a confused representation. The confusing prediction result is processed using the inverse function of the invertible function to obtain the true prediction result, wherein the confusing prediction result is obtained by the first party using the prediction layer to process the confusing representation; Based on the inverse function and the corresponding method of the fusion processing, the target gradient is processed to determine the first gradient propagated to the first representation layer and the second gradient of the second representation layer, and the first gradient and the second gradient are sent to the first party and the second party respectively; wherein the target gradient is obtained by the first party based on the real prediction result and the sample label.

15. The method according to claim 14, wherein a Trusted Execution Environment (TEE) is deployed in the trusted third party, wherein, Both obtaining the obfuscated characterization and obtaining the true prediction results are performed in the TEE environment.

16. An apparatus for jointly training a business prediction model by two parties to protect data privacy, the business prediction model comprising a first representation layer and a prediction layer deployed in a first party, and a second representation layer deployed in a second party; The device is integrated into the second party and includes: The representation unit is configured to process the second feature portion of the training samples held by the party using the second representation layer to obtain the second representation; The fusion unit is configured to perform a fusion process on the second representation and the first representation to obtain a fused representation; wherein the first representation is obtained by the first party processing a first feature portion of the training samples it holds using the first representation layer; The encryption unit is configured to process the fused representation using a randomly generated reversible function to obtain a confused representation. The decryption unit is configured to process the obfuscated prediction result using the inverse function of the invertible function to obtain the true prediction result, wherein the obfuscated prediction result is obtained by the first party using the prediction layer to process the obfuscated representation; The gradient processing unit is configured to process the target gradient according to the inverse function and the corresponding method of the fusion processing, thereby determining the first gradient propagated to the first representation layer and the second gradient of the second representation layer. A gradient sending unit is configured to send the first gradient to the first party, wherein the target gradient is obtained by the first party based on the true prediction result and the sample label.

17. An apparatus for jointly training a business prediction model by two parties to protect data privacy, the business prediction model comprising a first representation layer and a prediction layer deployed in a first party, and a second representation layer deployed in a second party; The device is integrated into the first party and includes: The representation unit is configured to process the first feature portion of the training samples held by the first representation layer to obtain a first representation; The prediction unit is configured to process the obfuscated representation received from the second party using the prediction layer to obtain an obfuscated prediction result; wherein the obfuscated representation is obtained by the second party processing the fused representation using a randomly generated invertible function, the fused representation is obtained by fusing the first representation and the second representation, and the second representation is obtained by processing the second feature part of the training sample using the second representation layer; The gradient calculation unit is configured to calculate the target gradient propagated to the representation layer using the true prediction result and the sample labels of the training samples; the true prediction result is obtained by the second party by processing the confused prediction result using the inverse function of the invertible function; The update unit is configured to update the model parameters in the first representation layer by receiving a first gradient from the second party; the first gradient is obtained by the second party processing the target gradient according to the inverse function and the corresponding method of the fusion process.

18. An apparatus for multi-party joint training of a business prediction model to protect data privacy, the business prediction model comprising a first feature representation layer and a prediction layer deployed in a first party, and a second feature representation layer deployed in a second party; The device is applied to a trusted third party among the multiple parties, including: The characterization receiving unit is configured to receive from the first party a first characterization obtained by processing a first feature portion of a training sample using a first characterization layer, and to receive from the second party a second characterization obtained by processing a second feature portion of a training sample using a second characterization layer. The fusion unit is configured to perform a fusion process on the first representation and the second representation to obtain a fused representation; The encryption unit is configured to process the fused representation using a randomly generated reversible function to obtain a confused representation. The decryption unit is configured to process the obfuscated prediction result using the inverse function of the invertible function to obtain the true prediction result, wherein the obfuscated prediction result is obtained by the first party using the prediction layer to process the obfuscated representation; The gradient processing unit is configured to process the target gradient according to the inverse function and the corresponding method of the fusion processing, thereby determining the first gradient propagated to the first representation layer and the second gradient of the second representation layer. The gradient sending unit is configured to send the first gradient and the second gradient to the first party and the second party respectively; wherein the target gradient is obtained by the first party based on the true prediction result and the sample label.

19. A computer-readable storage medium having a computer program stored thereon, wherein, When the computer program is executed in the computer, it causes the computer to perform the method according to any one of claims 1-15.

20. A computing device comprising a memory and a processor, wherein, The memory stores executable code, and when the processor executes the executable code, it implements the method of any one of claims 1-15.

Images