A privacy-preserving multi-party joint training method and device

Abstract

The embodiment of the present specification provides a privacy-protecting multi-party joint training method and device. A second device and a plurality of first devices respectively own a second partial model and a plurality of first partial models in a complete model. Any one of the first devices determines an extracted feature containing privacy data of a sample through the first partial model of the first device, and determines a first gradient slice for updating the first partial model based on the extracted feature. Through data interaction based on multi-party secure calculation between the plurality of first devices and the second device, the second device obtains aggregated features aggregated from the plurality of extracted features; the second device determines output data of the second partial model by using the aggregated features, determines a second gradient slice for updating the first partial model based on the output data and the aggregated features, and sends the second gradient slice to the plurality of first devices; and the plurality of first devices respectively update the first partial model based on the second gradient slice and the first gradient slice determined by the first device.

Description

Technical Field

[0001] This specification relates to one or more embodiments in the field of computer technology, and in particular to a method and apparatus for multi-party joint training that protects privacy. Background Technology

[0002] With the development of artificial intelligence technology, neural networks have been gradually applied to fields such as risk assessment, speech recognition, facial recognition, and natural language processing. The neural network structures for different application scenarios have become relatively fixed, and more training data is needed to achieve better model performance. In fields such as healthcare and finance, different companies or institutions possess different data samples. Jointly training these sample data will greatly improve model accuracy. Therefore, multi-party joint training models have become a research hotspot in recent years. In the process of multi-party joint training models, devices from multiple participating parties complete joint training through data transmission between them. During iterative training, a large amount of data transmission is required between devices. Simultaneously, data privacy protection should be considered when participating parties transmit data.

[0003] Therefore, we hope to find an improved solution that can protect data privacy as much as possible and improve the overall processing efficiency between devices during the multi-party joint training of models. Summary of the Invention

[0004] This specification describes one or more embodiments of a privacy-preserving multi-party collaborative training method and apparatus to maximize data privacy and improve the overall processing efficiency between devices. The specific technical solution is as follows.

[0005] In a first aspect, the embodiments provide a privacy-preserving multi-party joint training method, wherein a second device and multiple first devices respectively possess a second part model and multiple first part models in a business prediction model, the method comprising:

[0006] Any first device determines the extracted features of its own samples through its own first part model, and determines the first gradient slice for updating the first part model based on the extracted features;

[0007] Through data interaction based on multi-party secure computation, multiple first devices and the second device enable the second device to obtain aggregated features after aggregating multiple extracted features;

[0008] The second device uses the aggregation features to determine the output data of the second part of the model, determines the second gradient slice for updating multiple first part models based on the output data and the aggregation features, and sends it to multiple first devices;

[0009] Multiple first devices respectively receive the second gradient slice sent by the second device, and update their own first part model based on the second gradient slice and the first gradient slice determined by themselves.

[0010] In one implementation, multiple first devices and the second device interact through data exchange based on a secret sharing algorithm, enabling the second device to obtain aggregated features after aggregating multiple extracted features.

[0011] In one implementation, any first device splits its extracted features into multiple feature fragments based on a secret sharing algorithm, and sends the multiple feature fragments to other first devices respectively, and receives the feature fragments sent by the other first devices; it uses its own feature fragments and the feature fragments of the other first devices to determine its own secret state features, and sends them to the second device;

[0012] The second device receives dense state features sent by multiple first devices respectively, and determines the aggregated feature based on the multiple dense state features.

[0013] In one embodiment, the step of determining the aggregated feature based on multiple dense-state features includes determining the sum or mean of the multiple dense-state features as the aggregated feature.

[0014] In one implementation, the output data is a label prediction value; the second device, when determining a second gradient slice for updating multiple first part models based on the output data and the aggregated features, includes:

[0015] The prediction loss of the sample is determined based on the output data;

[0016] The second gradient slice is determined using the prediction loss and the aggregation features.

[0017] In one implementation, the business forecasting model further includes a third model configured in a third device; the method further includes:

[0018] The second device sends the output data to the third device and determines a third gradient slice for updating the second part of the model based on the output data and the second part of the model.

[0019] The third device uses the output data and the third part of the model to determine the predicted label value of the sample, uses the predicted label value to determine the prediction loss of the sample, and determines a fourth gradient slice for updating the first part of the model and the second part of the model based on the prediction loss and the output data, and sends it to the second device.

[0020] The second device receives the fourth gradient slice sent by the third device, updates the second part of the model using the fourth gradient slice and the third gradient slice; determines the fifth gradient slice using the output data and the aggregated features, and determines the second gradient slice based on the fifth gradient slice.

[0021] In one implementation, the step of determining the second gradient slice based on the fifth gradient slice includes:

[0022] The second gradient slice is determined by multiplying the fourth gradient slice and the fifth gradient slice.

[0023] In one implementation, the third device also sends the fourth gradient fragment to the first device;

[0024] The second device directly identifies the fifth gradient slice as the second gradient slice;

[0025] The first device also receives a fourth gradient slice sent by the third device, and updates its own first part model based on the fourth gradient slice, the second gradient slice, and the first gradient slice determined by itself.

[0026] In one implementation, the step of determining the first gradient slice for updating the first part of the model based on the extracted features includes:

[0027] Before receiving the second gradient slice sent by the second device, a first gradient slice for updating the first part of the model is determined based on the extracted features.

[0028] In one implementation, the step of determining the first gradient slice for updating the first part of the model based on the extracted features includes:

[0029] The first gradient slice is determined based on the partial derivatives of the extracted features with respect to the first part of the model.

[0030] In one implementation, the step of updating the first part of the model based on the second gradient partition and the first gradient partition determined by itself includes:

[0031] Based on the product of the second gradient slice and the first gradient slice determined by itself, the first gradient of the first part of the model is determined;

[0032] The first gradient is used to update the first part of the model.

[0033] Secondly, the embodiment provides a privacy-preserving multi-party joint training method, wherein a second device and multiple first devices respectively possess a second part model and multiple first part models in a business prediction model, and the method is executed through any one of the first devices, including:

[0034] The first part of the model determines the extraction features of the samples it possesses;

[0035] Based on the extracted features, a first gradient slice is determined for updating the first part of the model;

[0036] Through data interaction based on multi-party secure computation between the first device and the second device, the second device obtains aggregated features after aggregating multiple extracted features, and determines a second gradient slice for updating multiple first part models based on the output data and the aggregated features, and sends it to the first device; wherein, the output data is determined based on the aggregated features and the second part models;

[0037] Receive the second gradient fragment sent by the second device;

[0038] The first part of the model is updated based on the second gradient partition and the first gradient partition.

[0039] Thirdly, the embodiment provides a privacy-preserving multi-party joint training method, wherein a second device and multiple first devices respectively possess a second part model and multiple first part models in a business prediction model, and the method is executed through the second device, including:

[0040] Through data interaction based on multi-party secure computation with multiple first devices, aggregated features are obtained after aggregating the extracted features of multiple first devices; wherein, the multiple extracted features are determined using samples and first part models of their respective first devices;

[0041] The output data of the second part of the model is determined using the aggregation features;

[0042] Based on the output data and the aggregation features, a second gradient slice is determined for updating multiple first part models;

[0043] The second gradient slice is sent to multiple first devices so that any one of the first devices can update the first part of the model based on the second gradient slice and the first gradient slice; wherein the first gradient slice is determined based on its respective extracted features and the first part of the model.

[0044] Fourthly, the embodiments provide a privacy-preserving multi-party joint training system, including a second device and multiple first devices; wherein the second device and the multiple first devices respectively possess a second part model and multiple first part models in a business prediction model;

[0045] Any first device is used to determine the extracted features of its own samples through its own first part model, and to determine a first gradient slice for updating the first part model based on the extracted features.

[0046] Multiple first devices and the second device are used to interact through data based on multi-party secure computation, so that the second device obtains aggregated features after aggregating multiple extracted features;

[0047] The second device is configured to determine the output data of the second part model using the aggregation features, determine the second gradient slice for updating multiple first part models based on the output data and the aggregation features, and send it to multiple first devices;

[0048] Multiple first devices are configured to receive second gradient slices sent by the second device, and update their own first part of the model based on the second gradient slices and their own determined first gradient slices.

[0049] Fifthly, the embodiment provides a privacy-preserving multi-party joint training device, wherein a second device and multiple first devices respectively possess a second part model and multiple first part models in a business prediction model, and the device is deployed in any one of the first devices, including:

[0050] The first determining module is configured to determine the extracted features of its own samples through its own first part model;

[0051] The second determining module is configured to determine a first gradient slice for updating the first part of the model based on the extracted features.

[0052] The first interaction module is configured to, through data interaction based on multi-party secure computation with other first devices and second devices, enable the second device to obtain aggregated features after aggregating multiple extracted features, and enable the second device to determine a second gradient slice for updating multiple first part models based on the output data and the aggregated features, and send it to the first device; wherein, the output data is determined based on the aggregated features and the second part models;

[0053] The first receiving module is configured to receive the second gradient fragment sent by the second device;

[0054] The first update module is configured to update the first part of the model based on the second gradient slice and the first gradient slice.

[0055] Sixthly, the embodiment provides a privacy-preserving multi-party joint training device, wherein a second device and multiple first devices respectively possess a second part model and multiple first part models in a business prediction model, and the device is deployed in the second device, comprising:

[0056] The second interaction module is configured to obtain aggregated features after aggregating the extracted features of multiple first devices through data interaction based on multi-party secure computation with multiple first devices; wherein the multiple extracted features are determined using samples and a first part of the model of their respective first devices.

[0057] The third determining module is configured to determine the output data of the second part of the model using the aggregated features;

[0058] The fourth determining module is configured to determine a second gradient slice for updating multiple first part models based on the output data and the aggregated features;

[0059] The first sending module is configured to send the second gradient slice to multiple first devices, so that any one of the first devices can update the first part of the model based on the second gradient slice and the first gradient slice; wherein the first gradient slice is determined based on its respective extracted features and the first part of the model.

[0060] In a seventh aspect, an embodiment provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method described in any one of the first to third aspects.

[0061] Eighthly, an embodiment provides a computing device including a memory and a processor, wherein the memory stores executable code, and the processor, when executing the executable code, implements the method described in any one of the first to third aspects.

[0062] In the methods and apparatus provided in the embodiments of this specification, the first device and the second device interact through data interaction based on multi-party secure computation, enabling the second device to obtain aggregated features after aggregating the extracted features from multiple first devices, thereby better protecting the data privacy of the extracted features. Simultaneously, the second device determines a second gradient partition for updating the first part of the model, sharing some of the processing tasks of the first device during the model update process, allowing the first device to update the model more quickly. Therefore, the embodiments of this specification can better protect data privacy and comprehensively consider the computing power of the devices, improving the overall processing efficiency between the devices. Attached Figure Description

[0063] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0064] Figure 1 This is a schematic diagram illustrating an implementation scenario of one embodiment disclosed in this specification;

[0065] Figure 2 A flowchart illustrating a privacy-preserving multi-party joint training method provided for an embodiment;

[0066] Figure 3 This is a schematic diagram illustrating an application scenario of a multi-party joint training method provided in an embodiment.

[0067] Figure 4 A flowchart illustrating a privacy-preserving multi-party joint training method provided for an embodiment;

[0068] Figure 5 A flowchart illustrating a privacy-preserving multi-party joint training method provided for an embodiment;

[0069] Figure 6 A schematic block diagram of a privacy-preserving multi-party joint training system provided for an embodiment;

[0070] Figure 7 A schematic block diagram of a privacy-preserving multi-party joint training device provided for an embodiment;

[0071] Figure 8 This is a schematic block diagram of a privacy-preserving multi-party joint training device provided for an embodiment. Detailed Implementation

[0072] The solution provided in this specification will now be described with reference to the accompanying drawings.

[0073] Figure 1 This is a schematic diagram illustrating an implementation scenario of one embodiment disclosed in this specification. In this scenario, multiple devices jointly train a service prediction model. Figure 1 Devices 1, 2, and 3 in the model possess partial models from the business forecasting model, while device 4 possesses partial models from the business forecasting model.

[0074] The model training process can be divided into a forward computation phase and a backward computation phase. In the forward computation phase, the device uses the samples and the business prediction model to determine the prediction results for the samples, and uses the difference between the prediction results and the sample labels to determine the prediction loss. In the backward computation phase, the device uses the prediction loss to update the business prediction model.

[0075] Figure 1 In the forward computation phase, devices 1 through 3 use their respective partial models to obtain extracted features from the samples and send these features to device 4. Device 4 obtains aggregated features from multiple extracted features and continues model training using its own partial model. Since devices 1 through 3 possess the former part of the model and device 4 possesses the latter part, in the backward computation phase, the gradients used to update the model propagate from back to front. Device 4 can determine the gradient slices used to update the partial models of devices 1 through 3 and sends these gradient slices to devices 1 through 3. Device 1 uses its own computed gradient slices and the gradient slices sent by device 4 to update its own partial model. Devices 2 and 3 also perform the same model update process. Figure 1 Devices 1 to 3 are the owners of the sample, and their number is only an example. In actual applications, the number can be much larger.

[0076] The aforementioned joint training can also be called split learning, which means breaking down the entire model learning process into smaller parts, with different devices undertaking parts of the model's computational tasks. The aforementioned business prediction model, used for business prediction of objects, can be implemented using deep neural networks (DNN), convolutional neural networks (CNN), recurrent neural networks (RNN), or graph neural networks (GNN), etc.

[0077] Devices 1 through 4 mentioned above can be different participating devices. For example, devices 1 through 3 can be different sample owner devices, and device 4 can be a server, a label owner, or possess both label data and sample data. In different application scenarios, the devices can be implemented by different participating parties. Specifically, the above multi-party joint training can be applied in different network architectures. For example, it can be applied in a client-server architecture consisting of a server and two or more sample owner devices, or in a peer-to-peer network architecture consisting of two or more sample owner devices.

[0078] Sample owners can correspond to different service platforms, which use their computer equipment to transmit data with other devices. Service platforms can be banks, hospitals, medical examination institutions, or other institutions or organizations. Therefore, different sample owners can possess different attribute feature values ​​of the same sample, which belongs to a vertical scenario of sample data distribution. The sample data in a sample owner's data can be the business data of an object. For example, an object can be, but is not limited to, a user, a product, a transaction, or an event. The object's business data can include object feature data, such as, but is not limited to, at least one of the following feature groups: basic attribute features of the object, historical behavior features of the object, relationship features of the object, interaction features of the object, and physical indicators of the object. The sample data from multiple sample owners constitutes the entire sample space. Sample data is the private data of the sample owner and cannot be sent to external parties in plaintext.

[0079] In vertical scenarios with diverse sample data distributions, a complete business prediction model can be broken down into several partial models, each trained by a different participating device using its own sample data. The embodiments described in this specification can be applied to different network architectures. The participating devices can include several sample-owning devices or a server (which does not possess sample data). During multiple iterations of model training, the participating devices can jointly train the business prediction model through data transmission with each other.

[0080] In multi-party joint training, data transmission occurs between different devices, requiring the device that owns the samples to pay attention to data privacy protection when sending data out. At the same time, the limited computing power of some participating devices makes the overall computational efficiency of the joint training process relatively low.

[0081] To better protect data privacy while improving overall processing efficiency among devices, this specification provides a privacy-preserving multi-party joint training method. For ease of description, we will first set aside specific application scenarios and network architectures, and only take the first and second devices among multiple devices performing multi-party joint training as examples to illustrate the core inventive solution of the embodiments. In this method, the second device and multiple first devices respectively possess a second part model and multiple first part models in a business prediction model. In step S210, any one of the first devices determines the extracted features of its own samples through its own first part model. In step S220, the first devices determine the first gradient slices for updating their respective first part models based on their respective extracted features. In step S230, the multiple first devices and the second device interact through data interaction based on multi-party secure computation (MPC), enabling the second device to obtain aggregated features after aggregating multiple extracted features. In step S240, the second device uses the aggregated features to determine the output data of the second part model, determines the second gradient slices for updating the multiple first part models based on the output data, and sends them to the multiple first devices. In step S250, multiple first devices respectively receive the second gradient slice sent by the second device, and update their own first part model based on the second gradient slice and the first gradient slice determined by themselves.

[0082] Because there is a clear mapping relationship between extracted features and sample data, the sample data can be inferred from the extracted features, leading to the leakage of privacy data. In this embodiment, the devices interact with each other through multi-party secure computation, enabling the second device to obtain aggregated features and protecting the data privacy of the extracted features. Simultaneously, because the second device determines the gradient partitioning, it shares some of the computational tasks with the first device, allowing the first device to update the model more quickly.

[0083] Secure Multi-Party Computation (MPC) is a data privacy protection technology that can be used for multi-party participation. Its implementation methods include homomorphic encryption, obfuscated circuits, unintentional transmission, and secret sharing. By employing MPC, secure interactive computation of extracted features can be achieved between multiple devices. This allows devices to obtain aggregated features from multiple extracted features without sending plaintext extracted features.

[0084] The following describes the embodiments in detail with reference to the flowchart.

[0085] Figure 2This is a flowchart illustrating a privacy-preserving multi-party joint training method provided in an embodiment. For ease of description, device 100 represents the first device, and device 200 represents the second device. The business prediction model is jointly trained by at least several devices 100 and one device 200. Each device 100 possesses a first part of the business prediction model (denoted as NN1), and device 200 possesses a second part of the business prediction model (denoted as NN2).

[0086] A business forecasting model can contain multiple computational layers. The first part, model NN1, can be the first few consecutive computational layers in the business forecasting model, and the second part, model NN2, can be the middle or last consecutive computational layers. The first part, model NN1, and the second part, model NN2, can be consecutive; that is, the output of the first part, model NN1, can be the input of the second part, model NN2. The first part, model NN1, precedes the second part, model NN2. For example, if the business forecasting model contains 100 computational layers, the first part, model NN1, can be the first 50 computational layers, and the second part, model NN2, can be the last 50 computational layers. Alternatively, the first part, model NN1, and the second part, model NN2, can be discontinuous; that is, other computational layers can exist in between.

[0087] There are multiple devices 100, and different devices 100 possess feature values ​​of different attributes for all samples. Therefore, the number of computational layers in the first part of the model of different devices 100 can be the same or different, and the model parameters therein can also be the same or different.

[0088] At the beginning of joint training, the management device can split the first part of the business prediction model NN1 and the second part of the model NN2 from the sample owner's sample attributes or the operation that the device needs to perform, and then distribute them to the corresponding devices respectively.

[0089] For example, when device 100 is the sample owner device, the management device can determine the corresponding computational layer and the neurons in the computational layer based on the sample attributes owned by the device, thus obtaining the first part of the model NN1. This first part of the model NN1 is used to extract deep features of the sample based on the original feature data of the sample, thus obtaining the extracted features.

[0090] In a client-server architecture, device 200 can be a server. The management device can determine multiple computational layers from the business prediction model as needed, serving as a second part model NN2 within the server. This second part model NN2 is used to further extract some or all of the feature data from the samples, and the server is used to perform computational processing on the multiple computational layers of this second part model.

[0091] In a peer-to-peer network architecture, device 200 can be the tag-owning device. The management device can use the last few computational layers in the business prediction model as a second part model NN2, which is used to determine the prediction result for the sample based on the feature data extracted from the sample.

[0092] Multiple devices 100 and 200 (i.e., the first device and the second device) can be implemented using any device, equipment, platform, device cluster, etc., with computing and processing capabilities. The model training process may include multiple iterations, where any model iteration may include the following steps S210 to S250.

[0093] In step S210, any device 100 determines the extracted features of its own samples using its own first part model NN1. All devices 100 perform this step, that is, determine the extracted features of their own samples using their own first part model NN1. Multiple devices 100 obtain their own extracted features. These extracted features are features extracted by the first part model NN1 from the original features of the samples, and they carry privacy data.

[0094] For any device 100, it can select a certain number of samples from its own sample set, input their original features into the first part model NN1, and determine the extracted features of the samples through the first part model NN1.

[0095] In step S220, multiple devices 100 each determine a first gradient slice g1 for updating their own first part of the model based on their own extracted features. Each device 100 calculates its own first gradient slice g1 independently of the others.

[0096] The following describes the process of determining the first gradient partition g1 for any device 100.

[0097] When the first part of the model NN1 contains one computational layer, the device 100 can determine the first gradient slice g1 corresponding to that computational layer in the first part of the model NN1 based on extracted features. When the first part of the model NN1 contains multiple computational layers, the device 100 can determine the first gradient slice g1 corresponding to each of the multiple computational layers in the first part of the model NN1 based on extracted features. That is, each computational layer corresponds to a first gradient slice g1.

[0098] When determining the first gradient slice g1, device 100 can determine the first gradient slice g1 based on the partial derivatives of the extracted features with respect to the first part of the model NN1. For example, it can be based on... Determine the first gradient segment g1, where b is the extracted feature.

[0099] When the first part of the model NN1 contains one or more computational layers, for each computational layer, the first gradient slice g1 of that computational layer can be determined based on the partial derivatives of the extracted features with respect to that layer. Since there are input-output relationships between multiple computational layers, the first gradient slice g1 of multiple computational layers can be determined according to the order of the computational layers from back to front. This back-to-front order refers to the direction opposite to the data transmission in the forward computation stage.

[0100] In one implementation, the first gradient slice g1 can be determined using the computation graph of the first partial model NN1. For example, device 100 can first acquire the computation graph of the first partial model NN1, and then use the computation graph to determine the partial derivatives of the extracted features with respect to multiple model parameters in the first partial model NN1, and determine the first gradient slice g1 based on the multiple partial derivatives. The computation graph includes multiple computation layers in the first partial model NN1, and the data input-output relationships between neurons in different computation layers.

[0101] In practice, for any model parameter in any computational layer, the partial derivative of that model parameter can be determined using the path from the extracted features to that model parameter in the computation graph. After determining the partial derivatives of all model parameters in that computational layer in this way, the partial derivatives of all model parameters are constructed into a matrix to obtain the first gradient slice g1 in matrix form for that computational layer.

[0102] In step S230, multiple devices 100 and 200 interact through MPC-based data exchange, enabling device 200 to obtain aggregated features after aggregating multiple extracted features.

[0103] In one implementation, the aforementioned multiple devices 100 and 200 can interact via data exchange based on algorithms such as secret sharing algorithms or homomorphic encryption, enabling device 200 to obtain aggregated features after aggregating multiple extracted features. The implementation process is explained below using the secret sharing algorithm as an example.

[0104] Any device 100, based on the secret sharing algorithm, splits its own extracted features into multiple feature fragments and sends the multiple feature fragments to other devices 100 respectively, and receives the feature fragments sent by other devices 100; it uses its own feature fragments and the feature fragments of other devices 100 to determine its own secret state features, and sends them to device 200.

[0105] Device 200 receives dense-state features sent by multiple devices 100 respectively, and determines aggregate features based on the multiple dense-state features. Device 200 can determine the aggregate feature as the sum or mean of the multiple dense-state features.

[0106] When splitting and extracting features, device 100 can split based on the number of devices 100. For example, if the number of devices 100 is m, the extracted features can be split into m feature fragments. For any device 100, the m feature fragments obtained from its splitting can be reconstructed to obtain the extracted features. Reconstruction can be an addition operation, or an operation of multiplying by a set coefficient and then adding. Even if m-1 feature fragments are sent to other devices 100, the other devices 100 cannot reconstruct the extracted features based on these m-1 feature fragments. Therefore, this implementation method is easy to implement and can effectively protect the privacy data of the extracted features from being leaked.

[0107] When determining the dense state features, device 100 can reconstruct them by combining its own feature fragments with the feature fragments of other devices 100. The dense state features of any device 100 are obtained by reconstructing the feature fragments of multiple devices 100. When device 200 receives multiple dense state features, it cannot separate the feature fragments of each device 100 from the multiple dense state features, and therefore cannot reconstruct the extracted features of device 100.

[0108] For ease of explanation, the above implementation method is illustrated below using three devices 100 as an example, and each device 100 is referred to as Device 1, Device 2, and Device 3 respectively. The extracted features are represented in matrix form.

[0109] Device 1 splits its own extracted feature matrix L1 into fragments [L1]1, [L1]2 and [L1]3, and sends [L1]2 and [L1]3 to Device 2 and Device 3 respectively;

[0110] Device 2 splits its own extracted feature matrix L2 into fragments [L2]1, [L2]2 and [L2]3, and sends [L2]1 and [L2]3 to Device 1 and Device 3 respectively;

[0111] Device 3 splits its own extracted feature matrix L3 into fragments [L3]1, [L3]2 and [L3]3, and sends [L3]1 and [L3]2 to Device 1 and Device 2 respectively.

[0112] Next, device 1 can calculate the dense state feature [L1] using the formula [L1] = [L1]1 + [L2]1 + [L3]1, and send [L1] to device 200;

[0113] Device 2 can calculate the dense state feature [L2] using the formula [L2] = [L1]² + [L2]² + [L3]², and send [L2] to device 200;

[0114] Device 3 can calculate the dense state feature [L3] using the formula [L3] = [L1]3 + [L2]3 + [L3]3, and send [L3] to device 200.

[0115] In this way, device 200 can receive [L1] sent by device 1, [L2] sent by device 2, and [L3] sent by device 3, and use formula L 聚合 = [L1] + [L2] + [L3] or L 聚合 = ([L1]+[L2]+[L3]) / 3, to obtain the aggregated features.

[0116] Throughout the entire data interaction and transmission process, device 100 did not send the extracted features of the plaintext to the outside, which enabled device 200 to obtain the aggregated features.

[0117] In step S240, device 200 uses aggregated features to determine the output data of the second part model NN2, determines the second gradient slice g2 for updating multiple first part models NN1 based on the output data and aggregated features, and sends it to multiple devices 100. Multiple devices 100 respectively receive the second gradient slice g2 sent by device 200.

[0118] When determining the output data of the second part model NN2 using aggregated features, device 200 can directly input the aggregated features into the second part model NN2, or it can preprocess the aggregated features and input the preprocessed second input data c into the second part model NN2.

[0119] In different application scenarios, the device 200 has different implementation methods when determining the second gradient fragment g2, which will be described in detail later and will not be repeated here.

[0120] When determining the second gradient slice g2, partial derivatives can be used. For privacy reasons, device 200 cannot extract features from every single device 100, but instead obtains aggregated features. Therefore, the second gradient slice g2 determined based on the output data and aggregated features is a gradient slice applicable to multiple first-part models NN1. Thus, device 200 determines the same second gradient slice g2 for different devices 100.

[0121] When sending the second gradient fragment g2, device 200 can send the second gradient fragment g2 to different devices 100 one by one, or it can broadcast the second gradient fragment g2 to different devices 100.

[0122] In step S250, multiple devices 100 update their own first part model NN1 based on the second gradient partition g2 and the first gradient partition g1 determined by themselves. In this embodiment, when updating the first part model NN1, the complete gradient needs to be used for updating, and the first gradient partition g1 and the second gradient partition g2 are used to determine the complete gradient.

[0123] Any device 100 can determine a first gradient G1 for its own first part model NN1 based on the product of the second gradient segment g2 and its own determined first gradient segment g1, and then update the first part model NN1 using the first gradient G1. This first gradient G1 is the complete gradient used to update the first part model NN1.

[0124] When the first part of the model NN1 contains multiple computation layers, the device 100 can update the model parameters in any computation layer by using the first gradient partition g1 and the second gradient partition g2 of that computation layer.

[0125] Specifically, for any given computational layer, the first gradient G1 can be determined based on the product of its corresponding first gradient slice g1 and second gradient slice g2. Then, the device 100 can update the computational layer using the first gradient G1. For multiple computational layers, the model parameters of each layer can be updated sequentially from back to front.

[0126] In this embodiment, after obtaining the model's output data, device 200 can determine the second gradient partition g2 based on the output data and aggregated features, thereby sharing some of the computational load of device 100. When device 200 has higher computing power, it can share some of the computational load of device 100, enabling device 100 to complete the model update to the computation stage more quickly and improving the overall efficiency of the model training process.

[0127] Now let's return to steps S210 to S240 above.

[0128] In step S210, after obtaining the extracted features of the sample, device 100 transmits them to other computational layers of the model in step S230. Without receiving the gradient of the predicted loss for this model iteration from subsequent computational layers, it cannot perform the model update operation. Therefore, it remains in a waiting state, waiting for device 200 to feed back gradient and other data to device 100.

[0129] To further improve overall data processing efficiency, in this embodiment, device 100 can also make full use of the waiting time to determine the first gradient fragment g1, i.e., execute step S220. That is, device 100 can perform the step of determining the first gradient fragment g1 based on extracted features before receiving the second gradient fragment g2 sent by device 200 (in step S240), thereby making full use of device 100's waiting time. After receiving the second gradient fragment g2 from device 200, device 100 can use the first gradient fragment g1 and the second gradient fragment g2 to more quickly update the first part of the model NN1, thereby improving the overall processing efficiency between device 100 and device 200.

[0130] The following explanations will focus on steps S240 and S250 in different application scenarios.

[0131] In peer-to-peer network application scenarios, multiple devices 100 are sample owners and devices 200 are label owners. The business prediction model is divided into two parts, with device 200 responsible for the calculation of the latter part of the model and the loss calculation.

[0132] In step S240, the output data of the second part model NN2 is the label prediction value.

[0133] When determining the second gradient partition g2 based on the output data and aggregation features, the device 200 can specifically determine the prediction loss of the sample based on the output data; then, it uses the prediction loss and aggregation features to determine the second gradient partition g2.

[0134] When determining the second gradient partition g2, it can be based on the partial derivative of the prediction loss with respect to the aggregated features. Specifically, the computational graph of the second part model NN2 can be obtained, and the partial derivative of the prediction loss with respect to the aggregated features can be used to determine the second gradient partition g2. Here, the prediction loss can be a numerical value, and the aggregated feature L... 聚合 It can be a numerical matrix.

[0135] In this implementation scenario, the output data can be understood as the prediction result of the sample, which is the label prediction value (i.e., prediction data) obtained after the sample is processed by the first part model NN1 and the second part model NN2.

[0136] When determining the prediction loss, device 200 can utilize the difference between the predicted label values ​​and the labeled data of the samples to determine the prediction loss of the samples. Common loss functions can be used to determine the prediction loss of the samples. From inputting the samples into the business prediction model to obtaining the predicted data of the samples and determining the prediction loss of the samples, this process can be considered the forward computation stage. Next, device 200 can continue to perform the backward computation stage.

[0137] Device 200 can determine the gradient data for updating the second part model NN2 based on the prediction loss, and use this gradient data to update the second part model NN2, that is, to update the model parameters of the second part model NN2. The second part model NN2 may contain multiple computational layers. Updating the model parameters in multiple computational layers using the aforementioned gradient data can be done in a conventional manner, the specific method of which will not be detailed here.

[0138] Any of the above first gradient slices g1 can be a Jacobian matrix obtained by using the partial derivatives of the extracted features with respect to the computational layers in the first part of the model NN1, and the second gradient slice g2 can be a Jacobian matrix obtained by using the partial derivatives of the prediction loss with respect to the aggregated features. When the computational layer W1 in the first part of the model NN1 is represented by a two-dimensional matrix, and the aggregated features are also represented by a two-dimensional matrix, the first gradient slice g1 of the computational layer W1 can be represented by a four-dimensional matrix.

[0139] In step S250, when calculating the product of the first gradient piece g1 and the second gradient piece g2, in order to perform matrix multiplication on the specified dimensions of the first gradient piece g1 and the second gradient piece g2, the first gradient piece g1 and the second gradient piece g2 can be flattened in a preset manner. Based on the product of the flattened first gradient piece g1 and the second gradient piece g2, the first gradient G1 is obtained.

[0140] The following explanation will focus on steps S240 and S250 in the context of a client-server architecture application scenario.

[0141] Figure 3 This is a schematic diagram illustrating an application scenario of a multi-party joint training method provided in this embodiment. In this application scenario, multiple devices 100 are sample-owning devices, and device 200 is a server. The business prediction model also includes a third part, model NN3, configured in a third device (represented by device 300), which is the label owner. Device 300 (i.e., the third device) can be implemented using any device, equipment, platform, device cluster, etc., with computing and processing capabilities.

[0142] In practical applications, the computing power of sample-owning devices is relatively low, making it impossible to perform complex model calculations, while servers have higher configurations and can handle more demanding computational tasks. Therefore, the business prediction model can be split, with the sample-owning devices receiving the first few layers, the tag-owning devices receiving the last few layers, and the servers receiving the middle layers. Servers typically receive the majority of the middle computational layers, leveraging their superior computing power to handle most of the model calculations.

[0143] Thus, the business prediction model is divided into three parts: the first part, model NN1, owned by device 100 (the sample owner); the second part, model NN2, owned by device 200 (the server); and the third part, model NN3, owned by device 300 (the tag owner). Sample data is input into each part of the model in the order of first part model NN1, then second part model NN2, and finally third part model NN3 to complete the forward computation phase of model training.

[0144] During the forward computation phase, device 200 sends the output data to device 300 and determines the third gradient slice g3 for updating the second part model NN2 based on the output data and the second part model NN2.

[0145] Device 300 receives the output data sent by device 200, uses the output data and the third part model NN3 to determine the predicted label value of the sample, and uses the predicted label value to determine the prediction loss of the sample.

[0146] During the backward computation phase, device 300 can directly update the third part model NN3 based on the prediction loss. The specific implementation can be carried out in a conventional manner, and will not be elaborated here.

[0147] To enable devices 200 and 100 to update parts of their models, device 300 further determines a fourth gradient slice g4 for updating the first part of the model NN1 and the second part of the model NN2 based on the aforementioned prediction loss and output data, and sends this fourth gradient slice g4 to device 200. Specifically, the fourth gradient slice g4 can be determined based on the partial derivative of the prediction loss with respect to the output data a. Specifically, the computation graph of the third part of the model NN3 can be obtained, and the partial derivative of the prediction loss with respect to the output data can be calculated using this computation graph to obtain the fourth gradient slice g4. Here, the output data a can be in the form of a numerical matrix.

[0148] Then, device 200 receives the fourth gradient fragment g4 sent by device 300, and updates its own second part model NN2 using the fourth gradient fragment g4 and the third gradient fragment g3. Next, when determining the second gradient fragment g2, device 200 can specifically use the output data and aggregated features to determine the fifth gradient fragment g5, determine the second gradient fragment g2 based on the fifth gradient fragment g5, and send the second gradient fragment g2 to device 100.

[0149] Before receiving the fourth gradient slice g4 sent by device 300, device 200 can use the time waiting for device 300 to provide feedback on the fourth gradient slice g4 to determine the third gradient slice g3, thereby making full use of the training time and improving the processing efficiency of the device.

[0150] In this embodiment, the third gradient slice g3 and the fourth gradient slice g4 are used to determine the complete gradient for updating the second part of the model NN2, while the complete gradient for updating the first part of the model NN1 can be determined based on the fourth gradient slice g4, the second gradient slice g2, and the first gradient slice g1. When performing step S250, the device 100 needs to use the fourth gradient slice g4, the second gradient slice g2, and the first gradient slice g1, and various implementation methods can be used to update the first part of the model NN1.

[0151] In one implementation, when determining the second gradient partition g2 based on the fifth gradient partition g5, device 200 can use the product of the fourth gradient partition g4 and the fifth gradient partition g5 to determine the second gradient partition g2. In this way, device 200 can combine the information from the fourth gradient partition g4 into the second gradient partition g2, and device 100 can directly update its first part model NN1 based on the second gradient partition g2 and its own determined first gradient partition g1.

[0152] In another implementation, device 300 can directly send the fourth gradient fragment g4 to multiple devices 100. Device 200 directly identifies the fifth gradient fragment g5 as the second gradient fragment g2.

[0153] Any device 100 receives the fourth gradient segment g4 sent by device 300. When executing step S250, it updates its own first part model NN1 based on the fourth gradient segment g4, the second gradient segment g2 and the first gradient segment g1 determined by itself.

[0154] Specifically, the first gradient G1 can be determined based on the product of the fourth gradient slice g4, the second gradient slice g2, and the first gradient slice g1, and the first gradient G1 can be used to update the first part of the model NN1.

[0155] Steps S210 to S250 above can be understood as a model iteration process. During the training of the business prediction model, this iteration process can be executed multiple times until the model reaches a convergence condition. The convergence condition may include the prediction loss being less than a preset value, or the model iteration count reaching a preset number, etc. In a single model iteration process, the steps from step S210 to determining the prediction loss can be performed on a batch of samples or on a single sample.

[0156] The following examples illustrate this point. Figure 3 The illustrated embodiment will be used for explanation.

[0157] In the multi-party joint training task with vertical segmentation, assuming there are m training devices (i.e., sample-owning devices), multiple training devices perform PSI (Private Set Intersection) processing on the sample data in their respective sample training sets to obtain the sample space X of each training device. i Where i = 1, 2, ..., m. The labeled dataset is y in the training device k. k .

[0158] Step 1, Model Distribution and Initialization. The business prediction model is a DNN model W. The server acts as the management device, initializing model W and sequentially dividing it into training device i and model W1. i (i.e., the first part of the model), server model W s (i.e., the second part of the model) and the model W of the training device k. k (i.e., the third part of the model), and model W i The model W is distributed to each training device i. k Distribute to training device k.

[0159] Step 2, forward propagation of the training device. During any training iteration, the training device i reads its respective sample training set X. i The same batch of samples, with a sample size of n, are used to develop model W. i The forward propagation yields the output layer result L. i (i.e., feature extraction);

[0160] Training device i utilizes a secret-sharing algorithm to train L. i Split into two parts [L i ]1 and, and [L i ] j (i≠j) Transmit to other training devices i, and [L i ] i Keep it locally;

[0161] Training device i receives [L] sent by other training members i ] j This leads to the dense state feature [L] i ] = [L i ] i +[L i ] j +…, will [L] i Transmitted to the server;

[0162] Training device i utilizes model W i The calculation graph is used to calculate L. i For W i Each computational layer W i,l The partial derivative Jacobian matrix J i,l J i,l This is the first gradient partitioning.

[0163] Step 3, server forward propagation. The server receives the dense-state features [L] of all training devices. i (i = 1, 2), using L 聚合 =∑[L i Decrypting it yields the aggregated feature L. 聚合 The server utilizes L 聚合 Model W s The forward propagation yields the output data L. out and L out Send to training device k;

[0164] The server utilizes model W s The calculation graph is used to calculate L. out For model W s Each layer of the model W s,l The partial derivative Jacobian matrix J s,l (i.e., the third gradient partition), and L out For L 聚合 The partial derivative Jacobian matrix J s (That is, the fifth gradient partition).

[0165] Step 4, loss function calculation. Training member k receives output data L sent by the server. out And based on L out Continue with model W k The forward propagation yields the predicted label values. The trained member k combines the predicted label values ​​with the true label values ​​y. k Obtain the loss function value Loss (i.e., prediction loss), and use model W k Backpropagation is performed on the computation graph to update model W. k ;

[0166] Training member k using Wk The computational graph calculates the Loss with respect to the output data L. out The partial derivative Jacobian matrix J k (i.e., fourth gradient partitioning), and J k Send to the server.

[0167] Step 5, backpropagation. The server receives J sent by training member k. k And divide the fourth gradient into J k and the third gradient partition J s,l Flatten the matrix along the corresponding dimension, then perform matrix multiplication on the flattened matrix to obtain model W. s gradient G s Then, using formula W s =W s -αG s The model is updated, where α is a preset parameter.

[0168] For example, suppose L out The matrix dimension is 10*20, W s,l The matrix dimension is 5*4, J k The matrix dimension is 10*20, J s,l Based on L out and W s,l The obtained four-dimensional matrix has dimensions of 10*20*5*4. After flattening, J can be... k Flatten J into a 1*200 dimensional vector by concatenating rows. s,l Flatten it to 200*1*5*4. Then flatten J... k With J s,l Multiplying, we get the gradient G. s It is a 5*4 dimensional matrix.

[0169] The server will partition the fourth gradient J. k and the fifth gradient partition J s Flatten the matrix along the corresponding dimension, then perform matrix multiplication on the flattened matrix to obtain J. i (i.e., second gradient partitioning), and J i Send them to training member i respectively.

[0170] Training device i will divide the second gradient into slices J. i and the first gradient partition J i,l By flattening and matrix multiplying the corresponding dimensions, we obtain the model W for training device i. i gradient G i Therefore, according to formula W i =W i -αG i Update the model.

[0171] Repeat steps 2-5 above multiple times until the business prediction model converges, thus completing the model training.

[0172] The above Figure 3 The illustration shows only one implementation scenario in this specification, where the server undertakes a significant portion of the processing tasks for the sample owner's device, thereby minimizing the computational load on the sample owner's device. In specific applications, the server and sample owner's device can also employ other implementation methods to rationally coordinate computational tasks.

[0173] The above Figure 2 These explanations all focus on multi-device collaborative training methods. The following... Figure 4 and Figure 5 The method flow is then described using the first device and the second device as the executing entities, respectively. For specific implementation details, please refer to [link to relevant documentation]. Figure 2 The description of the illustrated embodiment will not be repeated here.

[0174] Figure 4 This is a flowchart illustrating a privacy-preserving multi-party joint training method provided in an embodiment. The second device and multiple first devices each possess a second part of the business prediction model and multiple first part models, respectively. The method is executed by any one of the first devices and includes the following steps S410–S450.

[0175] Step S410: Determine the extracted features of the samples owned by the user through its own first part model.

[0176] Step S420: Determine the first gradient slice for updating the first part of the model based on the extracted features.

[0177] Step S430: Through data interaction based on multi-party secure computation between the second device and other first and second devices, the second device obtains aggregated features after aggregating multiple extracted features, and the second device determines the second gradient slice for updating multiple first part models based on the output data, and sends it to the first device; wherein, the output data is determined based on the aggregated features and the second part models.

[0178] Step S440: Receive the second gradient fragment sent by the second device.

[0179] Step S450: Update the first part of the model based on the second gradient partition and the first gradient partition.

[0180] Figure 5This is a flowchart illustrating a privacy-preserving multi-party joint training method provided in an embodiment. The second device and multiple first devices each possess a second part of the business prediction model and multiple first part models, respectively. The method is executed by the second device and includes the following steps S510–S540.

[0181] Step S510: Through data interaction based on multi-party secure computation with multiple first devices, aggregated features are obtained after aggregating the extracted features from the multiple first devices. The multiple extracted features are determined using samples from their respective first devices and a first part of the model.

[0182] Step S520: Use aggregated features to determine the output data of the second part of the model.

[0183] Step S530: Determine the second gradient slices for updating multiple first part models based on the output data.

[0184] Step S540: The second gradient slice is sent to multiple first devices, so that any one of the first devices can update the first part of the model based on the second gradient slice and the first gradient slice. The first gradient slice is determined based on its respective extracted features and the first part of the model.

[0185] In this specification, the terms "first," "first part model," and "first gradient piece," as well as the corresponding "second," are used merely for ease of distinction and description and do not have any limiting meaning.

[0186] The foregoing description describes specific embodiments of this specification; other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than those shown in the embodiments, and the desired result may still be achieved. Furthermore, the processes depicted in the drawings do not necessarily need to follow the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0187] Figure 6 This is a schematic block diagram of a privacy-preserving multi-party joint training system provided for an embodiment. The system 600 includes a second device 620 and multiple first devices 610. The business prediction model is jointly trained at least through the first device 610 and the second device 620, and the second device 620 and the multiple first devices 610 respectively possess a second part of the business prediction model and multiple first part models. This system embodiment is similar to... Figure 2 The method embodiments shown correspond to those described.

[0188] Any first device 610 is used to determine the extracted features of its own samples through its own first part model, and to determine a first gradient slice for updating the first part model based on the extracted features.

[0189] The first device 610 and the second device 620 interact through data interaction based on multi-party secure computation, so that the second device 620 obtains aggregated features after aggregating multiple extracted features;

[0190] The second device 620 is used to determine the output data of the second part model using the aggregation features, determine the second gradient slice for updating the multiple first part models based on the output data and aggregation features, and send it to the multiple first devices 610;

[0191] Multiple first devices 610 are used to receive the second gradient fragment sent by the second device 620, and update their own first part model based on the second gradient fragment and the first gradient fragment determined by themselves.

[0192] In one embodiment, multiple first devices 610 and second devices 620 interact through data exchange based on a secret sharing algorithm, so that the second device 620 obtains aggregated features after aggregating multiple extracted features.

[0193] In one implementation, any first device 610 is configured to split its own extracted features into multiple feature fragments based on a secret sharing algorithm, and send the multiple feature fragments to other first devices 610 respectively, and receive the feature fragments sent by the other first devices 610; use its own feature fragments and the feature fragments of the other first devices 610 to determine its own secret state features, and send them to the second device 620;

[0194] The second device 620 is used to receive dense state features sent by multiple first devices 610 respectively, and to determine the aggregate feature based on the multiple dense state features.

[0195] In one embodiment, the second device 620, when determining the aggregate feature based on multiple dense state features, includes determining the sum or mean of the multiple dense state features as the aggregate feature.

[0196] In one implementation, the output data is a label prediction value; the second device 620, when determining a second gradient slice for updating multiple first part models based on the output data and aggregated features, includes:

[0197] The prediction loss of the sample is determined based on the output data;

[0198] The second gradient slice is determined using the prediction loss and the aggregation features.

[0199] In one implementation, the business forecasting model further includes a third part model configured in a third device 630; the method further includes:

[0200] The second device 620 is further configured to send the output data to the third device 630, and determine a third gradient slice for updating the second part of the model based on the output data and the second part of the model;

[0201] The third device 630 is used to receive output data sent by the second device 620, determine the label prediction value of the sample using the output data and the third part model, determine the prediction loss of the sample using the label prediction value, and determine a fourth gradient slice for updating the first part model and the second part model based on the prediction loss and the output data, and send it to the second device 620.

[0202] The second device 620 is further configured to receive a fourth gradient slice sent by the third device 630, update the second part of the model using the fourth gradient slice and the third gradient slice; determine a fifth gradient slice using the output data and the aggregated features, and determine the second gradient slice based on the fifth gradient slice.

[0203] In one embodiment, when the second device 620 determines the second gradient slice based on the fifth gradient slice, it includes determining the second gradient slice by using the product of the fourth gradient slice and the fifth gradient slice.

[0204] In one embodiment, the third device 630 is further configured to send the fourth gradient fragment to the first device 610;

[0205] The second device 620 is specifically used to directly determine the fifth gradient slice as the second gradient slice;

[0206] The first device 610 is further configured to receive the fourth gradient slice sent by the third device, and update its own first part model based on the fourth gradient slice, the second gradient slice and the first gradient slice determined by itself.

[0207] In one implementation, the first device 610 is specifically configured to determine a first gradient slice for updating the first part of the model based on the extracted features before receiving the second gradient slice sent by the second device 620.

[0208] In one implementation, the first device 610 is specifically used to determine the first gradient slice based on the partial derivatives of the extracted features with respect to the first part of the model.

[0209] In one implementation, when the first device 610 updates its first part model based on the second gradient partition and its own determined first gradient partition, it includes:

[0210] Based on the product of the second gradient slice and the first gradient slice determined by itself, the first gradient of the first part of the model is determined;

[0211] The first gradient is used to update the first part of the model.

[0212] Figure 7 This is a schematic block diagram of a privacy-preserving multi-party joint training device provided in an embodiment. The business prediction model is jointly trained using at least a first device and a second device, with the second device and multiple first devices each possessing a second part of the business prediction model and multiple first part models. This device embodiment is similar to... Figure 2 The method embodiment shown corresponds to the method executed by device 100. The device 700 is deployed in any of the first devices and includes:

[0213] The first determining module 710 is configured to determine the extracted features of its own samples through its own first part model;

[0214] The second determining module 720 is configured to determine a first gradient slice for updating the first part of the model based on the extracted features.

[0215] The first interaction module 730 is configured to, through data interaction based on multi-party secure computation with other first devices and second devices, enable the second device to obtain aggregated features after aggregating multiple extracted features, and enable the second device to determine a second gradient slice for updating multiple first part models based on output data and aggregated features, and send it to the first device; wherein, the output data is determined based on the aggregated features and the second part models;

[0216] The first receiving module 740 is configured to receive the second gradient fragment sent by the second device.

[0217] The first update module 750 is configured to update the first part of the model based on the second gradient slice and the first gradient slice.

[0218] In one implementation, the first interaction module 730 is specifically configured as follows:

[0219] Through data interaction based on a secret sharing algorithm between the second device and other first and second devices, the second device obtains aggregated features after aggregating multiple extracted features.

[0220] In one implementation, the first interaction module 730 includes:

[0221] The splitting submodule (not shown in the figure) is configured to split its own extracted features into multiple feature fragments based on the secret sharing algorithm, and send the multiple feature fragments to other first devices respectively;

[0222] The receiving submodule (not shown in the figure) is configured to receive feature fragments sent by the other first device;

[0223] The interaction submodule (not shown in the figure) is configured to determine its own dense-state features using its own feature fragments and the feature fragments of the other first devices, and send them to the second device so that the second device can determine the aggregated features based on multiple dense-state features.

[0224] In one implementation, the business forecasting model further includes a third part model, configured in a third device;

[0225] The first receiving module 740 is further configured to receive a fourth gradient slice sent by the third device; wherein the fourth gradient slice is used to update the first part model and the second part model, the fourth gradient slice is determined by the third device based on the prediction loss and the output data, the prediction loss is obtained based on the output data and the label prediction value of the sample determined by the third part model, and the output data is sent by the second device to the third device.

[0226] The first update module 750 is specifically configured to update its own first part of the model based on the fourth gradient partition, the second gradient partition, and the first gradient partition determined by itself.

[0227] In one implementation, the second determining module 720 is specifically configured as follows:

[0228] Before receiving the second gradient slice sent by the second device, a first gradient slice for updating the first part of the model is determined based on the extracted features.

[0229] In one implementation, the second determining module 720 is specifically configured to determine the first gradient slice based on the partial derivatives of the extracted features with respect to the first part of the model.

[0230] In one implementation, the first update module 750 is specifically configured to determine a first gradient for its own first part of the model based on the product of the second gradient partition and the first gradient partition determined by itself, and update the first part of the model using the first gradient.

[0231] Figure 8 This is a schematic block diagram of a privacy-preserving multi-party joint training device provided in an embodiment. The business prediction model is jointly trained using at least a first device and a second device, with the second device and multiple first devices each possessing a second part of the business prediction model and multiple first part models. This device embodiment is similar to... Figure 2 The method performed by device 200 in the illustrated embodiment corresponds to the method described. The device 800 is deployed in a second device and includes:

[0232] The second interaction module 810 is configured to obtain aggregated features after aggregating the extracted features of multiple first devices through data interaction based on multi-party secure computation with multiple first devices; wherein the multiple extracted features are determined using samples and a first part of the model of each of the first devices.

[0233] The third determining module 820 is configured to determine the output data of the second part of the model using the aggregation features;

[0234] The fourth determining module 830 is configured to determine a second gradient slice for updating multiple first part models based on the output data and aggregated features;

[0235] The first sending module 840 is configured to send the second gradient slice to multiple first devices, so that any one of the first devices can update the first part of the model based on the second gradient slice and the first gradient slice; wherein the first gradient slice is determined based on its respective extracted features and the first part of the model.

[0236] In one implementation, the second interaction module 810 is specifically configured to obtain aggregated features after aggregating the extracted features of the multiple first devices through data interaction with multiple first devices based on a secret sharing algorithm.

[0237] In one implementation, the second interaction module 810 includes:

[0238] The first submodule (not shown in the figure) is configured to receive secret features sent by multiple first devices respectively; wherein, the secret features of any one first device are determined by its own feature fragments and feature fragments sent by other first devices, and the feature fragments are obtained by the first device by splitting the extracted features based on the secret sharing algorithm;

[0239] The second submodule (not shown in the figure) is configured to determine the aggregation feature based on multiple dense state features.

[0240] In one implementation, the second submodule is specifically configured to determine the sum or mean of multiple dense-state features as the aggregated feature.

[0241] In one implementation, the output data is a predicted label value; the fourth determining module 830 is specifically configured as follows:

[0242] The prediction loss of the sample is determined based on the output data, and the second gradient slice is determined using the prediction loss and the aggregated features.

[0243] In one embodiment, the business forecasting model further includes a third part model, configured in a third device; the device further includes:

[0244] The fifth determining module (not shown in the figure) is configured to determine the third gradient slice for updating the second part of the model based on the output data and the second part of the model.

[0245] The second sending module (not shown in the figure) is configured to send the output data to the third device;

[0246] The second receiving module (not shown in the figure) is configured to receive the fourth gradient slice sent by the third device; wherein the fourth gradient slice is used to update the first part of the model and the second part of the model, and the fourth gradient slice is determined by the third device based on the prediction loss and the output data. The prediction loss is obtained based on the output data and the label prediction value of the sample determined by the third part of the model, and the output data is sent by the second device to the third device.

[0247] The second update module (not shown in the figure) is configured to update the second part of the model using the fourth gradient partition and the third gradient partition.

[0248] The fourth determining module 830 is specifically configured to determine the fifth gradient partition using the output data and the aggregation features, and to determine the second gradient partition based on the fifth gradient partition.

[0249] In one implementation, the fourth determining module 830, when determining the second gradient slice based on the fifth gradient slice, includes:

[0250] The second gradient slice is determined by multiplying the fourth gradient slice and the fifth gradient slice.

[0251] In one implementation, when the fourth determining module 830 determines the second gradient slice based on the fifth gradient slice, it includes directly determining the fifth gradient slice as the second gradient slice.

[0252] In one implementation, the fifth determining module is specifically configured to determine the third gradient slice based on the partial derivatives of the output data with respect to the model parameters in the second part of the model.

[0253] In one implementation, the second update module is specifically configured to determine a second gradient for the second part of the model based on the product of the fourth gradient slice and the third gradient slice, and update the second part of the model using the second gradient.

[0254] The above-described apparatus embodiments correspond to the method embodiments, and detailed descriptions can be found in the description of the method embodiments section, which will not be repeated here. The apparatus embodiments are derived based on the corresponding method embodiments and have the same technical effects as the corresponding method embodiments; detailed descriptions can be found in the corresponding method embodiments.

[0255] This specification also provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform... Figures 1 to 5 Any one of the methods described.

[0256] This specification also provides a computing device, including a memory and a processor, wherein the memory stores executable code, and the processor executes the executable code to implement... Figures 1 to 5 Any one of the methods described.

[0257] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments for storage media and computing devices are basically similar to the method embodiments, so they are described more simply; relevant parts can be referred to the descriptions of the method embodiments.

[0258] Those skilled in the art will recognize that the functions described in the embodiments of the present invention in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.

[0259] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, or improvements made based on the technical solutions of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for privacy-preserving federated training of multiple parties, wherein, The second device and the plurality of first devices respectively own a second partial model and a plurality of first partial models in a business prediction model, and the method comprises: Any one of the first devices determines an extracted feature of a sample owned by itself through the first partial model of itself, and determines a first gradient slice for updating the first partial model based on the extracted feature; the first gradient slice is not a complete gradient; Through data interaction based on multi-party secure calculation between the plurality of first devices and the second device, the second device obtains an aggregated feature after aggregation of a plurality of extracted features; The second device determines output data of the second partial model by using the aggregated feature, determines a second gradient slice for updating the plurality of first partial models based on the output data and the aggregated feature, and sends the second gradient slice to the plurality of first devices; the second gradient slice is not a complete gradient; The plurality of first devices respectively receive the second gradient slice sent by the second device, determine a first gradient for the first partial model of itself based on a product of the second gradient slice and the first gradient slice determined by itself, update the first partial model of itself by using the first gradient, and the first gradient is a complete gradient.

2. The method of claim 1, wherein, Through data interaction based on a secret sharing algorithm between the plurality of first devices and the second device, the second device obtains an aggregated feature after aggregation of a plurality of extracted features.

3. The method of claim 2, wherein, Any one of the first devices splits its own extracted feature into a plurality of feature slices based on a secret sharing algorithm, sends the plurality of feature slices to other first devices respectively, receives feature slices sent by the other first devices, determines a ciphertext feature of itself by using the feature slice of itself and the feature slices of the other first devices, and sends the ciphertext feature to the second device; The second device receives the ciphertext features sent by the plurality of first devices respectively, and determines the aggregated feature based on the plurality of ciphertext features.

4. The method of claim 3, wherein the step of determining the aggregated feature based on the plurality of ciphertext features comprises determining a sum or mean value of the plurality of ciphertext features as the aggregated feature.

5. The method of claim 1, wherein, The output data is a label prediction value; when determining the second gradient slice for updating the plurality of first partial models based on the output data and the aggregated feature, the second device comprises: Determining a prediction loss of the sample based on the output data; Determining the second gradient slice by using the prediction loss and the aggregated feature.

6. The method of claim 1, wherein the business prediction model further comprises a third partial model and is configured in a third device; the method further comprises: The second device sends the output data to the third device, and determines a third gradient slice for updating the second partial model based on the output data and the second partial model; and The third device receives the output data sent by the second device, determines a third gradient slice for updating the third partial model based on the output data and the second partial model, and sends the third gradient slice to the second device. The third device determines a label prediction value of the sample by using the output data and the third partial model, determines a prediction loss of the sample by using the label prediction value, determines a fourth gradient slice for updating the first partial model and the second partial model based on the prediction loss and the output data, and sends the fourth gradient slice to the second device; The second device receives the fourth gradient slice sent by the third device, and updates the second partial model by using the fourth gradient slice and the third gradient slice. A fifth gradient slice is determined by using the output data and the aggregated features, and the second gradient slice is determined based on the fifth gradient slice.

7. The method of claim 6, wherein the step of determining the second gradient slice based on the fifth gradient slice comprises: The second gradient slice is determined by using the product of the fourth gradient slice and the fifth gradient slice.

8. The method of claim 6, wherein The third device further sends the fourth gradient slice to the first device; The second device directly determines the fifth gradient slice as the second gradient slice; The first device further receives the fourth gradient slice sent by the third device, and updates the first partial model of the first device based on the fourth gradient slice, the second gradient slice, and the first gradient slice determined by the first device itself.

9. The method of claim 1, wherein the step of determining the first gradient slice for updating the first partial model based on the extracted features comprises: The first gradient slice for updating the first partial model is determined based on the extracted features before receiving the second gradient slice sent by the second device.

10. The method of claim 1, wherein the step of determining the first gradient slice for updating the first partial model based on the extracted features comprises: The first gradient slice is determined based on the partial derivative of the first partial model with respect to the extracted features.

11. A method of privacy-preserving federated training of multiple parties, wherein, The second device and the plurality of first devices respectively own a second partial model and a plurality of first partial models in a business prediction model, and the method is executed by any one of the first devices, and the method comprises: Determining extracted features of a sample owned by the first device by using the first partial model of the first device; Determining a first gradient slice for updating the first partial model based on the extracted features; the first gradient slice is not a complete gradient; Through data interaction between the first device and other first devices and the second device based on multi-party secure calculation, the second device obtains aggregated features obtained by aggregating a plurality of extracted features, and the second device determines a second gradient slice for updating a plurality of first partial models based on output data and the aggregated features, and sends the second gradient slice to the first device; wherein the output data is determined based on the aggregated features and the second partial model; the second gradient slice is not a complete gradient; Receiving the second gradient slice sent by the second device; Determine a first gradient for the first part model of the second device based on the product of the second gradient slice and the first gradient slice, update the first part model of the second device by using the first gradient, and the first gradient is a complete gradient.

12. A method of privacy-preserving federated training, wherein, The second device and the plurality of first devices respectively own a second part model and a plurality of first part models in a business prediction model, and the method is executed by the second device, comprising: Obtaining aggregated features of the plurality of extracted features by data interaction between the second device and the plurality of first devices based on multi-party secure calculation; wherein the plurality of extracted features are respectively determined by using the sample and the first part model of each first device; Determine the output data of the second part model by using the aggregated features; Determine the second gradient slice for updating the plurality of first part models based on the output data and the aggregated features; the second gradient slice is not a complete gradient; Send the second gradient slice to the plurality of first devices, so that any one of the first devices updates the first part model by using the first gradient; wherein the first gradient is obtained based on the product of the second gradient slice and the first gradient slice, the first gradient slice is determined based on the extracted feature and the first part model, the first gradient slice is not a complete gradient, and the first gradient is a complete gradient.

13. A privacy-preserving multi-party federated training system comprising a second device and a plurality of first devices; wherein, The second device and the plurality of first devices respectively own a second part model and a plurality of first part models in a business prediction model; Any one of the first devices is configured to determine the extracted feature of the sample owned by itself by using the first part model of itself, and determine the first gradient slice for updating the first part model based on the extracted feature; The first gradient slice is not a complete gradient; The plurality of first devices and the second device are configured to obtain the aggregated features of the plurality of extracted features by data interaction based on multi-party secure calculation; The second device is configured to determine the output data of the second part model by using the aggregated features, determine the second gradient slice for updating the plurality of first part models based on the output data and the aggregated features, and send the second gradient slice to the plurality of first devices; the second gradient slice is not a complete gradient; The plurality of first devices are configured to respectively receive the second gradient slice sent by the second device, determine the first gradient for the first part model of itself based on the product of the second gradient slice and the first gradient slice determined by itself, update the first part model of itself by using the first gradient, and the first gradient is a complete gradient.

14. A privacy-preserving multi-party co-training apparatus, wherein, The second device and the plurality of first devices respectively own a second part model and a plurality of first part models in a business prediction model, and the device is deployed in any one of the first devices, comprising: The first determination module is configured to determine the extracted feature of the sample owned by itself by using the first part model of itself; The second determination module is configured to determine the first gradient slice for updating the first part model based on the extracted feature; the first gradient slice is not a complete gradient; The first interaction module is configured to, through data interaction between the first device and the second device based on multi-party secure computation, make the second device obtain aggregated features aggregated from the plurality of extracted features, and make the second device determine second gradient shards for updating the plurality of first partial models based on output data and the aggregated features, and send the second gradient shards to the first device; wherein the output data is determined based on the aggregated features and the second partial model; and the second gradient shards are incomplete gradients. The first receiving module is configured to receive the second gradient shards sent by the second device. The first updating module is configured to determine a first gradient for the first partial model of the first device based on a product of the second gradient shards and the first gradient shards, and update the first partial model using the first gradient, wherein the first gradient is a complete gradient.

15. A privacy-preserving multi-party joint training apparatus, wherein, The second device and the plurality of first devices respectively own a second partial model and a plurality of first partial models in a business prediction model, and the device is deployed in the second device and includes: The second interaction module is configured to, through data interaction between the first device and the second device based on multi-party secure computation, make the second device obtain aggregated features aggregated from the plurality of extracted features, and make the second device determine second gradient shards for updating the plurality of first partial models based on output data and the aggregated features, and send the second gradient shards to the first device; wherein the output data is determined based on the aggregated features and the second partial model; and the second gradient shards are incomplete gradients. The third determining module is configured to determine output data of the second partial model using the aggregated features. The fourth determining module is configured to determine second gradient shards for updating the plurality of first partial models based on the output data and the aggregated features; wherein the second gradient shards are incomplete gradients. The first sending module is configured to send the second gradient shards to the plurality of first devices, so that any one of the first devices updates the first partial model using a first gradient; wherein the first gradient is obtained based on a product of the second gradient shards and first gradient shards, the first gradient shards are determined based on respective extracted features and first partial models, the first gradient shards are incomplete gradients, and the first gradient is a complete gradient.

16. A computer readable storage medium having stored thereon a computer program which, when executed in a computer, causes the computer to perform the method of any one of claims 1-12.

17. A computing device comprising a memory and a processor, the memory having stored therein executable code which, when executed by the processor, implements the method of any one of claims 1-12.

Images