Training method and apparatus for federated model, and device and storage medium

Abstract

A training method and apparatus for a federated model, and a device and a storage medium. The method comprises: in the process of training a federated model, determining a first intermediate feature of the federated model in the current training epoch (S110); on the basis of the first intermediate feature, a data label, and a second feature acquired from a data provider, determining a reverse gradient of the federated model in the current training epoch (S120); on the basis of the reverse gradient and the first intermediate feature, determining a first update gradient, and using the first update gradient to update a first model parameter (S130); encrypting the reverse gradient to obtain target gradient ciphertext, and sending the target gradient ciphertext to the data provider (S140); and receiving second gradient update ciphertext sent by the data provider, decrypting same to obtain first gradient update plaintext, and sending the first gradient update plaintext to the data provider, such that the data provider updates the gradient of the federated model on the basis of the first gradient update plaintext (S150).

Description

Training method, device and equipment of federal model and storage medium

[0001] The present application claims priority to the Chinese patent application No. 202410962148.7, filed on July 18, 2024, to the Chinese Patent Office, the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the technical field of federated learning, for example, to a training method, device and equipment of federal model and storage medium. BACKGROUND

[0003] In the federal risk control model of a bank, the credit score model is a commonly used model, which is used to calculate the credit score of a user and calculate the credit level of the user according to the credit score (the bank can define the credit card limit according to the credit level). The bank generally only has the basic information of the user (such as age, gender and address, etc.) and the transaction information in the bank (such as monthly consumption, monthly deposit and half-year deposit, etc.). When the user is a new user, there will be inaccurate judgment due to lack of data, but federated learning can solve this problem. Federated learning can combine the data of at least two banks to train a federal model and obtain a federal credit score.

[0004] The data of a bank, especially a large bank, is often in the order of hundreds of millions of data, which cannot be trained by a single machine, so distributed model training is needed.

[0005] SUMMARY

[0006] The present application provides a training method, device and equipment of federal model and storage medium to realize distributed training of the federal model.

[0007] According to an aspect of the present application, a training method of a federal model is provided, which is applied to a first control node of a data application party, the data application party including at least one first computing node, one of the at least one first computing node being the first control node, and the method comprising:

[0008] In the training process of the federal model, the at least one first computing node is controlled to determine a first intermediate feature of the federal model in the current training round according to the first training data and the first model parameter of the current training round;

[0009] According to the first intermediate feature, the data label and the second feature obtained from the data provider, a reverse gradient of the federal model in the current training round is determined;

[0010] determine a first update gradient according to the reverse gradient and the first intermediate feature, and update the first model parameter by using the first update gradient;

[0011] encrypt the reverse gradient to obtain target gradient ciphertext, and send the target gradient ciphertext to the data provider, so that the data provider obtains second gradient update ciphertext according to the target gradient ciphertext and second training data;

[0012] receive and decrypt the second gradient update ciphertext sent by the data provider to obtain first gradient update plaintext, and send the first gradient update plaintext to the data provider, so that the data provider performs gradient update of the federated model according to the first gradient update plaintext.

[0013] According to another aspect of the present application, a federated model training method is provided, which is applied to a second control node of a data provider, the data provider including at least one second computing node, one of the at least one second computing node being the second control node, and the method comprising:

[0014] During the training of the federated model, the at least one second computing node is controlled to determine at least one second intermediate feature of the federated model at the current training round by using second training data and second model parameters of the current training round, and the at least one second intermediate feature is integrated to obtain a second feature;

[0015] receive target gradient ciphertext sent by the data application side, and split the target gradient ciphertext according to a second data set division manner to obtain second gradient ciphertext corresponding to each of the at least one second computing node;

[0016] control the at least one second computing node to obtain at least one second intermediate gradient update ciphertext according to the second gradient ciphertext corresponding to each of the at least one second computing node and the second training data;

[0017] integrate the at least one second intermediate gradient ciphertext to obtain second gradient update ciphertext, and send the second gradient update ciphertext to the data application side, so that the data application side decrypts the second gradient update ciphertext to obtain first gradient update plaintext;

[0018] receive the first gradient update plaintext sent by the data application side, and update the second model parameter according to the first gradient update plaintext.

[0019] According to another aspect of the present application, a training device of a federated model is provided, configured at a first control node of a data application party, the data application party comprising at least one first computing node, one of the at least one first computing node being the first control node, the device comprising:

[0020] a first intermediate feature determination module configured to control the at least one first computing node to determine a first intermediate feature of the federated model in a current training round based on first training data and first model parameters of the current training round in a training process of the federated model;

[0021] a reverse gradient determination module configured to determine a reverse gradient of the federated model in the current training round based on the first intermediate feature, data labels, and second features obtained from a data provider;

[0022] a first parameter update module configured to determine a first update gradient based on the reverse gradient and the first intermediate feature, and update the first model parameters using the first update gradient;

[0023] a second gradient update ciphertext determination module configured to encrypt the reverse gradient to obtain a target gradient ciphertext, and send the target gradient ciphertext to the data provider, so that the data provider obtains a second gradient update ciphertext based on the target gradient ciphertext and second training data;

[0024] a first gradient update plaintext determination module configured to receive and decrypt the second gradient update ciphertext sent by the data provider to obtain a first gradient update plaintext, and send the first gradient update plaintext to the data provider, so that the data provider performs gradient update of the federated model based on the first gradient update plaintext.

[0025] According to another aspect of the present application, a training device of a federated model is provided, configured at a second control node of a data provider, the data provider comprising at least one second computing node, one of the at least one second computing node being the second control node, the device comprising:

[0026] a second feature determination module configured to control the at least one second computing node to determine at least one second intermediate feature of the federated model in a current training round based on second training data and second model parameters of the current training round in a training process of the federated model, and integrate the at least one second intermediate feature to obtain a second feature;

[0027] The second gradient ciphertext determination module is configured to receive the target gradient ciphertext sent by the data application party, and split the target gradient ciphertext based on the second dataset partitioning method to obtain the second gradient ciphertext corresponding to each of the at least one second computing node.

[0028] The second intermediate gradient update ciphertext determination module is configured to control the at least one second computing node to obtain at least one second intermediate gradient update ciphertext based on its respective second gradient ciphertext and the second training data.

[0029] The first gradient update plaintext determination module is configured to integrate the at least one second intermediate gradient ciphertext to obtain a second gradient update ciphertext, and send the second gradient update ciphertext to the data application party so that the data application party can decrypt the second gradient update ciphertext to obtain the first gradient update plaintext.

[0030] The second parameter update module is configured to receive the first gradient update plaintext sent by the data application party, and update the second model parameters according to the first gradient update plaintext.

[0031] According to another aspect of this application, an electronic device is provided, the electronic device comprising:

[0032] At least one processor; and

[0033] A memory communicatively connected to the at least one processor; wherein,

[0034] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the training method of the federated model described in any embodiment of this application.

[0035] According to another aspect of this application, a computer-readable storage medium is provided that stores computer instructions for causing a processor to execute and implement the training method of the federated model described in any embodiment of this application.

[0036] According to another aspect of this application, a computer program product is provided, the computer program product comprising a computer program that, when executed by a processor, implements a training method for a federated model according to any embodiment of this application. Attached Figure Description

[0037] To illustrate the embodiments of this application, the accompanying drawings used in the description of the embodiments will be introduced below. The accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0038] Figure 1 is a flowchart of a training method for a federated model according to Embodiment 1 of this application;

[0039] Figure 2 is a flowchart of a training method for a federated model according to Embodiment 2 of this application;

[0040] Figure 3 is a schematic diagram of data interaction during the training process of a federated model according to Embodiment 3 of this application;

[0041] Figure 4 is a schematic diagram of the structure of a training device for a federated model according to Embodiment 4 of this application;

[0042] Figure 5 is a schematic diagram of the structure of a training device for a federated model according to Embodiment 5 of this application;

[0043] Figure 6 is a schematic diagram of the structure of an electronic device that implements the training method of the federated model in the embodiments of this application. Detailed Implementation

[0044] The embodiments of this application will now be described with reference to the accompanying drawings. These described embodiments are some related to this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort should fall within the scope of protection of this application.

[0045] The terms "first" and "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units listed, but may include other steps or units not listed or inherent to such processes, methods, products, or apparatus.

[0046] In the embodiments of this application, the collection, storage, use, processing, transmission, provision, and disclosure of training data and related gradient parameters of the federated model involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0047] Example 1

[0048] Figure 1 is a flowchart of a training method for a federated model according to Embodiment 1 of this application. This embodiment is applicable to situations where a federated model, such as a vertical federated model, is trained, for example, in a banking scenario where a federated risk control model, such as a vertical federated credit scoring model, is trained in a distributed manner. This method can be executed by a training device for the federated model, which can be implemented in hardware and / or software and integrated into an electronic device that carries the training function of the federated model, such as a first control node of the data application party, where the first control node can be a server; the data application party includes at least one first computing node, where the first computing node can also be a server. As shown in Figure 1, the method includes:

[0049] S110. During the training process of the federated model, the first computing node is controlled to determine the first intermediate feature of the federated model in the current training round based on the first training data and the first model parameters of the current training round.

[0050] In this embodiment, the federal model can be a bank's federal risk control model, such as a credit score model. In one embodiment, the federal model can be a vertical federal credit score model; for example, the equation of a vertical federal credit score model can be: The model parameters are iterated using gradient descent. For example, the gradient iteration method is as follows: θ k+1 =θ k -α(f(x)-y)x; where x represents the training data, y is the data label, θ is the model parameters, k represents the k-th training epoch, and θ k This represents the model parameters in the k-th training epoch, where α is the learning rate.

[0051] The data application party refers to the party that possesses data labels and initiates the vertical federated model training process. Optionally, the data application party includes a first control node and at least one first computation node; the first control node is the main process node (header) used to control the operation of the computation nodes of the data application party; the first computation node is the node used to perform data computation, controlled by the first control node. It should be noted that, in addition to controlling each first computation node and interacting with the data provider, the first control node also participates in the training of the federated model, that is, it participates in the computation of the first training data and the first model parameters, just like the first computation nodes; it can also be understood that the first control node is also a first computation node. For example, the data application party has four machines, such as servers, and randomly selects one machine as the first control node, and the remaining machines as first computation nodes.

[0052] The so-called first training data refers to the training data used for training the federated model at the first computing node. For example, it can be the basic information and transaction information of customers in a bank. The basic information includes the customer's age and gender, etc.; the transaction information includes bank transaction information, such as monthly consumption amount, monthly deposit amount and number of defaults, etc. It should be noted that the collection, storage, use, processing, transmission, provision and disclosure of basic information and transaction information all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0053] The first model parameters refer to the model parameters of the federated model in the current training epoch during the training process. The first intermediate features refer to the intermediate features generated during model training based on the first training data and the first model parameters, which can be represented in matrix or vector form.

[0054] For example, during the iterative training of the federated model, the first control node sends control commands to each first computing node. Correspondingly, each first computing node obtains a first intermediate feature based on its local first training data and the first model parameters of the current training epoch. Simultaneously, the first control node also obtains a first intermediate feature based on its local first training data and the first model parameters of the current training epoch. For example, the first model parameters are θ. A The first training data is X Ai Where i represents the machine of the i-th data application party (including the first control node and the first computing node); A represents the data application party; and the first intermediate feature is X. Ai θ A .

[0055] S120. Based on the first intermediate feature, the data label, and the second feature obtained from the data provider, determine the back gradient of the federated model in the current training round.

[0056] One optional approach involves receiving a second feature sent by the data provider, splitting the second feature based on the first dataset partitioning method to obtain a third intermediate feature corresponding to the first computing node; determining the target feature based on the first and third intermediate features; obtaining the target prediction result based on the target feature; and determining the backpropagation gradient of the federated model in the current training epoch based on the target prediction result and data labels. The second feature refers to the feature determined by the data provider based on its local training data and the parameters of the second model, and can be represented in matrix or vector form. The third intermediate feature refers to the features in the second feature corresponding to the first computing node and the first control node, respectively, and can be represented in matrix or vector form. The target feature refers to all features corresponding to each distributed node during the training of the federated model, i.e., each first computing node and each first control node, including the first and third intermediate features, and can be represented in matrix or vector form. The target prediction result refers to the result predicted by the federated model based on the target feature, such as the credit score probability value predicted by a credit scoring model. The data label refers to the ground truth value corresponding to the training data, such as the actual credit score. The backpropagation gradient refers to the difference between the target prediction result and the data label.

[0057] For example, the first control node receives the second feature X sent by the data provider. B θ B , where X B θ represents the training data owned by the data provider. B This indicates that the second model parameters are the model parameters of the federated model when the data provider trains the federated model. Then, the first control node splits the second feature based on the partitioning method of the first dataset, obtaining the third intermediate feature X corresponding to the first control node and each first computation node. Bi θ B , where X Bi This represents a feature in the second feature corresponding to the i-th machine of the data application party. Then, the first control node and each first computing node sum their respective first intermediate features and third intermediate features to obtain the target feature, M. i =X Ai θ a +X Bi θ B , of which M i This represents the target feature corresponding to the i-th machine in the data application. Next, the first control node and each first computing node, based on their respective target features, predict the corresponding target using a federated model. Among them, P iThis represents the target prediction result corresponding to the i-th machine in the data application. Finally, the difference between the target prediction result and the data label is used as the back gradient of the federated model in the current training round, i.e., G. i =P i -Y i Among them, G i Y represents the backpropagation gradient corresponding to the i-th machine in the data application; i This represents the data tag corresponding to the i-th machine of the data application party.

[0058] S130. Based on the back gradient and the first intermediate feature, determine the first update gradient and use the first update gradient to update the parameters of the first model.

[0059] In this embodiment, the first update gradient refers to the result obtained by multiplying the reverse gradient with the first intermediate feature.

[0060] For example, the first control node and each first computation node use the product of the backward gradient and the first intermediate feature as the first update gradient, i.e., G. i X Ai Then, each first computation node sends its corresponding first update gradient to the first control node; correspondingly, the first control node accumulates the collected first update gradients and the locally computed first update gradients to obtain the target update gradient GX. A Then, based on gradient iteration, the difference between the first model parameters of the current training epoch and the product of the target update gradient and the learning rate is used as the updated first model parameters, i.e., the first model parameters of the next training epoch, to update the first model parameters, i.e., θ. Ak+1 =θ Ak -αGX A Where k represents the k-th training round, θ Ak This represents the first model parameter in the k-th training round of the data application.

[0061] S140. Encrypt the reverse gradient to obtain the target gradient ciphertext, and send the target gradient ciphertext to the data provider so that the data provider can obtain the second gradient update ciphertext based on the target gradient ciphertext and the second training data.

[0062] In this embodiment, the second training data refers to the training data used for federated model training from the second computing node in the data provider. It should be noted that the second training data and the first training data can be of the same type or different types. The so-called second gradient update ciphertext refers to the data obtained after the data provider encrypts the target gradient ciphertext using the second training data.

[0063] An alternative approach involves controlling at least one first computing node to encrypt the reverse gradient using a public key to obtain at least one first gradient ciphertext; and integrating the at least one first gradient ciphertext to obtain the target gradient ciphertext.

[0064] The target gradient ciphertext refers to the encrypted data corresponding to the gradient descent update of the model parameters for the data application side. The first gradient ciphertext refers to the encrypted data corresponding to the backward gradient for each machine of the data application side.

[0065] For example, the first control node sends gradient encryption instructions to at least one first computing node; correspondingly, each first computing node encrypts its corresponding reverse gradient using its public key to obtain the first gradient ciphertext, denoted as [G]. i The first control node then sends the first gradient ciphertext to the first control node. It should be noted that the first control node also participates in the calculation of the first gradient ciphertext; that is, the first control node uses its public key to encrypt its corresponding reverse gradient, obtaining its own first gradient ciphertext. Next, the first control node merges the first gradient ciphertext sent by each first computing node with the first gradient ciphertext calculated locally to obtain the target gradient ciphertext, denoted as [G]. Finally, the first control node sends the target gradient ciphertext to the data provider, who can then obtain the second gradient update ciphertext based on the target gradient ciphertext and the second training data.

[0066] S150. Receive and decrypt the second gradient update ciphertext sent by the data provider to obtain the first gradient update plaintext, and send the first gradient update plaintext to the data provider so that the data provider can perform gradient updates of the federated model based on the first gradient update plaintext.

[0067] The first-gradient update plaintext refers to the data obtained by the data application party after decrypting the second-gradient update ciphertext.

[0068] An alternative approach involves receiving a second gradient update ciphertext sent by a data provider; splitting the second gradient update ciphertext based on a first dataset partitioning method to obtain a first intermediate gradient update ciphertext corresponding to a first computing node; controlling the first computing node to decrypt the first intermediate gradient update ciphertext using a private key to obtain a first intermediate gradient update plaintext; and integrating at least one first intermediate gradient update plaintext to obtain a first gradient update plaintext.

[0069] The first intermediate gradient update ciphertext refers to a portion of the second gradient update ciphertext corresponding to each machine of the data application party. The so-called first intermediate gradient update plaintext refers to the data obtained after decrypting the first intermediate gradient update ciphertext.

[0070] For example, the first control node receives the second gradient update ciphertext sent by the data provider, denoted as [GX]. B [+mask], then based on the first dataset partitioning method, the second gradient update ciphertext is split to obtain the first intermediate gradient update ciphertext corresponding to the first control node and each first computation node, denoted as [GX]. Bi +mask i The first intermediate gradient update ciphertext corresponding to each first computing node is sent to the corresponding first computing node. Correspondingly, each first computing node receives the corresponding first intermediate gradient update ciphertext, decrypts it using its private key, and obtains the first intermediate gradient update plaintext, denoted as GX. Bi +mask i Each node sends its corresponding first intermediate gradient update plaintext to the first control node. Simultaneously, the first control node decrypts its corresponding first intermediate gradient update ciphertext using its private key, obtaining its own first intermediate gradient update plaintext. Then, the first control node sums the first intermediate gradient update plaintext received from each first computing node with its locally computed first intermediate gradient update plaintext to obtain its own first gradient update plaintext, denoted as GX. B +mask; and send the first gradient update plaintext to the data provider so that the data provider can update the gradient of the federated model based on the first gradient update plaintext.

[0071] In this embodiment, during the training of the federated model, the first computing node is controlled to determine the first intermediate feature of the federated model in the current training epoch based on the first training data and the first model parameters of the current training epoch. Then, based on the first intermediate feature, data labels, and second features obtained from the data provider, the backpropagation gradient of the federated model in the current training epoch is determined. Subsequently, based on the backpropagation gradient and the first intermediate feature, the first update gradient is determined, and the first model parameters are updated using the first update gradient. Simultaneously, the backpropagation gradient is encrypted to obtain the target gradient ciphertext, which is sent to the data provider. The data provider then obtains the second gradient update ciphertext based on the target gradient ciphertext and the second training data. The data provider receives and decrypts the second gradient update ciphertext sent by the data provider to obtain the first gradient update plaintext, which is then sent to the data provider so that the data provider can update the gradient of the federated model based on the first gradient update plaintext. This embodiment of the application uses a distributed training method between the data application party and the data provider for the federated model. For large-scale training data, such as the hundreds of millions involved in banking, this can improve the training accuracy of the federated model.

[0072] As an optional approach in this application, during the training process of the federated model, before determining the first intermediate feature of the federated model in the current training round based on the first model parameters of the first training data, the method further includes: dividing the local training data of the data application party based on the first dataset partitioning method to obtain at least one first training data, and sending the at least one first training data to at least one first computing node respectively; generating a private key and a public key, sharing the public key and the private key with each of the at least one first computing node, and sending the public key to the data provider.

[0073] The local training data of the data application provider refers to the training data possessed by the data application provider. The public and private keys are used for encryption and decryption of the data during subsequent federated learning. The first dataset partitioning method refers to partitioning using a hash algorithm, such as hash(ID). A )mod m; where ID A The ID represents the local training data of the data application provider, m represents the number of machines owned by the data application provider, i.e., the number of first machines, and mod represents the modulo operator.

[0074] For example, the first control node divides the local training data of the data application party based on the first dataset partitioning method. For example, it may determine the number of first machines owned by the data application party, and then evenly divide the local training data into the number of parts of the first machine. Each part of the local training data is recorded as the first training data. Then, based on the intranet of the data application party, the first training data is sent to each first computing node. That is, the first control node keeps one part of the first training data locally on the first control node for distributed training of the federated model, and sends the other parts of the first training data to each first computing node.

[0075] Subsequently, the first control node generates a public key and a private key, and shares the public key and private key with each of at least one first computing node based on the data application party's intranet; at the same time, it sends the public key to the data provider's second control node.

[0076] By controlling the main process and distributing data to data users through the first control node, data transmission efficiency can be improved.

[0077] Example 2

[0078] Figure 2 is a flowchart of a training method for a federated model according to Embodiment 2 of this application. This embodiment is applicable to situations where a federated model, such as a vertical federated model, is trained, for example, in a banking scenario where a federated risk control model, such as a vertical federated credit scoring model, is trained in a distributed manner. This method can be executed by a training device for the federated model, which can be implemented in hardware and / or software and integrated into an electronic device that carries the training function of the federated model, such as a second control node of the data provider, where the second control node can be a server; the data provider also includes at least one second computing node, where the second computing node can also be a server. As shown in Figure 2, the method includes:

[0079] S210. During the training of the federated model, the second computing node is controlled to use the second training data and the second model parameters of the current training round to determine the second intermediate feature of the federated model in the current training round, and at least one second intermediate feature is integrated to obtain the second feature.

[0080] In this embodiment, the data provider refers to the party that provides the relevant data for training the federated model. Optionally, the data provider may include a second control node and at least one second computing node; wherein, the second control node is the main process node, i.e., the header, used to control the operation of the computing nodes of the data provider; the second computing node is the node used to perform data computation, controlled by the second control node. It should be noted that, in addition to controlling each second computing node and interacting with the data user, the second control node itself also participates in the training of the federated model, that is, it participates in the computation of the second training data and the second model parameters just like the second computing nodes. It can also be understood that the second control node is also a second computing node. For example, the data provider has 3 machines, such as servers, and randomly selects one machine as the second control node, and the remaining machines as second computing nodes.

[0081] The so-called second training data refers to the training data used for training the federated model on the second computing node. That is, the training data possessed by each machine of the data provider. For example, it could be the basic information and transaction information of customers in a bank. The basic information includes the customer's age and gender, etc.; the transaction information includes bank transaction information, such as the monthly consumption amount, the monthly deposit amount, and the number of defaults, etc. It should be noted that the collection, storage, use, processing, transmission, provision, and disclosure of the basic information and transaction information all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0082] The second model parameters refer to the model parameters in the current training epoch during the federated model training process at the data provider. The second intermediate features refer to the intermediate features generated during model training by each machine at the data provider based on its corresponding second training data and second model parameters; these can be represented in matrix or vector form. The second features refer to the intermediate features generated during model training based on the local training data and second model parameters of the data provider; these can also be represented in matrix or vector form.

[0083] For example, during the iterative training of the federated model, the second control node issues computation instructions to each second computation node. Correspondingly, each second computation node uses its local second training data and the second model parameters of the current training epoch to calculate the second intermediate feature of the federated model corresponding to each second computation node in the current training epoch, and sends its corresponding second intermediate feature to the second control node. Simultaneously, the second control node also calculates its second intermediate feature based on its local second training data and the second model parameters of the current training epoch. Then, the second control node accumulates the second intermediate features received from each second computation node and the second intermediate features calculated locally by the second control node to obtain the second feature. For example, the second model parameter is θ. B The second training data is X Bj Where j represents the machine of the j-th data provider (including the second control node and the second computing node); B represents the data provider; and the second intermediate feature is X. Bj θ B The second feature is X B θ B .

[0084] S220: Receive the target gradient ciphertext sent by the data application party, and split the target gradient ciphertext according to the second dataset partitioning method to obtain the second gradient ciphertext corresponding to the second computing node.

[0085] In this embodiment, the target gradient ciphertext refers to the encrypted data corresponding to the gradient update of the model parameters corresponding to the data application party during gradient descent. The second gradient ciphertext refers to the portion of the target gradient ciphertext corresponding to each machine of the data provider.

[0086] For example, the second control node receives the target gradient ciphertext [G] sent by the data application, and then splits the target gradient ciphertext according to the second dataset partitioning method to obtain the second gradient ciphertext corresponding to the second control node and each second computation node, denoted as [G]. j The second gradient ciphertext is then sent to the corresponding second computation node.

[0087] S230: Control the second computing node to obtain the second intermediate gradient update ciphertext based on the second gradient ciphertext and the second training data.

[0088] In one optional manner, the second computing node is controlled to obtain the candidate gradient update ciphertext based on the second gradient ciphertext and the second training data; the second computing node is controlled to generate the mask ciphertext, and obtain the second intermediate gradient update ciphertext based on the candidate gradient update ciphertext and the mask ciphertext.

[0089] The candidate gradient update ciphertext refers to the gradient ciphertext calculated based on the second gradient ciphertext and the second training data. The second intermediate gradient update ciphertext refers to the gradient ciphertext encrypted with the candidate gradient update ciphertext. The mask ciphertext refers to the data obtained by encrypting the mask, which is used to encrypt the candidate gradient update ciphertext.

[0090] For example, the second control node sends a gradient ciphertext update instruction to each second computation node; correspondingly, each second computation node multiplies its corresponding second gradient ciphertext with the second training data to obtain the corresponding candidate gradient update ciphertext, denoted as [G]. j ]X Bj And randomly generate a mask ciphertext with the same dimension as the candidate gradient update ciphertext, denoted as [mask]. j Then, the candidate gradient update ciphertext and the mask ciphertext are added together to obtain the corresponding second intermediate gradient update ciphertext, denoted as [G]. j X Bj +mask j Finally, the corresponding second intermediate gradient update ciphertexts are sent to the second control node. It should be noted that the second control node also performs the same data processing as the second computing node locally, that is, it obtains candidate gradient update ciphertexts based on the second gradient ciphertexts and the second training data, generates mask ciphertexts, and obtains the second intermediate gradient update ciphertexts based on the candidate gradient update ciphertexts and the mask ciphertexts.

[0091] Applying a mask to the candidate gradient update ciphertext can prevent data leakage between data users and data providers during data transmission.

[0092] S240. Integrate at least one second intermediate gradient ciphertext to obtain a second gradient update ciphertext, and send the second gradient update ciphertext to the data application party so that the data application party can decrypt the second gradient update ciphertext to obtain the first gradient update plaintext.

[0093] In this embodiment, the second gradient update ciphertext refers to the gradient ciphertext obtained by integrating the second intermediate gradient update ciphertext, that is, the data obtained by the data provider encrypting the target gradient ciphertext using the second training data. The first gradient update plaintext refers to the data obtained by the data user after decrypting the second gradient update ciphertext.

[0094] For example, the second control node accumulates the second intermediate gradient update ciphertext received from each second computing node and the second intermediate gradient update ciphertext computed locally by the second control node to obtain the second gradient update ciphertext [GX]. B +mask]; and then send the second gradient update ciphertext to the data application party. Correspondingly, the data application party decrypts the second gradient update ciphertext to obtain the first gradient update plaintext GX. B +mask.

[0095] S250: Receive the first gradient update plaintext sent by the data application party, and update the second model parameters according to the first gradient update plaintext.

[0096] An alternative approach involves splitting the plaintext of the first gradient update based on the partitioning method of the second dataset to obtain the plaintext of the second intermediate gradient update corresponding to the second computing node; controlling the second computing node to update the plaintext of the second intermediate gradient and the mask plaintext to obtain the second intermediate update gradient; integrating at least one second intermediate update gradient to obtain the second update gradient, and using the second update gradient to update the parameters of the second model.

[0097] The second intermediate gradient update plaintext refers to a portion of the second gradient plaintext corresponding to each machine of the data provider. The second intermediate gradient update is the data obtained after removing the mask from the second intermediate gradient update plaintext. The second update gradient is the result obtained by multiplying the backpropagation gradient by the second intermediate feature.

[0098] For example, the second control node receives the first gradient update plaintext sent by the data application, and then splits the first gradient update plaintext according to the second dataset partitioning method to obtain the second intermediate gradient update plaintext corresponding to the second control node and each second computing node, denoted as G. j X Bj +mask j The second control node sends the plaintext update of the second intermediate gradient to the corresponding second computation node. Then, the second control node sends the gradient plaintext update operation to each second computation node. Correspondingly, each second computation node removes the masked plaintext from its corresponding second intermediate gradient update plaintext to obtain the second intermediate update gradient, denoted as G. j X BjThe second intermediate update gradient is then sent to the second control node. Simultaneously, the second control node removes the masked plaintext from its local second intermediate gradient update plaintext to obtain its corresponding second intermediate update gradient. Furthermore, the second control node accumulates the second intermediate update gradients sent by each second computing node and the second intermediate update gradients calculated locally by the second control node to obtain the second update gradient, denoted as GX. B Finally, based on the gradient iteration method, the difference between the second model parameters corresponding to the current training epoch and the product of the second update gradient and the learning rate is used as the updated second model parameters, i.e., the second model parameters for the next training epoch, to achieve the update of the second model parameters, i.e., θ. Bk+1 =θ Bk -αGX B Where k represents the k-th training round, θ Bk This represents the second model parameter of the data provider in the k-th training round.

[0099] This embodiment of the application, during the training process of the federated model, controls the second computing node to use the second training data and the second model parameters of the current training epoch to determine the second intermediate feature of the federated model in the current training epoch. It then integrates at least one second intermediate feature to obtain the second feature. Next, it receives the target gradient ciphertext sent by the data application party and splits it according to the second dataset partitioning method to obtain the second gradient ciphertext corresponding to the second computing node. The second computing node then obtains the second intermediate gradient update ciphertext based on the second gradient ciphertext and the second training data. It further integrates at least one second intermediate gradient ciphertext to obtain the second gradient update ciphertext and sends it to the data application party, allowing the data application party to decrypt it to obtain the first gradient update plaintext. Finally, it receives the first gradient update plaintext sent by the data application party and updates the second model parameters accordingly. This embodiment of the application, through distributed training of the federated model by the data application party and the data provider, can improve the training accuracy of the federated model for large-scale training data, such as the hundreds of millions involved in banking.

[0100] As an optional approach in this application, during the training process of the federated model, before controlling the second computing node to use the second training data and the second model parameters of the current training round to determine the second intermediate feature of the federated model in the current training round, and before integrating at least one second intermediate feature to obtain the second feature, the method further includes: dividing the local training data of the data provider based on the second dataset partitioning method to obtain at least one second training data, and sending the at least one second training data to at least one second computing node respectively; receiving the public key sent by the data application party, and sharing the public key with each of the at least one second computing node.

[0101] The local training data provided by the data provider refers to the training data owned by the data provider. The second data partitioning method involves partitioning the data using a hash algorithm, such as hash(ID). B )mod n; where ID B The ID represents the local training data of the data provider, n represents the number of machines owned by the data provider, i.e., the number of second machines, and mod represents the modulo operator.

[0102] For example, the second control node divides the local training data of the data provider based on the second dataset partitioning method. For example, it may determine the number of second machines owned by the data provider, and then divide the local training data into the number of parts of the second machine. Each part of the local training data is recorded as the second training data. Then, based on the data provider's intranet, the second training data is sent to each second computing node. That is, the second control node itself keeps one part of the second training data locally for distributed training of the federated model, and sends the other parts of the second training data to each second computing node.

[0103] Furthermore, the second control node can receive the public key sent by the data application party and share the public key with each second computing node based on the data provider's intranet.

[0104] Using a second control node to control the main process and distribute data to the data provider can improve data transmission efficiency.

[0105] Example 3

[0106] Figure 3 is a schematic diagram of data interaction during the training process of a federated model according to Embodiment 3 of this application. This embodiment, based on the above embodiments, illustrates the training process of the federated model, i.e., the gradient update process, in cooperation between the data user and the data provider. Optionally, the data user includes its local training data X. A Data label Y, first model parameter θ AThe data application (guest) includes a first control node (header1, m0) and at least one first computing node, such as m1, m2, and m3. The data provider (host) includes its local training data X. B The second model parameter θ B The data provider includes a second control node (header2 n0) and at least one second computing node, such as n1 and n2. The first control node of the data user and the second control node of the data provider interact with each other via the external network. The first control node and at least one first computing node interact with each other via the data user's internal network. The second control node and at least one second computing node interact with each other via the data provider's internal network, as shown in Figure 3.

[0107] For example, the training process of a federated model can be:

[0108] First, the data preparation stage:

[0109] The first control node (header1 m0) of the data application (guest) partitions the local training data based on the first dataset partitioning method, obtaining at least one first training data point, and sends the first training data point to the first computing node; simultaneously, the first control node sends the first model parameters θ A Share with each first computing node. Generate a private key and a public key, share the public key en and the private key de with at least one first computing node respectively, and send the public key en to the second control node header2 n0 of the data provider.

[0110] The second control node (header2 n0) of the data provider host partitions the local training data based on the second dataset partitioning method, obtaining at least one second training data set, and sends the second training data set to the second computing node; simultaneously, the second control node sends the second model parameters θ B Share it with each second computing node. Receive the public key 'en' sent by the data application party and share it with at least one second computing node.

[0111] Then, the federated model training phase, i.e. the model parameter update process:

[0112] First, the determination of the first intermediate feature and the second intermediate feature.

[0113] For example, each of the first computing node and the first control node of the data application party calculates the first intermediate feature X based on the first training data and the first model parameters of the current training round. Ai θ AEach of the data provider's second computing nodes and second control nodes calculates the second intermediate feature X based on the second training data and the second model parameters of the current training round. Bj θ B .

[0114] Second, the aggregation and decomposition of the second feature by data providers and data users.

[0115] For example, the second computing nodes of the data provider send their respective second intermediate features to the second control node; correspondingly, the second control node accumulates the second intermediate features received from each second computing node and the second intermediate features calculated locally by the second control node to obtain the second feature X. B θ B and the second feature X B θ B The first control node sent to the data application user (guest).

[0116] After receiving the second feature, the first control node splits the second feature to obtain the third intermediate feature X corresponding to the first control node and each first computing node. Bi θ B Simultaneously, the first control node sends the third intermediate feature to the corresponding first computing node.

[0117] Third, the data application party obtains the target gradient ciphertext and sends it to the data provider.

[0118] For example, the first control node and each first computing node of the data application guest sum their respective first intermediate features and third intermediate features to obtain the target feature, M. i =X Ai θ A +X Bi θ B Based on the corresponding target features, the corresponding target prediction result is obtained using a federated model, i.e. The difference between the target prediction result and the data label is used as the back gradient of the federated model in the current training epoch, i.e., G. i =P i -Y i The reverse gradient corresponding to the first computing node is encrypted using the public key to obtain the first gradient ciphertext, denoted as [G]. i ].

[0119] Subsequently, each first computing node sends the first gradient ciphertext to the first control node; correspondingly, the first control node merges the first gradient ciphertext sent by each first computing node with the first gradient ciphertext computed locally by the first control node to obtain the target gradient ciphertext, denoted as [G]. Then, the first control node sends the target gradient ciphertext [G] to the second control node of the data provider.

[0120] Fourth, the data provider obtains the second-gradient update ciphertext and sends it to the data application provider.

[0121] For example, the second control node header2 n0 receives the target gradient ciphertext [G] sent by the data application. Then, based on the second dataset partitioning method, it splits the target gradient ciphertext to obtain the second gradient ciphertext corresponding to the second control node and each second computation node, denoted as [G]. j The second gradient ciphertext is then sent to the corresponding second computation node.

[0122] Then, each second computation node multiplies its corresponding second gradient ciphertext with the second training data to obtain the corresponding candidate gradient update ciphertext, denoted as [G]. j ]X Bj And randomly generate a mask ciphertext with the same dimension as the candidate gradient update ciphertext, denoted as [mask]. j Then, the candidate gradient update ciphertext and the mask ciphertext are added together to obtain the corresponding second intermediate gradient update ciphertext, denoted as [G]. j X Bj +mask j Finally, the corresponding second intermediate gradient update ciphertext is sent to the second control node.

[0123] Then, the second control node accumulates the second intermediate gradient update ciphertext received from each second computing node and the second intermediate gradient update ciphertext computed locally by the second control node to obtain the second gradient update ciphertext [GX]. B +mask).

[0124] Fifth, the data application party determines the first gradient update plaintext and sends it to the data provider, as well as the update of the first model parameters.

[0125] For example, the first control node header1 m0 updates the ciphertext [GX] to the second gradient based on the first dataset partitioning method. B The [+mask] is split to obtain the first intermediate gradient update ciphertext corresponding to the first control node and each first computation node, denoted as [GX]. Bi +mask iThe first intermediate gradient update ciphertext corresponding to each first computing node is sent to the corresponding first computing node. Correspondingly, each first computing node receives the corresponding first intermediate gradient update ciphertext, decrypts it using its private key, and obtains the first intermediate gradient update plaintext, denoted as GX. Bi +mask i Simultaneously, the first control node also uses its private key to decrypt the local first intermediate gradient update ciphertext, obtaining the first intermediate gradient update plaintext corresponding to the first control node.

[0126] Each first computing node sends its corresponding first intermediate gradient update plaintext to the first control node; the first control node accumulates the first intermediate gradient update plaintext received from each first computing node and the first intermediate gradient update plaintext calculated locally to obtain the first gradient update plaintext, denoted as GX. B +mask; and send the first gradient update plaintext to the second control node of the data provider.

[0127] Furthermore, the first control node and each first computation node use the product of the backward gradient and the first intermediate feature as the first update gradient, i.e., G. i X Ai Then, each first computation node sends its corresponding first update gradient to the first control node; correspondingly, the first control node accumulates the collected first update gradients and the locally computed first update gradients to obtain the target update gradient GX. A Then, based on the gradient iteration method, the difference between the first model parameter of the current training round and the product of the target update gradient and the learning rate is used as the updated first model parameter, which is the first model parameter of the next training round. The updated first model parameter is then analyzed and distributed to each first computing node to realize the update of the first model parameter.

[0128] Sixth, the processing of plaintext for the first gradient update, and the updating of the second model parameters.

[0129] The second control node, based on the partitioning method of the second dataset, splits the first gradient update plaintext to obtain the second intermediate gradient update plaintext corresponding to the second control node and each second computation node, denoted as G. j X Bj +mask j The second intermediate gradient update plaintext is sent to the corresponding second computation node.

[0130] Then, each second computation node removes the masked plaintext from its corresponding second intermediate gradient update plaintext to obtain the second intermediate update gradient, denoted as G. j X BjThe second intermediate update gradient is then sent to the second control node. Simultaneously, the second control node removes the masked plaintext from its local second intermediate gradient update plaintext to obtain the second intermediate update gradient corresponding to the second control node.

[0131] Then, the second control node accumulates the second intermediate update gradient sent by each second computing node and the second intermediate update gradient calculated locally by the second control node to obtain the second update gradient, denoted as GX. B Finally, based on the gradient iteration method, the difference between the second model parameter corresponding to the current training round and the product of the second update gradient and the learning rate is used as the updated second model parameter, i.e., the second model parameter of the next training round. The updated second model parameter is then shared with each second computing node to update the second model parameter.

[0132] In this application, encryption and decryption are performed on each computing node of both the data application party and the data provider. Data distribution and transmission are handled by the first control node of the data application party, which processes the data and distributes it to each first computing node, and the second control node of the data provider, which processes the data and distributes it to each second computing node. This allows for rapid data transmission within the intranet. Since the data encryption and decryption process is cumbersome, the ciphertext or plaintext is first split and then processed by its respective computing node, thereby improving model training efficiency.

[0133] In some embodiments, the initialization of the first model parameters and the second model parameters is randomly generated by the corresponding first control node and the second control node.

[0134] Example 4

[0135] Figure 4 is a schematic diagram of a training device for a federated model according to Embodiment 4 of this application. This embodiment is applicable to various scenarios involving training federated models, such as vertical federated models, for example, in a banking scenario where a federated risk control model, such as a vertical federated credit score model, is trained in a distributed manner. This method can be executed by a training device for the federated model, which can be implemented in hardware and / or software and integrated into an electronic device that carries the training function of the federated model, such as a first control node of the data application party, where the first control node can be a server; the data application party includes at least one first computing node, where the first computing node can also be a server; one of the at least one first computing node is the first control node. As shown in Figure 4, the device includes:

[0136] The first intermediate feature determination module 410 is configured to control at least one first computing node to determine the first intermediate feature of the federated model in the current training round based on the first training data and the first model parameters of the current training round during the training process of the federated model.

[0137] The back gradient determination module 420 is configured to determine the back gradient of the federated model in the current training round based on the first intermediate feature, the data label, and the second feature obtained from the data provider.

[0138] The first parameter update module 430 is configured to determine the first update gradient based on the back gradient and the first intermediate feature, and use the first update gradient to update the parameters of the first model.

[0139] The second gradient update ciphertext determination module 440 is configured to encrypt the reverse gradient to obtain the target gradient ciphertext, and send the target gradient ciphertext to the data provider so that the data provider can obtain the second gradient update ciphertext based on the target gradient ciphertext and the second training data.

[0140] The first gradient update plaintext determination module 450 is configured to receive and decrypt the second gradient update ciphertext sent by the data provider to obtain the first gradient update plaintext, and send the first gradient update plaintext to the data provider so that the data provider can perform gradient updates of the federated model based on the first gradient update plaintext.

[0141] This application embodiment, during the training of a federated model, controls at least one first computing node to determine the first intermediate feature of the federated model in the current training epoch based on the first training data and the first model parameters of the current training epoch. Then, based on the first intermediate feature, data labels, and second features obtained from the data provider, the backpropagation gradient of the federated model in the current training epoch is determined. Subsequently, based on the backpropagation gradient and the first intermediate feature, a first update gradient is determined, and the first model parameters are updated using the first update gradient. Simultaneously, the backpropagation gradient is encrypted to obtain the target gradient ciphertext, which is sent to the data provider. The data provider then obtains the second gradient update ciphertext based on the target gradient ciphertext and the second training data. The data provider receives and decrypts the second gradient update ciphertext sent by the data provider to obtain the first gradient update plaintext, which is then sent to the data provider so that the data provider can update the gradient of the federated model based on the first gradient update plaintext. This application embodiment, through distributed training of the federated model by the data application party and the data provider, can improve the training accuracy of the federated model for large-scale training data, such as the hundreds of millions involved in banking.

[0142] Optionally, the device also includes a data distribution module, which is configured as follows:

[0143] During the training of the federated model, based on the parameters of the first model in the current training round of the first training data, before determining the first intermediate feature of the federated model in the current training round, the local training data of the data application party is divided according to the first dataset partitioning method to obtain at least one first training data, and at least one first training data is sent to at least one first computing node respectively.

[0144] Generate a private key and a public key, share the public key and private key with each of at least one of the first computing nodes, and send the public key to the data provider.

[0145] Optionally, the back gradient determination module 420 is configured as follows:

[0146] The second feature sent by the data provider is split based on the first dataset partitioning method to obtain at least one third intermediate feature corresponding to each of the first computing nodes;

[0147] The target features are determined based on the first and third intermediate features;

[0148] The target prediction result is obtained based on the target characteristics;

[0149] Based on the target prediction results and data labels, determine the back gradient of the federated model in the current training round.

[0150] Optionally, the second gradient update ciphertext determination module 440 is configured as follows:

[0151] Control each of at least one first computing node to encrypt its corresponding reverse gradient using a public key, thereby obtaining at least one first gradient ciphertext;

[0152] Integrate at least one first-gradient ciphertext to obtain the target gradient ciphertext.

[0153] Optionally, the first gradient update plaintext determination module 450 is set to:

[0154] Receive the second-gradient update ciphertext sent by the data provider;

[0155] Based on the first dataset partitioning method, the second gradient update ciphertext is split to obtain at least one first intermediate gradient update ciphertext corresponding to each of the first computing nodes;

[0156] Control at least one first computing node to use its private key to decrypt its corresponding first intermediate gradient update ciphertext, thereby obtaining at least one first intermediate gradient update plaintext;

[0157] Integrate at least one first intermediate gradient update plaintext to obtain the first gradient update plaintext.

[0158] The training apparatus for the federated model provided in this application embodiment can execute the training method for the federated model provided in any embodiment of this application, and has the corresponding functional modules and beneficial effects of the execution method.

[0159] Example 5

[0160] Figure 5 is a schematic diagram of a training device for a federated model according to Embodiment 5 of this application. This embodiment is applicable to various situations involving the training of federated models, especially vertical federated models, such as distributed training of federated risk control models, like vertical federated credit scoring models, in a banking scenario. This method can be executed by a training device for the federated model, which can be implemented in hardware and / or software and integrated into an electronic device that carries the training function of the federated model, such as a second control node of a data provider, where the second control node can be a server; the data provider includes at least one second computing node, where the second computing node can also be a server; one of the at least two second computing nodes is a second control node. As shown in Figure 5, the device includes:

[0161] The second feature determination module 510 is configured to, during the training process of the federated model, control at least one second computing node to use the second training data and the second model parameters of the current training round to determine at least one second intermediate feature of the federated model in the current training round, and integrate the at least one second intermediate feature to obtain the second feature.

[0162] The second gradient ciphertext determination module 520 is configured to receive the target gradient ciphertext sent by the data application party, and split the target gradient ciphertext based on the second dataset partitioning method to obtain at least one second gradient ciphertext corresponding to each of the second computing nodes.

[0163] The second intermediate gradient update ciphertext determination module 530 is configured to control at least one second computing node to obtain at least one second intermediate gradient update ciphertext based on its corresponding second gradient ciphertext and second training data.

[0164] The first gradient update plaintext determination module 540 is configured to integrate at least one second intermediate gradient ciphertext to obtain a second gradient update ciphertext, and send the second gradient update ciphertext to the data application party so that the data application party can decrypt the second gradient update ciphertext to obtain the first gradient update plaintext.

[0165] The second parameter update module 550 is configured to receive the first gradient update plaintext sent by the data application party and update the second model parameters according to the first gradient update plaintext.

[0166] This embodiment of the application, during the training process of the federated model, controls at least one second computing node to use second training data and second model parameters of the current training epoch to determine at least one second intermediate feature of the federated model in the current training epoch, and integrates the at least one second intermediate feature to obtain a second feature. Then, it receives the target gradient ciphertext sent by the data application party, and splits the target gradient ciphertext according to the second dataset partitioning method to obtain the second gradient ciphertext corresponding to each of the at least one second computing node. It then controls at least one second computing node to obtain at least one second intermediate gradient update ciphertext based on its corresponding second gradient ciphertext and the second training data, and integrates the at least one second intermediate gradient ciphertext to obtain the second gradient update ciphertext. The second gradient update ciphertext is then sent to the data application party, allowing the data application party to decrypt the second gradient update ciphertext to obtain the first gradient update plaintext. Finally, it receives the first gradient update plaintext sent by the data application party and updates the second model parameters according to the first gradient update plaintext. This embodiment of the application, through distributed training of the federated model by the data application party and the data provider, can improve the training accuracy of the federated model for large-scale training data, such as the hundreds of millions involved in banking.

[0167] Optionally, the device also includes a data distribution module, which is configured as follows:

[0168] During the training of the federated model, at least one second computing node is controlled to use the second training data and the second model parameters of the current training round to determine at least one second intermediate feature of the federated model in the current training round. Before integrating the at least one second intermediate feature to obtain the second feature, the local training data of the data provider is divided according to the second dataset partitioning method to obtain at least one second training data, and the at least one second training data is sent to at least one second computing node respectively.

[0169] Receive the public key sent by the data application party and share the public key with each of at least one second computing node.

[0170] Optionally, the second intermediate gradient update ciphertext determination module 530 is configured as follows:

[0171] Control at least one second computing node to obtain candidate gradient update ciphertext based on its corresponding second gradient ciphertext and second training data;

[0172] Control at least one second computing node to generate its corresponding mask ciphertext, and update the ciphertext and mask ciphertext according to their respective candidate gradients to obtain at least one second intermediate gradient update ciphertext.

[0173] Optionally, the second parameter update module 550 is set to:

[0174] Based on the second dataset partitioning method, the first gradient update plaintext is split to obtain at least one second intermediate gradient update plaintext corresponding to each second computing node;

[0175] Control at least one second computing node to update the plaintext and mask plaintext according to their respective second intermediate gradients to obtain at least one second intermediate update gradient;

[0176] The second update gradient is obtained by integrating at least one second intermediate update gradient, and the second update gradient is used to update the second model parameters.

[0177] The training apparatus for the federated model provided in this application embodiment can execute the training method for the federated model provided in any embodiment of this application, and has the corresponding functional modules and beneficial effects of the execution method.

[0178] Example 6

[0179] Figure 6 is a schematic diagram of the structure of an electronic device implementing the training method of the federated model according to an embodiment of this application; Figure 6 shows a schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of this application. The electronic device can represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are illustrative.

[0180] As shown in Figure 6, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0181] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0182] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as the training methods of federated models.

[0183] In some embodiments, the training method for the federated model may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the training method for the federated model described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to execute the training method for the federated model by any other suitable means (e.g., by means of firmware).

[0184] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard parts (ASSPs), systems-on-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0185] Computer programs used to implement the methods of this application may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0186] In the context of this application, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. Examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, RAM, ROM, erasable programmable read-only memory (EPROM), flash memory, optical fiber, compact disc-read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0187] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a cathode ray tube (CRT), liquid crystal display (LCD), or monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0188] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0189] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system. It addresses the shortcomings of traditional physical hosts and Virtual Private Server (VPS) services, such as high management difficulty and weak business scalability.

[0190] The various processes shown above can be used to reorder, add, or delete steps. For example, the multiple steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of this application can be achieved.

Claims

1. A method for training a federated model, applied to a first control node of a data application party, the data application party comprising at least one first computing node, one of the at least one first computing node being the first control node, the method comprising: controlling the at least one first computing node to determine a first intermediate feature of the federated model in a current training round based on first training data and first model parameters of the current training round in a training process of the federated model; determining a back propagation gradient of the federated model in the current training round based on the first intermediate feature, data labels, and second features obtained from a data provider; determining a first update gradient based on the back propagation gradient and the first intermediate feature, and updating the first model parameters using the first update gradient; encrypting the back propagation gradient to obtain target gradient ciphertext, and sending the target gradient ciphertext to the data provider, so that the data provider obtains second gradient update ciphertext based on the target gradient ciphertext and second training data; receiving and decrypting the second gradient update ciphertext sent by the data provider to obtain first gradient update plaintext, and sending the first gradient update plaintext to the data provider, so that the data provider performs gradient update of the federated model based on the first gradient update plaintext. 2.The method of claim 1, wherein, before controlling the at least one first computing node to determine the first intermediate feature of the federated model in the current training round based on the first training data and the first model parameters of the current training round in the training process of the federated model, the method further comprises: dividing local training data of the data application party based on a first data set division manner to obtain at least one first training data, and sending the at least one first training data to the at least one first computing node respectively; generating a private key and a public key, sharing the public key and the private key to each of the at least one first computing node, and sending the public key to the data provider. The determining of the back propagation gradient of the federated model in the current training round based on the first intermediate feature, the data labels, and the second features obtained from the data provider comprises: receiving the second features sent by the data provider, splitting the second features based on the first data set division manner to obtain third intermediate features corresponding to the at least one first computing node respectively; determining target features based on the first intermediate features and the third intermediate features; obtaining target prediction results based on the target features; and determining the back propagation gradient of the federated model in the current training round based on the target prediction results and the data labels. The encrypting of the back propagation gradient to obtain the target gradient ciphertext comprises: controlling each of the at least one first computing node to encrypt the back propagation gradient corresponding thereto using the public key to obtain at least one first gradient ciphertext; and integrating the at least one first gradient ciphertext to obtain the target gradient ciphertext. ​ ​ ​ ​ ​ 3. The method of claim 1, wherein, ​ ​ ​ ​ ​ 4. The method of claim 2, wherein, ​ ​ ​ 5. The method of claim 2, wherein, The receiving and decrypting the second gradient update ciphertext sent by the data provider to obtain the first gradient update plaintext comprises: Receiving the second gradient update ciphertext sent by the data provider; Based on the first data set division mode, the second gradient update ciphertext is split to obtain the first intermediate gradient update ciphertext corresponding to each of the at least one first computing node; Control the at least one first computing node to use the private key to decrypt the first intermediate gradient update ciphertext corresponding to each of them to obtain at least one first intermediate gradient update plaintext; Integrate the at least one first intermediate gradient update plaintext to obtain the first gradient update plaintext.

6. A training method of a federated model, applied to a second control node of a data provider, the data provider comprising at least one second computing node, one node in the at least one second computing node being the second control node, the method comprising: During the training process of the federated model, controlling the at least one second computing node to determine at least one second intermediate feature of the federated model at the current training round using the second training data and the second model parameter of the current training round, and integrating the at least one second intermediate feature to obtain a second feature; Receiving the target gradient ciphertext sent by the data application party, and based on the second data set division mode, the target gradient ciphertext is split to obtain the second gradient ciphertext corresponding to each of the at least one second computing node; Control the at least one second computing node to obtain at least one second intermediate gradient update ciphertext according to the second gradient ciphertext corresponding to each of them and the second training data; Integrate the at least one second intermediate gradient ciphertext to obtain the second gradient update ciphertext, and send the second gradient update ciphertext to the data application party, so that the data application party decrypts the second gradient update ciphertext to obtain the first gradient update plaintext; Receiving the first gradient update plaintext sent by the data application party, and updating the second model parameter according to the first gradient update plaintext.

7. The method of claim 6, wherein, during the training process of the federated model, before controlling the at least one second computing node to determine at least one second intermediate feature of the federated model at the current training round using the second training data and the second model parameter of the current training round, and integrating the at least one second intermediate feature to obtain a second feature, the method further comprises: Based on the second data set division mode, the local training data of the data provider is divided to obtain at least one second training data, and the at least one second training data is sent to the at least one second computing node respectively; Receiving the public key sent by the data application party, and sharing the public key with each of the at least one second computing node.

8. The method of claim 6, wherein, The control the at least one second computing node to obtain at least one second intermediate gradient update ciphertext according to the second gradient ciphertext corresponding to each of them and the second training data, comprises: control the at least one second computing node to obtain a candidate gradient update ciphertext according to the respective corresponding second gradient ciphertext and the second training data; control the at least one second computing node to generate a respective corresponding mask ciphertext, and obtain the at least one second intermediate gradient update ciphertext according to the respective corresponding candidate gradient update ciphertext and the mask ciphertext.

9. The method of claim 6, wherein, The updating of the second model parameter according to the first gradient update plaintext comprises: splitting the first gradient update plaintext based on a second data set division manner to obtain a second intermediate gradient update plaintext corresponding to each of the at least one second computing node; controlling the at least one second computing node to obtain at least one second intermediate update gradient according to the respective corresponding second intermediate gradient update plaintext and a mask plaintext; integrating the at least one second intermediate update gradient to obtain a second update gradient, and updating the second model parameter by using the second update gradient.

10. A training device of a federated model, configured at a first control node of a data application party, the data application party comprising at least one first computing node, one of the at least one first computing node being the first control node, the device comprising: a first intermediate feature determination module configured to control the at least one first computing node to determine a first intermediate feature of the federated model in a current training round according to first training data and a first model parameter in the current training round in a training process of the federated model; a reverse gradient determination module configured to determine a reverse gradient of the federated model in the current training round according to the first intermediate feature, a data label, and a second feature obtained from a data provider; a first parameter update module configured to determine a first update gradient according to the reverse gradient and the first intermediate feature, and update the first model parameter by using the first update gradient; a second gradient update ciphertext determination module configured to encrypt the reverse gradient to obtain a target gradient ciphertext, and send the target gradient ciphertext to the data provider, so that the data provider obtains a second gradient update ciphertext according to the target gradient ciphertext and second training data; a first gradient update plaintext determination module configured to receive and decrypt the second gradient update ciphertext sent by the data provider to obtain a first gradient update plaintext, and send the first gradient update plaintext to the data provider, so that the data provider performs gradient update of the federated model according to the first gradient update plaintext.

11. A training device of a federated model, configured at a second control node of a data provider, the data provider further comprising at least one second computing node, one of the at least one second computing node being the second control node, the device comprising: The second feature determination module is configured to control the at least one second computing node to determine at least one second intermediate feature of the federated model in the current training round by using the second training data and the second model parameters of the current training round, and to integrate the at least one second intermediate feature to obtain a second feature. The second gradient ciphertext determination module is configured to receive target gradient ciphertext sent by the data application party, and split the target gradient ciphertext to obtain second gradient ciphertext corresponding to each of the at least one second computing node based on a second data set division manner. The second intermediate gradient update ciphertext determination module is configured to control the at least one second computing node to obtain at least one second intermediate gradient update ciphertext based on the second gradient ciphertext corresponding to each of the at least one second computing node and the second training data corresponding to each of the at least one second computing node. The first gradient update plaintext determination module is configured to integrate the at least one second intermediate gradient ciphertext to obtain second gradient update ciphertext, and send the second gradient update ciphertext to the data application party, so that the data application party decrypts the second gradient update ciphertext to obtain first gradient update plaintext. The second parameter update module is configured to receive the first gradient update plaintext sent by the data application party, and update the second model parameters based on the first gradient update plaintext.

12. An electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the federated model training method of any one of claims 1-5 or claims 6-9.

13. A computer readable storage medium, the computer readable storage medium stores computer instructions for enabling a processor to implement the federated model training method of any one of claims 1-5 or claims 6-9 when executed by the processor.

14. A computer program product, the computer program product comprises a computer program, the computer program implements the federated model training method according to any one of claims 1-5 or claims 6-9 when executed by a processor.

Images