Data processing method and device, federal learning system and electronic equipment

Abstract

The embodiment of the invention discloses a data processing method and device, a federal learning system and electronic equipment. A specific embodiment of the data processing method comprises the steps of determining a prediction result of a model according to inner product data in response to the received inner product data sent by a follow-up participant in a federated learning system; determining a data residual error according to the prediction result and the corresponding label data; performing encryption processing on the data residual, and sending encrypted residual data to participants except the main participant in the federated learning system; wherein the federal learning system is a system in which at least three parties participate. The implementation mode is related to a model training technology, and the follow-up participant can directly send the inner product data to the main participant. Therefore, the training efficiency of the combined learning model can be improved. And the main participant sends the encrypted residual data to other participants, so that the data security can be ensured while the weight update of each participant is realized.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of model training technology, and specifically to a data processing method, device, federated learning system, and electronic device. Background Art

[0002] The training of federated logistic regression is generally divided into two parts: forward propagation and back propagation. The prediction result can be obtained through the calculation of forward propagation. In back propagation, the gradient can be obtained by calculating the difference between the prediction result and the label value. Then, the weight can be updated using the idea of ​​gradient descent. In actual scenarios, each data party usually has different characteristics of samples. The purpose of federated learning is to jointly complete model training with various data parties to obtain better model effects. However, the data security of all parties must be guaranteed during the training process.

[0003] The above information disclosed in this Background section is only for enhancement of understanding of the background of the inventive concept and therefore it may contain information that does not form the prior art that is already known in this country to a person of ordinary skill in the art. Summary of the invention

[0004] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0005] Some embodiments of the present disclosure propose a data processing method, a data processing device, a federated learning system, an electronic device, a computer-readable medium, and a computer program product to solve one or more of the technical problems mentioned in the above background technology section.

[0006] In a first aspect, some embodiments of the present disclosure provide a data processing method for a master participant in a federated learning system, the method comprising: in response to receiving inner product data sent by a follower participant in the federated learning system, determining a prediction result of a model based on the inner product data; determining a data residual based on the prediction result and corresponding label data; encrypting the data residual, and sending the encrypted residual data to participants other than the master participant in the federated learning system; wherein the federated learning system is a system involving at least three parties.

[0007] In some embodiments, encrypting the data residual includes: splitting the data residual into multiple sub-residual data according to the number of participants in the federated learning system, wherein the number of the multiple sub-residual data is equal to the number of participants.

[0008] In some embodiments, the encrypted residual data is sent to participants other than the main participant in the federated learning system, including: selecting a specified number of sub-residual data from multiple sub-residual data to be retained in the main participant, wherein the specified number is one less than the number of participants; and reselecting a specified number of sub-residual data to be sent to the participants other than the main participant in the federated learning system respectively; wherein the sub-residual data obtained by each participant in the federated learning system are different.

[0009] In some embodiments, the method also includes: in response to receiving encrypted feature data sent by the follower participant, determining a first weight gradient of the follower participant based on the encrypted feature data and the encrypted residual data, wherein the encrypted feature data and the encrypted residual data are encrypted in the same manner; and sending the first weight gradient to the follower participant so that the follower participant updates its own weight data.

[0010] In some embodiments, based on the encrypted feature data and the encrypted residual data, a first weight gradient of the follower participant is determined, including: generating a random seed according to an elliptic curve data signature algorithm, and generating a random vector according to the random seed; generating the first weight gradient of the follower participant according to the encrypted feature data, the encrypted residual data and the random vector.

[0011] In some embodiments, the method further includes: performing reverse calculation based on the data residual to determine the weight gradient of the main participant; and updating its own weight data based on the weight gradient to continue model training.

[0012] In a second aspect, some embodiments of the present disclosure provide a data processing device, including: a prediction unit, configured to determine a prediction result of a model according to the inner product data in response to receiving inner product data sent by a follower participant in a federated learning system; a residual determination unit, configured to determine a data residual according to the prediction result and corresponding label data; an encryption unit, configured to encrypt the data residual, and send the encrypted residual data to participants other than the main participant in the federated learning system; wherein the federated learning system is a system involving at least three parties.

[0013] In some embodiments, the encryption unit is further configured to split the data residual into a plurality of sub-residual data according to the number of participants in the federated learning system, wherein the number of the plurality of sub-residual data is equal to the number of participants.

[0014] In some embodiments, the encryption unit is further configured to select a specified number of sub-residual data from multiple sub-residual data to be retained in the main participant, wherein the specified number is one less than the number of participants; and re-select a specified number of sub-residual data to be sent separately to participants other than the main participant in the federated learning system; wherein the sub-residual data obtained by each participant in the federated learning system are different.

[0015] In some embodiments, the data processing device also includes a first weight gradient determination unit, which is configured to determine the first weight gradient of the follower participant in response to receiving encrypted feature data sent by the follower participant, based on the encrypted feature data and encrypted residual data, wherein the encrypted feature data and the encrypted residual data are encrypted in the same way; and send the first weight gradient to the follower participant so that the follower participant updates its own weight data.

[0016] In some embodiments, the first weight gradient determination unit is further configured to generate a random seed according to the elliptic curve data signature algorithm, and to generate a random vector according to the random seed; and to generate a first weight gradient of the follower participant according to the encrypted feature data, the encrypted residual data and the random vector.

[0017] In some embodiments, the data processing device also includes a weight updating unit, which is configured to perform reverse calculation based on the data residual to determine the weight gradient of the main participant; and update its own weight data based on the weight gradient to continue model training.

[0018] In a third aspect, some embodiments of the present disclosure provide a data processing method for a follower participant in a federated learning system, the method comprising: in response to receiving encrypted residual data sent by a master participant in the federated learning system, determining a weight gradient of the follower participant based on the encrypted residual data, wherein the encrypted residual data is obtained using the data processing method described in any implementation method of the first aspect above; and updating its own weight data according to the weight gradient.

[0019] In some embodiments, the method also includes: splitting the feature data of the follower participant into multiple sub-feature data according to the number of participants in the federated learning system, wherein the number of the multiple sub-feature data is equal to the number of participants; selecting a specified number of sub-feature data from the multiple sub-feature data to be retained in the follower participant, wherein the specified number is one less than the number of participants; and reselecting a specified number of sub-feature data to be sent as encrypted feature data to the participants other than the follower participant in the federated learning system; wherein the sub-feature data obtained by each participant in the federated learning system are different.

[0020] In some embodiments, based on the encrypted residual data, a weight gradient of a follower participant is determined, including: in response to receiving sub-residual data sent by a main participant, determining a second weight gradient of the follower participant based on the sub-residual data and retained sub-feature data; and in response to receiving a corresponding weight gradient sent by a participant other than the follower participant, merging the corresponding weight gradient and the second weight gradient to obtain the weight gradient of the follower participant.

[0021] In some embodiments, determining a second weight gradient of the follower participant based on the sub-residual data and the retained sub-feature data includes: generating a random vector according to an elliptic curve data signature algorithm; and determining the second weight gradient of the follower participant based on the sub-residual data, the retained sub-feature data and the random vector.

[0022] In some embodiments, the method further includes: in response to receiving encrypted feature data sent by other follower participants in the federated learning system, determining the third weight gradient of the other follower participants based on the received encrypted feature data and sub-residual data, and sending the third weight gradient to the corresponding other follower participants.

[0023] In a fourth aspect, some embodiments of the present disclosure provide a data processing device, comprising: a gradient determination unit, configured to determine the weight gradient of a follower participant based on the encrypted residual data in response to receiving encrypted residual data sent by a master participant in a federated learning system, wherein the encrypted residual data is obtained using the data processing method described in any implementation method in the first aspect above; and a weight updating unit, configured to update its own weight data according to the weight gradient.

[0024] In some embodiments, the data processing device also includes a feature encryption unit, which is configured to split the feature data of the follower participant into multiple sub-feature data according to the number of participants in the federated learning system, wherein the number of the multiple sub-feature data is equal to the number of participants; select a specified number of sub-feature data from the multiple sub-feature data to be retained in the follower participant, wherein the specified number is one less than the number of participants; and re-select a specified number of sub-feature data to be sent as encrypted feature data to the participants other than the follower participant in the federated learning system; wherein the sub-feature data obtained by each participant in the federated learning system are different.

[0025] In some embodiments, the gradient determination unit is further configured to, in response to receiving sub-residual data sent by the main participant, determine a second weight gradient of the follower participant based on the sub-residual data and the retained sub-feature data; and in response to receiving a corresponding weight gradient sent by a participant other than the follower participant, merge the corresponding weight gradient and the second weight gradient to obtain the weight gradient of the follower participant.

[0026] In some embodiments, the gradient determination unit is further configured to generate a random vector according to an elliptic curve data signature algorithm; and determine a second weight gradient of the follower participant according to the sub-residual data, the retained sub-feature data and the random vector.

[0027] In some embodiments, the data processing device also includes a third weight gradient determination unit, which is configured to determine the third weight gradient of other follower participants in response to receiving encrypted feature data sent by other follower participants in the federated learning system based on the received encrypted feature data and sub-residual data, and send the third weight gradient to the corresponding other follower participants.

[0028] In a fifth aspect, some embodiments of the present disclosure provide a federated learning system, comprising: a master participant, configured to receive inner product data sent by a follower participant in the federated learning system, and execute the data processing method described in any implementation method in the first aspect above to update its own weight data, and send encrypted residual data to participants other than the master participant in the federated learning system; at least one follower participant, configured to send inner product data to the master participant, and receive the encrypted residual data sent by the master participant, and execute the data processing method described in any implementation method in the third aspect above to update its own weight data; wherein the federated learning system is a system involving at least three parties.

[0029] In a sixth aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by one or more processors, the one or more processors implement the data processing method described in any implementation method of the first aspect or the third aspect above.

[0030] In a seventh aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the data processing method described in any one of the implementations in the first aspect or the third aspect above is implemented.

[0031] In an eighth aspect, some embodiments of the present disclosure provide a computer program product, including a computer program, which, when executed by a processor, implements the data processing method described in any implementation of the first aspect or the third aspect.

[0032] The data processing method in some embodiments of the present disclosure can realize efficient multi-party federated logistic regression training, and at the same time, can ensure the data security of multiple parties. Specifically, since the weights, features and other data of each participant are only known to themselves, the follower participant in the federated learning system can directly send its inner product data to the main participant. This can not only ensure the data security of the follower participant, but also avoid the follower participant from encrypting the feature data. At the same time, it can avoid the main participant from decrypting the inner product data or the weight gradient in the process of updating the weight. This can simplify the data processing process and shorten the training time of each round of the model, thereby helping to improve the efficiency of model training. In addition, the data residual of the main participant is encrypted and the encrypted residual data is sent to other participants to achieve weight update. This can also ensure the data security of the main participant and prevent data leakage. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0034] Figure 1 is a flow chart of some embodiments of the data processing method for a main participant disclosed in the present invention;

[0035] Figure 2 are flow charts of other embodiments of the data processing method for a main participant disclosed in the present invention;

[0036] Figure 3 is a schematic diagram of the structure of some embodiments of the data processing device disclosed in the present invention;

[0037] Figure 4 is a flow chart of some embodiments of the data processing method for a follower participant disclosed in the present invention;

[0038] Figure 5 is a schematic diagram of the structure of some other embodiments of the data processing device disclosed in the present invention;

[0039] Figure 6 is a schematic diagram of some embodiments of the federated learning system of the present disclosure;

[0040] Figure 7 It is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0041] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0042] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0043] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0044] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0045] Figure 1 The process 100 of some embodiments of the data processing method according to the present disclosure is shown. The data processing method can be used for the main participants in the federated learning system. It should be noted that federated learning is generally a distributed machine learning technology. Its core idea is generally to build a global model based on virtual fusion data by performing distributed model training between multiple data sources with local data, without the need to exchange local individual or sample data, only by exchanging model parameters or intermediate results, so as to achieve a balance between data privacy protection and data sharing computing.

[0046] In the disclosed embodiment, the federated learning system may be a system involving at least three parties. That is, the federated learning system may include a main participant and at least one follower participant. For example, there may be a main participant, a follower participant and a trusted third party. For another example, there may be a main participant and at least two follower participants. Among them, the main participant is usually a data party with labeled (annotated) data. The follower participant is usually a data party that does not have labeled data but participates in feature learning. The trusted third party may be a data party that does not participate in feature learning but only participates in data processing. The method may include the following steps:

[0047] Step 101, in response to receiving inner product data sent by a follower participant in a federated learning system, determining a prediction result of a model according to the inner product data.

[0048] In some embodiments, the execution subject of the data processing method (for example Figure 6 The main participant Leader shown in the figure can communicate with other participants in the federated learning system through wired or wireless connections. Here, the main participant and follower participant in the federated learning system can set their own data respectively and initialize their own weights according to the feature dimensions. Then, these participants can perform feature learning and forward calculations. That is, the inner product of feature data and weight is calculated respectively. In addition, each follower participant can send its own inner product data to the main participant (i.e., the execution subject).

[0049] At this time, when the execution subject receives the inner product data sent by the follower participant, it can determine the prediction result of the model based on the inner product data. As an example, the execution subject can combine the inner product data of the follower participant with its own inner product data. Figure 6 In the federated learning system shown in the figure, the main participant can transfer its inner product data z l The inner product data z with the two follower participants f1 、z f2 The sum of is taken as the inner product result z of the model. Then the prediction result of the model can be calculated

[0050] It is understandable that since the follower participant sets his weight and the data is known only to him, it is usually safe to send the inner product data calculated by the follower participant directly to the master participant in the forward process as long as the feature dimension of the follower participant is greater than 1.

[0051] Step 102: determine the data residual according to the prediction result and the corresponding label data.

[0052] In some embodiments, the execution entity may determine the data residual based on the prediction result and the corresponding label data. As an example, the execution entity may calculate the difference between the prediction result and the label data as the data residual, that is,

[0053] Step 103: encrypt the data residual and send the encrypted residual data to the participants in the federated learning system except the main participant.

[0054] It is understandable that in the reverse calculation process, the follower participant often needs to use the label data of the main participant and the feature data of the follower participant when calculating the gradient. Therefore, in the reverse process, it is usually necessary to prevent the leakage of the label data of the main participant and ensure the data security of the follower participant.

[0055] Therefore, in some embodiments, the execution subject may encrypt the data residual obtained in step 102 to obtain encrypted residual data. And the encrypted residual data may be sent to the participants other than the master participant in the federated learning system. In this way, the follower participant can determine the weight gradient according to the encrypted residual data, and then update its own weight data.

[0056] It should be noted that the encryption processing method here is not limited. As an example, the execution subject can use homomorphic encryption (such as FHE, Paillier's method) to encrypt data. Among them, homomorphic encryption generally refers to the original data being homomorphically encrypted, the obtained ciphertext being subjected to a specific operation, and then the calculation result being homomorphically decrypted to obtain the plaintext, which is usually equivalent to the data result obtained by directly performing the same calculation on the original plaintext data. FHE is a fully homomorphic encryption method, that is, the encryption function satisfies both additive homomorphism and multiplicative homomorphism. Paillier is a homomorphic public key encryption algorithm, that is, ciphertext multiplication is equal to plaintext addition.

[0057] Optionally, the execution subject can split the data residual into multiple sub-residual data according to the number of all participants in the federated learning system. The number of multiple sub-residual data is equal to the number of participants. That is, the data residual can be split according to the rule that A=A1+A2+A3 by taking A as the target data and generating A1, A2, and A3. As an example, Figure 6 As shown, the execution entity can split the data residual dy into dy0, dy1 and dy2.

[0058] At this time, the execution subject can select a specified number of sub-residual data from multiple sub-residual data to be retained in the main participant. The specified number can be one less than the number of participants. And the execution subject can re-select a specified number of sub-residual data to send to the participants other than the main participant in the federated learning system. There are differences in the sub-residual data obtained by each participant in the federated learning system. In other words, the sub-residual data obtained by each participant is not exactly the same.

[0059] As an example, Figure 6As shown, the execution subject can retain dy0 and dy1. In addition, dy1 and dy2 can be sent to follower participant Follower1, and dy0 and dy2 can be sent to follower participant Follower2. In this way, follower participant Follower1 can determine the weight gradient based on dy1 and dy2 to update its own weight. And follower participant Follower2 can determine the weight gradient based on dy0 and dy2 to update its own weight. For details, please refer to Figure 4 The relevant description in the embodiments will not be repeated here.

[0060] It is understandable that the use of homomorphic encryption usually results in a huge time overhead. Even though special fully homomorphic encryption schemes can concatenate multiple sets of ciphertexts to reduce computing time, homomorphic encryption itself determines that a lot of time is needed for encryption and decryption and ciphertext operations, and one round of training often takes several hours. The above method of splitting the data into multiple sub-data and sending them to different participants does not need to rely on slower encryption technology. Federated logistic regression training is completed by using an algorithm that requires at least three-party secure computing. In this way, the training of the federated model can be completed efficiently and securely while protecting the label data of the main participant and the feature data of the follower participant.

[0061] It should be noted that most of the related federated logistic regression algorithms are two-party logistic regression based on homomorphic encryption. One type of solution requires the help of a trusted third party. This type of solution often uses semi-homomorphic encryption as a security measure. Therefore, it is necessary to perform Taylor expansion approximation on the nonlinear calculation formula in logistic regression and convert the loss function Approximate Among them, y i represents label data; θ T x i represents the inner product data; x i represents feature data; n represents the number of samples.

[0062] After entering the formal operation, the third-party assistant will first generate and distribute the key to the two participants for encryption of data. And the ciphertext and Sent to the main participant. The main participant calculates its own ciphertext gradient based on this The follower participant is then used to compute the critical part of the gradient (i.e. residual) is sent to the follower participants. The follower participants calculate their own gradients based on this Then, both parties use a trusted third party to decrypt the gradients, and then use the decrypted gradients to update the weights to complete this round of training.

[0063] Another type of solution does not require the participation of a trusted third party. The key is generated and distributed by the master party. Encrypted Sent to the main participant. The main participant decrypts it after receiving it and combines it with his own To calculate the prediction results and its own gradient g A Then, the residual of the ciphertext Sent to follower participants to calculate Subsequently, the follower participants need to use noise-adding methods to ensure Send it to the main participant in a safe way. Decrypt it with the help of the main participant's private key to obtain the plaintext gradient. Then both parties use the gradient to update the weight.

[0064] From the above description, we can see that the related federated logistic regression solutions usually cannot solve the problem of multiple participants (at least three parties) very well. In order to solve the scenario of multiple participants, it is usually necessary to repeat what two parties do. This will increase the time cost and the situation where the participants are stuck waiting. This will affect the training efficiency of the model.

[0065] In the data processing method in some embodiments of the present disclosure, since the weights, features and other data of each participant are only known to themselves, the follower participant in the federated learning system can directly send its inner product data to the main participant. This can not only ensure the data security of the follower participant, but also avoid the follower participant from encrypting the feature data. At the same time, it can avoid the main participant from decrypting the inner product data or the weight gradient in the process of updating the weight. This can simplify the data processing process and shorten the training time of each round of the model, thereby helping to improve the efficiency of model training. In addition, by encrypting the data residual of the main participant and sending the encrypted residual data to other participants to achieve weight update. This can also ensure the data security of the main participant and prevent data leakage. In other words, the embodiment of the present disclosure provides an efficient multi-party federated logistic regression algorithm. Under the premise of realizing lossless multi-party logistic regression training, the data security of multiple parties can be guaranteed, that is, the feature data security of the follower participant and the label data security of the main participant.

[0066] In some embodiments, the execution subject can perform reverse calculation based on the data residual obtained in step 102 to determine the weight gradient of the main participant. Then, the weight data of the main participant can be updated based on the weight gradient. As an example, Figure 6 The weight gradient of the main participant Leader shown in Among them, dy here is the data residual dy mentioned above i The updated weight can be wl ′=w l -α*dw l Among them, w l represents the weight of the main participant in the previous round; α is the hyperparameter learning rate.

[0067] Continue to refer Figure 2 , which shows a process 200 of another embodiment of the data processing method according to the present disclosure. The method comprises the following steps:

[0068] Step 201, in response to receiving encrypted feature data sent by a follower participant, determining a first weight gradient of the follower participant based on the encrypted feature data and the retained sub-residual data.

[0069] In some embodiments, during the reverse calculation process, the follower participant can also use the same encryption processing method as the main participant to send the encrypted feature data to the main participant to implement weight gradient calculation. Figure 4 The relevant description in the embodiments will not be repeated here.

[0070] At this time, the execution subject (main participant) of the data processing method, upon receiving the encrypted feature data sent by the follower participant, can determine the first weight gradient of the follower participant based on the encrypted feature data and the encrypted residual data (i.e., the retained sub-residual data). The encrypted feature data and the encrypted residual data can be encrypted in the same way. As an example, the execution subject can multiply the encrypted feature data with the sub-residual data retained by itself. Then, the multiplication results are added together to serve as the first weight gradient of the follower participant.

[0071] In some embodiments, the execution subject may also generate a random seed according to the elliptic curve data signature algorithm (ECDSA). For example, a zero-share random seed is generated. Then, a random vector is generated according to the random seed. At this time, the execution subject may generate a first weight gradient of the follower participant according to the encrypted feature data, the retained sub-residual data (i.e., the encrypted residual data), and the random vector. As an example, Figure 6 As shown, when the Leader receives the encrypted feature data x0 and x1 sent by Follower1, and the encrypted feature data v0 and v1 sent by Follower2, it can be calculated according to the following formula:

[0072]

[0073]

[0074] in, Represents the first weight gradient of Follower1; Represents the first weight gradient of Follower2; a1 and a2 are random number vectors generated by Leader based on its own zero-share random seed.

[0075] It is understandable that by generating random vectors to calculate the weight gradient, this will not affect the accuracy of the calculation results, but can further improve the security of the data.

[0076] Step 202: Send the first weight gradient to the follower participant so that the follower participant updates its own weight data.

[0077] In some embodiments, the execution subject may send the first weight gradient obtained in step 201 to the corresponding follower participant, so that the follower participant updates its own weight data. The update process here can be seen in Figure 4 The relevant description in the embodiments will not be repeated here.

[0078] The data processing method in the disclosed embodiment further enriches and improves the reverse calculation process of multi-party joint. The data residual and feature data are split and sent to different participants respectively. And the data obtained by each participant is different for interactive calculation. This can not only ensure the security of the data, but also reduce or avoid the traditional slower data encryption technology. Under the premise of protecting the data, efficient multi-party federated logistic regression training can be achieved.

[0079] Further references Figure 3 , as a response to the above Figure 1 , 2 In order to realize the data processing method shown in the figure, the present disclosure provides some embodiments of a data processing device. These data processing device embodiments are Figure 1 , 2 The data processing device can be specifically applied to various electronic devices.

[0080] like Figure 3 As shown, the data processing device 300 of some embodiments may include: a prediction unit 301, configured to determine the prediction result of the model according to the inner product data in response to receiving the inner product data sent by the follower participant in the federated learning system; a residual determination unit 302, configured to determine the data residual according to the prediction result and the corresponding label data; an encryption unit 303, configured to encrypt the data residual and send the encrypted residual data to the participants other than the main participant in the federated learning system; wherein the federated learning system is a system involving at least three parties.

[0081] In some embodiments, the encryption unit 303 may be further configured to split the data residual into a plurality of sub-residual data according to the number of participants in the federated learning system, wherein the number of the plurality of sub-residual data is equal to the number of participants.

[0082] In some embodiments, the encryption unit 303 can be further configured to select a specified number of sub-residual data from multiple sub-residual data to be retained in the main participant, wherein the specified number is one less than the number of participants; and re-select a specified number of sub-residual data to be sent to participants other than the main participant in the federated learning system respectively; wherein the sub-residual data obtained by each participant in the federated learning system are different.

[0083] In some embodiments, the data processing device 300 may also include a first weight gradient determination unit (not shown in the figure), which is configured to determine the first weight gradient of the follower participant in response to receiving the encrypted feature data sent by the follower participant, based on the encrypted feature data and the encrypted residual data, wherein the encrypted feature data and the encrypted residual data are encrypted in the same manner; and send the first weight gradient to the follower participant so that the follower participant updates its own weight data.

[0084] In some embodiments, the first weight gradient determination unit is further configured to generate a random seed according to the elliptic curve data signature algorithm, and to generate a random vector according to the random seed; and to generate a first weight gradient of the follower participant according to the encrypted feature data, the encrypted residual data and the random vector.

[0085] In some embodiments, the data processing device 300 may also include a weight updating unit (not shown in the figure), which is configured to perform reverse calculation based on the data residual to determine the weight gradient of the main participant; and update its own weight data based on the weight gradient to continue model training.

[0086] It is understood that the units described in the data processing device 300 are similar to those described in the reference Figure 1 , 2 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the data processing device 300 and the units included therein, and will not be described in detail here.

[0087] Reference below Figure 4 , which shows a process 400 of some embodiments of the data processing method according to the present disclosure. The data processing method can be used for a follower participant in a federated learning system. The method includes the following steps:

[0088] Step 401, in response to receiving encrypted residual data sent by a master participant in a federated learning system, determining a weight gradient of a follower participant based on the encrypted residual data.

[0089] In some embodiments, the execution subject of the data processing method (such as Figure 6 The follower participants Follower1 and Follower2 shown in the figure can also communicate with other participants in the federated learning system through wired or wireless connections. Here, the execution subject can send the inner product data learned directly to the main participant for model prediction. At the same time, the main participant can use the above Figure 1 , 2 The data processing method in the embodiment performs data residual encryption and sends the encrypted residual data to the execution subject.

[0090] Here, when the execution subject receives the encrypted residual data sent by the main participant, it can perform weight gradient calculation based on the encrypted residual data to determine its own weight gradient. As an example, the execution subject can calculate the encrypted weight gradient based on the encrypted residual data and its own feature data. Then, the gradient is decrypted through a trusted third party in the system to obtain the decrypted weight gradient.

[0091] In some embodiments, the execution entity can also split the feature data of the follower participant into multiple sub-feature data according to the number of participants in the federated learning system. The number of the multiple sub-feature data is equal to the number of participants. Then, a specified number of sub-feature data can be selected from the multiple sub-feature data to be retained in the follower participant. The specified number is one less than the number of participants. And a specified number of sub-feature data can be re-selected to be sent as encrypted feature data to the participants other than itself (the follower participant) in the federated learning system. There are differences in the sub-feature data obtained by each participant in the federated learning system.

[0092] As an example, Figure 6 As shown, Follower1 can split feature data x into x0, x1, and x2. It can also keep x1 and x2, send x0 and x1 to the Leader, and send x0 and x2 to Follower2. Similarly, Follower2 can split feature data v into v0, v1, and v2. It can also keep v0 and v2, send v0 and v1 to the Leader, and send v1 and v2 to Follower1.

[0093] At this time, if the execution subject receives the sub-residual data sent by the main participant, then based on the sub-residual data and the retained sub-feature data, the execution subject can determine the second weight gradient of the follower participant itself.

[0094] In some embodiments, the follower participant may also generate a random vector according to the elliptic curve data signature algorithm, and then determine the second weight gradient of the follower participant according to the sub-residual data, the retained sub-feature data and the random vector.

[0095] In addition, since the execution subject also sends the encrypted feature data to other participants (such as Leader and Follower2), and other participants will also receive the sub-residual data sent by the main participant, other participants will also perform corresponding weight gradient calculations. And the gradient calculation results will be sent to the execution subject. At this time, if the execution subject receives the corresponding weight gradient sent by a participant other than itself, it can merge the corresponding weight gradient with the second weight gradient to obtain its own weight gradient.

[0096] It is understandable that for other followers in the federated learning system, the execution subject is also the other participants. In other words, during the reverse interactive calculation, other followers will also send their own encrypted feature data to the execution subject.

[0097] In some embodiments, when the execution subject receives the encrypted feature data sent by other follower participants, it can also determine the third weight gradient of other follower participants according to the received encrypted feature data and sub-residual data, and can send the third weight gradient to the corresponding other follower participants.

[0098] As an example, Figure 6 As shown, Follower1 and Follower2 can also calculate the second weight gradient and the third weight gradient according to the following formula:

[0099] Follower1:

[0100]

[0101] Follower2:

[0102]

[0103] Among them, b1 and b2 are random number vectors generated by Follower1; c1 and c2 are random number vectors generated by Follower2; here, a i +b i +c i =0,a i is the i-th element of vector a.

[0104] At this point, the follower participants can merge the weight gradients:

[0105] Followerl:

[0106] Follower2:

[0107] in, Figure 6 In the figure, number ① indicates data sharing; number ② indicates gradient calculation; number ③ indicates synchronization of calculation results; number ④ indicates merging weight gradients; number ⑤ indicates updating weights.

[0108] Step 402, update own weight data according to weight gradient.

[0109] In some embodiments, the execution entity may adopt the above Figure 1 The relevant formula in the embodiment is used to update its own weight.

[0110] In the data processing method of the embodiment of the present disclosure, the follower participant can directly send his inner product data to the main participant. This can not only ensure the data security of the follower participant, but also avoid the follower participant from encrypting the feature data. At the same time, it can avoid the main participant from decrypting the inner product data or the weight gradient in the process of updating the weight. In turn, the data processing process can be simplified and the training time of each round of the model can be shortened, which helps to improve the efficiency of model training. In addition, the main participant encrypts the data residuals and sends the encrypted residual data to other participants to achieve weight update. This can also ensure the data security of the main participant and prevent data leakage. In other words, the embodiment of the present disclosure provides an efficient multi-party federated logistic regression algorithm. Under the premise of realizing lossless multi-party logistic regression training, the data security of multiple parties can be guaranteed, that is, the feature data security of the follower participants and the label data security of the main participant.

[0111] Continue to refer Figure 5 , as a response to the above Figure 4 In order to realize the data processing method shown in the figure, the present disclosure provides some embodiments of a data processing device. These data processing device embodiments are Figure 4 The data processing device can be specifically applied to various electronic devices.

[0112] like Figure 5 As shown, the data processing device 500 of some embodiments may include: a gradient determination unit 501, configured to determine the weight gradient of the follower participant based on the encrypted residual data in response to receiving the encrypted residual data sent by the master participant in the federated learning system, wherein the encrypted residual data adopts the above Figure 1 , 2In the embodiment, the data processing method described in any implementation manner is obtained; the weight updating unit 502 is configured to update its own weight data according to the weight gradient.

[0113] In some embodiments, the data processing device 500 may also include a feature encryption unit (not shown in the figure), which is configured to split the feature data of the follower participant into multiple sub-feature data according to the number of participants in the federated learning system, wherein the number of the multiple sub-feature data is equal to the number of participants; select a specified number of sub-feature data from the multiple sub-feature data to be retained in the follower participant, wherein the specified number is one less than the number of participants; and re-select a specified number of sub-feature data to be sent as encrypted feature data to the participants other than the follower participant in the federated learning system; wherein the sub-feature data obtained by each participant in the federated learning system are different.

[0114] In some embodiments, the gradient determination unit 501 can be further configured to, in response to receiving sub-residual data sent by the main participant, determine the second weight gradient of the follower participant based on the sub-residual data and the retained sub-feature data; and in response to receiving the corresponding weight gradient sent by a participant other than the follower participant, merge the corresponding weight gradient and the second weight gradient to obtain the weight gradient of the follower participant.

[0115] In some embodiments, the gradient determination unit 501 can be further configured to generate a random vector based on the elliptic curve data signature algorithm; and determine the second weight gradient of the follower participant based on the sub-residual data, the retained sub-feature data and the random vector.

[0116] In some embodiments, the data processing device 500 may also include a third weight gradient determination unit (not shown in the figure), which is configured to determine the third weight gradient of other follower participants in response to receiving encrypted feature data sent by other follower participants in the federated learning system based on the received encrypted feature data and sub-residual data, and send the third weight gradient to the corresponding other follower participants.

[0117] It is understood that the units described in the data processing device 500 are similar to those described in the reference Figure 4 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the data processing device 500 and the units included therein, and will not be described in detail here.

[0118] The embodiments of the present disclosure also provide a federated learning system. The federated learning system may be a system in which at least three parties participate. Figure 6 As shown, the federated learning system may include a master participant and at least one follower participant.

[0119] Here, the master participant can be used to receive the inner product data sent by the follower participant in the federated learning system and perform Figure 1 , 2 The data processing method described in any implementation of the embodiment is used to update its own weight data and send the encrypted residual data to the participants other than the master participant in the federated learning system. The follower participant can be used to send the inner product data to the master participant, receive the encrypted residual data sent by the master participant, and execute the above Figure 3 The data processing method described in any implementation manner in the embodiment is used to update its own weight data.

[0120] It should be noted that if the federated learning system includes a master participant, a follower participant, and a trusted third party, during reverse calculation, the master participant and the follower participant can also perform data residual splitting and feature data splitting respectively according to the above method, and send them to other participants. At this time, since the trusted third party does not participate in feature learning, there is no feature data, and there is no need to split the data. For the data received from other participants, the weight gradient can be calculated and returned according to the above method. In other words, for participants without feature learning, they can participate in reverse interactive calculations. This can protect the data of other participants.

[0121] The federated learning system of the disclosed embodiment can solve the problem of slow training of multi-party federated logistic regression algorithms. It can improve the training speed of federated logistic regression in multi-party scenarios to be close to the speed of plain text training, and ensure that the privacy information of multiple parties is not leaked.

[0122] It should be noted that the disclosed federated learning system can be applied to various scenarios, such as joint risk prediction, feature weight learning, potential user mining, user analysis in the banking and insurance industries, etc. A more common scenario is joint risk prediction, and in some risk control scenarios, more emphasis is placed on the interpretability of model weights. Therefore, some models with strong interpretability, including logistic regression, will be used. Sample features are often scattered in different data parties, and using more complete sample features can better perform risk analysis. When the participants are three parties, this solution can be used to efficiently and securely complete cross-domain logistic regression training. For another example, a bank needs to predict the behavior of a certain user, such as whether to take out a loan, repay a loan, etc. However, the bank may not have enough features to analyze and predict the user's behavior. In this scenario, the strategy of federated learning can be used to combine other authoritative data parties. Predictions can be made with the help of features from other angles, including but not limited to users' shopping habits, short video browsing preferences, etc.

[0123] Reference below Figure 7 , which shows a structural schematic diagram of an electronic device 700 suitable for implementing some embodiments of the present disclosure. Figure 7 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0124] like Figure 7 As shown, the electronic device 700 may include a processing device 701 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the electronic device 700 are also stored. The processing device 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0125] Typically, the following devices may be connected to the I / O interface 705: input devices 706 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 707 including, for example, a speaker, a vibrator, etc.; storage devices 708 including, for example, a hard disk, a magnetic disk, etc.; and communication devices 709. The communication devices 709 may allow the electronic device 700 to communicate with other devices wirelessly or by wire to exchange data. Figure 7 The electronic device 700 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 7 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0126] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 709, or installed from the storage device 708, or installed from the ROM 702. When the computer program is executed by the processing device 701, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

[0127] It should be noted that the computer-readable medium recorded in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0128] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0129] The computer-readable medium may be included in the electronic device; or it may exist independently without being installed in the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: in response to receiving the inner product data sent by the follower participant in the federated learning system, determines the prediction result of the model according to the inner product data; determines the data residual according to the prediction result and the corresponding label data; encrypts the data residual, and sends the encrypted residual data to the participants in the federated learning system other than the main participant; wherein the federated learning system is a system involving at least three parties including at least one follower participant.

[0130] Alternatively, in response to receiving the encrypted residual data sent by the master participant in the federated learning system, the weight gradient of the follower participant is determined based on the encrypted residual data, wherein the encrypted residual data adopts the above Figure 1 , 2 In the embodiment, the data processing method described in any implementation manner is obtained; updating the own weight data according to the weight gradient.

[0131] In addition, computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0132] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0133] The units described in some embodiments of the present disclosure may be implemented by software or by hardware. The units described may also be provided in a processor, for example, may be described as: a processor including a prediction unit, a residual determination unit, and an encryption unit; or a processor including a gradient determination unit and a weight update unit. The names of these units do not, in some cases, constitute limitations on the units themselves, for example, a prediction unit may also be described as a "unit for determining the prediction result of a model based on inner product data".

[0134] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0135] Some embodiments of the present disclosure further provide a computer program product, including a computer program, which implements any of the above-mentioned data processing methods when executed by a processor.

[0136] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) and the technical solutions formed.

Claims

1. A data processing method for a main participant in a federated learning system, the method include: In response to receiving inner product data sent by a follower participant in the federated learning system, determining a prediction result of the model according to the inner product data; Determine a data residual according to the prediction result and the corresponding label data; Encrypting the data residual, and sending the encrypted residual data to the participants in the federated learning system except the main participant; The federated learning system is a system involving at least three parties.

2. The data processing method according to claim 1, in, The encrypting the data residual includes: According to the number of participants in the federated learning system, the data residual is split into a plurality of sub-residual data, wherein the number of the plurality of sub-residual data is equal to the number of the participants.

3. The data processing method according to claim 2, in, The step of sending the encrypted residual data to the participants other than the main participant in the federated learning system includes: Selecting a specified number of sub-residual data from the plurality of sub-residual data to be retained in the primary participant, wherein the specified number is one less than the number of the participants; and Reselecting the specified number of sub-residual data to send them to the participants in the federated learning system except the main participant respectively; Among them, there are differences in the sub-residual data obtained by each participant in the federated learning system.

4. The data processing method according to claim 1, in, The method further comprises: In response to receiving the encrypted feature data sent by the follower participant, determining a first weight gradient of the follower participant based on the encrypted feature data and the encrypted residual data, wherein the encrypted feature data and the encrypted residual data are encrypted in the same manner; The first weight gradient is sent to the follower participant so that the follower participant updates its own weight data.

5. The data processing method according to claim 4, in, The determining, based on the encrypted feature data and the encrypted residual data, a first weight gradient of the follower participant comprises: Generate a random seed according to an elliptic curve data signature algorithm, and generate a random vector according to the random seed; A first weight gradient of the follower participant is generated according to the encrypted feature data, the encrypted residual data and the random vector.

6. The data processing method according to any one of claims 1 to 5, in, The method further comprises: Perform reverse calculation based on the data residual to determine the weight gradient of the main participant; Update its own weight data according to the weight gradient to continue model training.

7. A data processing device, include: A prediction unit, configured to, in response to receiving inner product data sent by a follower participant in the federated learning system, determine a prediction result of a model according to the inner product data; A residual determination unit, configured to determine a data residual according to the prediction result and the corresponding label data; an encryption unit, configured to encrypt the data residual and send the encrypted residual data to the participants other than the main participant in the federated learning system; The federated learning system is a system involving at least three parties including at least one follower participant.

8. A data processing method for a follower participant in a federated learning system, the method include: In response to receiving the encrypted residual data sent by the master participant in the federated learning system, determining the weight gradient of the follower participant based on the encrypted residual data, wherein the encrypted residual data is obtained by using the data processing method according to any one of claims 1 to 6; Update own weight data according to the weight gradient.

9. The data processing method according to claim 8, in, The method further comprises: According to the number of participants in the federated learning system, split the feature data of the follower participant into a plurality of sub-feature data, wherein the number of the plurality of sub-feature data is equal to the number of the participants; Selecting a specified number of sub-feature data from the plurality of sub-feature data to be retained in the follower participant, wherein the specified number is one less than the number of the participants; and Reselecting the specified number of sub-feature data as encrypted feature data and sending them to the participants in the federated learning system except the follower participant respectively; Among them, there are differences in the sub-feature data obtained by each participant in the federated learning system.

10. The data processing method according to claim 9, in, The determining the weight gradient of the follower participant based on the encrypted residual data includes: In response to receiving the sub-residual data sent by the master participant, determining a second weight gradient of the follower participant based on the sub-residual data and the retained sub-feature data; and In response to receiving a corresponding weight gradient sent by a participant other than the follower participant, the corresponding weight gradient and the second weight gradient are combined to obtain the weight gradient of the follower participant.

11. The data processing method according to claim 10, in, The determining, based on the sub-residual data and the retained sub-feature data, a second weight gradient of the follower participant comprises: Generate a random vector according to the elliptic curve data signature algorithm; and A second weight gradient of the follower participant is determined based on the sub-residual data, the retained sub-feature data and the random vector.

12. The data processing method according to claim 10, in, The method further comprises: In response to receiving encrypted feature data sent by other follower participants in the federated learning system, determining third weight gradients of other follower participants based on the received encrypted feature data and the sub-residual data, and sending the third weight gradient to the corresponding other follower participants.

13. A federated learning system, include: A master participant, configured to receive inner product data sent by a follower participant in the federated learning system, and execute the data processing method according to any one of claims 1 to 6 to update its own weight data, and send encrypted residual data to participants other than the master participant in the federated learning system; At least one follower participant, configured to send inner product data to the master participant, receive encrypted residual data sent by the master participant, and execute the data processing method according to any one of claims 8 to 12 to update its own weight data; The federated learning system is a system involving at least three parties.

14. An electronic device, include: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method as described in any one of claims 1-6 or 8-12.

15. A computer readable medium having a computer program stored thereon, in, When the computer program is executed by a processor, the data processing method as described in any one of claims 1-6 or 8-12 is implemented.

16. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the data processing method according to any one of claims 1 to 6 or 8 to 12.

Images