Distributed neural network training method and system
By splitting the gradient component values and substituting them into preset functions in distributed neural network training, the problem that users are unwilling to disclose private training data is solved, and the secure training calculation of distributed neural network is realized.
Patent Information
- Application Number
- CN202411535061.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-31
- Publication Date
- 2025-02-11
AI Technical Summary
In distributed neural network training, the private training data held by users are unwilling to be disclosed to other users, but the quality of lack of joint training will be greatly reduced.
By randomly splitting the gradient component value into preset integer copies and substituting the split gradient component value into the preset function, the user sends the function value to the operator, and the operator calculates and updates the distributed neural network.
Without leaking user's private training data, the training calculation of distributed neural network is implemented to ensure the security of gradient component values.
Smart Images
Figure CN119416867A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of distributed neural network training, and in particular to a distributed neural network training method and system. Background Art
[0002] A distributed neural network refers to a neural network whose parts are deployed on different servers, and each part of the neural network is owned and used by different users. The training of a distributed neural network generally requires users to upload training data to train the neural network. However, the training data held by users may involve private data assets that users do not want to disclose to other users. However, the quality of the neural network will be greatly reduced if all users lack joint training. Summary of the invention
[0003] The present application provides a distributed neural network training method and system, which can realize distributed training of distributed neural networks without leaking the user's original training data.
[0004] In a first aspect, the present application provides a distributed neural network training method. The method is applied to a system consisting of a neural network operator and multiple users holding distributed neural networks. The method includes: as a user, Use private training data to train the distributed neural network to obtain gradient component values; Randomly split the gradient component value into agreed first preset integer parts; Substituting the split gradient component values into at least two agreed preset functions respectively to obtain at least two substituted function values, wherein the preset functions are all first preset integer-element functions, and the sum of all the preset functions is the sum of each independent variable times a second preset constant, and the second preset constant is non-zero; All function values are sent to the operator so that the operator can obtain the sum of the gradient component values of all users by dividing the sum of all function values of all users by the agreed second preset constant. The operator trains the neural network and updates the distributed neural network based on the sum of the gradient component values.
[0005] In a second aspect, the present application provides a distributed neural network training method. The method is applied to a system consisting of a neural network operator and multiple users holding distributed neural networks. The method includes: as the operator, Receive all function values sent by all users, wherein the function values are obtained by each user randomly splitting the gradient component values obtained by training using private training data into agreed first preset integer parts, and substituting the split gradient component values into agreed at least two preset functions, wherein the preset functions are all first preset integer element functions, and the sum of all preset functions is the sum of each independent variable times a second preset constant, and the second preset constant is non-zero; The sum of all function values divided by the agreed second preset constant is equal to the sum of the gradient component values held by all users. The operator trains the neural network and updates the distributed neural network based on the sum of the gradient component values.
[0006] In a third aspect, the present application provides a distributed neural network training method. The method is applied to a system consisting of an operator of a neural network and multiple users holding distributed neural networks, and the method includes: Each user uses private training data to train the distributed neural network to obtain the gradient component value; Randomly split the gradient component value into agreed first preset integer parts; Substituting the split gradient component values into at least two agreed preset functions respectively to obtain at least two substituted function values, wherein the preset functions are all first preset integer-element functions, and the sum of all the preset functions is the sum of each independent variable times a second preset constant, and the second preset constant is non-zero; The user sends all the function values to the operator so that the operator can calculate the sum of all the function values; The operator calculates that the sum of all function values divided by the agreed second preset constant is equal to the sum of the gradient component values held by all users, and the operator trains the neural network and updates the distributed neural network based on the sum of the gradient component values.
[0007] Furthermore, there are m preset functions, and the jth preset function , where , is the second preset constant.
[0008] Furthermore, there are two preset functions, the first preset function is , the second preset function is , where p+1 is the second preset constant.
[0009] Furthermore, the first preset integer is not less than 5.
[0010] In a fourth aspect, the present application provides a distributed neural network training system. The system includes an operator and multiple user parties, the operator operates the neural network, the user party holds a distributed neural network of the neural network, the user party applies any one of the methods described in the first aspect above, and / or the operator applies any one of the methods described in the second aspect above, and / or the system applies any one of the methods described in the third aspect above.
[0011] By adopting the above technical solution, it is possible to implement training calculations of distributed neural networks, ensuring that the user's private training data and the original gradient components obtained by training with the private training data are not leaked.
[0012] It should be understood that the contents described in the Summary of the Invention are not intended to limit the key or important features of the embodiments of the present application, nor are they intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0013] The above and other features, advantages and aspects of the embodiments of the present application will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein: Figure 1 A flow chart of a distributed neural network training method in an embodiment of the present application is shown. DETAILED DESCRIPTION
[0014] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0015] In addition, the term "and / or" in this article is only a description of the association relationship between the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.
[0016] The present application provides a distributed neural network training method and system, which can realize the calculation of the sum of gradient component values held by multiple users under the premise of ensuring that the gradient component values held by the users are not leaked.
[0017] In a first aspect, the present application provides a distributed neural network training system, which includes an operator and multiple users, wherein the users hold and use the distributed neural network, and the operator operates the neural network.
[0018] The system realizes secure multi-party training of distributed neural networks and is specifically introduced in accordance with the disclosure of the second aspect of this application.
[0019] In a second aspect, the present application provides a distributed neural network training method. The method can be Figure 1 The user and operator parties cooperate in the implementation.
[0020] Figure 1 A flow chart of a distributed neural network training method in an embodiment of the present application is shown.
[0021] Reference Figure 1 , the method specifically comprises the following steps: S110: Each user uses private training data to train the distributed neural network to obtain gradient component values; the process of obtaining the gradient components through holding and training is performed locally on the user. S120: randomly splitting the gradient component value into agreed first preset integer parts; S130: Substituting the split gradient component values into at least two predetermined preset functions respectively to obtain at least two substituted function values, wherein the preset functions are all first preset integer-element functions, and the sum of all the preset functions is the sum of each independent variable times a second preset constant, and the second preset constant is non-zero; S140: The user sends all the substituted function values to the operator, so that the operator calculates the sum of all the function values; S150: The operator calculates that the sum of all function values divided by the agreed second preset constant is equal to the sum of the gradient component values held by all users, and the operator trains the neural network and updates the distributed neural network based on the sum of the gradient component values.
[0022] Specifically, there are m preset functions, and the jth preset function , where , is the second preset constant.
[0023] In one example, there are two preset functions, the first preset function is , the second preset function is , where p+1 is a second preset constant. The first preset integer is not less than 5.
[0024] The technology of substituting training data into a distributed neural network to obtain gradient component values is common knowledge and is not disclosed here. Only the secure multi-party summation method of gradient components is disclosed in detail. The following integrates the above content to introduce a specific example.
[0025] Suppose there are n users, and the users agree on a first preset integer p, where n is not less than 2 and p is not less than 5; each user randomly splits the gradient component value it holds into p parts, and obtains p split data, and the sum of the p split data is equal to the gradient component value. The split function is defined here , where d represents the gradient component value, Indicates split data.
[0026] Assume that the gradient component value of the nth user is , , ..., , The corresponding p split data are , , ..., , The corresponding p split data are , , ..., , ..., The corresponding p split data are , , ..., .
[0027] Define two p-ary functions, , , each user substitutes the gradient component value held by itself into the two functions respectively to obtain the function values.
[0028] Then n users substitute The obtained function values are ; ; ... ; n users enter The obtained function values are ; ; ... ; These function values are clear numerical results and can be sent to the operator. The operator can calculate the sum of the gradient component values held by the user based on the following steps: calculate ; calculate ; calculate That is, it is equal to p+1 times the gradient component values held by n users. Dividing it by p+1 can obtain the sum of the gradient component values held by n users.
[0029] Similarly, the average of the gradient component values held by n users can be calculated.
[0030] In a third aspect, the present application provides a distributed neural network training method. The method is applied to a system consisting of a neural network operator and multiple users holding distributed neural networks. The method includes: as a user, Use private training data to train the distributed neural network to obtain gradient component values; Randomly split the gradient component value into agreed first preset integer parts; Substituting the split gradient component values into at least two agreed preset functions respectively to obtain at least two substituted function values, wherein the preset functions are all first preset integer-element functions, and the sum of all the preset functions is the sum of each independent variable times a second preset constant, and the second preset constant is non-zero; All function values are sent to the operator so that the operator can obtain the sum of the gradient component values of all users by dividing the sum of all function values of all users by the agreed second preset constant. The operator trains the neural network and updates the distributed neural network based on the sum of the gradient component values.
[0031] In a fourth aspect, the present application provides a distributed neural network training method. The method is applied to a system consisting of a neural network operator and multiple users holding distributed neural networks. The method includes: as an operator, Receive all function values sent by all users, wherein the function values are obtained by each user randomly splitting the gradient component values obtained by training using private training data into agreed first preset integer parts, and substituting the split gradient component values into agreed at least two preset functions, wherein the preset functions are all first preset integer element functions, and the sum of all preset functions is the sum of each independent variable times a second preset constant, and the second preset constant is non-zero; The sum of all function values divided by the agreed second preset constant is equal to the sum of the gradient component values held by all users. The operator trains the neural network and updates the distributed neural network based on the sum of the gradient component values.
[0032] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the embodiments of the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.
[0033] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the aforementioned disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.
Claims
1. A distributed neural network training method, characterized in that: Applied to a system consisting of an operator of a neural network and multiple users holding distributed neural networks, the method comprises: as a user, Use private training data to train the distributed neural network to obtain gradient component values; Randomly split the gradient component value into agreed first preset integer parts; Substituting the split gradient component values into at least two agreed preset functions respectively to obtain at least two substituted function values, wherein the preset functions are all first preset integer-valued functions, and the sum of all the preset functions is the sum of each independent variable times a second preset constant, and the second preset constant is non-zero; All function values are sent to the operator so that the operator can obtain the sum of the gradient component values of all users by dividing the sum of all function values of all users by the agreed second preset constant. The operator trains the neural network and updates the distributed neural network based on the sum of the gradient component values.
2. A distributed neural network training method, characterized in that: Applied to a system consisting of an operator of a neural network and multiple users holding distributed neural networks, the method comprises: as the operator, Receive all function values sent by all users, wherein the function values are obtained by each user randomly splitting the gradient component values obtained by training using private training data into agreed first preset integer parts, and substituting the split gradient component values into agreed at least two preset functions, wherein the preset functions are all first preset integer element functions, and the sum of all preset functions is the sum of each independent variable times a second preset constant, and the second preset constant is non-zero; The sum of all function values divided by the agreed second preset constant is equal to the sum of the gradient component values held by all users. The operator trains the neural network and updates the distributed neural network based on the sum of the gradient component values.
3. A distributed neural network training method, characterized in that: Applied to a system consisting of an operator of a neural network and multiple users holding distributed neural networks, the method comprises: Each user uses private training data to train the distributed neural network to obtain the gradient component value; Randomly split the gradient component value into agreed first preset integer parts; Substituting the split gradient component values into at least two agreed preset functions respectively to obtain at least two substituted function values, wherein the preset functions are all first preset integer-valued functions, and the sum of all the preset functions is the sum of each independent variable times a second preset constant, and the second preset constant is non-zero; The user sends all the function values to the operator so that the operator can calculate the sum of all the function values; The operator calculates that the sum of all function values divided by the agreed second preset constant is equal to the sum of the gradient component values held by all users, and the operator trains the neural network and updates the distributed neural network based on the sum of the gradient component values.
4. The distributed neural network training method according to any one of claims 1 to 3, characterized in that: There are m preset functions, and the jth preset function , where , is the second preset constant.
5. The distributed neural network training method according to any one of claims 1 to 3, characterized in that: There are two preset functions. The first preset function is , the second preset function is , where p+1 is the second preset constant.
6. The distributed neural network training method according to any one of claims 1 to 3, characterized in that: The first preset integer is not less than 5.
7. A distributed neural network training system, characterized in that: The system includes an operator and multiple users, the operator operates the neural network, the user holds a distributed neural network of the neural network, the user applies the method described in any one of claims 1 and 4-6, and / or the operator applies the method described in any one of claims 2 and 4-6, and / or the system applies the method described in any one of claims 3 and 4-6.