A federated learning method, device, system, storage medium and electronic equipment

Abstract

The application discloses a federated learning method, device, system, storage medium and electronic equipment. The method comprises the following steps: in each global iteration process, receiving a modified gradient item in the current global iteration process; in each local iteration process in the current global iteration process, determining a local gradient item of the machine learning model in the current local iteration; correcting the local gradient item of the current local iteration based on the modified gradient item, updating the model parameter of the machine learning model based on the corrected target gradient item, and executing the next local iteration process based on the updated model parameter; and in the case that the local iteration process is completed, sending the model parameter change in the current global iteration process to the center server node. The embodiment of the application avoids the case that each computing node falls into a local optimal value in the local training process, and improves the generalization performance of the machine learning model obtained by training.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of federated learning, and in particular to a federated learning method, device, system, storage medium and electronic equipment. BACKGROUND

[0002] As a distributed machine learning framework, federated learning cooperatively trains a same model by multiple computing nodes. Since only model parameters are exchanged between the computing nodes and the center server node in the training process without exchanging original data, this training method can avoid sharing of original data of the computing nodes.

[0003] In the implementation of the present application, it is found that at least the following technical problems exist in the prior art: Due to the heterogeneity of local training data of the computing nodes, the optimal parameters of the computing nodes are often different, and the mean of these optimal parameters is often not the global optimal parameter. The difference in model parameters obtained by different computing nodes in local iteration is not conducive to the overall training of the model. SUMMARY

[0004] The present application provides a federated learning method, device, system, storage medium and electronic equipment to reduce the situation that the computing nodes fall into local optimal parameters in the training process.

[0005] According to an aspect of the present application, a federated learning method is provided, applied to a computing node, and the method comprises:

[0006] In each global iteration process, a correction gradient term in the current global iteration process is received;

[0007] In each local iteration process in the current global iteration process, a local gradient term of the machine learning model in the current local iteration is determined;

[0008] The local gradient term of the current local iteration is corrected based on the correction gradient term, the model parameter of the machine learning model is updated based on the corrected target gradient term, and the next local iteration process is performed based on the updated model parameter;

[0009] In the case of completing the local iteration process, the model parameter change in the current global iteration process is sent to the center server node, wherein the center server node determines the correction gradient term required for the next global iteration process based on the model parameter change in the current global iteration process.

[0010] According to another aspect of the present application, a federated learning method is provided, applied to a center server node, and the method comprises:

[0011] receive the model parameter changes sent by each computing node in any global iteration process;

[0012] update the model parameters based on the model parameter changes sent by each computing node, and determine the correction gradient term of the next global iteration process;

[0013] send the updated model parameters and the correction gradient term of the next global iteration process to each computing node, wherein each computing node performs the next global iteration process.

[0014] According to another aspect of the present application, a federated learning device integrated in a computing node device is provided, which comprises:

[0015] a global information acquisition module configured to receive the correction gradient term in the current global iteration process in any global iteration process;

[0016] a correction gradient term determination module configured to determine the local gradient term of the machine learning model in the current local iteration in any local iteration process in the current global iteration process;

[0017] a local parameter update module configured to correct the local gradient term of the current local iteration based on the correction gradient term, update the model parameters of the machine learning model based on the corrected target gradient term, and perform the next local iteration process based on the updated model parameters;

[0018] an information sending module configured to send the model parameter changes in the current global iteration process to a center server node in the case of completing the local iteration process, wherein the center server node determines the correction gradient term required for the next global iteration process based on the model parameter changes in the current global iteration process.

[0019] According to another aspect of the present application, a federated learning device integrated in a center server node device is provided, which comprises:

[0020] a local information receiving module configured to receive the model parameter changes sent by each computing node in any global iteration process;

[0021] a global update module configured to update the model parameters based on the model parameter changes sent by each computing node, and determine the correction gradient term of the next global iteration process;

[0022] an information sending module configured to send the updated model parameters and the correction gradient term of the next global iteration process to each computing node, wherein each computing node performs the next global iteration process.

[0023] According to another aspect of the present application, there is provided a federated learning system comprising a center server node and a plurality of computing nodes, wherein,

[0024] the center server node issues a correction gradient term, a global model parameter and a momentum parameter in a current global iteration process to each computing node;

[0025] the computing node receives the correction gradient term, the global model parameter and the momentum parameter, determines a local gradient term of the machine learning model in a current local iteration in a current global iteration process, corrects the local gradient term in the current local iteration based on the correction gradient term, updates a model parameter of the machine learning model based on the corrected target gradient term, and performs a next local iteration process based on the updated model parameter; and in the case of completing the local iteration process, sends a model parameter change and a local momentum parameter in the current global iteration process to the center server node;

[0026] the center server node determines a correction gradient term, a global model parameter and a global momentum parameter required for a next global iteration process based on the model parameter change and the local momentum parameter in the current global iteration process, and issues the correction gradient term, the global model parameter and the global momentum parameter required for the next global iteration process to each computing node, until the global iteration is completed to obtain a trained machine learning model.

[0027] According to another aspect of the present application, there is provided an electronic device, comprising:

[0028] at least one processor; and

[0029] a memory in communication connection with the at least one processor; wherein,

[0030] the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the federated learning method according to any one of the embodiments of the present application.

[0031] According to another aspect of the present application, there is provided a computer readable storage medium storing computer instructions for enabling a processor to implement the federated learning method according to any one of the embodiments of the present application when executed by the processor.

[0032] The technical scheme of the embodiment is characterized in that, in the process of local partial iteration performed by the computing nodes, the local gradient term in each partial iteration process is corrected by the correction gradient term issued by the central server node, the difference in model parameters caused by the difference in local gradient terms obtained on different computing nodes is reduced, the situation that each computing node falls into a local partial optimal value in the local training process is avoided, and the generalization performance of the machine learning model obtained by training is improved.

[0033] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0035] Figure 1 is a flowchart of a federated learning method provided by an embodiment of the present application;

[0036] Figure 2 is a flowchart of a federated learning method provided by an embodiment of the present application;

[0037] Figure 3 is a structural schematic diagram of a federated learning device provided by the third embodiment of the present application;

[0038] Figure 4 is a structural schematic diagram of a federated learning device provided by an embodiment of the present application;

[0039] Figure 5 is a structural schematic diagram of a federated learning system provided by an embodiment of the present application;

[0040] Figure 6 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0041] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings of the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should be within the scope of protection of the present application.

[0042] It should be noted that the terms "first", "second", and the like in the description and in the claims of the present application and in the above-described drawings are intended to distinguish similar objects and not necessarily describe a particular sequential or chronological order. It should be understood that the data thus used can be interchanged, where appropriate, so that the embodiments of the application described herein can be practiced in other than the illustrated or described order. Furthermore, the terms "comprise" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, processes, methods, systems, products, or devices that include a list of steps or units not necessarily limited to those clearly listed, but can include other steps or units not clearly listed or inherent to such processes, methods, products, or devices.

[0043] The federated learning process includes: 1. The computing node uses its own data to train the distributed model, and uploads the trained model to the center server node. 2. The center server node aggregates the uploaded model, and then distributes the same global model to multiple computing nodes. These two steps are iterated until the model converges. Among them, the federated learning system composed of multiple computing nodes and the center server node, the center server node can be configured in the center server device, and the computing node is configured in the computing node device, wherein different computing nodes can be configured in different computing node devices, or two or more computing nodes can be configured in the same computing node device, which is not limited here.

[0044] In this embodiment, the computing node and the server node jointly train to obtain a machine learning model. Here, the application scenario of federated learning is not limited, that is, the type of machine learning model and the function of the trained machine learning model are not limited. In some embodiments, the machine learning model can include but is not limited to a neural network model, a logistic regression model, etc., wherein the neural network model can include but is not limited to a convolutional neural network model CNN, a recurrent neural network model RNN, a long short-term memory network model LSTM, a residual network model ResNet50, etc. The federated learning of the machine learning model can be applied to image classification models, image segmentation models, image feature extraction models, image compression models, image enhancement models, image denoising models, image label generation models, text classification models, text translation models, text summary extraction models, text prediction models, keyword conversion models, text semantic analysis models, speech recognition models, audio denoising models, audio synthesis models, audio equalizer conversion models, weather prediction models, commodity recommendation models, article recommendation networks, action recognition models, face recognition models, facial expression recognition models, etc. The above application scenarios are only exemplary, and the application of the generation method of the neural model is not limited in this application.

[0045] In federated learning of a machine learning model, a computing node locally iteratively trains the machine learning model. Correspondingly, the computing node is pre-configured with sample data, and iteratively trains the machine learning model based on the sample data, where the training manner of the machine learning model is not limited, for example, can be supervised training, unsupervised training, etc., which can train the machine learning model and update network parameters.

[0046] The sample data can be image data, and the prediction result of the machine learning model is an image processing result; or the sample data is text data, and the prediction result of the machine learning model is a text processing result; or the sample data is audio data, and the prediction result of the machine learning model is an audio processing result.

[0047] For example, if the sample data is image data, the machine learning model can be an image classification model, and the prediction result output by the machine learning model can be an image classification result; or the machine learning model can be an image segmentation model, and the prediction result can be an image segmentation result; or the machine learning model can be an image feature extraction model, and the prediction result can be an image feature extraction result; or the machine learning model can be an image compression model, and the prediction result can be an image compression result; or the machine learning model can be an image enhancement model, and the prediction result can be an image enhancement result; or the machine learning model can be an image denoising model, and the prediction result can be an image denoising result; or the machine learning model can be an image label generation model, and the prediction result can be an image label, etc. If the sample data is text data, the machine learning model can be a text classification model, and the prediction result output by the machine learning model can be a text classification result; or the machine learning model can be a text prediction model, and the prediction result can be a text prediction result; or the machine learning model can be a text abstract extraction model, and the prediction result can be a text abstract extraction result; or the machine learning model can be a text translation model, and the prediction result can be a text translation result; or the machine learning model can be a keyword conversion model, and the prediction result can be a keyword conversion result; or the machine learning model can be a text semantic analysis model, and the prediction result can be a text semantic analysis result, etc. If the sample data is audio data, the machine learning model can be a speech recognition model, and the prediction result output by the machine learning model can be a speech recognition result; or the machine learning model can be an audio denoising model, and the prediction result can be an audio denoising result; or the machine learning model can be an audio synthesis model, and the prediction result can be an audio synthesis result; or the machine learning model can be an audio equalizer conversion model, and the prediction result can be an audio equalizer conversion result, etc.

[0048] The federal learning process includes multiple global iteration processes. In each global iteration process, the center server node sends model parameters of a machine learning model to each computing node. The computing node updates the model parameters by performing multiple local iterations locally, and uploads the trained model to the center server node. The center server node aggregates the uploaded models, completes one global iteration, and distributes the aggregated model parameters to multiple computing nodes as a global model for the next global iteration process.

[0049] Due to different data distributions of local training data on different computing nodes, that is, the heterogeneity of training data, the model parameters trained on different computing nodes are different, and there is a risk of falling into a local optimal value, which is not conducive to the overall model parameter update of the federal learning. To solve the above technical problems, the embodiment of the present application provides a federal learning method, Figure 1 is a flowchart of a federal learning method provided by the embodiment of the present application. The embodiment can be applied to the case of multiple local iterations of a machine learning model on a computing node. The method can be executed by a federal learning device, which can be realized in the form of hardware and / or software. The federal learning device can be configured in a computing node device, which can be an electronic device such as a computer, a mobile phone, a PC, etc. As shown in the figure, the method comprises the following steps. Figure 1

[0050] S110, in any global iteration process, receiving a correction gradient term in the current global iteration process.

[0051] S120, in any local iteration process in the current global iteration process, determining a local gradient term of the machine learning model in the current local iteration.

[0052] S130, correcting the local gradient term of the current local iteration based on the correction gradient term, updating the model parameters of the machine learning model based on the corrected target gradient term, and performing the next local iteration process based on the updated model parameters.

[0053] S140, in the case of completing the local iteration process, sending a model parameter change local momentum parameter in the current global iteration process to the center server node, wherein the center server node determines a correction gradient term required for the next global iteration process based on the model parameter change in the current global iteration process.

[0054] In any global iteration process, the computing node receives the model parameters distributed by the center server node, and optimizes the model parameters by multiple local iterations based on the local training data until the local iteration process is completed. ​

[0055] In the embodiment, the computing node also receives a correction gradient term issued by the center server node, the correction gradient term is determined by the center server node and is synchronously sent to all computing nodes in the current global iteration process, wherein the correction gradient term in the current global iteration process is obtained based on the model parameter change in the last global iteration process, and the correction gradient term in the first global iteration process can be set to zero.

[0056] In the current global iteration process, each computing node locally performs multiple local iteration processes, and in each iteration process, the model parameter can be updated by the gradient descent method. In the embodiment, in the process of updating the model parameter based on the gradient descent method, each computing node corrects the local gradient term of each local iteration process by the correction gradient term, and updates the model parameter based on the corrected gradient, thereby reducing the difference between the model parameters obtained by different computing nodes.

[0057] Optionally, the receiving the correction gradient term in the current global iteration process comprises: receiving the correction gradient term, the initial model parameter and the initial momentum parameter in the current global iteration process issued by the center server node. The correction gradient term, the initial model parameter and the initial momentum parameter in the current global iteration process are determined based on the iteration result of the last global iteration process. It should be noted that in the first global training process, the correction gradient term, the initial model parameter and the initial momentum parameter are initialization data, for example, the correction gradient term can be zero, i.e. g 0,α = 0, and the initial momentum parameter can be a preset value, i.e. The initial model parameter can be an initial value x0, which can be preset, for example, 0 or 0.5, etc., or can be obtained by an initialization operation on the model parameter, which is not limited.

[0058] The computing node locally performs k times of local iteration, wherein in each global iteration process, the number of local iterations can be the same or different, which is not limited. Before performing the local iteration, the computing node performs initialization setting based on the correction gradient term, the initial model parameter and the initial momentum parameter in the current global iteration process issued by the center server node, as the initial parameters of the local iteration.

[0059] Based on the initial model parameter, the corresponding parameter in the machine learning model is set, forming a machine learning model to be trained, and the machine learning model to be trained is subjected to multiple local iterations. In each local iteration process, the local momentum parameter in the current local iteration is determined, and the local gradient term is determined based on the local momentum parameter, which is used to update the model parameter in the current local iteration process.

[0060] Optionally, the determining the local gradient term of the machine learning model in the current local iteration comprises: determining a local momentum parameter of the machine learning model in the current local iteration, wherein the local momentum parameter comprises a first-order momentum and a second-order momentum; and determining the local gradient term of the machine learning model in the current local iteration based on the first-order momentum and the second-order momentum.

[0061] The determining the local momentum parameter of the machine learning model in the current local iteration comprises: determining a random gradient of the machine learning model in the current local iteration; determining a first-order momentum of the machine learning model in the current local iteration based on the random gradient and a first-order momentum of the machine learning model in a previous local iteration; and determining a second-order momentum of the machine learning model in the current local iteration based on the random gradient and a second-order momentum of the machine learning model in the previous local iteration. In each local iteration, a prediction result of the machine learning model to be trained in the current local iteration is input based on the local training data, and a loss function is determined based on the prediction result to determine the random gradient of the machine learning model in the current local iteration. The loss function can be pre-set and is not limited herein and can be adaptively changed according to different application scenarios. For example, the loss function can be denoted as f(x,ξ), where x represents a feature of a training sample data, and ξ represents a label of the training sample data. i (x t，k-1，i ,ξ t，k，i )。 t，k，i For any one training sample data, it can be denoted as ξ t，k，i =(ξ t，k , ifeature ,ξ t，k , ilabel )。 t，k，ifeature represents a feature of a training sample data participating in the kth iteration training in the tth round of the i th computing node device. x t，k，ilabel represents a label of a training sample data participating in the kth iteration training in the tth round of the i th computing node device. x t，k-1，i represents a model parameter before the current local iteration training. The random gradient where t represents a global iteration number, k represents a local iteration number, and i represents a computing node identifier. Specifically, a derivative of a model parameter in the machine learning model can be determined based on the loss function, and the derivative can be a derivative of different orders corresponding to different network layers. For example, when the machine learning model comprises three network layers, the derivative of the model parameter in the machine learning model comprises a first-order derivative, a second-order derivative, and a third-order derivative of the loss function with respect to the model parameter. The derivatives of different orders corresponding to the respective model parameters are combined to obtain the random gradient of the machine learning model in the current iteration, for example, the derivatives of different orders corresponding to the respective model parameters are combined in the form of a matrix or a vector to obtain the random gradient of the machine learning model in the current iteration.

[0062] The computing node is preset with hyperparameters, wherein the hyperparameters can include a first-order momentum-related hyperparameter β1 and a second-order momentum-related hyperparameter β2. Correspondingly, the first-order momentum of the current local iteration is calculated based on the random gradient, the first-order momentum of the last local iteration, and the first-order momentum-related hyperparameter β1. Specifically, the first-order momentum can be calculated based on the following formula:

[0063] m t,k,i = β1m t,k-1,i -(1-β1)g t,k,i

[0064] wherein m t,k-1,i is the first-order momentum corresponding to the k-1th local iteration in the tth global iteration process, i.e., the first-order momentum of the last local iteration, m t,k,i is the first-order momentum corresponding to the kth local iteration in the tth global iteration process, i.e., the first-order momentum of the current local iteration. g t,k,i is the random gradient in the current local iteration.

[0065] In some embodiments, the second-order momentum of the current local iteration is determined based on the random gradient, the second-order momentum of the last local iteration, and the second-order momentum-related hyperparameter β2. Specifically, the second-order momentum can be calculated based on the following formula:

[0066]

[0067] wherein v t,k-1,i is the second-order momentum corresponding to the k-1th local iteration in the tth global iteration process, i.e., the second-order momentum of the last local iteration, v t,k,i is the second-order momentum corresponding to the kth local iteration in the tth global iteration process, i.e., the second-order momentum of the current local iteration.

[0068] In some embodiments, the second-order momentum applied in the current local iteration is the maximum value of the second-order momentum calculated in the current iteration and the second-order momentum calculated in the last local iteration. Correspondingly, the second-order momentum of the current local iteration is determined based on the random gradient and the second-order momentum of the last local iteration, including: determining the current second-order momentum of the current local iteration based on the random gradient and the second-order momentum of the last local iteration; determining the second-order momentum of the current local iteration as the maximum value of the second-order momentum of the last local iteration and the current second-order momentum. Exemplarily, the target second-order momentum in the current local iteration is It should be noted that the dimensions of the first-order momentum, the second-order momentum, and the target second-order momentum are the same as the dimension of the random gradient, and the dimension can be represented as .

[0069] The local gradient term of the current local iteration is determined based on the first-order momentum and the second-order momentum. Specifically, the local gradient term of the current local iteration can be determined based on a ratio of the first-order momentum to the second-order momentum. For example, the local gradient term can be calculated based on the following formula: where the ratio of the first-order momentum to the target second-order momentum is used as the local gradient term.

[0070] On the basis of the above embodiments, in each local iteration process, the local gradient term of the current local iteration is modified by the correction gradient term to obtain a target gradient term, which is used to update the model parameters.

[0071] Optionally, the target gradient term of the current local iteration is determined based on the local gradient term and the correction gradient term, including: the local gradient term and the correction gradient term are weighted and fused based on a preset weight to obtain the target gradient term of the current local iteration, wherein the weight of the correction gradient term is not zero. For example, the weight of the local gradient term and the weight of the correction gradient term are set, and the sum of the weight of the local gradient term and the weight of the correction gradient term is 1. Accordingly, the target gradient term can be represented as where g t,α is the correction gradient term in the current global iteration process, a is the weight of the local gradient term, and 1-a is the weight of the correction gradient term. It should be noted that the weight of the correction gradient term is not zero, i.e. a≠1. In some embodiments, the weight a is a correction weight, which can be set in advance.

[0072] On the basis of the above embodiments, the model parameters in the current local iteration process are updated based on the target gradient term. For example, the update of the model parameters can be realized by the following formula: where x t,k-1,i is the model parameter before the current local iteration, x t,k,i is the updated model parameter after the current local iteration, and η l is the local learning rate of the computing node.

[0073] The computing node iteratively performs the above local iteration process until K local iterations are completed, and obtains the machine learning model updated by the computing node in the current global iteration process. The computing node locally completes the current global iteration part and sends the iteration result to the center server node. The iteration result can include the model parameter after iteration, or the model parameter change before and after iteration. In some embodiments, the iteration result includes the model parameter change x t,0,i -x t,K,i , where x t,K,i is the model parameter after k local iterations, and x t,0,i is the model parameter after k local iterations.

[0074] Each computing node sends the iteration result to the center server node, so that the center server node obtains the global model parameter of the current global iteration process and the correction gradient item in the next global iteration process based on the iteration result of each computing node, facilitating the next global iteration process. Wherein, the next global iteration process is determined based on the global model parameter obtained in the current global iteration process and the global model parameter before the current global iteration process.

[0075] In some embodiments, the iteration result can also include the local momentum parameter obtained in the last local iteration process. Correspondingly, the local momentum parameter is sent to the center server node, so that the center server node obtains the initial momentum parameter in the next global iteration process based on the local momentum parameter of each computing node, and performs the next global iteration.

[0076] The technical scheme of the embodiment reduces the difference between the model parameters obtained on different computing nodes caused by the difference between the local gradient items in each local iteration process by correcting the local gradient item in each local iteration process through the correction gradient item issued by the center server node in the process of local iteration on the computing node, avoids the situation that each computing node falls into the local optimal value in the local training process, and improves the generalization performance of the machine learning model obtained by training.

[0077] Figure 2 is a flowchart of a federated learning method provided by the embodiment of the application. The embodiment can be applied to the case where the center server node processes the iteration result of each computing node. The method can be executed by a federated learning device, which can be realized in the form of hardware and / or software. The federated learning device can be configured in a center server node device, which can be an electronic device such as a computer, a server, etc. As shown in the figure, the method comprises the following steps. Figure 2

[0078] S210, in any global iteration process, receiving the model parameter change sent by each computing node.

[0079] S220, updating the model parameter globally based on the model parameter change sent by each computing node, and determining the correction gradient item in the next global iteration process.

[0080] S230, sending the updated model parameter and the correction gradient item in the next global iteration process to each computing node, wherein each computing node performs the next global iteration process.

[0081] ​In this embodiment, the center server node sends the corrected gradient term, the initial model parameter of the current global iteration process and the initial momentum parameter to the computing nodes participating in the current global iteration in the process of each global iteration, so as to facilitate the local model training of each computing node based on the corrected gradient term, the initial model parameter of the current global iteration process and the initial momentum parameter, and to obtain the iteration result of the current global iteration respectively. In this embodiment, the iteration result includes the model parameter change.

[0082] Based on the model parameter change of each computing node, the center server node can obtain the mean value of the model parameter change, and update the global model parameter of the current iteration process based on the global learning rate and the mean value of the model parameter change, to realize the update of the global model parameter. For example, the update can be realized based on the following formula: Wherein, x t,0,i -x t,K,i is the model parameter change sent by the i th computing node, S is the number of computing nodes participating in the model training in the current global iteration process, η g is the global learning rate, x t is the global model parameter of the current iteration process, x t+1 is the updated global model parameter, i.e. the global model parameter of the next global iteration.

[0083] Further, based on the global model parameter x t before updating and the global model parameter x t+1 after updating, the corrected gradient term in the next global iteration process is determined. Specifically, the corrected gradient term in the next global iteration process can be calculated by the parameter difference between the global model parameter x t before updating and the global model parameter x t+1 after updating, the local iteration number K, the local learning rate and the global learning rate. For example, the update can be realized based on the following calculation process:

[0084] In some embodiments, the iteration result of each computing node further includes the local dynamic parameter, i.e. the local dynamic parameter obtained after K times of local iteration. Correspondingly, the center server node is further configured to determine the global dynamic parameter based on the local dynamic parameter of each computing node, and the global dynamic parameter is used as the initial dynamic parameter of each computing node in the next global iteration process. For example, the global dynamic parameter can be determined by the mean value of the local dynamic parameter of each computing node, for example Wherein, is the local dynamic parameter sent by the i th computing node, i.e. the target second order momentum.

[0085] The center server node sends the updated model parameters and the correction gradient item of the next global iteration process to each computing node, wherein the updated model parameters are used as the initial model parameters of each computing node for the next global iteration. In some embodiments, the global dynamic parameters can also be sent to each computing node.

[0086] It should be noted that in each global iteration process, only the local computing nodes perform local training, the number of computing nodes performing local training in different global iteration processes is the same, and the computing nodes performing local training can be different. The number of computing nodes or the proportion of computing nodes is set in advance, or random extraction is performed among all the computing nodes to determine the computing nodes performing local training in each global iteration process.

[0087] Correspondingly, the model parameters and the correction gradient item of the next global iteration process are sent to the computing nodes in each global iteration, including: extracting a preset proportion of computing nodes as the computing nodes of the next global iteration; and sending the updated model parameters and the correction gradient item of the next global iteration process to the extracted computing nodes. In each global iteration process, the local training is performed by extracting the local computing nodes, which reduces the number of computing nodes participating in the global iteration, reduces the communication cost and the computing cost, and improves the diversity of the training data applied in different global iteration processes and the generalization of the trained model.

[0088] The technical scheme provided in this embodiment processes the iteration results of each computing node at the center server node to obtain global model parameters, and determines the correction gradient item in the next global iteration process based on the global model parameters. By sending the correction gradient item to each computing node, the local gradient item of the computing node in the local iteration process is corrected and processed in the local training process of each computing node, the difference between the local gradient items of different computing nodes is reduced, and then the difference between the model parameters obtained by iteration is reduced, the accuracy of the global model parameters is improved, and the situation that each computing node falls into a local optimum is avoided.

[0089] Figure 3 is a structural schematic diagram of a federated learning device provided in Embodiment Three of the present application, which can be configured in a computing node device. As shown in Figure 3 , the device comprises:

[0090] The global information acquisition module 310 is configured to receive the correction gradient item in the current global iteration process in any global iteration process.

[0091] The correction gradient item determination module 320 is configured to determine the local gradient item of the machine learning model in the current local iteration in any local iteration process in the current global iteration process.

[0092] a local parameter updating module 330, configured to correct a local gradient term of a current local iteration based on the correction gradient term, update model parameters of the machine learning model based on the corrected target gradient term, and perform a next local iteration process based on the updated model parameters;

[0093] an information sending module 340, configured to send model parameter changes in the current global iteration process to a center server node in a case where the local iteration process is completed, where the center server node determines a correction gradient term and global model parameters required for a next global iteration process based on the model parameter changes in the current global iteration process.

[0094] In the above embodiment, optionally, the correction gradient term determination module 320 includes:

[0095] a local momentum parameter determination unit, configured to determine a local momentum parameter of the machine learning model in the current local iteration, where the local momentum parameter includes a first-order momentum and a second-order momentum;

[0096] a local gradient term determination unit, configured to determine the local gradient term of the current local iteration based on the first-order momentum and the second-order momentum.

[0097] Optionally, the local momentum parameter determination unit includes:

[0098] a random gradient determination sub-unit, configured to determine a random gradient of the machine learning model in the current local iteration;

[0099] a first-order momentum determination sub-unit, configured to determine a first-order momentum of the current local iteration based on the random gradient and a first-order momentum of a previous local iteration;

[0100] a second-order momentum determination sub-unit, configured to determine a second-order momentum of the current local iteration based on the random gradient and a second-order momentum of the previous local iteration.

[0101] Optionally, the second-order momentum determination sub-unit is configured to:

[0102] determine a current second-order momentum of the current local iteration based on the random gradient and the second-order momentum of the previous local iteration;

[0103] determine the second-order momentum of the current local iteration as a maximum value between the second-order momentum of the previous local iteration and the current second-order momentum.

[0104] In the above embodiment, optionally, the local parameter updating module 330 is configured to:

[0105] The local gradient term and the correction gradient term are fused by preset weights to obtain a target gradient term of the current local iteration, wherein the weight of the correction gradient term is not zero.

[0106] On the basis of the above-mentioned embodiments, the global information acquisition module 310 is used to:

[0107] The correction gradient term, the initial model parameter and the initial momentum parameter of the current global iteration process are received.

[0108] The federal learning device provided in the embodiments of the present application can execute the federal learning method provided in any of the embodiments of the present application, and has the corresponding function modules and beneficial effects of the execution method.

[0109] Figure 4 is a structural schematic diagram of a federal learning device provided in the embodiment of the present application, which can be configured in a center server node device. Figure 4 As shown in the figure, the device comprises:

[0110] The local information receiving module 410 is used to receive the model parameter changes sent by each computing node in each global iteration process.

[0111] The global updating module 420 is used to globally update the model parameters based on the model parameter changes sent by each computing node, and determine the correction gradient term of the next global iteration process.

[0112] The information sending module 430 is used to send the updated model parameters and the correction gradient term of the next global iteration process to each computing node, wherein each computing node performs the next global iteration process.

[0113] On the basis of the above-mentioned embodiments, the information sending module 430 is used to:

[0114] Extract a preset proportion of computing nodes as the computing nodes of the next global iteration;

[0115] Send the updated model parameters and the correction gradient term of the next global iteration process to the extracted computing nodes.

[0116] The federal learning device provided in the embodiments of the present application can execute the federal learning method provided in any of the embodiments of the present application, and has the corresponding function modules and beneficial effects of the execution method.

[0117] Figure 5 is a structural schematic diagram of a federal learning system provided in the third embodiment of the present application. As shown in the figure, Figure 5 The system comprises a center server node 510 and a plurality of computing nodes 520,

[0118] The center server node 510 distributes the correction gradient item, the global model parameter and the momentum parameter in the current global iteration process to each computing node;

[0119] The computing node 520 receives the correction gradient item, the global model parameter and the momentum parameter, determines the local gradient item of the machine learning model in the current local iteration in each local iteration process in the current global iteration process, corrects the local gradient item of the current local iteration based on the correction gradient item, updates the model parameter of the machine learning model based on the corrected target gradient item, and performs the next local iteration process based on the updated model parameter; in the case of completing the local iteration process, sends the model parameter change and the local momentum parameter in the current global iteration process to the center server node 510;

[0120] The center server node 510 determines the correction gradient item, the global model parameter and the global momentum parameter required for the next global iteration process based on the model parameter change and the local momentum parameter in the current global iteration process, and distributes the correction gradient item, the global model parameter and the global momentum parameter required for the next global iteration process to each computing node until the global iteration is completed, and the trained machine learning model is obtained.

[0121] The center server node sets the initialization model parameter x0, the hyperparameters: the local learning rate η l , the global learning rate η g , the correction weight α, the correction gradient g 0,α = 0, the number of selected devices S in each round, the hyperparameters β1 and β2 before the first global iteration. The initialized parameters β1 and β2, The global iteration process is a synchronization process of the center server node and the computing node. The local iteration process is a process of model training locally by each computing node. In the global iteration process, the center server node selects some computing nodes to perform local calculation, the center server node processes the collected model parameter change and the second-order momentum, and updates the stored information with the averaged model parameter and other states. For example, by the second-order momentum of the next global iteration is obtained, by the global model parameter of the next global iteration is obtained, by the correction gradient item of the next global iteration is obtained. The center server node then distributes the model parameter and other states to the computing node to facilitate the next round of calculation.

[0122] During local iteration, the selected S computation nodes compute the local iteration in parallel. The initial values ​​are the model parameters, second-order momentum, and corrected gradient terms distributed by the central server node in the previous round. Afterward, the computation nodes use local data to compute stochastic gradients. In the local iteration step, the AMSGrad method is used to update the model parameters. The gradient g of the model parameters is obtained through backpropagation of the loss function. t,k,i , where t represents the number of global synchronization cycles, k represents the local iteration round, and i represents the i-th computation node. From the gradient g t,k,i The state m in the previous local iteration round t,k-1,i and v t,k-1,i Calculate the current momentum m t,k,i With second momentum v t,k,i Here, both the momentum and second-order momentum dimensions are the same as the gradient dimension, which belongs to... Choose the current second-order momentum v. t,k,i The second-order momentum used in the previous iteration The larger one is used as the second-order momentum in the current iteration. This is determined by the current x... t,k-1,i Update the model parameters x when k is obtained t,k,i After K iterations, the compute nodes upload the current model parameters and other states to the central server node.

[0123] In this embodiment, the learning process satisfies (1) the continuity assumption, (2) the bounded variance assumption, (3) the bounded gradient assumption, and (4) the bounded device gradient difference assumption. When the global learning rate and the local learning rate satisfy... When we select and At that time, the convergence rate of our algorithm is:

[0124]

[0125] Where S is the number of selected worker nodes, K is the number of local loops, and T is the number of global loops.

[0126] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0127] like Figure 6 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0128] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0129] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as federated learning methods.

[0130] In some embodiments, the federated learning method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the federated learning method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to execute the federated learning method by any other suitable means (e.g., by means of firmware).

[0131] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (PLD), a computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0132] Computer programs used to implement the federated learning method of the present application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the computer program running on the processor implements the functions / operations specified in the flowcharts and / or block diagrams. The computer program can be executed entirely on a machine, partially on a machine, partially on a machine as a standalone software package and partially on a remote machine, or entirely on a remote machine or server.

[0133] The embodiments of the present application also provide a computer readable storage medium, which stores computer instructions for causing a processor to execute a federated learning method, the method comprising:

[0134] In any global iteration process, a correction gradient item in the current global iteration process is received;

[0135] In any local iteration process in the current global iteration process, a local momentum parameter of the machine learning model in the current local iteration is determined;

[0136] The model parameter of the machine learning model is updated based on the local momentum parameter of the current local iteration and the correction gradient item, and the next local iteration process is performed based on the updated model parameter;

[0137] In the case where the local iteration process is completed, the model parameter change and the local momentum parameter in the current global iteration process are sent to a center server node, wherein the center server node determines the correction gradient item required for the next global iteration process based on the model parameter change and the local momentum parameter in the current global iteration process.

[0138] And / or,

[0139] The computer instructions are for causing a processor to perform a federated learning method, the method comprising:

[0140] During each global iteration process, receiving model parameter changes sent by each computing node;

[0141] Based on the model parameter changes sent by each computing node, performing global update on the model parameters, and determining a correction gradient item of the next global iteration process;

[0142] Sending the updated model parameters and the correction gradient item of the next global iteration process to each computing node, wherein each computing node performs the next global iteration process.

[0143] In the context of the present application, a computer-readable storage medium can be a tangible medium which can contain or store the program for use by or in connection with an instruction execution system, apparatus, or device. The computer-readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the above. Alternatively, the computer-readable storage medium can be a machine-readable signal medium. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disc read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0144] To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0145] The systems and techniques described herein can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described herein, or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0146] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. A server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service.

[0147] It should be understood that the various forms of flow shown above can be re-ordered, added to, or deleted from without departing from the scope of the present disclosure. For example, the steps recited in the present disclosure can be performed in parallel, in series, or in a different order, and the present disclosure is not limited in this regard.

[0148] The specific embodiments described above are not intended to limit the scope of the present disclosure. Those skilled in the art will understand that various modifications, combinations, sub-combinations, and alternatives can be made to the specific embodiments without departing from the spirit and principles of the present disclosure. Any further modifications, equivalents, and / or alternatives come within the scope of the present disclosure as recited by the claims.

Claims

1. A federated learning method, characterized in that, Applied to a computing node, the method comprises: In any global iteration process, a modified gradient term in a current global iteration process is received, wherein the modified gradient term in the current global iteration process is based on a model parameter change of a previous global iteration process, and the model parameter change of the previous global iteration process is based on a difference between a global model parameter before updating and a global model parameter after updating in the previous global iteration process; in a first global iteration process, the modified gradient term is zero; In any local iteration process in the current global iteration process, a local gradient term of the machine learning model in the current local iteration is determined; The local gradient term of the current local iteration is modified based on the modified gradient term, and a model parameter of the machine learning model is updated based on the modified target gradient term, and a next local iteration process is performed based on the updated model parameter; the target gradient term is obtained by weighted fusion of the modified gradient term and the local gradient term; the weight of the modified gradient term is not zero; In the case where the local iteration process is completed, the model parameter change of the computing node is sent to a center server node, wherein the center server node globally updates the model parameter based on the model parameter changes sent by each computing node, obtains an updated global model parameter, and determines a modified gradient term required for a next global iteration process based on a difference between a global model parameter before updating and the updated global model parameter.

2. The method of claim 1, wherein, The determination of the local gradient term of the machine learning model in the current local iteration comprises: A local momentum parameter of the machine learning model in the current local iteration is determined, wherein the local momentum parameter comprises a first-order momentum and a second-order momentum; The local gradient term of the current local iteration is determined based on the first-order momentum and the second-order momentum.

3. The method of claim 2, wherein, The determination of the local momentum parameter of the machine learning model in the current local iteration comprises: A random gradient of the machine learning model in the current local iteration is determined; A first-order momentum of the current local iteration is determined based on the random gradient and a first-order momentum of a previous local iteration, and a second-order momentum of the current local iteration is determined based on the random gradient and a second-order momentum of the previous local iteration.

4. The method of claim 3, wherein, The determination of the second-order momentum of the current local iteration based on the random gradient and the second-order momentum of the previous local iteration comprises: A current second-order momentum of the current local iteration is determined based on the random gradient and the second-order momentum of the previous local iteration; The maximum value between the second-order momentum of the previous local iteration and the current second-order momentum is determined as the second-order momentum of the current local iteration.

5. The method of claim 1, wherein, The receiving of the modified gradient term in the current global iteration process comprises: The modified gradient term, initial model parameters and initial momentum parameters of the current global iteration process issued by the center server node are received.

6. A federated learning method, comprising: Applied to a center server node, the method comprises: In any global iteration process, model parameter changes sent by each computing node are received; updating the model parameters based on the model parameter changes sent by the computing nodes to obtain updated global model parameters, and determining a correction gradient item for a next global iteration process, wherein the correction gradient item for the next global iteration process is determined based on a difference between the global model parameters before updating and the updated global model parameters; sending the updated model parameters and the correction gradient item for the next global iteration process to the computing nodes, wherein each of the computing nodes performs local partial iteration, and the correction gradient item for the next global iteration process is used to correct a local gradient item for each partial iteration process.

7. The method of claim 6, wherein, The sending of the updated model parameters and the correction gradient item for the next global iteration process to the computing nodes comprises: extracting a preset proportion of the computing nodes as computing nodes for a next global iteration; sending the updated model parameters and the correction gradient item for the next global iteration process to the extracted computing nodes.

8. A federated learning apparatus, comprising: The device integrated in the computing node equipment comprises: a global information acquisition module configured to, in any global iteration process, receive a correction gradient item in a current global iteration process, wherein the correction gradient item in the current global iteration process is obtained based on model parameter changes in a previous global iteration process, and the model parameter changes in the previous global iteration process are based on a difference between global model parameters before updating and global model parameters after updating in the previous global iteration process; in a first global iteration process, the correction gradient item is zero; a correction gradient item determination module configured to, in any partial iteration process in the current global iteration process, determine a local gradient item of the machine learning model in a current partial iteration; a local parameter updating module configured to correct the local gradient item of the current partial iteration based on the correction gradient item, update model parameters of the machine learning model based on a target gradient item after correction, and perform a next partial iteration process based on the updated model parameters; the target gradient item is obtained by weighted fusion of the correction gradient item and the local gradient item; and the weight of the correction gradient item is not zero; an information sending module configured to, in a case where the partial iteration process is completed, send model parameter changes of the computing node to a central server node, wherein the central server node updates the model parameters based on the model parameter changes sent by the computing nodes to obtain updated global model parameters, and determines a correction gradient item required for a next global iteration process based on a difference between global model parameters before updating and the updated global model parameters.

9. A federated learning apparatus, comprising: The device integrated in the central server node equipment comprises: a local information receiving module configured to, in any global iteration process, receive model parameter changes sent by the computing nodes. The global updating module is configured to update the model parameters globally based on the model parameter changes sent by the computing nodes, to obtain updated global model parameters, and to determine a modified gradient item for a next global iteration process; wherein the modified gradient item for the next global iteration process is determined based on a difference between the global model parameters before updating and the updated global model parameters; The information sending module is configured to send the updated model parameters and the modified gradient item for the next global iteration process to the computing nodes, wherein the computing nodes perform local and partial iterations, and in the local and partial iteration process, the modified gradient item for the next global iteration process is used to modify a local gradient item for each partial iteration process.

10. A federated learning system, comprising: The system comprises a central server node and a plurality of computing nodes, wherein The central server node sends a modified gradient item in a current global iteration process, global model parameters, and momentum parameters to the computing nodes; wherein the modified gradient item in the current global iteration process is obtained based on model parameter changes in a previous global iteration process, the model parameter changes in the previous global iteration process are based on a difference between global model parameters before updating and global model parameters after updating in the previous global iteration process, and the modified gradient item in a first global iteration process is zero; The computing nodes receive the modified gradient item, the global model parameters, and the momentum parameters, determine a local gradient item for a current partial iteration in a current global iteration process, modify the local gradient item for the current partial iteration based on the modified gradient item, update model parameters of the machine learning model based on the modified target gradient item, and perform a next partial iteration process based on the updated model parameters; in the case where the partial iteration process is completed, the computing nodes send model parameter changes and local momentum parameters to the central server node; the target gradient item is obtained by weighted fusion of the modified gradient item and the local gradient item; and the weight of the modified gradient item is not zero; The central server node updates the model parameters globally based on the model parameter changes sent by the computing nodes, to obtain updated global model parameters, determines a modified gradient item, global model parameters, and global momentum parameters required for a next global iteration process based on a difference between the global model parameters before updating and the updated global model parameters and the local momentum parameters, and sends the modified gradient item, the global model parameters, and the global momentum parameters required for the next global iteration process to the computing nodes until the global iteration is completed, to obtain a trained machine learning model; wherein the modified gradient item for the next global iteration process is determined based on the difference between the global model parameters before updating and the updated global model parameters; The computing nodes perform local and partial iterations, and in the local and partial iteration process, the modified gradient item for the next global iteration process is used to modify a local gradient item for each partial iteration process.

11. An electronic device, comprising: The electronic device comprises: at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the federated learning method of any one of claims 1-5, and / or the federated learning method of any one of claims 6-7.

12. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions for causing the processor to implement the federated learning method of any one of claims 1-5, and / or the federated learning method of any one of claims 6-7 when executed.

Images