A momentum-based hierarchical compromise model federated learning method and device

Abstract

The application discloses a momentum-based hierarchical compromise model federated learning method, comprising the following steps: according to the group model and the global model issued by the corresponding edge node, updating the local momentum, the group momentum and the global momentum in the local device, obtaining the first local momentum, the first group momentum and the first global momentum, and training the local model according to the first local momentum; according to the first group momentum and the first global momentum, aggregating the second group momentum and the second global momentum in the edge node, training the group model according to the second group momentum, and aggregating the third global momentum in the center server according to the second global momentum, training the global model according to the third global momentum, and ending the iteration until the global model obtained in the latest iteration is determined to be converged. The application can improve the learning efficiency of the federated learning method.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of software engineering, and in particular to a momentum-based hierarchical trade-off model federated learning method and device. BACKGROUND

[0002] With the popularization of big data and machine learning, large model and other technologies, distributed learning in the massive data scenario has become the focus of research and industry. Federated learning is an encrypted distributed learning paradigm that allows multiple participants (such as user devices, edge devices, or cloud center servers) to jointly train a model without sharing raw data. The core idea of federated learning is to decentralize the model training process to local devices rather than centralizing data to a central server for processing. This allows raw data to remain local at all times, maximizing the protection of participants' privacy.

[0003] Although federated learning has significant advantages in protecting data privacy and efficiently utilizing distributed resources, it still faces many challenges in practical applications. Communication bottleneck between computing nodes and central nodes: increased communication overhead, due to different data distributions of each node, the required training time and training steps are also different. Some nodes may need more local training iterations to achieve similar model performance to other nodes, which leads to an imbalance in the frequency and amount of data uploaded by different nodes. Second, the trade-off between system model generalization ability and individualization: due to the large differences in data distribution of each node, a single global model may not be able to adapt well to the specific data characteristics of each node, resulting in limited generalization ability of the global model. Finally, low model convergence efficiency: in a heterogeneous data scenario, the gradients of each node may differ significantly, which leads to inconsistent directions of global gradient updates during aggregation, thus slowing down the convergence speed of the model, and even possibly causing the model to fall into a local optimum. The amount of data and computing power of different nodes may differ significantly, and stronger nodes may want to update the model faster, while weaker nodes may become a system bottleneck. This asymmetry further reduces the overall convergence efficiency of the model. SUMMARY

[0004] The present application is to overcome the defects of the prior art, and provides a momentum-based hierarchical trade-off model federated learning method and device, which can improve the learning efficiency of the federated learning method.

[0005] An embodiment of the present application provides a momentum-based hierarchical trade-off model federated learning method and device, comprising the following steps:

[0006] According to the group model and the global model issued by the corresponding edge node, the local momentum, the group momentum and the global momentum are updated in the local device, and the first local momentum, the first group momentum and the first global momentum are obtained, and the local model is trained according to the first local momentum; wherein one edge node corresponds to a plurality of local devices;

[0007] According to the first group momentum and the first global momentum, the second group momentum and the second global momentum are aggregated in the edge node, the group model is trained according to the second group momentum, and then the trained group model and the global model issued by the corresponding center server are issued to a plurality of local devices corresponding to the edge node; wherein one center server corresponds to a plurality of edge nodes;

[0008] According to the second global momentum, the third global momentum is aggregated in the center server, the global model is trained according to the third global momentum, and after the trained global model is issued to a plurality of edge nodes corresponding to the center server, the next round of iteration is started, and the iteration is ended until the global model obtained in the latest round of iteration is determined to be converged.

[0009] Further, according to the group model and the global model issued by the corresponding edge node, the local momentum, the group momentum and the global momentum are updated in the local device, and the first local momentum, the first group momentum and the first global momentum are obtained, and the local model is trained according to the first local momentum; wherein one edge node corresponds to a plurality of local devices;

[0010] According to the preset random subset and the preset local gradient calculated by the local model, the local momentum is updated, and the first local momentum is obtained; wherein the update formula of the local momentum is specifically:

[0011]

[0012] Wherein, is the first local momentum, is the local momentum, is a preset update coefficient, is the preset local gradient;

[0013] According to the group model and the global model issued by the corresponding edge node, the group momentum and the global momentum are updated respectively, and the first group momentum and the first global momentum are obtained.

[0014] Further, the local model is trained according to the first local momentum, specifically including:

[0015] According to the first local momentum, the local model is updated by momentum gradient descent method;

[0016] The updated local model is corrected based on the group model issued by the corresponding edge node to obtain the trained local model.

[0017] Preferably, the step of correcting the updated local model based on the group model issued by the corresponding edge node specifically includes:

[0018] Calculate the first square norm between the updated local model and the group model issued by the corresponding edge node;

[0019] The first square norm is used as a correction term to correct the updated local model; wherein the specific formula for the correction is:

[0020]

[0021] in, For the first The updated local model described in each iteration round For the first The group model issued by the corresponding edge node in each iteration round To preset the learning rate, This is a preset correction factor. Let be the first square norm.

[0022] Furthermore, the step of aggregating the second set of momentum and the second global momentum at the edge nodes based on the first set of momentum and the first global momentum specifically includes:

[0023] Based on the first set of momentum, the second set of momentum is obtained by aggregating them using a weighted average method; wherein, the aggregation weight of the first set of momentum is the ratio of the sample data volume of the dataset contained in the local device corresponding to the first set of momentum to the sample data volume of the dataset contained in the edge node;

[0024] Based on the first global momentum, the second global momentum is obtained by aggregating it using a weighted average method; wherein, the aggregation weight of the first global momentum is the ratio of the sample data volume of the dataset contained in the local device corresponding to the first global momentum to the sample data volume of the dataset contained in the edge node.

[0025] Furthermore, the step of using the second set of momentum training models specifically includes:

[0026] Based on the second set of momentum, the group model is updated using the momentum gradient descent method;

[0027] The updated group model is corrected based on the global model issued by the corresponding central server to obtain the trained group model.

[0028] Preferably, the updated group model is corrected according to the global model issued by the corresponding center server, and the correction specifically includes:

[0029] The second square norm between the updated group model and the global model issued by the corresponding center server is calculated.

[0030] The second square norm is taken as a correction term to correct the updated group model, and the formula of the correction is specifically:

[0031]

[0032] wherein, is the updated group model in the i-th iteration round, is the global model issued by the corresponding center server in the i-th iteration round, is the second group momentum, is a preset correction coefficient, is the second square norm. Further, the third global momentum is aggregated in the center server according to the second global momentum, and the global model is trained according to the third global momentum, and the aggregation specifically includes:

[0033] The third global momentum is aggregated by a weighted average method according to the second global momentum, and the aggregation weight of the second group momentum is the ratio of the sample data amount of the data set contained in the edge node to the sample data amount of the data set contained in the center server corresponding to the second group momentum.

[0034] The global model is trained by a momentum gradient descent method according to the third global momentum.

[0035] Further, the convergence condition of the global model is specifically:

[0036] The loss function value of the global model issued by the corresponding edge node is calculated according to the preset loss function in the local device.

[0037] The global loss value is obtained by aggregating the global model loss function values calculated by all the local devices.

[0038] When the global loss value reaches the minimum, it is determined that the global model converges.

[0039] Another embodiment of the application provides a hierarchical compromise model federated learning device based on momentum, which comprises a local module, an edge module and a center module.

[0040] Another embodiment of the application provides a hierarchical compromise model federated learning device based on momentum, which comprises a local module, an edge module and a center module.

[0041] ​The local module is used for updating local momentum, group momentum and global momentum in the local device according to the group model and the global model issued by the corresponding edge node, and correspondingly obtaining first local momentum, first group momentum and first global momentum, and training the local model according to the first local momentum; wherein one edge node corresponds to a plurality of local devices;

[0042] The edge module is used for aggregating second group momentum and second global momentum in the edge node according to the first group momentum and the first global momentum, training the group model according to the second group momentum, and then issuing the trained group model and the global model issued by the corresponding center server to a plurality of local devices corresponding to the edge node; wherein one center server corresponds to a plurality of edge nodes;

[0043] The center module is used for aggregating third global momentum in the center server according to the second global momentum, training the global model according to the third global momentum, and after issuing the trained global model to a plurality of edge nodes corresponding to the center server, starting the next round of iteration, and ending the iteration after determining that the global model obtained in the latest round of iteration converges.

[0044] Compared with the prior art, the beneficial effects of the present application are that:

[0045] By introducing the bidirectional collaborative optimization strategy, each level of model can improve the generalization ability based on the superior model correction, and at the same time, can improve the model individualization ability based on momentum aggregation, thereby further improving the overall model adaptability and learning efficiency of federated learning. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 A flowchart of a hierarchical compromise model federated learning method based on momentum is provided for an embodiment of the present application.

[0047] Figure 2 A structure diagram of a hierarchical compromise model federated learning framework based on momentum is provided for an embodiment of the present application.

[0048] Figure 3 A structure diagram of a hierarchical compromise model federated learning device based on momentum is provided for another embodiment of the present application. DETAILED DESCRIPTION

[0049] The drawings are only used for illustrative description, and cannot be understood as a limitation on the patent;

[0050] For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings can be omitted.

[0051] The technical solutions of the present application will be further described below in combination with the drawings and examples.

[0052] Referring to Figure 1 A flowchart of a momentum-based hierarchical compromise model federated learning method provided by an embodiment of the present application, comprising the following steps:

[0053] S1: updating local momentum, group momentum and global momentum in a local device according to a group model and a global model issued by a corresponding edge node, corresponding to obtain first local momentum, first group momentum and first global momentum, and training a local model according to the first local momentum; wherein one edge node corresponds to several local devices;

[0054] S2: aggregating to obtain second group momentum and second global momentum in the edge node according to the first group momentum and the first global momentum, training a group model according to the second group momentum, and then issuing the trained group model and the global model issued by the corresponding center server to the several local devices corresponding to the edge node; wherein one center server corresponds to several edge nodes;

[0055] S3: aggregating to obtain third global momentum in the center server according to the second global momentum, training a global model according to the third global momentum, and after issuing the trained global model to the several edge nodes corresponding to the center server, starting the next round of iteration until determining that the global model obtained in the latest round of iteration converges, ending the iteration.

[0056] For step S1, specifically, updating local momentum, group momentum and global momentum in a local device according to a group model and a global model issued by a corresponding edge node, corresponding to obtain first local momentum, first group momentum and first global momentum, specifically including:

[0057] updating the local momentum according to a preset random subset and a preset local gradient calculated by the local model, corresponding to obtain the first local momentum; wherein the update formula of the local momentum is specifically:

[0058]

[0059] wherein, is the first local momentum, is the local momentum, is a preset update coefficient, is the preset local gradient;

[0060] According to the group model and the global model issued by the corresponding edge node, the group momentum and the global momentum are updated respectively, and the first group momentum and the first global momentum are obtained correspondingly.

[0061] In a preferred embodiment, the local device uses a random subset of the local data during the implementation of each round of local iteration, and calculates the gradient and updates the local momentum using the local model. When updating the momentum, the parameters The influence of the current gradient on the momentum can be controlled, The general value , The larger the value is, the greater the influence of the historical gradient on the momentum is, and when , the momentum degenerates into the gradient. is a random gradient calculated by the local device through the stochastic gradient descent method.

[0062] The reason for selecting a random subset as the training data is to alleviate the non-independent and identically distributed phenomenon of the data to some extent and avoid overfitting. Similarly, the group momentum and the global momentum can also be updated using the group model and the global model issued by the edge node respectively. After completing the momentum update, the local model uploads the first local momentum, the first group momentum and the first global momentum obtained by updating to the corresponding edge node, thereby providing a data basis for implementing the reverse model feedback strategy.

[0063] Further, the training of the local model according to the first local momentum specifically includes:

[0064] updating the local model by the momentum gradient descent method according to the first local momentum;

[0065] correcting the updated local model according to the group model issued by the corresponding edge node to obtain the trained local model.

[0066] Preferably, the correcting the updated local model according to the group model issued by the corresponding edge node specifically includes:

[0067] calculating a first square norm between the updated local model and the group model issued by the corresponding edge node;

[0068] correcting the updated local model by taking the first square norm as a correction term; wherein the formula of the correction is specifically:

[0069]

[0070] wherein, is the updated local model in the i-th iteration round, is the group model issued by the corresponding edge node, is the global model issued by the corresponding edge node, and the group model corresponding to the edge node in the iteration round, a preset learning rate, a preset correction coefficient, the first square norm.

[0071] In a preferred embodiment, after the local device completes the local momentum update, the updated local momentum (i.e., the first local momentum) can be used instead of the gradient to update the local model by the momentum gradient descent method. Since the momentum contains not only the information of the current gradient, but also the information of all previous gradients to some extent, using the momentum to train the model can make the convergence direction of the model training more stable and the convergence process more efficient.

[0072] At the same time, the group model returned by the edge node in the previous iteration round is introduced as a correction term in the training process, and the local model is corrected by calculating the square norm of the local model and the group model to realize the forward model correction strategy.

[0073] The preferred embodiment uses the parameter to control the correction degree, when , the local device tends to personalized learning; when , the local device tends to assimilate with the group model. Among them, The value of can be modified by the local device according to the actual situation.

[0074] Through the above steps, the local device completes a complete local iteration implementation process. Since the data update frequency of the local devices in the same group is inconsistent due to factors such as region, user habit, and the stability between devices is also limited by factors such as hardware condition, therefore, in the preferred embodiment, the local devices are allowed to perform local iteration in parallel and asynchronously. That is, in a certain iteration round, when the local devices in a subset of the group have completed local iteration, it can be considered that the group has completed a successful local iteration implementation process.

[0075] For step S2, specifically, the second group momentum and the second global momentum are aggregated in the edge node according to the first group momentum and the first global momentum, specifically including:

[0076] The second group momentum is aggregated by weighted average method according to the first group momentum; wherein the aggregation weight of aggregating the first group momentum is the ratio of the sample data amount of the data set contained in the local device corresponding to the first group momentum to the sample data amount of the data set contained in the edge node;

[0077] According to the first global momentum, the second global momentum is aggregated by a weighted average method; wherein, the aggregation weight of the first global momentum is the ratio of the sample data amount of the data set contained in the local device corresponding to the first global momentum to the sample data amount of the data set contained in the edge node.

[0078] In a preferred embodiment, the edge node is regarded as a special node different from the local device, which can only serve as a group leader and is only responsible for maintaining the group model through the feedback information of the nodes in the group, without storing any data. The applicability of this edge node provides help.

[0079] The edge node aggregates the first group momentum uploaded by the local devices in the group by a weighted average method to obtain the second group momentum. In the aggregation process, the weight can generally use the ratio of the data amount of the node in the group to the data amount in the group, which means that the node with a large amount of data contributes more to the group model, so that the group model performs better on the node with a large amount of data, and improves the overall generalization ability.

[0080] At the same time, the edge node also needs to aggregate the first global momentum to obtain the second global momentum, and upload the second global momentum to the corresponding center server for training the global model.

[0081] Further, the training of the group model according to the second group momentum specifically includes:

[0082] updating the group model according to the second group momentum by a momentum gradient descent method;

[0083] correcting the updated group model according to the global model issued by the corresponding center server to obtain the trained group model.

[0084] Preferably, the correction of the updated group model according to the global model issued by the corresponding center server specifically includes:

[0085] calculating the second square norm between the updated group model and the global model issued by the corresponding center server;

[0086] using the second square norm as a correction term to correct the updated group model; wherein, the correction formula is specifically:

[0087]

[0088] wherein, is the updated group model in the i-th iteration round, is the global model issued by the corresponding center server in the i-th iteration round, is the global model issued by the corresponding center server in the i-th iteration round, a global model corresponding to the center server in each iteration round, for the second group of momentums, for a preset correction coefficient, for the second square norm.

[0089] In a preferred embodiment, after the second group of momentums is aggregated, the group model can be trained by the momentum gradient descent method, and the global model sent by the center server is introduced for correction in the training process.

[0090] After the group model is trained, the edge node also needs to synchronize the trained group model and the global model sent by the center server to all local devices in the group in the form of broadcasting, further reducing the communication overhead. It is worth noting that, since the periods of group aggregation and global aggregation are inconsistent, the edge node may synchronize the same global model to the local devices in multiple group aggregation periods. The preferred embodiment does not require the edge node to synchronize the latest global model to the local devices every time, because considering the instability of the edge node, the edge node may not be able to successfully receive the global model in every global aggregation period. Therefore, the edge node only needs to ensure that the global model stored by itself is synchronized to the local devices, which can provide a certain fault tolerance for the system.

[0091] Through the above steps, the edge node completes a complete group aggregation implementation process. In the preferred embodiment, the edge node only needs to bear the functions of calculation and communication, and does not need to store a large amount of data, which makes the selection of hardware devices of the edge node in the actual scene only need to consider the calculation and communication related indicators, and does not need to be equipped with large capacity storage, reducing the construction cost of the edge node. At the same time, since the edge node does not use any data of itself for calculation, the data returned by the edge node to the center server can better represent the distribution of the data in the group.

[0092] For step S3, specifically, the third global momentum is aggregated in the center server according to the second global momentum, and the global model is trained according to the third global momentum, specifically including:

[0093] The third global momentum is aggregated by the weighted average method according to the second global momentum; wherein the aggregation weight of the second group of momentums is the ratio of the sample data amount of the data set contained in the edge node to the sample data amount of the data set contained in the center server corresponding to the second group of momentums;

[0094] The global model is trained by the momentum gradient descent method according to the third global momentum.

[0095] In a preferred embodiment, the central server aggregates the global momentum uploaded by the edge nodes using a weighted average, with the weights being the ratio of the data volume within an edge node group to the total data volume. Considering the instability of edge nodes, this step is executed once the central server receives a certain amount of data feedback from edge nodes within a single period. After training is complete, the trained global model is distributed to all the edge nodes. Thus, the method of this invention completes a full federated learning process. Generally, the method of this invention repeats the above process continuously until the latest trained global model is determined to have converged.

[0096] Furthermore, the convergence condition of the global model is specifically as follows:

[0097] The loss function value of the global model issued by the corresponding edge node is calculated based on the preset loss function in the local device.

[0098] The global model loss function values ​​calculated by all the local devices are aggregated to obtain the global loss value;

[0099] The global model is considered to have converged when the global loss value reaches its minimum.

[0100] In a preferred embodiment, the method of the present invention can obtain an optimal global model after multiple iterations, such that... , so that through the function The aggregated result minimizes the global loss. Within the Fedavg (Federated Averaging) framework, the aggregation function... ;in, Let k be the number of dataset samples for the k-th local device. The number of samples in all datasets within the central server. The preset loss function is defined as follows.

[0101] Therefore, the optimization objective of federated learning can be modeled as:

[0102] Reference Figure 2 This is a schematic diagram of a federated learning framework based on a momentum-based hierarchical trade-off model, provided in an embodiment of the present invention. Figure 2 It can be seen that, based on existing federated learning algorithms, this invention introduces the following two technical improvements:

[0103] Firstly, hierarchical topology. The whole federated learning architecture is centered on multiple relay devices such as edge nodes in a certain regional range, and user devices are logically divided into groups according to certain rules (in actual scenarios, more consideration is given to dividing according to factors such as geography, device type, and user habits) to form a three-level topology: local device-edge node-center server. The edge node can be regarded as the group leader of the group. At the same time, it is stipulated that during the whole learning process, the local device only communicates with the edge node of the group, and the edge node synchronizes information with the nodes in the group through broadcasting to reduce the communication overhead in the group. The center server only communicates with the edge node of each group, that is, the group leader. In the process of model training, the local device sends the update to the edge node in the group after completing multiple rounds of local training, and the edge node aggregates these updates by weighting to train the group model maintained by the edge node. Similarly, the edge node sends the update to the center server after completing several rounds of training, and the center server also aggregates these updates by weighted average and updates the global model.

[0104] Secondly, the bidirectional collaborative optimization strategy. In general, the bidirectional collaborative optimization strategy includes a forward model correction strategy from top to bottom, that is, a backward model feedback strategy from bottom to top. These two strategies work collaboratively based on the hierarchical structure to dynamically balance the generalization and personalization ability of the models at each level. For the forward model correction strategy, in the process of training the models at each level, the model at the upper level is introduced as a correction term to correct the objective function, balance the differences between the local model and the group model, and the group model and the global model. This allows the models at each level to better fit the local data in a heterogeneous data scenario while also improving the generalization ability of the model to some extent and avoiding overfitting.

[0105] Based on the above two improvement strategies, the federated learning architecture provided by the application reduces unnecessary communication loss and reduces communication costs through a hierarchical topology. At the same time, the introduction of the bidirectional collaborative optimization strategy allows the models at each level to improve the generalization ability based on the upper model correction while also improving the model personalization ability based on momentum aggregation, thereby further improving the overall model adaptability of federated learning.

[0106] In addition, in the method described in the application, the learning rate of the local model, the correction coefficient, the threshold value of how many local device feedback information the edge node receives for intra-group aggregation, and the threshold value of the number of edge nodes received by the center server are all flexible adjustments according to actual needs in actual scenarios.

[0107] Reference Figure 3A structural schematic diagram of a momentum-based hierarchical compromise model federated learning device according to another embodiment of the present application comprises a local module 101, an edge module 102 and a center module 103;

[0108] The local module 101 is configured to update local momentum, group momentum and global momentum in the local device according to the group model and the global model issued by the corresponding edge node, so as to obtain first local momentum, first group momentum and first global momentum, and train the local model according to the first local momentum; one edge node corresponds to a plurality of local devices;

[0109] The edge module 102 is configured to aggregate second group momentum and second global momentum in the edge node according to the first group momentum and the first global momentum, train the group model according to the second group momentum, and then issue the trained group model and the global model issued by the corresponding center server to a plurality of local devices corresponding to the edge node; one center server corresponds to a plurality of edge nodes;

[0110] The center module 103 is configured to aggregate third global momentum in the center server according to the second global momentum, train the global model according to the third global momentum, and after issuing the trained global model to a plurality of edge nodes corresponding to the center server, start the next round of iteration until the global model obtained in the latest round of iteration converges, and end the iteration.

[0111] Obviously, the above embodiments of the present application are only examples for clearly illustrating the present application, and are not intended to limit the embodiments of the present application. Based on the above description, other different forms of changes or variations can be made by those skilled in the art. Here, it is not necessary and impossible to exhaust all the embodiments. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the claims of the present application.

Claims

1. A momentum-based hierarchical trade-off model federated learning method, characterized in that, Includes the following steps: Based on the group model and global model issued by the corresponding edge node, the local momentum, group momentum and global momentum are updated in the local device to obtain the first local momentum, the first group momentum and the first global momentum, and the local model is trained based on the first local momentum; wherein, one edge node corresponds to several local devices; Based on the first set of momentum and the first global momentum, a second set of momentum and a second global momentum are aggregated in the edge nodes. A group model is trained based on the second set of momentum, and then the trained group model and the corresponding global model issued by the central server are distributed to several local devices corresponding to the edge nodes; wherein one central server corresponds to several edge nodes; the step of training the group model based on the second set of momentum specifically includes: Based on the second set of momentum, the group model is updated using the momentum gradient descent method; The updated group model is corrected based on the global model issued by the corresponding central server to obtain the trained group model, specifically including: Calculate the second square norm between the updated group model and the global model issued by the corresponding central server; The second square norm is used as a correction term to modify the updated group model; wherein the specific formula for the correction is: in, For the first The updated group model described in each iteration round For the first The global model corresponding to each iteration round is issued by the central server. For the second set of momentum, This is a preset correction factor. It is the second square norm; Based on the second global momentum, a third global momentum is aggregated in the central server. A global model is trained based on the third global momentum. After the trained global model is distributed to the edge nodes corresponding to the central server, the next round of iteration begins. The iteration ends when the global model obtained in the latest round of iteration is determined to have converged.

2. The momentum-based hierarchical trade-off model federated learning method as described in claim 1, characterized in that, The step of updating the local momentum, group momentum, and global momentum in the local device based on the group model and global model issued by the corresponding edge node, and obtaining the first local momentum, the first group momentum, and the first global momentum, specifically includes: The local momentum is updated based on a preset random subset and a preset local gradient calculated by the local model, thereby obtaining the first local momentum; wherein, the update formula for the local momentum is specifically as follows: in, This is the first local momentum. This refers to the local momentum. To preset update coefficients, For the preset local gradient; Based on the group model and global model issued by the corresponding edge node, the group momentum and global momentum are updated respectively to obtain the first group momentum and the first global momentum.

3. The momentum-based hierarchical trade-off model federated learning method as described in claim 1, characterized in that, The step of training a local model based on the first local momentum specifically includes: Based on the first local momentum, the local model is updated using momentum gradient descent. The updated local model is corrected based on the group model issued by the corresponding edge node to obtain the trained local model.

4. The momentum-based hierarchical trade-off model federated learning method as described in claim 3, characterized in that, The step of correcting the updated local model based on the group model issued by the corresponding edge node specifically includes: Calculate the first square norm between the updated local model and the group model issued by the corresponding edge node; The first square norm is used as a correction term to correct the updated local model; wherein the formula for correcting the updated local model is as follows: in, For the first The updated local model described in each iteration round For the first The group model issued by the corresponding edge node in each iteration round To preset the learning rate, This is a preset correction factor. Let be the first square norm. This is a preset local gradient.

5. The momentum-based hierarchical trade-off model federated learning method as described in claim 1, characterized in that, The step of aggregating the second set of momentum and the second global momentum in the edge nodes based on the first set of momentum and the first global momentum specifically includes: Based on the first set of momentum, the second set of momentum is obtained by aggregating them using a weighted average method; wherein, the aggregation weight of the first set of momentum is the ratio of the sample data volume of the dataset contained in the local device corresponding to the first set of momentum to the sample data volume of the dataset contained in the edge node; Based on the first global momentum, the second global momentum is obtained by aggregating it using a weighted average method; wherein, the aggregation weight of the first global momentum is the ratio of the sample data volume of the dataset contained in the local device corresponding to the first global momentum to the sample data volume of the dataset contained in the edge node.

6. The method as described in claim 1, characterized in that, The step of aggregating a third global momentum in the central server based on the second global momentum, and training a global model based on the third global momentum, specifically includes: The third global momentum is obtained by aggregating the second global momentum using a weighted average method; wherein, the aggregation weight of the second set of momentum is the ratio of the sample data volume of the dataset contained in the edge node corresponding to the second set of momentum to the sample data volume of the dataset contained in the central server. The global model is trained using momentum gradient descent based on the third global momentum.

7. The method as described in claim 1, characterized in that, The convergence condition of the global model is as follows: The loss function value of the global model issued by the corresponding edge node is calculated based on the preset loss function in the local device. The global model loss function values ​​calculated by all the local devices are aggregated to obtain the global loss value; The global model is considered to have converged when the global loss value reaches its minimum.

8. A momentum-based hierarchical trade-off model federated learning device, characterized in that, include: Local modules, edge modules, and central modules; The local module is used to update the local momentum, group momentum, and global momentum in the local device according to the group model and global model issued by the corresponding edge node, thereby obtaining the first local momentum, the first group momentum, and the first global momentum, and training the local model based on the first local momentum; wherein, one edge node corresponds to several local devices; The edge module is used to aggregate a second set of momentum and a second global momentum in the edge nodes based on the first set of momentum and the first global momentum, and to train a group model based on the second set of momentum. Subsequently, the trained group model and the global model issued by the corresponding central server are distributed to several local devices corresponding to the edge nodes; wherein one central server corresponds to several edge nodes; the step of training the group model based on the second set of momentum specifically includes: Based on the second set of momentum, the group model is updated using the momentum gradient descent method; The updated group model is corrected based on the global model issued by the corresponding central server to obtain the trained group model, specifically including: Calculate the second square norm between the updated group model and the global model issued by the corresponding central server; The second square norm is used as a correction term to modify the updated group model; wherein the specific formula for the correction is: in, For the first The updated group model described in each iteration round For the first The global model corresponding to each iteration round is issued by the central server. For the second set of momentum, This is a preset correction factor. It is the second square norm; The central module is used to aggregate a third global momentum in the central server based on the second global momentum, train a global model based on the third global momentum, and after distributing the trained global model to the edge nodes corresponding to the central server, start the next round of iteration until the global model obtained in the latest round of iteration is determined to have converged, and then end the iteration.

Images