Federal learning training method and system supporting heterogeneous data

Abstract

The invention discloses a federated learning training method and system supporting heterogeneous data, belongs to the technical field of distributed machine learning, and is used for solving the technical problem that an existing federated learning technology cannot solve three major problems of data privacy, data heterogeneity and hierarchical communication bottleneck at the same time and lacks an integrated federated learning training framework. The method comprises the following steps that: a working node acquires a local model update quantity and performs quantitative compression based on local data, a historical update record and a latest global model to obtain a compressed update package and uploads the compressed update package to an affiliated edge node; the edge node aggregates the compressed update packet sent by each working node to obtain a regional aggregation update packet; carrying out second quantization compression on the regional aggregation update packet to obtain a final uploading packet, and uploading the final uploading packet to a central server to which the final uploading packet belongs; the central server globally aggregates the final upload packet sent by each edge node to obtain a global model update packet; and optimizing the global model according to the global model updating package to obtain an optimized model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of distributed machine learning, in particular to a federated learning training method and system supporting heterogeneous data. BACKGROUND

[0002] With the increasing strictness of data privacy regulations and the enhancement of user privacy awareness, the traditional mode of gathering data to the central server for model training is facing great challenges. Federated learning (FL) emerges as the times require, which is a new distributed machine learning paradigm that allows model training at the data generation site (such as user's mobile phone, hospital server), without uploading the user's original data, thereby protecting data privacy at the source.

[0003] Although federated learning has broad prospects, it still faces three major challenges in practical application: first, data heterogeneity: in federated learning, the data distribution of each participant (worker node) is usually non-independent and identically distributed (non-IID). This data heterogeneity will make the model update direction of local training differ greatly, and if a simple average is performed directly on the server, it will lead to slow convergence or even complete non-convergence of the global model. Second, communication bottleneck: federated learning usually involves a large number of device participants. If all devices directly communicate with the central server, it will cause huge network congestion and server load, which is not feasible in reality. Third, computational and communication efficiency: in order to reduce communication frequency, federated learning allows devices to perform multiple local iterations before uploading model updates. However, how to efficiently compress and transmit these model updates generated after multiple local iterations, especially under the background of non-IID data, is a pending problem.

[0004] Existing technologies attempt to solve these problems from different angles, but generally have shortcomings. For example, some methods introduce gradient compression, but ordinary compression algorithms may exacerbate the negative effects of non-IID data. Some other methods design hierarchical aggregation architecture (Hierarchical FL, HFL) to alleviate server bottlenecks, but lack collaborative design with compression algorithms and non-IID problem handling. Therefore, there is an urgent need for an integrated federated learning training framework that can simultaneously solve the three major challenges of data privacy, data heterogeneity, and hierarchical communication bottlenecks. SUMMARY

[0005] The embodiments of the present application provide a federated learning training method and system supporting heterogeneous data, which are used to solve the technical problem that existing federated learning technologies cannot simultaneously solve the three major problems of data privacy, data heterogeneity, and hierarchical communication bottlenecks, and lack an integrated federated learning training framework.

[0006] The embodiment of the present application adopts the technical solutions below:

[0007] In one aspect, the embodiment of the present application provides a federated learning training method supporting heterogeneous data, which comprises the following steps:

[0008] The edge node aggregates the compressed update packages sent by each working node to obtain a regional aggregated update package, and performs second quantization compression on the regional aggregated update package to obtain a final upload package and upload the final upload package to a central server belonging to the edge node;

[0009] The central server performs global aggregation on the final upload packages sent by each edge node to obtain a global model update package, and optimizes the global model according to the global model update package to obtain an optimized model.

[0010] In a feasible implementation, before the working node obtains a local model update amount and performs quantization compression based on local data, a historical update record and a latest global model to obtain a compressed update package and upload the compressed update package to a belonging edge node, the method further comprises the following steps:

[0011] The central server initializes a globally shared global model, and publishes the global model to all edge nodes;

[0012] The edge node forwards the received global model to all working nodes connected thereto;

[0013] Each working node comprises a historical update record for a DIANA clustering algorithm, and the initial value of the historical update record is zero.

[0014] In a feasible implementation, the working node obtains a local model update amount and performs quantization compression based on local data, a historical update record and a latest global model to obtain a compressed update package and upload the compressed update package to a belonging edge node, specifically comprising the following steps:

[0015] Each working node collects training data within a node connection range and saves the training data to a local device as local data;

[0016] After receiving the global model and a model training task, the working node performs multi-step training iteration on the global model through the local data to obtain a local model;

[0017] The working node calculates a change amount between the local model and the latest global model to obtain a local model update amount, and obtains the compressed update package according to the local model update amount and the historical update record, and uploads the compressed update package to a belonging edge node.

[0018] In one feasible implementation, the compressed update package is obtained based on the local model update volume and the historical update records, specifically including:

[0019] The working node retrieves the historical update record and calculates the difference between the local model update amount and the historical update record; then it quantizes and compresses the difference using a quantization compressor to obtain the compressed update package.

[0020] In one feasible implementation, after obtaining the compressed update package based on the local model update volume and the historical update records, the method further includes:

[0021] The working node updates the historical update record according to the compressed update package and preset rules, so that it can be used in the next iteration.

[0022] In one feasible implementation, the edge node aggregates the compressed update packets sent by each working node to obtain a regional aggregated update packet, specifically including:

[0023] The edge node collects compressed update packets sent by all the worker nodes under its jurisdiction, and calculates the average value of all compressed update packets to obtain the regional aggregated update packet.

[0024] In one feasible implementation, the regional aggregation update package undergoes a second quantization and compression to obtain the final upload package, which is then uploaded to the central server to which it belongs. Specifically, this includes:

[0025] The edge node performs a second quantization compression on the regional aggregated update package using a preset quantization compressor to obtain the final upload package, which is then uploaded to the central server to which the edge node belongs.

[0026] In one feasible implementation, the central server globally aggregates the final upload packets sent by each edge node to obtain a global model update packet, specifically including:

[0027] The central server collects the final upload packets sent by all edge nodes under its jurisdiction and calculates their average value to obtain the global model update packet.

[0028] In one feasible implementation, optimizing the global model based on the global model update package to obtain an optimized model specifically includes:

[0029] The central server substitutes the global model parameters obtained from the previous iteration, the global model update package, and the preset learning rate into the update formula to calculate the global model parameters after the current iteration.

[0030] This process is repeated until a preset number of iterations or a preset convergence criterion is reached, to obtain the final global model parameters and thus the optimized model.

[0031] On the other hand, embodiments of the present invention also provide a federated learning training system supporting heterogeneous data, the system comprising:

[0032] The first quantization and compression module is used to obtain the local model update volume through the working node based on local data, historical update records and the latest global model, and to quantize and compress it to obtain a compressed update package and upload it to the corresponding edge node.

[0033] The second quantization and compression module is used to aggregate the compressed update packages sent by each working node through the edge node to obtain a regional aggregated update package; and to perform a second quantization and compression on the regional aggregated update package to obtain a final upload package and upload it to the central server to which it belongs.

[0034] The model optimization module is used to globally aggregate the final upload packets sent by each edge node through the central server to obtain a global model update packet; and optimize the global model according to the global model update packet to obtain an optimized model.

[0035] Compared with the prior art, the federated learning training method and system for heterogeneous data provided in this embodiment of the invention have the following beneficial effects:

[0036] This invention proposes a hierarchical federated learning method to protect data privacy. The core idea of ​​this method is to construct a three-layer federated learning architecture of "worker node-edge node-server" and deploy a two-level quantization compression mechanism based on a quantization algorithm with momentum correction, specifically designed for data heterogeneity.

[0037] First, this invention follows the core principles of federated learning, whereby all training data is always stored on the local worker node (such as the user's device). The worker node performs multiple computational iterations locally to fully utilize local data for model training, and only the calculated "model update" rather than the original data is subsequently transmitted, fundamentally ensuring data privacy.

[0038] Secondly, after local training is completed on the worker nodes, this invention does not use traditional quantization methods, but instead employs the DIANA algorithm to compress the resulting "model update volume". DIANA effectively corrects update biases caused by the non-IID characteristics of local data by maintaining a historical update record locally on the worker nodes, thereby generating high-quality compressed updates that better represent the global direction. This is the core of this invention, specifically designed to handle data heterogeneity.

[0039] Finally, to avoid communication bottlenecks, this invention employs a hierarchical aggregation architecture. Worker nodes send DIANA-compressed updates to regional edge nodes instead of directly to the central server. Edge nodes are designed as stateless, lightweight relays. They are only responsible for aggregating updates from their subordinate worker nodes and performing a second quantization compression to further reduce the communication load on the central server. This design simplifies deployment and maintenance at the edge.

[0040] Through the above design, this invention "forwards" the complexity of dealing with data heterogeneity to the DIANA algorithm on the worker node side, while solving the communication efficiency and scalability issues through a hierarchical architecture and two-level quantization, forming a complete, efficient and privacy-preserving federated learning solution. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0042] Figure 1 A flowchart of a federated learning training method supporting heterogeneous data is provided in an embodiment of the present invention;

[0043] Figure 2 An algorithm logic diagram of a federated learning training method supporting heterogeneous data provided in an embodiment of the present invention;

[0044] Figure 3 This is a schematic diagram of the structure of a federated learning training system that supports heterogeneous data, provided in an embodiment of the present invention. Detailed Implementation

[0045] To enable those skilled in the art to better understand the technical solutions of this invention, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this invention.

[0046] This invention provides a federated learning training method that supports heterogeneous data, such as... Figure 1 As shown, the federated learning training method supporting heterogeneous data specifically includes steps S101-S103:

[0047] S101: The working node obtains the local model update volume based on local data, historical update records, and the latest global model, performs quantization and compression, obtains a compressed update package, and uploads it to the edge node to which it belongs.

[0048] Specifically, the central server initializes a globally shared global model and publishes it to all edge nodes. The edge nodes forward the received global model to all worker nodes connected to it. Each worker node contains a historical update record used for the DIANA clustering algorithm, with an initial value of zero.

[0049] As a feasible implementation method, this invention explicitly defines a three-tiered communication and computing architecture of "worker node-edge node-server". Worker nodes are responsible for local computation, edge nodes are responsible for regional aggregation and critical error compensation, and servers are responsible for the final update of the global model. One server manages multiple edge nodes, and one edge node manages multiple worker nodes, forming a standard hierarchical network structure.

[0050] Furthermore, each worker node aggregates the training data within its connection range and saves it locally as local data. After receiving the global model and the model training task, the worker node performs multi-step training iterations on the global model using the local data to obtain the local model.

[0051] Furthermore, the worker node calculates the change between the local model and the latest global model to obtain the local model update amount. Based on the local model update amount and historical update records, a compressed update package is obtained and uploaded to the corresponding edge node.

[0052] As a feasible implementation method, a compressed update package is obtained based on the local model update amount and historical update records. The specific method is as follows: the working node retrieves the historical update records and calculates the difference between the local model update amount and the historical update records; and the difference is quantized and compressed using a quantization compressor to obtain the compressed update package.

[0053] Furthermore, the working nodes update the historical update records according to the compressed update package and preset rules, so that they can be retrieved in the next iteration.

[0054] In one embodiment, the preset rule is: multiply the compressed update package by the preset step size parameter, and then add it to the historical update record of the previous iteration to obtain the updated historical update record.

[0055] After local training is completed on the worker nodes, this invention does not use traditional quantization methods, but instead employs the DIANA algorithm to compress the generated model update volume. DIANA effectively corrects update biases caused by the non-IID characteristics of local data by maintaining a historical update record locally on the worker nodes according to preset rules, thereby generating high-quality compressed updates that better represent the global direction. This is the core of this invention, specifically designed to handle data heterogeneity.

[0056] S102. The edge nodes aggregate the compressed update packets sent by each working node to obtain the regional aggregated update packet; the regional aggregated update packet is then subjected to a second quantization compression to obtain the final upload packet and uploaded to the central server to which it belongs.

[0057] Specifically, the edge node collects compressed update packets sent by all the worker nodes under its jurisdiction, calculates the average value of all compressed update packets, and obtains a regional aggregated update packet. Then, the edge node performs a second quantization compression on the regional aggregated update packet using a preset quantization compressor to obtain the final upload packet, which is then uploaded to the central server to which the edge node belongs.

[0058] S103. The central server performs global aggregation on the final upload packets sent by each edge node to obtain a global model update packet; the global model is optimized based on the global model update packet to obtain an optimized model.

[0059] Specifically, the central server collects the final upload packets sent by all the edge nodes under its jurisdiction, calculates their average value, and obtains the global model update packet.

[0060] Furthermore, the central server substitutes the global model parameters obtained from the previous iteration, the global model update package, and the preset learning rate into the update formula to calculate the global model parameters after the current iteration.

[0061] This process is repeated until the preset number of iterations or the preset convergence criterion is reached, to obtain the final global model parameters and thus the optimized model.

[0062] As a feasible implementation method, the specific iterative process of the federated learning training method for heterogeneous data provided by this invention is as follows: Figure 2 As shown:

[0063] First, during the system initialization phase, the central server initializes the global model parameters x0, the learning rate γ, and the quantization compressor Q in the worker nodes. wThe model consists of a sparsity compressor C in the edge nodes and a preset step size parameter α. The initial global model parameters x0 are broadcast down to all edge nodes, which then forward the parameters to all their connected worker nodes. A buffer is then created for each edge node to store compression errors, and the values ​​in this buffer are initialized to a zero vector, representing the initial error term for each edge node e. Among them, 0 d This represents the zero vector. A "local history" for the DIANA algorithm is created for each worker node (such as a user's mobile phone or a hospital server), and its initial value is set to zero.

[0064] Furthermore, the model parameters are calculated in T iterations. The following is the process for a single training iteration (executed in a loop):

[0065] Step 1: Operations on worker nodes (executed in parallel):

[0066] a. Retrieve the current global model x t And calculate the local stochastic gradient. Where t represents the t-th iteration and w represents the w-th working node.

[0067] b. Calculate the difference components and quantize them for compression, generating a compressed update package with a smaller data size. And send it to its corresponding edge node e. Wherein, This refers to the historical update records stored in worker node w.

[0068] c. Update the local historical gradient estimate (i.e., the historical update record mentioned above):

[0069] Step 2: Operations on edge nodes (executed in parallel):

[0070] a. Aggregated Update: Each edge node collects compressed update packets sent by all worker nodes under its jurisdiction and calculates their average value to form a regional aggregated update packet. Among them, W e This indicates that edge node e governs a total of W working nodes.

[0071] b. Error Compensation: Edge nodes perform a second quantization and compression on the regional aggregated update packet to obtain the final upload packet. Then update the error term. This difference is then stored in the local error cache for use in the next iteration.

[0072] c. Data Upload: Upload the final package Send to the central server.

[0073] Step 3: Operation of the central server:

[0074] a. Global Aggregation: The central server collects the final upload packets sent by all edge nodes. The average value is then calculated to obtain the final global model update package.

[0075] b. Updating the model: The central server uses the global model update package and the preset learning rate to update the global model parameters. Here, E represents the number of edge nodes managed by the central server.

[0076] Step 4, Process Loop and End: After the model update is completed, the new version of the global model parameters will be broadcast again at the start of the next iteration. The system repeats all the above steps until the training reaches the preset convergence criterion or number of iterations.

[0077] Final output: The final global model after training.

[0078] In one embodiment, multiple hospitals wish to jointly train a high-precision AI medical image diagnostic model (such as tumor recognition from CT scans). However, due to patient data privacy and industry regulations, the raw image data from any one hospital cannot leave its local server. Furthermore, the different equipment models, scanning parameters, and patient origins of each hospital lead to severe data heterogeneity. Using this invention, each hospital's server acts as a "working node," performing multiple rounds of model training on local data. The resulting model updates are compressed using the DIANA algorithm, specifically designed for heterogeneity, and then sent to regional "edge nodes" (such as a regional medical cloud platform). The edge nodes aggregate and further compress the updates from multiple hospitals before submitting them to the final "central server." This solution effectively solves the training challenges caused by data heterogeneity without exposing patient privacy and constructs a scalable regional collaborative network.

[0079] In another embodiment, multiple banks or financial institutions wish to collaboratively build a more robust credit risk assessment model to identify potential fraudulent activities. Each institution's user data is a core and highly sensitive asset that must never be shared. Furthermore, different banks have vastly different customer groups and business types, resulting in significant data heterogeneity. Applying this invention, each bank's internal server acts as a "working node," using its own data for local training. Its specially designed quantization and compression mechanism ensures that even if a bank's customer characteristics are highly unique, its uploaded model updates will not excessively interfere with the stability of the global model. These high-quality updates are efficiently aggregated through a layered architecture, ultimately forming a risk control model on a central server that incorporates the wisdom of all parties and has stronger generalization capabilities, while ensuring the highest level of data privacy protection for all parties.

[0080] Experiments were conducted on different datasets and models. Under the condition of equal data transmission, this invention achieved higher model accuracy compared to other methods, demonstrating that it achieved optimal training efficiency and model accuracy while maintaining a high compression ratio and low communication cost. This indicates that this invention achieves a balance between communication efficiency, training speed, and accuracy under a multi-layer communication architecture, which is superior to existing mainstream hierarchical training methods.

[0081] In addition, embodiments of the present invention also provide a federated learning training system that supports heterogeneous data, such as... Figure 3 As shown, the federated learning training system 300 that supports heterogeneous data specifically includes:

[0082] The first quantization and compression module 310 is used to obtain the local model update volume and perform quantization and compression based on local data, historical update records and the latest global model through the working node, to obtain a compressed update package and upload it to the edge node to which it belongs.

[0083] The second quantization and compression module 320 is used to aggregate the compressed update packages sent by each working node through the edge node to obtain a regional aggregated update package; and to perform a second quantization and compression on the regional aggregated update package to obtain a final upload package and upload it to the central server to which it belongs.

[0084] The model optimization module 330 is used to globally aggregate the final upload packets sent by each edge node through the central server to obtain a global model update packet; and optimize the global model according to the global model update packet to obtain an optimized model.

[0085] The various embodiments in this invention are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, devices, and non-volatile computer storage media are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0086] The foregoing has described specific embodiments of the present invention. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0087] The above description is merely an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations can be made to the embodiments of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the embodiments of the present invention should be included within the protection scope of the present invention.

Claims

1. A federated learning training method supporting heterogeneous data, characterized in that, The method includes: Based on local data, historical update records, and the latest global model, the worker node obtains the local model update volume, performs quantization and compression, obtains a compressed update package, and uploads it to the edge node to which it belongs. The edge nodes aggregate the compressed update packets sent by each working node to obtain a regional aggregated update packet; the regional aggregated update packet is then subjected to a second quantization compression to obtain the final upload packet, which is then uploaded to the central server to which it belongs. The central server globally aggregates the final upload packets sent by each edge node to obtain a global model update packet; the global model is then optimized based on the global model update packet to obtain an optimized model.

2. The federated learning training method for supporting heterogeneous data according to claim 1, characterized in that, Before the worker node obtains the local model update volume based on local data, historical update records, and the latest global model, performs quantization and compression to obtain a compressed update package, and uploads it to the corresponding edge node, the method further includes: The central server initializes a globally shared global model and publishes the global model to all edge nodes; The edge node forwards the received global model to all the worker nodes it is connected to; Each working node contains a historical update record for the DIANA clustering algorithm, and the initial value of the historical update record is zero.

3. The federated learning training method for supporting heterogeneous data according to claim 1, characterized in that, Based on local data, historical update records, and the latest global model, the worker node obtains the local model update volume, performs quantization and compression, obtains a compressed update package, and uploads it to its affiliated edge node. Specifically, this includes: Each worker node aggregates the training data within its connection range and saves it locally as local data; After receiving the global model and the model training task, the working node performs multi-step training iterations on the global model using local data to obtain the local model; The working node calculates the change between the local model and the latest global model to obtain the local model update amount; based on the local model update amount and the historical update records, it obtains the compressed update package and uploads it to the corresponding edge node.

4. The federated learning training method for supporting heterogeneous data according to claim 3, characterized in that, Based on the local model update volume and the historical update records, the compressed update package is obtained, specifically including: The working node retrieves the historical update record and calculates the difference between the local model update amount and the historical update record; then it quantizes and compresses the difference using a quantization compressor to obtain the compressed update package.

5. The federated learning training method for supporting heterogeneous data according to claim 4, characterized in that, After obtaining the compressed update package based on the local model update volume and the historical update records, the method further includes: The working node updates the historical update record according to the compressed update package and preset rules, so that it can be used in the next iteration.

6. The federated learning training method for supporting heterogeneous data according to claim 1, characterized in that, The edge nodes aggregate the compressed update packets sent by each working node to obtain a regional aggregated update packet, which specifically includes: The edge node collects compressed update packets sent by all the worker nodes under its jurisdiction, and calculates the average value of all compressed update packets to obtain the regional aggregated update packet.

7. The federated learning training method for supporting heterogeneous data according to claim 1, characterized in that, The regional aggregated update package undergoes a second quantization and compression to obtain the final upload package, which is then uploaded to the central server to which it belongs. Specifically, this includes: The edge node performs a second quantization compression on the regional aggregated update package using a preset quantization compressor to obtain the final upload package, which is then uploaded to the central server to which the edge node belongs.

8. The federated learning training method for supporting heterogeneous data according to claim 1, characterized in that, The central server globally aggregates the final upload packets sent by each edge node to obtain a global model update packet, which specifically includes: The central server collects the final upload packets sent by all edge nodes under its jurisdiction and calculates their average value to obtain the global model update packet.

9. The federated learning training method for supporting heterogeneous data according to claim 1, characterized in that, The global model is optimized based on the global model update package to obtain an optimized model, specifically including: The central server substitutes the global model parameters obtained from the previous iteration, the global model update package, and the preset learning rate into the update formula to calculate the global model parameters after the current iteration. This process is repeated until a preset number of iterations or a preset convergence criterion is reached, to obtain the final global model parameters and thus the optimized model.

10. A federated learning training system supporting heterogeneous data, characterized in that, The system includes: The first quantization and compression module is used to obtain the local model update volume through the working node based on local data, historical update records and the latest global model, and to quantize and compress it to obtain a compressed update package and upload it to the corresponding edge node. The second quantization and compression module is used to aggregate the compressed update packages sent by each working node through the edge node to obtain a regional aggregated update package; and to perform a second quantization and compression on the regional aggregated update package to obtain a final upload package and upload it to the central server to which it belongs. The model optimization module is used to globally aggregate the final upload packets sent by each edge node through the central server to obtain a global model update packet; and optimize the global model according to the global model update packet to obtain an optimized model.

Images