Rapid and stable convergence model heterogeneous federated learning method, system and device

Abstract

The invention discloses a rapid and stable convergence model heterogeneous federated learning method and system, and belongs to the field of data protection machine learning, and the method comprises the steps: 1, a server initializes global model parameters, and creates a parameter buffer region of unused neurons in each hidden layer of a model; 2, at the beginning of each round, randomly selecting a part of or all clients to participate in the training of the current round; step 3, according to the capacity of each client participating in training, extracting a sub-model for the client from the parameter buffer area by using no-put-back random sampling, and sending the sub-model to the client for local training; step 4, a global aggregation step: after all participating clients finish local training, the server collects sub-model parameters uploaded by each client and performs weighted average aggregation by using a global aggregation step size attenuation mechanism to obtain a global model after the current round of aggregation; and step 5, if it is judged that the specified number of rounds is not reached or the model performance convergence is not reached, circularly executing the step 2 to the step 4, otherwise, ending. The method can improve convergence stability.

Description

Technical Field

[0001] The present invention relates to the field of distributed machine learning technology for data protection, and in particular to a method and system for fast and stable convergence of heterogeneous federated learning models. Background Art

[0002] Federated Learning (FL) is a distributed machine learning technology that allows multiple participants (also known as clients) to collaboratively train models without sharing local data. A basic algorithm for implementing federated learning is the FedAvg algorithm, which is the basic federated learning parameter weighted average aggregation algorithm (see reference [1]). In this FedAvg algorithm, each client trains a model with the same structure locally (i.e., model isomorphism) and only transmits the model parameters to the cloud-side server, where the parameters are weighted averaged and aggregated. However, in actual applications, the device resources (such as memory, network bandwidth, etc.) and computing power (hereinafter collectively referred to as client "capacity") of different clients may vary significantly, so their ability to train models also varies significantly, and it is necessary to introduce model heterogeneity to the original FedAvg algorithm.

[0003] The closest prior art to the present invention is a heterogeneous federated learning method based on sub-model extraction. This method allows each client to extract a sub-model from the global model for local training. The size of the sub-model is adjusted based on the client's capacity, and all sub-models are part of the global model, making it compatible with the weighted average parameter aggregation mechanism of the FedAvg algorithm. However, existing heterogeneous federated learning methods based on sub-model extraction have at least the following technical drawbacks:

[0004] (I) Low parameter space coverage efficiency: There are three main sub-model extraction schemes: static extraction, rolling extraction, and random extraction. (1) Static extraction (such as the HeteroFL algorithm in reference [2], a model heterogeneous federated learning algorithm based on static extraction) will cause part of the model's parameter space to never be extracted by some clients, making it impossible to train these parameters on the data of these clients; (2) Rolling extraction (such as the FedRolex algorithm in reference [3], a model heterogeneous federated learning algorithm based on rolling extraction) has a slow parameter space coverage speed and the frequency of different parameters being extracted (trained) is uneven; (3) The existing random extraction scheme is a simple random extraction with replacement (such as the FD-m algorithm in reference [4], a model heterogeneous federated learning algorithm based on simple random extraction that has been uniformly improved with residual connections), which cannot guarantee the integrity of parameter space coverage. The low parameter space coverage efficiency directly affects the convergence speed and final performance of the global model.

[0005] (2) Poor convergence stability: Existing methods exhibit poor convergence stability when faced with dual data and model heterogeneity. Data heterogeneity leads to client drift, making it difficult for the global model to reach an optimal solution. Furthermore, since the parameters of heterogeneous sub-models are only a portion of the global model, the mapping space they fit differs from that of the complete global model, further exacerbating convergence instability. This dual heterogeneity of data and sub-models leads to unstable convergence of the global model, which directly affects the final performance of the global model.

[0006] These shortcomings limit the convergence speed, final performance, and applicability of the global model in real-world scenarios. Therefore, it is imperative to provide a federated learning approach that can efficiently cover the parameter space, improve convergence stability, and adapt to data heterogeneity and client model heterogeneity.

[0007] In view of this, the present invention is proposed.

[0008] References [1] McMahan B, Moore E, Ramage D, et al. Communication-efficient learning of deep networks from decentralized data [C] / / Artificial intelligence and statistics. PMLR, 2017: 1273-1282.

[0009] References[2]Diao E,Ding J,Tarokh V.Heterofl:Computation and communication efficient federated learning for heterogeneous clients[J].arXivpreprint arXiv:2010.01264,2020.

[0010] References [3] Alam S, Liu L, Yan M, et al. Fedrolex: Model-heterogeneous federated learning with rolling sub-model extraction [J]. Advances in neuralinformation processing systems, 2022, 35: 29677-29690.

[0011] References [4] Caldas S, J,McMahan HB,et al.Expanding the reach offederated learning by reducing client resource requirements[J].arXiv preprintarXiv:1812.07210,2018. Summary of the Invention

[0012] The purpose of the present invention is to provide a method, system and device for model heterogeneous federated learning with fast and stable convergence, which can efficiently cover the parameter space, improve convergence stability and adapt to data heterogeneity and client model heterogeneity, thereby solving the above-mentioned technical problems existing in the prior art.

[0013] The purpose of the present invention is achieved through the following technical solutions:

[0014] A fast and stable convergence method for heterogeneous federated learning of models, used in a federated learning system consisting of a server and multiple clients with different capacities, including:

[0015] Step 1, initialization step: The server initializes the global model parameters and creates a parameter buffer to record the unused neurons in each hidden layer of the global model;

[0016] Step 2: At the beginning of each round of training, the server randomly selects some or all clients to participate in the current round of training;

[0017] Step 3, sub-model allocation step: The server will extract a sub-model for each client participating in the training from the parameter buffer using random sampling without replacement, and send it to the client for local training;

[0018] Step 4, global aggregation step: After all participating clients complete local training, the server collects the sub-model parameters uploaded by each client and uses the global aggregation step decay mechanism to perform weighted average aggregation to obtain the global model after this round of aggregation;

[0019] Step 5: If it is determined that the specified number of rounds has not been reached or the global model performance has converged, then loop through steps 2 to 4 for multiple rounds of training, otherwise terminate.

[0020] A fast and stable convergence model heterogeneous federated learning system that implements the fast and stable convergence model heterogeneous federated learning method described in the present invention is composed of a server and multiple clients with different capacities, wherein the server is respectively connected to each client, can initialize global model parameters, and create a parameter buffer to record unused neurons in each hidden layer of the global model; and at the beginning of each round of training, randomly select some or all clients to participate in the current round of training until a specified number of rounds is reached or the global model performance converges; based on the capacity of each client participating in the training, a sub-model is extracted from the parameter buffer using random sampling without replacement for the client, and sent to the corresponding participating client for local training; the sub-model parameters uploaded by all participating clients after completing local training are collected and weighted averaged using a global aggregation step decay mechanism to obtain the global model after the current round of aggregation, and multiple rounds of training are repeated until a specified number of rounds is reached or the global model performance converges;

[0021] After receiving the sub-model assigned by the server, each client can use its own local data to train the sub-model. After the local training is completed, the sub-model parameters are uploaded back to the server for global aggregation.

[0022] A processing device comprising:

[0023] at least one memory for storing one or more programs;

[0024] At least one processor is capable of executing one or more programs stored in the memory. When the one or more programs are executed by the processor, the processor is enabled to implement the method described in the present invention.

[0025] Compared with the existing technology, the method, system and device for fast and stable convergence of heterogeneous federated learning models provided by the present invention have the following beneficial effects:

[0026] By enabling the server to adopt a random sampling without replacement strategy for sub-model extraction, clients of different capacities can train different parts of the global model parameters in different rounds, ensuring that the sub-models trained by clients of different capacities can efficiently cover the parameter space of the global model, improving the parameter space coverage efficiency, and thus improving the convergence speed and final performance of the global model; in the global aggregation step, by introducing a global aggregation step attenuation mechanism, the global model is allowed to accept more updates from sub-models in the early stage of training to quickly improve the global model performance, and reduce the degree of acceptance of sub-model updates in the later stage, which can alleviate the client drift problem caused by data heterogeneity and sub-model heterogeneity, improve convergence stability, and thus improve the convergence stability and final performance of the global model. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0028] Figure 1 A flowchart of a model heterogeneous federated learning method with fast and stable convergence provided by an embodiment of the present invention.

[0029] Figure 2 Schematic diagram of extracting a single hidden layer sub-model for the model heterogeneous federated learning method provided in an embodiment of the present invention.

[0030] Figure 3 Schematic diagram of maintaining consistent neuron indexes in the two layers before and after the residual connection during the sub-model extraction process of the model heterogeneous federated learning method provided in an embodiment of the present invention.

[0031] Figure 4 Comparison of the global top-1 test accuracy curves of the model heterogeneous federated learning method (FedEcover) provided in an embodiment of the present invention and other existing methods under different communication rounds on the CIFAR-10 dataset in a scenario with 10 clients (all participating in each round).

[0032] Figure 5 Comparison of the global top-1 test accuracy curves of the model heterogeneous federated learning method (FedEcover) provided in an embodiment of the present invention and other existing methods under different communication rounds on the CIFAR-100 dataset in a scenario with 10 clients (all participating in each round).

[0033] Figure 6 Comparison of the global top-1 test accuracy curves of the model heterogeneous federated learning method (FedEcover) provided in an embodiment of the present invention and other existing methods under different communication rounds on the Tiny ImageNet dataset in a scenario with 10 clients (all participating in each round).

[0034] Figure 7 Comparison of the global top-1 test accuracy curves of the model heterogeneous federated learning method (FedEcover) provided in an embodiment of the present invention and other existing methods under different communication rounds on the CIFAR-10 dataset in a scenario with 100 clients (20% participation in each sampling round).

[0035] Figure 8Comparison of the global top-1 test accuracy curves of the model heterogeneous federated learning method (FedEcover) provided in an embodiment of the present invention and other existing methods under different communication rounds on the CIFAR-100 dataset in a scenario with 100 clients (20% participation in each sampling round).

[0036] Figure 9 Comparison of the global top-1 test accuracy curves of the model heterogeneous federated learning method (FedEcover) provided in an embodiment of the present invention and other existing methods under different communication rounds on the TinyImageNet dataset in a scenario with 100 clients (20% participation in each sampling round).

[0037] Figure 10 The convergence accuracy and speedup ratio of the model heterogeneous federated learning method (FedEcover) provided in the embodiment of the present invention are compared with other existing methods under different degrees of data heterogeneity on the CIFAR-100 dataset with all 10 clients participating in each round.

[0038] Figure 11 The convergence accuracy and acceleration ratio of the model heterogeneous federated learning method (FedEcover) provided in the embodiment of the present invention are compared with other existing methods on the CIFAR-100 dataset with 100 clients sampling 20% ​​per round and different degrees of data heterogeneity. DETAILED DESCRIPTION

[0039] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the specific content of the present invention. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments, and do not constitute a limitation of the present invention. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0040] First, the following terms may be used in this article:

[0041] The term “and / or” means that either or both of them can be realized at the same time. For example, X and / or Y includes both “X” or “Y” and “X and Y”.

[0042] The terms "include," "comprises," "contains," "has," or other similar expressions should be interpreted as non-exclusive. For example, "including certain technical features (such as raw materials, components, ingredients, carriers, dosage forms, materials, dimensions, parts, components, mechanisms, devices, steps, procedures, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products, or manufactured articles, etc.) should be interpreted as including not only the technical features explicitly listed, but also other technical features known in the art that are not explicitly listed.

[0043] The term "consisting of" excludes any technical features not explicitly listed. If used in a claim, this term renders the claim closed, excluding any technical features other than those explicitly listed, except for conventional impurities associated with them. If this term appears only in a clause of a claim, it limits only the elements explicitly listed in that clause; elements listed in other clauses are not excluded from the claim as a whole.

[0044] Unless otherwise specified or limited, the terms "mounted," "connected," "connect," and "fixed" should be interpreted broadly. For example, they can refer to fixed, detachable, or integral connections; mechanical or electrical connections; direct or indirect connections through an intermediary; and internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in this document based on specific circumstances.

[0045] The terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings and are only for the convenience and simplification of description, and do not explicitly or implicitly indicate that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore should not be understood as a limitation to this document.

[0046] The scheme provided by the present invention is described in detail below. The contents not described in detail in the examples of the present invention belong to the prior art known to professionals in this field. If specific conditions are not specified in the examples of the present invention, they are carried out according to conventional conditions in the field or conditions recommended by the manufacturer. If the manufacturer of the reagents or instruments used in the examples of the present invention is not specified, they are all conventional products that can be purchased commercially.

[0047] like Figure 1As shown, an embodiment of the present invention provides a fast and stable convergence model heterogeneous federated learning method for a federated learning system consisting of a server and multiple clients with different capacities, including:

[0048] Step 1, initialization step: The server initializes the global model parameters and creates a parameter buffer to record the unused neurons in each hidden layer of the global model;

[0049] Step 2: At the beginning of each round of training, the server randomly selects some or all clients to participate in the current round of training;

[0050] Step 3, sub-model allocation step: The server will extract a sub-model for each client participating in the training from the parameter buffer using random sampling without replacement, and send it to the client for local training;

[0051] Step 4, global aggregation step: After all participating clients complete local training, the server collects the sub-model parameters uploaded by each client and uses the global aggregation step decay mechanism to perform weighted average aggregation to obtain the global model after this round of aggregation;

[0052] Step 5: If it is determined that the specified number of rounds has not been reached or the global model performance has converged, then loop through steps 2 to 4 for multiple rounds of training, otherwise terminate.

[0053] Preferably, in the above method, each client participating in the training trains the sub-model assigned to it by the server in the following manner, including:

[0054] Step 31: After receiving the sub-model assigned by the server, each client participating in the training trains the sub-model using its own local data.

[0055] Step 32: After the local training is completed, the client uploads the sub-model parameters back to the server, which performs global aggregation.

[0056] Preferably, in step 2 of the above method, the capacity of the client refers to the device resources of the client, which include computing power, storage capacity and bandwidth; the capacity of the client is expressed by parameter c i Indicates that 0 <c i <1, which means the ratio of the number of neurons in each hidden layer supported by the client to the number of neurons in the corresponding layer of the global model.

[0057] Preferably, in step 2 of the above method, the server extracts a sub-model for each client from the parameter buffer using random sampling without replacement according to the capacity of each client in the following manner, including:

[0058] For each client, randomly select neurons from the unused parameter buffer in proportions that match the capacity of the client to build the sub-model layer by layer;

[0059] Refill the parameter buffer when it is empty to ensure that every neuron is selected evenly;

[0060] During the sub-model extraction process, for the layers with residual connections, the neuron index of the previous layer in the residual connection is used as the sub-model extraction index of the next layer, so that the neurons in each layer in the residual connection correspond to each other.

[0061] Preferably, in step 4 of the above method, the server collects the sub-model parameters uploaded by each client and adopts a global aggregation step decay mechanism to perform weighted average aggregation in the following manner, including:

[0062] By adjusting the global aggregation step η during the aggregation process g , to control the degree of acceptance of the aggregation process for updates from heterogeneous sub-models. The aggregation formula for each sub-model parameter is:

[0063]

[0064] in, are the global model parameters at the beginning of the t+1th round; the superscript t indicates the tth round of training; is the global model parameter at the beginning of the tth round of training; represents the set of clients participating in the tth round of training; represents the parameter change of client i after the local training is completed in the tth round of training; p i Represents the weight of client i in weighted averaging.

[0065] The embodiment of the present invention also provides a model heterogeneous federated learning system with fast and stable convergence, which consists of a server and multiple clients with different capacities, wherein:

[0066] The server is connected to each client for communication, and can initialize the global model parameters and create a parameter buffer to record the unused neurons in each hidden layer of the global model. At the beginning of each round of training, it randomly selects some or all of the clients to participate in the current round of training until the specified number of rounds is reached or the global model performance converges. Based on the capacity of each client participating in the training, it uses random sampling without replacement to extract a sub-model for the client from the parameter buffer and sends it to the corresponding client participating in the training for local training. It collects the sub-model parameters uploaded by all participating clients after completing local training and uses a global aggregation step decay mechanism for weighted average aggregation to obtain the global model after the current round of aggregation. Multiple rounds of training are repeated until the specified number of rounds is reached or the global model performance converges.

[0067] After receiving the sub-model assigned by the server, each client can use its own local data to train the sub-model. After the local training is completed, the sub-model parameters are uploaded back to the server for global aggregation.

[0068] Preferably, in the above system, the capacity of the client refers to the device resources of the client, which include computing power, storage capacity and bandwidth; the capacity of the client is expressed by parameter c i Indicates that 0 <c i <1, which means the ratio of the number of neurons in each hidden layer supported by the client to the number of neurons in the corresponding layer of the global model.

[0069] Preferably, in the above system, the server extracts a sub-model for each client from the parameter buffer using random sampling without replacement according to the capacity of each client in the following manner, including:

[0070] For each client, randomly select neurons from the unused parameter buffer in proportions that match the capacity of the client to build the sub-model layer by layer;

[0071] Refill the parameter buffer when it is empty to ensure that every neuron is selected evenly;

[0072] During the sub-model extraction process, for the layers with residual connections, the neuron index of the previous layer in the residual connection is used as the sub-model extraction index of the next layer, so that the neurons in each layer in the residual connection correspond to each other.

[0073] Preferably, in the above system, the server collects the sub-model parameters uploaded by each client and adopts a global aggregation step decay mechanism to perform weighted average aggregation in the following manner, including:

[0074] By adjusting the global aggregation step η during the aggregation process g , to control the degree of acceptance of the aggregation process for updates from heterogeneous sub-models. The aggregation formula for each sub-model parameter is:

[0075]

[0076] in, are the global model parameters at the beginning of the t+1th round; the superscript t indicates the tth round of training; is the global model parameter at the beginning of the tth round of training; represents the set of clients participating in the tth round of training; represents the parameter change of client i after the local training is completed in the tth round of training; p i Represents the weight of client i in weighted averaging.

[0077] The embodiment of the present invention further provides a processing device, comprising:

[0078] at least one memory for storing one or more programs;

[0079] At least one processor is capable of executing one or more programs stored in the memory. When the one or more programs are executed by the processor, the processor is enabled to implement the method described in the present invention.

[0080] In summary, the fast and stable convergence model heterogeneous federated learning method and system provided by the embodiments of the present invention enables the server to adopt a sub-model extraction method with a random sampling strategy without replacement, so that clients of different capacities train different parts of the global model parameters in different rounds, ensuring that the sub-models trained by clients of different capacities can efficiently cover the parameter space of the global model, improve the parameter space coverage efficiency, and thus improve the convergence speed and final performance of the global model; in the global aggregation step, by introducing the global aggregation step attenuation mechanism, the global model is allowed to accept more updates from the sub-model in the early stage of training to quickly improve the global model performance, and reduce the degree of acceptance of sub-model updates in the later stage, which can alleviate the client drift problem caused by data heterogeneity and sub-model heterogeneity, improve convergence stability, and thus improve the convergence stability and final performance of the global model.

[0081] In order to more clearly demonstrate the technical solution and technical effects provided by the present invention, the solution provided by the embodiment of the present invention is described in detail with reference to specific embodiments below.

[0082] Example 1

[0083] like Figure 1 As shown, this embodiment provides a fast and stable convergence model heterogeneous federated learning method (referred to as FedEcover), which is used in a federated learning system consisting of a server and multiple clients with different capacities. This method achieves fast and stable convergence of global model federated learning under the dual challenges of data heterogeneity and model heterogeneity through efficient sub-model extraction and the introduction of a global step-size decay mechanism (GSD). Specifically, the method includes:

[0084] (1) Server-side processing:

[0085] Step 1, initialization phase: The server first initializes the global model parameters and creates a parameter buffer to record the neurons that have not been used in each hidden layer.

[0086] Step 2: At the beginning of each round of training: the server randomly selects some or all clients to participate in the current round of training.

[0087] Step 3, sub-model allocation phase: For each client participating in the training, the server will extract a sub-model for the client from the parameter buffer according to the client's capacity. The parameter extraction process uses random sampling without replacement. Specifically, a certain proportion of neurons (determined by the client's capacity) are randomly selected layer by layer for each client from the unused parameter buffer to build the sub-model layer by layer. When the parameter buffer is empty, it is refilled to ensure that each neuron can be evenly selected. This process is as follows Figure 2 In the sub-model extraction process, for the layers with residual connections, the neuron index of the previous layer in the residual connection is used as the sub-model extraction index of the next layer to ensure the correspondence between the neurons in each layer in the residual connection and avoid the problem of model performance degradation caused by inconsistent residual connections, such as Figure 3 shown.

[0088] Step 4, global aggregation phase: When all participating clients complete local training, the server collects the sub-model parameters uploaded by these clients for weighted average aggregation. During the aggregation process, a global aggregation step decay mechanism is adopted to adjust the global aggregation step size η g (equivalent to the learning rate) to control the degree of acceptance of the aggregation process to updates from heterogeneous sub-models. The aggregation formula is as follows:

[0089]

[0090] The superscript t indicates the tth round, represents the set of clients participating in round t, represents the parameter change of client i after the local training in round t, p i is the weight of client i in the weighted average. A larger aggregation step size is used in the early stages of federated training to quickly improve global model performance. A smaller aggregation step size is used in the later stages to reduce the impact of client drift on the global model and improve convergence stability.

[0091] Step 5: Repeat steps 2 to 5 above for multiple rounds of training until the specified number of rounds is reached or the global model performance converges.

[0092] (2) The client processes the sub-model assigned by the server as follows:

[0093] Step 31: The client trains the sub-model using its own local data.

[0094] Step 32: After local training is completed, the client uploads the sub-model parameters back to the server.

[0095] The above process loop is also executed for multiple rounds until the specified number of rounds is reached or the global model performance converges.

[0096] The model heterogeneous federated learning method of the present invention has many significant advantages, as follows:

[0097] (1) Improve the global model parameter space coverage efficiency and convergence speed:

[0098] (a) Technical Features: This invention employs a sub-model extraction scheme based on random sampling without replacement. By maintaining a buffer of unused neurons on the server side, it extracts a sub-model of appropriate size for each client based on its capacity. This scheme avoids missed training or uneven repeated training in the parameter space, thereby achieving efficient coverage of the global model parameter space and accelerating global model performance. Furthermore, this scheme increases the diversity of sub-models trained across different clients and rounds, exerting a regularization effect similar to that of model ensembles, helping to improve the generalization capability of the global model.

[0099] (b) Experimental Results: With all 10 clients participating, our method achieves comparable convergence accuracy to FedAvg on the CIFAR-10 dataset with only 10 rounds of communication, while FedAvg requires 250 rounds. On the CIFAR-100 dataset, our method requires only 11 rounds, while FedAvg requires 124 rounds. On the Tiny ImageNet dataset, our method requires only 5 rounds, while FedAvg requires 109 rounds. On all three datasets, our method requires fewer rounds than existing heterogeneous models based on sub-model extraction (i.e., HeteroFL, FedRolex, and FD-m). This demonstrates that our method converges faster. Similarly, with a total of 100 clients and 20% random sampling per round, our method also achieves the best speedup on all three datasets, as shown in Tables 1 and 2.

[0100] Table 1 shows the communication rounds (speedup ratio) required for different methods to achieve the same accuracy as the average convergence accuracy of FedAvg when all 10 clients participate.

[0101]

[0102] Where "\" indicates that the method never reaches the target value.

[0103] Table 2 shows the communication rounds (speedup ratio) required for different methods to achieve the same accuracy as the average accuracy of FedAvg when 20% of the 100 clients are randomly selected to participate.

[0104]

[0105] Where "\" indicates that the method never reaches the target value.

[0106] (2) Improve convergence stability and final performance:

[0107] (a) Technical Features: The method of the present invention introduces a global aggregate step-size decay mechanism (GSD), which allows the global model to receive more updates from heterogeneous sub-models in the early stages of training to quickly improve accuracy. In the later stages of training, the update amplitude is reduced to mitigate client drift caused by data heterogeneity and sub-model heterogeneity.

[0108] (b) Experimental results: Figures 4 to 9 As shown, from the curve of the accuracy changing with the rounds during the federated training process, the curve of the method of the present invention is more stable and has less fluctuation, indicating that its convergence stability has been significantly improved, effectively solving the problem of unstable convergence of traditional methods in complex scenarios. Through an efficient sub-model extraction scheme and a global aggregation step decay mechanism, the global model accuracy of the method of the present invention on multiple models and datasets is superior to the existing model heterogeneous methods based on sub-model extraction and the basic FedAvg method, and can better utilize the data of clients with different device capabilities to learn a more expressive global model. As shown in Table 3, on the CIFAR-10 dataset, the global model of the method of the present invention has a final converged top-1 test accuracy of 75.25% when all 10 clients participate, which is a significant improvement over the traditional FedAvg method (59.63%). On the CIFAR-100 dataset, the accuracy of the method of the present invention is 41.64%, while FedAvg is only 33.38%. On the Tiny ImageNet dataset, the proposed method achieved an accuracy of 26.84% and a FedAvg of 21.79%. Furthermore, the proposed method outperformed existing heterogeneous model methods based on sub-model extraction (i.e., HeteroFL, FedRolex, and FD-m) on these datasets. Similarly, with a total of 100 clients and 20% random sampling per round, the proposed method also achieved the best accuracy test performance on the three datasets, as shown in Table 4.

[0109] Table 3 shows the global top-1 test accuracy (%) when all 10 clients participate.

[0110]

[0111] Table 4 shows the global top-1 test accuracy (%) when 20% of the 100 clients are randomly selected to participate.

[0112]

[0113] (3) Adapting to different data distributions and client model heterogeneity:

[0114] (a) Technical Features: This method adapts to the heterogeneous data distribution and models of different clients through efficient sub-model extraction and global aggregation mechanisms. Sub-model extraction is based on the client's device capabilities, ensuring that each client can perform effective local training within its resource constraints. Global aggregation comprehensively considers updates from different clients, enabling the global model to accommodate clients with different data distributions and device capabilities.

[0115] (b) Experimental results: Figure 10 and Figure 11 As shown in the figure, under various data heterogeneity levels and two client number distributions, in most cases, the method of the present invention outperforms the basic FedAvg method and the existing model heterogeneity method based on sub-model extraction in terms of test accuracy and acceleration ratio, showing good adaptability and being able to effectively cope with complex challenges in real-world scenarios.

[0116] Example 2

[0117] The present invention relates to a model heterogeneity framework for federated learning. The present invention aims to address data heterogeneity and client device capability heterogeneity, achieving rapid and stable global model convergence through an efficient sub-model extraction scheme and a global aggregation step-size decay mechanism. The following are specific examples of the present invention:

[0118] (1) System initialization:

[0119] (11) Client settings: Assume that there are N (e.g., 10, 100) clients in the system, each of which has different computing power, storage capacity, bandwidth and other equipment resources (collectively referred to as "capacity"). The capacity of the client is determined by the parameter c i (0 <c i <1) indicates the ratio of the number of neurons in each hidden layer supported by each client to the number of neurons in the corresponding layer of the global model.

[0120] (12) Data distribution: The data distribution of clients is non-independent and identically distributed (Non-IID), meaning that data from different clients have different distribution characteristics. The imbalance in data distribution is generated using Dirichlet distribution, where each client has a different proportion of samples assigned to different categories.

[0121] (13) Model initialization: global model parameters θ g Initialized to random values.

[0122] (2) Sub-model extraction:

[0123] (21) Buffer maintenance: Maintain a buffer B on the server = {B1, B2, ..., B L}(L is the number of hidden layers in the global model), which is used to record the unused neuron indexes in each hidden layer.

[0124] (22) Sub-model extraction algorithm: For each client i, at the beginning of each round of training, according to its capacity c i , randomly select a certain proportion of neuron indexes from each layer buffer to construct a sub-model θ i The specific steps are as follows:

[0125] (211) For each layer l (1≤l≤L), calculate the number of neurons n that client i needs to extract in this layer i,l =c i ·n g,l , where n g,l is the total number of neurons in the lth layer of the global model.

[0126] (212) If buffer B l The number of neurons remaining in is less than n i,l , then B l All neuron indices in B are assigned to client i, and B is refilled with all neuron indices of this layer of the global model l .

[0127] (213) From B l Randomly select the remaining required number of neuron indices and add them to the sub-model index set S of client i i,l (If there is a residual connection in the model, it is necessary to keep the neuron indexes selected by the two layers before and after the residual connection consistent).

[0128] (214) According to S i,l Extract the parameter elements in the corresponding global model and construct the sub-model θ of client i i .

[0129] (3) Local training: Each client i uses its local data D i Pair model θ i Perform local training and update the sub-model parameters using an optimization algorithm (such as Adam). Set hyperparameters such as the number of iterations and learning rate for local training based on actual conditions. After training is complete, upload the sub-model parameters.

[0130] (4) Parameter aggregation:

[0131] (41) Sparse aggregation mechanism: The server updates the global model based on the sub-model parameters uploaded by the client.

[0132] For each parameter w g, only aggregates the updated values ​​of clients that contain this parameter. The specific formula is:

[0133]

[0134] The superscript t indicates the tth round, Indicates that the t-th round neutron model contains parameter w g The client collection, Represents the parameter w after client i completes local training in round t g value.

[0135] (42) Global aggregation step attenuation mechanism: Introducing the global aggregation step hyperparameter η during the aggregation process g , and adopts a decay mechanism. The initial value is 1.0, and it decays once every certain number of rounds T (such as 10 rounds), with a decay coefficient of γ (0<γ≤1). After decay, η g Used to control the magnitude of global model updates and reduce the impact of client drift on global model performance in the later stages of federated training.

[0136] (5) Experimental plan:

[0137] (51) Dataset selection: Experiments were conducted using image datasets such as CIFAR-10, CIFAR-100, and Tiny ImageNet.

[0138] (52) Model architecture: Specifically, two model architectures, CNN and ResNet-18, were used for the experiment. The CNN model contains three convolutional layers and one fully connected layer. The ResNet-18 model uses the implementation in the torchvision library and replaces the default batch normalization layer with a static batch normalization layer.

[0139] (53) Experimental setup: Two experimental scenarios are set, namely, a small number of clients with full participation (10 clients, all participating in each round) and a large number of clients with sampling (100 clients, 20% randomly selected to participate in each round). The degree of data heterogeneity is controlled by setting the α parameter of the Dirichlet distribution, and the model heterogeneity is controlled by customizing different c i The value is set based on the ratio of the total number of clients, for example, {1.0:5%, 0.75:10%, 0.5:15%, 0.25:20%, 0.1:50%}. A reference hyperparameter setting is shown in Table 5.

[0140] (54) Results evaluation: The classification accuracy and convergence performance of the global model are evaluated using the global test set. The model is compared with FedAvg, HeteroFL, FedRolex, and FD-m.

[0141] Table 5 shows the experimental hyperparameter settings.

[0142]

[0143] (6) Application Examples

[0144] (61) Federated Learning on Mobile Devices: In mobile edge networks, the computing and storage capabilities of different mobile devices vary significantly. The proposed method can extract sub-models based on the device capacity for local training, fully utilizing the data resources of each device and improving the performance of the global model.

[0145] (62) Federated learning of medical data: In the medical field, data distribution varies greatly among different hospitals or medical institutions, and data privacy requirements are high. The method of the present invention can achieve cross-institutional federated learning of medical data while protecting data privacy, thereby building a more accurate disease diagnosis model.

[0146] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims. The information disclosed in the background technology section of this article is only intended to deepen the understanding of the overall background technology of the present invention, and should not be regarded as an admission or any form of implication that the information constitutes prior art already known to those skilled in the art.

Claims

1. A fast and stable convergence model heterogeneous federated learning method, characterized by: Used in federated learning systems consisting of a server and multiple clients with different capacities, including: Step 1, initialization step: The server initializes the global model parameters and creates a parameter buffer to record the unused neurons in each hidden layer of the global model; Step 2: At the beginning of each round of training, the server randomly selects some or all clients to participate in the current round of training; Step 3, sub-model allocation step: The server will extract a sub-model for each client participating in the training from the parameter buffer using random sampling without replacement, and send it to the client for local training; Step 4, global aggregation step: After all participating clients complete local training, the server collects the sub-model parameters uploaded by each client and uses the global aggregation step decay mechanism to perform weighted average aggregation to obtain the global model after this round of aggregation; Step 5: If it is determined that the specified number of rounds has not been reached or the global model performance has converged, then loop through steps 2 to 4 for multiple rounds of training, otherwise terminate.

2. The fast and stable convergence model heterogeneous federated learning method according to claim 1 is characterized in that: In the method, each client participating in the training trains the sub-model assigned to it by the server in the following manner, including: Step 31: After receiving the sub-model assigned by the server, each client participating in the training trains the sub-model using its own local data. Step 32: After the local training is completed, the client uploads the sub-model parameters back to the server, which performs global aggregation.

3. The fast and stable convergence model heterogeneous federated learning method according to claim 1 or 2, characterized in that: In step 2, the client's capacity refers to the device resources of the client, which include computing power, storage capacity and bandwidth. The client's capacity is expressed by parameter c. i Indicates that 0 <c i <1, which means the ratio of the number of neurons in each hidden layer supported by the client to the number of neurons in the corresponding layer of the global model.

4. The fast and stable convergence model heterogeneous federated learning method according to claim 3 is characterized in that: In step 2, the server extracts a sub-model for each client from the parameter buffer using random sampling without replacement according to the capacity of each client, including: For each client, randomly select neurons from the unused parameter buffer in proportions that match the capacity of the client to build the sub-model layer by layer; Refill the parameter buffer when it is empty to ensure that every neuron is selected evenly; During the sub-model extraction process, for the layers with residual connections, the neuron index of the previous layer in the residual connection is used as the sub-model extraction index of the next layer, so that the neurons in each layer in the residual connection correspond to each other.

5. The fast and stable convergence model heterogeneous federated learning method according to claim 1 or 2, characterized in that: In step 4, the server collects the sub-model parameters uploaded by each client and uses the global aggregation step decay mechanism to perform weighted average aggregation in the following manner, including: By adjusting the global aggregation step η during the aggregation process g , to control the degree of acceptance of the aggregation process for updates from heterogeneous sub-models. The aggregation formula for each sub-model parameter is: in, are the global model parameters at the beginning of the t+1th round; the superscript t indicates the tth round of training; is the global model parameter at the beginning of the tth round of training; represents the set of clients participating in the tth round of training; represents the parameter change of client i after the local training is completed in the tth round of training; p i Represents the weight of client i in weighted averaging.

6. A fast and stable convergence model heterogeneous federated learning system that implements the fast and stable convergence model heterogeneous federated learning method according to any one of claims 1 to 5, characterized in that: It consists of a server and multiple clients with different capacities, among which, The server is connected to each client for communication, and can initialize the global model parameters and create a parameter buffer to record the unused neurons in each hidden layer of the global model. At the beginning of each round of training, it randomly selects some or all of the clients to participate in the current round of training until the specified number of rounds is reached or the global model performance converges. Based on the capacity of each client participating in the training, it uses random sampling without replacement to extract a sub-model for the client from the parameter buffer and sends it to the corresponding client participating in the training for local training. It collects the sub-model parameters uploaded by all participating clients after completing local training and uses a global aggregation step decay mechanism for weighted average aggregation to obtain the global model after the current round of aggregation. Multiple rounds of training are repeated until the specified number of rounds is reached or the global model performance converges. After receiving the sub-model assigned by the server, each client can use its own local data to train the sub-model. After the local training is completed, the sub-model parameters are uploaded back to the server for global aggregation.

7. The model heterogeneous federated learning system with fast and stable convergence according to claim 6 is characterized in that: The client capacity refers to the device resources of the client, which include computing power, storage capacity and bandwidth. The client capacity is expressed by parameter c i Indicates that 0 <c i <1, which means the ratio of the number of neurons in each hidden layer supported by the client to the number of neurons in the corresponding layer of the global model.

8. The fast and stable convergence model heterogeneous federated learning system according to claim 7 is characterized in that: In the system, the server extracts a sub-model for each client from the parameter buffer using random sampling without replacement according to the capacity of the client, including: For each client, randomly select neurons from the unused parameter buffer in proportions that match the capacity of the client to build the sub-model layer by layer; Refill the parameter buffer when it is empty to ensure that every neuron is selected evenly; During the sub-model extraction process, for the layers with residual connections, the neuron index of the previous layer in the residual connection is used as the sub-model extraction index of the next layer, so that the neurons in each layer in the residual connection correspond to each other.

9. The fast and stable convergence model heterogeneous federated learning system according to claim 6 or 7, characterized in that: In the system, the server collects the sub-model parameters uploaded by each client and uses the global aggregation step decay mechanism to perform weighted average aggregation as follows: By adjusting the global aggregation step η during the aggregation process g , to control the degree of acceptance of the aggregation process for updates from heterogeneous sub-models. The aggregation formula for each sub-model parameter is: in, are the global model parameters at the beginning of the t+1th round; the superscript t indicates the tth round of training; is the global model parameter at the beginning of the tth round of training; represents the set of clients participating in the tth round of training; represents the parameter change of client i after the local training is completed in the tth round of training; p i Represents the weight of client i in weighted averaging.

10. A processing device, characterized in that include: at least one memory for storing one or more programs; At least one processor is capable of executing one or more programs stored in the memory, and when the one or more programs are executed by the processor, the processor is capable of implementing the method according to any one of claims 6 to 9.

Images