Federal learning method and device and medium
Optimizing the federated learning model through fine-grained decoupling and attention mechanisms, the problems of poor convergence and insufficient personalization of traditional methods under heterogeneous data are solved, and a stronger balance between personalization and global generalization capabilities is achieved.
Patent Information
- Application Number
- CN202510389836.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-31
- Publication Date
- 2025-07-25
AI Technical Summary
Traditional federated learning methods have poor convergence when processing highly heterogeneous data, lack personalized solutions, and are difficult to adapt to the specific diagnostic needs of each hospital, and the embedding of global information and personalized information in client data cannot be fully utilized.
By fine-grained decoupling of the global federated learning model, it is divided into global feature extractor, personalized feature extractor, global header and personalized header, combining attention mechanism and correlation alignment loss function, dynamically adjust feature weights, and enhance the model's adaptability and personalization ability to client data.
It improves the training accuracy and robustness of the model under non-independent and homogeneous data distribution, enhances the client's personalization ability and global generalization ability, and ensures that the personalized model performance of each client is optimal.
Smart Images
Figure CN120373419A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure belongs to the technical field of artificial intelligence, and particularly relates to a federated learning method, device, and medium. Background Art
[0002] In practical applications of federated learning, such as in medical diagnosis, the data of each medical institution is usually non-independent and identically distributed. Different hospitals have different patient groups and medical devices, so there is heterogeneity in the data characteristics of each institution. Through federated learning technology, multiple medical institutions can collaboratively train a diagnostic model without sharing patient data, thus ensuring that patient privacy is not leaked.
[0003] Traditional federated learning methods usually adopt a single global model and are difficult to adapt to the specific diagnostic needs of each hospital. Moreover, they are unbalanced. When dealing with highly heterogeneous data, traditional federated learning methods are difficult to fully adapt to the local data of each client and achieve high performance.
[0004] The problems existing in current federated learning methods are poor convergence when dealing with highly heterogeneous data and the lack of personalized solutions.
[0005] In response to the above problems, personalized federated learning methods have been proposed, which focus on creating customized models for each client rather than a single global model. Most personalized federated learning methods enhance the local personalization ability by aggregating the local personalized models trained on client data with the global model, that is, using the global model as the global information to enrich the client's local personalized model, that is, staying at the level of a complete model. Some personalized federated learning methods conduct more in-depth research and processing on the model, dividing a complete model into two parts: global and personalized, that is, decoupling the federated model, dividing the model into a personalized model and a global model. By sharing the global model through the server and retaining the personalized model locally on the client, the federated learning model can have both personalized ability and global generalization ability. However, the method of only separating the model still has deficiencies in dealing with the global information and personalized information of the client. Although the model parameters are generated based on data, since the model parameters are a high-level generalization and compression of the original data, it is difficult for them to completely retain and reflect the specific information in the data, resulting in the inability to fully utilize the embedding of global information and personalized information in the client data, and there are limitations in separating this information. Summary of the Invention
[0006] The present disclosure proposes a federated learning method, device, and medium to solve the above technical problems.
[0007] According to a first aspect of the present disclosure, a federated learning method is provided. The method includes: using a received global feature extractor, a personalized feature extractor, a global head, a personalized head, and a global attention mechanism module to overwrite a local model, wherein the global feature extractor, the personalized feature extractor, the global head, and the personalized head are obtained by fine-grained decoupling of the basic layer of the global federated learning model, and the global attention mechanism module is obtained from the global federated learning model; freezing the global feature extractor, updating the parameters of the personalized feature extractor of the local model, extracting feature information, and dynamically adjusting the weight allocation of the global feature and the personalized feature by using the global attention mechanism module, wherein the feature information is separated to obtain the global feature and the personalized feature; updating the parameters of the personalized head of the local model based on the personalized feature, and not updating the parameters of the global head of the local model; uploading the personalized feature extractor, the global attention mechanism module, the global head, and the personalized head with updated parameters to a central server, wherein the central server performs weighted average aggregation on the personalized feature extractor, the global attention mechanism module, the global head, and the personalized head uploaded by each client to update the global federated learning model, and the parameters of the global feature extractor of the global federated learning model are updated based on the feedback from the client.
[0008] In some embodiments, after not updating the parameters of the global head of the global model, it further includes: using a loss function to balance the correlation between the global feature and the personalized feature, wherein the loss function includes a cross-entropy loss function and a correlation alignment loss function.
[0009] In some embodiments, using the received global feature extractor to overwrite the global feature extractor and the personalized feature extractor of the local model, that is: Using the received global head to overwrite the global head of the local model, that is: Using the global attention mechanism module to overwrite the attention mechanism module of the local model, that is: θ k ←θ glob ; wherein, represents the personalized feature extractor of the local model, w fe represents the global feature extractor of the local model, w hd represents the global head of the local model, θ k represents the attention mechanism module of the local model, represents the global feature extractor, represents the global head, θ glob represents the global attention module.
[0010] In some embodiments, update the parameters according to wherein, w kdenotes the parameters to be optimized for the k-th local model, and η denotes the learning rate. denotes the gradient of the objective function g with respect to the parameter w k Updating the parameters of the personalized feature extractor of the local model includes: According to perform the update. denotes the personalized feature extractor of the local model, η denotes the learning rate, and f loss denotes the loss function of the local model.
[0011] In some embodiments, extracting the feature information and dynamically adjusting the weight allocation of the global feature and the personalized feature by using the global attention mechanism module, wherein separating the feature information to obtain the global feature and the personalized feature includes: (h g , h k ) ← θ k (h), where θ k (h) represents that the feature information is calculated through the global attention mechanism module, h g denotes the global feature, and h k denotes the personalized feature; denotes the change amount of the parameter θ k .
[0012] In some embodiments, updating the parameters of the personalized head of the local model includes: According to perform the update. denotes the parameter 's change amount.
[0013] In some embodiments, the central server performs weighted average aggregation on the global heads uploaded by each client, including:
[0014] According to perform the weighted average, where the local head after the weighted average, w hd denotes the global head of the local model, denotes the personalized head; According to perform the aggregation, where D k denotes the data volume used for training of the k local models, and D denotes the total data volume used for training of all local models.
[0015] According to a second aspect of the present disclosure, there is provided a federated learning device, including: a covering module, configured to cover a local model by using a received global feature extractor, a personalized feature extractor, a global head, a personalized head, and a global attention mechanism module, wherein the global feature extractor, the personalized feature extractor, the global head, and the personalized head are obtained by performing fine-grained decoupling on a basic layer of a global federated learning model, and the global attention mechanism module is obtained from the global federated learning model; an adjustment module, configured to freeze the global feature extractor, update parameters of the personalized feature extractor of the local model, extract feature information, and dynamically adjust the weight allocation of the global feature and the personalized feature by using the global attention mechanism module, wherein the feature information is separated to obtain the global feature and the personalized feature; a personalized head parameter update module, configured to update parameters of the personalized head of the local model based on the personalized feature, and not update parameters of the global head of the local model; an uploading module, configured to upload the personalized feature extractor, the global attention mechanism module, the global head, and the personalized head with updated parameters to a central server, wherein the central server performs weighted average aggregation on the personalized feature extractor, the global attention mechanism module, the global head, and the personalized head uploaded by each client, updates the global federated learning model, and parameters of the global feature extractor of the global federated learning model are updated based on client feedback.
[0016] According to a third aspect of the present disclosure, there is provided a federated learning device, including: a memory; and a processor coupled to the memory, the processor being configured to execute the federated learning method as described above based on instructions stored in the memory.
[0017] According to a fourth aspect of the present disclosure, there is provided a computer-readable storage medium, on which computer program instructions are stored, and when the instructions are executed by a processor, the federated learning method as described above is implemented.
[0018] By adopting the above technical solutions, the beneficial technical effects that can be achieved by the embodiments of the present disclosure are as follows: by performing fine-grained decoupling on the global federated learning model, a global feature extractor, a personalized feature extractor, a global head, and a personalized head are obtained, which are used for separately processing the global features and personalized features of client data, enhancing the adaptability of the model to non-independent synchronous data of each client; by dynamically adjusting the weights of the separation of the global features and personalized features of the client by means of the attention mechanism, the ability of the model to capture the specific needs of the client is enhanced, and the utilization efficiency of the local data features of the client is improved; in balancing the global generalization ability and the personalization ability of the client, the correlation alignment loss function is used to ensure that the performance of the personalized model of each client reaches the optimal, and at the same time, the generalization ability of the federated learning model is not lost. Description of the Drawings
[0019] The accompanying drawings, which form a part of the specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.
[0020] Referring to the accompanying drawings, the present disclosure can be more clearly understood from the following detailed description.
[0021] Figure 1 is a flowchart showing a federated learning method according to some embodiments of the present disclosure.
[0022] Figure 2 is a schematic diagram showing a model part for inference work according to some embodiments of the present disclosure.
[0023] Figure 3 is a schematic diagram showing the cooperation between client local training and server global aggregation according to some embodiments of the present disclosure.
[0024] Figure 4 is a block diagram showing a federated learning device according to some embodiments of the present disclosure.
[0025] Figure 5 is a block diagram showing a federated learning device according to some other embodiments of the present disclosure.
[0026] Figure 6 is a block diagram showing a computer system for implementing some embodiments of the present disclosure. Detailed Description of Specific Embodiments
[0027] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that: unless otherwise specifically stated, the relative arrangements of components and steps, numerical expressions and values set forth in these embodiments do not limit the scope of the present disclosure.
[0028] At the same time, it should be understood that, for the sake of convenience of description, the dimensions of the various parts shown in the drawings are not drawn in actual proportional relationship.
[0029] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or its uses.
[0030] Techniques, methods, and devices known to those of ordinary skill in the relevant art may not be discussed in detail, but where appropriate, such techniques, methods, and devices should be considered as part of the specification.
[0031] In all the examples shown and discussed here, any specific values should be understood as merely exemplary and not as a limitation. Thus, other examples of the exemplary embodiments may have different values.
[0032] It should be noted that like reference numerals and letters refer to like items in the following figures, and thus, once an item is defined in one figure, further discussion thereof is not required in subsequent figures.
[0033] Currently, in practical applications of federated learning, such as in medical diagnosis, the data of each medical institution is usually non-independent and identically distributed. Different hospitals have different patient groups and medical devices, so the data characteristics of each institution are heterogeneous. Through federated learning technology, multiple medical institutions can collaboratively train a diagnostic model without sharing patient data, thus ensuring that patient privacy is not leaked.
[0034] Traditional federated learning methods usually adopt a single global model and are difficult to adapt to the specific diagnostic needs of each hospital. Moreover, they are unbalanced. When dealing with highly heterogeneous data, traditional federated learning methods are difficult to fully adapt to the local data of each client and achieve high performance.
[0035] The problems existing in current federated learning methods are poor convergence when dealing with highly heterogeneous data and the lack of personalized solutions.
[0036] To address the above problems, personalized federated learning methods have been proposed, which focus on creating customized models for each client rather than a single global model. Most personalized federated learning methods enhance local personalization capabilities by aggregating the locally personalized models trained on client data with the global model, that is, using the global model as the global information to enrich the locally personalized models of clients, that is, staying at the level of a complete model. Some personalized federated learning methods conduct more in-depth research and processing on the model, dividing a complete model into two parts: global and personalized, that is, decoupling the federated model and dividing the model into a personalized model and a global model. By sharing the global model through the server and retaining the personalized model locally on the client, the federated learning model can have both personalized capabilities and global generalization capabilities. However, the method of only separating the model still has deficiencies in dealing with the global information and personalized information of clients. Although the model parameters are generated based on data, since the model parameters are highly generalized and compressed of the original data, it is difficult for them to completely retain and reflect the specific information in the data, resulting in the inability to fully utilize the embedding of global information and personalized information in client data and limitations in separating this information.
[0037] In view of this, the present disclosure proposes a federated learning method. By performing fine-grained decoupling on the global federated learning model, a global feature extractor, a personalized feature extractor, a global head, and a personalized head are obtained, which are used for separately processing the global features and personalized features of client data, enhancing the adaptability of the model to the non-independent and synchronous data of each client; by means of an attention mechanism, the weights of the separation of the global features and personalized features of the client are dynamically adjusted, enhancing the model's ability to capture the specific needs of the client and improving the utilization efficiency of the local data features of the client; in terms of balancing the global generalization ability and personalized ability of the client, a correlation alignment loss function is used to ensure that the performance of the personalized model of each client reaches the optimal, without losing the generalization ability of the federated learning model.
[0038] In the present disclosure, a personalized federated learning optimization method combining an attention mechanism, fine-grained decoupling of a federated learning model, and a correlation alignment loss function is proposed, realizing a more stable personalized federated learning method, greatly improving the accuracy and robustness of the model training under non-independent and identically distributed data, and enhancing the client personalization ability and global generalization ability of the federated learning model.
[0039] The last connection layer of the federated learning model is used as the model head, and another part of the basic layer of the model is used as the feature extractor, and further fine-grained decoupling is performed, divided into a global feature extractor, a personalized feature extractor, a global head, and a personalized head. A separable attention mechanism module is added to separate the global feature information and personalized feature information of the client data, and the correlation alignment loss function is used to balance the global information and personalized information of the client single, forming an overall architecture of a federated learning model that works in coordination with each other. When the client communicates with the central server, only the personalized feature server, the global head, the personalized head, and the attention mechanism module are uploaded to the central server for aggregation to form a new federated learning model.
[0040] The central server is used to initialize the global federated model parameters, receive the local model weight parameters uploaded by the client, and send the aggregated global model parameters to the edge client.
[0041] The client is used to receive the global model parameters sent by the central server and perform local training on the client, including a global feature extractor, a global head, and a global attention mechanism module. The global feature extractor is used to overwrite the global feature extractor and personalized feature extractor of the client local, the global head is used to overwrite the global head of the client local (local model), the global attention module is used to overwrite the local attention module, and the personalized head remains unchanged to retain the personalized information of the client.
[0042] When training a local federated learning model on the client side, where: The global feature extractor is shared by all clients and is responsible for extracting feature information common across clients, reflecting potential common patterns and cross-domain knowledge in the dataset, such as edge or texture features in an image classification task, and remains frozen during client training. This module integrates the parameter updates fed back by clients through a server aggregation algorithm to maintain generalization ability in a distributed environment; The personalized feature extractor is customized for each client, and its parameters are continuously iteratively updated according to the client data features during local client training, focusing on extracting specific features of the local data distribution, which is closely related to the unique task requirements or data characteristics of the client, such as regional differences in user behavior in different regions, giving the model flexibility to adapt to specific needs; The attention mechanism module serves as the dynamic adjustment core. The attention mechanism module intelligently adjusts the weight allocation of global features and personalized features in prediction according to local data characteristics, and iteratively updates the model parameters during local training. This module is implemented based on the attention mechanism module and generates an adaptive weight vector by calculating the correlation or context dependence between features. When the local data is highly consistent with the global pattern, higher weights are preferentially assigned to global features. When the uniqueness of local data is significant, the contribution of personalized features is highlighted, thereby enhancing the adaptability and prediction accuracy of the model to heterogeneous data; The global head and the personalized head respectively perform distribution modeling of the prediction task based on global features and personalized features. The model parameters of the global head remain unchanged during local client training to retain global information, while the personalized head is continuously iteratively updated with local data. The global head uses shared features to provide consistent outputs across clients, enhancing generalization ability. The personalized head combines local features to meet specific task requirements. The two work together through weighted fusion to ensure the dual goals of the model's generalization of global knowledge and local personalized optimization.
[0043] Figure 1 is a flowchart showing a federated learning method according to some embodiments of the present disclosure. As Figure 1 shown, the federated learning method includes steps S101 to step S110.
[0044] In step S101, receive a global feature extractor, a personalized feature extractor, a global head, a personalized head, and a global attention mechanism module, where the global feature extractor, the personalized feature extractor, the global head, and the personalized head are obtained by fine-grained decoupling of the basic layer of the global federated learning model, and the global attention mechanism module is obtained from the global federated learning model.
[0045] As Figure 2 shown, the solid lines represent the model parts for inference work, including the personalized feature extractor global head w hd 、personalized head and attention mechanism module θ k. During the model prediction process, the frozen feature extractor w fe remains inactive and is only used for local learning during training. It combines with the correlation alignment loss function to balance the global generalization ability and personalization ability of the client's federated learning model. The model training process adopts a distributed optimization strategy of distributed iteration, as Figure 3 shown, which reflects the collaboration between the client's local training and the server's global aggregation, and communication occurs between the central server and the client.
[0046] Assign the initialized model parameters to the global model parameters of the first round, that is: w init represents the initialized model parameters, including the global feature extractor global head and the global attention mechanism module θ glob , represents the global model parameters of the first round.
[0047] In some embodiments, wherein, represents the parameters of the local model of the client, represents the global model parameters after r rounds of training.
[0048] In step S102, use the global feature extractor to overwrite the global feature extractor and the personalized feature extractor of the local model, use the global head to overwrite the global head of the local model, and use the global attention mechanism module to overwrite the attention mechanism module of the local model.
[0049]
[0050] Among them, represents the personalized feature extractor of the client (local model); w fe represents the global feature extractor of the client; w hd represents the global head of the client; θ k represents the attention (mechanism) module of the client; represents the personalized head of the client, which is randomly initialized by the client at the 0th round of training and has the same dimension as the global head. It remains unchanged during the stage when the client receives the global model sent by the server and only updates the parameters during the local training iteration of the client.
[0051] In step S103, initialize the personalized feature extractor of the local model and the personalized head of the local model.
[0052] In step S104, freeze the global feature extractor.
[0053] In step S105, according to the characteristics of the local dataset, the parameters of the personalized feature extractor of the local model are updated to extract feature information.
[0054] In step S106, the global attention mechanism module is used to separate the feature information to obtain global features and personalized features, and the global attention mechanism module is used to dynamically adjust the weight distribution of the global features and personalized features.
[0055] In step S107, based on the personalized features, the parameters of the personalized head of the local model are updated, and the parameters of the global head of the local model are not updated.
[0056] In step S108, the loss function is used to balance the correlation between the global features and the personalized features. Among them, the loss function includes the cross-entropy loss function and the correlation alignment loss function.
[0057] After the client receives these parameters, it uses the and of the server to initialize the local w fe 、 and w hd respectively. During the local training process, the parameters of w fe and w hd are frozen to keep them unchanged. The freezing strategy aims to protect the global model parameters from the influence of local training, so as to retain the global generalization information. At the same time, the client uses the local data to update the parameters of the personalized feature extractor and the personalized head to extract more personalized information that conforms to the characteristics of the client's own data, so as to capture the feature information specific to the local dataset. The above process fully reflects the flexibility in personalized federated learning. Among them, the attention module θ k intelligently separates the global and personalized features by analyzing the characteristics of the local data, and at the same time updates the parameters of the global feature extractor, the personalized feature extractor and the classification head to improve the model performance and adapt to local needs:
[0058] (h g , g k )←θ k (h),
[0059] h is the feature vector extracted from the data by the personalized feature extractor, and θ k (h) represents the calculation of the feature vector through the attention mechanism. h g represents the separated global feature, and h k represents the personalized feature of the client. Represents the global head prediction output of client k, out k Represents the final prediction result of the client model. w k Represents the parameters to be optimized for the model of the k-th client, and η represents the learning rate. Represents the objective function g with respect to the parameter w k Gradient.
[0060] Furthermore, it can be decomposed into
[0061] Among them, f loss Represents the loss function of the client model, cross-entropy loss function:
[0062] And the correlation alignment loss function: The sum is the loss function, and x k Represents the input data of the k-th client model; y k Represents the label of the input data of the k-th client model; out k Represents the label prediction of the client model for the input data of the k-th client. And f(x i，j ; w fe ) are the feature outputs of the input x i，j under the model and w fe respectively; is the covariance of the feature and f(x i，j ; w fe ); is the standard deviation of the feature and f(x i，j ; w fe ); N is the number of samples; M is the number of features.
[0063] In step S109, the personalized feature extractor, global attention mechanism module, global head, and personalized head with updated parameters are uploaded to the central server. Among them, the central server performs weighted average aggregation on the personalized feature extractor, global attention mechanism module, global head, and personalized head uploaded by each client, and updates the global federated learning model. The parameters of the global feature extractor of the global federated learning model are updated based on the client feedback.
[0064] Furthermore, Among them, is the global head w hd and the personalized head It is obtained by weighted average and is the local header finally uploaded by the client to the server; λ is a hyperparameter; model aggregation can be decomposed into: The server uses an aggregation algorithm to integrate the parameters of the global feature extractor, update the global model, and adjust the weights according to the data volume or task importance to ensure fairness.
[0065] In step S110, the above steps are repeatedly executed until the global federated learning model converges.
[0066] The above steps are iteratively executed, and the model performance is gradually improved through local optimization and global aggregation until the preset convergence conditions are reached, such as the loss function being stable or the accuracy meeting the standard.
[0067] The following provides the algorithm pseudocode for readers' reference.
[0068] Input: Client data D k ; Initialized global feature extractor Initialized global header Initialize the global attention module θ glob ; Client local model learning rate η; Client participation rate ρ; Total number of iterative training times T.
[0069] Output: Client personalized learning models w1,..., w N , global model w T .
[0070] / / Executed by the server; The server distributes θ glob , initialize the client global feature extractor w fe , personalized feature extractor Client global header w hd , attention module θ k ; For iteration t = 0,..., T do; The server randomly selects K clients participating in training according to ρ; The server distributes and to the selected clients; For Client k ∈ K in parallel do; / * Client k initializes the model: Use to overwrite w fe , while freezing w fe ; Use to overwrite w hd , while freezing w hd ; Use to overwrite θ k ; Generate a new vector v with client sample features k。* / / / Client training, update the client while updating Θ k ;
[0071]
[0072] Client i uploads θ k to the server; end for; / / Server aggregation; Server calculates D = ∑ k∈K D k , and obtains:
[0073] w t+1 ← {w fe，t+1 , w hd，t+1 , θ t+1}; end for;
[0074] return w T .
[0075] Figure 4 is a block diagram showing a federated learning device according to some embodiments of the present disclosure. As Figure 4 shown, the federated learning device includes a receiving module 401, a covering module 402, an initialization module 403, a freezing module 404, a first update module 405, an adjustment module 406, a second update module 407, a balancing module 408, an uploading module 409, and a repeated execution module 410.
[0076] The receiving module 401 is configured to receive a global feature extractor, a personalized feature extractor, a global head, a personalized head, and a global attention mechanism module, wherein the global feature extractor, the personalized feature extractor, the global head, and the personalized head are obtained by performing fine-grained decoupling on the basic layer of the global federated learning model, and the global attention mechanism module is obtained from the global federated learning model;
[0077] The covering module 402 is configured to use the global feature extractor to cover the global feature extractor and the personalized feature extractor of the local model, the global head to cover the global head of the local model, and the global attention mechanism module to cover the attention mechanism module of the local model;
[0078] The initialization module 403 is configured to initialize the personalized feature extractor of the local model and the personalized head of the local model;
[0079] The freezing module 404 is configured to freeze the global feature extractor;
[0080] The first update module 405 is configured to update the parameters of the personalized feature extractor of the local model according to the characteristics of the local data set, and extract feature information;
[0081] The adjustment module 406 is configured to separate the feature information by using the global attention mechanism module to obtain global features and personalized features, and dynamically adjust the weight distribution of the global features and the personalized features by using the global attention mechanism module;
[0082] The second update module 407 is configured to update the parameters of the personalized head of the local model based on the personalized features, and does not update the parameters of the global head of the local model;
[0083] The balance module 408 is configured to balance the correlation between the global features and the personalized features by using a loss function, where the loss function includes a cross-entropy loss function and a correlation alignment loss function;
[0084] The upload module 409 is configured to upload the personalized feature extractor, the global attention mechanism module, the global head, and the personalized head with updated parameters to the central server, where the central server performs weighted average aggregation on the personalized feature extractor, the global attention mechanism module, the global head, and the personalized head uploaded by each client, and updates the global federated learning model. The parameters of the global feature extractor of the global federated learning model are updated based on the client feedback;
[0085] The repeated execution module 410 is configured to repeatedly execute the above steps until the global federated learning model converges.
[0086] In this device, by performing fine-grained decoupling on the global federated learning model, a global feature extractor, a personalized feature extractor, a global head, and a personalized head are obtained for separately processing the global features and personalized features of the client data, enhancing the adaptability of the model to the non-independent synchronous data of each client; by using the attention mechanism to dynamically adjust the weights of the separation of the global features and personalized features of the client, enhancing the model's ability to capture the specific needs of the client and improving the utilization efficiency of the local data features of the client; in balancing the global generalization ability and the personalized ability of the client, by using the correlation alignment loss function, ensuring that the performance of the personalized model of each client reaches the optimal, and at the same time not losing the generalization ability of the federated learning model
[0087] Figure 5 It is a block diagram showing a federated learning device according to some other embodiments of the present disclosure.
[0088] Such as Figure 5As shown, the federated learning device 500 includes a memory 510; and a processor 520 coupled to the memory 510. The memory 510 is used to store instructions for implementing corresponding embodiments of the federated learning method. The processor 520 is configured to execute the federated learning method in any of the embodiments of the present disclosure based on the instructions stored in the memory 510.
[0089] Figure 6 is a block diagram showing a computer system for implementing some embodiments of the present disclosure. As Figure 6 shown, the computer system 600 may be in the form of a general-purpose computing device. The computer system 600 includes a memory 610, a processor 620, and a bus 630 connecting different system components.
[0090] The memory 610 may include, for example, a system memory, a non-volatile storage medium, etc. The system memory stores, for example, an operating system, application programs, a boot loader, and other programs. The system memory may include a volatile storage medium, such as a random access memory (RAM) and / or a cache memory. The non-volatile storage medium stores, for example, instructions for implementing corresponding embodiments of at least one of the business opportunity allocation methods. The non-volatile storage medium includes, but is not limited to, a disk memory, an optical memory, a flash memory, etc.
[0091] The processor 620 may be implemented in the form of a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates, or discrete hardware components such as transistors. Correspondingly, each of the modules such as a receiving module, a coverage module, an initialization module, a freezing module, a first update module, an adjustment module, a second update module, a balancing module, an uploading module, and a repeated execution module may be implemented by a central processing unit (CPU) running instructions for executing corresponding steps in the memory, or may be implemented by a dedicated circuit for executing corresponding steps.
[0092] The bus 630 may use any of a variety of bus structures. For example, the bus structure includes, but is not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, and a Peripheral Component Interconnect (PCI) bus.
[0093] The computer system 600 may further include an input / output interface 640, a network interface 650, a storage interface 660, etc. These interfaces 640, 650, 660, the memory 610, and the processor 620 may be connected through a bus 630. The input / output interface 640 may provide a connection interface for input / output devices such as a display, a mouse, and a keyboard. The network interface 650 provides a connection interface for various networking devices. The storage interface 660 provides a connection interface for external storage devices such as a floppy disk, a USB flash drive, and an SD card.
[0094] Here, various aspects of the present disclosure have been described with reference to the flowcharts and / or block diagrams of methods, apparatuses, and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of the blocks, can be implemented by computer-readable program instructions.
[0095] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable devices to produce a machine, such that the device implementing the functions specified in one or more blocks in the flowchart and / or block diagram is generated by the execution of the instructions by the processor.
[0096] These computer-readable program instructions can also be stored in a computer-readable memory, and these instructions cause the computer to work in a specific manner, thereby producing a manufactured article including instructions for implementing the functions specified in one or more blocks in the flowchart and / or block diagram.
[0097] The present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects.
[0098] So far, the federated learning method, apparatus, and medium according to the present disclosure have been described in detail. To avoid obscuring the concept of the present disclosure, some details known in the art have not been described. Those skilled in the art can fully understand how to implement the technical solutions disclosed herein based on the above description.
[0099] Although some specific embodiments of the present disclosure have been described in detail by way of examples, those skilled in the art should understand that the above examples are only for illustration and not for limiting the scope of the present disclosure. Those skilled in the art should understand that the above embodiments can be modified without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.
Claims
1. A federated learning method, characterized in that, The method includes: Using the received global feature extractor, personalized feature extractor, global head, personalized head, and global attention mechanism module to overwrite the local model, wherein the global feature extractor, personalized feature extractor, global head, and personalized head are obtained by fine-grained decoupling of the basic layer of the global federated learning model, and the global attention mechanism module is obtained from the global federated learning model; Freezing the global feature extractor, updating the parameters of the personalized feature extractor of the local model, extracting feature information, and dynamically adjusting the weight allocation of the global feature and the personalized feature by using the global attention mechanism module, wherein the feature information is separated to obtain the global feature and the personalized feature; Updating the parameters of the personalized head of the local model based on the personalized feature, and not updating the parameters of the global head of the local model; Uploading the personalized feature extractor, global attention mechanism module, global head, and personalized head with updated parameters to the central server, wherein the central server performs weighted average aggregation on the personalized feature extractor, global attention mechanism module, global head, and personalized head uploaded by each client, updates the global federated learning model, and the parameters of the global feature extractor of the global federated learning model are updated based on the feedback of the client.
2. The federated learning method according to claim 1, wherein After not updating the parameters of the global head of the global model, it further includes: using a loss function to balance the correlation between the global feature and the personalized feature, wherein the loss function includes a cross-entropy loss function and a correlation alignment loss function.
3. The federated learning method according to claim 1, wherein Use the received global feature extractor to overwrite the global feature extractor and the personalized feature extractor of the local model, i.e.: Use the received global head to overwrite the global head of the local model, i.e.: Use the global attention mechanism module to overwrite the attention mechanism module of the local model, i.e.: θ k ←θ glob ; where represents the personalized feature extractor of the local model, w fe represents the global feature extractor of the local model, w hd represents the global head of the local model, θ k represents the attention mechanism module of the local model, represents the global feature extractor, represents the global head, θ glob represents the global attention module.
4. The federated learning method according to claim 1, wherein According to update the parameters, where w k represents the parameters to be optimized for the k-th local model, η represents the learning rate, represents the gradient of the objective function g with respect to the parameter w k ; Updating the parameters of the personalized feature extractor of the local model includes: According to for updating represents the personalized feature extractor of the local model, η represents the learning rate, and f loss represents the loss function of the local model.
5. The federated learning method according to claim 4, wherein Extracting feature information, and dynamically adjusting the weight allocation of the global feature and the personalized feature by using the global attention mechanism module, wherein separating the feature information to obtain the global feature and the personalized feature includes: (h g ,h k ) ← θ k (h), θ k (h) indicates that the feature information is calculated through the global attention mechanism module, h g represents the global feature, h k represents the personalized feature; Indicates the change in the parameter θ k of.
6. The federated learning method according to claim 5, wherein Updating the parameters of the personalized head of the local model includes: Update according to and indicates the change in parameter .
7. The federated learning method according to claim 6, characterized in that The central server performing weighted average aggregation on the global heads uploaded by each client includes: According to perform weighted averaging, where the local head after weighted averaging, w hd represents the global head of the local model, represents the personalized head; According to perform aggregation, where D k represents the amount of data used for training k local models, and D represents the total amount of data used for training all local models.
8. A federated learning device, characterized in that, Including: An overwriting module for using the received global feature extractor, personalized feature extractor, global head, personalized head, and global attention mechanism module to overwrite the local model, wherein the global feature extractor, personalized feature extractor, global head, and personalized head are obtained by fine-grained decoupling of the basic layer of the global federated learning model, and the global attention mechanism module is obtained from the global federated learning model; An adjustment module for freezing the global feature extractor, updating the parameters of the personalized feature extractor of the local model, extracting feature information, and dynamically adjusting the weight allocation of the global feature and the personalized feature by using the global attention mechanism module, wherein the feature information is separated to obtain the global feature and the personalized feature; A personalized head parameter update module for updating the parameters of the personalized head of the local model based on the personalized feature, and not updating the parameters of the global head of the local model; An upload module, configured to upload the personalized feature extractor, global attention mechanism module, global head, and personalized head after parameter update to a central server, where the central server performs weighted average aggregation on the personalized feature extractor, global attention mechanism module, global head, and personalized head uploaded by each client respectively to update the global federated learning model, and the parameters of the global feature extractor of the global federated learning model are updated based on client feedback.
9. A federated learning device, characterized in that, Comprising: A memory; And A processor coupled to the memory, the processor being configured to execute the federated learning method according to any one of claims 1 to 7 based on instructions stored in the memory.
10. A computer-readable storage medium, characterized in that, Computer program instructions are stored thereon, and when the instructions are executed by a processor, the federated learning method according to any one of claims 1 to 7 is implemented.