Federal learning method and system for non-independent identically distributed data
By introducing a leader student model and a supernetwork feature selection mechanism into federated learning, combined with a self-distillation mechanism, the problems of slow model convergence and performance fluctuation under non-independent and identically distributed data are solved, and efficient and robust federated learning in heterogeneous data environments is achieved.
Patent Information
- Application Number
- CN202511438067.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-10
- Publication Date
- 2026-01-20
AI Technical Summary
Existing federated learning models suffer from slow convergence, large performance fluctuations, local optimum traps, and expanded attack surfaces under non-independent and identically distributed data. Furthermore, existing methods lack dynamic adaptability, have low computational efficiency, insufficient privacy protection, and inadequate theoretical guarantees.
By randomly selecting a leader student model in each round of communication, the supernetwork filters and integrates client features, and adopts a multilayer perceptron scoring and self-distillation mechanism to dynamically adjust feature importance and knowledge transfer, thereby achieving collaborative optimization of global and local knowledge.
It significantly improves the convergence speed and accuracy of federated learning on non-independent and identically distributed data, and provides robustness and stability, making it suitable for scenarios such as healthcare and the Internet of Things.
Smart Images
Figure CN121365710A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of data heterogeneity processing and system robustness enhancement in machine learning, and in particular to a federated learning method and system for non-independent and identically distributed data. BACKGROUND
[0002] With the in-depth application of artificial intelligence technology in medical, financial, Internet of Things and other fields, federated learning has become the core technology to solve the data island problem due to its data privacy protection characteristics. In the federated learning scenario, each participant trains the model on the local data and only shares the model parameters rather than the original data, thereby achieving a balance between data privacy protection and model collaborative training. However, a fundamental challenge in the actual deployment of federated learning is the non-independent and identically distributed (non-independent and identically distributed) data problem. Unlike the traditional machine learning assumption of independent and identically distributed data, the data in federated learning usually has significant distribution differences: 1. Vertical non-independent and identically distributed: different participants have different features of the same sample; 2. Horizontal non-independent and identically distributed: different participants have the same features of different samples; 3. Mixed non-independent and identically distributed: both feature and sample distribution differences exist. These uneven data distributions result in: slow model convergence: the gradient directions of each client are inconsistent, making it difficult to effectively aggregate the global model; significant performance fluctuations: the model performance differs greatly among different clients, making it difficult for the global model to take into account all participants; local optimal trap: easily trapped in local optimal solution, unable to achieve theoretical optimal performance; expanded attack surface: data distribution differences provide more operational space for malicious participants, etc.
[0003] Current research on non-independent and identically distributed problems is still relatively small, and existing methods still have the following limitations: first, insufficient static adaptability: most methods assume fixed data distribution, lacking the ability to respond to dynamic changes; second, low computational efficiency: complex adaptive mechanisms increase system computing and communication overhead; third, privacy-performance trade-off: some solutions require additional information exchange, which may compromise privacy protection goals; fourth, lack of theoretical guarantees: theoretical analysis and performance guarantees for extreme non-independent and identically distributed scenarios are still insufficient.
[0004] Therefore, constructing a federated learning framework with dynamic distribution awareness, high computational efficiency, and privacy protection characteristics, and systematically addressing the challenges posed by non-independent and identically distributed data, has important theoretical and practical value for promoting the reliable application of federated learning in real-world scenarios. This not only requires algorithm-level innovation, but also requires collaborative innovation in system design, communication optimization, and privacy protection mechanisms. SUMMARY
[0005] The embodiment of the application aims to provide a federated learning method and system for non-independent and identically distributed data, which realizes the cooperation of local data characteristics and global knowledge, is suitable for federated learning optimization in a data heterogeneous scene, and helps to improve the robustness and practical value of medical AI models in distributed training.
[0006] To achieve the above-mentioned purpose, the embodiment of the application provides a federated learning method for non-independent and identically distributed data, which comprises:
[0007] The server randomly selects one client as a leader student model in each round of communication, and other clients perform local model training on heterogeneous data sets;
[0008] The super network extracts features from the network models trained by each client, receives the local update embedding vectors sent by each client, and scores the importance of the features based on a multilayer perceptron;
[0009] The super network selects the top-k features of each client according to the importance score, and sends the filtered high-value features to the server;
[0010] After the server receives the top-k features of each client, it aligns the heterogeneous features of different clients layer by layer based on the network model hierarchical structure, and generates a comprehensive feature map by fusing the aligned features;
[0011] The server sends the fused feature map to the leader student model, and the leader student absorbs the aggregated knowledge and updates to the global model through a self-distillation mechanism;
[0012] The leader student model is converted into a normal client in the next round of communication to participate in local training, and its knowledge as a global model is contributed to the system.
[0013] Preferably, the server randomly selects one client as a leader student model in each round of communication, which comprises:
[0014] In the process of randomly selecting the leader student model, before the start of each round of federated learning training, a client is determined as the leader student model L0 from all local clients participating in data training through random sampling, wherein the model will undertake the core task of receiving global fused features and performing self-distillation to ensure the randomness and fairness of global knowledge transmission in the training process.
[0015] Preferably, the other clients perform local model training on heterogeneous data sets, which comprises:
[0016] The test client needs to fully participate in the T-round federated learning training process, and synchronously collect the feature parameters generated by the local model during data training; the loss of each client single-round training is calculated, and the loss function is based on the task demand to adopt the standard cross-entropy loss, and the expression is:
[0017]
[0018] Where B i is the batch size of the local sample of the client, w i is the local model parameter, f i (x k ; w i ) is the prediction output of the model to the sample x k , and y k is the real label of the sample; the model parameter is updated according to the gradient descent strategy, and the update formula is:
[0019]
[0020] The parameter change is calculated as:
[0021]
[0022] to reflect the parameter adjustment range of the local model during data training; the key parameters such as features and embedding vectors output by each client network model are uploaded to the super network model for subsequent processing.
[0023] Preferably, the super network extracts features from the network model trained by each client, receives the local update embedding vector sent by each client, and scores the importance of the features based on a multilayer perceptron; the super network selects the top-k features of each client according to the importance score, and sends the filtered high-value features to the server, including:
[0024] After the super network receives the features and embedding vectors sent by each client, the mean value of the features of different network layers of the client is calculated first, and for a three-dimensional feature map (channel number C, height H, width W), the mean value calculation formula is:
[0025]
[0026] Where is the feature map of the jth layer network of the ith client, is used to simplify the feature dimension and retain the key information in the layer; the embedding vector v i of the client is spliced with the feature mean value of the corresponding network layer to obtain the combined feature:
[0027]
[0028] The information fusion of the vector features and the layer mean features is implemented; the combined features after splicing are input into a multi-layer perception (MLP) in the super network, and the importance scores of the features are calculated, and the scoring formula is:
[0029]
[0030] wherein W1 and W2 are MLP weight matrices, b1 and b2 are bias terms, and σ is an activation function;
[0031] According to the scoring results, the top-k features with high importance in each client are selected, and the high-value features are uploaded to the server side.
[0032] Preferably, after the server receives the top-k features of each client, the heterogeneous features of different clients are aligned layer by layer based on the network model hierarchical structure, and the aligned features are fused to generate a comprehensive feature map, including:
[0033] After the server receives the high-value features sent by the super network, the features are classified and aligned according to the network level to which the features belong, and the features from different clients but belonging to the same network level are integrated into a feature set, and the expression is:
[0034]
[0035] wherein N is the number of clients participating in feature uploading, to ensure the dimension consistency of features in the same level; the aligned features are input into a feature fusion module, first performing linear combination on the multi-client features through a 1x1 convolution layer to reduce the feature channel dimension and enhance the cross-client feature correlation, and then sequentially performing a batch normalization (BN) operation and a ReLU activation function to obtain the fused feature map, and the formula is:
[0036]
[0037] wherein W is a convolution weight, and b is a bias term, and this process can effectively eliminate the feature deviation caused by the data distribution difference of different hospital clients; the feature map fused by each network layer is sent to a pre-determined leader student model L0.
[0038] Preferably, the server sends the fused feature map to the leader student model, and the leader student absorbs and updates the aggregated knowledge into a global model through a self-distillation mechanism, including:
[0039] After the leader student model L0 receives the feature map fused by each network layer sent by the server, the global knowledge contained in the fused feature map is taken as a teacher signal, and a self-distillation task is performed, and the distillation loss adopts a mean square error to calculate a single level loss, and the formula is:
[0040]
[0041] wherein is the feature output of the leader student model corresponding to the layer of the network itself; after self-distillation of all network layers is completed, the loss of each layer is weighted and aggregated to obtain the total self-distillation loss, and the expression is:
[0042]
[0043] wherein J is the total number of network layers participating in distillation; the local training loss of the leader student model is calculated at the same time, and the cross-entropy loss is used based on the local sample set, and the formula is:
[0044]
[0045] The local training loss and the total self-distillation loss are combined, a weight coefficient λ is introduced to balance the contributions of the two, and the final training loss of the leader student model L0 is obtained, and the formula is:
[0046]
[0047] Ensure that the model retains the characteristics of the local data while fully absorbing global knowledge.
[0048] In addition, the present application also provides a federated learning system for non-independent and identically distributed data, which comprises:
[0049] a server, which randomly selects a client as a leader student model in each round of communication, and other clients perform local model training on heterogeneous data sets; and after receiving the top-k features of each client, the server aligns the heterogeneous features of different clients layer by layer based on the network model layer structure, and generates a comprehensive feature map by fusing the aligned features; the fused feature map is sent to the leader student model, and the leader student absorbs and aggregates knowledge through a self-distillation mechanism and is updated to a global model;
[0050] a super network, which is used to extract features from the network models trained by each client, receives the local updated embedding vectors sent by each client, and scores the importance of the features based on a multi-layer perceptron; and according to the importance score, the top-k features of each client are selected, and the filtered high-value features are sent to the server.
[0051] In addition, the present application also provides a machine-readable storage medium, which stores instructions for causing a machine to execute the above-mentioned federated learning data heterogeneity solution based on multi-objective optimization.
[0052] In addition, the application further provides a processor, characterized by being used for running a program, wherein the program is used for executing the federated learning method for non-identically distributed data when being run.
[0053] Through the above technical solution, the application constructs a federated learning method for non-identically distributed data aiming at the core problems of slow model convergence and large performance fluctuation of existing federated learning on non-identically distributed data, so as to systematically alleviate the challenges brought by data heterogeneity (such as different hospital disease distribution differences and different sample annotation standards), evaluate the robustness performance of the model under different heterogeneity degrees, and provide an efficient technical path for federated learning model performance optimization; in the mechanism, the system randomly selects a leader student model from the clients at the start of each round of federated communication, the remaining clients perform regular training based on local data, the super network is then activated and extracts deep features and model parameter update embedding vectors of each client, the Top-k key features are selected after the information value of the features is scored by the embedded MLP module and uploaded to the server, the server aligns the features according to the network level and generates a unified feature map containing global knowledge by using a multi-source fusion strategy (such as attention weighted fusion), and then the unified feature map is distributed to the leader student model to perform self-distillation (the local and global knowledge are coupled by taking the unified feature map as a "teacher signal"), and the dynamic rotation design of the leader student role is introduced (the global knowledge is returned to the normal client in the next round).
[0054] Other features and advantages of the embodiments of the application will be described in detail in the following specific embodiments. BRIEF DESCRIPTION OF DRAWINGS
[0055] The accompanying drawings are included to provide a further understanding of the embodiments of the application, and constitute a part of the specification, and are used together with the following specific embodiments to explain the embodiments of the application, but do not constitute a limitation on the embodiments of the application. In the drawings:
[0056] Figure 1 is a system model schematic diagram of the federated learning method for non-identically distributed data of the application.
[0057] Figure 2 is a working flow schematic diagram of the federated learning method for non-identically distributed data of the application. DETAILED DESCRIPTION
[0058] The specific embodiments of the embodiments of the present application are described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to illustrate and explain the embodiments of the present application, and are not used to limit the embodiments of the present application.
[0059] As shown in Figure 1 The present application proposes a system model diagram of a federated learning method for non-independent and identically distributed data, which mainly includes the following key steps:
[0060] The server randomly selects a client as the leader student model in each round of communication, and other clients perform local model training on heterogeneous data sets;
[0061] The super network extracts features from the network models trained by each client, receives the local update embedding vectors sent by each client, and scores the importance of the features based on a multi-layer perceptron;
[0062] The super network selects the top-k features of each client according to the importance score, and sends the filtered high-value features to the server;
[0063] After the server receives the top-k features of each client, it aligns the heterogeneous features of different clients layer by layer based on the network model hierarchy, and generates a comprehensive feature map by fusing the aligned features;
[0064] The server sends the fused feature map to the leader student model, which absorbs the aggregated knowledge through a self-distillation mechanism and updates to a global model;
[0065] The leader student is converted into a normal client in the next round of communication to participate in local training, and its knowledge as a global model is contributed to the system, effectively alleviating the difference in non-independent and identically distributed data distribution and optimizing the model performance.
[0066] As shown in Figure 2 A federated learning method for non-independent and identically distributed data includes the following steps:
[0067] Step 1: Randomly select a leader student model;
[0068] Step 2: Normal training and client feature collection;
[0069] Step 3: Super network update and feature importance scoring;
[0070] Step 4: Feature alignment and fusion to generate a feature map;
[0071] Step 5: The leader student model performs self-distillation.
[0072] In this implementation method, the method of randomly selecting a leader student model in step 1 includes:
[0073] Step 11: In the process of randomly selecting a leader student model: before the start of each round of federated learning training, a client is determined as the leader student model L0 through random sampling from all local clients participating in data training, which will undertake the core task of receiving global fusion features and performing self-distillation to ensure the randomness and fairness of global knowledge transmission during training.
[0074] The normal training and client feature collection method in step 2 includes:
[0075] Step 21: The test client needs to complete the T-round federated learning training process and synchronously collect the feature parameters generated by the local model during data training; the single-round training loss of each client is calculated, and the loss function is based on the task requirement to use the standard cross-entropy loss, the expression is:
[0076]
[0077] Where B i is the local sample batch size of the client, w i is the local model parameter, f i (x k ; w i ) is the prediction output of the model for the sample x k , and y k is the true label of the sample.
[0078] Step 22: Update the model parameters according to the gradient descent strategy, and the update formula is:
[0079]
[0080] The parameter change is calculated as:
[0081]
[0082] to reflect the parameter adjustment amplitude of the local model during data training.
[0083] Step 24: Upload the features, embedding vectors, and other key parameters output by each client network model to the super network model for subsequent processing.
[0084] The super network update and feature importance scoring method in step 3 includes:
[0085] Step 31: In the super network update and feature importance scoring section: after the super network receives the features and embedding vectors sent by each client, it first calculates the mean of the features of different network layers of the client. For a three-dimensional feature map (channel number C, height H, width W), the mean calculation formula is:
[0086]
[0087] wherein is the feature map of the jth layer network of the ith client, for simplifying the feature dimension and retaining the key information within the layer;
[0088] Step 32: splice the embedding vector v i with the feature mean of the corresponding network layer to obtain the combined features:
[0089]
[0090] to realize the information fusion of the vector features and the layer mean features;
[0091] Step 33: input the spliced combined features into the multi-layer perception (MLP) in the super network to calculate the importance scores of each feature, and the score formula is:
[0092]
[0093] wherein W1 and W2 are the MLP weight matrices, b1 and b2 are the bias terms, and σ is the activation function;
[0094] Step 34: according to the score results, select the top k important features in each client, and upload these high-value features to the server side.
[0095] The feature alignment and fusion method in step 4 to generate the feature map includes:
[0096] Step 41: in the process of generating the feature map by feature alignment and fusion: after the server receives the high-value features sent by the super network, it classifies and aligns them according to the network level to which the features belong, and integrates the features from different clients but belonging to the same network layer into a feature set, expressed as:
[0097]
[0098] wherein N is the number of clients participating in feature uploading, ensuring the dimension consistency of features in the same level;
[0099] Step 42: input the aligned features into the feature fusion module, first perform linear combination on the multi-client features through the 1x1 convolution layer to reduce the feature channel dimension and enhance the cross-client feature correlation, then perform batch normalization (BN) operation and ReLU activation function in turn, to obtain the fused feature map, expressed as:
[0100]
[0101] Wherein, W is the convolution weight, b is the bias term, this process can effectively eliminate the feature deviation caused by the difference of different hospital client data distribution;
[0102] Step 43: the feature map fused by each network layer The leader student model L0 is sent to the predetermined leader student model L0.
[0103] The leader student model in step 5 performs a self-distillation method, which includes:
[0104] Step 51: when the leader student model performs the self-distillation operation: after the leader student model L0 receives the feature map fused by each network layer sent by the server, the global knowledge contained in the fused feature map is used as the "teacher signal" to perform the self-distillation task, and the mean square error is used to calculate the single level loss, and the formula is:
[0105]
[0106] Wherein, is the feature output of the corresponding level of the leader student model itself;
[0107] Step 52: after completing the self-distillation of all network levels, the loss of each level is weighted and aggregated to obtain the total self-distillation loss, and the expression is:
[0108]
[0109] Wherein, J is the total number of network levels participating in the distillation;
[0110] Step 53: calculate the local training loss of the leader student model, and use the cross entropy loss based on the local sample set, and the formula is:
[0111]
[0112] Combine the local training loss with the total self-distillation loss, introduce the weight coefficient λ to balance the contribution of the two, and obtain the final training loss of the leader student model L0, and the formula is:
[0113]
[0114] Ensure that the model retains the characteristics of the local data while fully absorbing the global knowledge.
[0115] To illustrate the technical solutions, take the medical image classification scenario of diagnosing benign and malignant breast cancer pathological sections as an example: the federated learning system contains 5 benign hospital clients (A-E), the local data is not independent and identically distributed (the disease subtypes of each hospital and the sample size are significantly different, for example, A has 1200 sections and benign accounts for 60%, B has 800 sections and malignant accounts for 70%), the total training round is 15 rounds, the initial model is ResNet-18, the super network MLP input dimension is 256, the top-k screening k = 50, and the distillation weight λ = 0.3. In terms of process, the server first distributes the initial model, each client performs 3 rounds of local training with cross-entropy loss (learning rate 0.001), and synchronously extracts the feature maps of each convolutional layer and the 256-dimensional embedding vectors of the fully connected layer; in the first round, randomly select hospital C as the leading student, and after the super network receives the features of each client, first calculate the mean value of the feature maps according to the channel, height and width dimensions (for example, the mean value of the layer 3 features of hospital A is calculated by summing the pixel values of each dimension and then taking the average), then concatenate the mean value and the embedding vector as a combined feature, and after quantifying the score by MLP, select the top-50 high-value features to upload to the server; the server classifies the features according to the network level, unifies the feature dimensions of different clients at the same level through dimension mapping (such as zero padding), and then inputs the fusion module to reduce the dimension through 1x1 convolution, batch normalization and ReLU activation, to generate global feature maps and send them to hospital C; hospital C uses the fusion feature maps as the “teacher signal” to calculate the MSE distillation loss of each level and aggregate them with weighted, and combines the local cross-entropy training loss to obtain the final loss, which is updated as the global model through back propagation.
[0116] After the first round, hospital C becomes a normal client, and a new leading student is randomly selected in each subsequent round, and the process of “super network feature processing-server alignment fusion-leading student distillation update” is repeated until 15 rounds of training are completed. This example accurately selects pathological key features through a super network, eliminates data heterogeneity bias through a server, and integrates global knowledge through dynamic distillation, effectively addressing the problem of non-independent and identically distributed data, and is suitable for federated learning scenarios such as medical care and Internet of Things.
[0117] Preferably, the non-independent and identically distributed data solution method is suitable for medical care, Internet of Things and other scenarios, and has good cross-scene robustness and detection generalization ability.
[0118] Those skilled in the art will appreciate that embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0119] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.
[0120] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.
[0121] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.
[0122] In one typical configuration, the computing device includes one or more processors (CPU's), input / output interfaces, network interfaces, and memory.
[0123] The memory can include non-persistent memory and / or persistent memory, for example, read only memory (ROM) and / or flash memory, for example, in the form of a computer readable storage medium. The memory is an example of computer readable media.
[0124] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.
[0125] It should also be noted that the terms "comprising", "containing", or any other variant thereof are intended to cover non-exclusive inclusions, such that a process, method, article or apparatus that comprises a list of elements does not only include those elements, but can also include other elements not expressly listed or inherent to such process, method, article or apparatus. Without more limitations, the element defined by the phrase "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.
[0126] Those skilled in the art will appreciate that embodiments of the present application can be provided as a method, system or computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) containing computer usable program code.
[0127] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application shall be included in the scope of claims of the present application.
Claims
1. A federated learning method for non-identically distributed data, characterized in that, The federated learning method for non-independent and identically distributed data comprises the following steps: The server randomly selects one client as a leader student model in each round of communication, and other clients perform local model training on heterogeneous data sets; The super network extracts features from the network models trained by each client, receives the local update embedding vectors sent by each client, and scores the importance of the features based on a multi-layer perceptron; The super network selects the top-k features of each client according to the importance score, and sends the filtered high-value features to the server; After receiving the top-k features of each client, the server aligns the heterogeneous features of different clients layer by layer based on the network model hierarchy, and generates a comprehensive feature map by fusing the aligned features; The server sends the fused feature map to the leader student model, which absorbs the aggregated knowledge through a self-distillation mechanism and updates to a global model; The leader student model is converted into a normal client in the next round of communication to participate in local training, and its knowledge as a global model is contributed to the system.
2. The federated learning method for non-identically distributed data according to claim 1, wherein, The server randomly selects one client as a leader student model in each round of communication, which comprises the following steps: During the process of randomly selecting a leader student model, before the start of each round of federated learning training, a client is determined as a leader student model L0 from all local clients participating in data training through random sampling, wherein the model will undertake the core task of receiving global fused features and performing self-distillation to ensure the randomness and fairness of global knowledge transmission during training.
3. The federated learning method for non-identically distributed data according to claim 2, wherein, The other clients perform local model training on heterogeneous data sets, which comprises the following steps: The test client needs to fully participate in the T-round federated learning training process and synchronously collect the feature parameters generated by the local model during data training; the single-round training loss of each client is calculated, and the loss function is based on the task demand and adopts the standard cross-entropy loss, the expression is: where B i is the batch size of the local samples of the client, w i is the local model parameters, f i (x k ; w i ) is the predicted output of the model for the sample x k , y k is the true label of the sample; the model parameters are updated according to the gradient descent strategy, and the update formula is: At the same time, the parameter change is calculated as: to reflect the parameter adjustment amplitude of the local model during data training; the key parameters such as features and embedding vectors output by the network model of each client are uploaded to the super network model for subsequent processing.
4. The federated learning method for non-identically distributed data according to claim 3, characterized in that, The super network extracts features from the network models trained by each client, receives the local update embedding vectors sent by each client, and scores the importance of the features based on a multi-layer perceptron; The super network selects the top-k features of each client according to the importance score, and sends the filtered high-value features to the server, which comprises the following steps: After receiving the features and embedding vectors sent by each client, the super network first calculates the mean value of the features of different network layers, and for a three-dimensional feature map (channel number C, height H, width W), the mean value calculation formula is: wherein is the feature map of the jth layer network of the ith client, to simplify the feature dimension and retain the key information within the layer; the embedding vector v i is the feature mean of the corresponding network layer is spliced to obtain the combined features: to realize the information fusion of vector features and layer mean value features; the combined features after splicing are input into the multi-layer perceptron (MLP) in the super network, and the importance score of each feature is calculated, and the score formula is: where W1 and W2 are the MLP weight matrices, b1 and b2 are the bias terms, and σ is the activation function; According to the score result, the top-k features with high importance in each client are selected and uploaded to the server.
5. The federated learning method for non-identically distributed data according to claim 4, wherein, The server receives the top-k features of each client, aligns the heterogeneous features of different clients layer by layer based on the network model hierarchy, and fuses the aligned features to generate a comprehensive feature map, including: After the server receives the high-value features sent by the super network, the features are classified and aligned according to the network level to which they belong, and the features from different clients but belonging to the same network level are integrated into a feature set, expressed as: Where N is the number of clients participating in feature uploading, ensuring the dimension consistency of features in the same level; the aligned features are input into the feature fusion module, first performing linear combination on the multi-client features through a 1x1 convolution layer to reduce the feature channel dimension and enhance the cross-client feature correlation, then sequentially performing batch normalization (BN) operation and ReLU activation function to obtain the fused feature map, expressed as: Wherein W is the convolution weight, b is the bias term, this process can effectively eliminate the feature deviation caused by the different hospital client data distribution difference; the feature map fused by each network layer Directly send to the predetermined leader student model L0.
6. The federated learning method for non-identically distributed data according to claim 5, characterized in that, The server sends the fused feature map to the leader student model, and the leader student absorbs and aggregates knowledge through the self-distillation mechanism and updates to a global model, including: After the leader student model L0 receives the network level fused feature map sent by the server, it takes the global knowledge contained in the fused feature map as the teacher signal, performs the self-distillation task, and calculates the single level loss using mean square error as the distillation loss, expressed as: wherein is the feature output of the student model itself corresponding to the hierarchy level; after self-distillation of all network levels is completed, the loss of each level is weighted and aggregated to obtain the total self-distillation loss, and the expression is: Where J is the total number of network levels participating in distillation; the local training loss of the leader student model is also calculated, and the cross-entropy loss is used based on the local sample set, expressed as: The local training loss and the total self-distillation loss are combined with a weight coefficient λ to balance their contributions, and the final training loss of the leader student model L0 is obtained, expressed as: Ensure that the model retains the characteristics of local data while fully absorbing global knowledge.
7. A federated learning system for non-identically distributed data, characterized in that, The federated learning system for non-independent and identically distributed data includes: a server that randomly selects a client as a leader student model in each round of communication, and other clients perform local model training on heterogeneous data sets; and after receiving the top-k features of each client, aligns the heterogeneous features of different clients layer by layer based on the network model hierarchy, and fuses the aligned features to generate a comprehensive feature map; sends the fused feature map to the leader student model, and the leader student absorbs and aggregates knowledge through the self-distillation mechanism and updates to a global model; a super network for extracting features from the network models trained by each client, receiving the local updated embedding vectors sent by each client, and scoring the importance of the features based on a multi-layer perceptron; and selecting the top-k features of each client according to the importance score, and sending the filtered high-value features to the server.
8. A machine-readable storage medium, characterized in that, The machine-readable storage medium stores instructions for causing the machine to execute the federated learning data heterogeneity solution based on multi-objective optimization of any one of claims 1-6.
9. A processor, comprising: A program for running, wherein the program is executed to perform the federated learning method for non-independent and identically distributed data of any one of claims 1-6.