A model-heterogeneous federated learning method, system, device, medium, and product

Abstract

The present invention relates to the field of federated learning technology, specifically disclosing a model-heterogeneous federated learning method, system, device, medium, and product. The method comprises the following steps: S1: a server synchronously initializes a global classifier to all clients; S2: each client updates a local model based on its local private dataset; the local model includes a representation extractor and a classifier; S3: each client generates a representation using its representation extractor, then calculates an entangled representation and uploads it to the server; S4: the server updates the classifier parameters based on the entangled representations of each client and sends them to the client; S5: steps S2-S4 are repeated until the iteration round is reached. The present invention can effectively alleviate the problem of model performance degradation caused by inconsistent client model architectures and inconsistent data distribution, thereby improving the learning ability of the federated learning system in a heterogeneous environment.

Description

Technical Field

[0001] The present invention relates to the field of federated learning technology, and in particular to a federated learning method, system, device, medium and product for heterogeneous models. Background Art

[0002] Currently, data security and privacy protection are receiving increasing attention. Traditional machine learning paradigms typically rely on centralizing distributed data to a centralized server for unified modeling and analysis. This process carries a high risk of privacy leakage during data transmission and centralized storage. To address this challenge, federated learning technology has emerged. Its fundamental concept is to keep the data static while moving the model. Specifically, different clients train models locally using private data, exchanging only necessary model parameters or gradient information without transmitting private data. By collaboratively updating model parameters across multiple clients, federated learning effectively utilizes distributed data resources while significantly reducing the risk of private data leakage. It has become a key research direction in privacy-preserving machine learning.

[0003] In federated learning systems, different clients often have significant differences in computing power, hardware environment, and data distribution, making it difficult for them to adopt a unified model architecture. Specifically, the representation extractors in the model architecture are heterogeneous, while the classifier structure remains consistent. In this scenario, the traditional Federated Averaging (FedAVG) algorithm has significant limitations because it cannot directly and effectively aggregate model parameters with inconsistent structures.

[0004] FedGH (Yi L, Wang G, Liu X, et al. Fedgh: Heterogeneous federated learning with generalized global header[C] / / ACM MM. 2023: 8686-8696) employs the following core concept: Each client computes the class center (i.e., class mean) of its local data representations for a specific class and uploads this representation to the server. The server fine-tunes its classifier based on the class means uploaded by all clients and then synchronizes the updated classifier to each client, replacing its local classifier. However, this approach has design limitations. Because the class mean only models representations of the same class and ignores representations of different classes, the trained classifier is limited in its generalization ability, thus affecting overall model performance.

[0005] A Chinese invention, application publication number CN119670916A, discloses a federated learning method and apparatus based on feature comparison optimization and dynamic integration of classifiers. The client uses a trained local feature extractor to extract features from its local private dataset to calculate local feature prototypes corresponding to various feature vectors. These local feature prototypes are then combined into a local feature prototype set, which is then uploaded to a server. This invention cannot handle heterogeneous client models and requires uploading a local model and local feature prototype set, resulting in high communication overhead and potential privacy concerns.

[0006] Therefore, there is an urgent need to design a federated learning method that takes into account privacy protection, model performance, and communication overhead at the same time, so as to achieve higher-quality cross-client collaborative modeling. Summary of the Invention

[0007] The present invention aims to address the aforementioned issues. To this end, it provides a model-heterogeneous federated learning method, system, device, medium, and product. This method fuses all sample representations from each client into a unified, entangled representation, which is uploaded to a server for classifier training. This effectively mitigates the degradation of model performance caused by inconsistent client model architectures (model heterogeneity) and inconsistent data distribution (data heterogeneity), thereby improving the learning capabilities of federated learning systems in heterogeneous environments.

[0008] The present invention provides a federated learning method for heterogeneous models, which adopts the following technical solution:

[0009] S1: The server initializes the global classifier synchronously to all clients;

[0010] S2: Each client updates a local model based on its local private dataset; the local model includes a representation extractor and a classifier;

[0011] S3: Each client generates a representation using its representation extractor, then calculates an entangled representation and uploads it to the server;

[0012] The computation process of the entangled representation is as follows: all samples in the local private dataset are generated into corresponding representations through the representation extractor, and the average pooling operation is applied to the representations, and the average pooled representations are fused into an entangled representation;

[0013] S4: The server updates the classifier parameters based on the entanglement representation of each client and sends it to the client;

[0014] S5: Repeat steps S2-S4 until the iteration round is reached.

[0015] Furthermore, the calculation formula for entanglement representation is as follows:

[0016]

[0017] in, Indicates the The entangled representation of each client, Indicates the clients in the category The sample set, Indicates the The set of categories owned by a client, Indicates the Client categories The normalized weight of represents the average pooling operation, Indicates the A sample of clients The corresponding representation, express The corresponding one-hot encoded labels.

[0018] Furthermore, the normalized weight vector Each element is initially uniformly distributed randomly sampled and then normalized by dividing by the sum of all its elements so that they sum to 1, , Indicates the The normalized weight of client class 1, Indicates the Client categories The normalized weights.

[0019] Furthermore, the clients have heterogeneous representation extractors, while the classifier adopts a unified architecture across all clients.

[0020] Furthermore, in S2, The optimization goal of each client is:

[0021]

[0022] in, Represents the first cross entropy loss function, which is used to measure the sample The model prediction results and their corresponding labels The difference between Indicates that by optimizing Model parameters of the client's local model , to minimize its average classification loss on the local private dataset, Indicates the The total number of samples from clients.

[0023] Furthermore, in S4, the optimization goal of the server is:

[0024]

[0025] in, represents the second cross entropy loss function, Represents the parameters of the classifier Optimize to minimize the corresponding loss function value, Indicates that the parameter is The classifier, Representation category The corresponding one-hot encoded labels, Indicates the Client categories The normalized weight of Indicates the The entangled representation of each client, Indicates the The set of categories owned by a client, Indicates the total number of clients.

[0026] The present invention also provides a model heterogeneous federated learning system, which adopts the following technical solution: including: 1 server and multiple clients,

[0027] The server is used to synchronize and initialize the global classifier to all clients; update the parameters of the classifier according to the entanglement representation of each client, and send them to the client;

[0028] The client is used to independently update a local model based on its local private data set, where the local model includes a representation extractor and a classifier. The client generates a representation using its representation extractor, then calculates an entangled representation and uploads it to the server.

[0029] The present invention also provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned model heterogeneous federated learning method when executing the computer program.

[0030] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which implements the above-mentioned model heterogeneous federated learning method when executed by a processor.

[0031] The present invention also provides a computer program product, including a computer program, which implements the above-mentioned model heterogeneous federated learning method when executed by a processor.

[0032] The above one or more technical solutions in the embodiments of the present invention have at least one of the following technical effects:

[0033] 1. The present invention fuses all sample representations of each client into a unified entangled representation and uploads it to the server to train the classifier. Since the entangled representation integrates information from all categories, using these entangled representations to train the classifier can improve the generalization ability of the classifier to a certain extent. At the same time, since each client only uploads one entangled representation, it is difficult for an attacker to reversely infer the original sample through a representation inversion attack. Therefore, it has strong privacy protection capabilities and significantly reduces communication overhead. Therefore, while ensuring model performance, the present invention takes into account both privacy protection and communication overhead, and has good application prospects and promotion value.

[0034] 2. This invention applies average pooling to the representations, mapping them uniformly to a common representation space for processing. This effectively compresses the representation dimensions while preserving their core semantic information and eliminates representation differences caused by different model architectures. The average pooled representations are uniformly mapped to a common low-dimensional representation space, achieving consistency in representation dimensions across all clients.

[0035] 3. Each client of the present invention only uploads one entanglement representation. Accordingly, the present invention designs the client's optimization target as the model parameters of the local model, and designs the server's optimization target as the parameters of the global classifier. The parameters of the global classifier are used to update the parameters of the local classifier of each client. Under the premise of ensuring the privacy of client data, a group of local models can be collaboratively learned to achieve the optimal average classification accuracy on each client.

[0036] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0038] Figure 1 It is a flow chart of the method provided by the present invention.

[0039] Figure 2 This is a comparison chart of the privacy protection experiment results provided by the present invention.

[0040] Figure 3 It is a structural block diagram of the system provided by the present invention. DETAILED DESCRIPTION

[0041] To make the purpose, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the present invention. Obviously, the embodiments described are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. The following embodiments are used to illustrate the present invention, but are not used to limit the scope of the present invention.

[0042] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiment of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0043] The following combination Figures 1 to 3 The present invention is further described in detail, and a method, system, device, medium, and product for federated learning with heterogeneous models of the present invention are described:

[0044] First, the federated learning architecture used in this embodiment is defined.

[0045] Definition clients and 1 server, total number of categories is . No. The private dataset of a client is represented as , ,in, Indicates the Client's samples, express The corresponding one-hot encoded labels, Indicates the The total number of samples of clients, .

[0046] The local model consists of two parts: the representation extractor and the classifier. Let the local model be expressed as , ,in, Representation Extractor, represents a classifier, Represents the composite operation of functions, represents the model parameters of the local model, , represents the parameters that characterize the extractor, Represents the parameters of the classifier.

[0047] It should be noted that there are some clients, such as Client and clients, with heterogeneous representation extractors but the same classifier. Client-side representation extractor Hedi Client-side representation extractor The network structure is different. Client-side classifier , No. Client-side classifier All classifiers use a unified architecture across all clients.

[0048] The learning goal of this embodiment is to collaboratively learn a set of local models while ensuring the privacy of client data, so that they can achieve the optimal average classification accuracy on their respective clients.

[0049] In this embodiment, Figure 1 As shown, a federated learning method for heterogeneous models is provided, including the following steps:

[0050] S1: The server initializes the global classifier synchronously to all clients.

[0051] The client receives the global classifier sent by the server in the form of broadcast, completes local initialization, and realizes local deployment.

[0052] S2: Each client independently updates the local model based on its local private dataset; the local model includes a representation extractor and a classifier.

[0053] Specifically, The optimization goal of a client can be expressed as:

[0054]

[0055] in, Represents the first cross entropy loss function, which is used to measure the sample The model prediction results and their corresponding labels The difference between Indicates that by optimizing Model parameters of the client's local model , to minimize its average classification loss on the local private dataset.

[0056] S3: Each client generates a representation using its representation extractor, then calculates an entangled representation and uploads it to the server.

[0057] To address the problem of representation heterogeneity caused by differences in client model architectures, this embodiment first applies an average pooling operation to the representations generated by the clients, mapping them uniformly to a common representation space for processing. Specifically, by applying average pooling to the representations of each client, it is possible to effectively compress the representation dimensions while retaining their core semantic information and eliminate representation differences caused by different model architectures. After average pooling, the representations are uniformly mapped to a common low-dimensional representation space, thereby achieving consistency in the representation dimensions of each client.

[0058] The entangled representation is calculated as follows: all samples in the local private dataset are generated corresponding representations by the representation extractor, average pooling is applied to the representations, and the average pooled representations are fused into a entangled representation. The entangled representation is used to train the classifier in step S4.

[0059] The calculation formula for entanglement representation is as follows:

[0060]

[0061] in, Indicates the The entangled representation of each client, Indicates the clients in the category The sample set, Indicates the The set of categories owned by a client, Indicates the Client categories The normalized weight of represents the average pooling operation, Indicates the A sample of clients Corresponding representation.

[0062] Normalized weight vector Each element is initially uniformly distributed randomly sampled and then normalized by dividing by the sum of all its elements so that they sum to 1, , Indicates the The normalized weight of client class 1, Indicates the Client categories The normalized weights. is the probability distribution of each entangled representation belonging to different categories, which can be regarded as its corresponding label encoding.

[0063] The client will entangle the representation and its corresponding weight vector Upload to the server.

[0064] S4: The server updates the classifier parameters based on the entanglement representation of each client and sends it to the client.

[0065] After receiving the entanglement representations uploaded by each client and their corresponding weight vector sets back, ,The server uses these entangled representations and their corresponding weight vectors to update the classifier.

[0066] To this end, the server's optimization goals are defined as:

[0067]

[0068] in, Represents the parameters of the classifier Optimize to minimize the corresponding loss function value, represents the second cross entropy loss function, Indicates that the parameter is The classifier, Representation category The corresponding one-hot encoded labels, represents the total number of clients. By minimizing the aforementioned loss function, the server can jointly train a classifier using the entangled representations uploaded by all clients. Because each entangled representation effectively integrates representation information from multiple categories, the classifier trained based on this representation has stronger generalization capabilities, thereby accurately classifying training samples from each client.

[0069] Subsequently, the server broadcasts the updated global classifier parameters to all clients to replace the client's local classifier parameters, and then proceeds to the next round of federated learning iteration.

[0070] S5: Repeat steps S2-S4 until the iteration round is reached.

[0071] This example verifies the effectiveness of this method through the following experiments.

[0072] This example compares the classification accuracy of this method with the existing method FedGH on the CIFAR-10, CIFAR-100, and TinyImageNet datasets, using the same heterogeneous data setup. Table 1 shows the experimental results, which demonstrate that this method outperforms FedGH on all datasets, validating its effectiveness.

[0073] Table 1 Classification accuracy comparison table

[0074]

[0075] Figure 2 The privacy protection effect of this method and FedGH on some images sampled from the TinyImageNet dataset is compared. Since FedGH trains the global classifier based on the class mean of each category, it may leak category-related information. For example, Figure 2 (a) is the original image. Figure 2 In (b), we can observe features related to the “fish” category, and thus infer the existence of this category. However, this method fuses the representations of all categories in each client into a unified entangled representation, making it difficult to infer any specific original sample information from this representation, including its category, such as Figure 2 The above results show that this method has significant advantages in protecting client privacy.

[0076] Table 2 compares the communication overhead of our method and FedGH on various datasets. It can be observed that our method significantly reduces the communication overhead during the model upload phase. This is primarily due to the fact that in our method, each client only needs to upload a single entangled representation that incorporates information from all classes, while FedGH requires each client to upload its local class mean for all classes, resulting in higher communication overhead. These results demonstrate that our method significantly reduces communication overhead.

[0077] Table 2 Communication overhead comparison table (unit: 10 3 Number of parameters)

[0078]

[0079] This embodiment also provides a federated learning system with heterogeneous models, such as Figure 3 As shown, the technical solution adopted is as follows: including: 1 server and Clients,

[0080] The server is used to synchronize and initialize the global classifier to all clients; update the parameters of the classifier according to the entanglement representation of each client, and send them to the client;

[0081] The client is configured to independently update a local model based on its local private dataset. The local model includes a representation extractor and a classifier. The client generates a representation using its representation extractor, then calculates an entangled representation and uploads it to the server. After receiving the global classifier parameters from the server, the client uses these parameters to update its local classifier.

[0082] The working process of this system is:

[0083] S1: The server initializes the global classifier synchronously to all clients.

[0084] S2: Each client updates its local model based on its local private dataset; the local model includes a representation extractor and a classifier. The client optimizes the model parameters of the local model to minimize its average classification loss on the local private dataset.

[0085] S3: Each client generates a representation using its representation extractor, then calculates an entangled representation and uploads it to the server. The entangled representation is calculated as follows: All samples in the local private dataset are generated corresponding representations by the representation extractor, and average pooling is applied to the representations, fusing the average-pooled representations into a single entangled representation.

[0086] S4: The server updates the classifier parameters based on the entanglement representation of each client and sends them to the client. The server optimizes the classifier parameters to minimize the corresponding loss function value.

[0087] S5: Repeat steps S2-S4 until the iteration round is reached.

[0088] The clients have heterogeneous representation extractors, and the classifier uses a unified architecture across all clients. This system can collaboratively learn a set of local models to achieve the best average classification accuracy on each client while ensuring client data privacy.

[0089] This embodiment further provides a computer device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-mentioned federated learning method for heterogeneous models, including the following steps:

[0090] S1: The server initializes the global classifier synchronously to all clients;

[0091] S2: Each client updates a local model based on its local private dataset; the local model includes a representation extractor and a classifier;

[0092] S3: Each client generates a representation using its representation extractor, then calculates an entangled representation and uploads it to the server;

[0093] The computation process of the entangled representation is as follows: all samples in the local private dataset are generated into corresponding representations through the representation extractor, and the average pooling operation is applied to the representations, and the average pooled representations are fused into an entangled representation;

[0094] S4: The server updates the classifier parameters based on the entanglement representation of each client and sends it to the client;

[0095] S5: Repeat steps S2-S4 until the iteration round is reached.

[0096] This embodiment further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for implementing the above-mentioned federated learning method for heterogeneous models includes the following steps:

[0097] S1: The server initializes the global classifier synchronously to all clients;

[0098] S2: Each client updates a local model based on its local private dataset; the local model includes a representation extractor and a classifier;

[0099] S3: Each client generates a representation using its representation extractor, then calculates an entangled representation and uploads it to the server;

[0100] The computation process of the entangled representation is as follows: all samples in the local private dataset are generated into corresponding representations through the representation extractor, and the average pooling operation is applied to the representations, and the average pooled representations are fused into an entangled representation;

[0101] S4: The server updates the classifier parameters based on the entanglement representation of each client and sends it to the client;

[0102] S5: Repeat steps S2-S4 until the iteration round is reached.

[0103] This embodiment further provides a computer program product, including a computer program. When the computer program is executed by a processor, the computer program implements the above-mentioned federated learning method for heterogeneous models, including the following steps:

[0104] S1: The server initializes the global classifier synchronously to all clients;

[0105] S2: Each client updates a local model based on its local private dataset; the local model includes a representation extractor and a classifier;

[0106] S3: Each client generates a representation using its representation extractor, then calculates an entangled representation and uploads it to the server;

[0107] The computation process of the entangled representation is as follows: all samples in the local private dataset are generated into corresponding representations through the representation extractor, and the average pooling operation is applied to the representations, and the average pooled representations are fused into an entangled representation;

[0108] S4: The server updates the classifier parameters based on the entanglement representation of each client and sends it to the client;

[0109] S5: Repeat steps S2-S4 until the iteration round is reached.

[0110] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A federated learning method for heterogeneous models, characterized by: The following steps are involved: S1: The server initializes the global classifier synchronously to all clients; S2: Each client updates a local model based on its local private image dataset; the local model includes a representation extractor and a classifier; S3: Each client generates a representation using its representation extractor, then calculates an entangled representation and uploads it to the server; The computation process of the entangled representation is as follows: all samples in the local private image dataset generate corresponding representations through the representation extractor, and the average pooling operation is applied to the representations, and the average pooled representations are fused into an entangled representation; The calculation formula for entanglement representation is as follows: in, Indicates the The entangled representation of each client, Indicates the clients in the category The sample set, Indicates the The set of categories owned by a client, Indicates the Client categories The normalized weight of represents the average pooling operation, Indicates the A sample of clients The corresponding representation, express The corresponding one-hot encoded labels; S4: The server updates the classifier parameters based on the entanglement representation of each client and sends it to the client to replace the client's classifier parameters, thus proceeding to the next round of federated learning iteration; In S4, the optimization goals of the server are: in, represents the second cross entropy loss function, Represents the parameters of the classifier Optimize to minimize the corresponding loss function value, Indicates that the parameter is The classifier, Representation category The corresponding one-hot encoded labels, Indicates the Client categories The normalized weight of Indicates the The entangled representation of each client, Indicates the The set of categories owned by a client, Indicates the total number of clients; S5: Repeat steps S2-S4 until the iteration round is reached.

2. A federated learning method for heterogeneous models according to claim 1, characterized in that: Normalized weight vector Each element of is initially randomly sampled from a uniform distribution and then normalized by dividing by the sum of all its elements so that the sum is 1. , Indicates the The normalized weight of client class 1, Indicates the Client categories The normalized weights.

3. The federated learning method for heterogeneous models according to claim 1, wherein: The clients have heterogeneous representation extractors, while the classifier adopts a unified architecture across all clients.

4. The federated learning method for heterogeneous models according to claim 1, wherein: In S2, The optimization goal of each client is: in, Represents the first cross entropy loss function, which is used to measure the sample The model prediction results and their corresponding labels The difference between Indicates that by optimizing Model parameters of the client's local model , to minimize its average classification loss on a local private image dataset, Indicates the The total number of samples of clients, For the A client's private dataset.

5. A federated learning system with heterogeneous models, characterized by: A method for performing a model heterogeneous federated learning method according to any one of claims 1 to 4, comprising: a server and a plurality of clients, The server is used to synchronize and initialize the global classifier to all clients; update the parameters of the classifier according to the entanglement representation of each client, and send them to the client; The client is used to independently update a local model based on its local private image dataset, where the local model includes a representation extractor and a classifier. The client generates a representation using its representation extractor, then calculates an entangled representation and uploads it to the server.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the method for federated learning of heterogeneous models according to any one of claims 1 to 4 is implemented.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for federated learning of heterogeneous models according to any one of claims 1 to 4 is implemented.

8. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, it implements a model heterogeneous federated learning method according to any one of claims 1 to 4.

Images