Federal learning method, system and device for model isomerism, medium and product

Abstract

The invention relates to the technical field of federated learning, and particularly discloses a model heterogeneous federated learning method, system and device, a medium and a product, and the method comprises the steps: S1, enabling a server to synchronously initialize a global classifier to all clients; s2, updating a local model by each client based on a local private data set of the client; the local model comprises a representation extractor and a classifier; s3, each client generates a representation by using a representation extractor thereof, then calculates to obtain an entanglement representation, and uploads the entanglement representation to the server; s4, the server updates the parameters of the classifier according to the entanglement representation of each client, and issues the parameters to the client; and S5, repeating the steps S2-S4 until an iteration round is reached. According to the method, the problem of model performance reduction caused by inconsistent client model architecture and inconsistent data distribution can be effectively relieved, so that the learning ability of a federal learning system in a heterogeneous environment is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of federated learning, and particularly to a federated learning method, system, device, medium and product with heterogeneous models. Background Art

[0002] Currently, data security and privacy protection issues have received increasing widespread attention. Traditional machine learning paradigms usually rely on collecting distributed data to a central server for unified modeling and analysis, which has a high risk of privacy leakage in the data transmission and centralized storage processes. To address this challenge, federated learning technology has emerged. Its basic concept is that data does not move while the model moves, that is, different clients use private data for model training locally, and only exchange necessary model parameters or gradient information without transmitting private data. By collaboratively updating model parameters among multiple clients, federated learning significantly reduces the risk of private data leakage while effectively utilizing distributed data resources, and has become an important direction in current privacy-preserving machine learning research.

[0003] In a federated learning system, different clients usually have significant differences in computing power levels, hardware environments, and data distributions, making it difficult for different clients to adopt a unified model architecture. Specifically, the feature extractors in the model architecture are heterogeneous, while the classifier structures are consistent. In this scenario, the traditional Federated Averaging (FedAVG) algorithm has significant limitations because it cannot effectively aggregate model parameters with inconsistent structures.

[0004] FedGH (Yi L, Wang G, Liu X, et al. Fedgh: Heterogeneous federated learning with generalized global header[C] / / ACM MM. 2023: 8686-8696), the core idea is that each client calculates the class center (i.e., the class mean) of the sample features of local data under specific categories and uploads it to the server side. Based on the class mean information uploaded by all clients, the server side fine-tunes the classifier and then synchronizes the updated classifier to each client to replace its local classifier. However, this method has certain limitations in design. Since the class mean only models the feature information of the same category and ignores the feature information of different categories, the trained classifier has limited generalization ability, thus affecting the overall model performance.

[0005] The Chinese invention with the application publication number CN119670916A discloses a federated learning method and device based on feature comparison optimization and classifier dynamic integration. The client uses the trained local feature extractor to extract features from its local private dataset to calculate the local feature prototypes corresponding to various feature vectors, and forms the local feature prototype set with the local feature prototypes, and then uploads the local model and the local feature prototype set to the server. This invention cannot handle the problem of heterogeneous client models, and it is necessary to upload the local model and the local feature prototype set, resulting in a large communication overhead and certain private data leakage problems.

[0006] Therefore, there is an urgent need to design a federated learning method that takes into account privacy protection, model performance, and communication overhead at the same time to achieve higher-quality cross-client collaborative modeling. Summary of the Invention

[0007] The present invention aims to solve the above problems. To this end, the present invention provides a federated learning method, system, device, medium, and product for heterogeneous models. The present invention fuses all sample representations of each client into a unified entangled representation and uploads it to the server to train a classifier, which can effectively alleviate the problem of model performance degradation caused by inconsistent client model architectures (model heterogeneity) and inconsistent data distributions (data heterogeneity), thereby improving the learning ability of the federated learning system in a heterogeneous environment.

[0008] The present invention provides a federated learning method for heterogeneous models, and the technical solution adopted is as follows: including the following steps: S1: The server synchronously initializes the global classifier to all clients; S2: Each client updates its local model based on its local private dataset; the local model includes a representation extractor and a classifier; S3: Each client uses its representation extractor to generate a representation, and then calculates an entangled representation and uploads it to the server; The calculation process of the entangled representation is as follows: all samples of the local private dataset generate corresponding representations through the representation extractor, apply an average pooling operation to the representations, and fuse the representations after average pooling into an entangled representation; S4: The server updates the parameters of the classifier according to the entangled representations of each client and distributes them to the clients; S5: Repeat steps S2 - S4 until the iteration round is reached.

[0009] Furthermore, the calculation formula of the entangled representation is as follows: Where represents the entangled representation of the th client, represents the The sample set belonging to the category in the th client, denotes the set of categories owned by the th client, and denotes the normalized weight of the th client category represents the average pooling operation, denotes the sample of the th client and its corresponding representation, denotes the corresponding one-hot encoded label. [[ID=(27)]]

[0010] Furthermore, each element of the normalized weight vector is initially randomly sampled from a uniform distribution and then normalized by dividing by the sum of all its elements so that the sum is 1. , denotes the normalized weight of the th client category 1, denotes the normalized weight of the th client category .

[0011] Furthermore, the clients have heterogeneous representation extractors, while the classifier adopts a unified architecture across all clients.

[0012] Furthermore, in S2, the optimization objective of the th client is: where denotes the first cross-entropy loss function, which is used to measure the difference between the model prediction result of the sample and its corresponding label , denotes minimizing the average classification loss of the local model of the th client on its local private dataset by optimizing the model parameters of the local model, denotes the total number of samples of the th client.

[0013] Furthermore, in S4, the optimization objective of the server is: where denotes the second cross-entropy loss function, denotes the parameters of the classifier Optimize it to minimize the value of its corresponding loss function. denote a classifier with parameters ; denote the one-hot encoded label corresponding to the class ; denote the -th client class of the normalized weight; denote the entanglement representation of the -th client; denote the -th client's set of classes owned; denote the total number of clients.

[0014] The present invention also provides a federated learning system with heterogeneous models, and the technical solution adopted is as follows: including: 1 server and multiple clients, The server is used to synchronously initialize the global classifier to all clients; update the parameters of the classifier according to the entanglement representations of each client, and send them to the clients; The client is used to independently update the local model based on its local private dataset. The local model includes a feature extractor and a classifier; the client uses its feature extractor to generate a representation, then calculates an entanglement representation, and uploads it to the server.

[0015] The present invention also provides a computer device, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the above-mentioned federated learning method with heterogeneous models.

[0016] The present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the above-mentioned federated learning method with heterogeneous models.

[0017] The present invention also provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the above-mentioned federated learning method with heterogeneous models.

[0018] One or more of the above technical solutions in the embodiments of the present invention have at least one of the following technical effects: 1. The present invention fuses all sample representations of each client into a unified entangled representation and uploads it to the server to train a classifier. Since the entangled representation incorporates information from all categories, training the classifier using these entangled representations can, to a certain extent, enhance the generalization ability of the classifier. Meanwhile, since each client only uploads one entangled representation, it is difficult for an attacker to reverse-infer the original samples through the representation by reverse attack. Therefore, it has strong privacy protection capabilities and significantly reduces the communication overhead. Thus, while ensuring the model performance, the present invention takes into account privacy protection and communication overhead, and has good application prospects and promotion value.

[0019] 2. The present invention applies an average pooling operation to the representation, maps it uniformly to a common representation space for processing, can effectively compress the representation dimension while retaining its core semantic information, and eliminates the representation differences caused by different model architectures. The representation after average pooling is uniformly mapped to a common low-dimensional representation space, thereby achieving the consistency of the representation dimensions of each client.

[0020] 3. Each client of the present invention only uploads one entangled representation. Correspondingly, the present invention designs the optimization objective of the client as the model parameters of the local model, designs the optimization objective of the server as the parameters of the global classifier, and uses the parameters of the global classifier to update the parameters of the local classifier of each client, which can, on the premise of ensuring the client data privacy, collaboratively learn a set of local models to achieve the optimal average classification accuracy on their respective clients.

[0021] Additional aspects and advantages of the present invention will be given in part in the following description, become apparent in part from the following description, or be learned through the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] To more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0023] Figure 1 is a flowchart of the method provided by the present invention.

[0024] Figure 2 is a comparison diagram of the privacy protection experimental results provided by the present invention.

[0025] Figure 3 is a structural block diagram of the system provided by the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0026] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below with reference to the accompanying drawings in the present invention. Apparently, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present invention without creative efforts shall fall within the protection scope of the present invention. The following embodiments are used to illustrate the present invention, but cannot be used to limit the scope of the present invention.

[0027] In the description of this specification, the descriptions referring to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. mean that the specific features, structures, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the embodiments of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, without conflict, those skilled in the art can combine the different embodiments or examples described in this specification and the features of the different embodiments or examples.

[0028] The following Figures 1 to 3 will further elaborate on the present invention, and describe a model heterogeneous federated learning method, system, device, medium, and product of the present invention: First, define the federated learning architecture adopted in this embodiment.

[0029] Define clients and 1 server, and the total number of categories is . The th client's private dataset is denoted as , , where represents the th sample of the th client, represents the corresponding one-hot encoded label, represents the total number of samples of the th client, .

[0030] The local model includes two parts, namely a feature extractor and a classifier. Let the local model be denoted as , , where represents the feature extractor, represents the classifier, represents the composite operation of functions, represent the model parameters of the local model, , represent the parameters of the feature extractor, represent the parameters of the classifier.

[0031] It should be noted that there are some clients, such as the th client and the th client, which have heterogeneous feature extractors, but their classifiers are the same. That is, the feature extractor of the th client and the feature extractor of the th client have different network structures, and the classifiers of the th client , the th client's classifier and so on, all classifiers adopt a unified architecture among all clients.

[0032] The learning objective of this embodiment is to collaboratively learn a set of local models on the premise of ensuring client data privacy, so that the optimal average classification accuracy is achieved on each client.

[0033] In this embodiment, as Figure 1 shown, a federated learning method with model heterogeneity is provided, including the following steps: S1: The server synchronously initializes the global classifier to all clients.

[0034] The client receives the global classifier sent by the server in broadcast form and completes local initialization to achieve local deployment.

[0035] S2: Each client independently updates its local model based on its local private dataset; the local model includes a feature extractor and a classifier.

[0036] Specifically, the optimization objective of the th client can be expressed as: where represents the first cross-entropy loss function, which is used to measure the difference between the model prediction result of the sample and its corresponding label , represents minimizing the average classification loss of the th client's local model by optimizing the model parameters of the

[0037] S3: Each client uses its feature extractor to generate features, then calculates an entangled feature, and uploads it to the server.

[0038] To solve the problem of heterogeneous features caused by differences in the model architectures of each client, in this embodiment, an average pooling operation is first applied to the features generated by the clients to map them uniformly to a common feature space for processing. Specifically, by applying average pooling to the features of each client, the core semantic information can be retained while effectively compressing the feature dimension and eliminating the feature differences caused by different model architectures. The features after average pooling are uniformly mapped to a common low-dimensional feature space, thereby achieving the consistency of the feature dimensions of each client.

[0039] The calculation process of the entangled feature is as follows: All samples of the local private dataset generate corresponding features through the feature extractor, an average pooling operation is applied to the features, and the features after average pooling are fused into an entangled feature. The entangled feature is used to train the classifier in step S4.

[0040] The calculation formula of the entangled feature is as follows: Where represents the entangled feature of the -th client, represents the set of samples belonging to class in the -th client, represents the set of classes owned by the -th client, represents the normalized weight of class in the -th client, represents the average pooling operation, represents the -th sample of the -th client,

[0041] The normalized weight vector The elements of each item are initially randomly sampled from a uniform distribution and then normalized by dividing by the sum of all its elements so that the sum is 1, , represents the normalized weight of class 1 of the -th client, represents the normalized weight of class [[ID=5??]] in the -th client. is the probability distribution of each entangled feature belonging to different classes and can be regarded as the corresponding label encoding.

[0042] The client uploads the entanglement representation and its corresponding weight vector to the server.

[0043] S4: The server updates the parameters of the classifier according to the entanglement representations of each client and distributes them to the clients.

[0044] After receiving the entanglement representations uploaded by each client and their corresponding weight vector sets , , the server uses these entanglement representations and their corresponding weight vectors to update the classifier.

[0045] For this purpose, the optimization objective of the server is defined as: where represents optimizing the parameters of the classifier to minimize its corresponding loss function value, represents the second cross-entropy loss function, represents the classifier with parameters , represents the one-hot encoded label corresponding to the category , represents the total number of clients. By minimizing the above loss function, the server can jointly train a classifier using the entanglement representations uploaded by all clients. Since each entanglement representation effectively integrates the representation information of multiple categories, the classifier trained based on this representation can have stronger generalization ability, thereby achieving accurate classification of the training samples of each client.

[0046] Subsequently, the server broadcasts the updated parameters of the global classifier to all clients to replace the parameters of the local classifier of the clients, and then conducts the next round of federated learning iteration.

[0047] S5: Repeat steps S2 - S4 until the number of iteration rounds is reached.

[0048] This embodiment verifies the effectiveness of this method through the following experiments.

[0049] In this embodiment, under the same data heterogeneity setting, this method and the existing method FedGH are respectively compared for the classification accuracy on the datasets CIFAR - 10, CIFAR - 100, and TinyImageNet. The experimental results are shown in Table 1. The experimental results in Table 1 show that the classification accuracy of this method on all datasets is better than that of FedGH, verifying the effectiveness of this method.

[0050] Table 1 Comparison table of classification accuracy

[0051] Figure 2 Shows the comparison of the privacy protection effects of the proposed method and FedGH on the sampled partial images in the TinyImageNet dataset. Since FedGH trains the global classifier based on the class means of each category, it may leak category-related information. For example, Figure 2 In (a), it is the original image. In Figure 2 In (b), features related to the "fish" category can be observed, thus inferring the existence of this category. While the proposed method fuses the representations of all categories in each client into a unified entangled representation, making it difficult to reverse any specific original sample information from this representation, including its category, as shown in Figure 2 In (c). The above results show that the proposed method has significant advantages in protecting client privacy.

[0052] Table 2 shows the comparison of the communication overhead effects of the proposed method and FedGH on different datasets. It can be observed that the communication overhead of the proposed method in the model uploading stage is significantly lower than that of FedGH. This is mainly due to the fact that in the proposed method, each client only needs to upload an entangled representation that fuses all category information, while FedGH requires each client to upload the class means of all its local categories, resulting in a higher communication overhead. The above results show that the proposed method significantly reduces the communication overhead.

[0053] Table 2 Comparison of communication overhead (unit: 10 3 Number of parameters)

[0054] This embodiment also provides a federated learning system with heterogeneous models. As shown in Figure 3 The technical solution adopted is as follows: It includes: 1 server and clients. The server is used to synchronously initialize the global classifier to all clients; update the parameters of the classifier according to the entangled representations of each client, and send them to the clients. The client is used to independently update the local model based on its local private dataset. The local model includes a feature extractor and a classifier; the client uses its feature extractor to generate a representation, and then calculates an entangled representation and uploads it to the server. After receiving the parameters of the global classifier sent by the server, the client uses these parameters to update its local local classifier.

[0055] The working process of this system is as follows: S1: The server synchronously initializes the global classifier to all clients.

[0056] S2: Each client updates its local model based on its local private dataset; the local model includes a feature extractor and a classifier. The client optimizes the model parameters of the local model to minimize its average classification loss on the local private dataset.

[0057] S3: Each client uses its feature extractor to generate features, then calculates an entangled feature, and uploads it to the server. The calculation process of the entangled feature is as follows: all samples of the local private dataset generate corresponding features through the feature extractor, apply average pooling operation to the features, and fuse the features after average pooling into an entangled feature.

[0058] S4: The server updates the parameters of the classifier according to the entangled features of each client and distributes them to the clients. The server optimizes the parameters of the classifier to minimize the corresponding loss function value.

[0059] S5: Repeat steps S2 - S4 until the number of iteration rounds is reached.

[0060] The clients have heterogeneous feature extractors, and the classifier adopts a unified architecture among all clients. This system can collaboratively learn a set of local models while ensuring the data privacy of the clients, enabling them to achieve the optimal average classification accuracy on their respective clients.

[0061] This embodiment also provides a computer device, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the above-mentioned federated learning method with heterogeneous models, including the following steps: S1: The server synchronously initializes the global classifier to all clients; S2: Each client updates its local model based on its local private dataset; the local model includes a feature extractor and a classifier; S3: Each client uses its feature extractor to generate features, then calculates an entangled feature, and uploads it to the server; The calculation process of the entangled feature is as follows: all samples of the local private dataset generate corresponding features through the feature extractor, apply average pooling operation to the features, and fuse the features after average pooling into an entangled feature; S4: The server updates the parameters of the classifier according to the entangled features of each client and distributes them to the clients; S5: Repeat steps S2 - S4 until the number of iteration rounds is reached.

[0062] This embodiment also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the above-mentioned federated learning method with heterogeneous models, including the following steps: S1: The server synchronously initializes the global classifier to all clients; S2: Each client updates its local model based on its local private dataset; the local model includes a feature extractor and a classifier; S3: Each client uses its feature extractor to generate features, then calculates an entangled feature, and uploads it to the server; The calculation process of the entangled feature is as follows: all samples of the local private dataset generate corresponding features through the feature extractor, apply average pooling operation to the features, and fuse the features after average pooling into an entangled feature; S4: The server updates the parameters of the classifier according to the entangled features of each client and distributes them to the clients; S5: Repeat steps S2 - S4 until the number of iteration rounds is reached.

[0063] This embodiment also provides a computer program product, including a computer program, which when executed by a processor implements the above - mentioned federated learning method with heterogeneous models, including the following steps: S1: The server synchronously initializes the global classifier to all clients; S2: Each client updates its local model based on its local private dataset; the local model includes a feature extractor and a classifier; S3: Each client uses its feature extractor to generate features, then calculates an entangled feature, and uploads it to the server; The calculation process of the entangled feature is as follows: all samples of the local private dataset generate corresponding features through the feature extractor, apply average pooling operation to the features, and fuse the features after average pooling into an entangled feature; S4: The server updates the parameters of the classifier according to the entangled features of each client and distributes them to the clients; S5: Repeat steps S2 - S4 until the number of iteration rounds is reached.

[0064] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A federated learning method with model heterogeneity, characterized in that, It includes the following steps: S1: The server synchronously initializes the global classifier to all clients; S2: Each client updates its local model based on its local private dataset; The local model includes a feature extractor and a classifier; S3: Each client uses its feature extractor to generate features, then calculates an entangled feature, and uploads it to the server; The calculation process of the entangled feature is as follows: All samples of the local private dataset generate corresponding features through the feature extractor, apply average pooling operation to the features, and fuse the features after average pooling into an entangled feature; S4: The server updates the parameters of the classifier according to the entangled features of each client, and distributes them to the clients; S5: Repeat steps S2 - S4 until the number of iteration rounds is reached.

2. The federated learning method with model heterogeneity as claimed in claim 1, wherein, The calculation formula of the entangled feature is as follows: Among them, represents the entanglement representation of the th client, represents the sample set belonging to the category in the th client, represents the category set owned by the th client, represents the th client category after normalized weight, represents the average pooling operation, represents the th client's sample corresponding representation, represents corresponding one-hot encoded label.

3. The federated learning method with model heterogeneity according to claim 2, wherein, Normalized weight vector Each element of the vector is initially randomly sampled from a uniform distribution and then normalized by dividing it by the sum of all its elements so that its sum is 1. , denotes the normalized weight of the th client of class 1, denotes the th client of class and its normalized weight.

4. The federated learning method with model heterogeneity according to claim 1, wherein The clients have heterogeneous feature extractors, while the classifier adopts a unified architecture among all clients.

5. A federated learning method with model heterogeneity as described in claim 1, characterized in that, In S2, the optimization objective of the th client is: Among them, represents the first cross-entropy loss function, which is used to measure the model prediction result of the sample and its corresponding label difference between them, represents optimizing the model parameters of the local model of the th client to minimize its average classification loss on the local private dataset, represents the total number of samples of the th client.

6. The federated learning method with model heterogeneity as claimed in claim 1, wherein, In S4, the optimization objective of the server is: Among them, represents the second cross-entropy loss function, represents optimizing the parameters of the classifier to minimize its corresponding loss function value, represents a classifier with parameters of, represents the one-hot encoded label corresponding to the class of, represents the th client class of the normalized weight, represents the entanglement representation of the th client, represents the th set of classes owned by the client, represents the total number of clients.

7. A federated learning system with model heterogeneity, characterized in that, For implementing a federated learning method with model heterogeneity as described in any one of claims 1 to 6, it includes: 1 server and multiple clients, The server is used to synchronously initialize the global classifier to all clients; update the parameters of the classifier according to the entangled features of each client, and distribute them to the clients; The client is used to independently update its local model based on its local private dataset, and the local model includes a feature extractor and a classifier; the client uses its feature extractor to generate features, then calculates an entangled feature, and uploads it to the server.

8. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements a federated learning method with model heterogeneity as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements a federated learning method with model heterogeneity as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements a federated learning method with model heterogeneity as described in any one of claims 1 to 6.

Images