Heterogeneous federated learning method of data and models based on adaptive aggregation prototype

Abstract

This application proposes a data and model heterogeneous federated learning method based on adaptive aggregated prototypes, which relates to the field of artificial intelligence technology. In the training initialization phase, the client performs supervised learning training on the private data domain; on the server side, the inter-domain distribution similarity matrix is ​​adaptively maintained based on the local prototypes uploaded by each client, and the matrix is ​​used to personalize the aggregated global prototype for each client; on the client side, the personalized global prototype generated by the server in each round of iteration is accepted, the difference between the local prototype and the global prototype of the same type is reduced, the distance with the heterogeneous global prototype is increased, and the correct classification effect of the classifier on the global prototype is optimized; in supervised learning, the distance between the feature extractor output and the global prototype of the same type is reduced; after completing the above training, the local prototype is generated and transmitted to the server. This application can effectively improve the accuracy of the model and the security of collaboration, reduce the negative impact of the heterogeneity of private data, and have the advantage of communication overhead.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a data and model heterogeneous federated learning method based on adaptive aggregation prototypes. Background Art

[0002] Federated learning is an emerging machine learning paradigm focused on data privacy. Its initial goal is to achieve joint modeling and deep learning training while protecting mobile user privacy and data security. Federated learning can be divided into two phases: local training and server aggregation. In the local training phase, clients receive a global model and train it on private data. In the server aggregation phase, the server receives local models sent by clients and aggregates them to generate a new global model. However, in practical applications, traditional federated learning suffers from performance degradation due to the statistical heterogeneity of private data.

[0003] Most personalized federated learning algorithms still perform collaborative training by aggregating model parameters. This collaborative approach incurs significant communication overhead for both clients and servers and requires all clients to have identical model structures and sizes, significantly limiting the application scenarios of federated learning. The exposure of model structure threatens the privacy of private data (GradInversion algorithm). In practical applications, clients often already have local heterogeneous models that are well-suited to local computing resources and private data. To address these issues, heterogeneous federated learning has emerged. It allows clients to have heterogeneous models and collaborate by sharing low- or high-dimensional knowledge about the data, rather than transmitting model parameters. Currently, mainstream heterogeneous federated learning achieves collaboration by using a public dataset. The principle is to aggregate client outputs on the public dataset and use knowledge distillation (KD) techniques to enable clients to learn features from the global data domain. The performance of these methods depends on the choice of the public dataset. When the public dataset is unrelated to the private data, local knowledge transfer is limited, making it difficult to effectively guide global optimization.

[0004] Prototype-based heterogeneous federated learning methods share embedding space class features (class prototypes) to achieve collaborative training. Existing algorithms for generating global prototypes are mainly divided into two types: weighted average and generator-generated global prototypes. The weighted average method aggregates prototypes only from the perspective of data volume, which is easily limited by the performance of local heterogeneous models. The generated global prototype is prone to deviate from the embedding space class features; the difficulty of the generator generating global prototypes lies in the training of the generator. Due to the limited number of local prototypes uploaded in each round of iteration, the generator training loss is large and unstable, making it difficult to converge effectively. In addition, existing algorithms all generate a single prototype for the global data domain, and do not fully consider the difference between the distribution of the global data domain and the private data domain. When the private data domain distribution is far away from the class center, the single global prototype is easy to mislead the local model, causing its performance to deteriorate. Figure 3 This phenomenon is demonstrated in the figure showing the distribution of the private data domains of four clients in the embedding space. The blue solid line in the middle divides the left and right sides into two categories of data. Different colors represent the private data domains of different clients. The diamond represents the global prototype generated by the server based on each client. The black dotted line represents the distance from the private data domain to the global prototype. For the private domain of the yellow client, the distance from the embedding space feature of the data in the class on the left to the global prototype of the label class is greater than the distance to the global prototype of the class on the right. The single global prototype fails to capture the embedding space class features of the private data domain well.

[0005] Therefore, existing federated learning methods have problems such as the heterogeneity of private data leading to reduced local model accuracy, the aggregation performance of client-side heterogeneous models relying on the selection of public datasets, the transmission of model parameters and gradients and other information being prone to reverse deduction attacks leading to private data leakage, high communication overhead in the collaboration phase, and the inability of the generated global prototype to effectively describe the embedding space characteristics of the private data domain. Summary of the Invention

[0006] The present application aims to solve one of the technical problems in the related art at least to a certain extent.

[0007] To this end, the first purpose of this application is to propose a data and model heterogeneous federated learning method based on adaptive aggregate prototypes to improve the accuracy of local models in the face of data heterogeneity, reduce communication overhead in the collaboration phase, and reduce the difference between global prototypes and local prototypes.

[0008] The second objective of this application is to provide an electronic device.

[0009] The third object of this application is to provide a computer-readable storage medium.

[0010] A fourth object of this application is to provide a computer program product.

[0011] To achieve the above objectives, the first embodiment of the present application proposes a data and model heterogeneous federated learning method based on an adaptive aggregate prototype, including:

[0012] S1. Before participating in collaborative training, each client performs supervised learning pre-training locally using private data;

[0013] S2. Each client generates a local prototype and sends it to the server. The server adaptively maintains an inter-domain distribution similarity matrix based on the local prototypes uploaded by each client, and uses this matrix to aggregate a personalized global prototype for each client, generates a personalized global prototype, and sends it to the corresponding client.

[0014] S3. Each client receives the personalized global prototype sent by the server and performs optimization training on the local feature extractor and classifier respectively;

[0015] S4. Each client performs supervised learning locally using private data, and adds the difference loss between the embedding spatial features of the private data and the personalized global prototype to the supervised learning loss.

[0016] Repeat the above steps S2-S4 until the predetermined number of iterations expires.

[0017] Optionally, the S2 includes:

[0018] Each client uses the current local feature extractor to extract class embedding space features for each data category on its private data, namely local prototypes, and uploads the extracted local prototypes to the server;

[0019] The server maintains an inter-domain distribution similarity matrix and adaptively updates the inter-domain distribution similarity matrix according to the local prototypes uploaded by each client;

[0020] Using the updated inter-domain distribution similarity matrix, a new round of personalized global prototypes is generated for each client through linear combination and sent to the corresponding client.

[0021] Optionally, the formula for generating a local prototype by client i is:

[0022]

[0023] Among them, P i,c is the local prototype of category c, D i,c It is client i’s private data D i A subset of D i The data composition of all categories c in f i is a feature extractor, feature extractor f i The parameter is θ i ;

[0024] The inter-domain distribution similarity matrix is ​​defined as Expressed as:

[0025]

[0026] where s 1i ,(i∈M) represents the similarity coefficient of the private data domain of client 1 and client i. The value is distributed between [0,1]. The larger the value, the more similar the private data domain distribution of the two clients is. Conversely, the larger the value, the greater the difference in the private data domain distribution of the two clients. Since the similarity of private data domains has no directionality, the inter-domain distribution similarity matrix s is a symmetric matrix with the property ∑ m∈M s im =1,(i∈M);

[0027] The optimization objective of the inter-domain distribution similarity matrix s is defined as:

[0028]

[0029] Among them, s mean Represents the weighted average initial matrix of the inter-domain distribution similarity matrix s, whose value is defined as ||ss mean || is a regularization term, which is used to ensure that the generated personalized global prototype has a certain degree of generalization ability, prevent the corresponding coefficient from approaching zero due to large differences in private data domains, and promote collaborative communication training between different clients;

[0030] The server calculates the aggregate formula for the personalized global prototype of client i as follows:

[0031]

[0032] in, is the personalized global prototype of data category c of client i, s i1 …s iM ,(iεM) represents the domain similarity of private data of client i and all other clients.

[0033] Optionally, the S3 includes:

[0034] After receiving the personalized global prototype sent by the server, each client performs local collaborative training and uses the corresponding personalized global prototype to optimize the training of the local feature extractor, so that the local prototype is closer to the same type of personalized global prototype and farther away from the different type of personalized global prototype. In addition, the corresponding personalized global prototype is used to optimize the training of the local classifier so that the classifier learns the spatial features embedded in the personalized global prototype.

[0035] Optionally, the local collaborative training is performed in multiple rounds.

[0036] Optionally, during the local collaborative training phase, the feature extractor f of client i i The loss function for:

[0037]

[0038] in, The collection elements in dist represent the feature extractor f i The Euclidean distance between the embedding space features output for category c data x and the personalized global prototype;

[0039] In the local collaborative training phase, the classifier h of client i i The loss function for:

[0040]

[0041] Among them, σ() c Represents the probability value of category c in the output value of the softmax function, h i is the classifier of client i, classifier h i The parameter is w i .

[0042] Optional, the total loss function for the local training phase in S4 for:

[0043]

[0044] Among them, μ is a hyperparameter used to balance the supervised learning loss and the difference loss with the personalized global prototype in the total loss function.

[0045] To achieve the above-mentioned purpose, a second embodiment of the present application provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor;

[0046] The memory stores computer-executable instructions;

[0047] The processor executes the computer-executable instructions stored in the memory to implement the method as described in any one of the first aspects.

[0048] To achieve the above-mentioned purpose, the third embodiment of the present application proposes a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the method as described in any one of the first aspects.

[0049] To achieve the above-mentioned objectives, the fourth embodiment of the present application proposes a computer program product, which implements any one of the methods in the first aspect when executed by a processor.

[0050] The data and model heterogeneous federated learning method, device, electronic device, and storage medium provided in this application based on adaptive aggregate prototype have the following beneficial effects:

[0051] 1. This application implements communication and collaborative training of heterogeneous local models with low communication overhead. The communication overhead of the federated learning algorithm based on the transmission of model parameters and gradients for aggregation is positively correlated with the scale of local model parameters. Taking Resnet as an example, the model parameter size of Resnet18~Resnet152 ranges from 44MB to 223MB. The communication overhead of the client in a round of global iteration is approximately between 88MB and 446MB. In order to have better model accuracy, small-scale networks are used very rarely, and this type of algorithm cannot be applied to scenarios with heterogeneous local models. In federated learning with heterogeneous models, different clients have different model architectures and cannot be directly aggregated by transmitting gradients and model parameters. Some algorithms use public data to obtain the logits output of each client's local model. The default batch size of the public data set per round is 2000. Taking Cifar10 as an example, the communication overhead of transmitting public data per round is about 24MB, and the communication overhead of the logits returned by each client is about 783KB. Some algorithms additionally transmit small models to achieve collaborative training. The generator needs to be downloaded in each round of iteration, which still has a large communication overhead. The present invention realizes communication and collaboration among all clients by transmitting prototypes. The method of transmitting prototypes can realize collaborative training in both homogeneous and heterogeneous models, and has lower communication overhead compared with other methods.

[0052] 2. This application offers improved privacy reliability. Traditional federated learning algorithms expose the structure of the local model and information related to each round of local training, making them vulnerable to reverse inference attacks and resulting in the leakage of private local data. This present invention utilizes the transmission of local prototypes. This communication method does not reveal the specific structure of the local model. Furthermore, the local prototype is the embedding space mean of a category of local data, and this low-dimensional data is irreversible. Therefore, adversaries cannot perform data reverse inference attacks, resulting in improved privacy reliability.

[0053] 3. This application improves the accuracy of local models under data heterogeneity. The global prototype is generated using a simple weighted average, which is easily affected by the performance of heterogeneous models and the global prototype is unstable; a trainable global prototype is maintained on the server side, but the number of local prototypes uploaded in each round is small, and its training is extremely difficult and difficult to converge effectively. The above methods all generate a single type of prototype for the global data domain, without considering the distribution differences in the private data domain. The generated global prototype and the class features of the private data domain are prone to offset. This application is based on adaptive personalized aggregation of global prototypes, constructs an inter-domain distribution similarity matrix, and adapts in a gradient descent manner. The inter-domain distribution similarity matrix should be maintained and updated, and the private data domain embedding space class features should be condensed from the perspective of the global data domain. A personalized global prototype is generated for each client to improve the accuracy of the local model under data heterogeneity.

[0054] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0056] Figure 1 A flowchart of a data and model heterogeneous federated learning method based on an adaptive aggregation prototype provided in an embodiment of the present application;

[0057] Figure 2 A framework diagram of a data and model heterogeneous federated learning method based on an adaptive aggregation prototype provided in an embodiment of the present application;

[0058] Figure 3 This is a schematic diagram of the embedding space distribution diagram of the personalized global prototype and the client private data domain provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] The following describes in detail embodiments of the present application, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0060] In recent years, the field of artificial intelligence has flourished, with researchers continuously proposing high-performance large-scale neural networks. Deep learning has been rapidly applied in various fields, greatly facilitating human life. High-quality deep learning models are often built on large amounts of training data. However, concerns about user privacy and data security are growing. AI companies are misusing personal data, sensitive user data leaks are frequent, and privacy-related laws are gradually being introduced. As a result, many AI application areas face the problem of "data silos," and local datasets are increasingly unable to meet the needs of traditional centralized machine learning training.

[0061] Federated learning is an emerging machine learning paradigm focused on data privacy. Its initial goal is to achieve joint modeling and deep learning training while protecting mobile user privacy and data security. Federated learning can be divided into two phases: local training and server aggregation. In the local training phase, clients receive a global model and train it on private data. In the server aggregation phase, the server receives local models sent by clients and aggregates them to generate a new global model. However, in practical applications, traditional federated learning suffers from performance degradation due to the statistical heterogeneity of private data.

[0062] Most personalized federated learning algorithms still perform collaborative training by aggregating model parameters. This collaborative approach incurs significant communication overhead for both clients and servers and requires all clients to have identical model structures and sizes, significantly limiting the application scenarios of federated learning. The exposure of model structure threatens the privacy of private data (GradInversion algorithm). In practical applications, clients often already have local heterogeneous models that are well-suited to local computing resources and private data. To address these issues, heterogeneous federated learning has emerged. It allows clients to have heterogeneous models and collaborate by sharing low- or high-dimensional knowledge about the data, rather than transmitting model parameters. Currently, mainstream heterogeneous federated learning achieves collaboration by using a public dataset. The principle is to aggregate client outputs on the public dataset and use knowledge distillation (KD) techniques to enable clients to learn features from the global data domain. The performance of these methods depends on the choice of the public dataset. When the public dataset is unrelated to the private data, local knowledge transfer is limited, making it difficult to effectively guide global optimization.

[0063] Prototype-based heterogeneous federated learning methods share embedding space class features (class prototypes) to achieve collaborative training. Existing algorithms for generating global prototypes are mainly divided into two types: weighted average and generator-generated global prototypes. The weighted average method aggregates prototypes only from the perspective of data volume, which is easily limited by the performance of local heterogeneous models. The generated global prototype is prone to deviate from the embedding space class features; the difficulty of the generator generating global prototypes lies in the training of the generator. Due to the limited number of local prototypes uploaded in each round of iteration, the generator training loss is large and unstable, making it difficult to converge effectively. In addition, existing algorithms all generate a single prototype for the global data domain, and do not fully consider the difference between the distribution of the global data domain and the private data domain. When the private data domain distribution is far away from the class center, the single global prototype is easy to mislead the local model, causing its performance to deteriorate. Figure 2 This phenomenon is demonstrated in the figure showing the distribution of the private data domains of four clients in the embedding space. The blue solid line in the middle divides the left and right sides into two categories of data. Different colors represent the private data domains of different clients. The diamond represents the global prototype generated by the server based on each client. The black dotted line represents the distance from the private data domain to the global prototype. For the private domain of the yellow client, the distance from the embedding space feature of the data in the class on the left to the global prototype of the label class is greater than the distance to the global prototype of the class on the right. The single global prototype fails to capture the embedding space class features of the private data domain well.

[0064] Therefore, existing federated learning methods have problems such as the heterogeneity of private data leading to reduced local model accuracy, the aggregation performance of client-side heterogeneous models relying on the selection of public datasets, the transmission of model parameters and gradients and other information being prone to reverse deduction attacks leading to private data leakage, high communication overhead in the collaboration phase, and the inability of the generated global prototype to effectively describe the embedding space characteristics of the private data domain.

[0065] To address this issue, the present invention provides a data and model heterogeneous federated learning method based on adaptive aggregation prototypes. Figure 1 This is a flow chart of a data and model heterogeneous federated learning method based on an adaptive aggregation prototype provided in an embodiment of the present application. Figure 1 As shown, the method includes the following steps:

[0066] S1. Before participating in collaborative training, each client performs supervised learning pre-training using private data locally.

[0067] In the embodiment of the present application, the federated learning scenario involves multiple clients, different clients have local models with different structures and each client has its own private data set. These data sets may differ not only in data content, but also in data distribution. In order for these clients to participate in federated learning and benefit from it, each client first needs to be pre-trained locally. Through this diverse pre-training process, the federated learning system can better adapt to different data environments and application requirements, thereby improving the generalization ability and adaptability of the overall model. Local models of different structures also bring diversity to the federated learning system, which helps to improve the model's adaptability to new data and unknown distributions.

[0068] like Figure 2 As shown, before participating in collaborative training, M clients use heterogeneous local models and Supervised learning on .

[0069] S2. Each client generates a local prototype and sends it to the server. The server adaptively maintains the inter-domain distribution similarity matrix based on the local prototypes uploaded by each client, and uses this matrix to personalize and aggregate the global prototype for each client, generate a personalized global prototype and send it to the corresponding client.

[0070] In the embodiment of the present application, each client uses the current local feature extractor to extract the class embedding space features for each data category on its private data, i.e., the local prototype, and uploads the extracted local prototype to the server. The generation formula of the local prototype generated by client i is:

[0071]

[0072] Among them, P i,c is the local prototype of category c, D i,c It is client i’s private data D i A subset of D i The data composition of all categories c in f i is a feature extractor, feature extractor f i The parameter is θ i .

[0073] Subsequently, each client sends the extracted local prototype to the server.

[0074] In the embodiment of the present application, the server maintains an inter-domain distribution similarity matrix, which is parameterized and adaptively updated according to the local prototypes uploaded by each client. The inter-domain distribution similarity matrix is ​​defined as Expressed as:

[0075]

[0076] Among them, s 1i ,(i∈M) represents the similarity coefficient of the private data domain of client 1 and client i. The value is distributed between [0,1]. The larger the value, the more similar the private data domain distribution of the two clients is. Conversely, the larger the value, the greater the difference in the private data domain distribution of the two clients. Since the similarity of private data domains has no directionality, the inter-domain distribution similarity matrix s is a symmetric matrix with the property ∑ m∈M s im =1,(i∈M).

[0077] It can be understood that since the dimension of the inter-domain distribution similarity matrix s is The distribution of the private data domain is not independent and identically distributed. In this embodiment of the application, the prototype of the missing category of the client is filled with the global average prototype. The calculation formula of the global average prototype of category c is:

[0078]

[0079] Furthermore, in order to better measure the data domain differences between different clients, the inter-domain distribution similarity matrix s is adaptively updated in each collaborative communication phase, and s is parameterized and optimized using a gradient descent method. In the embodiment of the present application, the optimization objective of the inter-domain distribution similarity matrix s is defined as:

[0080]

[0081] Among them, s mean Represents the weighted average initial matrix of the inter-domain distribution similarity matrix s, whose value is defined as ||ss mean || is a regularization term, which is used to ensure that the generated personalized global prototype has a certain generalization ability, prevent the corresponding coefficient from approaching zero due to large differences in private data domains, and promote collaborative communication training among different clients.

[0082] Furthermore, in the tth round of global communication and collaboration, this application defines the adaptive update formula of the inter-domain distribution similarity matrix s as follows:

[0083]

[0084] Among them, s t is the updated inter-domain distribution similarity matrix in round t, s t-1 is the matrix from the previous round, σ represents the softmax function, and η1 is a hyperparameter that controls the magnitude of each adaptive update of s. In the adaptive update formula for s, this application considers the similarity in the distribution of private data domains between different clients, effectively enhancing collaborative training between clients with similar data distributions.

[0085] Then, the embodiment of the present application uses the updated inter-domain distribution similarity matrix to generate a new round of personalized global prototypes for each client linear combination and sends it to the corresponding client. The server calculates the aggregation formula of the personalized global prototype of client i as follows:

[0086]

[0087] in, is the personalized global prototype of data category c of client i, s i1 …s iM ,(i∈M) represents the domain similarity of the private data of client i and all other clients.

[0088] Finally, the server sends the generated personalized global prototype to each corresponding client.

[0089] S3. Each client receives the personalized global prototype sent by the server and optimizes and trains the local feature extractor and classifier respectively.

[0090] In the embodiment of the present application, after each client receives the personalized global prototype sent by the server, it performs local collaborative training. It should be noted that the local collaborative training can be performed in multiple rounds.

[0091] Specifically, the corresponding personalized global prototype is used to optimize the training of the local feature extractor, so that the local prototype is closer to the personalized global prototype of the same type and farther away from the personalized global prototype of different types. In the local collaborative training phase, the feature extractor f of client i i The loss function for:

[0092]

[0093] in, The collection elements in dist represent the feature extractor f i The Euclidean distance between the embedding space features output for category c data x and the personalized global prototype. This loss function can effectively enhance the compactness of the embedding space features output by the feature extractor for data of the same category, while expanding the differences in the embedding space features of data of different categories and enhancing the separation of class features in the embedding space.

[0094] And, the corresponding personalized global prototype is used to optimize the training of the local classifier, so that the classifier learns the spatial class features embedded in the personalized global prototype. In the local collaborative training phase, the classifier h of client i i The loss function for:

[0095]

[0096] Among them, σ() c Represents the probability value of category c in the output value of the softmax function, h i is the classifier of client i, classifier h i The parameter is w i The main purpose of this loss function is to make the classifier h i Learning personalized global prototype P i g , and can correctly classify it, improve h i The recognition ability of embedded spatial features.

[0097] S4. Each client uses private data to perform supervised learning locally, and adds the difference loss between the embedded spatial features of the private data and the personalized global prototype to the supervised learning loss.

[0098] In this embodiment of the application, the client uses private data for supervised learning and adds the difference loss between the embedding space features of the private data and the personalized global prototype to the supervised learning loss, so that the local feature extractor is more similar to the global embedding space class features when mapping each type of data to the embedding space. The total loss function of the local training phase is for:

[0099]

[0100] Among them, μ is a hyperparameter used to balance the supervised learning loss and the difference loss with the personalized global prototype in the total loss function.

[0101] Finally, it is determined whether the number of iteration rounds has been reached. If not, each client regenerates a local prototype and sends it to the server again, that is, repeating the above steps S2-S4 until the predetermined number of iteration rounds expires.

[0102] To prove the effectiveness of the present invention, this application also conducted experiments under two data heterogeneous settings, namely pathological settings and practical settings. In the pathological setting, this application uses two types of data in Cifar10, 10 types of data in Cifar100, and 20 types of data in Tiny ImageNet as private data for each client, where the amount of data is unbalanced, that is, the amount of data for different clients and different classes is different, and there is no redundancy in the data. In the practical setting, this application uses Dirichlet distribution Dir(β) to sample each data class and sends the sampled data to each client, where the β value is set to 0.1 by default. All data sets are divided into training sets and test sets according to 75% and 25%. It is also compared with other federated learning methods.

[0103] The relevant setting parameters are shown in Table 1.

[0104] Table 1

[0105]

[0106] According to Table 1, for pathological settings:

[0107] On the Cifar10 dataset, this application achieved an accuracy of 87.99%, an improvement over the highest accuracy of 86.94% achieved by other methods (FedGen). On the Cifar100 dataset, this application achieved an accuracy of 44.53%, surpassing the highest accuracy of 42.98% achieved by other methods (FedMD). On the Tiny ImageNet dataset, this application achieved an accuracy of 33.15%, surpassing the highest accuracy of 32.43% achieved by other methods (FedDistill).

[0108] For the actual setup:

[0109] On the Cifar10 dataset, this application achieved an accuracy of 87.44%, surpassing the highest accuracy of 86.64% among other methods (FedGen). On the Cifar100 dataset, this application achieved an accuracy of 61.31%, surpassing the highest accuracy of 59.55% among other methods (FedGen). On the Tiny ImageNet dataset, this application achieved an accuracy of 40.94%, significantly surpassing the highest accuracy of 38.97% among other methods (FedDistill).

[0110] The comparison results show that the proposed method performs well across different datasets and in different data heterogeneity settings. In particular, the proposed method has a more pronounced advantage on the Tiny ImageNet dataset, which may be attributed to its stronger ability to handle data imbalance and heterogeneity.

[0111] In addition, the performance of this application in the practical setting is also better than that in the pathological setting. This may be because the Dirichlet distribution is used for data sampling in the practical setting. This sampling method may be closer to the distribution of data in the real world, and the method of this application is just suitable for processing this type of data distribution.

[0112] In summary, the experimental results demonstrate the effectiveness of this application, especially in dealing with data heterogeneity and imbalance

[0113] In order to implement the above embodiments, the present application also proposes an electronic device, comprising: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method provided by the above embodiments.

[0114] In order to implement the above embodiments, the present application also proposes a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the methods provided by the above embodiments.

[0115] In order to implement the above embodiments, the present application also proposes a computer program product, including a computer program, which implements the methods provided by the above embodiments when executed by a processor.

[0116] The collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in this application are in compliance with relevant laws and regulations and do not violate public order and good morals.

[0117] It is important to note that personal information collected from users should be used for legitimate and reasonable purposes and should not be shared or sold beyond these legitimate uses. Furthermore, such collection / sharing should be conducted only after receiving the user's informed consent, including but not limited to notifying the user to read the user agreement / user notice and sign an agreement / authorization that includes the relevant user information before using the feature. Furthermore, any necessary steps must be taken to safeguard and secure access to such personal information and ensure that others with access to personal information comply with its privacy policy and procedures.

[0118] This application contemplates providing implementations that allow users to selectively block the use or access of personal information data. Specifically, this disclosure contemplates providing hardware and / or software to prevent or block access to such personal information data. Risks can be minimized by limiting data collection and deleting data once it is no longer needed. Furthermore, where applicable, such personal information can be de-identified to protect user privacy.

[0119] In the descriptions of the foregoing embodiments, the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are mutually inconsistent.

[0120] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of such features. Throughout the description of this application, "plurality" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0121] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application belong.

[0122] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and a portable compact disc read-only memory (CDROM). Furthermore, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing it in another suitable manner if necessary, and then storing it in a computer memory.

[0123] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0124] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0125] In addition, the functional units in the various embodiments of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into a module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0126] The storage medium mentioned above may be a read-only memory, a magnetic disk, or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present application. Persons skilled in the art may make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application.

Claims

1. A data and model heterogeneous federated learning method based on adaptive aggregation prototype, used in federated learning scenarios, characterized by: The following steps are involved: S1. Before participating in collaborative training, each client uses private data to perform supervised learning pre-training locally. Clients use heterogeneous local models, in private heterogeneous data Supervised learning on S2. Each client generates a local prototype and sends it to the server. The server adaptively maintains an inter-domain distribution similarity matrix based on the local prototypes uploaded by each client, and uses this matrix to aggregate a personalized global prototype for each client, generates a personalized global prototype, and sends it to the corresponding client. S3. Each client receives the personalized global prototype sent by the server and performs optimization training on the local feature extractor and classifier respectively; S4. Each client performs supervised learning locally using private data, and adds the difference loss between the embedded spatial features of the private data and the personalized global prototype to the supervised learning loss. Repeat the above steps S2-S4 until the predetermined number of iterations is reached; Said S2 comprises: Each client uses the current local feature extractor to extract class embedding space features for each data category on its private data, namely local prototypes, and uploads the extracted local prototypes to the server; The server maintains an inter-domain distribution similarity matrix and adaptively updates the inter-domain distribution similarity matrix according to the local prototypes uploaded by each client; Using the updated inter-domain distribution similarity matrix, a new round of personalized global prototypes is generated for each client through linear combination, and sent to the corresponding client; Client The generation formula for generating local prototypes is: in, is a category The local prototype, Is the client Private data A subset of All categories The data composition of It is a feature extractor, feature extractor The parameters are ; The inter-domain distribution similarity matrix is ​​defined as , expressed as: in Indicates client 1 and client The private data domain similarity coefficient of The larger the value, the more similar the private data domain distribution of the two clients is. Conversely, the greater the difference in the private data domain distribution of the two clients is. Since the similarity of private data domains has no directionality, the inter-domain distribution similarity matrix is a symmetric matrix with the property .

2. The method according to claim 1, characterized in that The inter-domain distribution similarity matrix The optimization objective is defined as: in, Represents the inter-domain distribution similarity matrix The weighted average initial matrix of , It is a regularization term used to ensure that the generated personalized global prototype has a certain degree of generalization ability, prevent the corresponding coefficient from approaching zero due to large differences in private data domains, and promote collaborative communication training between different clients; Server Computing Client The aggregation formula of the personalized global prototype is: in, For the client Data categories The personalized global prototype of Represents the client Domain similarity with all other clients' private data.

3. The method according to claim 2, characterized in that The S3 includes: After receiving the personalized global prototype sent by the server, each client performs local collaborative training and uses the corresponding personalized global prototype to optimize the training of the local feature extractor, so that the local prototype is closer to the same type of personalized global prototype and farther away from the different type of personalized global prototype. In addition, the corresponding personalized global prototype is used to optimize the training of the local classifier so that the classifier learns the spatial features embedded in the personalized global prototype.

4. The method according to claim 3, characterized in that The local collaborative training is performed for multiple rounds.

5. The method according to claim 3, characterized in that During the local collaborative training phase, the client Feature extractor The loss function for: in, , The collection elements in represent feature extractors For categories data The Euclidean distance between the output embedding space features and the personalized global prototype; During the local collaborative training phase, the client Classifier The loss function for: in, Indicates the output value of the softmax function for the category The probability value of Is the client Classifier, classifier The parameters are .

6. The method according to claim 5, characterized in that The total loss function of the local training phase in S4 for: in, is a hyperparameter used to balance the supervised learning loss and the difference loss with the personalized global prototype in the total loss function.

7. An electronic device, characterized in that: include: a processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 6 when executed by a processor.

9. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 6 when the computer program is executed by a processor.

Images