Data and model heterogeneous federated learning method based on adaptive aggregation prototype

Abstract

The invention provides a data and model heterogeneous federal learning method based on an adaptive aggregation prototype, and relates to the technical field of artificial intelligence. In a training initialization stage, a client performs supervised learning training on a private data field; the method comprises the following steps: at a server side, adaptively maintaining an inter-domain distribution similar matrix according to a local prototype uploaded by each client side, and carrying out personalized aggregation on a global prototype for each client side by using the matrix; at a client, receiving a personalized global prototype generated by each round of iteration of the server, reducing the difference between a local prototype and a similar global prototype, increasing the distance between the local prototype and a heterogeneous global prototype, and optimizing the correct classification effect of a classifier on the global prototype; in supervised learning, the distance between the output of a feature extractor and a similar global prototype is reduced; and after the training is completed, generating a local prototype and transmitting the local prototype to a server. The method can effectively improve the model accuracy and cooperation security, reduces the negative influence of private data heterogeneity, and has the advantage of communication overhead.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a data and model heterogeneous federated learning method based on an adaptive aggregation prototype. Background Art

[0002] Federated learning is an emerging machine learning paradigm for data privacy protection. Its initial goal is to achieve joint modeling and complete deep learning training on the basis of protecting mobile user privacy and data security. Federated learning can be divided into two stages: local training stage and server aggregation stage. In the local training stage, the client receives the global model for training on private data. In the server aggregation stage, the server receives the local model sent by the client and aggregates it to generate a new round of global models. However, in practical applications, traditional federated learning has a performance degradation problem due to the statistical heterogeneity of private data.

[0003] Most personalized federated learning algorithms still perform collaborative training by aggregating model parameters. This collaborative approach brings huge communication overhead to the client and server, and requires all clients to have the same model structure and size, which greatly limits the application scenarios of federated learning. The exposure of the model structure threatens the privacy of private data (GradInversion algorithm). In actual application scenarios, clients often have built local heterogeneous models, which can adapt well to local storage resources and private data. In order to solve these problems, heterogeneous federated learning came into being, which allows clients to have heterogeneous models and achieve collaboration by sharing low-dimensional or high-dimensional knowledge of data instead of transmitting model parameters. At present, mainstream heterogeneous federated learning achieves collaboration by using a public dataset. Its principle is to aggregate the output of the client to the public dataset and use the knowledge distillation (KD algorithm) technology to enable the client to learn the features in the global data domain. The performance of this type of method depends on the selection of the public dataset. When the public dataset is not related to the private data, the local knowledge transfer is limited and cannot guide the global optimization direction well.

[0004] Prototype-based heterogeneous federated learning methods share embedding space class features (class prototypes) to achieve collaborative training. Existing algorithms generate global prototypes in two main types: weighted average and generator-generated global prototypes. The weighted average method aggregates prototypes only from the perspective of data volume, which is easily limited by the performance of local heterogeneous models. The generated global prototype is prone to deviate from the embedding space class features. The difficulty of the generator generating a global prototype lies in the training of the generator. Due to the limited number of local prototypes uploaded in each round of iteration, the generator training loss is large and unstable, making it difficult to converge effectively. In addition, existing algorithms all generate a single prototype for the global data domain, without fully considering the difference between the distribution of the global data domain and the private data domain. When the private data domain distribution is far away from the class center, the single global prototype is easy to mislead the local model, causing its performance to deteriorate. Figure 3 This phenomenon is demonstrated in the figure which shows the distribution of private data domains of four clients in the embedding space. The blue solid line in the middle divides the left and right sides into two categories of data. Different colors represent the private data domains of different clients. The diamond represents the global prototype generated by the server according to each client. The black dotted line represents the distance from the private data domain to the global prototype. For the private domain of the yellow client, the distance from the embedding space feature of the data in the class on the left to the global prototype of the label class is greater than the distance to the global prototype of the class on the right. The single global prototype fails to capture the embedding space class features of the private data domain well.

[0005] Therefore, existing federated learning methods have problems such as the heterogeneity of private data leading to reduced local model accuracy, the aggregation performance of client-side heterogeneous models relying on the selection of public data sets, the transmission of model parameters and gradient information being prone to reverse deduction attacks leading to private data leakage, high communication overhead in the collaboration phase, and the inability of the generated global prototype to effectively describe the embedding space characteristics of the private data domain. Summary of the invention

[0006] The present application aims to solve one of the technical problems in the related art at least to some extent.

[0007] To this end, the first purpose of this application is to propose a data and model heterogeneous federated learning method based on adaptive aggregate prototypes, to improve the accuracy of local models when facing data heterogeneity, to reduce the communication overhead in the collaboration phase, and to reduce the difference between the global prototype and the local prototype.

[0008] The second objective of the present application is to provide an electronic device.

[0009] The third object of the present application is to provide a computer-readable storage medium.

[0010] A fourth object of the present application is to provide a computer program product.

[0011] To achieve the above objectives, the first embodiment of the present application proposes a data and model heterogeneous federated learning method based on an adaptive aggregation prototype, including:

[0012] S1. Before participating in collaborative training, each client uses private data to perform supervised learning pre-training locally;

[0013] S2. Each client generates a local prototype and sends it to the server. The server adaptively maintains the inter-domain distribution similarity matrix based on the local prototypes uploaded by each client, and uses the matrix to aggregate the global prototype for each client, generates a personalized global prototype and sends it to the corresponding client.

[0014] S3, each client receives the personalized global prototype sent by the server, and performs optimization training on the local feature extractor and classifier respectively;

[0015] S4. Each client uses the private data to perform supervised learning locally, and adds the difference loss between the embedded spatial features of the private data and the personalized global prototype to the supervised learning loss;

[0016] Repeat the above steps S2-S4 until the predetermined number of iterations expires.

[0017] Optionally, S2 includes:

[0018] Each client uses the current local feature extractor to extract class embedding space features, i.e., local prototypes, for each data category on its private data, and uploads the extracted local prototypes to the server;

[0019] The server maintains an inter-domain distribution similarity matrix, and adaptively updates the inter-domain distribution similarity matrix according to the local prototypes uploaded by each client;

[0020] Using the updated inter-domain distribution similarity matrix, a new round of personalized global prototypes is generated for each client through linear combination, and sent to the corresponding client.

[0021] Optionally, the generation formula for the local prototype generated by client i is:

[0022]

[0023] Among them, P i,c is the local prototype of category c, D i,c It is client i private data D i A subset of i The data composition of all categories c in f i is a feature extractor, feature extractor f i The parameter is θ i ;

[0024] The inter-domain distribution similarity matrix is ​​defined as It is expressed as:

[0025]

[0026] where s 1i ,(i∈M) represents the similarity coefficient of the private data domain of client 1 and client i. The value is distributed between [0,1]. The larger the value, the more similar the private data domain distribution of the two clients is. On the contrary, the larger the value, the greater the difference in the private data domain distribution of the two clients is. Since the similarity of private data domains has no directionality, the inter-domain distribution similarity matrix s is a symmetric matrix with the property ∑ m∈M s im =1,(i∈M);

[0027] The optimization objective of the inter-domain distribution similarity matrix s is defined as:

[0028]

[0029] Among them, s mean Represents the weighted average initial matrix of the inter-domain distribution similarity matrix s, whose value is defined as ||ss mean || is a regularization term, which is used to ensure that the generated personalized global prototype has a certain generalization ability, prevent the corresponding coefficient from approaching zero due to large differences in private data domains, and promote collaborative communication training between different clients;

[0030] The server calculates the aggregate formula for the personalized global prototype of client i as follows:

[0031]

[0032] in, is the personalized global prototype of data category c of client i, s i1 …s iM ,(iεM) represents the domain similarity of private data of client i and all other clients.

[0033] Optionally, S3 includes:

[0034] After receiving the personalized global prototype sent by the server, each client performs local collaborative training and uses the corresponding personalized global prototype to optimize the training of the local feature extractor, so that the local prototype is closer to the same personalized global prototype and farther away from the different personalized global prototype. In addition, the corresponding personalized global prototype is used to optimize the training of the local classifier so that the classifier learns the spatial class features embedded in the personalized global prototype.

[0035] Optionally, the local collaborative training is performed in multiple rounds.

[0036] Optionally, during the local collaborative training phase, the feature extractor f of client i i The loss function for:

[0037]

[0038] in, The collection elements in dist represent the feature extractor f i The Euclidean distance between the embedding space features output for category c data x and the personalized global prototype;

[0039] In the local collaborative training phase, the classifier h of client i i The loss function for:

[0040]

[0041] Among them, σ() c represents the probability value of category c in the output value of the softmax function, h i is the classifier of client i, classifier h i The parameter is w i .

[0042] Optional, the total loss function for the local training phase in S4 for:

[0043]

[0044] Among them, μ is a hyperparameter used to balance the supervised learning loss and the difference loss with the personalized global prototype in the total loss function.

[0045] To achieve the above-mentioned purpose, a second aspect of the present application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0046] The memory stores computer-executable instructions;

[0047] The processor executes the computer-executable instructions stored in the memory to implement the method as described in any one of the first aspects.

[0048] To achieve the above-mentioned purpose, the third aspect embodiment of the present application proposes a computer-readable storage medium, in which computer-readable storage medium is stored computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method as described in any one of the first aspects.

[0049] To achieve the above-mentioned purpose, the fourth aspect of the present application provides a computer program product, which implements any method in the first aspect when executed by a processor.

[0050] The data and model heterogeneous federated learning method, device, electronic device and storage medium based on adaptive aggregation prototype provided in this application have the following beneficial effects:

[0051] 1. This application realizes the communication and collaborative training of heterogeneous local models with low communication overhead. The communication overhead of the federated learning algorithm based on the transmission of model parameters and gradients for aggregation is positively correlated with the scale of local model parameters. Taking Resnet as an example, the model parameter size range of Resnet18~Resnet152 is between 44MB and 223MB, and the communication overhead of the client in a round of global iteration is approximately between 88MB and 446MB. In order to have better model accuracy, small-scale networks are used very rarely, and this type of algorithm cannot be applied to scenarios with heterogeneous local models. In model heterogeneous federated learning, different clients have different model architectures and cannot be directly aggregated by transmitting gradients and model parameters; some algorithms use public data to obtain the logits output of each client's local model. The default batch size of the public data set per round is 2000. Taking Cifar10 as an example, the communication overhead of transmitting public data per round is about 24MB, and the communication overhead of logits returned by each client is about 783KB; some algorithms transmit small models additionally to achieve collaborative training, and the generator needs to be downloaded in each round of iteration, which still has a large communication overhead. The present invention realizes communication and collaboration among all clients by transmitting prototypes. The method of transmitting prototypes can realize collaborative training in both homogeneous and heterogeneous models, and has smaller communication overhead than other methods.

[0052] 2. This application has better privacy reliability. Traditional federated learning algorithms will expose the structure of the local model and the relevant information of each round of local training, which is prone to reverse deduction attacks, resulting in privacy leakage of local private data. The present invention takes the form of transmitting local prototypes. This communication method will not disclose the specific structure of the local model, and the local prototype is the embedding space mean of a class of local data. This low-dimensional data is irreversible. Therefore, the adversary cannot perform data reverse reasoning attacks, and thus has better privacy reliability.

[0053] 3. This application improves the accuracy of local models in the case of data heterogeneity. The global prototype is generated using a simple weighted average, which is easily affected by the performance of heterogeneous models and the global prototype is unstable; a trainable global prototype is maintained on the server side, but the number of local prototypes uploaded in each round is small, and its training is extremely difficult and difficult to converge effectively. The above methods all generate a single type of prototype for the global data domain, without considering the distribution differences in the private data domain, and the generated global prototype and private data domain class features are prone to offset. This application is based on adaptive personalized aggregation of global prototypes, constructs an inter-domain distribution similarity matrix, and adapts in a gradient descent manner. The inter-domain distribution similarity matrix should be maintained and updated, and the private data domain embedding space class features should be condensed from the perspective of the global data domain. A personalized global prototype is generated for each client to improve the accuracy of the local model in the case of data heterogeneity.

[0054] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0056] Figure 1 A flowchart of a data and model heterogeneous federated learning method based on an adaptive aggregation prototype provided in an embodiment of the present application;

[0057] Figure 2 A framework diagram of a data and model heterogeneous federated learning method based on an adaptive aggregation prototype provided in an embodiment of the present application;

[0058] Figure 3 A schematic diagram of the embedded space distribution diagram of the personalized global prototype and the client private data domain provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0060] In recent years, the field of artificial intelligence has flourished. Researchers have continuously proposed large neural networks with good performance. Deep learning has been rapidly applied in various fields, greatly facilitating human life. High-quality deep learning models are often built on a large amount of training data. However, the attention to issues related to user privacy and data security has continued to increase. Artificial intelligence companies have abused personal privacy data, and user sensitive privacy data has been frequently leaked. Privacy protection-related laws have gradually been introduced. There are "data islands" in many application fields of artificial intelligence, and the scale of local data sets has gradually been unable to meet the needs of traditional centralized machine learning training.

[0061] Federated learning is an emerging machine learning paradigm for data privacy protection. Its initial goal is to achieve joint modeling and complete deep learning training on the basis of protecting mobile user privacy and data security. Federated learning can be divided into two stages: local training stage and server aggregation stage. In the local training stage, the client receives the global model for training on private data. In the server aggregation stage, the server receives the local model sent by the client and aggregates it to generate a new round of global models. However, in practical applications, traditional federated learning has a performance degradation problem due to the statistical heterogeneity of private data.

[0062] Most personalized federated learning algorithms still perform collaborative training by aggregating model parameters. This collaborative approach brings huge communication overhead to the client and server, and requires all clients to have the same model structure and size, which greatly limits the application scenarios of federated learning. The exposure of the model structure threatens the privacy of private data (GradInversion algorithm). In actual application scenarios, clients often have built local heterogeneous models, which can adapt well to local storage resources and private data. In order to solve these problems, heterogeneous federated learning came into being, which allows clients to have heterogeneous models and achieve collaboration by sharing low-dimensional or high-dimensional knowledge of data instead of transmitting model parameters. At present, mainstream heterogeneous federated learning achieves collaboration by using a public dataset. Its principle is to aggregate the output of the client to the public dataset and use the knowledge distillation (KD algorithm) technology to enable the client to learn the features in the global data domain. The performance of this type of method depends on the selection of the public dataset. When the public dataset is not related to the private data, the local knowledge transfer is limited and cannot guide the global optimization direction well.

[0063] Prototype-based heterogeneous federated learning methods share embedding space class features (class prototypes) to achieve collaborative training. Existing algorithms generate global prototypes in two main types: weighted average and generator-generated global prototypes. The weighted average method aggregates prototypes only from the perspective of data volume, which is easily limited by the performance of local heterogeneous models. The generated global prototype is prone to deviate from the embedding space class features. The difficulty of the generator generating a global prototype lies in the training of the generator. Due to the limited number of local prototypes uploaded in each round of iteration, the generator training loss is large and unstable, making it difficult to converge effectively. In addition, existing algorithms all generate a single prototype for the global data domain, without fully considering the difference between the distribution of the global data domain and the private data domain. When the private data domain distribution is far away from the class center, the single global prototype is easy to mislead the local model, causing its performance to deteriorate. Figure 2 This phenomenon is demonstrated in the figure which shows the distribution of private data domains of four clients in the embedding space. The blue solid line in the middle divides the left and right sides into two categories of data. Different colors represent the private data domains of different clients. The diamond represents the global prototype generated by the server according to each client. The black dotted line represents the distance from the private data domain to the global prototype. For the private domain of the yellow client, the distance from the embedding space feature of the data in the class on the left to the global prototype of the label class is greater than the distance to the global prototype of the class on the right. The single global prototype fails to capture the embedding space class features of the private data domain well.

[0064] Therefore, existing federated learning methods have problems such as the heterogeneity of private data leading to reduced local model accuracy, the aggregation performance of client-side heterogeneous models relying on the selection of public data sets, the transmission of model parameters and gradient information being prone to reverse deduction attacks leading to private data leakage, high communication overhead in the collaboration phase, and the inability of the generated global prototype to effectively describe the embedding space characteristics of the private data domain.

[0065] To address this problem, the present application provides a data and model heterogeneous federated learning method based on an adaptive aggregation prototype. Figure 1 A schematic diagram of a data and model heterogeneous federated learning method based on an adaptive aggregation prototype provided in an embodiment of the present application. Figure 1 As shown, the method comprises the following steps:

[0066] S1. Before participating in collaborative training, each client uses private data to perform supervised learning pre-training locally.

[0067] In an embodiment of the present application, the federated learning scenario involves multiple clients, different clients have local models with different structures and each client has its own private data set. These data sets may differ not only in data content, but also in data distribution. In order for these clients to participate in federated learning and benefit from it, each client first needs to be pre-trained locally. Through this diverse pre-training process, the federated learning system can better adapt to different data environments and application requirements, thereby improving the generalization and adaptability of the overall model. Local models of different structures also bring diversity to the federated learning system, which helps to improve the model's adaptability to new data and unknown distributions.

[0068] like Figure 2 As shown in the figure, before participating in collaborative training, M clients use heterogeneous local models and private heterogeneous data Supervised learning on .

[0069] S2. Each client generates a local prototype and sends it to the server. The server adaptively maintains the inter-domain distribution similarity matrix based on the local prototypes uploaded by each client, and uses the matrix to personalize and aggregate the global prototype for each client, generates a personalized global prototype and sends it to the corresponding client.

[0070] In the embodiment of the present application, each client uses the current local feature extractor to extract the class embedding space feature, i.e., the local prototype, on its private data for each data category, and uploads the extracted local prototype to the server. The generation formula of the local prototype generated by client i is:

[0071]

[0072] Among them, P i,c is the local prototype of category c, D i,c It is client i private data D i A subset of i The data composition of all categories c in f i is a feature extractor, feature extractor f i The parameter is θ i .

[0073] Then, each client sends the extracted local prototype to the server.

[0074] In the embodiment of the present application, the server maintains an inter-domain distribution similarity matrix, which is parameterized and adaptively updated according to the local prototypes uploaded by each client, wherein the inter-domain distribution similarity matrix is ​​defined as It is expressed as:

[0075]

[0076] Among them, s 1i ,(i∈M) represents the similarity coefficient of the private data domain of client 1 and client i. The value is distributed between [0,1]. The larger the value, the more similar the private data domain distribution of the two clients is. On the contrary, the larger the value, the greater the difference in the private data domain distribution of the two clients is. Since the similarity of private data domains has no directionality, the inter-domain distribution similarity matrix s is a symmetric matrix with the property ∑ m∈M s im =1,(i∈M).

[0077] It can be understood that since the dimension of the inter-domain distribution similarity matrix s is The distribution of the private data domain is not independent and identically distributed. In the embodiment of the present application, the prototype of the missing category of the client is filled with the global average prototype, where the calculation formula of the global average prototype of category c is:

[0078]

[0079] Furthermore, in order to better measure the data domain differences between different clients, the inter-domain distribution similarity matrix s is adaptively updated in each collaborative communication phase, and s is parameterized and optimized using a gradient descent method. In the embodiment of the present application, the optimization target of the inter-domain distribution similarity matrix s is defined as:

[0080]

[0081] Among them, s mean Represents the weighted average initial matrix of the inter-domain distribution similarity matrix s, whose value is defined as ||ss mean || is a regularization term, which is used to ensure that the generated personalized global prototype has a certain generalization ability, prevent the corresponding coefficient from approaching zero due to large differences in private data domains, and promote collaborative communication training among different clients.

[0082] Furthermore, in the tth round of global communication and collaboration, the present application defines an adaptive update formula for the inter-domain distribution similarity matrix s as follows:

[0083]

[0084] Among them, s t is the updated inter-domain distribution similarity matrix in round t, s t-1 is the matrix of the previous round, σ represents the softmax function, and η1 is a hyperparameter that controls the amplitude of each adaptive update of s. In the adaptive update formula of s, this application considers the similarity of distribution between private data domains of different clients, which can effectively enhance the collaborative training between clients with similar data distribution.

[0085] Then, the embodiment of the present application uses the updated inter-domain distribution similarity matrix to generate a new round of personalized global prototypes for each client linear combination and sends it to the corresponding client. The server calculates the aggregation formula of the personalized global prototype of client i as follows:

[0086]

[0087] in, is the personalized global prototype of data category c of client i, s i1 …s iM ,(i∈M) represents the domain similarity of private data of client i and all other clients.

[0088] Finally, the server sends the generated personalized global prototype to each corresponding client.

[0089] S3. Each client receives the personalized global prototype sent by the server and optimizes and trains the local feature extractor and classifier respectively.

[0090] In the embodiment of the present application, after each client receives the personalized global prototype sent by the server, it performs local collaborative training. It should be noted that the local collaborative training can be performed in multiple rounds.

[0091] Specifically, the corresponding personalized global prototype is used to optimize the training of the local feature extractor, so that the local prototype is closer to the same personalized global prototype and farther from the different personalized global prototype. i The loss function for:

[0092]

[0093] in, The collection elements in dist represent the feature extractor f i The Euclidean distance between the embedding space features output by the category c data x and the personalized global prototype. This loss function can effectively enhance the compactness of the embedding space features output by the feature extractor for the same category of data, while expanding the differences in the embedding space features of different categories of data and enhancing the separability of the embedding space class features.

[0094] And, the corresponding personalized global prototype is used to optimize the local classifier training, so that the classifier learns the spatial class features embedded in the personalized global prototype. In the local collaborative training stage, the classifier h of client i i The loss function for:

[0095]

[0096] Among them, σ() c represents the probability value of category c in the output value of the softmax function, h i is the classifier of client i, classifier h i The parameter is w i The main purpose of this loss function is to make the classifier h i Learning a personalized global prototype P i g , and can correctly classify it, improving h i The recognition ability of embedded space features.

[0097] S4. Each client uses private data to perform supervised learning locally, and adds the difference loss between the embedded spatial features of the private data and the personalized global prototype to the supervised learning loss.

[0098] In the embodiment of the present application, the client uses private data for supervised learning, and adds the difference loss between the embedding space features of the private data and the personalized global prototype to the supervised learning loss, so that the local feature extractor is more similar to the global embedding space class features when mapping each type of data to the embedding space. The total loss function of the local training phase is for:

[0099]

[0100] Among them, μ is a hyperparameter used to balance the supervised learning loss and the difference loss with the personalized global prototype in the total loss function.

[0101] Finally, it is determined whether the number of iteration rounds has been reached. If not, each client regenerates a local prototype and sends it to the server again, that is, the above steps S2-S4 are repeated until the predetermined number of iteration rounds expires.

[0102] To prove the effectiveness of the present invention, this application also conducted experiments under two data heterogeneous settings, namely pathological settings and practical settings. In the pathological setting, this application uses two types of data in Cifar10, 10 types of data in Cifar100, and 20 types of data in Tiny ImageNet as private data for each client, where the amount of data is unbalanced, that is, the amount of data for different clients and different classes is different, and there is no redundancy in the data. In the practical setting, this application uses Dirichlet distribution Dir (β) to sample each data class, and sends the sampled data to each client, where the β value is set to 0.1 by default. All data sets are divided into training sets and test sets according to 75% and 25%. And compared with other federated learning methods.

[0103] The relevant setting parameters are shown in Table 1.

[0104] Table 1

[0105]

[0106] According to Table 1, for pathological settings:

[0107] On the Cifar10 dataset, this application achieved an accuracy of 87.99%, which is higher than the highest accuracy of 86.94% (FedGen) among other methods. On the Cifar100 dataset, this application achieved an accuracy of 44.53%, which is higher than the highest accuracy of 42.98% (FedMD) among other methods. On the Tiny ImageNet dataset, this application achieved an accuracy of 33.15%, which is better than the highest accuracy of 32.43% (FedDistill) among other methods.

[0108] For the actual setup:

[0109] On the Cifar10 dataset, the accuracy of this application is 87.44%, which is higher than the highest accuracy of 86.64% (FedGen) among other methods. On the Cifar100 dataset, the accuracy of this application is 61.31%, which is higher than the highest accuracy of 59.55% (FedGen) among other methods. On the Tiny ImageNet dataset, the accuracy of this application is 40.94%, which is significantly higher than the highest accuracy of 38.97% (FedDistill) among other methods.

[0110] From the comparison results, it can be seen that the present application shows good performance under different data sets and different data heterogeneity settings. Especially on the Tiny ImageNet data set, the advantages of the present application are more obvious, which may be attributed to the present application's stronger ability to handle data imbalance and heterogeneity.

[0111] In addition, the performance of the present application in the practical setting is also better than that in the pathological setting. This may be because the Dirichlet distribution is used for data sampling in the practical setting. This sampling method may be closer to the distribution of data in the real world, and the method of the present application is just suitable for processing this type of data distribution.

[0112] In summary, the experimental results demonstrate the effectiveness of this application, especially in dealing with data heterogeneity and imbalance

[0113] In order to implement the above embodiments, the present application also proposes an electronic device, comprising: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method provided by the above embodiments.

[0114] In order to implement the above embodiments, the present application also proposes a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the methods provided by the above embodiments.

[0115] In order to implement the above embodiments, the present application also proposes a computer program product, including a computer program, which implements the methods provided by the above embodiments when executed by a processor.

[0116] The collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in this application are in compliance with relevant laws and regulations and do not violate public order and good morals.

[0117] It should be noted that personal information from users should be collected for legitimate and reasonable purposes and should not be shared or sold outside of these legitimate uses. In addition, such collection / sharing should be carried out after receiving the user's informed consent, including but not limited to notifying the user to read the user agreement / user notice and sign the agreement / authorization including authorization of relevant user information before the user uses the function. In addition, any necessary steps should be taken to protect and safeguard access to such personal information data and ensure that others who have access to personal information data comply with its privacy policy and procedures.

[0118] The present application is expected to provide an implementation scheme for users to selectively block the use or access of personal information data. That is, the present disclosure is expected to provide hardware and / or software to prevent or block access to such personal information data. Once the personal information data is no longer needed, the risk can be minimized by limiting data collection and deleting the data. In addition, when applicable, such personal information is de-identified to protect the privacy of the user.

[0119] In the description of the aforementioned embodiments, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0120] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0121] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0122] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0123] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0124] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0125] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0126] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A data and model heterogeneous federated learning method based on adaptive aggregation prototype, characterized in that: The following steps are involved: S1. Before participating in collaborative training, each client uses private data to perform supervised learning pre-training locally; S2. Each client generates a local prototype and sends it to the server. The server adaptively maintains the inter-domain distribution similarity matrix based on the local prototypes uploaded by each client, and uses the matrix to aggregate the global prototype for each client, generates a personalized global prototype and sends it to the corresponding client. S3, each client receives the personalized global prototype sent by the server, and performs optimization training on the local feature extractor and classifier respectively; S4. Each client uses the private data to perform supervised learning locally, and adds the difference loss between the embedded spatial features of the private data and the personalized global prototype to the supervised learning loss; Repeat the above steps S2-S4 until the predetermined number of iterations expires.

2. The method according to claim 1, characterized in that The S2 comprises: Each client uses the current local feature extractor to extract class embedding space features, i.e., local prototypes, for each data category on its private data, and uploads the extracted local prototypes to the server; The server maintains an inter-domain distribution similarity matrix, and adaptively updates the inter-domain distribution similarity matrix according to the local prototypes uploaded by each client; Using the updated inter-domain distribution similarity matrix, a new round of personalized global prototypes is generated for each client through linear combination, and sent to the corresponding client.

3. The method according to claim 2, characterized in that The generation formula for the local prototype generated by client i is: Among them, P i,c is the local prototype of category c, D i,c It is client i private data D i A subset of i The data composition of all categories c in f i is a feature extractor, feature extractor f i The parameter is θ i ; The inter-domain distribution similarity matrix is ​​defined as It is expressed as: where s 1i ,(i∈M) represents the similarity coefficient of the private data domain of client 1 and client i. The value is distributed between [0,1]. The larger the value, the more similar the private data domain distribution of the two clients is. On the contrary, the larger the value, the greater the difference in the private data domain distribution of the two clients is. Since the similarity of private data domains has no directionality, the inter-domain distribution similarity matrix s is a symmetric matrix with the property ∑ m∈M s im =1,(i∈M); The optimization objective of the inter-domain distribution similarity matrix s is defined as: Among them, s mean Represents the weighted average initial matrix of the inter-domain distribution similarity matrix s, whose value is defined as ||ss mean || is a regularization term, which is used to ensure that the generated personalized global prototype has a certain generalization ability, prevent the corresponding coefficient from approaching zero due to large differences in private data domains, and promote collaborative communication training between different clients; The server calculates the aggregate formula for the personalized global prototype of client i as follows: in, is the personalized global prototype of data category c of client i, s i1 …s iM ,(i∈M) represents the domain similarity of private data of client i and all other clients.

4. The method according to claim 3, characterized in that: The S3 includes: After receiving the personalized global prototype sent by the server, each client performs local collaborative training and uses the corresponding personalized global prototype to optimize the training of the local feature extractor, so that the local prototype is closer to the same personalized global prototype and farther away from the different personalized global prototype. In addition, the corresponding personalized global prototype is used to optimize the training of the local classifier so that the classifier learns the spatial class features embedded in the personalized global prototype.

5. The method according to claim 4, characterized in that The local collaborative training is performed for multiple rounds.

6. The method according to claim 4, characterized in that In the local collaborative training phase, the feature extractor f of client i i The loss function for: in, The collection elements in dist represent the feature extractor f i The Euclidean distance between the embedding space features output for category c data x and the personalized global prototype; In the local collaborative training phase, the classifier h of client i i The loss function for: Among them, σ() c represents the probability value of category c in the output value of the softmax function, h i is the classifier of client i, classifier h i The parameter is w i .

7. The method according to claim 6, characterized in that The total loss function of the local training phase in S4 for: Among them, μ is a hyperparameter used to balance the supervised learning loss and the difference loss with the personalized global prototype in the total loss function.

8. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

10. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 7 when being executed by a processor.

Images