Federal learning method and system based on hybrid expert model

Abstract

The invention provides a federated learning method and system based on a hybrid expert model, and relates to the technical field of machine learning, and the method comprises the steps: carrying out the initialization of a local gating network of a current client through employing a global gating network, obtaining an initial local gating network # imgabs0 #, carrying out the feature extraction of a local data set, and obtaining a local data set; k target expert models Sk with the highest adaptation degree with the data features are screened out from the multiple expert models or a required target expert model # imgabs1 # is downloaded from a server side by using a Top-K-based adaptive expert selection mechanism to ensure that all the selected expert models can be used locally; and the client uses local data to perform forward calculation on the target expert model, obtains training feedback parameters of the target expert based on a prediction result and a real label, and optimizes the target expert model through a back propagation algorithm # imgabs2, and the client performs weighted summation on calculation results of the target expert models to obtain a comprehensive prediction result. And the generalization ability and performance of the model are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of machine learning, and in particular, to a federated learning method and device based on a mixture of experts model. Background Art

[0002] In the current field of machine learning, federated learning (FL), as an emerging distributed learning paradigm, aims to enable multiple participants to collaboratively train a model while protecting data privacy. It allows different clients to perform training locally using their respective data, and only upload the local model parameters to the server for aggregation. The server then performs a weighted average of the uploaded local models to generate a new global model, thus avoiding the direct transmission of sensitive data and effectively solving the problems of data silos and privacy protection. At the same time, the mixture of experts (MoE) model, as an architecture that can adaptively select the most suitable expert sub-model to process specific data, has gradually attracted wide attention. By combining multiple expert models and having a gating network select the most suitable expert model or assign weights for each input sample, complex data patterns can be efficiently processed, improving the performance and generalization ability of the model. In the federated learning scenario, the introduction of the mixture of experts model allows the data distributions of different clients to be learned through different expert models, enabling each client to select and optimize relevant expert models according to its specific data characteristics.

[0003] However, there are still some problems in the current method of combining federated learning with the mixture of experts model. On the one hand, in the current federated learning scenario, the number of clients has shown an explosive growth trend, and to meet the requirements of complex tasks, the scale of the expert models is also continuously expanding. If all expert models are directly downloaded from the server to the clients, it will lead to a large amount of unnecessary data transmission. In fact, clients often only need specific target expert models to process local tasks. In this case, distributing a large-scale all expert models from the server to numerous clients without discrimination will not only occupy a huge amount of network bandwidth resources, causing network transmission channels to become congested, but also bring a heavy computational burden to the clients, resulting in a significant extension of the response time, severely restricting the operating efficiency and performance of the system.

[0004] On the other hand, directly aggregating the local models of clients makes it difficult to fully adapt to the differences in data characteristics of different clients. In actual application scenarios, the data of different clients often has the characteristic of non-independent and identically distributed. This is because different clients may be in different geographical locations, usage environments, and application scenarios, and their local data will show diversity and differences. When facing such a complex data distribution situation, a single aggregation method may not be able to capture this diversity. Due to the inability to fully consider the unique characteristics of the data of different clients, this aggregation method will ignore some important information, thus limiting the generalization ability of the model.

[0005] Therefore, how to optimize the burden of server data transmission and calculation and improve the adaptability and generalization ability of the model has become a key technical problem that needs to be solved urgently in the combination of federated learning and the mixture of experts model. Summary of the Invention

[0006] The purpose of the present invention is: to reduce the burden of server data transmission and calculation, improve the system operation efficiency and performance; enhance the adaptability and generalization ability of the model to the data characteristics of different clients. The present invention provides a federated learning method and system based on the mixture of experts model.

[0007] The technical solution of the embodiment of the present application is implemented as follows:

[0008] The first aspect of the embodiment of the present application provides a federated learning method based on the mixture of experts model, including:

[0009] Obtain the global gating network from the server And use the global gating network to initialize the local gating network of the current client to obtain the initial gating network

[0010] Based on the initial gating network, obtain the data characteristics of the local data of the current client, and use the Top-K based adaptive expert selection mechanism to screen out the K local target expert models S with the highest fitness to the data characteristics from multiple expert models k , where, represents the specialization score of the jth expert on the local dataset of the ith client. If the target expert model is not in the current client, download the required non-local target expert model from the server

[0011] Train the target expert model based on the local data, calculate the training feedback parameters of the target expert model based on the prediction results of the target expert model and the true labels of the samples in the local dataset, and optimize the target expert model through the backpropagation algorithm

[0012] Determine the weights of each target expert model based on the professional scores of the experts, perform weighted summation on the calculation results of each target expert model to obtain a comprehensive prediction result; calculate the training feedback parameters of the local gating network based on the comprehensive prediction result and the true labels of the samples in the local dataset, train and optimize the local gating network, and use the optimized gating network Transmit to the server; the optimized gating network is used to aggregate and generate a new global gating network

[0013] Optionally, the method of using the Top-K based adaptive expert selection mechanism to screen out the target expert model with the highest fitness for the data features from the local expert models of multiple clients includes:

[0014] Determine the data features based on the task requirements and local data;

[0015] Use the Top-K based adaptive expert selection mechanism and the data features to screen out the target expert model S with the highest fitness for the data features from multiple expert models k ;

[0016] If the target expert model is not the expert model of the current client, download the target expert model to the current client.

[0017] Optionally, the method of training the target expert model based on the local data to obtain the training feedback parameters of the target expert model includes:

[0018] Train the target expert model based on the local data to obtain the training feedback parameters of the target expert model;

[0019] Use the backpropagation algorithm and the training feedback parameters to adjust and optimize the target expert model to obtain the final target expert model

[0020] Optionally, the method of transmitting the gating network to the server includes:

[0021] Perform weighted summation on the calculation results of each target expert model to obtain a comprehensive prediction result. Calculate the training feedback parameters of the local gating network based on the comprehensive prediction result and the true labels of the samples in the local dataset, and train and optimize the local gating network;

[0022] Transmit the updated gating network of each client to the server

[0023] In the second aspect of the embodiments of the present application, a federated learning system based on a mixture of experts model is provided, including: a server and multiple clients; the clients include a local gating network and an expert model;

[0024] The server is configured to transmit a global gating network to multiple clients, receive multiple updated gating networks transmitted by the clients, and aggregate the updated gating networks to generate a new global gating network; the server coordinates the operations between the clients to ensure the collaborative training and update of the model;

[0025] The client is configured to obtain the global gating network from the server, and use the global gating network to initialize the local gating network to obtain an initial gating network; obtain the data features of the local data based on the initial gating network, and use the Top-K based adaptive expert selection mechanism to screen out the top K target expert models S with the highest fitness to the data features from multiple expert models k and download the required target expert models; train the target expert models based on the local data, calculate the training feedback parameters of the target expert models based on the prediction results of the target expert models and the true labels of the samples in the local data set, and optimize the target expert models through the backpropagation algorithm Perform weighted summation based on the calculation results of each target expert model to obtain a comprehensive prediction result; calculate the training feedback parameters of the local gating network based on the comprehensive prediction result and the true labels of the samples in the local data set, train and optimize the local gating network, and transmit the optimized gating network to the server; the clients are edge devices with large differences in the amount of data they own and their own computing power, training models locally and communicating with the central server;

[0026] Optionally, the client is specifically configured to:

[0027] Determine the data features based on the task requirements and local data;

[0028] Use the Top-K based adaptive expert selection mechanism and the data features to screen out the target expert model S with the highest fitness to the data features from multiple expert models k ;

[0029] If the target expert model is not the expert model of the current client, download the target expert model to the current client.

[0030] Optionally, the client is specifically configured to:

[0031] Train the target expert model based on the local data and determine the training feedback parameters;

[0032] Adjust and optimize the target expert model by using the backpropagation algorithm and the training feedback parameters to obtain the final target expert model

[0033] Optionally, the client is further configured to:

[0034] Determine the weights of each target expert model based on the professional scores of the experts, perform weighted summation on the calculation results of each target expert model to obtain a comprehensive prediction result. Calculate the training feedback function of the local gating network based on the comprehensive prediction result and the true labels of the samples in the local dataset, and train and optimize the local gating network;

[0035] A third aspect of the embodiments of the present application provides an electronic device, including a processor and a memory; the memory stores a computer program, wherein the computer program, when executed by the processor, implements the federated learning method based on the hybrid expert model described in the first aspect.

[0036] A fourth aspect of the embodiments of the present application provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the steps of the method described in the first aspect are implemented.

[0037] Compared with the prior art, the beneficial effects brought by the technical solution provided by the present application are:

[0038] The present invention provides a federated learning method and device based on a hybrid expert model. By obtaining a global gating network from a server And use the global gating network to initialize the local gating network of the current client to obtain an initial local gating network Obtain the data features of the local data of the current client based on the initial gating network, and use the Top-K based adaptive expert selection mechanism to screen out the K target expert models S with the highest fitness with the data features from multiple expert models k , if the target expert model is not on the current client, the client will download the required target expert model from the server to ensure that all selected expert models can be used locally; aggregate to generate a new global gating network Train the target expert model based on the local data, calculate the training feedback parameters of the target expert model based on the prediction results of the target expert model and the true labels of the samples in the local dataset, and optimize the target expert model through the backpropagation algorithm Perform weighted summation on the calculation results of each target expert model to obtain a comprehensive prediction result. Calculate the training feedback parameters of the local gating network based on the comprehensive prediction result and the true labels of the samples in the local dataset, train and optimize the local gating network, and the optimized gating network Transmit to the server and aggregate to generate a new global gating network By maintaining an expert model and a local gating network at each client in federated learning, pull experts from the server according to the output of the gating network. Each expert model can focus on processing a specific type of data distribution. The gating network can dynamically select the most suitable expert model according to the characteristics of the client data, so as to better adapt to different data distributions, thereby improving the generalization ability and performance of the model. Brief Description of the Drawings

[0039] Figure 1 It is a schematic flowchart of a federated learning method based on a hybrid expert model provided by an embodiment of the present application;

[0040] Figure 2 It is an implementation effect diagram of a federated learning method based on a hybrid expert model provided by an embodiment of the present application;

[0041] Figure 3 It is a schematic structural diagram of a federated learning system based on a hybrid expert model provided by an embodiment of the present application;

[0042] Figure 4 It is a schematic structural diagram of an electronic device provided by an embodiment of the present application. Detailed Embodiments

[0043] Hereinafter, embodiments of the present application will be described with reference to the drawings. However, it should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present application. In the following detailed description, for the sake of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present application. However, obviously, one or more embodiments can also be implemented without these specific details. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessarily confusing the concepts of the present application.

[0044] The terms used herein are only for describing specific embodiments and are not intended to limit the present application. The terms "including", "comprising", etc. used herein indicate the presence of the described features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0045] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification, and should not be interpreted in an idealized or overly rigid manner.

[0046] Some block diagrams and / or flowcharts are shown in the accompanying drawings. It should be understood that some blocks or combinations of blocks in the block diagrams and / or flowcharts can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when executed by the processor, these instructions can create a device for implementing the functions / operations illustrated in these block diagrams and / or flowcharts.

[0047] In some embodiments, refer to Figure 1 , Figure 1 is a schematic flowchart of the federated learning method based on the mixture-of-experts model provided by the embodiments of the present application; the federated learning method based on the mixture-of-experts model provided by the embodiments of the present application includes:

[0048] S110, download the latest global gating network from the server and use the global gating network to initialize the local gating network of the current client to obtain an initial local gating network

[0049] In this embodiment, a server is connected to multiple clients, jointly forming an information interaction system. These clients first download the latest global gating network from the server. This global gating network is used to initialize the local gating network of each client.

[0050] S120, obtain the data features of the local data of the current client based on the initial gating network, and use the Top-K based adaptive expert selection mechanism to screen out the top K local target expert models S with the highest fitness to the data features k , wherein, represents the specialization score of the jth expert on the local dataset of the ith client. If the target expert model is not on the current client, download the required non-local target expert model from the server

[0051] In this embodiment, data features are extracted from local data based on task requirements, and then the Top-K based adaptive expert selection mechanism is used to screen out the target expert model S with the highest fitness to the data features from multiple expert models k ;

[0052] In some embodiments, S120, using the Top-K based adaptive expert selection mechanism to screen out the target expert model with the highest fitness to the data features from multiple expert models, includes:

[0053] Determine data features based on task requirements and local data;

[0054] Use the Top-K based adaptive expert selection mechanism and data characteristics to screen out the target expert model S with the highest fitness to the data characteristics from multiple expert models k ;

[0055] If the target expert model is not the expert model of the current client, download the target expert model to the current client.

[0056] The strategy of the Top-K based adaptive expert selection. Specifically, the gating network uses a multi-layer perceptron model to dynamically output the next expert selection decision according to information such as the input image features and the currently selected expert combination. Compared with the traditional Top-K selection, this expert selection mechanism can effectively improve the classification accuracy of the global model by continuous learning and optimization.

[0057] In this embodiment, at the input end of the MLP neural network, the information related to expert selection needs to be vectorized as input features. These input features include: the image feature vector F j , the currently selected expert information ε t and the classification accuracy Acc of the global model t .

[0058] The image feature vector F j represents the image feature vector of the current device, which is a d-dimensional vector obtained by the front-end image feature extraction network, where d is the dimension of the image feature vector.

[0059] F j =[f1,f2,…,f d

[0060] The currently selected expert information ε t represents the currently selected expert combination. It is represented by a binary vector with a vector length of the number of clients n, where each position e i represents whether the expert E i is selected (1 means selected, 0 means not selected), where e i ∈{0,1}:

[0061] ε t =[e1,e2,…,e n

[0062] The classification accuracy Acc of the global model t represents the classification accuracy of the current federated learning model.

[0063] The output representation of the gating network is a probability distribution of expert selection, indicating the probability of each unselected expert being selected in the current state. Each value p i ​​Indicates the probability of selecting expert E i : Output = [p1, p2, …, p n ;

[0064] To optimize the expert selection strategy of the MLP network, the policy gradient method is adopted, and the improvement of classification accuracy is taken as the main optimization goal. Through multiple rounds of federated learning and feedback training, the MLP can learn to select the expert combination that maximizes the classification accuracy under different states.

[0065] The expert selection based on policy gradient consists of the state space of the gating network, the action space of the gating network, and the reward.

[0066] The state space of the gating network, each state of the gating network is composed of the current image feature F j , the set of selected experts ε t , and the current global model accuracy Acc t : S t = (F j , ε t , Acc t );

[0067] The action space A t of the gating network includes selecting a certain expert ε t to join the current expert set, or terminating the selection process:

[0068]

[0069] The reward function, after each expert selection, uses the improvement of the classification accuracy ΔAcc of the global model as the reward. This reward function is adjusted in combination with resource consumption to balance the improvement of accuracy and resource use:

[0070] R t = ΔAcc - α · resourse

[0071] where α represents the parameter for adjusting the resource consumption weight, and resourse represents the computational resources and communication overhead costs consumed when selecting experts.

[0072] The goal of policy gradient is to maximize the expected cumulative reward. Among them, πθ is the adaptive expert selection strategy defined by the gating network, and the optimized objective function is:

[0073] J(θ) = E πθ [R t

[0074] The specific update rule of the policy parameters is as follows:

[0075] ​

[0076] The gating network gradually learns the optimal expert selection strategy through the policy gradient method. After each round of training, the model adjusts the strategy according to the feedback of the classification accuracy change and resource consumption, making the expert selection process tend to maximize the accuracy improvement while reasonably controlling the resource usage.

[0077] The output of the gating network is a vector of length N, representing the selection probability of each expert, where is the specialization score of the j-th expert. Based on these scores, the gating network will select the top k expert models that are most suitable for the current data.

[0078]

[0079] Calculate the scores of all experts, sort the scores, and select the Top k experts with the highest scores. S k is the set of indices of the selected Top k experts:

[0080]

[0081] In this embodiment, based on the result of the adaptive expert selection of the gating network, the target expert model S k is obtained. If the target expert model is not on the current client, the client will download the required target expert model from the server to ensure that all selected expert models can be used locally.

[0082] The expert model is only transmitted between clients without passing through the server. This strategy can optimize the transmission efficiency, ensure that the expert models of each client remain efficient and accurate, reduce the burden on the central server, and enable the expert models of each client to directly obtain the latest knowledge, ensuring that the expert models can quickly adapt to the needs of each client. This decentralized transmission method not only improves the efficiency of the system but also guarantees the accuracy and applicability of the model. After training on client i, the updated expert model parameters can be directly transmitted to other clients in need without passing through the central server. Transmitting the expert network parameters between clients reduces the dependence on the central server and significantly reduces the bandwidth pressure on the central server. Each time there is an update, only the required expert network parameters are transmitted, rather than the entire dataset or a large-scale model. This approach can effectively reduce the amount of data transmitted, thereby improving the overall transmission efficiency. At the same time, direct transmission between clients reduces the time delay of data traveling back and forth to the central server. Direct communication between clients can complete parameter updates and synchronization faster, thus accelerating the speed of model training and update. The central server does not need to process and store a large amount of expert network data and only needs to be responsible for coordination and monitoring. This can reduce the computing and storage burden on the server, enabling it to concentrate resources on handling other key tasks, such as aggregating and coordinating the global model.

[0083] Since each client can adjust and optimize the parameters of the expert model according to its specific dataset, the expert models of each client can better adapt to its unique environment and needs. This personalized adaptation improves the accuracy and generalization ability of the model, making the expert models of each client perform better on local tasks. Clients can update and adapt to changing data patterns more quickly without waiting for synchronization from the central server. This flexibility enables the system to respond more quickly to changes in data distribution or the environment, enhancing the overall dynamic adaptation ability of the system and making the system more efficient and effective in a distributed environment.

[0084] S130. Train the target expert model based on local data, calculate the training feedback parameters of the target expert model based on the prediction results of the target expert model and the true labels of the samples in the local dataset, and optimize the target expert model through the backpropagation algorithm.

[0085] In some embodiments, S130. Training the target expert model based on local data to obtain training feedback parameters includes:

[0086] Train the target expert model based on the local data to obtain the training feedback parameters of the target expert model;

[0087] Adjust and optimize the target expert model by using the backpropagation algorithm and the training feedback parameters to obtain the final target expert model

[0088] In this embodiment, each customer holds a target expert model, which is trained according to the specific needs and data of the customer. Each client uses its local dataset D i Train the expert model. For the given local dataset D i , where x k is the input data and y k is the true label

[0089] In the forward prediction stage, the forward calculation output of the target expert model is the prediction result The calculation formula is as follows

[0090]

[0091] where is the prediction output of the target expert model for the input data x k

[0092] Based on the prediction result and the true label y k , define the loss function to measure the prediction error of the model. The loss function is expressed as

[0093]

[0094] Calculate the gradient of the loss function with respect to the model parameters through the backpropagation algorithm, and update the target expert model using the gradient descent method. The formula is as follows

[0095]

[0096] where η represents the learning rate and t represents the current iteration round

[0097] S140. Determine the weights of each target expert model based on the specialization scores of the experts, perform weighted summation on the calculation results of each target expert model to obtain the comprehensive prediction result. Calculate the training feedback parameters of the local gating network based on the comprehensive prediction result and the true labels of the samples in the local dataset, train and optimize the local gating network, and transmit the optimized gating network to the server; the optimized gating network is used to aggregate and generate a new global gating network Distribute the new global gating network to all clients as the initial local gating network parameters for the next round of training ​

[0098] The goal is to optimize the parameters of the local gating network to adapt to local data. The ultimate aim of this approach is to enhance the performance of the local model by effectively utilizing global information while reducing the data transmission volume. It can minimize the amount of data transmitted between the client and the server because only the parameters of the gating network are transmitted, rather than the complete dataset.

[0099] In some embodiments, S140, train and optimize the local gating network according to the local dataset, and transmit the latest local gating network to the server;

[0100] Determine the weights of each target expert model based on the specialization scores of the experts, perform weighted summation on the calculation results of each target expert model to obtain a comprehensive prediction result. Calculate the training feedback parameters of the local gating network based on the comprehensive prediction result and the true labels of the samples in the local dataset, and train and optimize the local gating network.

[0101] Transmit the updated gating network of each client to the server

[0102] In this embodiment, for specific classes or data, only the gating network is transmitted between the client and the server, which means that the gating network is the only part exchanged between the client and the server. Each client downloads the latest global gating network parameters from the cloud server. The client uses the downloaded global gating network to initialize its local gating network, and the initial parameters of the local gating network are provided by the global gating network. Each client performs local training on its local dataset D i using the weighted output of the target expert model to calculate the training feedback parameters of the local gating network based on the comprehensive prediction result and the true labels of the samples in the local dataset. Specifically, for each input data x k , the gating network selects an appropriate expert model for forward calculation, and calculates the loss function based on the actual output and the true label y k to obtain the training feedback parameters for backpropagation to update the local gating network.

[0103] In the forward calculation, the predicted output of the model is:

[0104]

[0105] where represents the selection weight of the i-th expert model calculated by the gating network based on the input x k , Denotes the output of the $i$-th expert model. The final prediction result is obtained by weighted summation of the outputs of all selected experts.

[0106] The client uses the loss function to calculate the training feedback parameters through the backpropagation method, and obtains the updated gating network.

[0107]

[0108] After each round of training, client $i$ uploads the update of its local gating network to the cloud server. The cloud server receives the updates from all clients for global model aggregation, and uses the size of the data volume of each client $|D|$ i as the weight to calculate the global gating network by weighted average:

[0109]

[0110] A new global gating network is generated after the updated gating networks of each client are aggregated on the server to ensure the generalization ability of the global gating network. In the global aggregation stage, the client uploads the updated gating network to the server, and the server receives and aggregates the gating networks uploaded by all clients to form a new global gating network, and then distributes it to each client for the next round of training. Through this cyclic process, the client and the server cooperate to complete the system function, realize efficient distributed training and aggregation, improve the performance and generalization ability of the expert model, and at the same time ensure data privacy and security.

[0111] In an example, the client downloads the global gating network from the server cloud and initializes the local model. The client extracts features from the local dataset, and uses the Top-K based adaptive expert selection mechanism to screen out the top $K$ target expert models $S$ with the highest fitness to the data features from multiple expert models. k, download the target expert model from the server. During the training process, the system continuously checks whether the model converges. After convergence, the updated local gating network model is uploaded to the cloud. The cloud aggregates the models of all clients and generates a new global gating network to complete one iteration. This process ensures data privacy and improves model performance and flexibility. This training method always keeps the data local, and only the model parameters or gradients are uploaded, avoiding the leakage of raw data and enhancing the privacy of user data. Since it is the model parameters rather than large-scale raw data that are transmitted, the bandwidth usage requirement is reduced, improving system efficiency. Through the selection mechanism of the mixture-of-experts model, different clients select the most suitable expert for training according to their local data, enhancing the pertinence and overall performance of the model. The system can dynamically adjust the expert model according to the needs of different clients to achieve personalized training, with stronger flexibility and adaptability. Even if some clients cannot work, the training results of other clients can still be aggregated, which helps the system maintain normal operation and improve real-time response ability.

[0112] In another example, refer to Figure 2 , Figure 2 is the implementation effect diagram of a federated learning method based on a mixture-of-experts model provided by an embodiment of the present application. Compared with the traditional federated learning method, this solution can update model parameters more efficiently and reduce the communication burden. When 10 transmission rounds are performed, the communication overhead of this solution is reduced by approximately 104.27 MB compared with the traditional federated learning method, and such a reduction rate reaches 30.5%. This data fully shows that in the case of fewer rounds, this solution can already demonstrate good communication cost control ability. And as the number of rounds increases, this advantage becomes more obvious. At 30 rounds, the communication overhead is reduced by approximately 312.81 MB; when the number of rounds further increases to 50 rounds, the reduction in communication overhead reaches approximately 512.34 MB. This shows that the solution of the present invention can effectively reduce the communication cost and is suitable for efficient model training in scenarios with limited bandwidth.

[0113] In the embodiment of the present invention, by maintaining an expert model and a local gating network at each client in federated learning, it is judged whether to pull an expert from the server according to the output of the gating network. Each expert model can focus on processing a specific type of data distribution. The gating network can dynamically select the most suitable expert model according to the characteristics of the client data, so as to better adapt to different data distributions, and further improve the generalization ability and performance of the model.

[0114] In some embodiments, refer to Figure 3 , Figure 3Schematic diagram of a federated learning system based on a mixture of experts model provided by an embodiment of the present application; An embodiment of the present application provides a federated learning system based on a mixture of experts model, including: a server 210 and a plurality of clients 220; The client includes a local gating network 221 and an expert model 222;

[0115] The server 210 is configured to transmit a global gating network to a plurality of clients, receive the updated local gating networks transmitted by the plurality of clients, and aggregate the updated gating networks to generate a new global gating network;

[0116] The client 220 is configured to obtain a global gating network from the server, and use the global gating network to initialize the local gating network to obtain an initial gating network; Based on the initial gating network, obtain the data features of local data, and use a Top-K based adaptive expert selection mechanism to select the top K target expert models S with the highest fitness to the data features from a plurality of expert models k and download the required target expert models; Train the target expert models based on the local data, calculate the training feedback parameters of the target expert models based on the prediction results of the target expert models and the true labels of the samples in the local data set, and optimize the target expert models through the backpropagation algorithm Perform weighted summation based on the calculation results of each target expert model to obtain a comprehensive prediction result. Calculate the training feedback parameters of the local gating network based on the comprehensive prediction result and the true labels of the samples in the local data set, train and optimize the local gating network, and transmit the optimized gating network to the server;

[0117] In some embodiments, the client 220 is specifically configured to:

[0118] Determine the data features based on the task requirements and local data;

[0119] Use a Top-K based adaptive expert selection mechanism and the data features to select the target expert model S with the highest fitness to the data features from a plurality of expert models k ;

[0120] If the target expert model is not the expert model of the current client, download the target expert model to the current client.

[0121] In some embodiments, the client 220 is specifically configured to:

[0122] Train the target expert models based on the local data, and determine the training feedback parameters of the target expert models;

[0123] The target expert model is adjusted and optimized by using the backpropagation algorithm and the training feedback parameters of the target expert model to obtain the final target expert model.

[0124] In some embodiments, the client 220 is further specifically configured to:

[0125] The calculation results of each target expert model are used for weighted summation to obtain a comprehensive prediction result. Based on the comprehensive prediction result and the true labels of the samples in the local dataset, the training feedback parameters of the local gating network are calculated, and the local gating network is trained and optimized.

[0126] The federated learning system based on the hybrid expert model provided in the embodiments of the present application can implement each process in the corresponding embodiments of the above-mentioned federated learning method based on the hybrid expert model. To avoid repetition, it will not be elaborated here.

[0127] It should be noted that the federated learning system based on the hybrid expert model provided in the embodiments of the present application and the federated learning method based on the hybrid expert model provided in the embodiments of the present application are based on the same application concept. Therefore, the specific implementation of this embodiment can refer to the implementation of the above-mentioned federated learning method based on the hybrid expert model, and the repeated parts will not be elaborated.

[0128] In some embodiments, please refer to Figure 4 , Figure 4 is a schematic structural diagram of an electronic device provided in an embodiment of the present application. An electronic device 300 provided in an embodiment of the present application includes a processor 310 and a memory 320; the memory 320 stores a computer program, wherein the computer program, when executed by the processor, implements the above-mentioned federated learning method based on the hybrid expert model.

[0129] Specifically, the processor 310 may include, for example, a general microprocessor, an instruction set processor, and / or a related chipset and / or a dedicated microprocessor (such as an application specific integrated circuit (ASIC)), and so on. The processor 310 may also include on-board memory for caching purposes. The processor 310 may be a single processing unit or multiple processing units for performing different actions of the method flow according to the embodiments of the present application.

[0130] The memory 320 may be, for example, any medium capable of containing, storing, transmitting, propagating, or transporting instructions. For example, the memory 320 may include, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, device, or propagation medium. Specific examples of the memory 320 include: a magnetic storage device, such as a magnetic tape or a hard disk drive (HDD); an optical storage device, such as a compact disc (CD-ROM); it may also be a random access memory (RAM) or a flash memory; and / or a wired / wireless communication link.

[0131] The present application also provides a computer-readable medium, on which a computer program is stored, and when the program is executed by a processor, it implements the above-mentioned federated learning method based on a mixture of experts model. The computer-readable medium may be included in the device / device / system described in the above embodiments; or it may exist alone and not be assembled into the device / device / system. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiments of the present application is implemented.

[0132] According to an embodiment of the present application, the computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination of the above. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer-readable storage medium may be any tangible medium that contains or stores a program, and the program can be used by or in combination with an instruction execution system, device, or device. And in the present application, the computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries the computer-readable program code. Such a propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium may also be any computer-readable medium other than the computer-readable storage medium, and the computer-readable medium can send, propagate, or transmit a program for use by or in combination with an instruction execution system, device, or device. The program code included on the computer-readable medium can be transmitted by any appropriate medium, including but not limited to: wireless, wired, optical fiber cable, radio frequency signal, etc., or any suitable combination of the above.

[0133] Those skilled in the art will understand that the features recited in the various embodiments and / or claims of the present application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly recited in the present application. In particular, without departing from the spirit and teachings of the present application, the features recited in the various embodiments and / or claims of the present application can be combined and / or combined in various ways. All such combinations and / or combinations fall within the scope of the present application. Therefore, the scope of the present application should not be limited to the above embodiments, but should be determined not only by the appended claims, but also by the equivalents of the appended claims.

Claims

1. A federated learning method based on a mixture of experts model, applied to a client, characterized in that, Including: Obtain the global gating network from the server And initialize the local gating network of the current client using the global gating network to obtain an initial gating network Obtain the data features of the local data of the current client based on the initial gating network, and use the adaptive expert selection mechanism based on Top-K to screen out the top K local target expert models S with the highest fitness to the data features from multiple expert models k , wherein, represents the specialization score of the j-th expert on the local dataset of the i-th client. If the target expert model is not on the current client, download the required non-local target expert model from the server Train the target expert model based on the local data, calculate the training feedback parameters of the target expert model based on the prediction results of the target expert model and the true labels of the samples in the local dataset, and optimize the target expert model through the backpropagation algorithm Determine the weights of each target expert model based on the professional scores of experts, and perform weighted summation on the calculation results of each target expert model to obtain a comprehensive prediction result; Calculate the training feedback parameters of the local gating network based on the comprehensive prediction results and the true labels of the samples in the local dataset, train and optimize the local gating network, and transfer the optimized gating network to the server; the optimized gating network is used to aggregate and generate a new global gating network 2. The federated learning method based on the mixture of experts model according to claim 1, wherein The Top-K based adaptive expert selection mechanism filters out the target expert model with the highest fitness to the data characteristics from multiple expert models, including: Determine the data characteristics based on the task requirements and local data; Select the target expert model S with the highest fitness for the data characteristics from multiple expert models by using the Top-K based adaptive expert selection mechanism and the data characteristics k ; If the target expert model is not the expert model of the current client, download the target expert model to the current client.

3. The federated learning method based on the mixture of experts model according to claim 1, characterized in that Training the target expert model based on the local data to obtain training feedback parameters, including: Training the target expert model based on the local data to obtain the training feedback parameters of the target expert model; Adjust and optimize the target expert model using the backpropagation algorithm and the training feedback parameters to obtain the final target expert model 4. The federated learning method based on the mixture of experts model according to claim 1, characterized in that, Transmitting the updated gating network to the server, including: Calculating training feedback parameters based on the labels obtained from the weighted output of the target expert model, and training and optimizing the local gating network; Transmit the updated gating network of each of the clients to the server 5. A federated learning system based on a hybrid expert model, including a server and multiple clients; the clients include local gating networks and expert models; The server is configured to transmit a global gating network to multiple clients, receive multiple updated gating networks transmitted by the clients, and aggregate the updated gating networks to generate a new global gating network; the server coordinates the operations between the clients to ensure the collaborative training and update of the model; The client is configured to obtain the global gating network from the server and use the global gating network to initialize the local gating network to obtain an initial gating network; Obtain the data features of the local data based on the initial gating network, and use the Top-K based adaptive expert selection mechanism to screen out the top K target expert models S with the highest fitness to the data features from multiple expert models k and download the required target expert models; train the target expert models based on the local data, calculate the training feedback parameters of the target expert models based on the prediction results of the target expert models and the true labels of the samples in the local dataset, and optimize the target expert models through the backpropagation algorithm Perform weighted summation based on the calculation results of each target expert model to obtain a comprehensive prediction result; calculate the training feedback parameters of the local gating network based on the comprehensive prediction result and the true labels of the samples in the local dataset, train and optimize the local gating network, and transmit the optimized gating network to the server.

6. The federated learning system based on the mixture of experts model according to claim 5, wherein The client is specifically configured as: Determine the data characteristics based on the task requirements and local data; Using the Top-K based adaptive expert selection mechanism and the data features, screen out the target expert model S with the highest fitness for the data features from multiple expert models k ; If the target expert model is not the expert model of the current client, download the target expert model to the current client.

7. The federated learning system based on the mixture of experts model according to claim 5, wherein The client is specifically configured as: Train the target expert model based on the local data to determine the training feedback parameters of the target expert model; Adjust and optimize the target expert model by using the backpropagation algorithm and the training feedback parameters to obtain the final target expert model 8. The federated learning system based on the mixture of experts model according to claim 5, wherein The client is specifically further configured as: Calculate training feedback parameters using the labels obtained from the weighted output of the target expert model, and train and optimize the local gating network.

9. An electronic device, comprising a processor and a memory; the memory stores a computer program, wherein, The computer program, when executed by the processor, implements the federated learning method based on a hybrid expert model according to any one of claims 1 to 4.

10. A computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 4.

Images