Adaptive constraint federated learning method, device and system based on reinforcement learning

Abstract

The invention provides an adaptive constraint federated learning method, device and system based on reinforcement learning, and the method comprises the steps: determining a hyper-parameter of an action of a current round based on a local reinforcement learning agent unit according to a global model weight and a local model weight of a previous round; performing adaptive constraint on the local model based on hyper-parameters in the training process, sending the local model weight of the round to a server in the federated learning system for aggregation, and determining the reward of the current round according to the global model weight and the local model weight corresponding to the previous round and the current round based on a reinforcement learning agent unit; and forming a tuple sample of the current round and storing the tuple sample. According to the method, the influence of data heterogeneity can be reduced, and the reliability and effectiveness of model training and the performance of the trained model can be improved; independent decision making is carried out through the local reinforcement learning agent, and extra data communication between the server and the client is not needed.

Description

Technical Field

[0001] The present application relates to the field of federated learning technology, and in particular to an adaptive constrained federated learning method, device and system based on reinforcement learning. Background Art

[0002] Deep learning is increasingly being used to solve complex problems, and models are typically trained on public or private datasets. The quality of these datasets has a significant impact on model performance; public datasets typically have a balanced distribution, while private datasets may excel in certain areas but lack comprehensive coverage. However, due to privacy concerns, individuals prefer to keep their own data, which can lead to an imbalanced data distribution and poor model performance across organizations. Federated Learning (FL) addresses this problem by integrating information from multiple sources without sharing the data. Federated learning involves clients training models on local datasets, which are then aggregated on servers to enhance model performance over multiple rounds of iterations.

[0003] The emergence of Compute-First Networking (CFN) has brought new vitality to federated learning. By integrating discrete computing resources, large-scale CFN systems provide an ideal environment for federated learning tasks, making them more convenient. However, both pure federated learning and federated learning combined with CFN face the problem of data heterogeneity. The different data distributions of each client make it difficult for the global model to converge and generalize effectively. In addition, when the data classification of each client is highly unbalanced, the global model often forgets the useful information of other clients when it aggregates after local training.

[0004] At present, most studies on data heterogeneity in federated learning analyze models in layers and use regularization constraints to control the differences between local training models and global models to prevent global information loss caused by local training. However, this approach imposes global constraints and regulations on the model, applies uniform rules to all models participating in federated training, and ignores the special circumstances of individual clients. Other approaches that focus on fixing or constraining the model can prevent the client model from being adversely affected by heterogeneous data on model performance, but the classifier layer is easily affected by the data and is prone to change, ignoring the useful information about data distribution that may be contained in the model.

[0005] Therefore, there is an urgent need to design a federated learning method that can solve the problem of model performance degradation caused by data heterogeneity between clients under the federated learning framework. Summary of the invention

[0006] In view of this, the embodiments of the present application provide an adaptive constrained federated learning method, device and system based on reinforcement learning to eliminate or improve one or more defects existing in the prior art.

[0007] One aspect of the present application provides an adaptive constraint federated learning method based on reinforcement learning, comprising:

[0008] Extract the global model weight and the local model weight of the previous round from the tuple samples of the previous round pre-stored in the local experience pool, and determine the hyperparameters of the action of the current round based on the global model weight and the local model weight of the previous round based on the local reinforcement learning agent unit;

[0009] Using each data sample in the local training set to train the local model corresponding to the global model weight of the previous round, and adaptively constraining the local model based on the hyperparameters during the training process to obtain the local model weight of the local model in the current round, and sending the local model weight to the server in the federated learning system, so that the server aggregates the local model weights received in the current round to obtain the global model weight of the current round;

[0010] Receive the global model weight of the current round sent by the server, and determine the reward of the current round based on the local reinforcement learning agent unit according to the global model weight and the local model weight corresponding to the previous round and the current round respectively;

[0011] According to the global model weight and the local model weight of the previous round, the action of the current round, the reward of the current round, and the global model weight and the local model weight of the current round, a tuple sample of the current round is constructed and stored in the local experience pool.

[0012] In some embodiments of the present application, before extracting the global model weight and the local model weight of the previous round from the tuple samples of the previous round pre-stored in the local experience pool, the method further includes:

[0013] Build a local experience pool;

[0014] Initialize the action model and evaluation model corresponding to the reinforcement learning agent unit;

[0015] Divide each data sample in the local data into a training set and a validation set respectively;

[0016] and, receiving initial global model parameters sent by a server in the federated learning system;

[0017] In the initial round, the local model corresponding to the initial global model parameter is trained based on the local data and the preset initial hyperparameters to obtain the local model weight of the initial round, and the local model weight is sent to the server in the federated learning system, so that the server aggregates the local model weights received in the initial round to obtain the global model weight of the initial round;

[0018] The global model weight of the initial round is received from the server.

[0019] In some embodiments of the present application, the extracting of the global model weight and the local model weight of the previous round from the tuple sample of the previous round pre-stored in the local experience pool, and determining the hyperparameters of the action of the current round based on the global model weight and the local model weight of the previous round based on the local reinforcement learning agent unit, includes:

[0020] Searching whether the local experience pool contains a tuple sample of the previous round, and if so, extracting the global model weight and local model weight of the previous round as the state of the previous round from the tuple sample of the previous round;

[0021] Based on the action model in the reinforcement learning agent unit, hyper parameters of the action of the current round are generated according to the global model weight and the local model weight of the previous round.

[0022] In some embodiments of the present application, the locally based reinforcement learning agent unit determines the reward of the current round according to the global model weight and the local model weight corresponding to the previous round and the current round, respectively, including:

[0023] Obtain the accuracy of the global model weight and the local model weight corresponding to the previous round and the current round on the verification set;

[0024] The reward for the current round is determined according to the preset weight change parameters, and the accuracy of the global model weights and local model weights corresponding to the previous round and the current round on the verification set; wherein the weight change parameters are used to determine according to the round value corresponding to the current round, and different round values ​​correspond to different weight change parameters.

[0025] In some embodiments of the present application, it also includes:

[0026] If the number of the tuple samples in the local experience pool has reached a preset quantity threshold, multiple tuple samples other than the tuple samples in the previous round in the local experience pool are extracted, and the reinforcement learning agent unit is trained based on the extracted tuple samples.

[0027] In some embodiments of the present application, it also includes:

[0028] Receiving an index of similar client devices sent by a server in a federated learning system, wherein the server searches for corresponding similar client devices for each client device that sends the local model weights according to the similarity between the local model weights received in the current round, and sends the unique identifier of the similar client device corresponding to each client device to each client device;

[0029] Based on the index of the similar client device, copying a plurality of the tuple samples from an experience pool of the similar client device;

[0030] After the local reinforcement learning agent unit is trained based on each of the tuple samples extracted from the local experience pool, the local reinforcement learning agent unit is trained based on each of the tuple samples copied from the experience pool of the similar client device.

[0031] In some embodiments of the present application, the data sample includes: an image data sample;

[0032] The local model weights include: weights of local classifier heads;

[0033] The global model weight includes: the weight of the global classifier head.

[0034] Another aspect of the present application provides an adaptive constraint federated learning device based on reinforcement learning, comprising:

[0035] A reinforcement learning action determination module is used to extract the global model weight and the local model weight of the previous round from the tuple samples of the previous round pre-stored in the local experience pool, and determine the hyperparameters of the action of the current round based on the global model weight and the local model weight of the previous round based on the local reinforcement learning agent unit;

[0036] An adaptive constrained federated learning module, used to train the local model corresponding to the global model weight of the previous round using each data sample in the local training set, and adaptively constrain the local model based on the hyperparameters during the training process to obtain the local model weight of the local model in the current round, and send the local model weight to the server in the federated learning system, so that the server aggregates the local model weights received in the current round to obtain the global model weight of the current round;

[0037] A reinforcement learning reward determination module, configured to receive the global model weight of the current round sent by the server, and determine the reward of the current round based on the local reinforcement learning agent unit according to the global model weight and the local model weight corresponding to the previous round and the current round respectively;

[0038] The experience pool storage module is used to construct a tuple sample of the current round and store it in the local experience pool based on the global model weight and local model weight of the previous round, the action of the current round, the reward of the current round, and the global model weight and local model weight of the current round.

[0039] A third aspect of the present application provides a federated learning system, comprising: a server and various client devices in communication connection with the server;

[0040] Each of the client devices is used to execute the adaptive constrained federated learning method based on reinforcement learning.

[0041] The fourth aspect of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the adaptive constrained federated learning method based on reinforcement learning when executing the computer program.

[0042] The fifth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the adaptive constrained federated learning method based on reinforcement learning.

[0043] The sixth aspect of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the adaptive constrained federated learning method based on reinforcement learning.

[0044] The adaptive constrained federated learning method based on reinforcement learning provided in the present application extracts the global model weights and local model weights of the previous round from the tuple samples of the previous round pre-stored in the local experience pool, and determines the hyperparameters of the action of the current round based on the global model weights and local model weights of the previous round based on the local reinforcement learning agent unit; uses each data sample in the local training set to train the local model corresponding to the global model weight of the previous round, and adaptively constrains the local model based on the hyperparameters during the training process to obtain the local model weight of the local model in the current round, and sends the local model weight to the server in the federated learning system, so that the server aggregates the local model weights received in the current round to obtain the global model weight of the current round; receives the global model weight of the current round sent by the server, and based on the local reinforcement learning agent unit, aggregates the global model weights and local model weights corresponding to the previous round and the current round respectively. The local model weight determines the reward of the current round; according to the global model weight and local model weight of the previous round, the action of the current round, the reward of the current round, and the global model weight and local model weight of the current round, the tuple sample of the current round is formed and stored in the local experience pool, which can solve the problem of the degradation of model performance due to the inconsistency between the data distribution of each client and the data distribution of the test domain caused by the heterogeneity of data between clients under the federated learning framework, and can adaptively adjust the correlation between the local model of each client and the global model to reduce the impact of data heterogeneity, which can effectively improve the reliability and effectiveness of model training, and thus improve the model performance; at the same time, each client has a local reinforcement learning agent to make independent decisions, so that each client can fully and autonomously adapt to the training conditions and make decisions, without the need for additional data communication between the server and the client, thereby effectively reducing the resource consumption and time cost of data communication between the server and the client.

[0045] Additional advantages, purposes, and features of the present application will be partially described in the following description, and will become partially apparent to those skilled in the art after studying the following, or may be learned from the practice of the present application. The purposes and other advantages of the present application can be achieved and obtained by the structures specifically pointed out in the specification and the drawings.

[0046] Those skilled in the art will understand that the purposes and advantages that can be achieved by the present application are not limited to the above specific description, and the above and other purposes that can be achieved by the present application will be more clearly understood based on the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The drawings described herein are used to provide a further understanding of the present application, constitute a part of the present application, and do not constitute a limitation of the present application. The components in the drawings are not drawn to scale, but are only for the purpose of illustrating the principles of the present application. In order to facilitate the illustration and description of some parts of the present application, the corresponding parts in the drawings may be enlarged, that is, they may become larger relative to other components in the exemplary device actually manufactured according to the present application. In the drawings:

[0048] Figure 1 This is a first flow chart of an adaptive constrained federated learning method based on reinforcement learning in one embodiment of the present application.

[0049] Figure 2 This is a second flow chart of the adaptive constrained federated learning method based on reinforcement learning in one embodiment of the present application.

[0050] Figure 3 This is a schematic diagram of the execution architecture corresponding to the adaptive constrained federated learning method based on reinforcement learning in the video of an application example of this application.

[0051] Figure 4 Schematic diagram of the structure of an adaptive constrained federated learning device based on reinforcement learning in one embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the implementation modes and the accompanying drawings. Here, the illustrative implementation modes and descriptions of the present application are used to explain the present application, but are not intended to limit the present application.

[0053] It should also be noted here that in order to avoid obscuring the present application due to unnecessary details, only the structures and / or processing steps closely related to the scheme according to the present application are shown in the accompanying drawings, while other details that are not very relevant to the present application are omitted.

[0054] It should be emphasized that the term “include / comprises” when used herein refers to the presence of features, elements, steps or components, but does not exclude the presence or addition of one or more other features, elements, steps or components.

[0055] It should also be noted that, unless otherwise specified, the term “connection” herein may refer not only to a direct connection but also to an indirect connection involving an intermediate.

[0056] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0057] In order to solve the problem of model performance degradation due to data heterogeneity between clients under the federated learning framework, the embodiments of the present application respectively provide an adaptive constrained federated learning method based on reinforcement learning, an adaptive constrained federated learning device based on reinforcement learning for executing the adaptive constrained federated learning method based on reinforcement learning, a federated learning system, an electronic device, a computer-readable storage medium, and a computer program product, which combine reinforcement learning (RL) with federated learning (FL) to model the hyperparameter selection problem as a dynamic decision-making process. RL is used to make the best decision in various situations. It adaptively adjusts the correlation between the local model of each client and the global model in each round of local training to reduce the impact of data heterogeneity. Each client makes independent decisions without the need for additional data communication between the server and the client.

[0058] The details are described in detail through the following examples.

[0059] Based on this, the embodiment of the present application provides an adaptive constraint federated learning method based on reinforcement learning that can be implemented by an adaptive constraint federated learning device based on reinforcement learning, see Figure 1 The adaptive constraint federated learning method based on reinforcement learning specifically includes the following contents:

[0060] Step 100: Extract the global model weights and local model weights of the previous round from the tuple samples of the previous round pre-stored in the local experience pool, and based on the local reinforcement learning agent unit, determine the hyperparameters of the action of the current round according to the global model weights and local model weights of the previous round.

[0061] In one or more embodiments of the present application, the adaptive constrained federated learning device based on reinforcement learning can be a functional module in a client device, and the client device can be referred to as a client for short. In the adaptive constrained federated learning device based on reinforcement learning of each client device, a reinforcement learning agent unit, experience pool, local data and local model uniquely corresponding to the client are provided.

[0062] It is understandable that reinforcement learning is a machine learning method that aims to learn the optimal strategy through the interaction between the agent and the environment to maximize the cumulative reward. In reinforcement learning, the reinforcement learning agent unit refers to the agent in reinforcement learning. The agent is an entity that can perform actions, which can be called an agent. The agent selects an action based on the current state, enters the next state after executing the action, and obtains the corresponding reward. The goal of the agent is to find a strategy through continuous trial and learning to maximize the total reward obtained in the long term. The combination of federated learning and reinforcement learning has led to a research direction that has emerged in recent years. Learning strategies through reinforcement learning can assist federated learning training.

[0063] In one or more embodiments of the present application, each client device and server used to execute the adaptive constrained federated learning method based on reinforcement learning together constitute a federated learning system, and the rounds mentioned in the present application refer to each iteration round in the number of iterations of federated learning in the federated learning system. It can be understood that for the current round, if there is a previous round, it means that the current round is not the initial iteration round, that is, the steps 100 to the following steps 400 provided in the embodiments of the present application are general adaptive constrained federated learning processes for non-initial rounds.

[0064] Federated learning is a distributed machine learning method that allows multiple parties (such as devices or institutions) to jointly train a global model without sharing their local data. In federated learning, each party trains the model on local data and only sends the model parameters or gradients to a central server for aggregation, rather than uploading the data itself. This approach not only protects data privacy, but also reduces the cost and risk of data transmission.

[0065] Step 200: Use each data sample in the local training set to train the local model corresponding to the global model weight of the previous round, and adaptively constrain the local model based on the hyperparameters during the training process to obtain the local model weight of the local model in the current round, and send the local model weight to the server in the federated learning system, so that the server aggregates the local model weights received in the current round to obtain the global model weight of the current round.

[0066] In step 200, performing adaptive constraints on the local model refers to performing adaptive constraints on the local model with respect to the global model (i.e., the global model weights of the previous round). The purpose of the adaptive constraints is to enable the client to dynamically adjust the local training update direction according to its own situation in a dynamic scenario, and to adaptively adjust the correlation between the local model of each client and the global model to reduce the impact of data heterogeneity.

[0067] In one or more embodiments of the present application, the model may be a classification model for image or speech classification or a prediction model for image, speech, text, etc., all models applicable to federated learning. In one example, the local data may be local image data, and each data sample in the local data is divided into a training set and a validation set, respectively, and the data samples include image data samples; the model includes: a classification model (also known as an image classification model), which is used to output a type recognition result of the image data sample according to the input image data sample. When the client device trains the local model, the data samples used are provided with type labels corresponding to each of the image data samples.

[0068] Correspondingly, the local model and global model mentioned in one or more embodiments of the present application are different states of the same model. For example, the personalized classification model trained locally on the client device becomes the local model, and the classification model obtained by aggregating the weights of various personalized classification models on the server is the global model. Correspondingly, the local model weight includes: the weight of the local classifier head; the global model weight includes: the weight of the global classifier head.

[0069] Step 300: Receive the global model weight of the current round sent by the server, and determine the reward of the current round based on the local reinforcement learning agent unit according to the global model weight and local model weight corresponding to the previous round and the current round respectively.

[0070] Step 400: Based on the global model weights and local model weights of the previous round, the action of the current round, the reward of the current round, and the global model weights and local model weights of the current round, a tuple sample of the current round is constructed and stored in the local experience pool.

[0071] In step 400, the global model weight and local model weight of the previous round t-1 are the state s of the current round t t , the global model weight and local model weight of the current round t are the state s of the next round t+1 t+1 Therefore, the next round needs to use the global model weight and local model weight of the current round to perform step 100, and the action a of the current round t t It is also the hyperparameter τ of the action in the current round t , the tuple sample also contains the reward r of the current round t Based on this, the tuple samples collected by each agent are (s t ,a t ,r t ,s t+1 ), t>1.

[0072] Specifically, for the i-th client device, the state s of the current round t of the i-th client device is t The global model weight of the i-th client device in the previous round t-1 and local model weights Composition, recorded as The state s of the i-th client device in the next round t+1 t+1 The global model weight of the i-th client device in the current round t and local model weights Composition, recorded as The action a of the current round t of the i-th client device t Recorded as The reward r of the current round t of the i-th client device t Denoted as r i t , then for the i-th client device, the tuple sample of the t-th round obtained in step 400 is The tuple sample of the previous round t-1 obtained in step 100 is In one or more embodiments of the present application, action a may also be represented by γ.

[0073] In addition, the local experience pool of the i-th client device can be recorded as P i , in the tth round, the local experience pool of the i-th client device can record At round t, the collective experience pool of all agents is expressed as Wherein, k is the total number of client devices currently participating in federated learning in the federated learning system.

[0074] It should be noted that, since the action selection of this application is in a continuous space, this application adopts the PPO (Proximal Policy Optimization) algorithm when selecting the reinforcement learning method. By introducing the truncation mechanism, PPO avoids performance fluctuations caused by drastic changes in strategy, thereby improving the stability and convergence speed of the training process. In addition, due to the nature of the task of this application, the data generated for the agent in federated learning is relatively scarce. The PPO algorithm can be effectively trained with limited experience replay and low data requirements. These advantages become more significant when combined with federated learning. Overall, in the context of heterogeneous data and dynamic environments, PPO can prevent training instability due to limited data and poor performance problems that other reinforcement learning algorithms may encounter.

[0075] From the above description, it can be seen that the adaptive constrained federated learning method based on reinforcement learning provided in the embodiment of the present application can solve the problem of decreased model performance caused by the inconsistency between the data distribution of each client and the data distribution of the test domain due to data heterogeneity between clients under the federated learning framework. It can adaptively adjust the correlation between the local model of each client and the global model to reduce the impact of data heterogeneity, effectively improve the reliability and effectiveness of model training, and thus improve model performance; at the same time, each client has a local reinforcement learning agent to make independent decisions, so that each client can fully and autonomously adapt to the training conditions and make decisions, without the need for additional data communication between the server and the client, thereby effectively reducing the resource consumption and time cost of data communication between the server and the client.

[0076] In order to further improve the application effectiveness and reliability of adaptive constrained federated learning based on reinforcement learning, in an adaptive constrained federated learning method based on reinforcement learning provided in an embodiment of the present application, see Figure 2 The step 100 in the adaptive constrained federated learning method based on reinforcement learning also specifically includes the following contents:

[0077] Step 010: Build a local experience pool;

[0078] Step 020: Initialize the action model and evaluation model corresponding to the reinforcement learning agent unit.

[0079] Specifically, the action model is the Actor model in reinforcement learning. For the i-th client device, its action model is denoted by θ i The evaluation model is the Critic model in reinforcement learning. For the i-th client device, its evaluation model is recorded as Q i .

[0080] Step 030: Divide each data sample in the local data into a training set and a validation set respectively.

[0081] Specifically, local data refers to the local data set. For the i-th client device, its local data set is denoted as D i , and divide it into a training set and validation set

[0082] And, step 040: receiving initial global model parameters sent by a server in the federated learning system.

[0083] The initial global model parameters are denoted as

[0084] Step 050: In the initial round, the local model corresponding to the initial global model parameters is trained based on local data and preset initial hyperparameters to obtain the local model weights of the initial round, and the local model weights are sent to the server in the federated learning system so that the server aggregates the local model weights received in the initial round to obtain the global model weights of the initial round.

[0085] Specifically, taking a single client as an example, client i receives the In the initial global communication round, since there is no personalized model available to build the state for action selection, client i uses the preset initial hyperparameters Perform local training. Loss in the local training phase The function is:

[0086]

[0087] Among them, (x i ,y i )∈D i , represents the local model that is continuously updated in the current global round, and L(·,·,·) represents the cross-extraction loss function.

[0088] Step 060: Receive the global model weights of the initial round sent by the server.

[0089] Among them, for client i, in the first round, the global model weight of the initial round sent by the server is recorded as

[0090] In order to further improve the effectiveness and reliability of determining the hyperparameters of the current round of actions, in an adaptive constraint federated learning method based on reinforcement learning provided in an embodiment of the present application, see Figure 2 , step 100 in the adaptive constrained federated learning method based on reinforcement learning specifically includes the following contents:

[0091] Step 110: Check whether the local experience pool contains tuple samples of the previous round. If so, extract the global model weights and local model weights of the previous round as the state of the previous round from the tuple samples of the previous round.

[0092] In step 110, if the local experience pool does not contain tuple samples from the previous round, it means that the current round may be the initial round or an early round (such as the second round) in which tuple samples have not yet been generated. It is necessary to use the preset initial hyperparameters in the manner of steps 050 and 060. Perform local model training.

[0093] Step 120: Based on the action model in the reinforcement learning agent unit, the hyperparameters of the action in the current round are generated according to the global model weights and local model weights of the previous round.

[0094] In one example, after receiving the global model, the local client can observe the local model classifier head and the global classifier head in the previous round of training. i ,Q i ) uses this information to select the local training hyperparameter τ, which controls the extent to which the local model training is constrained by the global model.

[0095] In order to emphasize the performance comparison of the model in consecutive rounds, reduce the impact of rapid performance improvement when the model quickly fits the data in the early stage of training, and further improve the effectiveness and reliability of determining the hyperparameters of the action in the current round, in an adaptive constrained federated learning method based on reinforcement learning provided in an embodiment of the present application, see Figure 2 , step 300 in the adaptive constrained federated learning method based on reinforcement learning specifically includes the following contents:

[0096] Step 310: Obtain the accuracy of the global model weight and the local model weight corresponding to the previous round and the current round on the verification set;

[0097] Step 320: Determine the reward for the current round based on the preset weight change parameters and the accuracy of the global model weights and local model weights corresponding to the previous round and the current round on the verification set; wherein the weight change parameters are used to determine the round value corresponding to the current round, and different round values ​​correspond to different weight change parameters.

[0098] Specifically, the reward is used as the basis for the agent to evaluate the quality of the action. In the specific task of this application, the effectiveness of τ is most clearly reflected in the validation set. Therefore, in the method of this application, the reward of the tth round corresponding to the i-th client device is Set to:

[0099]

[0100] Among them, t represents the current communication round, represents the accuracy of client i on its validation set in round t, represents the accuracy of client i on its validation set in round t-1, acc t represents the accuracy of the global model on the t-th round of validation set, acc t-1 represents the accuracy of the global model on the validation set in the t-1th round, and β is a predefined weight change parameter.

[0101] In the reward setting of this application, this application emphasizes the performance comparison of the model in consecutive rounds. Therefore, this application mainly focuses on the difference in performance between consecutive rounds when determining rewards. In addition, this application hopes that the learned strategy can not only improve the performance of the model in local training, but also avoid excessive personalization of the model, which leads to poor generalization ability after aggregation. To this end, this application incorporates the performance of the global model into the reward calculation.

[0102] Considering the model training process, the model starts with random weights and quickly learns the characteristics and distribution of the data. The application observes significant performance improvements in early iterations. As training progresses, the model may approach the optimal solution, resulting in a slower rate of performance improvement. Therefore, the application introduces another factor in the reward calculation. This factor reduces the impact of rapid performance improvement when the model quickly fits the data in the early stages of training, while increasing the impact of the application's strategy on performance during the relatively stable stage of model training.

[0103] In order to further improve the application effectiveness and reliability of the reinforcement learning agent unit, in an adaptive constraint federated learning method based on reinforcement learning provided in an embodiment of the present application, see Figure 2 , the step 400 in the adaptive constrained federated learning method based on reinforcement learning further specifically includes the following contents:

[0104] Step 500: If the number of the tuple samples in the local experience pool has reached a preset quantity threshold, multiple tuple samples are extracted from other tuple samples in the local experience pool except the tuple samples in the previous round, and the reinforcement learning agent unit is trained based on the extracted tuple samples.

[0105] Among them, the quantity threshold can be set manually in advance, and a smaller value can be selected to increase the training frequency of the reinforcement learning agent unit for application reliability. For example, it can be set to any positive integer between 3 and 10.

[0106] In order to realize that multiple agents share samples according to data distribution, the efficiency of reinforcement learning training is improved without compromising privacy or increasing communication costs, so as to further improve the application effectiveness and reliability of reinforcement learning agent units, in an adaptive constrained federated learning method based on reinforcement learning provided in an embodiment of the present application, see Figure 2 , the step 500 in the adaptive constrained federated learning method based on reinforcement learning further specifically includes the following contents:

[0107] Step 600: Receive an index of similar client devices sent by a server in a federated learning system, wherein the server searches for corresponding similar client devices for each client device that sends the local model weights according to the similarity between each local model weight received in the current round, and sends the unique identifier of the similar client device corresponding to each client device to each client device.

[0108] Step 700: Based on the index of the similar client device, copy a plurality of the tuple samples from the experience pool of the similar client device.

[0109] Step 800: After training the local reinforcement learning agent unit based on each of the tuple samples extracted from the local experience pool, train the local reinforcement learning agent unit based on each of the tuple samples copied from the experience pool of the similar client device.

[0110] Specifically, in each round, the application uses the local model weight as the judgment criterion. For each client i, the application searches for the most similar client j whose update direction is highly consistent with that of client i:

[0111]

[0112] Client i uses its local experience pool After the update, the experience pool of similar client j will also be used Make additional updates.

[0113] In order to further illustrate the various embodiments of the above-mentioned adaptive constrained federated learning method based on reinforcement learning, the present application also provides a specific application example of an adaptive constrained federated learning method based on reinforcement learning, wherein the model specifically adopts a classification model (i.e., an image classification model), the local model weight specifically adopts the weight of the local classifier head (i.e., the weight of the bias classifier head of the client), and the global model weight specifically adopts the weight of the global classifier head (i.e., the classifier head weight of the global model after aggregation) as an example for illustration,

[0114] The execution architecture of this application example is as follows Figure 3 As shown in the figure, all clients have the same structure, feature extraction layer, classifier layer and private agent. The feature extraction layer extracts image features for classifier recognition and updates. The update direction is determined by the reinforcement learning agent strategy, and the agent is continuously updated during the training process. The server aggregates the model weights after local training and calculates the closest neighbor client for each client to assist in training. In federated learning, it is generally composed of 1 server and K clients, each client has a data set Where N i Indicates the amount of data owned by the i-th client.

[0115] The adaptive constraint federated learning method based on reinforcement learning provided in this specific application example specifically includes the following contents:

[0116] 1. Strengthening the personalized configuration of federated learning agents

[0117] Status: It is well known that during the local training process of federated learning, the model acquires information about the data distribution. This is the premise for federated learning to achieve a general global model without sharing data. Experimental evidence shows that non-IID data significantly affects the classifier layer. Differences in data distribution between different clients lead to biases in models trained based on the same global model initialization. After training on different data, the classifier heads tend to be dispersed in the vector space, resulting in poor generalization of the aggregated model.

[0118] On the contrary, this property of the classifier head indicates that they tend to learn information about the data distribution, because the deviations of different classifier heads come from the changes in different data distributions. Therefore, this application example uses the weights of the classifier head as the state: in represents the weight of the client's bias classifier head in the tth round of global training of federated learning, Represents the classifier head weights of the global model after the previous round of model aggregation, which are used as the initialization of the classifier head in round t. This state configuration not only avoids taking the entire model weight as a parameter, thereby reducing the burden of reinforcement learning, but also enables the agent to more directly observe the main reason for the degradation of federated learning performance caused by non-independent and identically distributed (non-IID) data.

[0119] Action: After receiving the global model, the local client observes the local model classifier head and the global classifier head in the previous training round. The agent uses this information to select the hyperparameter τ for local training, which controls the degree to which the local model training is constrained by the global model.

[0120] Reward: Reward is the basis for the agent to evaluate the quality of action. In the specific task of this application example, the effectiveness of τ is most clearly reflected on the validation set. Therefore, in the method of this application example, this application example sets the reward as:

[0121]

[0122] Among them, t represents the current communication round, represents the accuracy of client i on its validation set in round t, represents the accuracy of client i on its validation set in round t-1, acc t represents the accuracy of the global model on the t-th round of validation set, acc t-1Represents the accuracy of the global model on the validation set in the t-1th round, and β is a predefined weight change parameter. In the reward setting of this application example, this application example emphasizes the performance comparison of the model in consecutive rounds. Therefore, this application example focuses on the difference in performance between consecutive rounds when determining rewards. In addition, this application example hopes that the learned strategy can not only improve the performance of the model in local training, but also avoid excessive personalization of the model, which leads to poor generalization ability after aggregation. To this end, this application example incorporates the performance of the global model into the reward calculation.

[0123] Considering the model training process, where the model starts with random weights and quickly learns the characteristics and distribution of the data, this application example observes significant performance improvements in early iterations. As training progresses, the model may approach the optimal solution, resulting in a slower rate of performance improvement. Therefore, this application example introduces another factor into the reward calculation. This factor reduces the impact of rapid performance improvements in the early stages of training when the model quickly fits the data, while increasing the impact of this application example's strategy on performance during the relatively stable stage of model training.

[0124] 2. Experience Pool Collaboration Based on Data Distribution

[0125] In the specific task setting of this application example, each client has a private agent responsible for decision making and evaluation actions. The agent only needs to pay attention to its own state and construct a tuple (s t ,a t ,r t ,s t+1 ) and put it into the experience pool for experience replay. Unlike agents that only exist on the server side, it mainly focuses on the distribution of local data and the update direction related to data distribution information, but this will face a challenge: sparse sample availability. In this case, each individual agent is similar to a client in federated learning, and the samples in the experience replay pool represent the data held by the client. The tuple includes the local classifier head and the global classifier head, and sharing samples does not leak privacy. However, sharing all samples is similar to training K identical server-side agents in federated learning, which wastes precious computing resources. Therefore, this application proposes an example of experience pool sharing based on data distribution:

[0126] During the client training phase of federated learning, each agent collects tuples (s t ,a t ,r t ,s t+1 ) and add them to the experience pool In the tth round, the collective experience pool of all agents is expressed as In each round, this application example uses the weight of the classifier head as the judgment criterion. For each client i, this application example finds the most similar client j with a highly consistent update direction with client i:

[0127]

[0128] Client i uses its local experience pool After the update, the experience pool will also be used Make additional updates.

[0129] 3. Experience Pool Collaboration Based on Data Distribution

[0130] 1. Initialization: Before training begins, each client builds an experience pool P i , initialize the Actor model θ i and Critic model Q i , load the local dataset D i And divide it into training set and validation set At the same time, the server initializes the global model to obtain the initial global model parameters

[0131] 2. Local training: Taking a single client as an example, client i receives the In the initial global communication round (round 0), since there is no personalized model available to build the state for action selection, the client uses the preset hyperparameters Perform local training. The loss function in the local training phase is:

[0132]

[0133] Among them, (x i ,y i )∈D i , represents the local model that is continuously updated in the current global round, and l(·,·,·) represents the cross-extraction loss function.

[0134] When the global communication round is greater than 1, the agent of the reinforcement learning module starts to participate in local training and builds tuples using historical elements. and add them to P i ,in represents the weight of the classifier head. i When it is full, some samples will be extracted from it to update the agent (θ i ,Q i ). The agent will then j to extract samples from the dataset to acquire knowledge similar to the data distribution. Once all updates are completed, θi Select τ for local model training. The training loss function is almost the same as that of round 0.

[0135]

[0136] At the end of local training, we get

[0137] 3. Model aggregation: Upload to the server and aggregate In this phase, the server uses the EPC module to calculate the similarity between clients and assigns each client the index of the most similar client.

[0138] 4. Model Evaluation: exist Verify on Validate on the pre-split global validation set and get acc t , each client calculates the reward separately. Steps 2 to 4 are repeated cyclically until the total communication round T is reached.

[0139] That is to say, this application example proposes an adaptive constrained federated learning framework based on reinforcement learning, which integrates reinforcement learning and federated learning. Each client has its own private agent to make decisions, so that each client can fully adapt to training conditions and make decisions autonomously. This application example also proposes a method based on indirect data distribution, called the Experience Pool Collaboration (EPC) method. Multiple agents share samples according to data distribution, which improves the efficiency of reinforcement learning training without compromising privacy or increasing communication costs.

[0140] In summary, the adaptive constrained federated learning method based on reinforcement learning provided by the embodiments and application examples of the present application allows the client to dynamically select the local update direction according to the actual situation of each round of local training, avoids possible forgetting problems in model training, and improves the global image classification performance of the federated learning client model; utilizes the strong correlation between the classifier and the data distribution, calculates the closest client in each round for experience sample sharing, and improves the correctness of the reinforcement learning strategy.

[0141] The present application also provides a reinforcement learning-based adaptive constrained federated learning device for executing all or part of the reinforcement learning-based adaptive constrained federated learning method, see Figure 4 The adaptive constraint federated learning device based on reinforcement learning specifically includes the following contents:

[0142] The reinforcement learning action determination module 10 is used to extract the global model weights and local model weights of the previous round from the tuple samples of the previous round pre-stored in the local experience pool, and based on the local reinforcement learning agent unit, determine the hyperparameters of the action of the current round according to the global model weights and local model weights of the previous round.

[0143] The adaptive constrained federated learning module 20 is used to use each data sample in the local training set to train the local model corresponding to the global model weight of the previous round, and adaptively constrain the local model based on the hyperparameters during the training process to obtain the local model weight of the local model in the current round, and send the local model weight to the server in the federated learning system, so that the server aggregates the local model weights received in the current round to obtain the global model weight of the current round.

[0144] The reinforcement learning reward determination module 30 is used to receive the global model weight of the current round sent by the server, and based on the local reinforcement learning agent unit, determine the reward of the current round according to the global model weight and local model weight corresponding to the previous round and the current round respectively.

[0145] The experience pool storage module 40 is used to construct a tuple sample of the current round and store it in the local experience pool based on the global model weights and local model weights of the previous round, the action of the current round, the reward of the current round, and the global model weights and local model weights of the current round.

[0146] The embodiment of the adaptive constrained federated learning device based on reinforcement learning provided in the present application can be specifically used to execute the processing flow of the embodiment of the adaptive constrained federated learning method based on reinforcement learning in the above-mentioned embodiment. Its functions will not be repeated here, and reference can be made to the detailed description of the embodiment of the adaptive constrained federated learning method based on reinforcement learning.

[0147] The part of the adaptive constrained federated learning based on reinforcement learning performed by the adaptive constrained federated learning device based on reinforcement learning can be completed in the client device. The specific selection can be based on the processing capability of the client device and the limitations of the user's usage scenario. This application is not limited to this. If all operations are completed in the client device, the client device may also include a processor for specific processing of the adaptive constrained federated learning based on reinforcement learning.

[0148] The client device may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission with the server. The server may include a server on the task scheduling center side, and other implementation scenarios may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, or a server cluster consisting of multiple servers, or a server structure of a distributed device.

[0149] The server and the client device may communicate with each other using any suitable network protocol, including network protocols that have not yet been developed on the date of filing this application. The network protocols may include, for example, TCP / IP, UDP / IP, HTTP, HTTPS, etc. Of course, the network protocols may also include, for example, RPC (Remote Procedure Call Protocol) and REST (Representational State Transfer) protocols used on top of the above protocols.

[0150] From the above description, it can be seen that the adaptive constrained federated learning device based on reinforcement learning provided in the embodiment of the present application can solve the problem of decreased model performance caused by the inconsistency between the data distribution of each client and the data distribution of the test domain due to data heterogeneity between clients under the federated learning framework. It can adaptively adjust the correlation between the local model of each client and the global model to reduce the impact of data heterogeneity, effectively improve the reliability and effectiveness of model training, and thus improve model performance; at the same time, each client has a local reinforcement learning agent to make independent decisions, so that each client can fully and autonomously adapt to the training conditions and make decisions, without the need for additional data communication between the server and the client, thereby effectively reducing the resource consumption and time cost of data communication between the server and the client.

[0151] The present application also provides an adaptive constrained federated learning system based on reinforcement learning, and the adaptive constrained federated learning system specifically includes the following contents:

[0152] A server and each client device in communication with the server;

[0153] Each of the client devices is used for the adaptive constrained federated learning method based on reinforcement learning described in the aforementioned embodiment.

[0154] The embodiment of the present application also provides an electronic device, which may include a processor, a memory, a receiver and a transmitter, wherein the processor is used to execute the adaptive constraint federated learning method based on reinforcement learning mentioned in the above embodiment, wherein the processor and the memory may be connected via a bus or other means, such as by bus connection. The receiver may be connected to the processor and the memory via wired or wireless means.

[0155] The processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0156] As a non-transient computer-readable storage medium, the memory can be used to store non-transient software programs, non-transient computer executable programs and modules, such as the program instructions / modules corresponding to the adaptive constraint federated learning method based on reinforcement learning in the embodiment of the present application. The processor executes various functional applications and data processing of the processor by running the non-transient software programs, instructions and modules stored in the memory, that is, implementing the adaptive constraint federated learning method based on reinforcement learning in the above method embodiment.

[0157] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0158] The one or more modules are stored in the memory, and when executed by the processor, perform the adaptive constrained federated learning method based on reinforcement learning in the embodiment.

[0159] In some embodiments of the present application, the user equipment may include a processor, a memory, and a transceiver unit, which may include a receiver and a transmitter. The processor, memory, receiver, and transmitter may be connected through a bus system. The memory is used to store computer instructions, and the processor is used to execute the computer instructions stored in the memory to control the transceiver unit to send and receive signals.

[0160] As an implementation method, the functions of the receiver and the transmitter in the present application can be considered to be implemented through a transceiver circuit or a dedicated chip for transceiver, and the processor can be considered to be implemented through a dedicated processing chip, a processing circuit or a general chip.

[0161] As another implementation method, it is possible to use a general-purpose computer to implement the server provided in the embodiment of the present application, that is, to store the program code for implementing the functions of the processor, receiver, and transmitter in a memory, and the general-purpose processor implements the functions of the processor, receiver, and transmitter by executing the code in the memory.

[0162] The embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the aforementioned adaptive constrained federated learning method based on reinforcement learning are implemented. The computer-readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the technical field.

[0163] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the adaptive constrained federated learning method based on reinforcement learning.

[0164] It should be understood by those skilled in the art that the exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is specifically performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0165] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0166] In the present application, features described and / or illustrated for one embodiment may be used in the same manner or in a similar manner in one or more other embodiments, and / or combined with features of other embodiments or replace features of other embodiments.

[0167] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the embodiments of the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. An adaptive constrained federated learning method based on reinforcement learning, characterized in that: include: Extract the global model weight and the local model weight of the previous round from the tuple samples of the previous round pre-stored in the local experience pool, and determine the hyperparameters of the action of the current round based on the global model weight and the local model weight of the previous round based on the local reinforcement learning agent unit; Using each data sample in the local training set to train the local model corresponding to the global model weight of the previous round, and adaptively constraining the local model based on the hyperparameters during the training process to obtain the local model weight of the local model in the current round, and sending the local model weight to the server in the federated learning system, so that the server aggregates the local model weights received in the current round to obtain the global model weight of the current round; Receive the global model weight of the current round sent by the server, and determine the reward of the current round based on the local reinforcement learning agent unit according to the global model weight and the local model weight corresponding to the previous round and the current round respectively; According to the global model weight and the local model weight of the previous round, the action of the current round, the reward of the current round, and the global model weight and the local model weight of the current round, a tuple sample of the current round is constructed and stored in the local experience pool.

2. The adaptive constraint federated learning method based on reinforcement learning according to claim 1, characterized in that: Before extracting the global model weight and the local model weight of the previous round from the tuple samples of the previous round pre-stored in the local experience pool, the method further includes: Build a local experience pool; Initialize the action model and evaluation model corresponding to the reinforcement learning agent unit; Divide each data sample in the local data into a training set and a validation set respectively; and, receiving initial global model parameters sent by a server in the federated learning system; In the initial round, the local model corresponding to the initial global model parameter is trained based on the local data and the preset initial hyperparameters to obtain the local model weight of the initial round, and the local model weight is sent to the server in the federated learning system, so that the server aggregates the local model weights received in the initial round to obtain the global model weight of the initial round; The global model weight of the initial round is received from the server.

3. The adaptive constraint federated learning method based on reinforcement learning according to claim 2, characterized in that: The method extracts the global model weight and the local model weight of the previous round from the tuple samples of the previous round pre-stored in the local experience pool, and determines the hyperparameters of the action of the current round based on the global model weight and the local model weight of the previous round according to the local reinforcement learning agent unit, including: Searching whether the local experience pool contains a tuple sample of the previous round, and if so, extracting the global model weight and local model weight of the previous round as the state of the previous round from the tuple sample of the previous round; Based on the action model in the reinforcement learning agent unit, hyper parameters of the action of the current round are generated according to the global model weight and the local model weight of the previous round.

4. The adaptive constraint federated learning method based on reinforcement learning according to claim 2, characterized in that: The locally based reinforcement learning agent unit determines the reward of the current round according to the global model weight and the local model weight corresponding to the previous round and the current round, respectively, including: Obtain the accuracy of the global model weight and the local model weight corresponding to the previous round and the current round on the verification set; The reward for the current round is determined according to the preset weight change parameters, and the accuracy of the global model weights and local model weights corresponding to the previous round and the current round on the verification set; wherein the weight change parameters are used to determine according to the round value corresponding to the current round, and different round values ​​correspond to different weight change parameters.

5. The adaptive constraint federated learning method based on reinforcement learning according to claim 1, characterized in that: Also includes: If the number of the tuple samples in the local experience pool has reached a preset quantity threshold, multiple tuple samples other than the tuple samples in the previous round in the local experience pool are extracted, and the reinforcement learning agent unit is trained based on the extracted tuple samples.

6. The adaptive constraint federated learning method based on reinforcement learning according to claim 5, characterized in that: Also includes: Receiving an index of similar client devices sent by a server in a federated learning system, wherein the server searches for corresponding similar client devices for each client device that sends the local model weights according to the similarity between the local model weights received in the current round, and sends the unique identifier of the similar client device corresponding to each client device to each client device; Based on the index of the similar client device, copying a plurality of the tuple samples from an experience pool of the similar client device; After the local reinforcement learning agent unit is trained based on each of the tuple samples extracted from the local experience pool, the local reinforcement learning agent unit is trained based on each of the tuple samples copied from the experience pool of the similar client device.

7. The adaptive constraint federated learning method based on reinforcement learning according to any one of claims 1 to 6, characterized in that: The data samples include: image data samples; The local model weights include: weights of local classifier heads; The global model weight includes: the weight of the global classifier head.

8. An adaptive constrained federated learning device based on reinforcement learning, characterized in that: include: A reinforcement learning action determination module is used to extract the global model weight and the local model weight of the previous round from the tuple samples of the previous round pre-stored in the local experience pool, and determine the hyperparameters of the action of the current round based on the global model weight and the local model weight of the previous round based on the local reinforcement learning agent unit; An adaptive constrained federated learning module, used to train the local model corresponding to the global model weight of the previous round using each data sample in the local training set, and adaptively constrain the local model based on the hyperparameters during the training process to obtain the local model weight of the local model in the current round, and send the local model weight to the server in the federated learning system, so that the server aggregates the local model weights received in the current round to obtain the global model weight of the current round; A reinforcement learning reward determination module, configured to receive the global model weight of the current round sent by the server, and determine the reward of the current round based on the local reinforcement learning agent unit according to the global model weight and the local model weight corresponding to the previous round and the current round respectively; The experience pool storage module is used to construct a tuple sample of the current round and store it in the local experience pool based on the global model weight and local model weight of the previous round, the action of the current round, the reward of the current round, and the global model weight and local model weight of the current round.

9. A federated learning system, characterized in that: include: A server and each client device in communication with the server; Each of the client devices is used to execute the adaptive constrained federated learning method based on reinforcement learning as described in any one of claims 1 to 7.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the adaptive constrained federated learning method based on reinforcement learning as described in any one of claims 1 to 8 is implemented.

Images