Adaptive constrained federated learning method, device and system based on reinforcement learning

Abstract

The present application provides a method, device, and system for adaptive constrained federated learning based on reinforcement learning. The method includes: determining hyperparameters for the actions of the current round based on the global model weights and local model weights of the previous round based on a local reinforcement learning agent unit; adaptively constraining the local model based on the hyperparameters during training, sending the local model weights of the current round to a server in the federated learning system for aggregation; determining the reward for the current round based on the global model weights and local model weights corresponding to the previous and current rounds based on the reinforcement learning agent unit; and constructing and storing tuple samples for the current round. The present application can mitigate the impact of data heterogeneity and improve the reliability, effectiveness, and performance of the trained model; making independent decisions through the local reinforcement learning agent without the need for additional data communication between the server and the client.

Description

Technical Field

[0001] The present application relates to the field of federated learning technology, and in particular to a method, device, and system for adaptive constrained federated learning based on reinforcement learning. Background Art

[0002] Deep learning is increasingly being used to solve complex problems, with models typically trained on public or private datasets. The quality of these datasets has a significant impact on model performance; public datasets typically have a balanced distribution, while private datasets may excel in certain areas but lack comprehensive coverage. However, due to privacy concerns, individuals prefer to keep their data to themselves, which can lead to an imbalanced data distribution and, consequently, poor model performance across organizations. Federated Learning (FL) addresses this issue by integrating information from multiple sources without sharing the data. Federated Learning involves clients training models on local datasets, which are then aggregated on a server to enhance model performance over multiple rounds of iteration.

[0003] The emergence of Compute-First Networking (CFN) has brought new vitality to federated learning. By integrating discrete computing resources, large-scale CFN systems provide an ideal environment for federated learning tasks, making them more convenient. However, both pure federated learning and federated learning combined with CFN face the problem of data heterogeneity. The different data distributions of each client make it difficult for the global model to converge and generalize effectively. Furthermore, when the data classifications of each client are highly unbalanced, the global model often forgets useful information from other clients when it aggregates after local training.

[0004] Currently, most research on data heterogeneity in federated learning analyzes models in layers and employs regularization constraints to control the differences between locally trained models and the global model, preventing global information loss due to local training. However, this approach imposes global constraints and regulations on the model, applying uniform rules to all models participating in federated training while ignoring the unique circumstances of individual clients. Other approaches, which focus on fixing or constraining the model, can prevent client-side models from being adversely affected by heterogeneous data. However, the classifier layer is susceptible to changes due to data influence, ignoring the useful information about the data distribution that may be contained in the model.

[0005] Therefore, there is an urgent need to design a federated learning method that can solve the problem of model performance degradation caused by data heterogeneity between clients under the federated learning framework. Summary of the Invention

[0006] In view of this, the embodiments of the present application provide an adaptive constrained federated learning method, device and system based on reinforcement learning to eliminate or improve one or more defects in the prior art.

[0007] One aspect of the present application provides an adaptive constrained federated learning method based on reinforcement learning, comprising:

[0008] Extract the global model weights and local model weights of the previous round from the tuple samples pre-stored in the local experience pool, and determine the hyperparameters of the action of the current round based on the global model weights and local model weights of the previous round based on the local reinforcement learning agent unit;

[0009] Using each data sample in the local training set to train the local model corresponding to the global model weight of the previous round, and adaptively constraining the local model based on the hyperparameters during the training process to obtain the local model weight of the local model in the current round, and sending the local model weight to the server in the federated learning system, so that the server aggregates the local model weights received in the current round to obtain the global model weight of the current round;

[0010] Receiving the global model weight of the current round sent by the server, and determining the reward of the current round based on the local reinforcement learning agent unit according to the global model weight and local model weight corresponding to the previous round and the current round respectively;

[0011] According to the global model weight and local model weight of the previous round, the action of the current round, the reward of the current round, and the global model weight and local model weight of the current round, a tuple sample of the current round is constructed and stored in the local experience pool.

[0012] In some embodiments of the present application, before extracting the global model weights and local model weights of the previous round from the tuple samples of the previous round pre-stored in the local experience pool, the process further includes:

[0013] Build a local experience pool;

[0014] Initialize the action model and evaluation model corresponding to the reinforcement learning agent unit;

[0015] Divide each data sample in the local data into a training set and a validation set respectively;

[0016] and, receiving initial global model parameters sent by a server in the federated learning system;

[0017] In the initial round, the local model corresponding to the initial global model parameters is trained based on the local data and the preset initial hyperparameters to obtain the local model weights of the initial round, and the local model weights are sent to the server in the federated learning system so that the server aggregates the local model weights received in the initial round to obtain the global model weights of the initial round;

[0018] Receive the global model weight of the initial round sent by the server.

[0019] In some embodiments of the present application, extracting the global model weights and local model weights of the previous round from the tuple samples of the previous round pre-stored in the local experience pool, and determining, based on the local reinforcement learning agent unit, the global model weights and local model weights of the previous round as the hyperparameters of the action of the current round, includes:

[0020] Searching the local experience pool to see if it contains a tuple sample from the previous round, and if so, extracting the global model weight and local model weight of the previous round as the state of the previous round from the tuple sample from the previous round;

[0021] Based on the action model in the reinforcement learning agent unit, hyperparameters of the action in the current round are generated according to the global model weight and the local model weight of the previous round.

[0022] In some embodiments of the present application, the locally based reinforcement learning agent unit determines the reward for the current round based on the global model weights and local model weights corresponding to the previous round and the current round, respectively, including:

[0023] Obtain the accuracy of the global model weight and the local model weight corresponding to the previous round and the current round on the validation set;

[0024] The reward for the current round is determined based on the preset weight change parameters and the accuracy of the global model weights and local model weights corresponding to the previous round and the current round on the validation set; wherein the weight change parameters are used to be determined based on the round value corresponding to the current round, and different round values ​​correspond to different weight change parameters.

[0025] In some embodiments of the present application, further comprising:

[0026] If the number of the tuple samples in the local experience pool has reached a preset threshold, multiple tuple samples other than the tuple samples in the previous round in the local experience pool are extracted, and the reinforcement learning agent unit is trained based on the extracted tuple samples.

[0027] In some embodiments of the present application, further comprising:

[0028] Receiving an index of similar client devices sent by a server in the federated learning system, wherein the server searches for a corresponding similar client device for each client device that has sent the local model weight according to the similarity between the local model weights received in the current round, and sends a unique identifier of the similar client device corresponding to each client device to each client device;

[0029] Based on the index of the similar client device, copying a plurality of the tuple samples from the experience pool of the similar client device;

[0030] After training the local reinforcement learning agent unit based on each of the tuple samples extracted from the local experience pool, the local reinforcement learning agent unit is trained based on each of the tuple samples copied from the experience pool of the similar client device.

[0031] In some embodiments of the present application, the data sample includes: an image data sample;

[0032] The local model weights include: weights of local classifier heads;

[0033] The global model weight includes: the weight of the global classifier head.

[0034] Another aspect of the present application provides an adaptive constraint federated learning device based on reinforcement learning, comprising:

[0035] A reinforcement learning action determination module is used to extract the global model weights and local model weights of the previous round from the tuple samples of the previous round pre-stored in the local experience pool, and determine the hyperparameters of the action of the current round based on the global model weights and local model weights of the previous round based on the local reinforcement learning agent unit;

[0036] an adaptive constrained federated learning module, configured to train a local model corresponding to the global model weight of the previous round using each data sample in the local training set, and adaptively constrain the local model based on the hyperparameters during the training process to obtain a local model weight of the local model in the current round, and send the local model weight to a server in the federated learning system so that the server aggregates the local model weights received in the current round to obtain a global model weight of the current round;

[0037] a reinforcement learning reward determination module, configured to receive the global model weights for the current round sent by the server, and determine the reward for the current round based on the global model weights and local model weights corresponding to the previous round and the current round, respectively, based on the local reinforcement learning agent unit;

[0038] The experience pool storage module is used to construct a tuple sample of the current round and store it in the local experience pool based on the global model weight and local model weight of the previous round, the action of the current round, the reward of the current round, and the global model weight and local model weight of the current round.

[0039] A third aspect of the present application provides a federated learning system, comprising: a server and various client devices in communication with the server;

[0040] Each of the client devices is used to execute the adaptive constrained federated learning method based on reinforcement learning.

[0041] The fourth aspect of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the adaptive constrained federated learning method based on reinforcement learning when executing the computer program.

[0042] The fifth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the adaptive constrained federated learning method based on reinforcement learning.

[0043] The sixth aspect of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the adaptive constrained federated learning method based on reinforcement learning.

[0044] The adaptive constrained federated learning method based on reinforcement learning provided in the present application extracts the global model weights and local model weights of the previous round from the tuple samples of the previous round pre-stored in the local experience pool, and determines the hyperparameters of the action of the current round based on the global model weights and local model weights of the previous round based on the local reinforcement learning agent unit; uses each data sample in the local training set to train the local model corresponding to the global model weight of the previous round, and adaptively constrains the local model based on the hyperparameters during the training process to obtain the local model weight of the local model in the current round, and sends the local model weight to the server in the federated learning system so that the server aggregates the local model weights received in the current round to obtain the global model weight of the current round; receives the global model weight of the current round sent by the server, and based on the local reinforcement learning agent unit, aggregates the global model weights and The local model weight determines the reward of the current round; based on the global model weight and local model weight of the previous round, the action of the current round, the reward of the current round, and the global model weight and local model weight of the current round, a tuple sample of the current round is constructed and stored in the local experience pool. This can solve the problem of data heterogeneity between clients under the federated learning framework, which leads to inconsistent data distribution between each client and the test domain data distribution, resulting in decreased model performance. It can adaptively adjust the correlation between the local model of each client and the global model to reduce the impact of data heterogeneity, effectively improve the reliability and effectiveness of model training, and thus improve model performance. At the same time, each client has a local reinforcement learning agent to make independent decisions, so that each client can fully and autonomously adapt to training conditions and make decisions, without the need for additional data communication between the server and the client, thereby effectively reducing the resource consumption and time cost of data communication between the server and the client.

[0045] Additional advantages, purposes, and features of the present application will be described in part in the following description and will become apparent to those skilled in the art upon study of the following or may be learned from practice of the present application. The purposes and other advantages of the present application may be achieved and obtained by the structures specifically pointed out in the specification and drawings.

[0046] Those skilled in the art will understand that the purposes and advantages that can be achieved by the present application are not limited to the above specific description, and the above and other purposes that can be achieved by the present application will be more clearly understood based on the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The drawings described herein are intended to provide a further understanding of the present application, constitute a part of the present application, and do not constitute a limitation of the present application. The components in the drawings are not drawn to scale, but are only for the purpose of illustrating the principles of the present application. In order to facilitate the illustration and description of some parts of the present application, the corresponding parts in the drawings may be enlarged, that is, they may become larger than other components in the exemplary device actually manufactured according to the present application. In the drawings:

[0048] Figure 1 This is a first flow chart of an adaptive constrained federated learning method based on reinforcement learning in one embodiment of the present application.

[0049] Figure 2 This is a second flow chart of the adaptive constrained federated learning method based on reinforcement learning in one embodiment of the present application.

[0050] Figure 3 This is a schematic diagram of the execution architecture corresponding to the adaptive constrained federated learning method based on reinforcement learning in the video of an application example of this application.

[0051] Figure 4 Schematic diagram of the structure of an adaptive constrained federated learning device based on reinforcement learning in one embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail in conjunction with the embodiments and drawings. Here, the illustrative embodiments of this application and their descriptions are used to explain this application, but are not intended to limit this application.

[0053] It should also be noted here that in order to avoid obscuring the present application due to unnecessary details, the accompanying drawings only show structures and / or processing steps that are closely related to the scheme according to the present application, while other details that are not closely related to the present application are omitted.

[0054] It should be emphasized that the term "include / comprises" when used herein refers to the existence of features, elements, steps or components, but does not exclude the existence or addition of one or more other features, elements, steps or components.

[0055] It should also be noted that, unless otherwise specified, the term "connection" herein may refer not only to a direct connection but also to an indirect connection involving an intermediate.

[0056] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0057] In order to solve the problem of model performance degradation due to data heterogeneity between clients under the federated learning framework, the embodiments of the present application respectively provide an adaptive constrained federated learning method based on reinforcement learning, an adaptive constrained federated learning device based on reinforcement learning for executing the adaptive constrained federated learning method based on reinforcement learning, a federated learning system, an electronic device, a computer-readable storage medium and a computer program product, which combine reinforcement learning (RL) with federated learning (FL) and model the hyperparameter selection problem as a dynamic decision-making process. RL is used to make the best decision in various situations. It adaptively adjusts the correlation between the local model of each client and the global model in each round of local training to reduce the impact of data heterogeneity. Each client makes independent decisions without the need for additional data communication between the server and the client.

[0058] The details are described in detail through the following examples.

[0059] Based on this, the embodiment of the present application provides an adaptive constraint federated learning method based on reinforcement learning that can be implemented by an adaptive constraint federated learning device based on reinforcement learning, see Figure 1 The adaptive constrained federated learning method based on reinforcement learning specifically includes the following contents:

[0060] Step 100: Extract the global model weights and local model weights of the previous round from the tuple samples of the previous round pre-stored in the local experience pool, and determine the hyperparameters of the action of the current round based on the global model weights and local model weights of the previous round based on the local reinforcement learning agent unit.

[0061] In one or more embodiments of the present application, the adaptive constrained federated learning device based on reinforcement learning can be a functional module in a client device, and the client device can be simply referred to as a client. In the adaptive constrained federated learning device based on reinforcement learning of each client device, there is a reinforcement learning agent unit, experience pool, local data and local model uniquely corresponding to the client.

[0062] It's understandable that reinforcement learning is a machine learning method that aims to learn optimal strategies through the interaction between an agent and its environment to maximize cumulative rewards. In reinforcement learning, the "reinforcement learning agent" refers to the agent in reinforcement learning. An agent is an entity that can perform actions, also known as an agent. Based on its current state, the agent selects an action, enters the next state after executing that action, and receives a corresponding reward. The agent's goal is to find a strategy that maximizes the total reward received in the long term through continuous trial and error. The combination of federated learning and reinforcement learning has emerged as a research direction in recent years. Learning strategies through reinforcement learning can assist federated learning training.

[0063] In one or more embodiments of the present application, each client device and server used to execute the reinforcement learning-based adaptive constrained federated learning method together constitutes a federated learning system. The rounds mentioned in this application refer to each iteration round in the federated learning iteration number of the federated learning system. It is understood that for the current round, if there is a previous round, it means that the current round is not the initial iteration round. In other words, steps 100 to 400 provided in the embodiments of the present application are a general adaptive constrained federated learning process for non-initial rounds.

[0064] Federated learning is a distributed machine learning method that allows multiple participants (such as devices or institutions) to jointly train a global model without sharing their local data. In federated learning, each participant trains the model on local data and only sends the model parameters or gradients to a central server for aggregation, rather than uploading the data itself. This approach not only protects data privacy but also reduces the cost and risk of data transmission.

[0065] Step 200: Use each data sample in the local training set to train the local model corresponding to the global model weight of the previous round, and adaptively constrain the local model based on the hyperparameters during the training process to obtain the local model weight of the local model in the current round, and send the local model weight to the server in the federated learning system, so that the server aggregates the local model weights received in the current round to obtain the global model weight of the current round.

[0066] In step 200, adaptively constraining the local model refers to adaptively constraining the local model with respect to the global model (i.e., the global model weights of the previous round). The purpose of the adaptive constraint is to enable the client to dynamically adjust the local training update direction according to its own situation in a dynamic scenario, and to adaptively adjust the correlation between the local model of each client and the global model to reduce the impact of data heterogeneity.

[0067] In one or more embodiments of the present application, the model may be a classification model for image or speech classification or a prediction model for image, speech, text, etc., all models applicable to federated learning. In one example, the local data may be local image data, and each data sample in the local data is divided into a training set and a validation set, and the data samples include image data samples; the model includes: a classification model (that is, an image classification model), which is used to output a type recognition result of the image data sample according to the input image data sample. When the client device trains the local model, the data samples used are provided with type labels corresponding to each of the image data samples.

[0068] Correspondingly, the local model and global model mentioned in one or more embodiments of this application refer to different states of the same model. For example, a personalized classification model trained locally on a client device becomes the local model, while a classification model obtained by aggregating the weights of each personalized classification model on the server becomes the global model. Correspondingly, the local model weights include the weights of the local classifier head, and the global model weights include the weights of the global classifier head.

[0069] Step 300: Receive the global model weight of the current round sent by the server, and determine the reward of the current round based on the local reinforcement learning agent unit according to the global model weight and local model weight corresponding to the previous round and the current round respectively.

[0070] Step 400: Based on the global model weight and local model weight of the previous round, the action of the current round, the reward of the current round, and the global model weight and local model weight of the current round, a tuple sample of the current round is constructed and stored in the local experience pool.

[0071] In step 400, the global model weight and local model weight of the previous round t-1 are the state s of the current round t t , the global model weight and local model weight of the current round t are the state s of the next round t+1 t+1 Therefore, the next round needs to use the global model weight and local model weight of the current round to perform step 100, and the action a of the current round t t It is also the hyperparameter τ of the action in the current round t , the tuple sample also contains the reward r of the current round t , based on this, the tuple samples collected by each agent are (s t ,a t ,r t ,s t+1 ), t>1.

[0072] Specifically, for the i-th client device, the state s of the current round t of the i-th client device is t The global model weight of the i-th client device in the previous round t-1 and local model weights Composition, recorded as The state s of the i-th client device in the next round t+1 t+1 The global model weight of the i-th client device in the current round t and local model weights Composition, recorded as The action a of the current round t of the i-th client device t Recorded as The reward r of the current round t of the i-th client device t Denoted as r i t , then for the i-th client device, the tuple sample of the t-th round obtained in step 400 is The tuple sample of the previous round t-1 obtained in step 100 is In one or more embodiments of the present application, action a may also be represented by γ.

[0073] In addition, the local experience pool of the i-th client device can be recorded as P i , in the tth round, the local experience pool of the i-th client device can record At round t, the collective experience pool of all agents is expressed as Wherein, k is the total number of client devices currently participating in federated learning in the federated learning system.

[0074] It should be noted that, since the action selection of this application is in a continuous space, this application adopts the PPO (Proximal Policy Optimization) algorithm when selecting the reinforcement learning method. By introducing the truncation mechanism, PPO avoids performance fluctuations caused by drastic changes in strategy, thereby improving the stability and convergence speed of the training process. In addition, due to the nature of the task of this application, the data generated for the agent in federated learning is relatively scarce. The PPO algorithm can be effectively trained under limited experience replay and low data requirements. These advantages become more significant when combined with federated learning. Overall, in the context of heterogeneous data and dynamic environments, PPO can prevent training instability due to limited data and poor performance problems that other reinforcement learning algorithms may encounter.

[0075] From the above description, it can be seen that the adaptive constrained federated learning method based on reinforcement learning provided in the embodiment of the present application can solve the problem of data heterogeneity between clients under the federated learning framework, which leads to the inconsistency between the data distribution of each client and the data distribution of the test domain, resulting in the degradation of model performance. It can adaptively adjust the correlation between the local model of each client and the global model to reduce the impact of data heterogeneity, and can effectively improve the reliability and effectiveness of model training, thereby improving model performance; at the same time, each client has a local reinforcement learning agent to make independent decisions, so that each client can fully and autonomously adapt to the training conditions and make decisions, without the need for additional data communication between the server and the client, thereby effectively reducing the resource consumption and time cost of data communication between the server and the client.

[0076] In order to further improve the application effectiveness and reliability of adaptive constrained federated learning based on reinforcement learning, in an adaptive constrained federated learning method based on reinforcement learning provided in an embodiment of the present application, see Figure 2 The adaptive constrained federated learning method based on reinforcement learning further includes the following contents before step 100:

[0077] Step 010: Build a local experience pool;

[0078] Step 020: Initialize the action model and evaluation model corresponding to the reinforcement learning agent unit.

[0079] Specifically, the action model is the Actor model in reinforcement learning. For the i-th client device, its action model is denoted as θ i The evaluation model is the Critic model in reinforcement learning. For the i-th client device, its evaluation model is recorded as Q i .

[0080] Step 030: Divide each data sample in the local data into a training set and a validation set.

[0081] Specifically, local data is the local data set. For the i-th client device, its local data set is denoted as D i , and divide it into training set and validation set

[0082] And, step 040: receiving initial global model parameters sent by the server in the federated learning system.

[0083] The initial global model parameters are recorded as

[0084] Step 050: In the initial round, the local model corresponding to the initial global model parameters is trained based on local data and preset initial hyperparameters to obtain the local model weight of the initial round, and the local model weight is sent to the server in the federated learning system, so that the server aggregates the local model weights received in the initial round to obtain the global model weight of the initial round.

[0085] Specifically, taking a single client as an example, client i receives the In the initial global communication round, since there is no personalized model available to construct the state for action selection, client i uses the preset initial hyperparameters Perform local training. Loss during local training The function is:

[0086]

[0087] Among them, (x i ,y i )∈D i , represents the local model that is continuously updated in the current global round, and L(·,·,·) represents the cross-extraction loss function.

[0088] Step 060: Receive the global model weight of the initial round sent by the server.

[0089] For client i, in the first round, the global model weight of the initial round sent by the server is recorded as

[0090] In order to further improve the effectiveness and reliability of determining the hyperparameters of the action in the current round, in an adaptive constraint federated learning method based on reinforcement learning provided in an embodiment of the present application, see Figure 2 Step 100 in the adaptive constrained federated learning method based on reinforcement learning specifically includes the following:

[0091] Step 110: Check whether the local experience pool contains tuple samples from the previous round. If so, extract the global model weights and local model weights of the previous round as the state of the previous round from the tuple samples of the previous round.

[0092] In step 110, if the local experience pool does not contain tuple samples from the previous round, it means that the current round may be the initial round or an early round (such as the second round) that has not yet generated tuple samples. It is necessary to use the preset initial hyperparameters in the manner of steps 050 and 060. Perform local model training.

[0093] Step 120: Based on the action model in the reinforcement learning agent unit, hyperparameters for the action of the current round are generated according to the global model weights and local model weights of the previous round.

[0094] In one example, after receiving the global model, the local client can observe the local model classifier head and the global classifier head in the previous round of training. i ,Q i ) uses this information to select the local training hyperparameter τ, which controls the extent to which the local model training is constrained by the global model.

[0095] In order to emphasize the performance comparison of the model in consecutive rounds, reduce the impact of rapid performance improvement when the model quickly fits the data in the early stage of training, and further improve the effectiveness and reliability of determining the hyperparameters of the action in the current round, in an adaptive constrained federated learning method based on reinforcement learning provided in an embodiment of the present application, see Figure 2 Step 300 in the adaptive constrained federated learning method based on reinforcement learning specifically includes the following:

[0096] Step 310: Obtain the accuracy of the global model weight and the local model weight corresponding to the previous round and the current round on the validation set;

[0097] Step 320: Determine the reward for the current round based on the preset weight change parameters and the accuracy of the global model weights and local model weights corresponding to the previous round and the current round on the validation set; wherein the weight change parameters are used to determine the round value corresponding to the current round, and different round values ​​correspond to different weight change parameters.

[0098] Specifically, the reward is the basis for the agent to evaluate the quality of the action. In the specific task of this application, the effectiveness of τ is most clearly reflected on the validation set. Therefore, in the method of this application, the reward of the tth round corresponding to the i-th client device is Set to:

[0099]

[0100] Among them, t represents the current communication round, represents the accuracy of client i on its validation set in round t, represents the accuracy of client i on its validation set in round t-1, acc t Indicates the accuracy of the global model on the t-th round validation set, acc t-1 represents the accuracy of the global model on the validation set in the t-1th round, and β is a predefined weight change parameter.

[0101] In this application's reward setup, we emphasize comparing model performance across consecutive rounds. Therefore, we primarily focus on the differences in performance between consecutive rounds when determining rewards. Furthermore, we hope to learn strategies that not only improve model performance during local training but also prevent over-personalization of models, which can lead to poor generalization after aggregation. To this end, we incorporate global model performance into reward calculations.

[0102] Considering the model training process, which begins with random weights and rapidly learns the characteristics and distribution of the data, this application observes significant performance gains in early iterations. As training progresses, the model may approach the optimal solution, resulting in a slower rate of performance improvement. Therefore, this application introduces another factor into the reward calculation. This factor mitigates the impact of rapid performance gains in the early stages of training, when the model is quickly fitting the data, while increasing the impact of this application's strategy on performance during the relatively stable phase of model training.

[0103] In order to further improve the application effectiveness and reliability of the reinforcement learning agent unit, in the adaptive constraint federated learning method based on reinforcement learning provided in the embodiment of the present application, see Figure 2 The step 400 in the adaptive constrained federated learning method based on reinforcement learning further specifically includes the following contents:

[0104] Step 500: If the number of the tuple samples in the local experience pool has reached a preset threshold, multiple tuple samples other than the tuple samples in the previous round in the local experience pool are extracted, and the reinforcement learning agent unit is trained based on the extracted tuple samples.

[0105] The quantity threshold may be manually set in advance, and a smaller value may be selected to increase the training frequency of the reinforcement learning agent unit for application reliability. For example, it may be set to any positive integer between 3 and 10.

[0106] In order to achieve the sharing of samples among multiple agents according to data distribution, the efficiency of reinforcement learning training is improved without compromising privacy or increasing communication costs, so as to further improve the application effectiveness and reliability of reinforcement learning agent units, in an adaptive constrained federated learning method based on reinforcement learning provided in an embodiment of the present application, see Figure 2 The step 500 in the adaptive constrained federated learning method based on reinforcement learning further specifically includes the following contents:

[0107] Step 600: Receive the index of similar client devices sent by the server in the federated learning system, wherein the server searches for corresponding similar client devices for each client device that sends the local model weight according to the similarity between each local model weight received in the current round, and sends the unique identifier of the similar client device corresponding to each client device to each client device.

[0108] Step 700: Based on the index of the similar client device, copy a plurality of tuple samples from the experience pool of the similar client device.

[0109] Step 800: After training the local reinforcement learning agent unit based on each of the tuple samples extracted from the local experience pool, train the local reinforcement learning agent unit based on each of the tuple samples copied from the experience pool of the similar client device.

[0110] Specifically, in each round, this application uses the local model weight as the judgment criterion. For each client i, this application searches for the most similar client j whose update direction is highly consistent with that of client i:

[0111]

[0112] Client i uses its local experience pool After the update, the experience pool of similar client j will also be used Make additional updates.

[0113] In order to further illustrate the various embodiments of the above-mentioned adaptive constrained federated learning method based on reinforcement learning, the present application also provides a specific application example of the adaptive constrained federated learning method based on reinforcement learning, wherein the model specifically adopts a classification model (i.e., an image classification model), the local model weight specifically adopts the weight of the local classifier head (i.e., the weight of the bias classifier head of the client), and the global model weight specifically adopts the weight of the global classifier head (i.e., the classifier head weight of the aggregated global model) as an example for illustration.

[0114] The execution architecture of this application example is as follows Figure 3 As shown in the figure, all clients have the same structure, feature extraction layer, classifier layer and private agent. The feature extraction layer extracts image features for classifier recognition and update. The update direction is determined by the reinforcement learning agent strategy, and the agent is continuously updated during the training process. The server aggregates the model weights after local training and calculates the closest neighbor client for each client to assist in training. In federated learning, it is generally composed of 1 server and K clients, each client has a dataset where N i Indicates the amount of data owned by the i-th client.

[0115] The adaptive constrained federated learning method based on reinforcement learning provided in this specific application example specifically includes the following:

[0116] (1) Strengthening the personalized configuration of federated learning agents

[0117] Status: It is well known that during local training in federated learning, the model acquires information about the data distribution. This is a prerequisite for federated learning to achieve a universal global model without sharing data. Experimental evidence shows that non-independent and identically distributed (non-IID) data significantly impacts the classifier layer. Differences in data distribution between different clients lead to bias in models trained based on the same global model initialization. After training on different data, the classifier heads tend to disperse in the vector space, resulting in poor generalization of the aggregated model.

[0118] On the contrary, this characteristic of the classifier head indicates that they tend to learn information about the data distribution, because the deviations of different classifier heads come from the changes in different data distributions. Therefore, this application example uses the weights of the classifier head as the state: in represents the weight of the client's bias classifier head in the tth round of global training of federated learning, Represents the classifier head weights of the global model after the previous round of model aggregation, which serve as the initialization of the classifier head in round t. This state configuration not only avoids taking the entire model weight as a parameter, thereby reducing the burden of reinforcement learning, but also enables the agent to more directly observe the main reason for the degradation of federated learning performance due to non-independent and identically distributed (non-IID) data.

[0119] Action: After receiving the global model, the local client observes the local model classifier head and the global classifier head in the previous training round. The agent uses this information to select the hyperparameter τ for local training, which controls the degree to which the local model training is constrained by the global model.

[0120] Reward: Reward is the basis for the agent to evaluate the quality of the action. In the specific task of this application example, the effectiveness of τ is most clearly reflected on the validation set. Therefore, in the method of this application example, this application example sets the reward as:

[0121]

[0122] Among them, t represents the current communication round, represents the accuracy of client i on its validation set in round t, represents the accuracy of client i on its validation set in round t-1, acc t Indicates the accuracy of the global model on the t-th round validation set, acc t-1represents the accuracy of the global model on the validation set in round t-1, and β is a predefined weight change parameter. In the reward setting of this application example, this application example emphasizes comparing the performance of the model in consecutive rounds. Therefore, this application example focuses on the difference in performance between consecutive rounds when determining rewards. Furthermore, this application example hopes to learn a policy that not only improves the performance of the model in local training but also avoids excessive model personalization, which can lead to poor generalization ability after aggregation. To this end, this application example incorporates the performance of the global model into the reward calculation.

[0123] Considering the model training process, which begins with random weights and rapidly learns the characteristics and distribution of the data, this application example observes significant performance gains in early iterations. As training progresses, the model may approach the optimal solution, causing performance gains to slow. Therefore, this application example introduces another factor into the reward calculation. This factor mitigates the impact of rapid performance gains in the early stages of training, when the model is rapidly fitting the data, while increasing the impact of this application example's strategy on performance during the more stable phase of model training.

[0124] (2) Experience Pool Collaboration Based on Data Distribution

[0125] In the specific task setting of this application example, each client has a private agent responsible for decision making and evaluation of actions. The agent only needs to pay attention to its own state and build tuples (s t ,a t ,r t ,s t+1 ) and put it into the experience pool for experience replay. Unlike agents that only exist on the server side, it mainly focuses on the distribution of local data and the update direction related to data distribution information, but this will face a challenge: sparse sample availability. In this case, each individual agent is similar to a client in federated learning, and the samples in the experience replay pool represent the data held by the client. The tuple includes the local classifier head and the global classifier head, and sharing samples does not leak privacy. However, sharing all samples is similar to training K identical server-side agents in federated learning, which wastes precious computing resources. Therefore, this application proposes an example of experience pool sharing based on data distribution:

[0126] During the client training phase of federated learning, each agent collects tuples (s t ,a t ,r t ,s t+1 ) and add them to the experience pool In the tth round, the collective experience pool of all agents is expressed as In each round, this application example uses the weight of the classifier head as the judgment criterion. For each client i, this application example finds the most similar client j whose update direction is highly consistent with client i:

[0127]

[0128] Client i uses its local experience pool After the update, the experience pool will also be used Make additional updates.

[0129] (3) Experience pool collaboration based on data distribution

[0130] 1. Initialization: Before training begins, each client builds an experience pool P i , initialize the Actor model θ i and Critic model Q i , load the local dataset D i and divide it into training set and validation set At the same time, the server initializes the global model to obtain the initial global model parameters

[0131] 2. Local training: Taking a single client as an example, client i receives the data sent by the server. In the initial global communication round (round 0), since there is no personalized model available to build the state for action selection, the client uses the preset hyperparameters Perform local training. The loss function in the local training phase is:

[0132]

[0133] Among them, (x i ,y i )∈D i , represents the local model that is continuously updated in the current global round, and l(·,·,·) represents the cross-extraction loss function.

[0134] When the global communication round is greater than 1, the agent of the reinforcement learning module starts to participate in local training and builds tuples using historical elements. and add them to P i ,in Represents the weight of the classifier head. When P i When it is full, some samples will be extracted from it to update the agent (θ i ,Q i ). The agent will then j To extract samples from θ to obtain knowledge similar to the data distribution. Once all updates are completed, θi Select τ for local model training. The training loss function is almost the same as that of epoch 0.

[0135]

[0136] At the end of local training, we get

[0137] 3. Model aggregation: Upload to the server and aggregate In this phase, the server uses the EPC module to calculate the similarity between clients and assigns the index of the most similar client to each client.

[0138] 4. Model Evaluation: exist Verify on Verify on the pre-split global validation set and get acc t , each client calculates the reward separately. Steps 2 to 4 are repeated cyclically until the total number of communication rounds T is reached.

[0139] That is to say, this application example proposes an adaptive constrained federated learning framework based on reinforcement learning, which integrates reinforcement learning and federated learning. Each client has its own private agent to make decisions, so that each client can fully adapt to training conditions and make decisions autonomously. This application example also proposes a method based on indirect data distribution, called Experience Pool Collaboration (EPC) method. Multiple agents share samples according to data distribution, which improves the efficiency of reinforcement learning training without compromising privacy or increasing communication costs.

[0140] In summary, the adaptive constrained federated learning method based on reinforcement learning provided by the embodiments and application examples of the present application allows the client to dynamically select the local update direction according to the actual situation of each round of local training, avoid the forgetting problem that may exist in model training, and improve the global image classification performance of the federated learning client model; utilize the strong correlation between the classifier and the data distribution, calculate the closest client in each round for experience sample sharing, and improve the correctness of the reinforcement learning strategy.

[0141] The present application also provides a reinforcement learning-based adaptive constrained federated learning device for executing all or part of the reinforcement learning-based adaptive constrained federated learning method, see Figure 4 The adaptive constraint federated learning device based on reinforcement learning specifically includes the following contents:

[0142] The reinforcement learning action determination module 10 is used to extract the global model weights and local model weights of the previous round from the tuple samples of the previous round pre-stored in the local experience pool, and based on the local reinforcement learning agent unit, determine the hyperparameters of the action of the current round according to the global model weights and local model weights of the previous round.

[0143] The adaptive constrained federated learning module 20 is used to use each data sample in the local training set to train the local model corresponding to the global model weight of the previous round, and adaptively constrain the local model based on the hyperparameters during the training process to obtain the local model weight of the local model in the current round, and send the local model weight to the server in the federated learning system so that the server aggregates the local model weights received in the current round to obtain the global model weight of the current round.

[0144] The reinforcement learning reward determination module 30 is used to receive the global model weight of the current round sent by the server, and based on the local reinforcement learning agent unit, determine the reward of the current round according to the global model weight and local model weight corresponding to the previous round and the current round respectively.

[0145] The experience pool storage module 40 is used to construct a tuple sample of the current round and store it in the local experience pool based on the global model weight and local model weight of the previous round, the action of the current round, the reward of the current round, and the global model weight and local model weight of the current round.

[0146] The embodiment of the adaptive constrained federated learning device based on reinforcement learning provided in this application can be specifically used to execute the processing flow of the embodiment of the adaptive constrained federated learning method based on reinforcement learning in the above-mentioned embodiment. Its functions are not described in detail here, and reference can be made to the detailed description of the embodiment of the adaptive constrained federated learning method based on reinforcement learning.

[0147] The portion of the reinforcement learning-based adaptive constrained federated learning device that performs the reinforcement learning-based adaptive constrained federated learning can be completed in the client device. The specific selection can be based on the processing capabilities of the client device and the limitations of the user's usage scenario. This application is not limited to this. If all operations are completed in the client device, the client device may also include a processor for the specific processing of the reinforcement learning-based adaptive constrained federated learning.

[0148] The client device may include a communication module (i.e., a communication unit) that can establish a communication connection with a remote server to implement data transmission with the server. The server may include a server on the task scheduling center side, and in other implementation scenarios, may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, a server cluster consisting of multiple servers, or a server structure of a distributed device.

[0149] The server and the client device may communicate using any suitable network protocol, including network protocols that have not yet been developed as of the filing date of this application. Examples of such network protocols include TCP / IP, UDP / IP, HTTP, and HTTPS. Furthermore, examples of such network protocols include RPC (Remote Procedure Call Protocol) and REST (Representational State Transfer) protocols, which are used on top of the aforementioned protocols.

[0150] From the above description, it can be seen that the adaptive constrained federated learning device based on reinforcement learning provided in the embodiment of the present application can solve the problem of data heterogeneity between clients under the federated learning framework, which leads to the inconsistency between the data distribution of each client and the data distribution of the test domain, resulting in the degradation of model performance. It can adaptively adjust the correlation between the local model of each client and the global model to reduce the impact of data heterogeneity, and can effectively improve the reliability and effectiveness of model training, thereby improving model performance; at the same time, each client has a local reinforcement learning agent to make independent decisions, so that each client can fully and autonomously adapt to the training conditions and make decisions, without the need for additional data communication between the server and the client, thereby effectively reducing the resource consumption and time cost of data communication between the server and the client.

[0151] This application also provides an adaptive constrained federated learning system based on reinforcement learning, which specifically includes the following contents:

[0152] A server and each client device communicating with the server;

[0153] Each of the client devices is used for the adaptive constrained federated learning method based on reinforcement learning described in the aforementioned embodiment.

[0154] The present application also provides an electronic device that may include a processor, a memory, a receiver, and a transmitter. The processor is configured to execute the adaptive constraint federated learning method based on reinforcement learning described in the above embodiment. The processor and the memory may be connected via a bus or other means, with bus connection being used as an example. The receiver may be connected to the processor and the memory via a wired or wireless means.

[0155] The processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0156] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer executable programs, and modules, such as the program instructions / modules corresponding to the adaptive constrained federated learning method based on reinforcement learning in the embodiments of this application. The processor executes the non-transitory software programs, instructions, and modules stored in the memory to perform various processor functions and data processing, thereby implementing the adaptive constrained federated learning method based on reinforcement learning in the above method embodiments.

[0157] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory may optionally include a memory remotely located relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0158] The one or more modules are stored in the memory, and when executed by the processor, perform the adaptive constrained federated learning method based on reinforcement learning in the embodiment.

[0159] In some embodiments of the present application, the user equipment may include a processor, a memory and a transceiver unit, and the transceiver unit may include a receiver and a transmitter. The processor, memory, receiver and transmitter may be connected through a bus system. The memory is used to store computer instructions, and the processor is used to execute the computer instructions stored in the memory to control the transceiver unit to send and receive signals.

[0160] As an implementation method, the functions of the receiver and transmitter in this application can be considered to be implemented through a transceiver circuit or a dedicated transceiver chip, and the processor can be considered to be implemented through a dedicated processing chip, a processing circuit or a general-purpose chip.

[0161] As another implementation method, it is possible to use a general-purpose computer to implement the server provided in the embodiments of the present application. That is, the program code for implementing the functions of the processor, receiver, and transmitter is stored in a memory, and the general-purpose processor implements the functions of the processor, receiver, and transmitter by executing the code in the memory.

[0162] The present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the aforementioned adaptive constrained federated learning method based on reinforcement learning. The computer-readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the art.

[0163] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the adaptive constrained federated learning method based on reinforcement learning.

[0164] It should be understood by those skilled in the art that the various exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is specifically performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application. When implemented in hardware, it can be, for example, an electronic circuit, an application-specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of this application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0165] It should be understood that the present application is not limited to the specific configurations and processes described above and illustrated in the figures. For the sake of brevity, a detailed description of known methods is omitted here. In the above embodiments, several specific steps are described and illustrated as examples. However, the method process of the present application is not limited to the specific steps described and illustrated. Those skilled in the art can make various changes, modifications, and additions, or change the order of the steps after understanding the spirit of the present application.

[0166] In this application, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or replace features of other embodiments.

[0167] The above description is merely a preferred embodiment of the present application and is not intended to limit the present application. Those skilled in the art will appreciate that various modifications and variations of the present embodiment are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present application shall be included within the scope of protection of the present application.

Claims

1. An adaptive constrained federated learning method based on reinforcement learning, characterized in that: include: Extract the global model weights and local model weights of the previous round from the tuple samples pre-stored in the local experience pool, and determine the hyperparameters of the action of the current round based on the global model weights and local model weights of the previous round based on the local reinforcement learning agent unit; Using each data sample in the local training set to train the local model corresponding to the global model weight of the previous round, and adaptively constraining the local model based on the hyperparameters during the training process to obtain the local model weight of the local model in the current round, and sending the local model weight to the server in the federated learning system, so that the server aggregates the local model weights received in the current round to obtain the global model weight of the current round; Receiving the global model weight of the current round sent by the server, and determining the reward of the current round based on the local reinforcement learning agent unit according to the global model weight and local model weight corresponding to the previous round and the current round respectively; According to the global model weight and the local model weight of the previous round, the action of the current round, the reward of the current round, and the global model weight and the local model weight of the current round, a tuple sample of the current round is formed and stored in the local experience pool; If the number of the tuple samples in the local experience pool has reached a preset number threshold, extracting multiple tuple samples from the local experience pool other than the tuple samples in the previous round, and training the reinforcement learning agent unit based on each of the extracted tuple samples; Receiving an index of similar client devices sent by a server in the federated learning system, wherein the server searches for a corresponding similar client device for each client device that has sent the local model weight according to the similarity between the local model weights received in the current round, and sends a unique identifier of the similar client device corresponding to each client device to each client device; Based on the index of the similar client device, copying a plurality of the tuple samples from the experience pool of the similar client device; After training the local reinforcement learning agent unit based on each of the tuple samples extracted from the local experience pool, the local reinforcement learning agent unit is trained based on each of the tuple samples copied from the experience pool of the similar client device.

2. The adaptive constraint federated learning method based on reinforcement learning according to claim 1, characterized in that: Before extracting the global model weight and the local model weight of the previous round from the tuple samples of the previous round pre-stored in the local experience pool, the method further includes: Build a local experience pool; Initialize the action model and evaluation model corresponding to the reinforcement learning agent unit; Divide each data sample in the local data into a training set and a validation set respectively; and, receiving initial global model parameters sent by a server in the federated learning system; In the initial round, the local model corresponding to the initial global model parameters is trained based on the local data and the preset initial hyperparameters to obtain the local model weights of the initial round, and the local model weights are sent to the server in the federated learning system so that the server aggregates the local model weights received in the initial round to obtain the global model weights of the initial round; Receive the global model weight of the initial round sent by the server.

3. The adaptive constraint federated learning method based on reinforcement learning according to claim 2, characterized in that: The method extracts the global model weight and the local model weight of the previous round from the tuple samples of the previous round pre-stored in the local experience pool, and determines the hyperparameters of the action of the current round based on the global model weight and the local model weight of the previous round based on the local reinforcement learning agent unit, including: Searching the local experience pool to see if it contains a tuple sample from the previous round, and if so, extracting the global model weight and local model weight of the previous round as the state of the previous round from the tuple sample from the previous round; Based on the action model in the reinforcement learning agent unit, hyperparameters of the action in the current round are generated according to the global model weight and the local model weight of the previous round.

4. The adaptive constraint federated learning method based on reinforcement learning according to claim 2, characterized in that: The locally based reinforcement learning agent unit determines the reward for the current round according to the global model weight and the local model weight corresponding to the previous round and the current round, including: Obtain the accuracy of the global model weight and the local model weight corresponding to the previous round and the current round on the validation set; The reward for the current round is determined based on the preset weight change parameters and the accuracy of the global model weights and local model weights corresponding to the previous round and the current round on the validation set; wherein the weight change parameters are used to be determined based on the round value corresponding to the current round, and different round values ​​correspond to different weight change parameters.

5. The adaptive constrained federated learning method based on reinforcement learning according to any one of claims 1 to 4, characterized in that: The data samples include: image data samples; The local model weights include: weights of local classifier heads; The global model weight includes: the weight of the global classifier head.

6. An adaptive constrained federated learning device based on reinforcement learning, characterized in that: include: A reinforcement learning action determination module is used to extract the global model weights and local model weights of the previous round from the tuple samples of the previous round pre-stored in the local experience pool, and determine the hyperparameters of the action of the current round based on the global model weights and local model weights of the previous round based on the local reinforcement learning agent unit; an adaptive constrained federated learning module, configured to train a local model corresponding to the global model weight of the previous round using each data sample in the local training set, and adaptively constrain the local model based on the hyperparameters during the training process to obtain a local model weight of the local model in the current round, and send the local model weight to a server in the federated learning system so that the server aggregates the local model weights received in the current round to obtain a global model weight of the current round; a reinforcement learning reward determination module, configured to receive the global model weights for the current round sent by the server, and determine the reward for the current round based on the global model weights and local model weights corresponding to the previous round and the current round, respectively, based on the local reinforcement learning agent unit; An experience pool storage module, configured to construct a tuple sample for the current round based on the global model weight and the local model weight of the previous round, the action of the current round, the reward of the current round, and the global model weight and the local model weight of the current round, and store the tuple sample in the local experience pool; The adaptive constraint federated learning device based on reinforcement learning is further used to perform the following: If the number of the tuple samples in the local experience pool has reached a preset number threshold, extracting multiple tuple samples from the local experience pool other than the tuple samples in the previous round, and training the reinforcement learning agent unit based on each of the extracted tuple samples; Receiving an index of similar client devices sent by a server in the federated learning system, wherein the server searches for a corresponding similar client device for each client device that has sent the local model weight according to the similarity between the local model weights received in the current round, and sends a unique identifier of the similar client device corresponding to each client device to each client device; Based on the index of the similar client device, copying a plurality of the tuple samples from the experience pool of the similar client device; After training the local reinforcement learning agent unit based on each of the tuple samples extracted from the local experience pool, the local reinforcement learning agent unit is trained based on each of the tuple samples copied from the experience pool of the similar client device.

7. A federated learning system, characterized in that: include: A server and each client device communicating with the server; Each of the client devices is used to execute the adaptive constrained federated learning method based on reinforcement learning as described in any one of claims 1 to 5.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the adaptive constrained federated learning method based on reinforcement learning is implemented as described in any one of claims 1 to 5.

Images