Personalized federal learning method and system with energy-time perception based on deep reinforcement learning

Abstract

The invention belongs to but is not limited to the technical field of artificial intelligence, and discloses a personalized federated learning method with energy-time perception based on deep reinforcement learning, which introduces a personalized information layer into a local model to better adapt to heterogeneous data. Besides, in equipment heterogeneous, data heterogeneous and complex communication environments, an FL system framework is remodeled, a Markov decision process is constructed, and a personalized FL-DDPG algorithm based on a DDPG deep reinforcement learning algorithm is provided. According to the algorithm, clients participating in training in each round are screened, so that the convergence speed is increased and personalized information of users is protected while time and energy cost are constrained. Experimental results verify the convergence of Per-FL-DDPG, and contrast experiments show that the algorithm effectively controls time and energy consumption while improving the convergence speed, and ensures long-term stable operation of the system.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of artificial intelligence, and in particular relates to a personalized federated learning method and system with energy-time perception based on deep reinforcement learning. BACKGROUND

[0002] In recent years, the rapid progress of 5G technology has significantly improved the speed and quality of wireless communication, promoting the widespread application of Internet of Things (IoT) devices in daily life. At the same time, the commercialization of artificial intelligence (AI) and big data technology has accelerated the speed of data generation, but this has also made the collection of raw data increasingly complex. Privacy protection, legal regulations, and commercial factors have further exacerbated the difficulties of data acquisition and utilization. As the amount and quality of data increase, the complexity of the collection process also rises, and traditional big data processing methods have been unable to cope with the current challenges. Therefore, there is an urgent need for a new data processing solution to address the increasingly serious privacy protection and data analysis problems. In this context, federated learning (FL) as an innovative distributed machine learning technology has gradually attracted widespread attention.

[0003] To address data privacy and security challenges, Google proposed a FL method in 2017. With the popularity of mobile devices and IoT devices, users generate a large amount of data, which often contains sensitive information, and users are usually reluctant to upload such private data. Directly transmitting data to a central server may pose a risk of data leakage. The FL method allows clients to train locally using their own data, and only needs to upload model parameters after completion, without sending raw data, thus achieving effective data utilization while protecting privacy. However, traditional FL methods mainly focus on commonalities among users, relying on the accuracy of the global model to evaluate the performance of each round of FL, and all clients use the same model structure and parameters. Although the model can capture the individual characteristics of the clients during training, these individual characteristics may be averaged out after the server aggregates the model, making it unsuitable for scenarios that require personalization, which is a major challenge for the distribution of heterogeneous data clients.

[0004] In practical applications of FL, numerous challenges are faced due to the diversity of devices and data and the changes in complex environments. First, the system configurations of clients differ significantly, including CPU frequency, transmission power, and battery power, which have a direct impact on computing power and communication efficiency. Second, the running environments and task processing methods of different clients vary, resulting in significant differences in the size and independent and identically distributed (IID) degree of generated data. In particular, in traditional FL, non-IID data distribution can seriously hinder model performance and even lead to non-convergence. In addition, traditional FL randomly selects clients for training, which can select devices with poor performance or unstable channel quality, significantly increasing the time and energy consumption of training and data transmission. This random selection can lead to excessive consumption of resources, affecting the overall efficiency and sustainability of the system. For data heterogeneity, although traditional FL may perform well in the IID case, the effectiveness of model training will decrease significantly when the IID degree differs greatly among clients. Therefore, relying solely on the accuracy of the global model to evaluate the effectiveness of FL training is insufficient.

[0005] In view of the above analysis, the existing technical problems in the prior art are that traditional FL does not fully consider the personalized needs brought about by data heterogeneity and differences in client task processing. Current researches mainly focus on designing client selection mechanisms to accelerate training under non-IID conditions, but the consideration of computational heterogeneity and environmental changes is still insufficient. SUMMARY

[0006] To solve the problems in the prior art, the present application provides a personalized federated learning method and system with energy-time awareness based on deep reinforcement learning.

[0007] The present application is implemented as follows: a personalized federated learning method with energy-time awareness based on deep reinforcement learning, comprising the following steps:

[0008] S1, based on the device heterogeneity of clients, data heterogeneity, and complex environmental conditions, combining the data types in the FL task and the neural network adaptation, the entire FL scene is re-modeled, including the local client computing model, the upload model network model, and the personalized user accuracy model; specifically:

[0009] S11, a scenario for FL by a central server installed near a base station and a plurality of different clients is established, the central server and each client device are equipped with a single-core CPU for computing tasks and training models, a radio frequency transmitting and receiving device is used to communicate device information, and a certain storage space is used to store local data sets and task information;

[0010] S12, a new personalized local user model structure driven by a batch normalization layer (BNL) is established, a personalized user accuracy model is established to measure the performance of FL operation in a more differentiated real scene, and a dynamic time-varying channel model is established for model network transmission;

[0011] S2, an energy-time-aware personalized FL basic framework is designed based on the local client computing model, the uploaded model network model and the personalized user accuracy model, the system is modeled according to the framework, and the mathematical model of the entire system is obtained, under the framework, at the beginning of each communication round, the central server will select appropriate clients to perform this round of FL;

[0012] S3, based on the energy-time-aware personalized FL basic framework in S2, the system parameters of each client difference, the data distribution difference and the channel state difference are jointly considered, the energy consumed by the local client for local training and model uploading, the time spent and the global average personalized user accuracy under a specific communication round are calculated, and a multi-optimization problem is formulated according to the calculated values, so as to improve the convergence speed and protect the user personalized information while constraining the time cost and energy cost, and the multi-optimization objective function is represented as follows:

[0013]

[0014] wherein, GAPU (k) represents the global average personalized user accuracy of the kth round, E (k) represents the energy cost consumed by each client for model training and model parameter uploading in the kth round, T (k) represents the time cost consumed in the kth round of FL, , , , , , represents the weight of the three elements in the multi-optimization objective function, and is greater than 0, the three coefficients are to adjust the weight of the three key factors in the objective function, so as to reach the same order of magnitude or similar values;

[0015] S4, the entire system framework is modeled into a Markov process, a reasonable state and action space is designed, and a reasonable reward function is designed by using the objective function;

[0016] S5, the DDPG algorithm of deep reinforcement learning is used to optimize the objective function, and the convergence speed is improved while constraining the time cost and energy cost, and the user personalized information is protected.

[0017] Furthermore, in step S2, the entire energy-time-aware personalized FL framework consists of a central server and several clients. Both the central server and clients are equipped with single-core CPUs. The computing resources and energy costs of the central server are far greater than those of the framework clients. Therefore, the central server... Define, three-dimensional coordinates are The client set consists of express, , This represents the number of clients, with each client's coordinates as follows: Therefore, the distance between each client and the central server is represented as: .

[0018] Furthermore, in a personalized FL framework with energy-time awareness, the central server checks the client set before each round of training. The client selection process involves filtering clients within the system to choose a certain number with superior system parameters and good communication channel quality to participate in this round of FL. After the central server selects suitable clients, it only distributes the global model to the selected clients. Once these clients receive the global model, they reconstruct it using their own personalized information layer. The clients then use the reconstructed local model to extract data from their local database for local training. The size of each data point is defined by... express, It will vary depending on the data type. The number of batches required for each round of FL is represented by [the following]. Indicates the first The client in the first The number of CPU cycles per second (cycle / s) for the round; the number of CPU cycles for the client is dynamically changing, in the first round... Round The local training time for each client is:

[0019]

[0020] in This indicates the number of epochs of local training on the client side. This indicates the number of CPU cycles required for the client to train one bit of data;

[0021] Therefore, in the first Round The energy cost consumed by each client for local training is expressed as:

[0022]

[0023] in Indicates the first The energy efficiency factor of each client is related to the CPU architecture of the client. After the client completes local training with the reconstructed model, the client will save the parameter information of the personalized information layer of the trained local model. After saving, the client will use its own radio frequency transmission module to send the model parameter information to the central server.

[0024] Furthermore, after the client completes local training, it transfers the trained model to the RF transmission module, which then transfers it to the server. The communication between the server and the client uses the Orthogonal Frequency Division Multiple Access (OFDMA) protocol. The channel state is static within a time slot but dynamic across different time slots. The average channel gain between the central server and each client is assumed to follow a path loss model.

[0025]

[0026] Indicates the first Average channel gain between each client and the central server This represents the antenna gain for each client. Indicates the signal carrier frequency. The path loss index represents the path loss; while the channel gain between each ground user and the central server represents the channel gain. Since they follow an independent and identically distributed Rayleigh distribution, according to Shannon's theorem, the central server and... A client in Wheel channel transmission rate:

[0027]

[0028] in Indicates the first The transmission power from each client to the central server, Indicates the first Channel bandwidth between each client and the central server Indicates the first The noise power spectral density between each client and the central server. The data size of the client model is defined as follows: The data size of different models is related to the adaptation to the dataset, so the model upload time is:

[0029]

[0030] Therefore, in the first Round The energy consumed by each client uploading a local model is:

[0031] .

[0032] Furthermore, after each client has uploaded its local model, the central server aggregates the received client models. Once aggregation is complete, one round of FL (Flexible Modeling) is finished. The performance of FL computation in more personalized real-world scenarios is measured by local user accuracy. Specifically, a personalized information layer is added to the model network of each client to store the personalized information of each client. The model parameters of this personalized information layer are stored locally and not aggregated with the global model. This ensures that while the global model is optimized, the personalized features of each client are maintained. Local user accuracy is represented as follows:

[0033]

[0034] in Indicates the first The client in the first The accuracy of local users in the round. Indicates the first The client in the first The model network of the wheel requires two inputs, namely the first wheel... The client in the first Rounds of data used for training and the The client in the first The personalized information layer model information of the wheel. Therefore, in the first... The global average local user precision is expressed as:

[0035]

[0036] Furthermore, in the In a round, the total energy cost of FL is represented as the sum of the energy used for computation and uploading by all participating clients. For synchronous FL, the time cost is represented as the maximum sum of the time costs used for computation and uploading by participating clients, in the round. The time and energy costs of using wheels for FL are expressed as follows:

[0037]

[0038]

[0039] in Indicates the first The round selects the set of clients to participate in FL.

[0040] Further, combined with the system parameter difference of each client, the data distribution difference and the channel state difference, the energy consumption, the time overhead and the global average user precision of the local client in the local training and model uploading in a specific communication round are calculated, and based on the calculation results, a multi-objective optimization problem is comprehensively formulated:

[0041]

[0042]

[0043]

[0044]

[0045]

[0046] wherein and represent the lower bound and the upper bound of the CPU frequency of the local client, and represent the lower bound and the upper bound of the power of the radio frequency transmission module of the client, and represent the upper and lower bounds of the bandwidth allocation of the client.

[0047] Further, each round of training in the step S4 needs to be screened by the central server before the round, so as to ensure the long-term stable operation of the system, and the energy-time-aware personalized FL framework is established as an MDP, and one MDP is formulated as wherein represents the environment state, including channel information, frequency information and personalized user accuracy information, represents the action of the agent, since the output action of DDPG is continuous, the method of probability sampling is used to map the continuous space to the discrete space, represents the reward brought by the one-step action of the agent, is a state transition function, is a discount factor of the reward . .

[0048] Further, the target function is optimized by using the deep reinforcement learning DDPG algorithm in the step S5, and the steps of the DDPG algorithm are as follows:

[0049] (1) initialize various parameters, FL environment, agent network, and build experience replay pool buffer;

[0050] (2) Select action, the policy network π generates an action and adds exploration noise, executes the action in the current state, obtains the reward r and the next state s';

[0051] (3) Update the experience pool, store the current state s, action a, reward r, and next state s' in the experience buffer buffer;

[0052] (4) Sample from the experience buffer, randomly sample a small batch (s, a, r, s') from the experience buffer;

[0053] (5) Calculate the target action, use the target policy network π_target to generate the target action a' of the next state s';

[0054] (6) Calculate the target Q value, use the target network Q_target to calculate the target Q value, and use TD-target to approximate the output of the Q network ;

[0055] (7) Update the Q network using the TD algorithm, the loss function TD-error of the Q network is as follows: , where After obtaining the TD-error, the gradient of the TD-error is backpropagated, and the Adam optimization algorithm is used to update the neural network parameters of the Q network. When the Q network is updated, the policy gradient algorithm is used to update π network in a fixed iteration period. The following is the policy gradient formula for updating π network:

[0056]

[0057] where, is the action predicted by the π network when the state is ;

[0058] (8) Use soft update to stabilize the parameters of the target network, slowly migrate the parameters of the main network to the target network, and the neural network parameter soft update is as follows:

[0059] ,

[0060] where represents the parameters of the target network, and represents the parameters of the main network;

[0061] (9) Repeat steps (2) to (8) and continue iteration until the algorithm converges.

[0062] Another purpose of the present application is to provide a deep reinforcement learning-based personalized federated learning system with energy-time perception, comprising:

[0063] A client screening module, each client uploads environment and device information, and the central server selects superior performance clients according to the Per-FL-DDPG algorithm;

[0064] A global model issuing module, the Per-FL-DDPG algorithm selects clients, and the central server only issues the global model to these clients to save time and energy costs;

[0065] A personalized information adding module, after receiving the global model, the client calls its own personalized information and adds it to the global model;

[0066] A local training module, after adding personalized information, the client accesses its own data set, extracts a fixed number of data for local training;

[0067] A global model uploading module, after training, the client first stores personalized information, and then uploads the trained global model to the central server.

[0068] A new global model generating module, after receiving the local training model of all clients, the central server will converge to generate a new global model.

[0069] In combination with the above technical solutions and the technical problems solved, the technical solution to be protected by the present application has the following advantages and positive effects:

[0070] In view of the problems existing in the traditional FL, the application designs a novel FL framework, aiming to solve the challenges of traditional FL in device heterogeneity, data heterogeneity and complex environment. A personalized layer is introduced in the neural network of each client to save the "personalities" of the client data distribution, ensuring that the personalized information of each client is not completely eliminated after the model is converged by the central server. In this way, it better adapts to the data distribution and application requirements of different users, especially in the scene with strong data heterogeneity, and improves the applicability and accuracy of the model. In order to cope with the challenges brought by device heterogeneity and data heterogeneity, the application designs an intelligent client selection mechanism, which comprehensively considers the system parameters of each client, ensures that the client with better system performance, more stable channel quality and more reasonable data distribution can be selected for model training under the conditions of device heterogeneity, data heterogeneity and complex environment changes. In addition, the mechanism also evaluates the network environment and channel quality of the client, avoids selecting unstable clients, and reduces the time and energy consumption of training and model uploading, and improves the overall efficiency and sustainability of the FL system. Through this strategy, the resource waste caused by randomly selecting clients in complex environments can be effectively reduced, and the model convergence process can be accelerated.

[0071] Deep reinforcement learning (DRL) has gradually become an effective solution to optimization problems due to its flexibility and extensive environmental exploration capabilities. DRL allows agents to continuously interact with the environment and learn knowledge about optimization problems from it to continuously improve their performance in the environment. Through continuous learning, DRL agents can adapt to complex and variable environments, identify potential patterns that traditional methods cannot capture. To achieve reasonable client selection in complex environments and device heterogeneity, the application proposes a novel FL framework, which is modeled as a Markov decision process (MDP), and introduces the Per-FL-DDPG algorithm. This algorithm fully considers device heterogeneity and communication channel differences, and makes reasonable client selection in each round of FL, thereby effectively constraining the time and energy consumption of each round of training and improving the convergence speed. This method significantly optimizes resource allocation and improves the overall efficiency of the system, and is particularly suitable for application scenarios with strict energy and time constraints, such as edge computing and Internet of Things environments.

[0072] The technical scheme of the application solves the technical problem that people have been eager to solve but have failed to succeed: traditional federated learning fails to consider the communication between clients and servers and the differences between devices. The application designs a completely new method, jointly considers the convergence index of traditional federated learning, communication channel level and device heterogeneity, and proposes a reasonable client screening method to constrain the consumption of time and energy cost on the premise of ensuring that the personalized information of the client is not lost and improving the convergence accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0073] Figure 1 is a personalized federated learning method flowchart with energy-time perception based on deep reinforcement learning provided by an embodiment of the present application;

[0074] Figure 2 is a personalized federated learning scene graph with energy-time perception based on deep reinforcement learning provided by an embodiment of the present application;

[0075] Figure 3 is a Per-FL-DDPG training average reward graph provided by an embodiment of the present application;

[0076] Figure 4 is a global average personalized model accuracy comparison graph provided by an embodiment of the present application;

[0077] Figure 5 is a time cost comparison graph provided by an embodiment of the present application;

[0078] Figure 6 is an energy cost comparison graph provided by an embodiment of the present application;

[0079] Figure 7 is a system structure diagram of personalized federated learning with energy-time perception based on deep reinforcement learning provided by an embodiment of the present application. DETAILED DESCRIPTION

[0080] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0081] The embodiment of the application provides a personalized federated learning method with energy-time perception based on deep reinforcement learning, first, considering the device heterogeneity, data heterogeneity and complex environmental conditions of the client, combining the data type and neural network adaptation in the FL task, the whole FL scene is remodeled, including the local client computing model, the uploaded model network model and the personalized user accuracy model, then based on the created model, the energy-time perception personalized FL basic framework is designed, according to the framework, the system modeling is carried out, the mathematical model of the whole system is obtained, then the system parameters of each client difference, the different data distribution and the different channel state are considered, the energy consumed by the local client for local training and model uploading, the time spent and the global average personalized user accuracy under a certain communication round are calculated, according to the calculated values, a multi-optimization problem is formulated, the convergence speed is improved and the personalized information of the user is protected while the time cost and the energy cost are constrained, finally, the whole system framework is modeled into a Markov process, a reasonable state and action space are designed, a reasonable reward function is designed by using the objective function, and the DDPG algorithm of deep reinforcement learning is used to optimize the objective function.

[0082] As shown in Figure 1 The embodiment of the application provides a personalized federated learning method with energy-time perception based on deep reinforcement learning, and the method comprises the following steps:

[0083] S1, based on the device heterogeneity, data heterogeneity and complex environmental conditions of the client, combining the data type and neural network adaptation in the FL task, the whole FL scene is remodeled, including the local client computing model, the uploaded model network model and the personalized user accuracy model;

[0084] S11, a scene of FL performed by a central server installed near a base station and a plurality of different clients is established, the central server and each client device are provided with a single-core CPU for computing tasks and training models, a radio frequency transmitting and receiving device for communicating equipment information, and a certain storage space for storing local data sets and task information;

[0085] S12, an individual user accuracy model is established to measure the performance of the FL operation in a more different real scene, in order to better adapt to the complex environmental changes in the real scene, a dynamic time-varying channel model is established to transmit the model network;

[0086] S2, based on the local client computing model, upload model network model and personalized user accuracy model, an energy-time-aware personalized FL basic framework is designed, and a mathematical model of the entire system is obtained based on the framework. Under this framework, at the beginning of each communication round, the central server will select appropriate clients to perform this round of FL;

[0087] As shown in Figure 2 , the entire energy-time-aware personalized FL framework is composed of a central server and several clients. The central server and the clients are equipped with single-core CPUs. The computing resources and energy costs of the central server are much larger than those of the clients. The central server is defined as , and the three-dimensional coordinates are . The client set is represented by , and , represents the number of clients, and the coordinates of each client are , so the distance between each client and the central server is represented as .

[0088] S3, based on the energy-time-aware personalized FL basic framework in S2, the system parameters of each client difference, the data distribution difference and the channel state difference are considered, and the energy consumed by the local client for local training and model uploading, the time spent and the global average personalized user accuracy under a specific communication round are calculated. According to the calculated values, a multi-optimization problem is formulated to achieve the purpose of improving the convergence speed and protecting the user's personalized information while constraining the time cost and energy cost. The multi-optimization objective function is represented as follows:

[0089]

[0090] where represents the global average personalized user accuracy of the th round, represents the energy cost of the th round of each client for model training and model parameter uploading, represents the time cost consumed in the th round of FL, , , represents the weight of the three elements in the multi-optimization objective function, and is greater than 0. The three coefficients are to adjust the weights of the three key factors in the objective function, so that they reach the same order of magnitude or similar values;

[0091] As shown in Figure 2As shown, in the personalized FL framework with energy-time awareness, the central server will select a certain number of clients with better system parameters and better communication channel quality to participate in this round of FL before each round of training . After the central server selects the appropriate clients, in order to save energy and time costs, the central server will only issue the global model to the clients selected by the central server. When these clients receive the global model, they will reconstruct the global model combined with their own personalized information layer. Then the clients will use the reconstructed local model to extract data from the local database for local training. The size of each data is defined by , which varies with different data types. The Batch-Size number required for each round of FL is denoted by , and the CPU cycle number of the th client in the th round is denoted by . Since the CPU operation of the client is affected by temperature changes, task backlog, and complex environmental conditions, it is considered that the CPU cycle number of the client is dynamically changing. Therefore, the local training time of the th client in the th round is:

[0092]

[0093] where denotes the number of rounds of local training of the client, and denotes the number of CPU cycles required for the client to train one data.

[0094] Therefore, the energy cost consumed by the th client for local training in the th round is:

[0095]

[0096] where denotes the energy efficiency factor of the th client, which is related to the CPU structure of the client. After the client completes the local training with the reconstructed model, the client will save the parameter information of the personalized information layer of the trained local model for use in the next training. After saving, the client will use its own radio frequency transmission module to send the model parameter information to the central server.

[0097] When the client local training is completed, the client will transfer the trained model to the radio frequency transmission module, and the radio frequency transmission module will transfer it to the server. Due to the complex environmental conditions and electromagnetic interference between the client and the central server, the channel quality between the server and the client is often not stable, which increases the uncertainty in the model transmission process. In order to minimize the communication interference between the central server and each client, the communication between the server and the client is designed to use the Orthogonal Frequency Division Multiple Access (OFDMA) protocol. The channel state is static within a time slot, but dynamic between different time slots. It is determined that the average channel gain between the central server and each client follows the path loss model:

[0098]

[0099] represents the average channel gain between the th client and the central server, represents the antenna gain of each client, represents the signal carrier frequency, represents the path loss exponent. The channel gain between each ground user and the central server obeys the independent and identically distributed Rayleigh distribution, so according to Shannon's theorem, the channel transmission rate of the central server and th client in rounds is:

[0100]

[0101] where represents the transmission power of the th client to the central server, represents the channel bandwidth between the th client and the central server, represents the noise power spectral density between the th client and the central server. The data size of the client model is defined as . The data size of different models is related to the adaptation of the data set, so the model upload time is:

[0102]

[0103] So the energy consumed by the th client to upload the local model in the th round is:

[0104] .

[0105] After each client has uploaded its local model, the central server aggregates the received client models. Once aggregation is complete, one round of FL (Flexible Learning) is finished. Traditional FL learning severely neglects the personalized characteristics of each client, resulting in significant loss of individual client information during model aggregation. This deficiency often leads to poor performance of the global model when handling personalized tasks, especially under heterogeneous data distribution and significant differences in device operating conditions and tasks. The model's generalization ability and personalization adaptability are both limited. Traditional global model accuracy metrics are no longer applicable to more personalized real-world scenarios. Therefore, local user accuracy is proposed to measure the performance of FL computation in more personalized real-world scenarios. The specific implementation method involves adding a personalized information layer to the model network of each client to store the personalized information of each client. The model parameters of the personalized information layer are stored locally and not with the global model aggregation. This ensures that the personalized characteristics of each client are maintained while the model is optimized globally. Local user accuracy is represented as follows:

[0106]

[0107] in Indicates the first The client in the first The accuracy of local users in the round, Indicates the first The client in the first The model network of the wheel requires two inputs, namely the first wheel... The client in the first Rounds of data used for training and the The client in the first The personalized information layer model information of the wheel. Therefore, in the first... The global average local user precision is expressed as:

[0108] .

[0109] In steps S2 and S3, since the central server is often installed near a computer room or base station with a stable power output and possesses extremely powerful computing resources, the transmitted device information occupies very little memory (often only a few bytes) in the initial stage of device information communication. Therefore, the time and energy required for the central server to perform model aggregation and device information communication are ignored. The FL total energy cost is represented as the sum of the energy used by all participating clients for computation and uploading, and the time cost for synchronous FL is represented as the maximum of the time cost used by participating clients for computation and uploading, because a round of synchronous FL is not completed until the client with the longest computation and uploading time is finished, so the time cost for the first round of FL is represented as the maximum of the time cost used by participating clients for computation and uploading. The time cost and energy cost for FL in the i-th round are represented as:

[0110]

[0111]

[0112] wherein represents the energy cost of the i-th round of FL, represents the time cost of the i-th round of FL.

[0113] In combination with the system parameter differences, data distribution differences and channel state differences of each client, the energy consumption, time overhead and global average user accuracy of the local client during local training and model uploading in a specific communication round are calculated, and based on these calculation results, a multi-objective optimization problem is comprehensively formulated:

[0114]

[0115]

[0116]

[0117]

[0118]

[0119] wherein and represent the lower bound and upper bound of the CPU frequency of the local client, and represent the lower bound and upper bound of the power of the radio frequency transmission module of the client, and represent the upper and lower bounds of the bandwidth allocation of the client.

[0120] S4, the entire system framework is modeled as a Markov process, a reasonable state and action space is designed, and a reasonable reward function is designed by using the objective function; an MDP is formulated as wherein represents the environment state, including channel information, frequency information and personalized user accuracy information, represents the agent action, because the output action of DDPG is continuous, a probability sampling method is used to map the continuous space to the discrete space, representing the reward brought by the agent performing one step action, is a state transition function, is a reward discount factor, ;

[0121] S5, using deep reinforcement learning DDPG algorithm to optimize the objective function, while improving the convergence speed and protecting the user personalized information in the constraint time cost and energy cost; using deep reinforcement learning DDPG algorithm to optimize the objective function, DDPG algorithm is a deep reinforcement learning algorithm for continuous action space, which belongs to Actor-Critic framework. It combines the advantages of policy gradient method and Q learning, and is suitable for decision-making problems in high-dimensional and continuous action environment. Using DDPG algorithm to optimize the optimization problem, the purpose of improving the convergence speed and protecting the user personalized information while constraining the time cost and energy cost is achieved, and the main steps of DDPG algorithm are as follows:

[0122] (1) initialize various parameters, FL environment, agent network, and build experience replay buffer buffer;

[0123] (2) select action, policy network π generates action and adds exploration noise, executes the action in the current state, obtains reward r and next state s';

[0124] (3) update experience pool, store the current state s, action a, reward r, next state s' in the experience buffer buffer;

[0125] (4) sample from experience buffer, randomly sample a small batch (s, a, r, s') from the experience buffer;

[0126] (5) calculate target action, use target policy network π_target to generate the target action a' of the next state s';

[0127] (6) calculate target Q value, use target network Q_target to calculate target Q value, use TD-target to approximate the output of Q network ;

[0128] (7) update Q network using TD algorithm, the loss function TD-error of Q network is represented as follows: , wherein After obtaining the TD-error, the gradient of the TD-error is back-propagated, and the neural network parameters of the Q network are updated using the Adam optimization algorithm, and when the Q network is updated, the strategy gradient algorithm is used to update the pi network in a fixed iteration period, and the following is the derived strategy gradient formula for updating the pi network:

[0129]

[0130] wherein, is the action predicted by the pi network in the state .

[0131] (8) The parameters of the target network are stabilized by soft updating, and the parameters of the main network are slowly migrated to the target network, so as to avoid overestimation of the evaluation function and improve the stability of the system, and the neural network parameter soft updating is as follows:

[0132] ,

[0133] wherein represents the parameters of the target network, and represents the parameters of the main network.

[0134] (9) Repeat steps (2) to (8) to continue iteration until the algorithm converges.

[0135] As shown in Figure 7 , the embodiment of the application provides a deep reinforcement learning-based personalized federated learning system with energy-time perception, which comprises:

[0136] A client screening module, each client uploads environment and device information, and the central server selects superior performance clients according to the Per-FL-DDPG algorithm;

[0137] A global model issuing module, the Per-FL-DDPG algorithm selects clients, and the central server only issues the global model to these clients to save time and energy costs;

[0138] A personalized information adding module, after receiving the global model, the client calls its own personalized information and adds it to the global model;

[0139] A local training module, after adding personalized information, the client accesses its own data set, extracts a fixed number of data for local training;

[0140] A global model uploading module, after training, the client first stores the personalized information, and then uploads the trained global model to the central server.

[0141] A new global model generation module, after the central server receives all the local training models of the clients, it will be converged to generate a new global model.

[0142] As Figure 3 shown, it represents the average reward of each round of reinforcement learning algorithm agent interacting with the environment. It can be seen in detail that in the initial stage of model training, the reward of the algorithm is constantly increasing. This is because the agent is constantly learning prior knowledge in the interaction with the environment, so that the output action is biased towards the direction of increasing reward. In the later stage of training, the average reward value tends to be stable. This is because the agent continues to interact with the environment, and through continuous learning and adjustment of the strategy, it optimizes its output, thereby effectively improving the accuracy and efficiency of decision-making.

[0143] As Figure 4 shown, it represents the interaction of the trained agent with the test environment configured with No-IID CIFAR10 dataset and shows the global average personalized model accuracy at each communication round. The fixed communication round is 100 rounds, in order to observe the performance and learning effect of the agent in the stable environment. It can be clearly seen that with the increase of communication rounds, the global average model accuracy of the two algorithms is constantly increasing and finally fluctuates around a relatively fixed value. The global average personalized model accuracy of Per-FL-DDPG algorithm is significantly higher than Fedavg, which shows that Per-FL-DDPG algorithm effectively preserves user personalized information and improves convergence speed under the condition of No-IID dataset.

[0144] As Figure 5 and Figure 6 shown, it represents the time and energy cost consumption of each communication round in the training of the federated learning model. Then the costs are accumulated to get the time and energy cost of a whole round. This method will help to comprehensively evaluate the performance of different algorithms in diversified environments, Figure 5 It shows the time consumption of Per-FL-DDPG algorithm and Fedavg algorithm under the conditions of IID and No-IID. It can be clearly seen that the time cost of Per-FL-DDPG algorithm is less than that of Fedavg algorithm. In Figure 6The energy consumption of Per-FL-DDPG algorithm and Fedavg algorithm under IID and No-IID conditions is shown. It can be seen that the energy cost of Per-FL-DDPG algorithm is less than that of Fedavg algorithm, which shows the superior performance of Per-FL-DDPG algorithm. Through continuous optimization of reinforcement learning, the agent can effectively explore a better strategy, which not only improves the adaptability of the model in different environments, but also significantly reduces the energy and time cost. In addition, Per-FL-DDPG algorithm shows stronger robustness and flexibility in the face of diversified data distribution, making it have higher practical value in actual application.

[0145] Embodiment one: Energy-time aware personalized FL training process under heterogeneous client conditions

[0146] In this embodiment, the central server and 100 clients constitute an FL scene, each client is equipped with a 6GHz single-core CPU, different sizes of local data sets, and different radio bandwidths (5-20 MHz). The central server first constructs a local computing model and a dynamic channel model according to S1, and reflects the data distribution differences of different users through a personalized user accuracy model. At the beginning of each round of training, the server selects participants based on the energy-time aware personalized FL framework, combined with the current CPU occupancy rate, local model performance and instantaneous channel gain of the client. According to S3, the energy consumption, upload energy consumption and time cost of each candidate client in this round are calculated, and the optimal participation set is determined through a multi-optimization objective function. On this basis, through DDPG optimization of the reward function, the selected client maintains a faster convergence speed under the constraints of energy consumption and time.

[0147] Embodiment two: DDPG driven client selection and training optimization under complex channel environment

[0148] In this embodiment, the FL system is in a complex outdoor wireless environment, and the client position changes dynamically. The channel fading adopts a time-varying Rayleigh fading model. The central server first constructs an upload network model according to S1, and updates the transmission rate in real time. Then at the beginning of each round, the server evaluates the current channel state, local load, and local model performance of each client, and inputs them as state into the DDPG agent of deep reinforcement learning. The DDPG agent trains the model according to the reward function, and selects the optimal participating client. During the entire training process, the system continuously updates the personalized user accuracy and iterates through the DDPG strategy, ultimately achieving faster global model convergence and improving user personalized accuracy under the constraints of energy and time.

[0149] It should be noted that embodiments of the present invention can be implemented in hardware, software, or a combination of both. The hardware portion can be implemented using dedicated logic; the software portion can be stored in memory and executed by a suitable instruction execution system, such as a microprocessor or dedicated-design hardware. Those skilled in the art will understand that the above-described devices and methods can be implemented using computer-executable instructions and / or included in processor control code, for example, such code provided on a carrier medium such as a disk, CD, or DVD-ROM, a programmable memory such as read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The devices and modules of the present invention can be implemented by hardware circuitry such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field-programmable gate arrays, programmable logic devices, etc., or by software executed by various types of processors, or by a combination of the above-described hardware circuitry and software, such as firmware.

[0150] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions, and improvements made by those skilled in the art within the scope of the technology disclosed in the present invention, and within the spirit and principles of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A deep reinforcement learning based personalized federated learning method with energy-time awareness, characterized in that, Comprise the following steps: S1, based on the client's device heterogeneity, data heterogeneity and complex environment conditions, combined with the data type and neural network adaptation in FL task, the whole FL scene is re-modeled, including local client computing model, uploading model network model and personalized user accuracy model; Specifically: S11, a scenario is established in which a central server installed near a base station and a plurality of different clients perform FL, the central server and each client device are equipped with a single-core CPU for computing tasks and training models, a radio frequency transmitting and receiving device is used to communicate device information, and a certain storage space is used to store local data sets and task information; S12, a personalized user accuracy model is established to measure the performance of FL operation in a more differentiated real scene, and a dynamic time-varying channel model is established for model network transmission; S2, based on the local client computing model, uploading model network model and personalized user accuracy model, an energy-time-aware personalized FL basic framework is designed, the system is modeled according to the framework, and the mathematical model of the whole system is obtained, under this framework, at the beginning of each communication round, the central server will select appropriate clients to perform this round of FL; S3, based on the energy-time-aware personalized FL basic framework in S2, the system parameters of each client difference, the data distribution difference and the channel state difference are considered, the energy consumed by the local client for local training and model uploading, the time spent and the global average personalized user accuracy under a specific communication round are calculated, and a multi-optimization problem is formulated according to the calculated values, so as to improve the convergence speed and protect the user's personalized information while constraining the time cost and energy cost, the multi-optimization objective function is represented as follows: ； wherein, represents the global average personalized user accuracy of the wheel, represents the global average personalized user accuracy of the wheel, represents the energy cost consumption of each client for model training and model parameter uploading in the wheel, represents the time cost consumed in the FL of the wheel, represents the weights of the three elements in the multi-optimization objective function, and are all greater than 0. The three coefficients are to adjust the weights of the three key factors in the objective function so that they reach the same order of magnitude or similar values. S4, the whole system framework is modeled into a Markov process, a reasonable state and action space is designed, and a reasonable reward function is designed by using the objective function; S5, the DDPG algorithm of deep reinforcement learning is used to optimize the objective function, which improves the convergence speed and protects the user's personalized information while constraining the time cost and energy cost.

2. The deep reinforcement learning based personalized federated learning method with energy-time awareness of claim 1, wherein, The whole energy-time-aware personalized FL framework in step S2 is composed of a central server and several clients, both of which are equipped with single-core CPUs, and the computing resources and energy cost of the central server are much larger than those of the clients, so the central server is used as The three-dimensional coordinates are defined as The client set is represented by , , The number of clients is represented by N, and the coordinates of each client are represented by , so the distance between each client and the central server is represented by .

3. The deep reinforcement learning based personalized federated learning method with energy-time awareness of claim 1, wherein, In the personalized FL framework with energy-time awareness, the central server selects a certain number of clients with better system parameters and better communication channel quality to participate in this round of FL before each round of training . After the central server selects the appropriate clients, the central server will only issue the global model to the clients selected by the central server. When these clients receive the global model, they will reconstruct the global model combined with their own personalized information layer. Then the clients will extract data from the local database for local training using the reconstructed local model. The size of each data is defined by , which changes with the type of data. The Batch-Size required for each round of FL is represented by , the number of cycles of the CPU of the th client in the th round is represented by . The CPU cycle number of the client is dynamically changing. The local training time of the th client in the th round is: ； wherein denotes the number of rounds of client local training, denotes the number of CPU cycles required by the client to train on one data. In the first wheel the energy cost consumed by the local training of each client is represented as: ； in Indicates the first The energy efficiency factor of each client is related to the CPU architecture of the client. After the client completes local training with the reconstructed model, the client will save the parameter information of the personalized information layer of the trained local model. After saving, the client will use its own radio frequency transmission module to send the model parameter information to the central server.

4. The deep reinforcement learning based personalized federated learning method with energy-time awareness of claim 3, wherein, When the client local training is completed, the client transfers the trained model to the radio frequency transmitting module, and the radio frequency transmitting module transfers the model to the server, the communication between the server and the client adopts the orthogonal frequency division multiple access (OFDMA) protocol, the channel state is static within a time slot, but dynamic within different time slots, and the average channel gain between the central server and each client is determined to follow the path loss model: ； denotes the average channel gain between the central server and the kth client, denotes the antenna gain of each client, denotes the signal carrier frequency, denotes the path loss exponent; and the channel gain between each ground user and the central server obeys Rayleigh distribution with independent and identical distribution, so according to Shannon theorem, the channel transmission rate of the central server and the kth client in n rounds is: Cn= Wlog (1+ Pn / k), where W is the bandwidth of the channel. ； wherein denotes the transmission power of the th client to the central server, denotes the channel bandwidth between the th client and the central server, denotes the noise power spectral density between the th client and the central server; the data size defining the client model is The data size of different models is related to the adaptation of and the dataset, so the model upload time is: ； In the first wheel the energy consumed by the local model uploaded by the client is: 。 5. The deep reinforcement learning based personalized federated learning method with energy-time awareness of claim 4, wherein, When each client uploads the local model, the central server will gather the received client model, and when the gathering is completed, it indicates the end of a round of FL; the performance of FL operation in the personalized stronger real scene is measured by the local user accuracy; specifically, a personalized information layer is added to the model network of each client to store the personalized information of each client, and the model parameters of the personalized information layer are saved locally and not aggregated with the global model, which ensures that the personalized features of each client are maintained while the global model is optimized; the local user accuracy is represented as follows: ； wherein represents the local user precision of the th client in the th round, represents the model network of the th client in the th round, which requires two inputs, respectively, the data used for training of the th client in the th round and the personalized information layer model information of the th client in the th round; so the global average local user precision in the th round is represented as: 。 6. The deep reinforcement learning based personalized federated learning method with energy-time awareness of claim 5, wherein, In the first round, the FL total energy cost is represented as the sum of the energy used by all participating FL clients for computation and upload, and the time cost is represented as the maximum of the time cost used by participating FL clients for computation and upload, in the first round for FL, the time cost and energy cost are represented as: ； ； wherein indicates the number of rounds of selecting a set of clients to participate in FL.

7. The deep reinforcement learning based personalized federated learning method with energy-time awareness of claim 6, wherein, Based on the system parameter differences, data distribution differences and channel state differences of each client, the energy consumption, time overhead and global average user accuracy of the local client during local training and model uploading in a specific communication round are calculated, and based on these calculation results, a multi-objective optimization problem is formulated: ； ； ； ； ； wherein and fmin and fmax represent the lower and upper bounds of the local client CPU frequency, and Pmin and Pmax represent the lower and upper bounds of the client radio frequency transmission module power, and Bmin and Bmax represent the lower and upper bounds of the client bandwidth allocation.

8. The deep reinforcement learning based personalized federated learning method with energy-time awareness of claim 1, wherein, The step S4 is screened by the central server before each round of training to ensure long-term stable operation of the system, and the energy-time-aware personalized FL framework is established as an MDP, and one MDP is formulated as wherein represents the environment state, including channel information, frequency information and personalized user accuracy information, represents the agent action, since the output action of DDPG is continuous, a method of probability sampling is used to map the continuous space to the discrete space, represents the reward brought by the agent performing one step of action, is a state transition function, is a reward discount factor, .

9. The deep reinforcement learning based personalized federated learning method with energy-time awareness of claim 1, wherein, The step S5 uses a deep reinforcement learning DDPG algorithm to optimize the target function, and the steps of the DDPG algorithm are as follows: (1) initialize various parameters, FL environment, Agent network, and build an experience replay buffer buffer; (2) select action, the policy network π generates action and adds exploration noise, executes the action in the current state, and obtains reward r and next state s'; (3) update the experience pool, store the current state s, action a, reward r, and next state s' in the experience buffer buffer; (4) sample from the experience buffer, randomly sample a small batch (s, a, r, s') from the experience buffer; (5) calculate the target action, use the target policy network π_target to generate the target action a' of the next state s'; (6) Compute target Q-values, using the target network Q_target to compute the target Q-values, approximating the output of the Q network using TD-target ; (7) The Q network is updated using the TD algorithm, and the loss function TD-error of the Q network is represented as follows: wherein After obtaining the TD-error, the gradient of the TD-error is back-propagated, and the neural network parameters of the Q network are updated using the Adam optimization algorithm. When the Q network is updated, the policy gradient algorithm is used to update the π network in a fixed iteration period. The following is the policy gradient formula for updating the π network: ； wherein, is the action predicted by the π network in state . (8) use soft update to stabilize the parameters of the target network, slowly migrate the parameters of the main network to the target network, and the neural network parameter soft update is as follows: ， wherein denote parameters of the target network, while denote parameters of the host network; (9) repeat steps (2) to (8) and continue iteration until the algorithm converges.

10. A deep reinforcement learning based personalized federated learning system with energy-time awareness according to the method of any one of claims 1-9, characterized in that, It includes: Client screening module, each client uploads environment and device information, and the central server selects superior clients according to the Per-FL-DDPG algorithm; Global model distribution module, the Per-FL-DDPG algorithm selects clients, and the central server only distributes the global model to these clients to save time and energy costs; Personalized information adding module, after receiving the global model, the client calls its own personalized information and adds it to the global model; Local training module, after adding personalized information, the client accesses its own data set, extracts a fixed number of data for local training; Global model upload module, after training, the client first stores the personalized information and then uploads the trained global model to the central server. New global model generation module, after receiving all client local training models, the central server will aggregate them to generate a new global model.

Images