Reputation-based client scheduling deep reinforcement learning optimization method

Abstract

The embodiment of the invention relates to the technical field of wireless federated learning, in particular to a reputation-based client scheduling deep reinforcement learning optimization method, which comprises the following steps of: defining and initializing an environment state of a client, and designing actions and rewards; optimal action selection and random sampling are carried out through an actor network, and client scheduling is obtained; determining bandwidth allocation, transmission rate and CPU frequency based on client scheduling; instructing each client to execute local federal learning and upload an update model to update the reputation value of each client; obtaining a reward based on the feedback signal, updating the current state, the current action, the current reward, the current state value and the future state, and putting into a replay buffer area; and if a set maximum iteration number threshold value is reached or a preset convergence judgment condition is met, obtaining optimized actor network and reviewer network loss function parameters, and further obtaining optimal client scheduling. The method can reduce federal learning delay and optimize client scheduling.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of wireless federated learning, and in particular to a reputation-based client scheduling deep reinforcement learning optimization method. Background Art

[0002] Federated machine learning is a machine learning framework that can effectively help multiple institutions use data and conduct machine learning modeling while meeting the requirements of user privacy protection, data security and government regulations. As a distributed machine learning paradigm, federated learning can effectively solve the problem of data silos, allowing participants to conduct joint modeling without sharing data. It can technically break data silos and achieve artificial intelligence collaboration. Federated learning defines a machine learning framework, under which virtual models are designed to solve the problem of different data owners collaborating without exchanging data. The virtual model is the optimal model for all parties to aggregate data together, and each region serves local goals based on the model. Federated learning requires that the modeling result should be infinitely close to the traditional model, that is, the result of aggregating the data of multiple data owners in one place for modeling. Under the federation mechanism, the identity and status of each participant are the same, and a shared data strategy can be established. Since data is not transferred, user privacy will not be leaked or data specifications will not be affected.

[0003] Federated learning is a new type of distributed machine learning that allows multiple participants to train machine learning models locally. In recent years, federated learning has attracted widespread attention from academia and industry due to its ability to disperse raw data and not upload it to cloud servers, thereby preventing the leakage of user privacy data. However, while enjoying the benefits of federated learning, it also faces some challenges. First, federated learning enables multiple participants to train machine learning models while protecting data privacy, but since the server cannot monitor the training process conducted locally by each participant, the participants can tamper with the local training model, posing a security risk to the global model of federated learning. The proposed FL protocol design has been proven to be vulnerable to attacks from attackers inside and outside the system, thereby compromising data privacy and system robustness. Second, federated learning encounters problems such as resource imbalance and inefficiency. In addition, during the update of the federated learning synchronization model, differences in channel conditions and terminal client computing resources may lead to discrete problems. In the real world, heterogeneous federated learning faces security issues related to model fairness. The contributions of clients participating in collaborative learning to the cooperation may differ, and this difference may be exacerbated by heterogeneity. At the same time, most of the proposed federated learning frameworks ignore the differences in contributions of participating clients in the collaborative process, so there may be some "free-riding" participants who want to learn from federated communication without contributing useful information. In addition, the global model obtained through joint training may be biased towards clients with large data volumes or frequent occurrences, and the overall loss function may implicitly favor or disadvantage certain clients.

[0004] Through the above analysis, we can find that the current federated learning has the following problems:

[0005] 1) Malicious actors may intentionally manipulate data or models to influence the results of federated learning, thereby undermining the credibility and accuracy of the entire system.

[0006] 2) Federated learning may encounter problems such as resource imbalance and inefficiency.

[0007] 3) During the update of the federated learning synchronization model, differences in channel conditions and terminal client computing resources may lead to discrete problems, which in turn affects the effectiveness of federated learning. Summary of the invention

[0008] In view of this, an embodiment of the present application proposes a reputation-based client scheduling deep reinforcement learning optimization method, which can minimize the training delay of federated learning, while optimizing client scheduling, client CPU frequency, bandwidth allocation and transmission rate, maintaining user fairness and model integrity, and effectively improving the client's ability to deal with malicious attacks.

[0009] In a first aspect, an embodiment of the present application proposes a reputation-based client scheduling deep reinforcement learning optimization method, comprising the following steps:

[0010] S1, define and initialize the client's environment state, and design actions and rewards; the client's environment state includes channel gain, reputation value, and CPU available resources;

[0011] S2, through the actor network, selects the optimal action according to the client's environmental state, and randomly samples the selected optimal action to obtain the client schedule;

[0012] S3, determines bandwidth allocation, transmission rate, and CPU frequency based on client scheduling and client environment status;

[0013] S4, instructing each client to perform local federated learning training and upload an updated model, and updating the reputation value of each client based on each updated model;

[0014] S5, based on the feedback signal received when executing the optimal action, the reward is obtained, and the current state, current action, current reward, current state value and future state are updated and placed in the replay buffer;

[0015] S6, if the set maximum number of iterations reaches the threshold, or the preset convergence judgment condition is met, the optimized actor network loss function parameters and critic network loss function parameters are obtained, otherwise, S2 to S6 are repeated;

[0016] S7, through the optimized actor network, the optimal action is selected according to the client's environmental state, and random sampling is performed based on the selected optimal action to obtain the optimal client scheduling.

[0017] Optionally, in S1, define and initialize the client's environment state, including:

[0018] Define the client's environmental state in the τth iteration as s(τ), which is expressed by the formula:

[0019] s(τ)=[s1(τ),s2(τ),…,s k (τ),…,s K (τ)];

[0020]

[0021] Where K represents the total number of clients, s k (τ) represents the state information of the kth client in the τth iteration, h k (τ) represents the channel gain of the kth client in the τth iteration, RE k(τ) represents the reputation value of the k-th client in the τ-th iteration, represents the CPU available resources of the k-th client in the τ-th iteration, and the client's environment state is initialized to τ = 1;

[0022] In S1, design actions, including:

[0023] Define the action in the τth iteration as a(τ)=[a k (τ)],a k (τ) represents the probability that the kth client is selected;

[0024] In S1, design rewards include:

[0025] Define the reward in the τth iteration as r(τ) = -t total (τ), t total (τ) represents the total time required for all clients to execute the decision.

[0026] Optionally, in S2, the optimal action selection is performed according to the environmental state of the client through the actor network, including:

[0027] In the τth iteration, the state signal s is obtained from the environment through the actor network. k (τ);

[0028] Based on s k (τ), according to the strategy π, select the optimal action and determine action a k (τ) and reputation value RE k (τ);

[0029] Based on action a k (τ) and reputation value RE k (τ) Calculate the client's probability vector v k (τ), v k (τ) is expressed as:

[0030]

[0031]

[0032] Where A(τ) represents the set of available clients, represents the reputation required by the kth client;

[0033] Random sampling is performed based on the selected optimal action to obtain the client scheduling, including:

[0034] According to action a k (τ) randomly samples the client schedule x(τ) = [x k (τ)],xk (τ)∈{0,1};

[0035] Among them, x k (τ) = 0 means that the kth client is not selected in the τth iteration, x k (τ)=1 means that the kth client is selected in the τth iteration.

[0036] Optionally, S3 specifically includes:

[0037] Through the KKT condition, according to the client scheduling x(τ) and the CPU available resources f ava (τ), based on the Lagrangian dual decomposition method, we get the bandwidth allocation α(τ) and the transmission rate o(τ);

[0038] According to the client scheduling x(τ), bandwidth allocation α(τ) and transmission rate o(τ), the optimal solution of CPU frequency f(τ) is found based on the heuristic adaptive search algorithm.

[0039] Optionally, in S4, the reputation value of each client is updated based on each update model, which is achieved by the following formula:

[0040]

[0041] Among them, pos k represents the positive behavior of the kth client, neg k represents the negative behavior of the kth client, b k represents the trustworthiness of the kth client, y k represents the uncertainty of the kth client.

[0042] Optionally, S5 specifically includes:

[0043] When performing the optimal action, receive the feedback signal r(τ) and use r(τ) as the current reward;

[0044] The current state s(τ), current action a(τ), current reward r(τ), current state value V π [s(τ);Ω] and the future state s(τ+1) are placed in the replay buffer;

[0045] V π The expression of [s(τ);Ω] is:

[0046] V π [s(τ);Ω]=E π {r(τ)+γ·V π [s(τ+1);Ω]};

[0047] Among them, Ω is the commentator network loss function parameter, and γ is the preset discount factor.

[0048] Optionally, S6 specifically includes:

[0049] Determine whether the size of the replay buffer ζ reaches a preset upper limit of the replay buffer size ψ;

[0050] If ζ<ψ, set τ=τ+1 and repeat S2 to S6;

[0051] If ζ ≥ ψ, then through the deep reinforcement learning network, according to the current state s(τ), current action a(τ), current reward r(τ), current state value V π [s(τ); Ω], the future state s(τ+1) and the discount factor γ are used to calculate the actor network loss function parameter θ and the critic network loss function parameter Ω.

[0052] Optionally, the actor network loss function parameter θ is solved as:

[0053]

[0054] Among them, τ (θ) represents the difference between executing the new strategy θ and executing the old strategy θ old The probability ratio, clip<Π τ (θ),1-ξ,1+ξ> represents Π τ (θ) has a lower bound 1-ξ and an upper bound 1+ξ, if Π τ (θ) is close to 1, which means that the new strategy θ is consistent with the old strategy θ. old The difference between represents the advantage value of an action relative to the average expected value, λ represents the preset bias-variance trade-off factor, δ(τ) represents the time differential error, and Ω old represents the critic network loss function parameters before updating;

[0055] The solution formula for the critic network loss function parameter Ω is:

[0056]

[0057] in, Represents the estimate of the state-value function.

[0058] Through the above method, the present application proposes an optimization framework for reputation-based wireless federated learning aggregation, which realizes the integrated optimization of communication and computing resources. In this framework, a reputation model is introduced to evaluate the quality of the client. On the premise of ensuring that the client model is trustworthy, a minimum-maximum optimization framework is introduced to ensure fair treatment of all participating clients. The proposed optimization problem is a mixed integer nonlinear programming problem that is difficult to solve directly. In order to solve this mixed integer nonlinear programming problem, the present application develops a deep reinforcement learning algorithm for client scheduling and an iterative algorithm for radio resource allocation to optimize the actor network loss function parameters and the critic network loss function parameters, so as to obtain the best delay fairness between clients. In addition, the present application not only performs well in reducing delay, but also skillfully balances the convergence efficiency and total training delay, and effectively improves the client's ability to deal with malicious attacks. In general, the present application can minimize the training delay of federated learning, while optimizing client scheduling, client CPU frequency, bandwidth allocation and transmission rate, while adhering to user fairness and model integrity.

[0059] In the second aspect, an embodiment of the present application proposes a server, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a reputation-based client-scheduling deep reinforcement learning optimization method as described in the first aspect above.

[0060] In the third aspect, an embodiment of the present application proposes a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, it can implement a reputation-based client-scheduling deep reinforcement learning optimization method as described in the first aspect above.

[0061] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the related technologies, the drawings required for use in the embodiments of the present application or the related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0063] Figure 1is a flowchart of a reputation-based client scheduling deep reinforcement learning optimization method provided in one embodiment of the present application;

[0064] Figure 2 is a schematic diagram of the network architecture of a deep reinforcement learning network provided in one embodiment of the present application;

[0065] Figure 3 is a detailed flow chart of a reputation-based client scheduling deep reinforcement learning optimization method provided in one embodiment of the present application;

[0066] Figure 4 is a convergence diagram of a deep reinforcement learning network provided in another embodiment of the present application;

[0067] Figure 5 This is a schematic diagram of a comparison of the federated learning test accuracy of the present application and other algorithms under different attacks, provided in another embodiment of the present application;

[0068] Figure 6 It is a schematic diagram comparing the total delay of the present application and other algorithms under different attacks, provided in another embodiment of the present application;

[0069] Figure 7 It is a schematic diagram of the comparison of the delay of the present application and other algorithms under the average client, the best client and the worst client provided in another embodiment of the present application;

[0070] Figure 8 It is a structural diagram of a server provided in another embodiment of the present application. DETAILED DESCRIPTION

[0071] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the embodiments of the present application will be described in detail below in conjunction with the accompanying drawings. In the various embodiments of the present application, many technical details are proposed in order to make the reader better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical scheme claimed in the present application can also be implemented. The division of the following embodiments is only for the convenience of description, and the specific implementation mode of the present application should not constitute any limitation. The various embodiments can be combined with each other and referenced to each other under the premise of no contradiction.

[0072] An embodiment of the present application proposes a reputation-based client-scheduling deep reinforcement learning optimization method, which is applied to a server. The implementation details of the reputation-based client-scheduling deep reinforcement learning optimization method proposed in this embodiment are described in detail below. The following content is only the implementation details provided for the convenience of understanding and is not necessary for the implementation of this solution.

[0073] The specific process of the reputation-based client scheduling deep reinforcement learning optimization method proposed in this embodiment can be as follows: Figure 1 As shown, including:

[0074] S1, define and initialize the client's environment state, and design actions and rewards; the client's environment state includes channel gain, reputation value, and CPU available resources.

[0075] In the specific implementation, the server first needs to define and initialize the client's environment state, and also needs to complete the design of actions and rewards. The state, action, and reward are all for the deep reinforcement learning network. The network structure of the deep reinforcement learning network used by the server can be as follows: Figure 2 As shown in the figure, the deep reinforcement learning network is specifically composed of an actor network (actor network) and a critic network (critic network).

[0076] Simply put, the actor network is responsible for interacting with the client environment, selecting the action to be performed at the next moment, and obtaining interaction samples, while the critic network is responsible for calculating the target value.

[0077] In detail, the critic network evaluates the performance of the current state and outputs a state value to guide the actor network. The input of the actor network is the current state and the output is the action value. After the action is executed, the environment will provide reward information and information feedback for the next moment. The state, action, and reward information will be saved in the replay buffer until the size of the replay buffer reaches the upper limit. The two networks will update their network parameters based on the data in the replay buffer. In the initialization phase, the network randomly outputs actions and values. As the network is updated, the state value of the critic network becomes more accurate and the output action of the actor network becomes more optimized. The critic network learns a state value estimation function based on the data collected from the interaction between the actor and the environment. The state value estimation function can provide an estimate for each state, indicating what the expected benefit of the current strategy is. It can evaluate the effectiveness of the action in the current state, thereby assisting the actor to update the strategy.

[0078] In one example, the server defines and initializes the client's environment state, including:

[0079] Define the client's environmental state in the τth iteration as s(τ), s(τ) is expressed by the formula:

[0080] s(τ)=[s1(τ),s2(τ),…,s k (τ),…,s K (τ)];

[0081]

[0082] Where K represents the total number of clients, s k (τ) represents the state information of the kth client in the τth iteration, h k (τ) represents the channel gain of the kth client in the τth iteration, RE k (τ) represents the reputation value of the k-th client in the τ-th iteration, It represents the CPU available resources (also called CPU frequency) of the k-th client in the τ-th iteration. The client's environment state is initialized to τ = 1, that is, s(1).

[0083] In one example, the server designs actions including:

[0084] Define the action in the τth iteration as a(τ)=[a k (τ)],a k (τ) represents the probability that the kth client is selected.

[0085] In one example, a server design reward includes:

[0086] Define the reward in the τth iteration as r(τ) = -t total (τ), t total (τ) represents the total time required for all clients to execute the decision.

[0087] In one example, the server can be regarded as a policy agent in a deep reinforcement learning network. The policy agent provides a dual-network model in which the actor network and the critic network are decoupled for optimization. The mean square error between the target value of the critic network and the estimated value of the actor network is used as the loss. Through interactive learning between the client environment and the policy agent, the optimal deep reinforcement learning network is obtained.

[0088] S2, through the actor network, selects the optimal action according to the client's environmental state, and randomly samples the selected optimal action to obtain the client scheduling.

[0089] In the specific implementation, after the server completes the initialization, it can select the optimal action according to the client's environmental state through the actor network, and perform random sampling based on the selected optimal action to obtain the client scheduling.

[0090] In one example, the server selects the optimal action according to the client's environmental state through the actor network, including: in the τth iteration, obtaining the state signal s from the environment through the actor network k (τ), based on s k (τ), according to the strategy π, select the optimal action and determine action a k (τ) and reputation value RE k(τ), then based on action a k (τ) and reputation value RE k (τ) Calculate the client's probability vector v k (τ).

[0091] In one example, v k (τ) is expressed by the following formula:

[0092]

[0093]

[0094] Where A(τ) represents the set of available clients, represents the reputation required by the kth client.

[0095] In one example, the server randomly samples the optimal action selected to obtain the client schedule, specifically, according to action a k (τ) randomly samples the client schedule x(τ) = [x k (τ)],x k (τ)∈{0,1}; where x k (τ) = 0 means that the kth client is not selected in the τth iteration, x k (τ)=1 means that the kth client is selected in the τth iteration.

[0096] S3, determines bandwidth allocation, transmission rate and CPU frequency based on client scheduling and client environment status.

[0097] In a specific implementation, after obtaining the client scheduling, the server can determine the bandwidth allocation, transmission rate and CPU frequency based on the client scheduling and the client's environmental status.

[0098] In an example, the server can use the KKT condition to schedule x(τ) based on the client and the known CPU available resources f ava (τ), based on the Lagrangian dual decomposition method, the bandwidth allocation α(τ) and the transmission rate o(τ) are obtained. Then, according to the client scheduling x(τ), the bandwidth allocation α(τ) and the transmission rate o(τ), the optimal solution of the CPU frequency f(τ) is found based on the heuristic adaptive search algorithm.

[0099] S4, instructs each client to perform local federated learning training and upload an updated model, and updates the reputation value of each client based on each updated model.

[0100] In the specific implementation, after determining the bandwidth allocation, transmission rate and CPU frequency, the server instructs each selected client to perform local federated learning training and upload an updated model, and updates the reputation value of each client based on each updated model (actually based on the performance of each client in the federated learning training).

[0101] In one example, the server updates the reputation value of each client based on each update model, which can be achieved by the following formula: Among them, pos k represents the positive behavior of the kth client, neg k represents the negative behavior of the kth client, b k represents the trustworthiness of the kth client, y k represents the uncertainty of the kth client.

[0102] S5, based on the feedback signal received when executing the optimal action, the reward is obtained, and the current state, current action, current reward, current state value and future state are updated and placed in the replay buffer.

[0103] In the specific implementation, when the server performs the optimal action, it receives the timely feedback signal r(τ) and uses r(τ) as the current reward. Then it calculates the current state s(τ), the current action a(τ), the current reward r(τ), the current state value V π [s(τ);Ω] and the future state s(τ+1) are placed in the replay buffer.

[0104] In one example, V π The expression of [s(τ);Ω] is:

[0105] V π [s(τ);Ω]=E π {r(τ)+γ·V π [s(τ+1);Ω]};

[0106] Among them, Ω is the commentator network loss function parameter, and γ is the preset discount factor.

[0107] S6, if the set maximum number of iterations threshold is reached, or the preset convergence judgment condition is met, the optimized actor network loss function parameters and critic network loss function parameters are obtained, otherwise, S2 to S6 are repeated.

[0108] In the specific implementation, there are two conditions for the server to end the iteration. One is that the number of iterations τ reaches the set maximum number of iterations T, and the other is that the size of the replay buffer ζ reaches the preset upper limit of the replay buffer size ψ. The priority of the judgment of the number of iterations τ is higher than the judgment of the size of the replay buffer ζ.

[0109] For the size of the replay buffer ζ: if ζ<ψ, the server sets τ=τ+1 and repeats S2 to S6; if ζ≥ψ, the server uses the deep reinforcement learning network (that is, the actor network and the critic network) to calculate the reward based on the current state s(τ), the current action a(τ), the current reward r(τ), and the current state value V π [s(τ);Ω], the future state s(τ+1), and the discount factor γ are used to calculate the actor network loss function parameters θ and the critic network loss function parameters Ω.

[0110] In one example, the actor network loss function parameter θ is solved as:

[0111]

[0112] Among them, τ (θ) represents the difference between executing the new strategy θ and executing the old strategy θ old The probability ratio, clip<Π τ (θ),1-ξ,1+ξ> represents Π τ (θ) has a lower bound 1-ξ and an upper bound 1+ξ, if Π τ (θ) is close to 1, which means that the new strategy θ is consistent with the old strategy θ. old The difference between represents the advantage value of an action relative to the average expected value, λ represents the preset bias-variance trade-off factor, δ(τ) represents the time differential error, and Ω old Represents the critic network loss function parameters before updating.

[0113] In one example, the loss parameter Ω of the critic network is solved as:

[0114]

[0115] in, Represents the estimate of the state-value function.

[0116] S7, through the optimized actor network, the optimal action is selected according to the client's environmental state, and random sampling is performed based on the selected optimal action to obtain the optimal client scheduling.

[0117] In the specific implementation, after solving the actor network loss function parameters θ and the critic network loss function parameters Ω, the server can use the optimized actor network to select the optimal action according to the client's environmental state, and perform random sampling based on the selected optimal action to obtain the optimal client scheduling.

[0118] In an example, the detailed process of a reputation-based client scheduling deep reinforcement learning optimization method proposed in this embodiment can be as follows: Figure 3 shown.

[0119] This embodiment provides an optimization framework for reputation-based wireless federated learning aggregation, which realizes the integrated optimization of communication and computing resources. In this framework, a reputation model is introduced to evaluate the quality of the client. Under the premise of ensuring that the client model is trustworthy, a minimum-maximum optimization framework is introduced to ensure fair treatment of all participating clients. The proposed optimization problem is a mixed integer nonlinear programming problem that is difficult to solve directly. In order to solve this mixed integer nonlinear programming problem, this embodiment develops a deep reinforcement learning algorithm for client scheduling and an iterative algorithm for radio resource allocation to optimize the actor network loss function parameters and the commentator network loss function parameters, so as to obtain the best delay fairness between clients. In addition, this embodiment not only performs well in reducing delay, but also skillfully balances convergence efficiency and total training delay, and effectively improves the client's ability to deal with malicious attacks. In general, this embodiment can minimize the training delay of federated learning, while optimizing client scheduling, client CPU frequency, bandwidth allocation and transmission rate, while adhering to user fairness and model integrity.

[0120] The step division of the above methods is only for clear description. When implemented, they can be combined into one step or some steps can be split and decomposed into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this application; adding insignificant modifications to the algorithm or process or introducing insignificant designs without changing the core design of the algorithm and process are all within the scope of protection of this application.

[0121] In another embodiment, Figure 4 As shown, Figure 4 The convergence of the optimization algorithm in a reputation-based client scheduling deep reinforcement learning optimization method (hereinafter referred to as MNIST) proposed in this application is demonstrated. As the main algorithm framework of this application, its reward is the target value of the optimization problem. Figure 4 It can be observed that in the case of slight jitter, the rewards eventually converge.

[0122] exist Figure 5 In the , the federated learning test accuracy of MNIST and other algorithms, including RP, FedAvg and RR, is compared under different attacks. The solid curve represents the average of 10 experiments, and the shaded envelope represents the upper and lower limits of the accuracy. Figure 5As can be seen from the figure, as the proportion of malicious attacks increases, the convergence of FedAvg and RR is seriously damaged, while MNIST and RP still maintain good convergence. The reason for this phenomenon is that FedAvg and RR cannot identify clients containing malicious data attacks, which leads to model contamination. It is worth noting that under the condition of non-independent and identically distributed, MNIST has a faster convergence speed and better convergence accuracy than the RP algorithm.

[0123] Figure 6 The comparison of the total delay between MNIST and RP, FedAvg and RR under different attacks is shown. In order to observe the change of time delay more intuitively, we added two settings, namely Attack = 30 and Attack = 50. Figure 6 It can be observed that MNIST has significant differences compared to other algorithms in the scenarios of Attack = 20, 30, 40, and 50. This phenomenon is because MNIST prioritizes clients with low latency while ensuring quality. As the number of malicious clients increases, the number of available clients decreases, and MNIST selects clients with lower latency from the remaining clients. RP always prioritizes clients with good reputation.

[0124] As a result, MNIST is very efficient, striking a balance between convergence performance and latency.

[0125] from Figure 7 It can be seen that the latency comparison of MNIST and other algorithms under different schemes and clients is shown in Figure 2. In order to highlight the efficiency of resource optimization of MNIST, the following schemes are set for comparison.

[0126] 1)ROA, randomly sets the transmission rate and optimizes the client scheduling variable, CPU frequency variable and bandwidth ratio variable; 2)RBA, randomly sets the bandwidth ratio and optimizes the client scheduling variable, CPU frequency variable and transmission rate variable; 3)RFA, randomly sets the CPU main frequency and optimizes the client scheduling variable, bandwidth ratio variable and transmission rate variable; 4)RXA, randomly sets the client scheduling and optimizes the CPU frequency variable, bandwidth ratio variable and transmission rate variable.

[0127] exist Figure 7 We further compared MNIST( Figure 7The average client latency, optimal client latency, and worst client latency between ROA, RBA, RFA, and RXA are shown in Figure 1. It can be seen intuitively that the latency of ROA and RBA exceeds that of RFA and RXA, so optimizing the transmission rate and bandwidth ratio is more conducive to reducing latency than CPU frequency and client scheduling. In addition, compared with MNIST, the ROA algorithm sacrifices the efficiency of other clients to improve the average latency, which will cause serious waste of resources. Moreover, compared with other algorithms, MNIST has slight differences in the latency of the best and worst clients, balancing the latency of each client to achieve efficiency and fairness.

[0128] The above results verify the superiority of MNIST in achieving balanced and fair resource allocation.

[0129] Another embodiment of the present application provides a server, whose structure is as follows: Figure 8 As shown, it includes: at least one processor 301; and a memory 302 that is communicatively connected to the at least one processor 301; wherein the memory 302 stores instructions that can be executed by the at least one processor 301, and the instructions are executed by the at least one processor 301 so that the at least one processor 301 can execute a reputation-based client-scheduling deep reinforcement learning optimization method described in the above-mentioned method embodiments.

[0130] Among them, the memory and the processor are connected in a bus manner, and the bus may include any number of interconnected buses and bridges, and the bus connects various circuits of one or more processors and memories together. The bus can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and will not be further described in this article. The bus interface is responsible for providing an interface between the bus and the transceiver. The transceiver can be one component or multiple components, such as multiple receivers and transmitters, providing a unit for communicating with various other devices on a transmission medium. The data processed by the processor is transmitted on a wireless medium through an antenna, and further, the antenna also receives data and transmits the data to the processor.

[0131] The processor is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory can be used to store data used by the processor when performing operations.

[0132] Another embodiment of the present application proposes a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it can implement a reputation-based client-scheduling deep reinforcement learning optimization method described in the above method embodiments.

[0133] That is, those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, and the program is stored in a storage medium, including a number of instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, ROM (Read-Only Memory), RAM (Random Access Memory), disk or optical disk and other media that can store program codes.

[0134] Those skilled in the art will appreciate that the above embodiments are specific embodiments for implementing the present application, and in actual applications, various changes may be made thereto in form and detail without departing from the spirit and scope of the present application.

Claims

1. A reputation-based client scheduling deep reinforcement learning optimization method, characterized in that: include: S1, define and initialize the client's environment state, and design actions and rewards; the client's environment state includes channel gain, reputation value, and CPU available resources; S2, through the actor network, selects the optimal action according to the client's environmental state, and randomly samples the selected optimal action to obtain the client schedule; S3, determines bandwidth allocation, transmission rate, and CPU frequency based on client scheduling and client environment status; S4, instructing each client to perform local federated learning training and upload an updated model, and updating the reputation value of each client based on each updated model; S5, based on the feedback signal received when executing the optimal action, the reward is obtained, and the current state, current action, current reward, current state value and future state are updated and placed in the replay buffer; S6, if the set maximum number of iterations reaches the threshold, or the preset convergence judgment condition is met, the optimized actor network loss function parameters and critic network loss function parameters are obtained, otherwise, S2 to S6 are repeated; S7, through the optimized actor network, the optimal action is selected according to the client's environmental state, and random sampling is performed based on the selected optimal action to obtain the optimal client scheduling.

2. A reputation-based client scheduling deep reinforcement learning optimization method according to claim 1, characterized in that: In S1, define and initialize the client's environment state, including: Define the client's environmental state in the τth iteration as s(τ), which is expressed by the formula: s(τ)=[s1(τ),s2(τ),…,s k (t),…,s K (t)]; Where K represents the total number of clients, s k (τ) represents the state information of the kth client in the τth iteration, h k (τ) represents the channel gain of the kth client in the τth iteration, RE k (τ) represents the reputation value of the k-th client in the τ-th iteration, represents the CPU available resources of the k-th client in the τ-th iteration, and the client's environment state is initialized to τ = 1; In S1, design actions, including: Define the action in the τth iteration as a(τ)=[a k (τ)],a k (τ) represents the probability that the kth client is selected; In S1, design rewards include: Define the reward in the τth iteration as r(τ) = -t total (τ), t total (τ) represents the total time required for all clients to execute the decision.

3. A reputation-based client scheduling deep reinforcement learning optimization method according to claim 2, characterized in that: In S2, the actor network is used to select the optimal action based on the client's environmental state, including: In the τth iteration, the state signal s is obtained from the environment through the actor network. k (τ); Based on s k (τ), according to the strategy π, select the optimal action and determine action a k (τ) and reputation value RE k (τ); Based on action a k (τ) and reputation value RE k (τ) Calculate the client's probability vector v k (τ), v k (τ) is expressed as: Where A(τ) represents the set of available clients, represents the reputation required by the kth client; Random sampling is performed based on the selected optimal action to obtain the client scheduling, including: According to action a k (τ) randomly samples the client schedule x(τ) = [x k (τ)],x k (τ)∈{0,1}; Among them, x k (τ) = 0 means that the kth client is not selected in the τth iteration, x k (τ)=1 means that the kth client is selected in the τth iteration.

4. A reputation-based client scheduling deep reinforcement learning optimization method according to claim 3, characterized in that: S3, specifically includes: Through the KKT condition, according to the client scheduling x(τ) and the CPU available resources f ava (τ), based on the Lagrangian dual decomposition method, we get the bandwidth allocation α(τ) and the transmission rate o(τ); According to the client scheduling x(τ), bandwidth allocation α(τ) and transmission rate o(τ), the optimal solution of CPU frequency f(τ) is found based on the heuristic adaptive search algorithm.

5. A reputation-based client scheduling deep reinforcement learning optimization method according to claim 4, characterized in that: In S4, the reputation value of each client is updated based on each update model, which is achieved through the following formula: Among them, pos k represents the positive behavior of the kth client, neg k represents the negative behavior of the kth client, b k represents the trustworthiness of the kth client, y k represents the uncertainty of the kth client.

6. A reputation-based client scheduling deep reinforcement learning optimization method according to claim 5, characterized in that: S5, specifically includes: When performing the optimal action, receive the feedback signal r(τ) and use r(τ) as the current reward; The current state s(τ), current action a(τ), current reward r(τ), current state value V π [s(τ);Ω] and the future state s(τ+1) are placed in the replay buffer; V π The expression of [s(τ);Ω] is: V π [s(τ);Ω]=E π {r(τ)+γ·V π [s(τ+1);Ω]}; Among them, Ω is the commentator network loss function parameter, and γ is the preset discount factor.

7. The reputation-based client scheduling deep reinforcement learning optimization method according to claim 5, characterized in that: S6, specifically includes: Determine whether the size of the replay buffer ζ reaches a preset upper limit of the replay buffer size ψ; If ζ<ψ, set τ=τ+1 and repeat S2 to S6; If ζ ≥ ψ, then through the deep reinforcement learning network, according to the current state s(τ), current action a(τ), current reward r(τ), current state value V π [s(τ); Ω], the future state s(τ+1) and the discount factor γ are used to calculate the actor network loss function parameter θ and the critic network loss function parameter Ω.

8. The reputation-based client scheduling deep reinforcement learning optimization method according to claim 7, characterized in that: The solution formula for the actor network loss function parameter θ is: Among them, τ (θ) represents the difference between executing the new strategy θ and executing the old strategy θ old The probability ratio, clip<Π τ (θ),1-ξ,1+ξ> represents Π τ (θ) has a lower bound 1-ξ and an upper bound 1+ξ, if Π τ (θ) is close to 1, which means that the new strategy θ is consistent with the old strategy θ. old The difference between represents the advantage value of an action relative to the average expected value, λ represents the preset bias-variance trade-off factor, δ(τ) represents the time differential error, and Ω old represents the critic network loss function parameters before updating; The solution formula for the critic network loss function parameter Ω is: in, Represents the estimate of the state-value function.

9. A server, characterized in that: include: at least one processor; and, a memory communicatively coupled to the at least one processor; In which, the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a reputation-based client-scheduling deep reinforcement learning optimization method as described in any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it can implement a reputation-based client scheduling deep reinforcement learning optimization method as described in any one of claims 1 to 8.

Images