A risk-averse federated security reinforcement learning approach

Abstract

This paper discloses a risk-averse federated secure reinforcement learning method that improves federated training efficiency while avoiding dangerous conditions such as model aggregation failure. The method first establishes a policy risk assessment mechanism, using privacy protection level and model training accuracy as evaluation criteria. It then designs an R network based on long-term risk value and a Q network based on long-term discounted expected benefit. The server then uses a deep reinforcement learning algorithm to select participating nodes, comprehensively improving the accuracy and speed of federated learning model training while further protecting user data privacy.

Description

Technical Field

[0001] The present invention relates to the field of network security technology, and in particular to a federated security reinforcement learning method based on risk avoidance. Background Art

[0002] Federated learning addresses the drawbacks of traditional centralized model training by enabling edge devices to collaboratively train global models. This allows all models to be trained without leaving their local data, ensuring user data privacy while improving the real-time nature of model parameter updates. However, the federated learning training process still presents challenges such as the leakage of user sensitive information, low training accuracy due to limited computing power, and slow parameter transfer. Therefore, federated learning must balance model training accuracy and speed, jointly optimizing the participating node selection strategy across all edge devices in the face of unknown environmental variations.

[0003] Reinforcement learning is a model-free algorithm that allows network users to continuously interact with the environment through trial and error, optimizing strategies such as the selection of participating nodes in federated learning when the network topology changes. For example, Lu et al. proposed a secure participating node selection method based on reinforcement learning, using a deep deterministic policy gradient algorithm to select nodes participating in model training in federated learning to improve efficiency. Chinese patent application publication number CN113191484 invented a method and system for intelligent selection of federated learning clients based on deep reinforcement learning; Chinese patent application publication number CN113467952 discloses a distributed federated learning collaborative computing method and system. However, the above method traverses and explores all possible training strategy sets, including dangerous strategies that cause the training model accuracy to fall below system requirements or leak privacy.

[0004] Unlike the aforementioned federated learning parameter optimization method based on general reinforcement learning, this patent design incorporates a policy risk assessment mechanism for federated learning, avoiding dangerous strategies that could cause the training model to fall below requirements or leak privacy, thereby improving model training accuracy and speed while enabling safe exploration. This method uses reinforcement learning to obtain the optimal federated learning participating node selection strategy, without requiring foreknowledge of network topology changes, while avoiding dangerous strategies. This improves model training accuracy and speed, as well as privacy protection. Summary of the Invention

[0005] The purpose of this invention is to design a risk-averse federated secure reinforcement learning method that improves federated training efficiency while avoiding dangerous conditions such as model aggregation failure. This method first constructs a policy risk assessment mechanism, using privacy protection level and model training accuracy as evaluation criteria. It then designs an R network based on long-term risk value and a Q network based on long-term discounted expected benefit. The server then uses a deep reinforcement learning algorithm to select participating nodes, comprehensively improving the accuracy and speed of federated learning model training while further protecting user data privacy.

[0006] The present invention comprises the following steps:

[0007] Step 1: Assume a mobile edge computing system contains M edge devices and N participating nodes;

[0008] Step 2: Edge device i constructs a dual deep neural network consisting of an R network and a Q network, 1≤i≤M, and initializes the learning rate α, discount factor δ, and weight parameters of the R network and Q network and

[0009] Step 3: Edge device i observes the number of optional participating nodes within the communication range

[0010] Step 4: Edge device i constructs the current system state vector

[0011] In step 4, the method for the edge device i to construct the system state vector may be: at time k, the edge device i observes the number of optional participating nodes within the current communication range and the bandwidth of the channel between edge device i and each optional participating node The channel gain between edge device i and participating node j is estimated using the channel estimation method Where 1≤i≤M, According to the feedback information, edge device i estimates the training accuracy based on the global model quality trained at the last moment And measure its privacy protection level based on the feedback of the participating nodes at the last moment And measure the difference before and after to get the delay of federated training at the last moment and energy consumption Based on the above information, edge device i constructs its state vector as follows

[0012]

[0013] Step 5: Edge device i selects participating nodes;

[0014] In step 5, edge device i converts the state vector Input into R network and E network respectively. Among them, R network outputs the long-term risk value of all participating node selection strategies in the current state The Q network outputs the long-term discounted expected benefits of all participating node selection strategies in the current state Constructing the participating node selection strategy distribution function by improving the Boltzmann distribution function

[0015]

[0016] Among them, w i Measure the weight of the long-term risk value on the strategy selection of the edge device i. According to the strategy distribution function, the edge device i obtains the node selection measurement

[0017] In step 5, the participating node decision is defined as in Representative node j is selected as the participating node, Indicates that node j is not selected as a participating node,

[0018] Step 6: All selected participating nodes send local model parameters to edge device i;

[0019] In step 6, participating node j sets the local model parameters With fixed power P i Sent to edge device i, where I(·) is an indicator function. If the variable in the brackets is true, the value of the indicator function is 1, otherwise it is 0.

[0020] Step 7: Edge device i aggregates the local model parameters of all participating nodes to obtain a global model parameter, and then distributes this global model parameter to the participating nodes for parameter update;

[0021] In step 7, edge device i receives After the local model parameters of the participating nodes are obtained, the federated averaging algorithm is used to aggregate these parameters and then update the global model parameters. Then Sent to the selected participating nodes.

[0022] Step 8: Edge device i calculates the benefit;

[0023] In step 8, edge device i estimates the training accuracy based on the quality of the trained global model Measure the privacy protection level of participating nodes based on their feedback And measure the difference before and after to get the current federated training delay and energy consumption Then calculate the benefit at the current moment as follows

[0024]

[0025] Among them, i1 ,υ i2 ,υ i3 are weight coefficients, which respectively measure the importance of the privacy protection level, latency, and energy consumption of participating nodes in the benefits.

[0026] Step 9: Edge device i evaluates the risk value;

[0027] In step 9, the required value of the edge device i model training accuracy is defined as The training delay requirement is The privacy protection level requirement for participating nodes is Define the risk value of edge device i as and evaluate as follows

[0028]

[0029] Step 10: Edge device i stores the experience including state vector, participating node selection strategy, benefit, and risk value into the experience pool;

[0030] In step 10, edge device i converts the state vector Participating node selection strategy benefit and risk value Constructed as an experience sequence And stored in the experience pool D.

[0031] Step 11: Edge device i randomly samples Z experiences from the experience pool to construct a batch sample;

[0032] In step 11, edge device i randomly samples Z experiences to construct batch samples Among them, g(η)~U(1,k), U(1,k) is uniformly distributed.

[0033] Step 12: Use the Adam optimization algorithm or stochastic gradient descent algorithm to update the weight parameters of the R network and Q network and

[0034] In step 12, edge device i uses the Adam optimization algorithm to update the weight parameters of the R network and Q network as follows

[0035]

[0036]

[0037] Step 13: Repeat steps 3 to 12 until M edge devices learn a stable participating node selection strategy, i.e., benefit converges to a stable value.

[0038] This paper uses model training accuracy, training latency, and the required privacy protection level of participating nodes as evaluation factors to assess the risk value of strategies and establish a safe exploration mechanism. An R-network based on long-term risk value is designed to avoid exploring dangerous strategies that do not meet the requirements. Combined with a Q-network based on long-term discounted expected benefit, a policy distribution function based on a modified Boltzmann distribution is constructed. This optimizes the participating node selection strategy of each edge device, improves the accuracy and speed of model training, and protects the privacy of participating nodes. DETAILED DESCRIPTION

[0039] In order to more clearly understand the technical content of the present invention, the following embodiments are given to illustrate in detail.

[0040] A risk-averse-based federated secure reinforcement learning approach includes the following steps:

[0041] Step 1: The network includes 3 edge devices and up to 6 optional participating nodes;

[0042] Step 2: Set the participating node selection strategy set. The optional strategy of edge device i is A i ={{0,1},{0,1},{0,1},{0,1},{0,1},{0,1}}.

[0043] Step 3: Edge device i builds a dual-depth network consisting of an R network and a Q network, 1≤i≤3. The R network and the Q network have the same structure, both consisting of fully connected layers, including an input layer, two hidden layers, and an input layer. The input layer consists of 22 neurons, the two hidden layers consist of 64 and 128 neurons respectively, and the output layer consists of 36 neurons. Set the learning rate α to 0.5, the discount factor δ to 0.7, and the risk value weight w i is 1.2, and the benefit function weight parameter υ i1 ,υ i2 ,υ i3 are 1.5, 1, and 1 respectively, the random sampling experience number Z = 32, and the required value of model training accuracy The training delay requirement is The privacy protection level requirement for participating nodes is Initialize the weight parameters of the R network and Q network and

[0044] Step 4: Edge device i observes the number of optional participating nodes within the communication range and the bandwidth of the channel between edge device i and each optional node The channel gain between edge device i and participating node j is estimated using the channel estimation method Where 1≤i≤3,

[0045] Step 5: Based on the feedback information, edge device i calculates the training accuracy of the global model quality obtained by training in the past 100 times And measure the privacy protection level of the participating nodes based on their performance at the last moment Then based on the delay of the federated training at the last moment and energy consumption Build together

[0046]

[0047] Step 6: Edge device i Input into R network and Q network at the same time to obtain strategy distribution Then select the participating node selection strategy

[0048] Step 7: Participating node j sets the local model parameters With fixed power P i Sent to edge device i, where The local model has a local model for each participating node. It can be understood that each participating node maintains a dual-depth network by itself. The local model parameters here can be "network weight parameters of the local dual-depth network".

[0049] Step 8: Edge device i receives the After the parameters of the participating nodes are calculated, the federated averaging algorithm is used to aggregate these parameters and then update the global model parameters. Then Sent to the selected participating nodes.

[0050] Step 9: Edge device i estimates the current model training accuracy based on the results of the previous 99 model trainings and the current model training. And based on the feedback from participating nodes, the privacy protection level of participating nodes is estimated Through the current federated training delay and energy consumption calculate as follows

[0051]

[0052] Step 10: Train the model to the correct accuracy Privacy protection level of participating nodes and the latency of federated training and corresponding requirements and Compare and obtain the risk value of edge device i Right now

[0053]

[0054] Step 11: Edge device i is constructed as an experience sequence And stored in the experience pool D.

[0055] Step 12: Edge device i randomly samples 32 experiences and constructs batch samples

[0056] Step 13: Edge device i uses the Adam optimization algorithm to update the weight parameters of the R network and the E network as follows

[0057]

[0058]

[0059] Step 14: Repeat steps 4 to 13 until the M edge devices learn a stable participating node selection strategy.

[0060] The above embodiments are only preferred embodiments of the present invention and should not be considered to limit the scope of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of the patent of the present invention.

Claims

1. A risk-averse federated security reinforcement learning method, characterized in that: The following steps are involved: 1) Obtain a mobile edge computing system, assuming that the mobile edge computing system includes M edge devices and N nodes; 2) constructing a dual-depth network model for the i-th edge device among the M edge devices, wherein the dual-depth network model includes an R network and a Q network, and initializing the weight parameters of the R network and the Q network and Initialize learning rate α and discount factor δ; 3) The number of optional participating nodes within the communication range of the edge device i at time k 4) The edge device i constructs the state vector of the mobile edge computing system at time k The specific process is as follows: At time k, the edge device i obtains the number of participating nodes within the communication range Get the bandwidth of the channel between the edge device i and each participating node Estimate the channel gain between the edge device i and the participating node j through the channel estimation method Where 1≤i≤M, according to Information, the edge device i estimates the training accuracy of the dual-depth network model at the last moment And measure the privacy protection level of node j at the last moment And measure the latency of training the global model And the energy consumption of training the global model Then, the state vector constructed by the edge device i is as follows: 5) The participating node selected by the edge device i in step 3) sends the local model parameters to the edge device, where the local model is a dual-depth network model; 6) The edge device i aggregates the local model parameters sent by all participating nodes to obtain global model parameters, and then distributes the global model parameters to the participating nodes for parameter update; 7) computing efficiency of the edge device i; 8) The edge device i evaluates a risk value; 9) The edge device i stores the experience including the state vector, the node selection strategy, the benefit, and the assessed risk value into an experience pool; 10) The edge device i randomly samples Z experiences from the experience pool and constructs a batch sample based on the sampled Z experiences; 11) Update the weight parameters of the dual-depth network and 12) Repeat steps 3) to 11) until the M edge devices learn a stable participating node selection strategy.

2. A risk-averse federated security reinforcement learning method according to claim 1, characterized in that: In step 5), the edge device i converts the state vector Input into the R network and Q network in the dual-depth network model respectively; wherein, the R network outputs the long-term risk value of the selection strategy of the node at time k The long-term discounted expected benefit of the node selection strategy at time k of the Q network output And the distribution function of the node selection strategy is constructed using the improved Boltzmann distribution function: The node is defined as in Representative node j is selected as the participating node, Indicates that node j is not selected as a participating node, A i is the action space of edge device i; w i It is the weight of the long-term risk value on the strategy selection of the edge device i node. is the action space A of edge device i i An action component in .

3. The risk-averse federated security reinforcement learning method according to claim 2, wherein: In step 5), the participating node j sets the local model parameters With fixed power P i Sent to the edge device i, where I(·) is the indicator function, when When it is true, the value of the indicator function is 1, otherwise it is 0.

4. The risk-averse federated security reinforcement learning method according to claim 3, wherein: In step 6), the edge device i receives After the local model parameters of the participating nodes are calculated, the federated averaging algorithm is used to aggregate all local model parameters, and then the global model parameters are updated. Then the global model parameters Sent to the participating nodes.

5. The risk-averse federated security reinforcement learning method according to claim 4, characterized in that: In step 7), the specific steps of calculating the benefit of the edge device i are as follows: the edge device i estimates the training accuracy of the dual-depth network model Estimating the privacy protection level of participating nodes And measure the latency of training the global model And the energy consumption of training the global model Calculate the current benefit for: Among them, i1 ,υ i2 ,υ i3 is the weight coefficient.

6. A risk-averse federated security reinforcement learning method according to claim 5, characterized in that: In step 8), the specific steps of evaluating the risk value of the edge device i are as follows: defining the required value of the edge device i model training accuracy as The latency requirement for training the global model is The privacy protection level requirement for participating nodes is Define the risk value of edge device i for:

7. The risk-averse federated security reinforcement learning method according to claim 6, wherein: In step 9), edge device i converts the state vector Participating node selection strategy benefit and risk value Constructed as an experience sequence And stored in the experience pool D. At this time, there are k experience sequences in the experience pool D.

8. The risk-averse federated security reinforcement learning method according to claim 7, characterized in that: In step 10), edge device i randomly samples Z experiences from the experience pool D to construct a batch sample in It represents the g(η)th experience taken from the experience pool D, and obeys the uniform distribution U(1,k), that is, g(η)~U(1,k).

9. The risk-averse federated security reinforcement learning method according to claim 8, characterized in that: In step 11), edge device i uses the Adam optimization algorithm to update the weight parameters of the R network and the Q network as follows: Among them, φ is the weight parameter and δ is the discount factor for updating the weight parameter.

Images