Distributed cloud environment botnet detection method based on federal reinforcement learning

Abstract

The invention discloses a distributed cloud environment botnet detection method based on federal reinforcement learning. The method comprises the following steps: constructing a cloud environment botnet detection model; modeling a node selection problem as a Markov decision process; constructing a reinforcement learning agent network; training the reinforcement learning agent network; and participating in the federal learning training process by using the intelligent agent. According to the method, a reinforcement learning agent is trained to intelligently select a client node set participating in training in each round of training, and the federated learning process is accelerated, so that the negative influence of a data isomerism problem in a distributed cloud environment on the federated learning process is reduced, and the reliability of the federated learning process is improved. And finally, the detection efficiency of the botnet in the distributed cloud environment is effectively improved.

Description

Technical Field

[0001] The present invention relates to a distributed cloud environment botnet detection method based on federated reinforcement learning, and belongs to the technical field of cloud environment network security and machine learning. Background Art

[0002] A botnet is a large-scale network attack tool controlled by malware. It forms a network by infecting a large number of devices and attacks the target through remote control. Once the devices are infected and become zombie hosts, attackers can remotely control these devices and carry out various malicious activities. Compared with traditional network environments, the network structure of cloud environments is more complex, making traditional network security issues more severe in cloud environments. IoT devices that are ubiquitous in cloud environments usually have weak computing and storage capabilities, and relatively limited security measures, so they are easy to become targets of botnet attacks. Once IoT devices are infected on a large scale, attackers may pose a serious threat to the security of the cloud computing environment. For example, they may launch DDoS attacks, send spam, steal information, etc., or directly affect the functions of IoT devices, such as resource exhaustion or service unavailability. More seriously, these attacks may have a negative impact on the entire cloud computing environment, such as network congestion, data leakage, etc., which seriously endangers the security and reliability of the cloud computing environment.

[0003] Many existing botnet detection methods use centralized data processing and training schemes. This approach not only ignores the issues of data privacy and device heterogeneity, but also in cloud environments with large data scale and limited computing and communication resources, centralizing data to a central server for processing will lead to huge bandwidth consumption and unacceptable time delays. In addition, users' private data may be exposed during transmission, increasing the risk of data leakage and abuse. One solution is to use federated learning distributed training to replace the centralized training scheme. However, this scheme also faces the problem of data heterogeneity in distributed cloud environments. The data of different nodes may come from different regions, different users or different application scenarios, resulting in large differences in their data distribution. This difference makes it possible for the local models of some nodes to deviate from the optimization direction of the global model during the federated learning process, reducing the training efficiency. Moreover, due to the differences in computing resources between nodes, inappropriate node selection strategies will lead to inefficient resource utilization, further increasing communication and computing costs. Summary of the invention

[0004] In order to solve the problem of detecting botnets in the above-mentioned distributed cloud environment, the present invention provides a distributed cloud environment botnet detection method based on federated reinforcement learning. The method first combines the characteristics of distributed cloud environment botnets to construct a cloud environment botnet detection model based on time-space feature fusion, and then intelligently selects federated learning participating nodes through deep reinforcement learning during the distributed training process, so as to ultimately improve the detection efficiency of botnet detection in the distributed cloud environment.

[0005] The technical solution adopted in the present invention is:

[0006] A distributed cloud environment botnet detection method based on federated reinforcement learning includes the following steps:

[0007] S1: Aiming at the characteristics of botnets in distributed cloud environments, a botnet detection model based on spatiotemporal feature fusion is constructed;

[0008] S2: Model the client node selection problem for each round of training in federated learning as a Markov decision process;

[0009] S3: DDQN reinforcement learning algorithm based on priority experience replay mechanism to build a reinforcement learning agent network;

[0010] S4: Training the reinforcement learning agent network;

[0011] S5: Use the trained agent to participate in the federated learning training process.

[0012] Furthermore, S1 is specifically: the botnet detection model based on spatiotemporal feature fusion is specifically composed of a deep residual network ResNet-18 and a denoising autoencoder model LSTM-DAE based on a long short-term memory network. ResNet-18 is used to extract the spatial features of network traffic data, and LSTM-DAE is used to extract the temporal features of network traffic data.

[0013] Further, S2 is specifically: the Markov decision process specifically includes a state space S, an action space A and a reward function R; wherein the state S at time t t is the local model weight of all client nodes, the action space is to select 1 node from all client nodes in each round of communication, and the reward function is set to represents the accuracy of the validation set under the global model, ω represents the set target accuracy, and α represents a constant greater than 1.

[0014] Further, S3 is specifically as follows: the reinforcement learning agent network structure specifically includes a target network and a training network, both of which are composed of three fully connected layers. The number of neurons in these layers is 256, 256 and 50 respectively. The capacity of the experience pool is 1000, and 5 samples are randomly sampled from it for training each time. The discount factor γ is set to 0.9, and the action selection strategy adopts the ε-greedy strategy, where ε is set to 0.1.

[0015] Further, S4 is specifically as follows: calculating a time difference target, defining a loss function according to the time difference target, and minimizing the loss function by a gradient descent method to update the network parameters of the training network. The target network is updated every 5 trainings. The Adam optimizer is used for network update, and the learning rates of the target network and the training network are both set to 0.0001.

[0016] Furthermore, S5 is specifically as follows: inputting the state information at time t into the intelligent agent network, the network outputs a Q value, the Q value is used to evaluate the contribution of each client, and the clients corresponding to the top k Q values ​​are used as the best client subset in the current state; the client nodes in the selected subset download the current global model parameters from the central server, and perform model training locally; each client uploads its updated local model parameters to the central server; the central server executes a federated aggregation algorithm to update the global model; and the above process is continuously executed until the global model reaches the target accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is an overall flow chart of a distributed cloud environment botnet detection method based on federated reinforcement learning disclosed in an embodiment of the present invention;

[0018] Figure 2 It is a structural diagram of a distributed cloud environment botnet detection model of the present invention;

[0019] Figure 3 It is a schematic diagram of the PER-DDQN network structure of the present invention. DETAILED DESCRIPTION

[0020] In order to enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application are clearly and completely described below. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0021] refer to Figure 1This embodiment provides a distributed cloud environment botnet detection method based on federated reinforcement learning, which specifically includes the following steps:

[0022] Step 1: Based on the characteristics of botnets in distributed cloud environments, a botnet detection model based on spatiotemporal feature fusion is constructed. Specifically, it consists of a deep residual network ResNet-18 and a denoising autoencoder model LSTM-DAE based on a long short-term memory network. Among them, ResNet-18 is used to extract the spatial features of network traffic data, and LSTM-DAE is used to extract the temporal features of network traffic data. Then, the feature vector of spatiotemporal feature fusion is obtained through vector concatenation operation. The specific network structure of ResNet-18 can be divided into 6 blocks. Block1 includes a convolution layer with 3 input channels and 64 output channels, a 7×7 convolution kernel, a step size of 2, and a padding size of 3. It is followed by a batch normalization layer and a ReLU activation layer, as well as a 3×3 maximum pooling layer with a step size of 2 and a padding size of 1. Block2 to Block5 constitute the core part of the network. Each Block consists of two residual blocks, each of which includes convolutional layers and batch normalization layers with different configurations, and cooperates with activation layers to enhance nonlinear expression capabilities. In particular, the step size of the convolutional layer in Block2 is 1, and the number of channels remains at 64; while in Block3 to Block5, the step size of some convolutional layers is increased to 2 to gradually increase the abstract level of features, and the number of channels is increased accordingly, which are 128, 512, and 1024 respectively. The network finally compresses features through the 1×1 average pooling layer and Flatten layer of Block6 to facilitate subsequent feature fusion.

[0023] The LSTM-DAE structure consists of an encoder and a decoder, where the number of hidden neurons in the LSTM layer of the encoder and decoder is set to 256, 128, 64 and 128, 256 respectively, and the LSTM layer for time series feature extraction is set to 64 neurons. In the Dropout layer of LSTM-DAE, the dropout probability is set to 0.5 to reduce the risk of overfitting of the model.

[0024] Classification function selection: Use Softmax() as the activation function, which is used in the multi-classification process. Its formula is as follows:

[0025]

[0026] Among them, Z i is the output value of the ith node, C is the number of output nodes, that is, the number of categories of classification; the output value of the multi-classification is converted into a probability distribution ranging from [0,1] and 1 through the Softmax() function;

[0027] Loss function selection: The cross entropy loss function is used. In the binary classification problem, its formula is as follows:

[0028]

[0029] Among them, y i represents the label of sample i, the positive class is 1, the negative class is 0, and p i Indicates the probability that sample i is predicted to be a positive class;

[0030] Optimization function selection: Adopt Adam optimizer to accelerate parameter update during training.

[0031] Step 2: Model the node selection problem in federated learning as a Markov decision process. In the Markov decision framework, define the following four-tuple components (S, A, R, P): state space S. The state S at time t t Indicated as S t =(w t (1) , w t (2) , w t (3) , ..., w t (N) ), where w t (1) , w t (2) , w t (3) , ..., w t (N) is the local model weight of N client nodes. Action space A. Theoretically, the action space is to select K nodes from N client nodes in each round of communication. However, this design will lead to an overly large action space. Therefore, in practice, only one node is selected in each round of action selection, that is, the actual action space A = {1, 2, ..., N}. Reward function R. The reward after each round w t represents the accuracy of the validation set under the global model, ω represents the set target accuracy, and α represents a constant greater than 1. Transition probability P. represents the probability of transitioning to the next state after taking an action in the current state. The definitions of state and action form a strategy ∏: S→A, which is a mapping from state space to action space.

[0032] definition is the cumulative reward from time t to time T. Let s t is the state at time t, a t For t The action selected according to the strategy π in the state can be calculated by calculating the following state-action function Q π (st , a t ) Evaluate the decision:

[0033]

[0034] Where γ∈[0,1] is a discount factor, which is used to measure the importance of future rewards. The larger the γ, the greater the impact of future rewards on current decisions. The above formula can be further written as:

[0035]

[0036] Step 3: Build a reinforcement learning agent network based on the DDQN reinforcement learning algorithm based on the priority experience replay mechanism, refer to Figure 3 ,The reinforcement learning agent network structure specifically includes the target network and the training network,both consist of three fully connected layers. The number of neurons in these layers are 256, 256, and 50,respectively.

[0037] Step 4: Train the reinforcement learning agent network. To alleviate the problem of overestimation of Q value in the original DQN algorithm, DDQN introduces two neural networks with different parameters: a training network is used to select the optimal action, and the other target network is used to evaluate the value of this action. This mechanism can reduce the accumulation of positive errors caused by a single valuation network. Specifically, the algorithm updates the parameters of the training network during training and keeps the parameters of the target network unchanged. After a certain number of iterations, the parameters of the training network are copied to the target network. The temporal difference target for calculating DDQN is as follows:

[0038]

[0039] Select action a * It is based on the training network Q θ The output of the action is calculated using the target network This strategy switching effectively separates the two steps of action selection and value assessment, thereby reducing estimation bias. The loss function is defined according to the temporal difference target, and the loss function is minimized by the gradient descent method to update the network parameters of the training network. The target network is updated every 5 training times. The Adam optimizer is used for network update, and the learning rates of the target network and the training network are set to 0.0001. The capacity of the experience pool is 1000, and 5 samples are randomly sampled from it for training each time. The discount factor γ is set to 0.9, and the action selection strategy adopts the ε-greedy strategy, where ε is set to 0.1.

[0040] Step 5: Use the trained agent to participate in the federated learning training process. Input the state information at time t into the agent network, and the network outputs the Q value, which is used to evaluate the contribution of each client, and the clients corresponding to the top k Q values ​​are used as the best client subset in the current state; the client nodes in the selected subset download the current global model parameters from the central server and perform model training locally; each client uploads its updated local model parameters to the central server; the central server executes the federated aggregation algorithm to update the global model; the above process is continuously performed until the global model reaches the target accuracy.

[0041] Finally, it should be noted that the method steps described in the embodiments disclosed in the present invention can be configured to build a programming environment for implementation. Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A distributed cloud environment botnet detection method based on federated reinforcement learning, characterized in that: The following steps are involved: S1: Aiming at the characteristics of botnets in distributed cloud environments, a botnet detection model based on spatiotemporal feature fusion is constructed; S2: Model the client node selection problem for each round of training in federated learning as a Markov decision process; S3: DDQN reinforcement learning algorithm based on priority experience replay mechanism to build a reinforcement learning agent network; S4: Training the reinforcement learning agent network; S5: Use the trained agent to participate in the federated learning training process.

2. The distributed cloud environment botnet detection method based on federated reinforcement learning according to claim 1 is characterized in that: The botnet detection model based on spatiotemporal feature fusion in S1 is specifically composed of a deep residual network ResNet-18 and a denoising autoencoder model LSTM-DAE based on a long short-term memory network; wherein ResNet-18 is used to extract the spatial features of network traffic data, and LSTM-DAE is used to extract the temporal features of network traffic data, and finally a feature vector of spatiotemporal feature fusion is obtained through a vector concatenation operation.

3. The distributed cloud environment botnet detection method based on federated reinforcement learning according to claim 1 is characterized in that: The Markov decision process in S2 specifically includes: state space S, action space A and reward function R; state S at time t t is the local model weight of all client nodes, the action space is to select 1 node from all client nodes in each round of communication, and the reward function is set to where w t represents the accuracy of the validation set under the global model, ω represents the set target accuracy, and α represents a constant greater than 1.

4. The distributed cloud environment botnet detection method based on federated reinforcement learning according to claim 1 is characterized in that: The reinforcement learning agent network structure in S3 specifically includes a target network and a training network, both of which are composed of three fully connected layers. The number of neurons in the three fully connected layers is 256, 256 and 50 respectively. The capacity of the experience pool is 1000, and 5 samples are randomly sampled from it for training each time. The discount factor γ is set to 0.9, and the action selection strategy adopts the ε-greedy strategy, where ε is set to 0.

1.

5. The distributed cloud environment botnet detection method based on federated reinforcement learning according to claim 1 is characterized in that: The reinforcement learning agent network training process in S4 specifically includes: The temporal difference target is calculated, and the loss function is defined according to the temporal difference target. The loss function is minimized by the gradient descent method to update the network parameters of the training network. The target network is updated every 5 training times. The Adam optimizer is used for network update, and the learning rates of the target network and the training network are both set to 0.0001.

6. The distributed cloud environment botnet detection method based on federated reinforcement learning according to claim 3 is characterized in that: The federated learning training process in which the agent in S5 participates specifically includes: The state S at time t t Information is input into the agent network, and the network outputs a Q value, which is used to evaluate the contribution of each client, and the clients corresponding to the top k Q values ​​are used as the best client subset in the current state; the client nodes in the selected subset download the current global model parameters from the central server and perform model training locally; each client uploads its updated local model parameters to the central server; the central server executes the federated aggregation algorithm to update the global model; The above process is continuously performed until the global model reaches the target accuracy.

Images