A botnet traffic detection method based on deep reinforcement learning

Abstract

The application designs a botnet traffic detection method based on deep reinforcement learning, constructs an intelligent OneR-DQN detection model based on the OneR classifier in machine learning and the deep Q network of deep reinforcement learning, and faces the existing botnet traffic dataset; first, the dataset is merged, data is preprocessed, and features that can be used for classification and training are reserved; second, the OneR classifier is used to test and judge each feature in the dataset one by one, and appropriate features are selected for training by the DQN model; finally, the experience pool mechanism unique to DQN is used to continuously extract independent experience and training samples for cross-training, thereby improving the accuracy of detection; the mixed dataset of the four CIC datasets CIC-IDS2017, CIC-DoS2017, CIC-IDS2018 and CIC-DDoS2019 is used to verify the detection model constructed by the application, and the result shows that the accuracy of the model constructed by the application is 99.08%, and the artificial intelligence automatic identification and judgment for botnet traffic are realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of network traffic security and artificial intelligence, and particularly relates to a botnet traffic detection method based on deep reinforcement learning. BACKGROUND

[0002] Nowadays, botnet intrusion methods are constantly being updated. Many hosts are infected when they enter phishing websites with the same interface as normal websites, and are then used by botnet families to attack other websites when the users do not use the hosts. Many websites are also forced to shut down due to distributed denial of service (DDoS) attacks. Although experts have built many detection models for IRC botnets in the early stage, the structure of botnets is also evolving from the early IRC mode to the HTTP and P2P modes. Hackers use DDoS, spam attacks, brute force attacks, penetration attacks and other attack methods to infect a large number of hosts, forming large and small botnet families. Many users may unknowingly be used to engage in illegal activities, which not only has a negative impact on the websites of major operators and merchants, but also affects the reputation and personal computers of individual users. Due to the end-to-end mode of P2P, it is difficult to know the location of the botnet operator host in the investigation, and only the detection ability can be continuously improved to avoid the host being infected by the botnet, which poses a great threat to the world's network security problem.

[0003] Existing detection methods include feature analysis, behavior analysis, machine learning, deep learning and other technologies, which detect and identify the traffic of botnets through real-time monitoring, traffic analysis, feature recognition and other means. Although there are many detection methods for botnet traffic, it is not satisfactory to use a general detection model to deal with various botnet datasets with changing feature states. Some models that perform well on some datasets perform poorly on other datasets, and the accuracy is a concern. SUMMARY

[0004] In view of the deficiencies in the prior art, the present application designs a botnet traffic detection method based on deep reinforcement learning, and proposes a botnet traffic joint measurement model, which is composed of the OneR algorithm in machine learning (ML) and the deep Q network (DQN) algorithm in deep reinforcement learning (DRL). The ML classifier is used to classify the feature values of the data set, and some feature values with the best judgment ability are selected. Then, the feature values are given to the DRL. The DRL has the ability to make one-sided professional judgments and to change its own strategy when the environment changes due to its own selection, so that the model can continuously adjust its feedback learning ability and make judgments on the continuously updated botnet data set. In addition, the DQN adds high dimensionality of deep learning to the original reinforcement learning (ML) Q-learning algorithm, so that the model can rely on high-dimensional feature input to judge whether the traffic is botnet traffic, thereby enhancing the accuracy and robustness of the model judgment.

[0005] A botnet traffic detection method based on deep reinforcement learning, specifically comprising the following steps:

[0006] Step 1: Obtain the botnet traffic data set, preprocess it, and obtain the botnet traffic data set with float features divided into training set and validation set;

[0007] Step 1.1: Obtain the botnet traffic data set by packet capture or download. For data sets with similar content and same labels, merge them according to the same labels;

[0008] Step 1.2: Delete the pollutant features that are not conducive to judgment and the features that have no distinguishing ability;

[0009] Step 1.3: Balance the data set by random undersampling and oversampling;

[0010] Step 1.4: Label encoding of features in the data set;

[0011] Step 1.5: After the above data set preprocessing is completed, divide the data set into training set and validation set, and store the divided data set locally;

[0012] Step 1.6: Only keep the float features of the data samples in the data set, and discard all other features;

[0013] Step 2: Based on Step 1, use the OneR classifier of machine learning to classify the features of the training set, and screen out the features for training the DQN model:

[0014] Step 2.1: Separate independent variables xs and dependent variables y; If the input target is not a list, then the independent variables are all columns except the target. If the target is a list, then the independent variables are all columns except the columns contained in the target;

[0015] Step 2.2: Use the decision tree classifier of machine learning as the core of OneR classifier, specifically:

[0016] DecisionTreeClassifier(max_depth=1,criterion='gini')

[0017] Where the depth of the decision tree is 1 layer, and the Gini coefficient is used as the standard for evaluating each node in the construction process of the decision tree classifier. Then fit the training set, and then predict the ability of various features using the prediction function:

[0018] preds=rootnode.predict(X_test[feature].array.reshape(-1,1))

[0019] preds_tr=rootnode.predict(X_train[feature].array.reshape(-1,1))

[0020] According to the above formula, calculate the evaluation index value between the prediction result and the true label. If it is greater than 0.5, save the feature name, index value and training set prediction result of this feature;

[0021] Step 2.4: According to the evaluation index value, screen out the features with direct distinguishing ability, specifically roc_auc_score>0.5, then take the average value of the test results of these features on the training set to get an overall prediction result vector; Calculate the ROC curve parameters between the overall prediction result and the true label on the training set; The ROC calculation method is:

[0022] ROC=TPR-FPR

[0023] Where TPR is the true positive rate, i.e. the probability of correct prediction, and FPR is the false positive rate, i.e. the probability of incorrect prediction. Finally, generate an image using the Matplotlib library according to the ROC curve and the area surrounded by the x-axis, i.e. the size of the AUC score, to determine the feature with the best performance;

[0024] Step 3: Construct a DQN algorithm detection model based on deep reinforcement learning, combine step 2 to get the final intelligent OneR-DQN detection model, and verify it using the validation set;

[0025] The DQN algorithm detection model is composed of four categories: agent, model, experience pool ReplayMemory and DQN algorithm;

[0026] Step 3.1: Build the Model class, which serves as the processing layer for input data;

[0027] Build a three-layer fully connected forward neural network, where the first and second layers use the ReLU activation function, and the last layer outputs the action space without using the activation function; define the value function, which outputs the Q values of all actions [Q(s,a1), Q(s,a2), Q(s,a3)…]; where Q represents the state, including the environment s and action a; Q(s,a1) represents the reward value obtained by the model in this operation, written as a sequence;

[0028] Step 3.2: Build the DQN algorithm class, which defines how the model judges the input features and distinguishes whether the traffic is abnormal, references the forward network of the model class, defines the γ factor specific to DQN as the decay factor of the reward value reward, sets the learning rate LR to allow DQN to continuously iterate towards the optimal solution; Then define the prediction predict, learning learn and synchronization model sync_target three functions; After each prediction, the model will obtain feedback and adjust to maximize the Q value;

[0029] The Q value calculation method of DQN is:

[0030]

[0031] Where maxQ represents the maximum Q value of the action during prediction; γ is the decay factor; y t represents the reward value of Q value, rt represents the loss function, and w is the parameter adjusted during neural network training;

[0032] The loss function calculation method of DQN is:

[0033] L i (θ i )=E (s,a,r,s')～U(D) [(y i -Q(s,a;θ i )) 2 ]

[0034] Where s represents state, a represents action, r represents immediate reward, s' represents next state, and θ represents parameter of neural network i represents parameter of neural network at i-th iteration, Q(s, a; θ i ) represents Q value function of action a taken in current state, U(D) represents experience set uniformly sampled from experience replay pool, D represents experience pool itself, y i represents target Q value, which is calculated as follows:

[0035]

[0036] Where decay factor γ is used to measure the influence of immediate reward on future reward, and The network where is located is called target network; by calculating loss function, the model constantly seeks optimal solution; DQN uses reward mechanism, experience replay and target network technology;

[0037] Then constantly update Q parameter, so that Q(s t ,a t ) approaches y t , using gradient descent to update, and the cycle continues until convergence;

[0038] Step 3.3: build Agent class, which is responsible for the decision-making process of the agent; define ε-greed greedy value, so that the agent randomly selects action with a certain probability for exploration; at the same time, the greedy value is constantly reduced, so that the training gradually converges; after prediction by the upper DQN, the Agent gets the input value, and builds the computation graph for predicting action and updating Q network, then defines the sample function to determine whether to return random operation or Agent predicted operation; define predict function, select the maximum Q value as the optimal action; finally, define feed function to record all the obtained values, and unify different variable types to float type;

[0039] Step 3.4: build ReplayMemory class, which is used to store data used by Agent, and cross these data with untrained data again to return to DQN algorithm and Agent class for further judgment;

[0040] The experience replay operation builds ReplayMemory experience pool to store n experiences obtained after Agent judgment, and only when the experience pool is full, the old data will be discarded, and the stored content is:

[0041]

[0042] Where (s t ,a t ,rt ,s t+1 ) represents a set of tuples of the agent performing actions in the environment, and these tuples are used to train the DQN model to predict the Q value of taking a certain choice in a given state; in training, a batch of experiences [(s t ,a t ,r t ,s t+1 ) … (s m ,a m ,r m ,s m+1 )] is randomly selected from the experience pool, so that the correlation between data can be eliminated, and the DQN model can be trained better;

[0043] Step 3.5: Define the training period and the reward value obtained after each agent operation, and observe the training according to the result obtained at each step; after the model reaches the training iteration number or meets the training accuracy requirement, the intelligent OneR-DQN detection model training is completed, then the verification set is put into the trained model, the model is verified, and the record of the judgment ability of the intelligent OneR-DQN detection model is generated;

[0044] Step 4: After the different botnet data sets are preprocessed according to step 1, they are input into the intelligent OneR-DQN detection model composed of step 2 and step 3 to obtain the attack detection judgment result.

[0045] The beneficial technical effects of the present application are as follows:

[0046] The present application is aimed at various types of botnet traffic attacks, and labels the traffic features by normalization, selects the most influential features using the OneR classifier, and on this basis, constructs a deep reinforcement learning DQN model, which gradually masters the judgment of botnet traffic features through continuous self-training, realizes efficient identification and detection of botnet traffic attacks. The present application uses a deep reinforcement learning deep Q network model to realize artificial intelligence automatic identification and judgment of botnet traffic. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 A flowchart of the present application embodiment for combining four types of data and screening out suitable features;

[0048] Figure 2 A flowchart of the present application embodiment for classifying botnet traffic features using a machine learning OneR classifier;

[0049] Figure 3 A block diagram of the intelligent detection method of the intelligent OneR-DQN detection model of the present application embodiment for botnet traffic. DETAILED DESCRIPTION

[0050] The application will be further described below in conjunction with the accompanying drawings and examples.

[0051] A botnet traffic detection method based on deep reinforcement learning, as shown in the accompanying drawings, specifically comprising the following steps: Figure 3

[0052] Step 1: Obtain the botnet traffic dataset, preprocess it, and obtain the botnet traffic dataset with float features divided into training set and validation set; the process is shown in the accompanying drawings Figure 1

[0053] Step 1.1: Download CIC dataset from CIC official website, including CIC-IDS2017, CIC-DoS2017, CIC-IDS2018, CIC-DDoS2019 four datasets, and merge them according to the same label.

[0054] The Label label in this paper is:

[0055] Among them, the eight labels are: normal traffic (Benign), distributed denial of service (DDoS), brute force (Bruteforce), penetration (Infiltration), botnet (Botnet), denial of service (DoS), web attack (Webattack) and port scan (Portscan), each label has four feature sets. Through the reading of pandas library, there are 9167581 data and 79 dimensional features.

[0056] Step 1.2: Use the pandas library in python to delete some features that are not conducive to judgment, that is, pollutant features, to reduce the data dimension; at the same time, delete the features that have no distinguishing ability, and ensure that the dataset has no null value (i.e. None value);

[0057] This paper manually retrieves and sorts the data set pollution severity according to the features, and uses the drop function to discard nine features:

[0058] Table 1: 9-dimensional features deleted from the data set with serious pollution;

[0059]

[0060] Although Active Std and Idle Std also affect the dataset, the degree of feature pollution on the dataset is not very large and not obvious enough, so they are retained.

[0061] ​​In the process of looking for pollution characteristics, this paper found that there are 11 features with 0 prediction ability, which have no meaning in any classification model, and even some feature values are None. For these features, this paper also uses the drop function to discard:

[0062] Table 2 11-dimensional features deleted without any meaning;

[0063]

[0064] Through the above three steps, this paper obtains a clean sample data set (CIC-Collection) composed of four data sets. There are detailed labels for single attacks, and there are corresponding more extensive attack categories, there is no None value, and there is no repeated sample. Contains 59-dimensional features, upon inspection, it is found that the ClassLabel and Label feature contents are consistent, so one dimension is deleted, and finally a CIC-Collection data set with 9167581 data and 58-dimensional features is obtained:

[0065] Table 3 CIC-Collection Botnet Dataset;

[0066]

[0067] Because the data set has too many data samples, when using the model to train according to the data samples in the subsequent use, some of them are randomly cut off for training and testing.

[0068] Step 1.3: Balance the data set. Use the imbalanced-learn library in python to randomly undersample and oversample the data set to obtain data samples that can be used for model training.

[0069] Step 1.4: Label encoding of the features of the data set to meet the calling rules of the DQN model.

[0070] Step 1.5: Divide the data set into 80% training set and 20% validation set, and save the data set in the file for subsequent use, where the training set is used to train the model, and the validation set is used to verify the performance of the model;

[0071] Step 1.6: Data samples except float type (float) features are not suitable for classifiers and model reading, so all other features are discarded through the drop function of the pandas library. The final features that can be used for OneR classifier and DQN model classification and training have 44 dimensions;

[0072] Step 2: After obtaining the CIC-Collection dataset by combining the four botnet traffic attack datasets, first use the OneR classifier of machine learning to perform feature classification, filter out the features suitable for DQN model training, and the operation flow chart is as shown in the accompanying Figure 2

[0073] Step 2.1: Separate independent variables (xs) and dependent variables (y). If the input target is not a list, then the independent variables are all columns except the target. If the target is a list, then the independent variables are all columns except the columns contained in the target.

[0074] Step 2.2: Use the decision tree classifier of machine learning as the core of OneR classifier, this step is:

[0075] DecisionTreeClassifier(max_depth=1,criterion='gini')

[0076] Wherein, the depth of decision tree is 1 layer, and the Gini coefficient is used as the standard for evaluating each node in the construction process of decision tree classifier; then fit the training set, and then predict the ability of various features, using the prediction function:

[0077] preds=rootnode.predict(X_test[feature].array.reshape(-1,1))

[0078] preds_tr=rootnode.predict(X_train[feature].array.reshape(-1,1))

[0079] Calculate the evaluation index value between the predicted result and the true label, if greater than 0.5, save the feature name, index value, expected result of the validation set and the predicted result of the training set of this feature;

[0080] Step 2.3: Save the predicted results in a list, and convert it into a data frame, call the metrics method in sklearn library to directly calculate the evaluation index value roc_auc_score, and arrange it in descending order to get the required features and index values:

[0081] pd.DataFrame(data=results,columns=['feature','roc_auc_score','fitted_models',

[0082] ​predictions['preds_train']).sort_values(by='roc_auc_score', ascending=False)

[0083] Step 2.4: According to the index value, the features with direct discrimination ability (roc_auc_score>0.5) are screened out, and then for these useful features, the average value of their respective test results on the validation set and the training set is obtained to get an overall prediction result vector. The ROC curve parameters between the overall prediction result and the true label on the training set are calculated. The ROC calculation method is:

[0084] ROC = TPR - FPR

[0085] Where TPR is the true positive rate, i.e. the probability of correct prediction, and FPR is the false positive rate, i.e. the probability of incorrect prediction. Finally, the obtained ROC curve is generated into an image using the Matplotlib library, so as to intuitively distinguish the best performing features. We screen the features for each attack according to the size of the AUC score attached to the generated image, and obtain the screening feature AUC score graph for each attack category as shown in Table 4.

[0086] Table 4 Screening feature AUC score graph for each attack category

[0087]

[0088] Although only the simplest feature classification is done in Step 2, the results obtained by these numerical values are very different from the results obtained by the more complex model in Step 3, but it cannot be denied that OneR classifier is still an excellent classifier for efficiently and quickly screening the most influential features in machine learning algorithms.

[0089] Step 3: For the features screened out in Step 2, a DQN algorithm detection model based on deep reinforcement learning is constructed:

[0090] The DQN model is composed of four categories: agent (Agent), model (Model), experience pool (ReplayMemory), and DQN algorithm.

[0091] Step 3.1: Construct the Model class, which serves as the processing layer for input data. A three-layer fully connected layer forward network is created, with the first and second layers using ReLU activation functions, and the last layer outputting dimensions for the action space, so no activation function is applied. Then define the value function to output the Q values for all actions [Q(s, a1), Q(s, a2), Q(s, a3)…], where Q is a state containing the environment s and action a. Q(s, a1) means how much reward value is obtained in this operation, which is the Q value. The reason for writing it as a sequence is that the judgment is continuous, and the Q value needs to be constantly collected to constantly change the strategy.

[0092] Step 3.2: Construct the DQN algorithm class, which defines how the model judges the input features and determines whether the traffic is abnormal. The forward network of the model class is referenced, and the DQN-specific gamma factor is defined as the reward decay factor. The learning rate (LR) is set to allow the DQN model to iteratively learn in one direction, so that it moves towards the optimal solution in each iteration. Then define the predict, learn, and sync_target functions. After each prediction, the model receives feedback and adjusts to maximize the Q value.

[0093] The Q value calculation method of DQN is:

[0094] y t =r t +γ·max a Q(s t+1 ,a;w)

[0095] Where maxQ represents the maximum Q value of the action during prediction. As the number of steps increases, it is impossible to obtain the maximum Q value at each step, so the decay factor γ is needed to reduce the impact of late decisions on the overall result; y t represents the reward value of the Q value, r t represents the loss function, and w is the parameter adjusted during neural network training. By constantly updating the Q value, the model tends to make its own judgments based on the Q value in a new environment, ultimately maximizing the Q value to maximize y t , completing the training of the model;

[0096] The loss function calculation method of DQN is:

[0097] L i (θ i )=E (s,a,r,s')～U(D) [(yi Q(s, a; θ i )) 2 ]

[0098] where s represents state, a represents action, r represents immediate reward, s' represents next state, θ i represents the parameters of the neural network at the i-th iteration, Q(s, a; θ i ) represents the Q value function of taking action a in the current state, U(D) represents the experience set sampled uniformly from the experience replay pool, D represents the experience pool itself, y i represents the target Q value, which is calculated as:

[0099]

[0100] where the decay factor γ measures the degree of influence of immediate reward on future reward, and the network is called target network, thereby preventing overfitting; by calculating the loss function, the model can constantly find the optimal solution. DQN uses reward mechanism, experience replay and target network technology, which eliminates the correlation between data and effectively solves the problems of unstable training and shock.

[0101] After that, the Q parameters are constantly updated so that Q(s t ,a t ) is as close to y t as possible, and the gradient descent is used for updating work, which is repeated until convergence.

[0102] The main process of the existing DQN model is as follows:

[0103]

[0104]

[0105] Step 3.3: Build the Agent class, which is responsible for the decision-making process of the agent. By defining the ε-greed value, the agent randomly selects actions with a certain probability for exploration. At the same time, the greed value is constantly reduced, so that the training gradually converges; after the upper DQN makes a prediction, the Agent gets the input value and builds a computation graph for predicting actions and updating the Q network, then defines the sample function to determine whether to return a random operation or the Agent's predicted operation. Define the predict function to select the maximum Q value as the optimal action. Finally, define the feed function to record all the obtained values, and uniformly convert different variable types to float type;

[0106] Step 3.4: Construct the ReplayMemory class, which is used to store the data used by the Agent, and pass these data back to the DQN algorithm and Agent class again for re-judgment.

[0107] Experience replay breaks up the sequence of data samples, eliminating the correlation between data, so that the data meets the independent and identically distributed, thereby reducing the variance of parameter update and improving the convergence speed. And can reuse experience, high data utilization, especially useful for data acquisition difficulties. When doing reinforcement learning, often the most time-consuming step is to interact with the environment, and training the network is relatively fast because using GPU training is very fast. Using the replay buffer can reduce the number of interactions with the environment, and experience does not need to come from a certain policy. Some experience obtained by the past policy can be used multiple times in the replay buffer and repeatedly reused.

[0108] The ReplayMemory storage pool constructed by the experience replay operation stores n experiences obtained after the Agent makes a judgment. Only when the storage pool is full will the old data be discarded. The stored content is:

[0109]

[0110] Among them, (s t ,a t ,r t ,s t+1 ) represents a set of tuples of the agent performing actions in the environment, and these tuples are used to train the DQN model to predict the Q value of taking a certain choice in a given state. In training, a batch of experiences [(s t ,a t ,r t ,s t+1 ) … (s m ,a m ,r m ,s m+1 )] is randomly selected from the experience pool, which can eliminate the correlation between data and make the DQN model train better.

[0111] Step 3.5: Define the training period and evaluate the reward value obtained after each agent operation, and generate the model's zombie network traffic judgment accuracy rate to observe the training situation. The calculation formula of the accuracy rate is:

[0112]

[0113] Wherein, TP represents the number of positive samples predicted as positive examples; FP represents the number of negative samples predicted as positive examples; TN represents the number of positive samples predicted as negative examples; FN represents the number of negative samples predicted as negative examples. After the model reaches the training iteration number or meets the training accuracy requirement, the intelligent detection model training is completed; then the verification set is put into the trained model, and the judgment ability of the DQN model is recorded.

[0114] Step 4: After the different botnet data sets are preprocessed according to steps 1.3-1.6, they are input into the intelligent OneR-DQN detection model composed of steps 2 and 3 to give the attack detection judgment result.

Claims

1. A method for detecting botnet traffic based on deep reinforcement learning, characterized in that, Specifically the following steps are: Step 1: Obtain botnet traffic dataset, preprocess it to obtain botnet traffic dataset with float features divided into training set and validation set; Step 2: Based on step 1, use machine learning OneR classifier to classify features of the training set, and select features for training DQN model; Step 3: Construct DQN algorithm detection model based on deep reinforcement learning, combine step 2 to obtain the final intelligent OneR-DQN detection model, and use the validation set to verify it; The DQN algorithm detection model is composed of four categories: agent, model, replay memory and DQN algorithm; Step 4: After data preprocessing according to step 1, input the intelligent OneR-DQN detection model composed of step 2 and step 3 to obtain attack detection judgment result; Step 2 is specifically: Step 2.1: Separate independent variables xs and dependent variables y; If the input target is not a list, then the independent variables are all columns except target; If target is a list, then the independent variables are all columns except the columns contained in target; Step 2.2: Use machine learning decision tree classifier as the core of OneR classifier, specifically: Wherein, the depth of decision tree is 1 layer, and gini coefficient is used as the standard for evaluating each node in the construction process of decision tree classifier; Then fit the training set, and then predict the ability of various features, using the prediction function: According to the above formula, calculate the evaluation index value between the prediction result and the true label, if greater than 0.5, save the feature name, index value and training set prediction result of this feature; Step 2.4: According to the evaluation index value, select the features with direct distinguishing ability, specifically roc_auc_score>0.5, then take the average value of the test results of these features on the training set to obtain an overall prediction result vector; Calculate the ROC curve parameters between the overall prediction result and the true label on the training set; The ROC calculation method is: Wherein, TPR is the true positive rate, that is, the probability of correct prediction, and FPR is the false positive rate, that is, the probability of incorrect prediction; Finally, the obtained ROC curve is generated into an image by using Matplotlib library, and the size of AUC score surrounded by ROC curve and x coordinate axis is used to judge the best performing feature; Step 3 is specifically: Step 3.1: Construct Model class as input data processing layer; A three-layer fully connected forward neural network is constructed, where the first and second layers use ReLU activation functions, and the last layer outputs the action space without using activation functions; define the value function, output the Q value of all actions [Q(s, a1), Q(s, a2), Q(s, a3)…]; where Q represents the state, including the environment s and action a; Q(s, a1) represents the reward value obtained by the model in the operation at this step, which is written in the form of a sequence; Step 3.2: Construct the DQN algorithm class, define how the model judges the input features, and distinguish whether the traffic is abnormal; refer to the forward network of the model class, define the DQN-specific The factor is set as the decay factor of the reward value reward, and the learning rate LR is set to make the DQN continuously iterate towards the optimal solution. Then define the prediction predict, learning learn and synchronization model sync_target three functions; after each prediction, the model will get feedback and adjust to maximize the Q value; The Q value calculation method of DQN is: where maxQ represents the maximum Q value of the action at the prediction time; is a decay factor; is a reward value representing the Q value, is a loss function, and w is a parameter adjusted during neural network training. The loss function calculation method of DQN is: wherein, represents a state, represents an action, represents an immediate reward, represents a next state, represents a parameter of the neural network at the i-th iteration, represents a Q-value function of taking an action in the current state, represents a uniformly sampled set of experiences from the experience replay pool, D is the experience pool itself, represents a target Q-value, which is computed as: where the decay factor to measure the degree of influence of the immediate return on the future return, The network where it is located is called the target network; by calculating the loss function, the model constantly seeks the optimal solution; DQN uses reward mechanism, experience replay and target network technology; After constantly update Q parameter, make that Approach , with gradient descent to update work, back and forth until convergence; Step 3.3: Build Agent class, responsible for the decision-making process of the agent; define -greed greed value, so that the agent randomly selects an action with a certain probability for exploration; at the same time, the greed value is constantly reduced, so that the training gradually converges; after the upper DQN predicts, the Agent obtains the input value, builds a calculation graph for predicting the action and updating the Q network, and then defines the sample function to determine whether to return a random operation or the operation predicted by the Agent; define the predict function to select the maximum Q value as the optimal action; finally, define the feed function to record all the obtained values, and uniformly convert different variable types to float type. Step 3.4: Construct the ReplayMemory class to store the data used by the Agent, and cross these data with untrained data again to pass back to the DQN algorithm and Agent class for further judgment; The ReplayMemory experience pool constructed by the experience replay operation stores n experiences obtained after the Agent judges. Only when the experience pool is full will the old data be discarded. The stored content is: wherein (s t ,a t ,r t ,s t+1 ) represents a set of tuples of the agent performing actions in the environment, and these tuples are used to train the DQN model to predict the Q value of taking a certain choice under a given state; in the training, a batch of experiences [(s t ,a t ,r t ,s t+1 )...(s m ,a m ,r m ,s m+1 )] is randomly selected from the experience pool, and the correlation between data is eliminated; Step 3.5: Define the training period and evaluate the reward value obtained after each agent operation, and observe the training situation according to the result obtained at each step; when the model reaches the training iteration number or meets the training accuracy requirement, the intelligent OneR-DQN detection model training is completed; put the validation set into the trained model, verify the model, and generate the record of the judgment ability of the intelligent OneR-DQN detection model.

2. The botnet traffic detection method based on deep reinforcement learning according to claim 1, characterized in that, Step 1 is as follows: Step 1.1: Obtain the botnet traffic dataset by packet capture or download. For datasets with similar content and the same label, merge them according to the same label; Step 1.2: Delete the pollutant features that are not conducive to judgment and the features that have no distinguishing ability; Step 1.3: Balance the dataset by random undersampling and oversampling; Step 1.4: Label encoding of features in the dataset; Step 1.5: After the pre-processing of the dataset in steps 1.1-1.4 is completed, divide the dataset into training set and validation set, and store the divided dataset locally; Step 1.6: Only keep the float features of the data samples in the dataset, and discard all other features.

Images