Data security dynamic protection method based on artificial intelligence

Abstract

The invention discloses a data security dynamic protection method based on artificial intelligence, and relates to the field of network security, and the method comprises the steps: employing an anomaly detection algorithm to recognize malicious data streams and attack features according to a data stream log; according to the identified malicious data flow, a multi-agent reinforcement learning algorithm is adopted, and a protection strategy is adjusted through multiple agents according to the current environment state and attack characteristics; constructing a simulation environment, and training and evaluating an anomaly detection algorithm and a multi-agent reinforcement learning algorithm by generating a confrontation sample and a confrontation strategy; wherein the adversarial sample is generated by performing feature transformation on the malicious data stream and the normal data stream, and the adversarial strategy is generated by performing random disturbance on the existing attack strategy. In view of poor adaptability of a protection strategy for malicious data streams in the prior art, the malicious data streams and attack features are identified through an anomaly detection algorithm, and multi-agent reinforcement learning, collaborative optimization of flow control and a honeypot deployment strategy are adopted, so that the adaptability is improved.

Description

Technical Field

[0001] The present application relates to the field of network security, and in particular to a data security dynamic protection method based on artificial intelligence. Background Art

[0002] In today's information age, network security has become the focus of attention in all walks of life. With the rapid development of Internet technology, network attack methods are becoming more complex and diverse. Malicious data flow, as a common form of network attack, poses a serious threat to the information security of enterprises and individuals. Traditional data security protection methods mostly use static protection strategies, which are difficult to cope with the ever-changing malicious data flow and attack characteristics, resulting in poor protection effects.

[0003] At present, the protection technologies for malicious data flows mainly include signature-based detection, anomaly-based detection and honeypot technology. Signature-based detection extracts signature codes of known attacks for matching, but it is difficult to detect unknown attack types; anomaly-based detection identifies abnormal behaviors that deviate from the normal model by establishing a normal behavior model, but it is easy to produce false positives and false negatives; honeypot technology attracts and analyzes attack behaviors by deploying bait systems, but it is difficult to fully cover all attack types. In addition, most existing protection strategies use static configurations and lack the ability to dynamically adjust according to changes in the network environment, resulting in poor protection effects. Summary of the invention

[0004] In response to the problem of poor adaptability of protection strategies against malicious data flows in the prior art, the present application provides a dynamic data security protection method based on artificial intelligence. It identifies malicious data flows and attack characteristics through anomaly detection algorithms, and adopts multi-agent reinforcement learning to collaboratively optimize traffic control and honeypot deployment strategies, thereby improving the adaptability of protection strategies.

[0005] The purpose of this application is achieved through the following technical solutions.

[0006] The present application provides a data security dynamic protection method based on artificial intelligence, including: obtaining data flow logs; according to the data flow logs, using anomaly detection algorithms to identify malicious data flows and attack features; wherein the malicious data flow is a data flow segment that is identified to contain attack features such as SQL injection, DDoS or command execution; according to the identified malicious data flow, using a multi-agent reinforcement learning algorithm, adjusting the protection strategy through multiple agents according to the current environment state and attack features; constructing a simulation environment, training and evaluating anomaly detection algorithms and multi-agent reinforcement learning algorithms by generating adversarial samples and adversarial strategies; wherein the adversarial samples are generated by performing feature transformation on malicious data flows and normal data flows, and the adversarial strategies are generated by performing random perturbations on existing attack strategies.

[0007] According to the data flow log, an anomaly detection algorithm is used to identify malicious data flows and attack features, including: extracting feature vectors from the data flow log, the feature vectors contain the source IP address, destination IP address, port number and data packet content of the data flow; clustering the feature vectors using a clustering algorithm based on a Gaussian mixture model to obtain K clusters; calculating the mean vector and covariance matrix of each cluster, and judging whether each cluster is an abnormal cluster according to the sample number proportion and Mahalanobis distance of each cluster; among them, clusters with a sample number proportion less than a threshold and a Mahalanobis distance greater than a threshold are judged as abnormal clusters. Class clusters, abnormal clusters represent potential unknown attack patterns; a recognition model is built based on the random forest algorithm; one-hot encoding is used to convert feature vectors into numerical representations, and a sample set is built in combination with labeled malicious data flow samples; the sample set is used to perform supervised training on the recognition model to obtain a recognition model that recognizes known attack types, and the trained recognition model is used to predict the feature vectors of the data flow to be detected, and predict the known attack type to which the malicious data flow belongs; based on the abnormal clusters and the predicted known attack types, the malicious data flow is determined based on rules and the corresponding attack features are extracted.

[0008] Furthermore, according to the abnormal clustering clusters and the predicted known attack types, malicious data flows are determined based on rules and corresponding attack features are extracted, including: if the data flow belongs to the abnormal clustering cluster but does not belong to the known attack type, it is determined to be an unknown attack, its abnormal features are extracted as attack features, and it is marked as an unknown attack; if the data flow belongs to a known attack type but does not belong to the abnormal clustering cluster, it is determined to be a malicious data flow of a known type, and its corresponding attack type and attack characteristics are extracted as attack features; if the data flow belongs to both the abnormal clustering cluster and the known attack type, it is determined to be a variant attack, its abnormal features and known attack features are extracted, and it is marked as a variant attack.

[0009] Preferably, the priority of rule matching is: Rule 3 (variant attack) takes precedence over Rule 1 (unknown attack) and Rule 2 (known attack). Rule 1 (unknown attack) and Rule 2 (known attack) are mutually exclusive and have the same priority. Rule 4 (normal data flow) has the lowest priority. Determine whether data flow x belongs to abnormal cluster C abnormal :Calculate the Mahalanobis distance d(x,C from x to the center of each abnormal cluster i ),i=1,2,......,N,if min{d(x,C i )}≤the ​​maximum Mahalanobis distance of the abnormal cluster, then x∈C abnormal. Extraction of attack features: The abnormal features of unknown attacks can be obtained through cluster analysis, such as the common features of data flows within the abnormal cluster. The features of known attack types can be pre-defined, such as the typical features of DDoS, SQL injection, XSS and other attacks. Variant attacks extract both abnormal features and known attack features to fully characterize their behavioral characteristics.

[0010] Further, judging whether each cluster is an abnormal cluster includes: using the expectation maximization EM algorithm to estimate the parameters of the extracted feature vector to obtain the mean vector, covariance matrix and mixing coefficient of K Gaussian distributions, where K is the preset number of clusters; according to the mixing coefficient and Gaussian distribution parameters, calculating the posterior probability that the feature vector of each data stream belongs to each cluster, and dividing each data stream into the cluster with the largest posterior probability to obtain K clusters; calculating the number of data streams in each cluster, and calculating the proportion of the number of each cluster to the total number of data streams; when the proportion is less than the threshold, judging the corresponding cluster to be a small cluster, and the small cluster represents It indicates that the number of data streams in the corresponding cluster is abnormally sparse; for each cluster, the mean vector of the feature vectors of all data streams in the corresponding cluster is calculated as the center point of the corresponding cluster; the Mahalanobis distance from the feature vector of each data stream in the cluster to the center point of the cluster is calculated; when the Mahalanobis distance of a data stream is greater than the threshold, the corresponding data stream is judged to be an outlier far away from the center of the cluster; the number of outliers in each cluster is counted, and the average Mahalanobis distance of all outliers in the corresponding cluster to the cluster center is calculated as the average deviation degree of the corresponding cluster; the first N clusters that belong to small clusters and have the largest average deviation degree are judged as abnormal clusters.

[0011] Furthermore, a recognition model is constructed based on the random forest algorithm to predict the feature vector of the data flow to be detected, including: extracting features from the marked malicious data flow samples and normal data flow samples to obtain a training sample set; based on the training sample set, random sampling is performed using the Bootstrap method, and some features are randomly selected from all features to construct multiple CART decision trees to form a random forest. When each decision tree splits a node, the best split feature is selected according to the randomly selected feature subset and the Gini index criterion; the grid search method is used to optimize the hyperparameters of the random forest, and the hyperparameters include the number of decision trees, the maximum depth of each decision tree, and the minimum number of samples for node splitting; the performance of the model under different hyperparameter combinations is evaluated through cross-validation, and the hyperparameter combination with the best model performance is selected; according to the optimal hyperparameter combination, the random forest model is trained using the training sample set to obtain a recognition model, and the trained recognition model is used to obtain the attack type prediction result of the input data flow.

[0012] Furthermore, according to the identified malicious data flow, a multi-agent reinforcement learning algorithm is adopted, and the protection strategy is adjusted according to the current environment status and attack characteristics through multiple agents, including: setting a flow control agent and a honeypot deployment agent according to the identified malicious data flow; wherein the flow control agent performs flow control according to the flow state of the malicious data flow, and the honeypot deployment agent performs honeypot trapping according to the attack state of the malicious data flow; constructing the state space of the flow control agent based on the flow size and flow mutation degree of the malicious data flow; constructing the state space of the honeypot deployment agent based on the source IP address and attack type of the malicious data flow; the flow control agent obtains the flow state of the malicious data flow through the state space, and the honeypot deployment agent obtains the attack state of the malicious data flow through the state space. The attack state of malicious data flow; different levels of flow limit thresholds are used as the action space of flow control agent; honeypot systems with different degrees of interaction are used as the action space of honeypot deployment agent; the flow control agent selects the flow limit threshold from the action space and performs flow control according to the flow state of malicious data flow, and the honeypot deployment agent selects the honeypot system from the action space and performs honeypot trapping according to the attack state of malicious data flow; among them, the honeypot system (HoneypotSystem) is an active defense network security technology, whose main purpose is to trap, deceive and divert the attention of attackers by deploying some virtual and plausible network resources, so as to protect the security of real systems. The honeypot system is usually composed of one or more honeypot nodes, each of which simulates a specific type of network resources, such as servers, databases, network devices, etc.

[0013] A global reward function is constructed based on the quantitative value of the degree of harm caused by malicious data flow to the system; S35, based on the flow change of malicious data flow before and after flow limiting, a local reward function r of the flow control agent is constructed limit ; Based on the number of malicious data flows captured per unit time, a local reward function r of the honeypot deployment agent is constructed decoy ; Among them, the local reward function r of the flow control agent limit The calculation formula is: Flow before and Flow after They represent the flow of malicious data flow before and after flow limiting respectively; the local reward function r of the honeypot deployment agent decoy The calculation formula is: N capture represents the number of malicious data flows captured by the honeypot within a unit time T; the flow control agent uses the global state and the local reward function r limit As input, the honeypot deploys the agent with a global state and a local reward function r decoyThe traffic control agent and the honeypot deployment agent obtain the global state through local state observation and communication, and obtain the global optimal protection strategy. The traffic control agent and the honeypot deployment agent obtain the global state through local state observation and mutual communication, adjust their respective policy network parameters, and share the global reward signal R. global , collaboratively optimize the protection strategy with the goal of maximizing the global reward.

[0014] Furthermore, a global reward function is constructed based on the quantitative value of the degree of harm caused by malicious data flow to the system, including: obtaining the flow state, attack state, flow control state and honeypot trapping state of the malicious data flow; wherein the flow state represents the flow characteristics of the malicious data flow; the attack state represents the attack characteristics of the malicious data flow; the flow control state represents the flow limiting effect of the flow control agent on the malicious data flow through the malicious flow reduction rate and the normal flow retention rate; the honeypot trapping state represents the trapping effect of the honeypot deployment agent on the malicious data flow through the honeypot capture rate and the amount of attack information collected; the flow state is quantified into the flow anomaly degree, the attack state is quantified into the attack danger degree, the flow control state is quantified into the flow limiting effect degree, and the honeypot trapping state is quantified into the trapping effect degree; a global reward function is constructed, and the flow anomaly degree, attack danger degree, flow limiting effect degree and trapping effect degree are used as inputs to calculate the global reward value. The global reward function is expressed as: R global =w1×R flow +w2×R attack +w3×R limit +w4×R decoy , where R global represents the global reward value, R flow Indicates the flow abnormality, R attack Indicates the attack risk, R limit Indicates the current limiting effect, R decoy It represents the trapping effectiveness, w1, w2, w3, w4 are weight coefficients.

[0015] Furthermore, the quantitative calculation formula of traffic anomaly is: Among them, Flow malicious Indicates the average flow rate of malicious data flow, Flow normal Indicates the average flow of normal data flow; flow anomaly R flow It is calculated by the difference between the traffic of malicious data flow and normal data flow. The greater the difference, the higher the abnormality. The quantitative calculation formula of attack risk is:

[0016] Among them, Attack i represents the i-th attack type of the data flow, w irepresents the weight of the danger level of the i-th attack type, and N represents the total number of attack types considered.

[0017] The quantitative calculation formula of current limiting effect is: Among them, Flow normal_after and Flow normal_before They represent the flow of normal data flow before and after flow limiting, respectively, and k1 and k2 are the weight coefficients of the two indicators. The quantitative calculation formula for trapping effect is: Among them, N capture Represents the number of malicious data flows captured by the honeypot, N total Indicates the total number of malicious data flows, Info collect It represents the amount of attack information collected within a unit time T, and k3 and k4 are the weight coefficients of the two indicators.

[0018] Furthermore, a policy network is constructed through a deep reinforcement learning algorithm, and the action probability distribution is output to generate a protective action, including: constructing a policy network of an intelligent agent, taking the global state as input, and outputting the action probability distribution; wherein the global state includes the flow state, attack state, flow control state, and honeypot trapping state of the malicious data flow; the intelligent agent obtains the corresponding protective action by random sampling according to the action probability distribution output by the policy network; wherein the protective action of the flow control intelligent agent is the flow limiting action, and the protective action of the honeypot deployment intelligent agent is the honeypot deployment action; the protective action After being used for malicious data flow, a new environment state is obtained, and the global reward value and the local reward value are calculated; wherein, the new environment state includes the flow state, attack state, flow control state and honeypot trapping state of the malicious data flow after the protection action is taken; the global state, protection action, global reward value and local reward value, and the new environment state constitute a four-tuple sample (s, a, r, s'), wherein s is the global state, a is the protection action, r is the global reward value and the local reward value, and s' is the new environment state; the time difference algorithm is used to calculate the time difference error TD of the four-tuple sample error ; Among them, the calculation formula of timing difference error is: TD error =r+γ(s',a')-Q(s,a), where Q(s,a) is the state-action value function, which represents the expected value of the long-term cumulative reward for taking action a in state s; γ is the discount factor, which represents the degree of attenuation of future rewards; Q(s',a') is the maximum Q value in the next state s', which is calculated based on the next state s' and the action probability distribution output by the policy network; the gradient descent algorithm is used to calculate the temporal difference error TD error As the loss function, the parameters of the policy network are updated so that the action probability distribution generated by the policy network approaches the optimal policy; the parameter update formula is: θ'=θ+α×TD error×▽θlogπθ(s,a), where θ is the parameter of the policy network, α is the learning rate, and πθ(s,a) is the probability that the policy network with parameter θ takes action a in state s; through multiple rounds of iterative optimization, the policy networks of the flow control agent and the honeypot deployment agent converge to the optimal strategy, with the goal of maximizing the global reward.

[0019] Furthermore, the flow control agent and the honeypot deployment agent collaboratively optimize the protection strategy through local state observation and mutual communication with the goal of maximizing the global reward, including: the flow control agent observes the flow state of the malicious data flow through the local state space, and obtains the honeypot trapping state of the malicious data flow through communication with the honeypot deployment agent; the honeypot deployment agent observes the attack state of the malicious data flow through the local state space, and obtains the flow control state of the malicious data flow through communication with the flow control agent; the flow control agent integrates the observed flow state and the honeypot trapping state obtained through communication into its own global state as the input of the policy network; the honeypot deployment agent integrates the observed attack state and the flow control state obtained through communication into its own global state as the input of the policy network; the flow control agent and the honeypot deployment agent respectively adjust the flow control strategy according to their own global states through the deep reinforcement learning algorithm. The parameters of their respective strategy networks are optimized to optimize their respective protection strategies; among them, the flow control agent outputs the optimized current limiting strategy, and the honeypot deployment agent outputs the optimized honeypot deployment strategy; the flow control agent and the honeypot deployment agent share the same global reward signal based on the global reward function, and collaboratively optimize their respective protection strategies with the goal of maximizing the global reward; the flow control agent shares the optimized current limiting strategy with the honeypot deployment agent, and the honeypot deployment agent shares the optimized honeypot deployment strategy with the flow control agent, and both agents further adjust their own strategy network parameters based on the shared strategy information; through multiple rounds of strategy interaction and strategy optimization, the flow control agent and the honeypot deployment agent form a strategy collaboration to obtain the optimal joint protection strategy under the global reward; the globally optimal joint protection strategy is applied to malicious data flows in the network environment, and the flow control and honeypot deployment strategies are dynamically adjusted.

[0020] Compared with the prior art, the advantages of this application are:

[0021] Gaussian mixture clustering can adaptively learn multiple normal patterns of data streams, capture the complexity of data distribution, and improve the accuracy of anomaly detection. Random forests make comprehensive use of the judgment results of multiple decision trees, reduce the risk of overfitting of a single model, and enhance the generalization and robustness of the detection model. By training a two-level detection model, Gaussian mixture clustering is first used to identify unknown attacks, and then random forests are used to distinguish known attacks from variant attacks to accurately identify various types of malicious data streams.

[0022] Decomposing the protection decision problem into two subtasks, flow control and honeypot deployment, reduces the complexity of the problem and improves the pertinence and professionalism of the decision. The flow control agent can dynamically adjust the flow limit threshold according to the flow characteristics of the malicious data flow, while minimizing the impact on normal business while defending against attacks. The honeypot deployment agent can dynamically adjust the configuration of the honeypot system according to the attack characteristics of the malicious data flow, realize active trapping and analysis of attackers, and improve the ability to obtain threat intelligence.

[0023] The global reward function comprehensively considers the performance indicators of multiple aspects such as detection, defense, and trapping, forming a global optimization goal, avoiding the local optimal trap that may be caused by the optimization of a single indicator. Incorporating traffic anomaly and attack danger into the reward function guides the agent to consider the detection rate and degree of harm of malicious data flow when optimizing the strategy, thereby improving the effectiveness and pertinence of protection. Incorporating the flow limiting effect and trapping effect into the reward function guides the agent to consider the timeliness of protection actions and the quality of threat intelligence acquisition when optimizing the strategy.

[0024] Deep reinforcement learning can handle high-dimensional, continuous state and action spaces, and is suitable for modeling complex network security scenarios, improving the expressiveness and adaptability of policy learning. The policy network directly learns the mapping relationship from state to action and outputs the action probability distribution, avoiding explicit modeling of state transitions and reward functions, reducing the difficulty of learning and computational overhead. Gradient descent is used to optimize network parameters to achieve continuous optimization and adaptive adjustment of the strategy.

[0025] Through state observation and communication mechanisms, agents can obtain each other's local state information and form a complete understanding of the global state, breaking through the limitations of the perception range of a single agent and improving the information completeness of decision-making. Sharing global reward signals allows different agents to optimize their own strategies while taking into account the global protection effect, avoiding the global performance loss that may be caused by local strategy optimization, and achieving the unification and coordination of strategy optimization. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is an exemplary flow chart of a data security dynamic protection method based on artificial intelligence shown in this application;

[0027] Figure 2 is an exemplary flow chart for determining whether a data flow is a known attack according to the present application;

[0028] Figure 3 is an exemplary flow chart of obtaining abnormal clustering clusters according to the present application;

[0029] Figure 4is an exemplary flow chart for obtaining a global optimal protection strategy according to the present application;

[0030] Figure 5 is an exemplary flow chart of updating policy network parameters according to the present application;

[0031] Figure 6 This is an exemplary flow chart for obtaining a globally optimal joint protection strategy as shown in the present application. DETAILED DESCRIPTION

[0032] The method and system provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0033] Figure 1 This is an exemplary flow chart of an artificial intelligence-based data security dynamic protection method shown in the present application, including: obtaining data flow logs; using an anomaly detection algorithm to identify malicious data flows and attack features based on the data flow logs; wherein the malicious data flow is a data flow segment that is identified to contain attack features such as SQL injection, DDoS or command execution; based on the identified malicious data flow, using a multi-agent reinforcement learning algorithm, through multiple agents, adjust the protection strategy according to the current environment state and attack features; constructing a simulation environment, by generating adversarial samples and adversarial strategies, training and evaluating the anomaly detection algorithm and the multi-agent reinforcement learning algorithm; wherein the adversarial samples are generated by performing feature transformation on malicious data flows and normal data flows, and the adversarial strategies are generated by randomly perturbing existing attack strategies.

[0034] Figure 2 It is an exemplary flowchart for judging whether a data flow is a known attack as shown in the present application, a data security dynamic protection method based on artificial intelligence, which intelligently identifies malicious data flows by real-time monitoring and analysis of data flows in a network environment, and dynamically optimizes flow control and honeypot deployment strategies using a multi-agent reinforcement learning algorithm, thereby realizing active defense against malicious data flows. According to the data flow log, an anomaly detection algorithm is used to identify malicious data flows and attack features, including: parsing each data flow record from the data flow log, extracting key fields, including: source IP address, destination IP address, source port number, destination port number, protocol type, and data packet content. Perform deep packet inspection (DPI) on the data packet content, extract application layer protocol features: HTTP request header fields (such as User-Agent, Referer, etc.), SQL statement keywords (such as select, drop, etc.), file name suffixes (such as .exe, .php, etc.), and vectorize the extracted features to form a data flow feature vector.

[0035] Clustering anomaly detection based on Gaussian mixture model (GMM), the data stream feature vector set X = {x1, x2, ..., xn}, the number of clusters K, the anomaly ratio threshold alpha, and the anomaly distance threshold beta. The EM algorithm is used to estimate the GMM parameters, and the parameters of K Gaussian distributions are randomly initialized: the mean vector μ k , the covariance matrix Σ k , mixing coefficient p ik , repeat the iteration until convergence: Step E: Calculate each data point x according to the current parameter value i The posterior probability γ of belonging to the kth distribution ik , Step M: Update the parameters of each distribution according to the posterior probability. Output estimated GMM parameters: mean μ k , covariance Σ k , mixing coefficient p ik Based on the estimated GMM parameters, data stream clustering is performed, and for each data stream feature vector x i : Calculate the posterior probability γ that belongs to each Gaussian distribution ik , x i Divide into cluster C with the largest posterior probability k In this case, k*=argmax k (γ ik ); Output K clusters {C1, C2, ..., C K}.

[0036] Figure 3 is an exemplary flow chart of obtaining abnormal clusters according to the present application, detecting abnormal clusters, and calculating each cluster C k The proportion of data flow If p k <alpha, then mark C k is a small cluster, indicating that the data flow is sparse and abnormal. For each cluster C k :Calculate the mean vector μ of all data streams in the cluster k , as the cluster center, Calculate for each data stream x i To the cluster center μ k The Mahalanobis distance d ik , If d ik > beta, then determine x i C k For each cluster C k : Calculate the average Mahalanobis distance d of outliers k , as the average deviation of the cluster, Among them, k C kThe internal outlier set, among all the small clusters, select the top N clusters with the largest average deviation

[0037] {C k1 ,C k2 ,.....,C kN}, {C k1 ,C k2 ,.....,C kN} is determined to be an abnormal cluster, representing a sparse abnormal pattern, and the abnormal cluster set {C k1 ,C k2 ,.....,C kN}.

[0038] S23, build a malicious data flow recognition model based on random forest, and the labeled malicious data flow sample set D m and normal data flow sample set D n , for malicious sample set D m and normal sample set D n Perform feature extraction: Extract basic features of the data stream, such as source IP, destination IP, port number, and protocol type. Perform deep packet inspection on the data packet content to extract application layer protocol features, attack signature features, etc. Perform encoding conversion on the extracted features, convert discrete features (such as protocol type) into one-hot encoding, and normalize continuous features (such as packet length) to the [0, 1] interval. Combine the extracted features into feature vectors to form the training sample set D. train : The feature vector of malicious samples is marked as positive, and the class label y = 1. The feature vector of normal samples is marked as negative, and the class label y = 0.

[0039] Construct a random forest based on Bootstrap sampling, input: training sample set D train , the number of trees in the random forest is T, the number of Bootstrap samples for each tree is N, create an empty random forest RF = {}, sample and generate T decision trees FORt = 1toTDO: from D train Randomly select N samples (with replacement) from the dataset to form the Bootstrap sample set D of the decision tree t. t ;

[0040] D t =Bootstrap(D train ,N).

[0041] Based on D t , recursively construct a CART decision tree: Tree t =buildTree(D t ); Add the decision tree Tree_t to the random forest RF; RF'=RF∪{Treet}; ENDFOR. buildTree(D t ): Recursively construct a CART decision tree. If the sample set of the current node is D', the number of samples is N', and the number of features is M. IF (N'<preset sample number threshold or the tree reaches the maximum depth or the sample categories in D' are all the same): mark the current node as a leaf node, take the majority category of the samples in D' as the predicted category of the leaf node, and return the leaf node. ELSE: Randomly select from M features features, forming a feature subset F'. Based on the Gini index criterion, the best split feature f is selected from F' best and threshold th best :FOReach feature finF'DO:FOReach threshold thof feature fDO:Divide D' into D according to (f, th) left ' and D right 'Two subsets. Calculate the Gini index: in,

[0042] END FOR; END FOR. According to (f best ,th best ) Split D' into left and right child nodes D lef t' and D right '. Recursively in D left ' and D right 'Build left and right subtrees respectively: left subtree =BuildTree(D left '), right subtree =BuildTree(D right '), returns to f best is the splitting feature, th best is the splitting threshold, left subtree and right subtree For the internal nodes of the left and right subtrees, return the random forest model RF, which contains T decision trees Output: Random Forest Model RF.

[0043] Random forest hyperparameter optimization, specify the hyperparameter search space of random forest: the number of trees T∈{100, 200, 300, 400, 500}, the maximum depth of the tree max_depth∈{10, 20, 30, 40, 50}, the minimum number of samples for node splitting min samples_split ∈{2, 5, 10, 20}, using k-fold cross validation, D trainDivide into k mutually exclusive subsets, select k-1 subsets as training sets each time, and the remaining 1 as validation set, repeat k times. depth ,min samples_split ): Based on the current hyperparameter combination, train the random forest model on each training set of the k-fold cross validation. Evaluate the performance indicators of the model on the corresponding validation set, such as accuracy, precision, recall, F1 value, etc. Take the average of the performance indicators of the k-fold validation as the performance score of the current hyperparameter combination. Select the hyperparameter combination with the best performance score (T*, max depth *,min samples_split *).

[0044] Train the optimized random forest model based on the optimized hyperparameters (T*, max depth *,min samples_split *), in the complete training set D train Retrain the random forest model on . Generate T decision trees, and the maximum depth of each tree is max depth *, the minimum number of samples for node splitting is min samples_split *. The optimized random forest model RF* is obtained for the identification of malicious data flows. The trained random forest model is used to predict the attack type and extract the feature vector x for the input data flow to be detected. The feature vector x is input into the random forest model RF*: for each decision tree in RF*, starting from the root node according to the feature value of x, x is recursively divided into corresponding leaf nodes. Each decision tree gives a prediction y of the category to which x belongs. t (x), which is the majority class label of the leaf node that x falls into. The prediction results for all trees Voting is performed to obtain the final attack type prediction y(x),

[0045] Output the attack type prediction result y(x)* of x. If y(x)*=1, x is predicted to be a malicious data flow; if y(x)*=0, x is predicted to be a normal data flow. The trained random forest recognition model RF* is input to predict the attack type y(x)* of the data flow to be detected x.

[0046] Determine malicious data flows based on rules and extract attack features. Input: abnormal cluster set C abnormal ={C1,C2,......,C N}, each cluster C iContains a set of data streams, the data stream attack type y(x)*∈{0,1,2,......,K} predicted by the random forest model, where 0 represents normal, 1~K represents K known attack types, and the data stream x to be analyzed. Rule 1: Determine unknown attack, if x∈Cabnormal and y(x)*==0: Determine data stream x as an unknown attack, extract the abnormal feature vector of x x As attack features, abnormal features may include: data packet length distribution, packet arrival time interval, source / destination IP and port abnormality, etc. Mark x as "unknown attack" and record its abnormal feature vector x Rule 2: Determine the known attack type if At the same time, y(x)*∈{1,2,......,K}: determine that the data stream x is a known attack, the attack type is the type corresponding to y(x)*, and extract the attack feature vector of type y(x)* x , the characteristics of different attack types can be pre-defined. For example, the characteristics of DDoS attacks may include a large number of requests from the same source IP in a short period of time, SYN flooding characteristics, etc., mark x as "known attack: Typey*(x)", record its attack type and corresponding attack feature vector x Rule 3: Determine variant attack, if x∈Cabnormal and y(x)*∈{1,2,......,K}: Determine data stream x as a variant attack, belonging to the variant of type y(x)*, and extract the abnormal feature vector of x x_abnormal and known attack signature vector of type y(x)* x_known , the method for extracting abnormal features is the same as Rule 1, and the method for extracting known attack features is the same as Rule 2. Mark x as "variant attack: Typey*(x)" and record its abnormal feature vector x_abnormal and known attack signature vector x_known Rule 4: Determine normal data flow if At the same time, y(x)*==0: Data stream x is judged to be a normal data stream, and there is no need to extract attack features, and x is marked as "normal". Output: Malicious judgment result of data stream x: unknown attack / known attack: Typey(x)* / variant attack: Typey(x)* / Normal, attack features of data stream x (if judged to be malicious): unknown attack: abnormal feature vector x , known attack: y(x)* type attack feature vector x , Variant attack: abnormal characteristics, vector x_abnormal and known attack signature vector x_known .

[0047] Figure 4This is an exemplary flow chart for obtaining the global optimal protection strategy shown in the present application. According to the identified malicious data flow, a multi-agent reinforcement learning algorithm is used to adjust the protection strategy through multiple agents according to the current environment state and attack characteristics, including:

[0048] S31: Set up traffic control agent and honeypot deployment agent, input: identified malicious data flow set X malicious ={x1,x2,......,x M}, extract traffic status features, and for each malicious data flow x i ∈X malicious : Extract x i Traffic statistics feature vector fi , including: Packet rate: the number of packets per unit time; Byte rate: the number of bytes per unit time; Number of connections: with x i Related concurrent TCP / UDP connections; flow duration: x i Duration from start to end; traffic burstiness: x i The magnitude of flow changes in a short period of time; construct x i The traffic state vector state fi =[vector fi ,y(x)*], where y(x i )* is x i The attack type.

[0049] Train the traffic control strategy and collect a set of malicious data flow samples X offline train and its flow state vector set Design flow control action space A f , including: Speed ​​limit: set x i The rate is limited to the specified threshold v limit Below; interruption: directly discard x i All packets of x i Redirect traffic to honeypots or sinkholes; Block: Block x i All communications between the source IP and the destination IP. Define the reward function for flow control:

[0050] r f (state f ,a f )=w1×(-cost(a f ))+w2×(-risk(state f '))+w3×flow limit (state f ');in:

[0051] cost(a f ) indicates action a f Cost of overhead; risk(state f ') means to execute a f Security risks faced by the post-system; flow limit (state f ') means to execute a f The degree of restriction on malicious traffic after that; w1, w2, w3 are weight coefficients. Using deep reinforcement learning algorithms (such as DQN, A3C, etc.), S train is the environmental state, with A f is the action space, with r f As the reward function, train the flow control strategy π f :S train →A f , so that the actions generated by the strategy can maximize the cumulative reward. Perform flow control online, for each identified malicious data flow x i ∈X malicious : Extract its traffic state vector state fi , state fi Input the trained flow control strategy π f , get the control action a fi =π f (state fi ), execute control action a fi , for malicious traffic x i Perform speed limiting, interruption, redirection or blocking, and track and record actions fi The execution effect is used to update the strategy π f .

[0052] Honeypot deploys intelligent agents to extract attack status features and i ∈X malicious : Extract x i Attack behavior characteristics vector ai , including: Attack frequency: the number of attacks per unit time, Attack traffic ratio: malicious traffic ratio x i Ratio of total traffic, attack diversity: x i The number of attack types included, attack duration: from x i The duration from the first attack to the last attack, construct x i The attack state vector state ai =[vector ai ,y(x i )*], where y(x i )* is x iTraining honeypot deployment strategy, collecting a set of malicious data flow samples X offline train and its attack state vector set Design honeypot deployment action space A h , including: deploying high-interaction honeypots: building honeypots that are highly similar to the real system to capture the attacker's detailed operations; deploying low-interaction honeypots: building a simplified honeypot system, simulating common services, and recording attack traffic; dynamically adjusting honeypots: dynamically adjusting the type, number, and configuration of honeypots according to changes in malicious traffic; not deploying honeypots: directly discarding certain malicious traffic without trapping; defining the reward function r for honeypot deployment h (state a ,a h ):

[0053] r h (state a ,a h )=w1×intel gain (a h )-w2×risk(a h )-w3×cost(a h ). Among them: Intel gain (a h ) means to execute a h The value of threat intelligence obtained; risk(a h ) means to execute a h The risk of exposing the honeypot; cost(a h ) means to execute a h resource overhead; w1, w2, w3 are weight coefficients. Using deep reinforcement learning algorithm, S train is the environmental state, with A h is the action space, with r h is the reward function, and the honeypot deployment strategy π is trained h :S train →A h , so that the actions generated by the strategy can maximize the long-term cumulative reward. Perform honeypot trapping online, for each identified malicious data flow x i ∈X malicious : Extract its attack state vector state ai . ai Input the trained honeypot deployment strategy π h , get the deployment action a hi =π h (state ai ), execute deployment action a hi , dynamically configure and manage honeypot resources, and detect malicious traffic x iImplement trapping, analyze malicious traffic and attack behaviors captured by honeypots, and track and record actions hi The execution effect is used to update the strategy π h Output: Traffic control strategy π based on reinforcement learning f And honeypot deployment strategy π h , for each malicious data flow x i : Flow control action a executed fi and its effect, the honeypot deployment action a executed hi and its trapping results.

[0054] S32, the state space of the traffic control agent is constructed to define the traffic state characteristics. For each malicious data flow x i ∈X malicious , extract the following traffic state features: Traffic size: represents x i The total flow rate per unit time is denoted as f size_i . Traffic mutation degree: represents x i The degree of deviation between the instantaneous flow and its average flow is recorded as Calculation formula: in, For x i The instantaneous flow rate, For x i The average traffic volume over a period of time.

[0055] Construct a traffic state vector and transform each malicious data flow x i The traffic state characteristics are combined into a state vector: Define the state space, the state space S of the flow control agent f is the traffic state vector set of all malicious data flows: S f ={state f1 ,state f2 ,......,state fM}. State space quantization, in order to facilitate the training and strategy search of reinforcement learning algorithms, it is necessary to quantize the state space S f Quantify: Normalize: Among them, min size and max size are the minimum and maximum flow sizes of all malicious data flows. Discretize: Among them, step mutation is the discrete step length of the traffic mutation degree, and round is the rounding function. The quantized traffic state vector is:

[0056] The state space of the honeypot deployment agent is constructed to define the attack state characteristics. For each malicious data flow x i ∈X malicious , extract the following attack status features: Source IP address: represents the source IP address that initiates malicious traffic, denoted as ip src_i Attack type: Indicates the attack type of malicious traffic, recorded as attack type_i , is a K-dimensional one-hot vector, where K is the total number of known attack types. Construct an attack state vector and convert each malicious data flow x i The attack state features are combined into a state vector:

[0057] state a_i =[ip src_i ,attack type_i ] Define the state space, the state space S of the honeypot deployment agent h is the attack state vector set of all malicious data flows: S h =[state a_1 ,state a_2 ,......,state a_M ]. State space embedding. Since the source IP address is high-dimensional and sparse, in order to facilitate learning the association between IP addresses and attack behaviors, it is necessary to embed the state space S h The IP addresses in the embedding process are processed: using word embedding algorithms such as Word2Vec, each IP address is embedded into a D-dimensional dense vector:

[0058] ip src_i '=embedding(ip src_i ), the embedded attack state vector: state a_i '=[ip src_i ',attack type_i ].

[0059] Preferably, the state of the traffic control agent is obtained, and for each decision time step t, the malicious data flow set X at the current moment is extracted malicious_t , for X malicious_t Each data stream x in i_t , calculate its quantized flow state vector state fi_t ',Will As the state observation of the traffic control agent at time t, the state of the honeypot deployment agent is obtained. For each decision time step t, the malicious data flow set X at the current moment is extracted malicious_t , for X malicious_t Each data stream x in i_t , calculate the embedded attack state vector stateai_t ',Will As a honeypot deployment agent, observe the state at time t.

[0060] Agent decision and update, the flow control agent will observe Input policy network π f , and obtain a set of flow control actions {a fi_t}, limit the speed and interrupt the flow of malicious data, and the honeypot deployment agent will observe Input policy network π h , and obtain a set of honeypot deployment actions {a hi_t}, trap malicious data flows, and after executing the action, obtain the instant reward r for flow control and honeypot deployment respectively. f_t and r h_t , enter the next time step t+1, and obtain the new state observation {state fi_t+1 '}, {state ai_t+1 '}, will (state t ,action t ,reward t ,state t+1 ) are stored in the experience replay pool, and a batch of quadruple samples are regularly sampled from the experience replay pool. The policy network parameters of the flow control and honeypot deployment agents are updated by optimizing the mean square error between the predicted Q value and the target Q value.

[0061] S33, action space design of traffic control agent, defines the traffic restriction level, sets N different levels of traffic restriction thresholds, and takes different intensity speed restriction measures against malicious data flows: Level 1: v limit_1 =v min , the most relaxed restriction, limiting malicious traffic to no more than v min ; Level 2: v limit_2 =v min +step v , slightly stricter than level 1, limiting malicious traffic to no more than v min +step v ;............;Level N: v limit_N =v max , the most stringent restriction, limiting malicious traffic to no more than v max , where v min and v max are the minimum and maximum speed limit thresholds, step v It is the speed limit step between adjacent levels.

[0062] Construct the flow control action space, the action space A of the flow control agent fContains N discrete actions, each corresponding to a flow restriction level: A f ={a f_1 ,a f_2 ,......,a f_N}, where action a f_i Indicates that malicious data flow is limited to v limit_i Below. Action selection and execution, for each decision time step t, the traffic control agent obtains the current malicious traffic state observation {state fi_t '} Input flow control strategy network π f (s,θ f ), and get the probability distribution π of each action f (a|s,θ f ), according to the ε-greedy strategy, randomly select an action a with probability ε fi_t ∈A f , otherwise choose the action with the highest probability Execute action a fi_t , the malicious data flow x i_t The rate limit is v limit_i the following.

[0063] The action space design of the honeypot deployment agent defines the honeypot interaction level and sets K honeypot systems with different interaction levels, which represent different degrees of active trapping of malicious data flows: Level 1: Low-interaction honeypots, such as Honeyd, simulate simple network topologies and services to trick attackers into scanning and connecting; Level 2: Medium-interaction honeypots, such as Cowrie, simulate common operating systems and applications and record attackers' shell commands; ...; Level K: High-interaction honeypots, such as real Web servers, allow attackers to obtain real system access rights and capture detailed intrusion processes. Construct the honeypot deployment action space, the action space A of the honeypot deployment agent h Contains K discrete actions, each action corresponds to a honeypot interaction level: A f ={a h_1 ,a h_2 ,......,a h_K}, where action a h_j Indicates the deployment of the jth level honeypot system for malicious data flow. Action selection and execution, for each decision time step t, the honeypot deployment agent obtains the current malicious traffic attack status observation The state ai_t 'Enter the honeypot deployment strategy network π f (s,θ f ), and get the probability distribution π of each action h (a|s,θ h), according to the ε-greedy strategy, randomly select an action a with probability ε hj_t ∈A h , otherwise choose the action with the highest probability Execute action a hj_t , for malicious data flow x i_t Deploy the j-th level honeypot system and start active trapping.

[0064] Preferably, the decision-making and coordination of the two agents are synchronized at time step t, respectively targeting the same batch of malicious data flows {x i_t} to make decisions, but the state information used is different: the flow control agent uses the flow state state fi_t ', select the speed limit action {a fi_t}; Honeypot deployment agent uses attack state {state ai_t '}, select honeypot action {a hj_t}; The actions of the two agents are coordinated and cooperated: flow control action {a fi_t}Suppress malicious traffic from the data traffic level to prevent malicious traffic from causing denial of service attacks on normal business, and provide a stable trapping environment for honeypot deployment. Honeypot deployment action {a hj_t Actively trap malicious traffic from the attack semantic level, so that malicious traffic is actively exposed, which is convenient for tracing and intelligence extraction, and in turn strengthens the effectiveness of traffic control; after executing the action, the traffic control and honeypot deployment agents receive instant rewards r f_t and r h_t : The reward r for flow control f_t Comprehensively consider the effect of malicious traffic restriction, the cost of killing normal traffic by mistake, and other factors; the reward for honeypot deployment is r h_t Comprehensively consider the value of obtaining attack intelligence, the risk of honeypot exposure and other factors. The two agents independently optimize their own strategy network π f (s,θ f ) and π h (s,θ h ): Using reinforcement learning algorithms such as policy gradient, according to the state transfer (state t ,action t ,reward t ,state t+1 ), adjust the policy network parameters θ f and θ h The goal of policy network optimization is to maximize the long-term cumulative reward, but flow control focuses on reducing risks, while honeypot deployment focuses on intelligence benefits.

[0065] A global reward function is constructed based on the quantitative value of the degree of harm caused by malicious data flow to the system; state acquisition and quantification, four state information of malicious data flow is obtained: traffic state: represents the traffic characteristics of malicious data flow, such as average traffic, traffic peak, etc.; attack state: represents the attack characteristics of malicious data flow, such as attack type, attack intensity, etc.; flow control state: represents the flow limiting effect of the flow control agent on the malicious data flow, such as the malicious traffic reduction rate, normal traffic retention rate, etc.; honeypot trapping state: represents the trapping effect of the honeypot deployment agent on the malicious data flow, such as the honeypot capture rate, the amount of attack information collected, etc. The four state information are quantified into four indicators: flow anomaly R flow : The difference between the traffic of malicious data flow and normal data flow. The greater the difference, the higher the abnormality. Quantitative formula: Among them, Flow malicious is the average flow rate of malicious data flow, Flow normal is the average flow of normal data flow; the attack risk R attack : Weighted average of various attack types of malicious data flows. The greater the weight of each attack type, the higher the overall risk. Quantification formula: Among them, Attack i is the i-th attack type of the data flow, w i is the danger level weight of the i-th attack type, N is the total number of attack types considered; the current limiting effect R limit : The weighted average of the reduction rate of malicious traffic and the retention rate of normal traffic by current limiting. The higher the reduction rate and the higher the retention rate, the better the effect. Quantitative formula: Among them, Flow bef ore and Flow after They are the flow rates of malicious data flows before and after flow limiting, Flow normal_before and Flow normal_after are the flow rates of normal data flows before and after flow limiting, respectively, and k1 and k2 are the weight coefficients of the two indicators. decoy : The weighted average of the honeypot's capture rate of malicious traffic and the amount of attack information collected. The higher the capture rate and the amount of information collected, the better the effect. Quantification formula:

[0066] Among them, N capture Represents the number of malicious data flows captured by the honeypot, N total Indicates the total number of malicious data flows, Info collect It represents the amount of attack information collected within a unit time T, and k3 and k4 are the weight coefficients of the two indicators.

[0067] The global reward function is constructed, taking the four quantitative indicators as input, constructing the global reward function, and calculating the global reward value Rglobal Expressed as: R global =w1×R flow +w2×R attack +w3×R limit +w4×R decoy , where R flow Indicates the flow abnormality, R attack Indicates the attack risk, R limit Indicates the current limiting effect, R decoy Represents the degree of trapping effectiveness, w1, w2, w3, and w4 are weight coefficients. The following principles are considered in setting the weight coefficients: the weights of w1 and w2 are relatively large, because abnormal traffic detection and attack risk assessment are the basis of malicious traffic defense and directly affect the security of the system; the weights of w3 and w4 are relatively small, because traffic control and honeypot trapping are defense measures taken on the basis of abnormal traffic detection and risk assessment, and are the deepening and extension of security protection; the weight relationship between w1 and w2 is determined by the relative importance of traffic anomalies and attack risks in actual scenarios; the weight relationship between w3 and w4 is determined by the relative effects of traffic control and honeypot trapping in actual scenarios. Global reward value R global Meaning: R global Quantitatively represents the degree of harm caused by malicious data flow to the system, R global The larger it is, the higher the degree of harm and the worse the defense effect. In the defense process, R should be minimized. global , so that the security threat to the system is minimized; R global The value range is [0, positive infinity), and a threshold R can be set threshold , when R global >R threshold , it means the system faces major security risks and urgent defensive measures are needed.

[0068] Preferably, the agent training and decision making uses the global reward function R global As the common optimization goal of the flow control agent and the honeypot deployment agent: the flow control agent affects the flow limiting effect R through actions (flow limiting threshold) limit , which in turn affects R global ; Honeypot deployment agents influence the trapping effect R through actions (honeypot type) decoy , which in turn affects R global ; The actions of the two agents will indirectly affect the traffic anomaly R flow and attack risk R attack , thus affecting R global .

[0069] In the reinforcement learning framework, using R global and its change ΔR globalTo guide the agent's strategy learning: R before and after state transition global The change in ΔR global As the immediate reward for the agent’s behavior, ΔR global The larger the value, the higher the degree of harm to the system caused by the behavior, and the greater the punishment should be given. During the training process, the agent learns to minimize the cumulative reward Σ(ΔR global ), so that in the long run, the security threat to the system is the lowest; in the decision-making process, the agent chooses to make ΔR global The minimum action makes each step as low as possible the harm of malicious traffic. The two agents use their own local state information and global reward information to train and make decisions: the traffic control agent uses < traffic state, traffic control state, R global> For learning samples, optimize the current limiting strategy to minimize R global ; Honeypot deployment agent with <attack state, honeypot trapping state, R global >For learning samples, optimize the honeypot strategy to minimize R global ; The two agents take different defensive measures based on their own local information, through R global Achieve global coordinated joint defense.

[0070] Design of local reward function for flow control agent, define local reward function r limit The goal of the traffic control agent is to minimize the flow of malicious data flow by dynamically adjusting the flow limit threshold. Therefore, the flow change ratio of malicious data flow before and after flow limit can be used as the basis for local rewards. limit The calculation formula is: Among them, Flow before Indicates the flow of malicious data flow before flow limiting, Flow after Indicates the traffic volume of malicious data flow after current limiting. limit The value range of Flow after = 0, that is, the malicious traffic is completely blocked after current limiting, r limit Reaching the maximum value 1; when Flow after =Flow before When the current limiting is ineffective against malicious traffic, r limit Reach the minimum value 0; when Flow after >Flow before When the flow restriction leads to an increase in malicious traffic, r limit is a negative value; therefore, r limit The value range is (-∞, 1], and the optimization goal of the agent is to maximize r limit, making it as close to 1 as possible. Local state and local reward, at each decision time step t, the flow control agent selects the flow limiting threshold action a based on the local state information of <flow state, flow control state> limit After the action is executed, the environment feeds back the flow state and flow control state at the next moment, as well as the local reward r calculated based on these states. limit ; State transition samples at time step t t ,a t ,r t ,s t+1 is used to train and optimize the flow control strategy, where r t It is calculated by the formula r limit .

[0071] Design of local reward function for honeypot deployment agent, define local reward function r decoy The goal of the honeypot deployment agent is to maximize the number of malicious data flows captured per unit time by dynamically selecting the honeypot type. Therefore, the number of malicious data flows captured per unit time can be directly used as the local reward. decoy The calculation formula is: Among them, N capture It represents the number of malicious data flows captured by the honeypot within a unit time T. In order to compare the trapping effects of different decision time steps, T is usually set to a fixed value, such as 1 minute, 10 minutes, etc. decoy The value range of r decoy The minimum value of r is 0, which means that no malicious data flow is captured per unit time; decoy There is no fixed upper limit for the maximum value of r, which depends on the overall scale of malicious data flows and the deployment effect of the honeypot; in practical applications, r can be estimated based on historical data and experience. decoy The normal value range of r is , and abnormal values ​​outside this range may indicate that the honeypot is exposed or attacked; therefore, the optimization goal of the intelligent agent is to maximize r decoy , making it as high as possible above the baseline level. Local state and local reward, at each decision time step t, the honeypot deployment agent selects the honeypot type action a based on the local state information of <attack state, honeypot trapping state> decoy After the action is executed, the environment will feedback the attack state and honeypot trapping state at the next moment, as well as the local reward r calculated based on these states. decoy , state transition sample at time step t t ,a t ,r t ,s t+1 > is used to train and optimize honeypot deployment strategies, where r_t is the r calculated by the formula decoy . ​​

[0072] Preferably, the two agents are based on their own local reward functions r limit and r decoy Learning local optimal policies: The flow control agent learns to maximize r limit , that is, minimize the malicious traffic after flow restriction; honeypot deployment agent learns to maximize r decoy , which is to maximize the malicious traffic captured per unit time; the local goals of the two agents seem independent, but in fact they affect each other: current limiting reduces the total amount of malicious traffic and reduces the capture space of the honeypot; honeypot captures malicious traffic, which is equivalent to current limiting in advance; the global reward function R global Unify the local contributions of the two agents into the evaluation of the global defense effect: limit Implicit in R global The current limiting effect R limit In, R limit Not only the decline rate of malicious traffic (i.e., r limit ), and the maintenance rate of normal flow is also considered; r decoy Implicit in R global The trapping effect R decoy In, R decoy Not only the amount of captured malicious traffic (i.e., r decoy ), and also considers the amount of collected attack information; R global In R limit and R decoy On the basis of flow and the risk level R attack , evaluate the defense effect from a more comprehensive perspective. global As the common learning goal of the two agents, the unification of local strategy and global strategy is achieved: while optimizing their own local strategies, the two agents are trying to minimize R global , that is, maximizing the global defense effect; local rewards provide timely and accurate strategy feedback, accelerating the learning process of the agent; and global rewards provide long-term and comprehensive optimization directions, avoiding the blindness of local strategies; through the dual drive of local rewards and global rewards, the traffic control and honeypot deployment agents form a closely coordinated and dynamically balanced joint defense system.

[0073] Figure 5 According to the exemplary flow chart of updating the policy network parameters shown in the present application, the policy network of the agent is constructed, and the flow control agent and the honeypot deployment agent respectively construct independent policy networks: the policy network of the flow control agent is based on the global state and the local reward function r limit As input; the policy network of the honeypot deployment agent takes the global state and the local reward function r decoyThe policy network structure of the two agents is similar, but the input and output are different. The input of the policy network: global state, which contains comprehensive information about malicious data flows, including: traffic state: reflecting the traffic characteristics of malicious data flows, such as traffic size, burstiness, etc.; attack state: reflecting the attack characteristics of malicious data flows, such as attack type, degree of danger, etc.; flow control state: reflecting the current flow control strategy, such as flow limiting threshold, flow limiting effect, etc.; honeypot trapping state: reflecting the current honeypot deployment strategy, such as honeypot type, trapping effect, etc. The output of the policy network: action probability distribution, the output of the traffic control agent is the probability distribution of flow limiting actions, each action corresponds to a flow limiting threshold, and the output of the honeypot deployment agent is the probability distribution of honeypot deployment actions, each action corresponds to a honeypot type. The action probability distribution reflects the possibility that the agent tends to choose various actions in the current state. The structure of the policy network adopts a multi-layer fully connected neural network, which can flexibly fit the complex mapping relationship from state to action probability distribution. The number of nodes in the input layer is equal to the dimension of the global state, and the number of nodes in the output layer is equal to the size of the action space. The number of hidden layers and the number of nodes in each layer can be adjusted according to actual needs to balance fitting ability and computational efficiency.

[0074] Generation of protection actions, the agent inputs the strategy network according to the global state, and the flow control agent converts the current global state s limit Input to its policy network π limit In the honeypot deployment agent, the current global state s decoy Input to its policy network π decoy The global state includes the traffic state, attack state, traffic control state and honeypot trapping state, which reflects the overall situation of the current malicious traffic. The policy network outputs the action probability distribution, and the policy network π_limit of the traffic control agent outputs the probability distribution p of the flow limiting action. limit ; Policy network π of honeypot deployment agent decoy Output the probability distribution p of the honeypot deployment action decoy ; The action probability distribution reflects the probability of the agent tending to choose each action in the current state. Random sampling is used to generate specific protection actions from the action probability distribution. The flow control agent selects the action according to the flow limiting action probability distribution p limit , using random sampling to generate specific current limiting actions a limit ; Honeypot deployment agent according to the honeypot deployment action probability distribution p decoy , using random sampling to generate specific honeypot deployment actions a decoy; Random sampling can strike a balance between exploring new strategies and utilizing existing strategies, avoiding falling into local optimality too early. In this embodiment, the flow control agent's flow limiting action space is {no flow limiting, flow limiting 10%, flow limiting 20%, flow limiting 30%}, and the action probability distribution in the current state is {0.2, 0.5, 0.2, 0.1}, then the flow limiting action generated by sampling may be "flow limiting 10%". The honeypot deployment action space of the honeypot deployment agent is {no deployment, deploy low-interaction honeypot, deploy high-interaction honeypot}, and the action probability distribution in the current state is {0.1, 0.6, 0.3}, then the honeypot deployment action generated by sampling may be "deploy low-interaction honeypot". The update of the environmental state, the protection action acts on the malicious data flow, and the flow limiting action a generated by the flow control agent limit Act on malicious data flows to restrict and filter them. Honeypot deployment action a generated by the honeypot deployment agent decoy Act on malicious data flows to trap and analyze them. Different combinations of protection actions will have different impacts and effects on malicious data flows.

[0075] When the environment state changes and enters a new state, the traffic state and attack state of the malicious data flow will change accordingly under the action of the protection action. At the same time, the flow control state and honeypot trapping state will also be updated to reflect the effect of the protection action. The new environment state s' includes various state components after the protection action takes effect, reflecting the latest changes in the malicious data flow situation. The new environment state reflects the immediate effect of the protection action. By comparing the changes between the new and old environment states, the immediate effect of the protection action can be evaluated. For example, the flow limiting action reduces the malicious flow and maintains the normal flow, indicating that the flow limiting effect is good. For another example, the honeypot deployment action causes a large amount of malicious traffic to enter the honeypot and the attack information is captured, indicating that the honeypot has played a role. Timely and accurate evaluation of the effect of the protection action provides an important basis for adjusting and optimizing the protection strategy.

[0076] Calculation of reward value: Calculate the global reward value based on the new environment state; the global reward value r global By the global reward function R global (s') is calculated; R global The overall protection effect of traffic control and honeypot deployment is comprehensively considered, reflecting the global defense level of the system; specifically, R global It can be designed based on factors such as the degree of abnormal malicious traffic, the degree of attack danger, the effect of traffic restriction, the effect of honeypot capture, etc. According to the new environmental state, the local reward value is calculated, and the local reward value r of the traffic control agent is limit By the local reward function r limit (s') is calculated, r limitIt mainly reflects the inhibitory effect of the current limiting action on malicious traffic, which can be designed based on indicators such as the malicious traffic change rate; the local reward value r of the honeypot deployment agent decoy By the local reward function r decoy (s') is calculated, r decoy It mainly reflects the trapping effect of honeypot deployment on malicious traffic. It can be designed based on indicators such as the number of malicious traffic captured per unit time, the relationship between global rewards and local rewards, and the global reward r global The coordinated protection effect of traffic control and honeypot deployment is evaluated as a whole, which is the common optimization goal of the two agents; the local reward r limit and r decoy Evaluate the protection effects of each from the local perspective of traffic control and honeypot deployment respectively; guide the agents to optimize their own strategies through local rewards and guide the agents to cooperate with each other through global rewards, so as to form linkage at the local and global levels; finally, the agents will converge to the optimal joint strategy that maximizes the global reward, achieving the global defense goal while meeting the local protection needs.

[0077] Construct a four-tuple sample. The components of the four-tuple sample are the global state s, including the current traffic state, attack state, traffic control state and honeypot trapping state; the traffic state describes the scale, rate and other characteristics of malicious traffic; the attack state describes the attack type, intensity and other characteristics of malicious traffic; the traffic control state describes the current flow limiting strategy, flow limiting effect and other information; the honeypot trapping state describes the current honeypot deployment strategy, trapping effect and other information; the global state s comprehensively describes the overall situation of malicious traffic at the current moment; the protection action a, the flow limiting action a selected by the traffic control agent limit , the honeypot deployment action a selected by the honeypot deployment agent decoy , the protection action a is generated by sampling the agent's strategy network under the global state s. Different combinations of protection actions reflect the agent's dynamic response strategy to the current malicious traffic situation. Reward value r, global reward value r global , which reflects the overall protection effect of flow control and honeypot deployment, and the local reward value r of the flow control agent limit , reflects the local effect of the current limiting action, and the local reward value r of the honeypot deployment agent decoy , reflecting the local effect of the honeypot deployment action; the reward values ​​of different granularities reflect the local and global impact of the agent's behavior; the new environment state s' includes the traffic state, attack state, traffic control state and honeypot trapping state after the protection action is executed; the new environment state s' reflects the immediate impact of the protection action on the malicious traffic situation; by comparing the new and old states, the effect of the protection action can be evaluated to guide subsequent decisions.

[0078] In the process of constructing the four-tuple sample, at each decision time step, the agent first obtains the current global state s; through the environment perception module, various state information is collected and processed to form a global state representation; the global state s is used as the input of the agent strategy network to generate protection actions; the agent inputs the global state s into the strategy network, samples and generates protection actions a; the flow control agent generates flow limiting actions a limit ; Honeypot deployment agent generates honeypot deployment action a decoy .

[0079] The protection action a reflects the decision result of the agent in the current state s. The protection action a acts on the malicious data flow, and the environment enters a new state s', and generates a reward value r. The new environment state s' includes all state components after the protection action. The global reward value r global Calculated according to the new state s', reflecting the overall protection effect; local reward value r limit and r decoy It is also calculated based on the new state s', reflecting the local effect; s, a, r, s' are combined into a four-tuple sample (s, a, r, s'); s is the state before the agent performs the action, a is the action selected by the agent in state s, r is the reward value obtained by the agent after performing action a, and s' is the new state entered by the environment after performing action a. The four-tuple sample records a complete step of the interaction between the agent and the environment, and the four-tuple sample (s, a, r, s') is stored in the experience replay pool. The experience replay pool is used to store the four-tuple samples generated by the interaction between the agent and the environment. By continuously interacting with the environment, the agent can continuously collect and accumulate four-tuple samples. The experience replay pool provides training data for the learning and strategy optimization of the agent.

[0080] Preferably, the four-tuple sample is the basic unit for training the policy network; the policy network learns the mapping relationship from state to action by fitting a large number of four-tuple samples; the policy network gradually masters the optimal decision-making strategy by optimizing the cumulative reward of the four-tuple samples; the four-tuple samples contain complete information about the interaction between the agent and the environment; the global state s provides the basis for decision-making, and the protective action a provides the decision result; the reward value r provides the decision evaluation, and the new state s' provides the decision impact; the four-tuple sample embodies the complete causal chain of "state-action-reward-new state". Quadruple samples support distributed learning of multiple agents. Each agent optimizes local strategies based on local reward values ​​using its own quadruple samples. Different agents learn collaborative strategies from each other's quadruple samples through global reward values. Quadruple samples help multiple agents achieve a balance between local optimization and global collaboration. Quadruple samples can be used for offline strategy evaluation and improvement. Through offline analysis of quadruple samples, the decision quality of agents in various states can be evaluated. Through offline strategy improvement algorithms, quadruple samples can be used to optimize the policy network offline. Offline optimization can accelerate the convergence of the policy network and reduce the cost of online interaction.

[0081] TD is calculated using the time difference (TD) algorithm error Temporal difference is a commonly used strategy evaluation method in reinforcement learning, which is used to estimate the state-action value function Q(s,a). For each four-tuple sample (s,a,r,s'), the temporal difference error TD is calculated according to the TD algorithm. error TD error Reflects the difference between the current Q value estimate and the actual reward and the maximum Q value of the next state, the temporal difference error TD error The calculation formula of TD error =r+γ(s',a')-Q(s,a), where r is the current reward value, γ is the discount factor, Q(s,a) is the current state-action value function, Q(s',a') is the maximum Q value of the next state, the discount factor γ (0<γ≤1) indicates the degree of attenuation of future rewards, the larger the γ, the higher the importance attached to future rewards, the significance of the state-action value function Q(s,a), Q(s,a) indicates the expected value of the long-term cumulative reward of taking action a in state s, Q(s,a) reflects the long-term benefit of choosing action a in the current state s, the larger the value of Q(s,a), the better the decision of choosing action a in state s, the calculation of the maximum Q value Q(s',a') of the next state, It means that in the next state s', the action a' that maximizes the Q value is selected. a' is sampled according to the action probability distribution of the current strategy network in state s'. Q(s',a') reflects the maximum expected benefit that can be obtained by following the current strategy in the next state s'.

[0082] Timing Differential Error TD error The role of TD error >0, indicating that the current Q value estimate is lower than the actual return, and the estimate of Q(s,a) needs to be improved; TD error <0, indicating that the current Q value estimate is higher than the actual return, and the estimate of Q(s,a) needs to be lowered; TD error =0, indicating that the current Q value estimate is accurate and consistent with the actual return; TD error The larger the absolute value of , the greater the difference between the current Q value estimate and the actual return, and the greater the adjustment required. error The positive or negative and size of can guide the adjustment direction and strength of the policy network parameters.

[0083] The parameters of the policy network are updated by using the gradient descent algorithm, and the temporal difference error TD error As the loss function of the policy network, the gradient descent algorithm is used to error In the descending direction, the policy network parameter θ is updated. Through parameter update, the action probability distribution generated by the policy network is continuously brought closer to the optimal policy. The update formula of the policy network parameter θ is θ'=θ+α×TD error ×▽θlogπθ(s,a), where α is the learning rate, πθ(s,a) is the probability that the policy network with parameter θ takes action a in state s, ▽θlogπθ(s,a) is the gradient of the logarithmic probability of πθ(s,a) with respect to parameter θ, and the role of the learning rate α. The learning rate α (0<α≤1) controls the step size of each parameter update; the larger α is, the larger the amplitude of each parameter adjustment is, the faster the learning speed is, but it may cause oscillation and instability; the smaller α is, the smaller the amplitude of each parameter adjustment is. The smaller the degree, the slower the learning speed, but it is more conducive to convergence to the optimal strategy. In practice, a learning rate that decays over time is usually used to balance the learning speed and stability. The meaning of the policy gradient ▽θlogπθ(s,a), ▽θlogπθ(s,a) represents the gradient of the logarithmic probability of πθ(s,a) with respect to the parameter θ. The direction of the gradient indicates the direction in which the parameter θ needs to be adjusted to increase the probability of πθ(s,a). The size of the gradient indicates the sensitivity of the probability of πθ(s,a) to the parameter θ. According to TD error Whether positive or negative, the gradient adjusts the parameters in the direction of increasing or decreasing the estimate of Q(s,a).

[0084] The effect of parameter update: Through parameter update, the action probability distribution πθ(s,a) generated by the policy network continues to move closer to the optimal policy π*(s,a); under the optimal policy π*(s,a), for each state s, the probability of the agent choosing action a will maximize the long-term cumulative reward; with the continuous optimization of the policy network parameters, the decision-making ability of the agent will continue to increase, and it will be able to make more accurate and efficient protection decisions. Multiple rounds of iterative optimization: the agent interacts with the environment in multiple rounds, starting from the initial state s0, and the agent interacts with the environment in time steps to form a sequence of states, actions, and rewards; at each time step t, the agent generates action a through the policy network based on the current state st. t , and observe the reward r of the environment feedback t and the next state s t+1 ; Multiple rounds of interaction generate a large number of four-tuple samples (s, a, r, s') to provide data support for the training of the policy network; the decision-making, learning, and updating processes are repeated in each round of interaction; in each round of interaction, the flow control agent and the honeypot deployment agent respectively perform the following processes: according to the current state, generate protection actions through the policy network (decision); according to the four-tuple samples generated by the protection action, calculate the time difference error (learning); according to the time difference error, use the gradient descent algorithm to update the policy network parameters (update); the decision-making, learning, and updating processes are intertwined to form a closed-loop feedback mechanism; through this mechanism, the agent can continuously adjust and improve the strategy according to the consequences of its own behavior; as the number of iterations increases, the policy network gradually converges; as the number of rounds of interaction between the agent and the environment increases, the parameters of the policy network will be continuously optimized; the action probability distribution generated by the policy network will gradually converge to the optimal strategy. The converged policy network can maximize the long-term cumulative reward of the agent and achieve the global optimal protection effect.

[0085] The dynamic balance characteristic of the optimal strategy: The optimal strategy aims to maximize the global reward while taking into account the overall effect of traffic control and honeypot deployment; at the same time, the optimal strategy will also weigh the local protection effect to ensure that traffic control and honeypot deployment perform well in their respective subtasks; the optimal strategy achieves a dynamic balance between global optimization and local optimization, which not only exerts the overall synergy effect but also does not ignore the local protection needs. The flexible adaptability of the optimal strategy: The optimal strategy generates the corresponding optimal protection decision based on the dynamic changes of malicious traffic. In the face of new attack states, the optimal strategy can quickly adjust the traffic control and honeypot deployment strategies to achieve precise protection. The optimal strategy has strong robustness and adaptability, and can flexibly respond to complex and changing malicious traffic scenarios.

[0086] Figure 6This is an exemplary flow chart for obtaining the global optimal joint protection strategy shown in the present application, obtaining the global state, local observation of the flow control agent, observing the flow state of the malicious data flow through the local state space, the flow state includes the scale, rate, distribution and other characteristics of the malicious flow, and the local observation provides local state information for the flow control agent. The local observation of the honeypot deployment agent, observing the attack state of the malicious data flow through the local state space, the attack state includes the attack type, intensity, frequency and other characteristics of the malicious flow, and the local observation provides local state information for the honeypot deployment agent. The flow control agent communicates with the honeypot deployment agent, the flow control agent obtains the honeypot trapping state of the malicious data flow through communication, and the honeypot deployment agent obtains the flow control state of the malicious data flow through communication, and the communication enables both agents to obtain each other's local state information. Integration of global states: The traffic control agent integrates the observed traffic state and the honeypot trapping state obtained through communication into its own global state. The honeypot deployment agent integrates the observed attack state and the traffic control state obtained through communication into its own global state. The global state integrates the complete situation of malicious data flows in the two dimensions of traffic control and honeypot deployment. The global state is used as the input of the policy network. The traffic control agent uses its own global state as the input of the policy network, and the honeypot deployment agent uses its own global state as the input of the policy network. The global state provides a comprehensive and accurate basis for the decision-making of the agent.

[0087] Collaborative optimization of flow control agent and honeypot deployment agent, state observation and information interaction of agents, state observation of flow control agent, flow control agent observes the flow state of malicious data flow through local state space, the flow state includes the flow size, flow change trend, flow distribution characteristics, etc. of malicious data flow; flow control agent uses flow monitoring and analysis tools to obtain the flow state information of malicious data flow in real time; state observation of honeypot deployment agent; honeypot deployment agent observes the attack state of malicious data flow through local state space; the attack state includes the attack type, attack intensity, attack frequency, etc. of malicious data flow; honeypot deployment agent uses honeypot system and intrusion detection system to obtain the attack state information of malicious data flow in real time. The state information interaction between agents: the flow control agent obtains the honeypot trapping state of the malicious data flow through communication with the honeypot deployment agent; the honeypot trapping state reflects the interaction of the malicious data flow with the honeypot system, such as detection frequency, penetration depth, etc.; the honeypot deployment agent obtains the flow control state of the malicious data flow through communication with the flow control agent; the flow control state reflects the impact of the flow limiting measures on the malicious data flow, such as flow limiting threshold, flow limiting effect, etc.; the state information interaction can be achieved through communication mechanisms such as RESTfulAPI and message queues.

[0088] State integration and policy optimization of intelligent agents, state integration of traffic control intelligent agents; the traffic control intelligent agent integrates the observed traffic state and the honeypot trapping state obtained through communication; the integrated state information forms the global state representation of the traffic control intelligent agent; the global state representation comprehensively reflects the behavioral characteristics of malicious data flows in the two dimensions of traffic and honeypot; the traffic control intelligent agent uses the global state as the input of the policy network for subsequent policy optimization. State integration of honeypot deployment intelligent agents, the honeypot deployment intelligent agent integrates the observed attack state and the traffic control state obtained through communication, and the integrated state information forms the global state representation of the honeypot deployment intelligent agent, the global state representation comprehensively reflects the behavioral characteristics of malicious data flows in the two dimensions of attack and traffic, and the honeypot deployment intelligent agent uses the global state as the input of the policy network for subsequent policy optimization.

[0089] Based on deep reinforcement learning strategy optimization, deep reinforcement learning models are established for the flow control agent and the honeypot deployment agent respectively, and the policy networks are trained and optimized through algorithms such as Actor-Critic and DDPG. The policy network of the flow control agent takes the global state as input and outputs the optimized flow limiting strategy, mainly adjusting parameters such as the flow limiting threshold and flow limiting duration. The policy network of the honeypot deployment agent takes the global state as input and outputs the optimized honeypot deployment strategy, mainly adjusting parameters such as the honeypot type and honeypot topology. The training of the policy network takes the maximization of the global reward as the optimization goal, and updates the network parameters through methods such as gradient ascent to make the protection strategy continuously tend to the optimal. Reward sharing and strategy coordination of intelligent agents, reward sharing based on the global reward function, the flow control agent and the honeypot deployment agent obtain the same global reward signal based on the common global reward function. The global reward signal comprehensively considers the protection effect of flow control and honeypot deployment, as well as the security and availability of the overall system. The global reward signal serves as the driving force for the optimization of the agent strategy, guiding the agent to adjust its strategy in the direction of the global optimum. The reward sharing mechanism enables the two agents to obtain consistent feedback during the optimization process and form strategic coordination.

[0090] Protection strategies are shared among agents. The flow control agent shares the optimized flow limiting strategy with the honeypot deployment agent, and the honeypot deployment agent shares the optimized honeypot deployment strategy with the flow control agent. Strategy sharing can be achieved through communication mechanisms such as RESTful API and message queues, and strategy parameters in JSON format are transmitted. The shared protection strategy provides additional optimization information for the agents, enabling them to consider each other's strategy trends when making decisions. Re-optimization based on shared strategies makes the strategy adjustments of both agents more coordinated and efficient.

[0091] Through strategy interaction, the traffic control agent and the honeypot deployment agent form strategy synergy. Through multiple rounds of strategy interaction and strategy optimization, the traffic control agent and the honeypot deployment agent make their respective protection strategies constantly tend to the best under the global reward. Each round of strategy interaction includes state observation, state integration, strategy optimization, strategy sharing and other links, so that the strategy optimization is gradually deepened in the collaboration. After multiple rounds of iterations, the protection strategies of both agents will form the best strategy synergy under the guidance of the global reward. Finally, traffic control and honeypot deployment will form a joint protection strategy to provide all-round and dynamic security protection for malicious data flows. Dynamic execution of joint protection strategy, environmental deployment of joint protection strategy, deploy the optimized flow limiting strategy of the traffic control agent to the online traffic cleaning system, control malicious traffic in real time, deploy the honeypot deployment strategy optimized by the honeypot deployment agent to the online honeypot array, and trap and analyze malicious traffic; through the policy delivery interface and configuration management tools, the joint protection strategy and network environment can be quickly adapted and dynamically updated.

[0092] Adjustment of joint protection strategy based on feedback. During the execution of the joint protection strategy, the attack effect of malicious data flow is continuously monitored, and feedback on the protection effect of flow control and honeypot deployment is collected. According to the feedback on the protection effect, the flow control agent and the honeypot deployment agent dynamically adjust the deployed joint protection strategy. The dynamic adjustment of the joint protection strategy can be triggered by setting a threshold, or executed regularly according to the time window. The dynamic adjustment enables the joint protection strategy to adapt to the changes of malicious data flow and maintain a high protection effectiveness. In the actual dynamic protection process, the optimized joint protection strategy is applied to the malicious data flow in the network environment. According to the changes in the flow state and attack state of the malicious data flow, the flow control agent dynamically adjusts the flow limit threshold to control the malicious flow; the honeypot deployment agent dynamically adjusts the deployment of the honeypot system to trap the malicious flow.

[0093] Preferably, an attack and defense confrontation simulation environment is constructed, and anomaly detection algorithms and multi-agent reinforcement learning algorithms are trained and evaluated by generating adversarial samples and adversarial strategies, including: based on the original malicious data flow and normal data flow samples, data enhancement technology is used to generate adversarial samples, the feature vector of the malicious data flow is perturbed, and the feature value range is changed by adding random noise to the feature vector to generate malicious adversarial samples; wherein the added noise amplitude is less than the detection threshold of the anomaly detection algorithm, so that the disturbed malicious adversarial samples can evade detection; the feature vector of the normal data flow is perturbed, and normal adversarial samples are generated by adding random noise similar to the malicious data flow characteristics to the feature vector; wherein the added noise amplitude is greater than the anomaly detection threshold, so that the disturbed normal adversarial samples are misjudged as malicious data flows; according to the attack strategy adopted by the original attacker, a strategy transformation technology is used to generate an adversarial strategy, and the attack actions in the original attack strategy are randomly replaced with other attack types. The attack action parameters in the original attack strategy are randomly perturbed, and the range of perturbation parameters is changed to generate an attack strategy with parameter changes; the original attack strategy is randomly combined with strategies of other attack types, and the attack action sequence is combined to generate a combined attack strategy; in a simulation environment, the generated adversarial samples and adversarial strategies are used to train the anomaly detection algorithm and the multi-agent reinforcement learning algorithm, and the generated malicious adversarial samples and normal adversarial samples are used to form a training sample set with the original malicious data flow samples and normal data flow samples for training and evaluating the anomaly detection algorithm; by adding adversarial samples, the robustness and generalization performance of the anomaly detection algorithm are improved; in the training process of the reinforcement learning agent, the generated adversarial strategy is added to the simulation environment as an additional attack behavior to compete with the defense agent; by introducing the adversarial strategy, the defense agent can adapt to more complex and changeable attack scenarios and improve the effectiveness of the protection strategy.

Claims

1. A data security dynamic protection method based on artificial intelligence, comprising: Get data flow logs; Based on the data flow log, anomaly detection algorithms are used to identify malicious data flows and attack features. Malicious data flows are data flow segments that contain attack features such as SQL injection, DDoS, or command execution. Based on the identified malicious data streams, a multi-agent reinforcement learning algorithm is used to adjust the protection strategy based on the current environment status and attack characteristics through multiple agents; Construct a simulation environment to train and evaluate anomaly detection algorithms and multi-agent reinforcement learning algorithms by generating adversarial samples and adversarial strategies. The adversarial samples are generated by transforming the features of malicious data streams and normal data streams, and the adversarial strategies are generated by randomly perturbing existing attack strategies. Among them, based on the data flow log, anomaly detection algorithms are used to identify malicious data flows and attack characteristics, including: Extract feature vectors from data flow logs. Feature vectors include source IP address, destination IP address, port number, and data packet content of the data flow. A clustering algorithm based on Gaussian mixture model is used to cluster feature vectors to obtain K clusters. The mean vector and covariance matrix of each cluster are calculated. According to the sample proportion and Mahalanobis distance of each cluster, whether each cluster is an abnormal cluster is determined. Among them, clusters with sample proportion less than the threshold and Mahalanobis distance greater than the threshold are determined to be abnormal clusters, which represent potential unknown attack patterns. The recognition model is built based on the random forest algorithm. One-hot encoding is used to convert feature vectors into numerical representations, and a sample set is built in combination with labeled malicious data flow samples. The sample set is used to perform supervised training on the recognition model to obtain a recognition model that can recognize known attack types. The trained recognition model is used to predict the feature vectors of the data flow to be detected, and the known attack type to which the malicious data flow belongs is predicted. According to the abnormal clustering clusters and the predicted known attack types, malicious data flows are determined based on rules and corresponding attack features are extracted.

2. The data security dynamic protection method based on artificial intelligence according to claim 1 is characterized by: Determine malicious data flows based on rules and extract corresponding attack features, including: If the data flow belongs to an abnormal cluster but does not belong to a known attack type, it is determined to be an unknown attack, and its abnormal features are extracted as attack features and marked as an unknown attack; If the data flow belongs to a known attack type but does not belong to an abnormal cluster, it is determined to be a known type of malicious data flow, and its corresponding attack type and attack characteristics are extracted as attack features; If the data flow belongs to both an abnormal cluster and a known attack type, it is determined to be a variant attack, and its abnormal features and known attack features are extracted and marked as a variant attack.

3. The data security dynamic protection method based on artificial intelligence according to claim 2 is characterized by: Determine whether each cluster is an abnormal cluster, including: The expectation maximization (EM) algorithm is used to estimate the parameters of the extracted feature vectors to obtain the mean vector, covariance matrix and mixing coefficient of K Gaussian distributions, where K is the preset number of clusters. According to the mixing coefficient and Gaussian distribution parameters, the posterior probability that the feature vector of each data stream belongs to each cluster is calculated, and each data stream is divided into the cluster with the largest posterior probability to obtain K clusters; Calculate the number of data flows in each cluster, and calculate the ratio of the number of each cluster to the total number of data flows; when the ratio is less than the threshold, the corresponding cluster is judged to be a small cluster, which means that the number of data flows in the corresponding cluster is abnormally sparse; For each cluster, the mean vector of the feature vectors of all data streams in the corresponding cluster is calculated as the center point of the corresponding cluster; the Mahalanobis distance from the feature vector of each data stream in the cluster to the center point of the cluster is calculated; when the Mahalanobis distance of a data stream is greater than the threshold, the corresponding data stream is judged as an outlier far away from the center of the cluster; Count the number of outliers in each cluster, and calculate the average Mahalanobis distance from all outliers in the corresponding cluster to the cluster center as the average deviation degree of the corresponding cluster; The first N clusters that belong to the small cluster and have the largest average deviation are determined as abnormal clusters.

4. The data security dynamic protection method based on artificial intelligence according to claim 3 is characterized by: Based on the random forest algorithm, a recognition model is built to predict the feature vectors of the data stream to be detected, including: Extract features from the marked malicious data flow samples and normal data flow samples to obtain a training sample set; Based on the training sample set, the Bootstrap method is used for random sampling. Some features are randomly selected from all features to build multiple CART decision trees to form a random forest. When each decision tree splits a node, the best split feature is selected based on the randomly selected feature subset and the Gini index criterion. The grid search method is used to optimize the hyperparameters of random forest, including the number of decision trees, the maximum depth of each decision tree, and the minimum number of samples for node splitting. The performance of the model under different hyperparameter combinations is evaluated through cross-validation, and the hyperparameter combination with the best model performance is selected. According to the optimal hyperparameter combination, the random forest model is trained using the training sample set to obtain a recognition model, and the trained recognition model is used to obtain the attack type prediction result of the input data stream.

5. The data security dynamic protection method based on artificial intelligence according to any one of claims 2 to 4 is characterized in that: Adopt multi-agent reinforcement learning algorithm to adjust the protection strategy, including: According to the identified malicious data flow, a flow control agent and a honeypot deployment agent are set; wherein the flow control agent performs flow control according to the flow state of the malicious data flow, and the honeypot deployment agent performs honeypot trapping according to the attack state of the malicious data flow; The state space of the flow control agent is constructed based on the flow size and flow mutation degree of the malicious data flow; the state space of the honeypot deployment agent is constructed based on the source IP address and attack type of the malicious data flow; the flow control agent obtains the flow state of the malicious data flow through the state space, and the honeypot deployment agent obtains the attack state of the malicious data flow through the state space; Different levels of flow restriction thresholds are used as the action space of the flow control agent; honeypot systems with different degrees of interaction are used as the action space of the honeypot deployment agent; the flow control agent selects the flow restriction threshold from the action space and performs flow control according to the flow state of the malicious data flow, and the honeypot deployment agent selects the honeypot system from the action space and performs honeypot trapping according to the attack state of the malicious data flow; Construct a global reward function based on the quantified value of the degree of harm caused by malicious data flow to the system; Based on the flow changes of malicious data flow before and after flow limiting, the local reward function r of the flow control agent is constructed limit ; Based on the number of malicious data flows captured per unit time, a local reward function r of the honeypot deployment agent is constructed decoy ; Among them, the local reward function r of the flow control agent is limit The calculation formula is: Flow before and Flow after They respectively represent the traffic volume of malicious data flows before and after current limiting; The local reward function r of the honeypot deployment agent decoy The calculation formula is: N capture It represents the number of malicious data flows captured by the honeypot within a unit time T; The flow control agent uses the global state and the local reward function r limit As input, the honeypot deploys the agent with a global state and a local reward function r decoy As input, a policy network is constructed through a deep reinforcement learning algorithm, the action probability distribution is output, and the protection action is generated; The traffic control agent and the honeypot deployment agent obtain the global status through local observation and communication, and obtain the global optimal protection strategy; The traffic control agent and the honeypot deployment agent obtain the global state through local state observation and mutual communication, adjust their respective policy network parameters, and share the global reward signal R global , collaboratively optimize the protection strategy with the goal of maximizing the global reward.

6. The data security dynamic protection method based on artificial intelligence according to claim 5 is characterized by: A global reward function is constructed based on the quantitative value of the degree of harm caused by malicious data flow to the system, including: Obtain the traffic state, attack state, traffic control state and honeypot trapping state of the malicious data flow; wherein the traffic state represents the traffic characteristics of the malicious data flow; the attack state represents the attack characteristics of the malicious data flow; the traffic control state represents the flow limiting effect of the traffic control agent on the malicious data flow through the malicious traffic reduction rate and the normal traffic retention rate; the honeypot trapping state represents the trapping effect of the honeypot deployment agent on the malicious data flow through the honeypot capture rate and the amount of attack information collected; Quantify the traffic status as traffic anomaly degree, quantify the attack status as attack risk degree, quantify the traffic control status as flow limiting effect degree, and quantify the honeypot trapping status as trapping effect degree; Construct a global reward function, take traffic anomaly, attack risk, flow limiting effect and trapping effect as input, calculate the global reward value, the global reward function R global It is expressed as: R global =w1×R flow +w2×R attack +w3×R limit +w4×R decoy Among them, R flow Indicates the flow abnormality, R attack Indicates the attack risk, R limit Indicates the current limiting effect, R decoy It represents the trapping effectiveness, w1, w2, w3, w4 are weight coefficients.

7. The data security dynamic protection method based on artificial intelligence according to claim 6 is characterized by: Quantitative calculation formula for traffic anomaly: Among them, Flow malicious Indicates the average flow rate of malicious data flow, Flow normal Indicates the average flow rate of normal data flow; Attack risk quantification calculation formula: Among them, Attack i represents the i-th attack type of the data flow, w i represents the weight of the danger level of the i-th attack type, and N represents the total number of attack types considered; The quantitative calculation formula of current limiting effect is: Among them, Flow normal_after and Flow normal_before They represent the flow of normal data flow before and after flow limiting respectively, and k1 and k2 are the weight coefficients of the two indicators.

8. The data security dynamic protection method based on artificial intelligence according to claim 7 is characterized by: Trapping effect quantitative calculation formula: Among them, N capture Represents the number of malicious data flows captured by the honeypot, N total Indicates the total number of malicious data flows, Info collect It represents the amount of attack information collected within a unit time T, and k3 and k4 are the weight coefficients of the two indicators.

9. The data security dynamic protection method based on artificial intelligence according to claim 8 is characterized by: Build a policy network through deep reinforcement learning algorithms, including: Construct a policy network of the intelligent agent, take the global state as input, and output the action probability distribution; the global state includes the traffic state, attack state, traffic control state and honeypot trapping state of the malicious data flow; According to the action probability distribution output by the strategy network, the agent uses random sampling to obtain the corresponding protection action; among them, the protection action of the flow control agent is the flow limiting action, and the protection action of the honeypot deployment agent is the honeypot deployment action; After applying the protection action to the malicious data flow, a new environment state is obtained, and a global reward value and a local reward value are calculated; wherein the new environment state includes the flow state, attack state, flow control state, and honeypot trapping state of the malicious data flow after the protection action is applied; The global state, protection action, global reward value and local reward value, and new environment state form a four-tuple sample (s, a, r, s'), where s is the global state, a is the protection action, r is the global reward value and local reward value, and s' is the new environment state; Use the time difference algorithm to calculate the time difference error TD of the four-tuple sample error ; Among them, the calculation formula of timing difference error is: TD error =r+γ(s',a')-Q(s,a) Among them, Q(s,a) is the state-action value function, which represents the expected value of the long-term cumulative reward for taking action a in state s; γ is the discount factor, which represents the degree of decay of future rewards; Q(s',a') is the maximum Q value in the next state s', which is calculated based on the next state s' and the action probability distribution output by the policy network; Using the gradient descent algorithm, the temporal difference error TD error is the loss function, and the parameters of the policy network are updated so that the action probability distribution generated by the policy network approaches the optimal policy. The parameter update formula is: Where θ is the parameter of the policy network, α is the learning rate, and πθ(s,a) is the probability that the policy network with parameter θ takes action a in state s; Through multiple rounds of iterative optimization, the policy networks of the flow control agent and the honeypot deployment agent converge to the optimal strategy with the goal of maximizing the global reward.

10. The data security dynamic protection method based on artificial intelligence according to claim 9 is characterized in that: Collaboratively optimize the protection strategy with the goal of maximizing the global reward, including: The flow control agent observes the flow state of the malicious data flow through the local state space, and obtains the honeypot trapping state of the malicious data flow through communication with the honeypot deployment agent; the honeypot deployment agent observes the attack state of the malicious data flow through the local state space, and obtains the flow control state of the malicious data flow through communication with the flow control agent; The traffic control agent integrates the observed traffic state and the honeypot trapping state obtained through communication into its own global state as the input of the policy network; the honeypot deployment agent integrates the observed attack state and the traffic control state obtained through communication into its own global state as the input of the policy network; The flow control agent and the honeypot deployment agent adjust the parameters of their respective policy networks and optimize their respective protection strategies according to their own global states through deep reinforcement learning algorithms. The flow control agent outputs the optimized flow limiting strategy, and the honeypot deployment agent outputs the optimized honeypot deployment strategy. The traffic control agent and the honeypot deployment agent share the same global reward signal based on the global reward function, and collaboratively optimize their respective protection strategies with the goal of maximizing the global reward. The flow control agent shares the optimized flow limiting strategy with the honeypot deployment agent, and the honeypot deployment agent shares the optimized honeypot deployment strategy with the flow control agent. Both agents further adjust their own strategy network parameters based on the shared strategy information. Through multiple rounds of strategy interaction and strategy optimization, the traffic control agent and the honeypot deployment agent form a strategy collaboration to obtain the optimal joint protection strategy under the global reward; Apply the globally optimal joint protection strategy to malicious data flows in the network environment, and dynamically adjust the traffic control and honeypot deployment strategies.

Images