A multi-step DDoS prediction poisoning attack and defense method based on reinforcement learning

Abstract

The application provides a multi-step DDoS prediction poisoning attack and defense method based on reinforcement learning, comprising the steps of setting a threat model and hypothesis, wherein an alarm label for constituting a local training data set is included; a multi-step DDoS prediction poisoning attack model is established according to the alarm label; the poisoning attack model is optimized by using a poisoning algorithm based on reinforcement learning; the optimized poisoning attack model is used to perform poisoning attack on a global prediction model of unlabeled data; and the poisoning attack is defended based on D-S evidence theory. The multi-step DDoS prediction poisoning attack method and defense strategy based on reinforcement learning provided by the application can not only poison the global prediction model of unlabeled data in a short time, but also continuously accelerate the poisoning speed by using accumulated experience; secondly, the defense strategy based on D-S evidence theory without verification data can effectively avoid abnormal aggregation.

Description

Technical Field

[0001] This invention relates to a multi-step DDoS prediction poisoning attack method based on reinforcement learning and a defense strategy for the poisoning attack method, which relates to the field of intelligent transportation. Background Technology

[0002] With the development of intelligent transportation systems (ITS), an increasing number of wireless sensors are being used for information services. For example, Figure 1 As shown, roadside cameras, vehicle-mounted sensors, induction coils, and embedded road sensors are all connected to base stations (BS) for data processing and computation. As edge infrastructure, roadside base stations provide convenient local services to mobile users (such as traffic broadcasts and driver warning support), and they are also a crucial bridge connecting thousands of sensors and backend servers. However, an increasing number of smart sensors (such as cameras) can now launch attacks against base stations, particularly Distributed Denial of Service (DDoS) attacks. Due to the sheer number of sensors, they can easily become "zombie base stations" to carry out DDoS attacks. Figure 1 As shown, wireless sensors within communication range can establish normal connections with the base station. However, if malicious sensors send a large number of useless interference packets to the base station, ultimately launching a DDoS attack, communication on the base station will be blocked, and many normally connected sensors will be disabled. Therefore, base stations should be able to predict whether they will be subject to DDoS attacks in the future, so as to take action before a disaster occurs.

[0003] To date, much research has been conducted on DDoS attack detection, rather than predicting it. Most people consider DDoS attacks to be a single-step attack. However, in reality, DDoS attacks can be broken down into multiple steps and executed in certain strategic phases, posing a significant security risk to systems. If DDoS attacks can be detected early, we can take preventative action. This is what defense systems need. Therefore, researchers have proposed several machine learning (ML) and correlation-based techniques to predict the probability of DDoS attacks, such as Bayesian network-based methods, attack graph-based methods, and Hidden Markov Model-based (HMM) methods. While these methods have achieved good results by analyzing data packets, the following problems still exist:

[0004] First, most of these ML-based or correlation-based technologies are centralized approaches, meaning base stations need to collect data from different areas and send it to a central cloud server. Applying these methods directly to base stations in our city presents challenges to defense systems due to the large-scale transmission to cloud servers and privacy concerns. Intelligent defense / attack prediction systems and low-latency response mechanisms for each roadside infrastructure are essential.

[0005] Secondly, even if an attack prediction system is installed in a roadside base station, the limited training data and the lack of data aggregation capabilities in areas with sparse communication traffic mean that the model cannot be effectively trained, further contributing to the system's inability to predict DDoS attacks in a timely manner.

[0006] Furthermore, if Hidden Markov Models (HMMs) are used in conjunction with federated learning to implement a multi-step DDoS prediction model—a distributed machine learning paradigm—it can alleviate the pressure on base stations and create a secure and effective prediction system capable of determining whether a base station is under DDoS attack. In federated learning, each base station can be considered a participant. Participants can train their models (HMMs) based on local datasets without exchanging raw, private data with others. These participants upload their local model parameters to a cloud server, where these parameters are aggregated to form a global model. The cloud server then distributes the global model to the participants, and the process is repeated until the model converges.

[0007] While federated learning frameworks leverage the advantages of collaborative training and avoid the leakage of sensitive privacy data, vulnerabilities in data reliability still exist. During federated learning, even if we ensure that uploaded parameters are protected by various cryptographically based tamper-proof techniques, we cannot guarantee that all participants are trustworthy. Participants may upload distracting parameters to the server with malicious intent, i.e., launching a poisoning attack. If a malicious participant continuously uploads its distracting parameters to the server, the global model will be poisoned and unable to converge. Thus, the overall network's defense capability will be reduced due to the poisoning attack. Therefore, it is essential to ensure that uploaded parameters are valid.

[0008] Currently, there is considerable research on poisoning attacks and defense strategies. Regarding poisoning, recent GAN-based poisoning attacks primarily target classifiers / classification models with labeled data. The principle is to generate data using a discriminator and generator, then assign incorrect labels to this data. This means the poisoner knows how to process the data, thus negatively impacting the model. However, GAN-based poisoning attacks cannot poison unlabeled prediction models because GAN poisoners do not know how to process the data and cannot poison it by flipping labels. Furthermore, among various prediction models, HMM-based prediction models are widely used, popular, and excellent multi-step DDoS prediction models. Therefore, focusing on poisoning attacks against unlabeled data is highly meaningful. In terms of defense, many studies on secure aggregation assume that the server has a validation dataset with labeled data to determine whether the uploaded local model is poisoned. However, in practice, this assumption is unrealistic because, due to privacy protection, the local training dataset is not uploaded to the server, and our dataset is also unlabeled. Summary of the Invention

[0009] The purpose of this invention is to provide a multi-step DDoS prediction poisoning attack method and its defense strategy based on reinforcement learning, which can solve the problem of poisoning attacks and defenses against unlabeled data models in the prior art.

[0010] To achieve the above objectives, the technical solution of the present invention provides a multi-step DDoS attack prediction and defense method based on reinforcement learning, characterized by comprising the following steps:

[0011] Step 1: Set up the threat model and hypotheses, including the alert labels that make up the local training dataset, including the following:

[0012] Each base station is considered a participant, capable of training its Hidden Markov Model (HMM) based on a local dataset without exchanging raw privacy data with others. Furthermore, each base station is equipped with a commonly used packet sniffer capable of detecting anomalous packets and generating relevant alerts. These alerts provide a common language to describe suspicious events and ultimately correspond to alert labels such as "information," "remote user," "remote attacker," "denial of service," and "buffer overflow." Each base station uses these generated alerts to train its local model. It is assumed that different base stations have training datasets of varying sizes, and each participant is unaware of the models of other participants in the federated learning framework. Base stations continuously collect sensor-base station communication traffic and train their local prediction models to monitor for DDoS attacks within their communication range. Then, all participants located in different geographical areas update their local models to a cloud server, assuming the cloud server does not have a validation dataset. The cloud server aggregates these local models into a global model and further distributes the global model to each base station to achieve DDoS prediction at the base stations. After predicting a DDoS attack, each base station subsequently updates its training dataset with accumulated data from a DDoS observation sequence.

[0013] A discrete first-order hidden Markov model is defined as a quintuple, as shown in equation (1):

[0014] (1)

[0015] In equation (1), S describes a set of hidden Markov model states corresponding to the sub-steps of a DDoS attack. It is a set of observations based on alarm tags; E is the transition probability matrix describing the state transitions in set S; Q represents the observation probability matrix; This represents the initial distribution vector of each Hidden Markov Model state, which indicates the probability that an intrusion can begin in each state.

[0016] Step 2: Based on the alert labels, establish a multi-step DDoS attack prediction model, which includes the following steps:

[0017] Step 201: Based on the alert labels, define the state space of the multi-step DDoS prediction poisoning attack model using the following formula:

[0018]

[0019] In the formula: This represents the state space of the multi-step DDoS prediction poisoning attack model at the current time t; j∈[1,5] represents the proportion of the j-th type of alarm label to the total number of alarm labels in the local training dataset; This indicates the last alarm label in the alarm sequence;

[0020] Step 202: Define the action space of the multi-step DDoS prediction poisoning attack model as... Among them, action a keep This indicates that an injection is performed at the end of the alarm sequence. Same alarm label; action a transit This indicates that the system is transitioning to another alert label;

[0021] Step 203: Based on the state space and action space, formalize the reward function;

[0022] Step 204: Based on the state space, action space, and reward function, establish a multi-step DDoS prediction poisoning attack model based on reinforcement learning;

[0023] Step 3: Based on reinforcement learning, use the poisoning algorithm to optimize the multi-step DDoS prediction poisoning attack model;

[0024] The optimized poisoning attack model is used to poison the global prediction model for unlabeled data.

[0025] Step 4: Defend against poisoning attacks based on DS evidence theory.

[0026] Preferably, in step 203, the instantaneous reward RI for time t is defined according to the following formula. t :

[0027]

[0028] In the formula, and This represents the attack probability calculated by the local model at times t-1 and t after an action is taken;

[0029] The delay reward RD for time t is defined by the following formula. t :

[0030]

[0031] In the formula, and This represents the attack probability calculated by the global model at times t-1 and t after an action is taken;

[0032] The total reward at time t is defined by the following formula, which is the reward function:

[0033]

[0034] In the formula, , These are the weights for immediate rewards and delayed rewards, respectively.

[0035] Preferably, step 3 further includes the following steps:

[0036] Step 301: In the dual deep Q network, establish an online neural network and a target neural network, wherein the online neural network and the target neural network have the same structure;

[0037] Step 302: Initialize the parameters of the online neural network, the target neural network, and the dual deep Q network. These parameters include the weights of the online neural network and the target neural network, as well as the memory capacity of the replay memory in the dual deep Q network.

[0038] Step 303: Based on the state space, action space, and reward function, optimize the weights of the online neural network using mini-batch gradient descent and update the weights of the target neural network to obtain a trained neural network.

[0039] Preferably, step 303 further includes the following steps:

[0040] Step 3031: Randomly select actions from the action space to balance exploration and utilization;

[0041] Step 3032: Obtain the state at the next moment based on the action selected in step 3031, and calculate the reward for the poisoning performance based on the reward function;

[0042] Step 3033: Store the current state, action, reward, and the state of the next moment in the replay memory;

[0043] Step 3034: Optimize the loss function of the online neural network using mini-batch gradient descent to obtain the optimized weights of the online neural network;

[0044] Step 3035: Update the weights of the target neural network based on the optimized weights of the online neural network to obtain the trained neural network.

[0045] Preferably, step 4 includes the following steps:

[0046] Step 401: Within the framework of the DS evidence theory, define a quaternion. ,in, The identification framework, also known as the hypothesis space containing all propositions, is... EV represents evidence; BPA represents a set of probability assignment functions; BEL represents a set of trust functions, for the l-th participant, , and These represent the trust functions for the proposition that the object being detected is benign and the trust functions for the proposition that the object being detected is malicious, respectively.

[0047] Step 402: Define the evidence for clustering, upload frequency, and historical trust, and their probability assignment functions respectively;

[0048] Step 403: Calculate the participant's trust function based on the probability allocation function of the evidence;

[0049] Step 404: Based on the trust function, filter and aggregate the local models of the participants to obtain the final aggregated global model;

[0050] Step 405: Use the aggregated global model to defend against poisoning attacks.

[0051] Preferably, the probability allocation function BPA is defined as:

[0052]

[0053] in, and Let represent the probability allocation functions for the benign and malicious propositions of the l-th participant under the i-th piece of evidence, respectively, where 1 ≤ i ≤ 3.

[0054] Preferably, step 402 includes the following steps:

[0055] Step 4021: Cluster the local models of the participants, and define the size of the cluster to which the participant belongs as the clustering evidence, with the probability assignment function as follows:

[0056]

[0057]

[0058] In the formula, L is the number of participants; C l This represents the size of the cluster to which the l-th participant belongs;

[0059] Step 4022: Define the number of times a participant uploads a local model in one round of aggregation as the upload frequency evidence, and the probability allocation function is:

[0060]

[0061]

[0062] In the formula, f l This represents the number of times the l-th participant uploads the local model in a round of aggregation;

[0063] Step 4023: Define the number of times a participant is identified as a benign participant in the aggregation as historical trust evidence, and the probability allocation function is:

[0064]

[0065]

[0066] In the formula, D represents the total number of aggregations; d l This represents the number of times the l-th participant has been historically identified as a benign participant.

[0067] Preferably, in step 403, the participant's trust function for:

[0068]

[0069] In the formula:

[0070]

[0071] In the formula, the coefficients The degree of conflict between pieces of evidence is defined as follows: .

[0072] Preferably, step 404 includes the following steps:

[0073] Step 4041: Sort all participants' trust functions in descending order and calculate the mean and median;

[0074] Step 4042: Set trusted pointers and untrusted pointers, and compare the average and median. Assign the larger value to the trusted pointer and the smaller value to the untrusted pointer.

[0075] Step 4043: Exclude local models of participants whose trust function is not greater than that of the untrusted pointer, and aggregate local models of participants whose trust function is not less than that of the trust pointer.

[0076] Step 4044: Treat the participants corresponding to the trust functions located between the untrusted pointers and the trust pointers as suspicious participants to obtain the final aggregated global model.

[0077] The attack method disclosed in this invention can not only successfully poison global prediction models using unlabeled data in a short time, but also continuously accelerate the poisoning speed by leveraging accumulated experience; secondly, the defense strategy for unverified data based on DS evidence theory can effectively avoid abnormal aggregation. Attached Figure Description

[0078] Figure 1 This diagram illustrates a DDoS attack on a base station in an intelligent transportation scenario using existing technologies.

[0079] Figure 2 The diagram shows a multi-step DDoS prediction architecture based on federated learning.

[0080] Figure 3 The diagram shows a flowchart of a multi-step DDoS prediction poisoning attack method and its defense strategy based on reinforcement learning disclosed in this invention.

[0081] Figure 4 This is a schematic diagram showing the test results of testing the performance of poisoning attack in one embodiment of the present invention;

[0082] Figure 5 The diagram shows the experimental results of testing the convergence time of a poisoning attack in one embodiment of the present invention.

[0083] Figure 6 This is a schematic diagram showing the test results of testing the average accuracy of a defense strategy in one embodiment of the present invention;

[0084] Figure 7 The diagram shows the test results of DDoS attack prediction performance under the first metric in one embodiment of the present invention.

[0085] Figure 8 This is a schematic diagram showing the test results of DDoS attack prediction performance under a second metric in one embodiment of the present invention.

[0086] Figure 9 The diagram shows the experimental results of testing DDoS attack prediction performance under a third metric in one embodiment of the present invention.

[0087] In the diagram: 1—Sensor; 2—Camera; 3—Base station; 4—Normal connection; 5—DDoS attack; 6—Cloud server; 7—Local model; 8—Global model; 9—Poisoned model; S1~S5—Steps. Detailed Implementation

[0088] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.

[0089] Figure 1This diagram illustrates a DDoS attack on a base station in a smart transportation scenario. Roadside cameras, vehicle-mounted sensors, induction coils, and embedded road sensors are all connected to base station 3 for data processing or computation. Base station 3, as edge infrastructure, provides convenient services to mobile users, such as traffic broadcasts and driving warning support. However, an increasing number of smart sensors, such as cameras 2, can launch attacks against the base station, particularly DDoS attacks 5. When malicious sensors 1 send a large number of useless interference packets to base station 3, ultimately launching a DDoS attack 5, communication on base station 3 will be blocked, and many normally connected sensors 1 will be disabled. To date, much research has been conducted on DDoS attack detection, with most considering it a single-step attack. However, in reality, a DDoS attack 5 can be broken down into multiple steps and executed in stages according to a specific strategy, posing a significant security risk to the system. In one embodiment of the present invention, a DDoS attack is broken down into five steps: (1) "IP scanning": scanning the IP network from a remote site; (2) "Sadmind Ping": allowing the attacker to distinguish which hosts are vulnerable to intrusion; (3) "Breach and buffer overflow": the attacker attempts to breach the host under the sadmind service in the second step; (4) "Install Mstream software": installing the Trojan Mstream DDoS software on the host; (5) "Start DDoS". If a DDoS attack can be detected early, we can take preventative measures, which is what a defense system needs.

[0090] Figure 2A multi-step DDoS prediction architecture based on federated learning is described. This architecture can be divided into two parts: base stations 3 and a cloud server 6. In one embodiment of the invention, each base station 3 can be considered a participant, which can train its Hidden Markov Model-based (HMM) model based on a local dataset without exchanging raw privacy data with others. Furthermore, each base station 3 is equipped with a commonly used packet sniffer called "Snort," which can detect anomalous packets and generate relevant alerts. These alerts provide a common language to describe suspicious events and ultimately correspond to alert labels including "information," "remote user," "remote attacker," "denial of service," and "buffer overflow." Each base station uses these generated alerts to train its local model. We assume that different base stations 3 have training datasets of different sizes, and that each participant is unaware of the models of other participants in the federated learning framework. The base stations continuously collect sensor-base station communication traffic and train their local prediction models to monitor whether they are vulnerable to DDoS attacks within their communication range. Then, all participants located in different geographical areas update their local models 7 to the cloud server 6. Cloud server 6 aggregates these local models 7 into a global model 8, and further distributes the global model 8 to each base station 3 to achieve DDoS prediction on base station 3. After predicting a DDoS attack, each base station 3 subsequently updates its training dataset with accumulated data through a DDoS observation sequence. In one embodiment of the present invention, we assume that cloud server 6 does not have a validation dataset.

[0091] Furthermore, in one embodiment of the present invention, we establish a multi-step DDoS prediction model based on HMM. A discrete first-order HMM is defined as a quintuple, as shown in equation (1) below:

[0092] (1)

[0093] In equation (1), S describes a set of HMM states corresponding to the sub-steps of a DDoS attack. In one embodiment of the present invention, the steps of a DDoS attack include "IP scanning", "Sadmind Ping", "breaking in and performing a buffer overflow", "installing Mstream software", and "starting DDoS". Therefore:

[0094] (2)

[0095] In equation (2), s1, s2, s3, s4, and s5 represent “IP scan”, “Sadmind Ping”, “break in and perform buffer overflow”, “install Mstream software”, and “start DDoS”, respectively.

[0096] In equation (1), This is a set of observations based on alert labels. In one embodiment of the present invention, the alert labels include "Information," "Remote User," "Remote Attacker," "Denial of Service," and "Buffer Overflow." Therefore:

[0097] (3)

[0098] In equation (3), , , , , These represent "information", "remote user", "remote attacker", "denial of service", and "buffer overflow", respectively.

[0099] In equation (1), E is the transition probability matrix describing the state transitions in set S. In one embodiment of the present invention, we define a 5×5 transition probability matrix, namely:

[0100] (4)

[0101] In equation (4), a hidden process is determined by a random variable x t This represents the state corresponding to time t in a Markov chain; the symbol e ij Indicates from state s i to state s j The transition probability.

[0102] In equation (1), Q represents the observation probability matrix, which in one embodiment of the present invention is expressed as:

[0103] (5)

[0104] In equation (5), an observation process is determined by a random variable o t This indicates that the corresponding alert label is sent from Snort; each element q in Q corresponds to an alert label sent from Snort. ik Indicates that in state s i Time observation symbol The probability of.

[0105] In equation (1), Let represent the initial distribution vector for each HMM state. It represents the probability that an intrusion can begin in each state. Therefore:

[0106] (6)

[0107] In one embodiment of the present invention, Each element in the array is uniformly set to 0.2.

[0108] At the start of HMM training, the transition probability matrix E and observation probability matrix Q can be computed through supervised training using the observation sequence and hidden states. In other words, it calculates the frequency of each observation in each HMM state and obtains the HMM parameters λ using the maximum likelihood method.

[0109] Let O represent a series of observations of variable length, whose elements belong to the observation set. After obtaining the parameters λ of the HMM, we can use forward or backward algorithms to calculate the probability P(Q|λ) of the observed sequence. Furthermore, given λ and the observed sequence, we can use the Viterbi algorithm to obtain the most likely corresponding state. Further, the final attack probability PA is calculated based on the number of alerts triggered by Snort, reflecting the current DDoS attack situation. The probability PA is expressed as equation (7):

[0110] (7)

[0111] In equation (7):

[0112] (8)

[0113] (9)

[0114] In equations (8) and (9), n j To train the dataset, the state s is in the middle. j The number of alarms, μ j This is the number of attack alerts detected in each state. In one embodiment of the present invention, the number of alerts can be dynamically adjusted according to the current intrusion.

[0115] DDoS attacks are characterized by multiple steps. This means that a DDoS attacker needs to make a series of preparations to launch the final DDoS attack. In formula (7), the numerator... The sum of the most likely state probabilities for each received alert is represented by the denominator Num, where Num represents the total number of DDoS attack alerts. If the HMM model can accurately predict the states, the sum of state probabilities will be close to 1 for each alert. As the number of alerts increases, the DDoS attack approaches its final initiation stage. Therefore, the attack probability PA represents the extent or range from the current alert to the final DDoS initiation stage. A high attack probability PA means it is closer to the final stage of the DDoS attack, and the base station needs to prepare for defense.

[0116] like Figure 3 As shown, this invention provides a multi-step DDoS attack prediction and poisoning attack method based on reinforcement learning and its defense strategy, including the following steps:

[0117] S1: Set up the threat model and hypotheses, including the alert labels that make up the local training dataset;

[0118] S2: Based on the alert labels, establish a multi-step DDoS prediction poisoning attack model;

[0119] S3: Based on reinforcement learning, the poisoning algorithm is used to optimize the poisoning attack model;

[0120] S4: Use the optimized poisoning attack model to poison the global prediction model for unlabeled data;

[0121] S5: Based on the DS evidence theory, it defends against poisoning attacks.

[0122] In a federated learning-based multi-step DDoS prediction architecture, one or more poisoners select certain base stations (3) as normal participants to infiltrate and engage in federated learning. These poisoners not only prevent their local base stations from predicting DDoS attacks correctly but also aim to poison the global model (8), further affecting the predictions of other base stations. To prevent the model from suddenly deviating from the original model, the poisoners attempt to poison the local model (7) of base station (3) by injecting anomalous alert data into its local training dataset. The poisoners can inject different alerts, which will alter the transition probability matrix in the Hidden Markov Model (HMM). When these alerts are used for training, the prediction model may become inaccurate.

[0123] In one embodiment of the invention, the poisoner is considered an agent interacting with the environment and taking actions based on the current state and reward. The poisoner can obtain uploaded parameters to evaluate its role in the current state and further provide feedback on rewards to guide policy learning. After multiple iterations, the agent can make actions that maximize the cumulative reward based on the current state and learn the optimal policy. Therefore, step S2 further includes:

[0124] S21: Based on the alarm labels, define the state space of the poisoning attack model using the following formula:

[0125]

[0126] Among them, symbols j∈[1,5] represents the proportion of the j-th type of alarm label to the total number of alarm labels in the local training dataset; in one embodiment of the present invention, the current step t... The value is calculated as follows:

[0127] (11)

[0128] In addition, symbols This represents the last alarm label in the alarm sequence. Since the poisoning attack ultimately needs to affect the Hidden Markov Model (HMM), and the HMM's observation sequence is a time-dependent sequence, the poisoner needs to inject an intervention alarm at the end of the observation sequence based on the last alarm label in the current observation sequence. The last alarm label in the current observation sequence will be different under different states. Based on the last alarm label... The poisoner can choose two partners. Related actions. Therefore, we will Set as a component of the state vector in formula (10).

[0129] S22: Define the action space of the poison attack model as... In one embodiment of the invention, to accelerate the poisoning process, when an agent performs an action, 10 or 10 pairs of alarms are injected into the local alarm sequence of the training data. Action a keep This indicates that the alert is injected at the end of the alert sequence along with the last alert (sc). The same 10 alarm labels; action a transit This indicates that the system is transitioning to another alert label. (Except for the last alert label.) In addition, the agent randomly selects one of the remaining four alarm tags as a transitional alarm, and then, at the end of the alarm sequence, the agent alternately injects the alarm tag with the last alarm tag. And 10 pairs of alarm labels for transitional alarms. For example, the local alarm sequence of the training data is If the agent selects an action during the proxy process, the transitional alert will be randomly selected as... Then, the poisoned training data became The two types of behavior described above will ultimately change the transition probability matrix in the HMM.

[0130] S23: Formalizing the reward function based on the state and action spaces. To verify whether the current state and action affect model poisoning, we need to determine whether the alarm injection is effective. Since the poisoner possesses a local training dataset from the base station, this raw dataset can be used to verify the performance of the model poisoning. Considering that the global model is periodically aggregated and distributed (i.e., it takes time), we formalize the two reward functions using both the local and global models to accelerate convergence.

[0131] Before the global model is distributed, the poisoner can use its local model to determine whether a certain action is effective in the current state. We use the final attack probability introduced in formula (7) as the calculation criterion for evaluating poisoning performance. Therefore, the immediate reward RI t The following can be calculated:

[0132] (12)

[0133] In equation (12), and This represents the attack probability calculated by the local model at times t-1 and t after an action is taken.

[0134] When the server aggregates the local model, it distributes a global model to each participant. After the poisoner receives the global model, the impact of their actions over that period can be measured. Delayed Rewards (RD) t The definition is as follows:

[0135] (13)

[0136] In equation (13), and This represents the attack probability calculated by the global model at times t-1 and t after an action is taken. If the global model has not been deployed at the current time, we set... .

[0137] Finally, the total reward is defined according to the following formula (14):

[0138] (14)

[0139] In equation (14), , These are the weights of immediate rewards and delayed rewards, respectively. In one embodiment of the invention, we set... , When the global or local model is poisoned, the attack probability will decrease, and then the reward R will be increased. t This will take effect; the increase in both immediate and delayed rewards will optimize the subject's action choices.

[0140] S24: Based on the state space, action space, and reward function, establish a poisoning attack model for multi-step DDoS prediction using reinforcement learning.

[0141] Furthermore, step S3 includes:

[0142] S31: Construct two deep neural networks with identical structures and use them to approximate the action-value function. (The θ sign is used in the original text.) Represents an online neural network, with Symbols This represents the target neural network.

[0143] S32: Initialize the parameters of the online neural network, the target neural network, and the Double Deep Q-network (DDQN), including the weights of the online neural network and the target neural network, as well as the memory capacity M of the replay memory in the DDQN.

[0144] S33: Based on the state space, action space, and reward function, the weights of the online neural network are optimized using the mini-batch gradient descent method, and the weights of the target neural network are updated to obtain a trained neural network.

[0145] In one embodiment of the present invention, to uniformly describe the degree of poisoning, a threshold tar ∈ [0,1] needs to be set to represent the degree of degradation in model performance. For an agent, it uses its original training dataset from its local base station and the threshold tar as input to the algorithm. After initialization, the agent uses a policy-based... of The greedy algorithm randomly selects actions from the action space to balance exploration and exploitation. After executing an action, the agent obtains the state for the next time step. The reward for the poisoning behavior is calculated based on the reward function. Then, the current state, action, reward, and the state for the next time step are stored in the replay memory. In the next step, the loss function of the online neural network is optimized using mini-batch gradient descent to obtain the optimized online neural network. The weights θ; where the loss function L(θ) is expressed as:

[0146] (15)

[0147] (16)

[0148] In equations (15) and (16), M s Indicates the size of the training set; ;symbol It is a discount factor with a value between 0 and 1.

[0149] Finally, based on the optimized weights of the online neural network, the weights of the target neural network are updated to obtain the trained neural network.

[0150] Furthermore, poisoning based on reinforcement learning may affect the global model of multi-step DDoS prediction based on HMM. Since the cloud server lacks a verification dataset, it is difficult to determine which participant is malicious. We can only infer the suspected poisoner using the uploaded information and the knowledge possessed by the cloud server, and obtain a robust global model. The DS evidence theory proposed by Dempster and Shafer is an effective method for inferring uncertain concepts based on a comprehensive formula integrating different pieces of evidence. Therefore, we utilize this theory to design our defense strategy. In one embodiment of the invention, the number of poisoners is less than 1 / 3 of the total participants; under this condition, a safe defense strategy can be found. Here, the cloud server will execute the defense strategy and periodically aggregate the global model. In each round of aggregation, we first use the DS evidence theory to quantify the evidence. Specifically, step S5 includes:

[0151] S51: Within the framework of the DS evidence theory, define a quadruple. .in, The identification framework, also known as the hypothesis space containing all propositions, is... Here, "benign" indicates that the participant is not the poisoner, while "malicious" indicates that the participant is the poisoner; EV represents evidence; BPA describes a set of basic probability assignment functions, defined as:

[0152] (17)

[0153] In equation (17), and Let represent the probability assignment functions for the benign and malicious propositions of the l-th participant under the i-th (1≤i≤3) pieces of evidence, respectively.

[0154] BEL represents a set of trust functions, for the l-th participant, we have:

[0155] (18)

[0156] In equation (18), and These represent the trust functions for the proposition that the object being tested is benign and the trust functions for the proposition that the object being tested is malicious, respectively.

[0157] S52: Define the evidence for clustering, upload frequency, and historical trust, and their probability assignment functions.

[0158] In federated learning, each participant uploads its parameters or model λ to a cloud server. For normal participants, their uploaded parameters will be similar. Therefore, we first use clustering methods to group the uploaded local models. There are many different clustering algorithms, such as hierarchical clustering, density-based clustering, and K-means. Compared to other algorithms, K-means has the advantages of simple implementation, low time complexity O(N), and good clustering results, where N represents the number of samples in the dataset. Therefore, in one embodiment of this invention, we choose the K-means algorithm to cluster the participants' local models. After obtaining the clustering results, we can observe that each local model will be divided into a cluster. Since we assume that normal participants are in the majority, normal participants are easily grouped together, while abnormal participants may be scattered throughout the clustering results. Therefore, we define the size of the cluster to which a participant belongs as the clustering evidence. Then, we have the following probability assignment function:

[0159] (19)

[0160] (20)

[0161] In equation (19), L is the number of participants; C l This indicates the size of the cluster to which the l-th participant belongs.

[0162] The poisoner needs to upload the latest local model to assess the impact of the current state and actions on the final poisoned target, resulting in a higher upload frequency than other normal participants. Therefore, we define the number of times a participant uploads the local model in a round of aggregation as the upload frequency evidence, and its probability assignment function is:

[0163] (twenty one)

[0164] (twenty two)

[0165] Among them, f l This represents the number of times the l-th participant uploads a local model in a round of aggregation.

[0166] On the cloud server side, it records whether a participant is a poisoner in each aggregation round. Therefore, we define the number of times a participant is judged as a benign participant in the aggregation as historical trust evidence. And its probability allocation function is:

[0167] (twenty three)

[0168] (twenty four)

[0169] Where D represents the total number of aggregations; d l This represents the number of times the l-th participant has been historically identified as a benign participant.

[0170] S53: Based on the probability allocation function of the evidence, the participant's trust function is calculated. In this invention, benign propositions and malicious propositions are incompatible; therefore, the trust function is:

[0171] (25)

[0172] in:

[0173] (26)

[0174] Among them, coefficient The degree of conflict between pieces of evidence is defined as follows:

[0175] (27)

[0176] Then, the fusion with the third piece of evidence is the same process as the steps described above.

[0177] S54: Based on the trust function, the local models of the participants are filtered and aggregated to obtain the final aggregated global model.

[0178] In obtaining the trust function Next, we must decide whether the local models of the l-th participant can be aggregated into a robust global model. First, we define the trust functions of all participants. Sort the values ​​in descending order and calculate the mean and median. Next, we set up two pointers, one called the "trusted pointer" and the other the "untrusted pointer." By comparing the mean and median, the value with the larger value is assigned to the "trusted pointer," and the value with the smaller value is assigned to the "untrusted pointer." Then, the trust function is eliminated. The participants, distrustful pointers represent distrustful pointers, and aggregate trust functions satisfy... The local model of the participants, where trustfulpointer represents the trust pointer; where the trust function satisfies Participants in this scenario are considered suspicious. Therefore, we arrive at the final aggregated global model. :

[0179] (28)

[0180] S55: Uses a global aggregation model to defend against poisoning attacks.

[0181] The steps described above are for clarity only. In practice, they can be combined into one step or some steps can be split into multiple steps. As long as they contain the same logical relationship, they are all within the protection scope of this invention. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, without changing the core design of the algorithm and process, are also within the protection scope of this invention.

[0182] To evaluate the poisoning attack method and its defense strategy proposed in this invention, in one embodiment of this invention, a test area was selected, and 10 base stations were randomly set up in the test area. The detailed information of the base stations is shown in the table below:

[0183]

[0184] Within the test area, we used Python to generate 2000 random points to simulate smart sensors, such as cameras and embedded road sensors. Simultaneously, a simulator called SUMO was used to generate 1000 vehicles whose trajectories were limited to the road topology. Experiments were then conducted using the DARPA2000 dataset, which contains multi-step DDoS attacks. The dataset contains 420 alerts. Under a federated learning framework, we randomly selected some attack alerts and assigned them to the simulated sensors and vehicles through sampling. Each base station collects alerts from sensors and vehicles within its communication range, making the training data different for each base station. This embodiment was conducted on a 64-bit Windows 10 operating system server equipped with an Intel Core i9-10920X 3.50GHz CPU processor and 64GB of memory.

[0185] First, the performance of the poisoning attack was tested and evaluated. The parameters used in DDQN are shown in the table below:

[0186]

[0187] like Figure 4 As shown, Figure 4 The results show the experimental performance of the poisoning attack. The horizontal axis represents the number of uploads for the local model, and the vertical axis represents the attack probability of the aggregated global model. When there is only one poisoner, the number of uploads (iterations) is 185, which reduces the attack probability from 0.96 to 0.86. When there are 2, 3, 4, and 5 poisoners, the number of uploads is 76, 53, 38, and 32 respectively, showing a gradual decreasing trend.

[0188] like Figure 5As shown, this embodiment further tests the convergence time for the poisoner to successfully complete the poisoning attack. The horizontal axis represents the number of learning iterations; a point on the horizontal axis signifies a complete learning convergence process, i.e., the attack probability decreases by 10%. We can see that the poisoning attack initially requires a significant amount of time to reach the poisoning target; subsequently, due to accumulated learning experience, the convergence time becomes lower and more stable.

[0189] In this embodiment, regarding attack probability, the defense strategy proposed in this invention was compared with three other defense algorithms (truncated mean, Krum, and median) under different numbers of poisoners. The experimental results are shown in the table below:

[0190]

[0191] The data in the table shows that the defense strategy proposed in this invention performs best regardless of the number of poisoners. When there is only one poisoner, the truncated mean, Krum, and median defense methods all work effectively; however, the truncated mean method outperforms the Krum and median methods because it eliminates the worst-case model and then takes the average. When there are 2 or 3 poisoners, the Krum and median methods are both good, with values ​​of 0.9576 and 0.9423 respectively, but the performance of the truncated mean method deteriorates because it takes the average and aggregates some toxic parameters. When there are more than three poisoners, the attack probabilities of the Krum, median, and truncated mean methods are difficult to predict for DDoS attacks. However, when the number of poisoners is less than 1 / 2, the defense method proposed in this invention still yields good results because the aggregation criteria consider many different pieces of evidence.

[0192] In another embodiment of the invention, we also conducted an ablation study, namely, allowing the defense strategy of the present invention to retain only any two pieces of evidence while lacking one piece. Specific experimental data shows that the attack probability of the defense strategy proposed in this invention is also higher than the other three groups. When the number of poisoners is less than 3, the attack probability is close to the result of retaining all three pieces of evidence; when the number of poisoners is greater than 3, selecting any two pieces of evidence is worse than selecting all three pieces of evidence. The ablation study verifies that selecting three pieces of evidence yields better results because it considers more factors to make the defense more robust.

[0193] like Figure 6As shown, we further tested the average accuracy of this defense strategy in excluding poisoners in federated learning. Note that when the number of poisoners is less than 3 / 10 of the participants, the accuracy in excluding poisoners is close to 100%. The discrepancy from 100% is because at the beginning of poisoning, the local model affected by the poisoning has a small impact on the global model and therefore cannot be directly eliminated. As the poisoning process progresses, the defense strategy can successfully detect and exclude it. However, when the number of poisoners exceeds 3 / 10, the accuracy in excluding poisoners decreases.

[0194] Furthermore, we experimented with the DDoS attack prediction performance of the global model. We used a Hidden Markov Model (HMM) to predict the intrusion probability of a DDoS attack as the number of alerts increases. Three performance metrics were set to evaluate the intrusion prediction performance. The first metric is prediction accuracy (PAR), which measures the proportion of correctly predicted states accumulated by the model, defined as:

[0195] (29)

[0196] The second metric is the Average State Prediction Probability (ASPP), which evaluates the average probability that the model predicts the current state based on the current alarm.

[0197] (30)

[0198] The third metric is the final attack probability, which is used to indicate the progress of the current DDoS attack based on the current alert level.

[0199] Figure 7 This diagram illustrates the experimental results under the first metric. The results show that the prediction accuracy remains consistently around 1, with an average of 0.995. However, when a poisoner is present among the participants, the global model's PAR decreases to approximately 0.9. Therefore, when the server aggregates poisoned models, it impacts the global model.

[0200] Figure 8 This diagram illustrates the experimental results under the second metric. Regardless of whether there is a poisoner, the initial ASPP value is only 0.664. This is because when the number of alarms is small, the observation sequence is too short to accurately calculate the probability of the current state. As the number of incoming alarms increases, the ASPP without a poisoner rapidly increases to 0.965; while the ASPP with one poisoner is lower, eventually reaching only 0.855. In summary, our global model can accurately predict the current state with a high probability.

[0201] Figure 9This diagram illustrates the experimental results under the third metric. The results show that the attack probability increases with the number of incoming alerts. The attack probability represents the distance a DDoS attack is from reaching its final target. In this experiment, the final attack probability for the global model without a poisoner reached 0.963. When the final attack probability reached 0.95, the number of incoming alerts was 315, far lower than the total number of alerts (420). This means they had sufficient time for proactive defense. However, the attack probability of the global model with one poisoner increased slowly; when the number of alerts reached 315, the attack probability was only 0.805, far less than 0.95.

[0202] As described above, the poisoning attack method provided by this invention can not only successfully poison a global prediction model using unlabeled data in a short time, but also continuously accelerate the poisoning speed by leveraging accumulated experience. Secondly, the defense strategy based on DS evidence theory and without verification data can effectively avoid abnormal clustering. Therefore, the multi-step DDoS prediction poisoning attack method and its defense strategy based on reinforcement learning provided by this invention effectively overcomes the various shortcomings of existing technologies and has high industrial application value.

[0203] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A method for multi-step DDoS prediction poisoning attack and its defense based on reinforcement learning, characterized in that, The method comprises the following steps: Step 1, setting a threat model and assumptions, wherein the alarm labels of the local training data set are included, and the alarm labels comprise the following contents: Each base station is considered as a participant, who can train their hidden Markov model based on local dataset without exchanging original privacy data with others; in addition, each base station is equipped with a commonly used packet sniffer, which can detect abnormal packets and generate related alerts, these alerts provide a common language to describe suspicious events, and finally correspond to some alert tags, including "information", "remote user", "remote attacker", "denial of service" and "buffer overflow"; each base station uses these generated alerts to train its local model; it is assumed that different base stations have different sizes of training data sets, and each participant cannot know the models of other participants in the federated learning framework; base stations continuously collect sensor-base station communication traffic and train their local prediction models to monitor whether they will be attacked by DDoS within their communication range; then, all participants located in different geographical areas update their local models to the cloud server, assuming that the cloud server does not have a validation dataset; the cloud server aggregates these local models into a global model, and further distributes the global model to each base station to achieve DDoS prediction on the base station; after predicting DDoS attack, each base station subsequently updates the cumulative data to its training dataset through the DDoS observation sequence; a discrete first-order hidden Markov model is defined as a five-tuple, as shown in the following formula (1): (1), in formula (1), S describes a set of hidden Markov model states corresponding to sub-steps of DDoS attack; is a set of observation results based on alert tags; E is a transition probability matrix describing state transitions in the set S; Q represents an observation probability matrix; represents an initial distribution vector of each hidden Markov model state, which represents the probability that an intrusion can start in each state; step 2, according to the alert tag, a multi-step DDoS prediction poisoning attack model is established, which specifically includes the following steps: step 201, according to the alert tag, the state space of the multi-step DDoS prediction poisoning attack model is defined using the following formula: , in which: represents the state space of the multi-step DDoS prediction poisoning attack model at the current time t; , j∈[1,5] represents the proportion of the jth alert tag in the total number of alert tags in the local training data set; represents the last alert tag in the alert sequence; step 202, the action space of the multi-step DDoS prediction poisoning attack model is defined as , in which, action a keep represents that the same alert tag as is injected at the end of the alert sequence; action a transit representing the transition to another alarm label; step 203, formalizing the reward function based on the state space and the action space; step 204, establishing a multi-step DDoS prediction poisoning attack model based on reinforcement learning according to the state space, the action space and the reward function; step 3, optimizing the multi-step DDoS prediction poisoning attack model using a poisoning algorithm based on reinforcement learning; using the optimized poisoning attack model to perform poisoning attack on the global prediction model of unlabeled data; step 4, defending against the poisoning attack based on D-S evidence theory.

2. The method of claim 1, wherein the method is based on a multi-step DDoS prediction poisoning attack and its defense method, characterized by, In said step 203, the immediate reward R1 at time t is defined according to the following formula t : , where and denote the probabilities of attack computed by the local model at times t-1 and t after taking the action. The delayed reward RD for time t is defined according to the following equation t : , where and denote the attack probabilities computed by the global model at times t-1 and t after taking the action The total reward at time t, i.e. the reward function, is defined according to the following equation: , where , are the weights of the immediate and delayed rewards, respectively.

3. The method of claim 1, wherein the method is based on a multi-step DDoS prediction poisoning attack and its defense method, characterized by, The step 3 further comprises the following steps: step 301, establishing an online neural network and a target neural network in the double deep Q network, wherein the online neural network and the target neural network comprise the same structure; step 302, initializing parameters of the online neural network, the target neural network and the double deep Q network, wherein the parameters comprise weights of the online neural network and the target neural network, and memory capacity of a replay memory in the double deep Q network; and step 303, optimizing the weights of the online neural network based on a state space, an action space and a reward function, and updating the weights of the target neural network to obtain a trained neural network.

4. The method of claim 3, wherein the method is based on a multi-step DDoS prediction poisoning attack and its defense method, characterized by, The step 303 further comprises the following steps: step 3031, randomly selecting an action from the action space to balance exploration and utilization; step 3032, obtaining a state at a next moment according to the action selected in the step 3031, and calculating a reward about poisoning performance according to the reward function; step 3033, storing the state at the current moment, the action, the reward and the state at the next moment in the replay memory; step 3034, optimizing a loss function of the online neural network by using a small batch gradient descent method to obtain optimized weights of the online neural network; and step 3035, updating the weights of the target neural network according to the optimized weights of the online neural network to obtain the trained neural network.

5. The method of claim 1, wherein the method is based on a multi-step DDoS prediction poisoning attack and its defense method, characterized by, The step 4 comprises the following steps: step 401, defining a four-tuple under the framework of D-S evidence theory , wherein, represents an identification framework, also known as a hypothesis space containing all propositions, that is, ; EV represents evidence; BPA represents a set of probability distribution functions; BEL represents a set of belief functions, for the lth participant, , and respectively represent the belief function of the proposition that the detected object is benign and the belief function of the proposition that the detected object is malicious; step 402, defining the evidence of clustering, uploading frequency and historical belief and the probability distribution function thereof respectively; step 403, calculating the belief function of the participant according to the probability distribution function of the evidence; step 404, screening and aggregating the local model of the participant according to the belief function to obtain a final aggregated global model; and step 405, using the aggregated global model to defend against poisoning attacks.

6. The method of claim 5, wherein the method is based on a multi-step DDoS prediction poisoning attack and its defense method, characterized by, The probability assignment function BPA is defined as: wherein, and BPA1(i) and BPA2(i) represent the probability assignment functions of the benign proposition and the malicious proposition of the ith evidence of the 1th participant, respectively, 1≤i≤3.

7. The method of claim 6, wherein the method is based on a multi-step DDoS prediction poisoning attack and its defense method, characterized by, The step 402 comprises the following steps: step 4021, clustering the local models of the participants, and defining the size of the cluster where the participant is located as a clustering evidence, and the probability assignment function is: , , wherein L is the number of participants; C l represents the size of the cluster where the lth participant is located; step 4022, defining the number of times that the local model uploaded by the participant in one round of aggregation as an uploading frequency evidence, and the probability assignment function is: , , wherein f l represents the number of times that the local model uploaded by the lth participant in one round of aggregation; step 4023, defining the number of times that the participant is determined as a benign participant in aggregation as a historical trust evidence, and the probability assignment function is: , , wherein D represents the total number of aggregations; d l represents the number of times that the lth participant is historically determined as a benign participant.

8. The method of claim 6, wherein the method is based on a multi-step DDoS prediction poisoning attack and its defense method, characterized by, In step 403, the trust function of the participant is: wherein: where the coefficients denote the degree of conflict between the evidence, defined as: .

9. The method of claim 5, wherein the method is based on a multi-step DDoS prediction poisoning attack and its defense method, characterized by, The step 404 comprises the following steps: step 4041, arranging trust functions of all participants in descending order, and calculating an average value and a median value; step 4042, setting a trust pointer and a distrust pointer, and comparing the average value and the median value, and assigning a value greater to the trust pointer and a value smaller to the distrust pointer; step 4043, excluding a local model of a participant whose trust function is not greater than the distrust pointer, and aggregating local models of participants whose trust functions are not smaller than the trust pointer; and step 4044, regarding participants whose trust functions are between the distrust pointer and the trust pointer as suspicious participants to obtain a final aggregated global model.

Images