A physical layer deception detection method based on deep reinforcement learning

Through deep reinforcement learning technology, using physical layer channel state information and Bayesian risk function, a state-behavior-benefit triple was established, and a dynamic selection method for detection thresholds was designed, which solved the problem of insufficient detection performance of spoofing attacks in wireless environments, and realized Fast and accurate detection and adaptability in dynamic environments.

CN114845304BActive Publication Date: 2024-11-12SOUTHEAST UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210454202.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-04-27
Publication Date
2024-11-12
Estimated Expiration
2042-04-27

AI Technical Summary

Technical Problem

The existing physical layer spoof detection methods are difficult to accurately select continuous detection thresholds in dynamic and complex wireless environments, which affects the detection performance of spoof attacks. In addition, traditional methods require prior channel knowledge and cannot adapt to diverse communication scenarios.

Method used

Using a deep reinforcement learning method, the physical layer channel state information is extracted by the receiver, and the state-behavior-benefit triple is established. The detection threshold dynamic selection method is designed using the deep deterministic strategy gradient framework, combining Bayesian risk function and depth Neural networks enable fast and accurate detection of physical layer spoofing attacks.

Benefits of technology

In complex wireless communication scenarios, fast and accurate detection of physical layer spoofing attacks is realized, and the appropriate detection threshold can be dynamically selected, which is suitable for dynamic environments, improves detection performance and stability, and makes up for the shortcomings of traditional methods.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114845304B_ABST
    Figure CN114845304B_ABST
Patent Text Reader

Abstract

The present invention provides a physical layer deception detection method based on deep reinforcement learning, which mainly solves the problem that the channel model or parameters are difficult to obtain and the fixed detection threshold is difficult to accurately select for the dynamic unknown wireless environment in the existing physical layer deception detection method. The implementation steps are: 1) Establishing a deception attack scenario, the receiver extracts the physical layer channel information between the sender and the receiver to characterize the physical layer fingerprint characteristics; 2) Establishing a binary hypothesis testing model; 3) Constructing a state value with the dynamic physical layer fingerprint characteristics, constructing a behavior value with the threshold value selection, and establishing a state-behavior-benefit triple with the Bayesian risk function as the instantaneous benefit function; 4) Based on the deep deterministic policy gradient framework, a detection threshold dynamic selection method is designed to detect physical layer deception attacks. The present invention can realize the dynamic and continuous selection of the detection threshold and is adaptive to the dynamic unknown environment, and effectively detects the physical layer deception attack.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a detection method, and in particular to a physical layer deception detection method based on deep reinforcement learning, which belongs to the field of wireless communication physical layer security. Background Art

[0002] With the large-scale deployment of 5G commercial use, the exploration and research of 6G is gradually underway. In the white paper "6G Network Architecture Vision and Key Technology Outlook" released on September 16, 2021, the IMT-2030 (6G) Network Technology Group proposed the establishment of an "active immune, elastic and autonomous" physical layer security endogenous protection system. The deep integration of physical layer security and deception detection is of key significance for solving the problem of wireless networks being vulnerable to deception attacks and deepening the concept of endogenous security.

[0003] The existing physical layer deception detection design methods mainly include traditional technologies based on fixed detection thresholds and new technologies based on artificial intelligence. The fixed detection threshold method can achieve the best detection performance in theory, but it requires prior knowledge of the known wireless channel environment. However, as wireless communications develop towards full coverage and giant connections, the wireless environment becomes dynamic, complex and unknown, and channel models or parameters are difficult to obtain, making it difficult to accurately select a fixed detection threshold. Artificial intelligence-based physical layer deception detection uses the design of intelligent networks to greatly improve network efficiency and is suitable for a variety of communication scenarios. At present, physical layer deception detection based on deep reinforcement learning includes physical layer deception detection based on deep Q networks. DQN, as an important value function algorithm, learns control strategies from high-dimensional state inputs, uses deep convolutional neural networks to approximate value functions, and combines experience replay and target networks to greatly improve the estimation accuracy and stability of value functions. However, most of the current research on deception attack detection is discrete threshold analysis, and few literature studies on continuous control of detection thresholds. The determination of detection thresholds is particularly important for the performance of deception attack detection. How to learn continuous thresholds for dynamic environments still needs to be further studied. Summary of the invention

[0004] The present invention aims at the problems existing in the prior art and provides a physical layer deception detection method based on deep reinforcement learning. This technical solution can quickly and accurately realize physical layer deception detection through physical layer channel state information.

[0005] In order to achieve the above object, the technical solution of the present invention is as follows: a physical layer deception detection method based on deep reinforcement learning, comprising the following steps:

[0006] Step 1: Establish a deception attack scenario, where the receiver extracts the physical layer channel state information between the sender and the receiver to represent the physical layer fingerprint features;

[0007] Step 2: Establish a binary hypothesis testing model;

[0008] Step 3: construct the state value with the dynamic physical layer fingerprint feature, construct the behavior value with the threshold value, and use the Bayesian risk function as the instantaneous benefit function to establish the state-behavior-benefit triple;

[0009] Step 4: Based on the deep deterministic policy gradient framework, a detection threshold dynamic selection method is designed to detect physical layer spoofing attacks.

[0010] Furthermore, as a preferred technical solution of the present invention, the deception attack scenario established in step 1 is that Alice and Bob are legitimate users, Alice is the transmitter, Bob is the receiver, and Eve is the deception user; the distance between the deception attacker and the legitimate transmitter and the legitimate receiver is greater than half a wavelength, that is, the illegal channel is unrelated to the legitimate channel; the deception attack is an active attack, that is, the deception attacker and the legitimate transmitter do not transmit signals at the same time, and the attack probability that Eve pretends to be Alice and sends a deception signal to Bob is y∈[0,Y MAX ), where the maximum attack probability Y MAX <1; the physical layer channel information between the sender and the receiver extracted by the receiver is the channel frequency response information of the signal between the sender and the receiver. This scheme has the following advantages: due to the uniqueness, time-varying and rapid changes with distance of the wireless transmission channel, even if the deceptive attacker uses network technology to modify its MAC address to impersonate the legitimate transmitter, it cannot modify or forge its physical layer channel state information. Therefore, this scheme detects deceptive attackers based on CFR information, has the characteristics of low implementation complexity and strong security performance, and is suitable for physical layer deception attack detection in wireless communications.

[0011] Further, as a preferred technical solution of the present invention, the binary hypothesis testing model in step 2 is expressed as:

[0012] D0:H t =H A ;

[0013] D1:H t ≠H A ;

[0014] Among them, D0 represents the state where the transmitting user is a legitimate transmitter, D1 represents the state where the transmitting user is an illegal attacker, and H A is the CFR information between Bob and Alice that has been stored, H t The CFR information received by Bob with the unknown transmitter.

[0015] Further, as a preferred technical solution of the present invention, in step 2, the test statistic L is compared with the detection threshold k to perform hypothesis testing, which can be expressed as:

[0016] L(H t ,H A )=||H t -H A || 2 ;

[0017]

[0018] Among them, ||·|| 2 is the Euclidean distance operator, and L is the Euclidean distance between the current CFR information and the historical CFR information. This scheme has the following advantages: due to the typical wireless environment with rich physical layer multipath, the CFR information is location-specific. In a sufficiently short period of time, the CFR information of the same sender and receiver Alice and Bob is highly similar, while the channel responses between the deceptive users Eve and Bob and Alice and Bob are very different; therefore, using the Euclidean distance as a benchmark to compare the difference between the current CFR information and the historical CFR information is objective and universal, and also lays the foundation for selecting the appropriate detection threshold through the deep reinforcement learning algorithm.

[0019] Further, as a preferred technical solution of the present invention, in step 3, the state value is constructed by the dynamic physical layer fingerprint feature, and the behavior value is constructed by the threshold value, which can be expressed as:

[0020] s t =[H t ]∈S;

[0021] k t ∈A;

[0022] Among them, s t represents the channel frequency response value obtained by Bob in time slot t, k t represents the detection threshold selected by Bob at time slot t, S is the state space, and A is the behavior space.

[0023] Furthermore, as a preferred technical solution of the present invention, the Bayesian risk function in step 3 can be expressed as:

[0024] r=(G1·(1-p f )-C1·p f )·(1-y)+(G0·(1-p m )-C0·p m )·y;

[0025] Among them, r is the instantaneous benefit function of Bob receiving the current data packet, G1 is the benefit of Bob receiving legally sent user data packets, G0 is the benefit of Bob rejecting illegally sent user data packets, C0 is the cost of Bob receiving illegally sent user data packets, C1 is the cost of Bob rejecting legally sent user data packets, and y is the attack probability. This scheme has the following advantages: due to the complexity of the communication scenario and the unknown physical layer channel environment model of the sender and receiver, it is particularly important to identify the source of the received data packet based on the known insufficient information; the receiver can combine the Markov decision process to establish a state-behavior-benefit triple, and select the best detection threshold by maximizing the expected cumulative benefit to detect fake attackers, thereby ensuring the communication security of the wireless network.

[0026] Furthermore, as a preferred technical solution of the present invention, in the detection threshold dynamic selection algorithm in step 4, the real critic network updates parameters by minimizing the loss function during training: the real actor network updates parameters by back propagation of expected reward gradients, and both networks use the Adam optimizer.

[0027] Further, as a preferred technical solution of the present invention, the network update method in step 4 specifically includes:

[0028] For the real critic network, the loss function used is:

[0029]

[0030]

[0031] For real actor networks, the expected return used is:

[0032]

[0033] Among them, θ q is the actual critic network parameter, θ μ is the actual actor network parameter, θ μ' is the target actor network parameter, θ q' is the target critic network parameter, λ is the discount factor, j is the number of samples, and j=1,...,N B , N B is the batch training size, r j is the instantaneous benefit value of the current data sample, s j+1The state of the next data sample. This scheme has the following advantages: it fully combines the decision-making advantages of reinforcement learning and the perception advantages of deep learning. It can solve complex control decision-making tasks by automatically extracting abstract features from data through deep neural networks; the deep deterministic policy gradient framework adds a real actor network and a target actor network on the basis of DQN to output continuous action values, while improving the stability and convergence of the actor-critic network.

[0034] Compared with the prior art, the present invention has the following advantages: in the face of complex actual communication scenarios and the physical layer channel environment model of unknown senders and receivers, the present invention proposes a method for realizing physical layer deception attack detection by utilizing the physical layer channel state information of the communication between the senders and receivers. The method combines the decision-making advantages of reinforcement learning and the perception advantages of deep learning, and establishes a state-behavior-benefit triple based on the deep deterministic policy gradient framework, so that the receiver can select the optimal detection threshold based on the difference of CFR information between different senders and receivers in continuous time by maximizing the expected cumulative benefit, so as to quickly and accurately detect physical layer deception attacks; at the same time, the method is adaptive to dynamic unknown environments, and can realize dynamic and continuous selection of detection thresholds, which makes up for the shortcomings of traditional fixed detection threshold methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a schematic diagram of a deception attack scenario of the present invention.

[0036] Figure 2 This is the framework of the method for dynamic selection of physical layer spoofing attack detection threshold based on DDPG in the present invention. DETAILED DESCRIPTION

[0037] In order to deepen the understanding of the present invention, the present embodiment is described in detail below with reference to the accompanying drawings.

[0038] Example 1: See Figure 1 , Figure 2 , a physical layer deception detection method based on deep reinforcement learning, comprising the following steps:

[0039] Step 1: Establish a deception attack scenario, where the receiver extracts the physical layer channel state information between the sender and the receiver to represent the physical layer fingerprint features;

[0040] Step 2: Establish a binary hypothesis testing model;

[0041] Step 3: construct the state value with the dynamic physical layer fingerprint feature, construct the behavior value with the threshold value, use the Bayesian risk function as the instantaneous benefit function, and establish the state-behavior-benefit triple;

[0042] Step 4: Based on the deep deterministic policy gradient framework, a detection threshold dynamic selection method is designed to detect physical layer spoofing attacks.

[0043] The deception attack scenario established in step 1 is that Alice and Bob are legitimate users, Alice is the transmitter, Bob is the receiver, and Eve is the deception user; the distance between the deception attacker and the legitimate transmitter and the legitimate receiver is greater than half a wavelength, that is, the illegal channel is unrelated to the legitimate channel; the deception attack is an active attack, that is, the deception attacker and the legitimate transmitter do not transmit signals at the same time, and the attack probability that Eve pretends to be Alice and sends a deception signal to Bob is y∈[0,Y MAX ), where the maximum attack probability Y MAX <1; the physical layer channel information between the sender and the receiver extracted by the receiver is the channel frequency response information of the signal between the sender and the receiver.

[0044] The binary hypothesis test model in step 2 is expressed as:

[0045] D0:H t =H A ;

[0046] D1:H t ≠H A ;

[0047] Among them, D0 represents the state where the transmitting user is a legitimate transmitter, D1 represents the state where the transmitting user is an illegal attacker, and H A is the CFR information between Bob and Alice that has been stored, H t The CFR information received by Bob with the unknown transmitter.

[0048] In step 2, the test statistic L is compared with the detection threshold k to perform hypothesis testing, which can be expressed as:

[0049] L(H t ,H A )=||H t -H A || 2 ;

[0050]

[0051] Among them, ||·|| 2 is the Euclidean distance operator, and L is the Euclidean distance between the current CFR information and the historical CFR information.

[0052] In step 3, the state value is constructed by the dynamic physical layer fingerprint feature, and the behavior value is constructed by the threshold value, which can be expressed as:

[0053] st =[H t ]∈S;

[0054] k t ∈A;

[0055] Among them, s t represents the channel frequency response value obtained by Bob in time slot t, k t represents the detection threshold selected by Bob at time slot t, S is the state space, and A is the behavior space.

[0056] In step 3, the Bayesian risk function is used as the instantaneous benefit function, which can be expressed as:

[0057] r=(G1·(1-p f )-C1·p f )·(1-y)+(G0·(1-p m )-C0·p m )·y;

[0058] Among them, r is the instantaneous benefit function of Bob receiving the current data packet, G1 is the benefit of Bob receiving legally sent user data packets, G0 is the benefit of Bob refusing illegally sent user data packets, C0 is the cost of Bob receiving illegally sent user data packets, C1 is the cost of Bob refusing legally sent user data packets, and y is the attack probability.

[0059] In the detection threshold dynamic selection algorithm in step 4, the real critic network updates parameters by minimizing the loss function during training; the real actor network updates parameters by back propagation of expected reward gradients, and both networks use the Adam optimizer.

[0060] The network updating method in step 4 specifically includes:

[0061] For the real critic network, the loss function used is:

[0062]

[0063]

[0064] For real actor networks, the expected return used is:

[0065]

[0066] Among them, θ q is the actual critic network parameter, θ μ is the actual actor network parameter, θ μ' is the target actor network parameter, θ q'is the target critic network parameter, λ is the discount factor, j is the number of samples, and j=1,...,N B , N B is the batch training size, r j is the instantaneous benefit value of the current data sample, s j+1 is the state of the next data sample.

[0067] Example 2: Figure 1 FIG. 1 is a schematic diagram of a deception attack scenario. The present invention proposes a physical layer deception detection method based on deep reinforcement learning. The method specifically includes the following steps:

[0068] Step 1: Establish a deception attack scenario. Alice and Bob are legitimate users. Alice is the transmitter, Bob is the receiver, and Eve is a deceptive user. Assume that the distance between the deceptive attacker and the legitimate transmitter and the legitimate receiver is greater than half a wavelength, that is, the illegal channel is unrelated to the legitimate channel. The deceptive attack is an active attack, that is, the deceptive attacker and the legitimate transmitter do not transmit signals at the same time. The attack probability that Eve pretends to be Alice and sends a deceptive signal to Bob is y∈[0,Y MAX ), where the maximum attack probability Y MAX <1. The physical layer channel information between the sender and the receiver extracted by the receiver is the channel frequency response information of the signal between the sender and the receiver.

[0069] Among them, Alice configures N A antennas, Bob has N B Antennas, Eve configuration N E antennas; the communication environment has N s scatterers, communication center frequency f0, channel bandwidth W, number of subcarriers N f .

[0070] Step 2: Establish a binary hypothesis test model. D0 represents the state where the transmitting user is a legitimate transmitter, and D1 represents the state where the transmitting user is an illegal attacker. It can be expressed as:

[0071] D0:H t =H A ;

[0072] D1:H t ≠H A ;

[0073] Among them, H A is the CFR information between Bob and Alice that has been stored, H t The CFR information received by Bob with the unknown transmitter.

[0074] Then, the Euclidean distance between the current CFR information and the historical CFR information is used as the test statistic L, i.e., L(H t ,H A )=||H t -H A || 2 ; where ||·|| 2 is the Euclidean distance operator. Then, L is compared with the detection threshold k for hypothesis testing. If L < k, the legitimate user Bob accepts the state D0, considers the transmitting user to be the legitimate user Alice, receives the data packet, and updates the historical CFR information; otherwise, accepts the state D1, considers the transmitting user to be the spoofing user Eve, discards the data packet. The above process can be expressed by the formula as follows:

[0075] Step 3: Construct the state value with dynamic physical layer fingerprint features. The channel frequency response value obtained by Bob at time slot t is s t , i.e., s t =[H t . Therefore, the state space S can be expressed as S={s1, s2,..., s t}; then, input s t into the realistic actor network, and the threshold selection for the current time slot t is k t . Therefore, the action space A can be expressed as A={k1, k2,..., k t}.

[0076] Based on this, taking the Bayesian risk function as the instantaneous benefit function, the instantaneous benefit function r for Bob to receive the current data packet can calculate the corresponding Bayesian risk values for spoofing detection according to whether the data packet is from a legitimate or illegal sending user, respectively.

[0077] When the data packet is from a legitimate sending user, the return of the receiver is:

[0078] r1=(G1·(1 - p f ) - C1·p f )·(1 - y);

[0079] where, G1 is the benefit for Bob to receive the data packet from the legitimate sending user, C1 is the cost for Bob to reject the data packet from the legitimate sending user, y is the attack probability, and p f is the false alarm probability, i.e., the probability that the data packet from the legitimate sender is misidentified as the data packet from the illegal attacker.

[0080] When the data packet is from an illegal sending user, the return of the receiver is:

[0081] r2=(G0·(1 - p m ) - C0·p m)·y;

[0082] Among them, G0 is the benefit of Bob refusing to send illegal user data packets, C0 is the cost of Bob receiving illegal user data packets, p m is the missed detection probability, that is, the probability that the data packet from the illegal attacker is mistaken for the data packet from the legitimate sender.

[0083] Therefore, the receiver's benefit function can be expressed as the sum of the receiver's rewards for packets from legitimate and illegitimate users, that is:

[0084] r=r1+r2=(G1·(1-p f )-C1·p f )·(1-y)+(G0·(1-p m )-C0·p m )·y;

[0085] Step 4: Figure 2 As shown in Figure 1, based on the deep deterministic policy gradient framework, a detection threshold dynamic selection method is designed to detect physical layer spoofing attacks. Specifically, for the unknown transmitter at the current time t, the receiver obtains the current CFR information based on the received data packet and converts it into state s t , input the real actor network and select the detection threshold k in combination with the behavioral noise I t Then, a hypothesis test is established to detect the data packet and calculate the instantaneous benefit value r of the current state. t , store sample data {s j ,a j ,r j ,s j+1} in the experience replay memory and sample a training batch of size N from it B A batch of sample data {s j ,a j ,r j ,s j+1} as the training data set. During training, the real critic network updates its parameters by minimizing the loss function, and the real actor network updates its parameters by back-propagating the expected return gradient. Both networks use the Adam optimizer.

[0086] For the real critic network, the loss function used is:

[0087]

[0088]

[0089] For real actor networks, the expected return used is:

[0090]

[0091] Among them, θ q is the actual critic network parameter, θ μ is the actual actor network parameter, θ μ' is the target actor network parameter, θ q' is the target critic network parameter, λ is the discount factor, j is the number of samples, and j=1,...,N B , N B is the batch training size, r j is the instantaneous benefit value of the current data sample, s j+1 After the state of the next data sample, the target actor network parameters and the target critic network parameters are soft updated at a certain interval T, that is, the convex combination of the current network parameters and the target network parameters is used for update. The soft update process can be expressed as:

[0092] θ q' =τθ q +(1-τ)θ q' ;

[0093] θ μ' =τθ μ +(1-τ)θ μ' ;

[0094] Among them, τ is the soft update coefficient, θ q' and θ μ' They are the target critic network parameters and the target actor network parameters respectively.

[0095] Based on the above method, the present invention provides a specific embodiment.

[0096] like Figure 1 As shown in the figure, a deception attack scenario is established. The position of the legitimate transmitter Alice is [0 0 2], the position of the illegal attacker Eve is [200 30 2], and the position of the legitimate receiver Bob is [250 7 3]. The units of the above three-dimensional coordinates are all meters. Alice, Bob, and Eve are all equipped with 64 antennas, the communication center frequency is 5GHz, the channel bandwidth is 10MHz, the number of subcarriers is 512, the number of scatterers is 5, and the distribution of the scattering cluster wireless channel is unknown. The deception attack is an active attack, that is, the deception attacker and the legitimate transmitter do not transmit signals at the same time. The attack probability of Eve pretending to be Alice to send a deception signal to Bob is 0.4. The physical layer channel information between the sender and the receiver extracted by the receiver is the CFR information of the signal between the sender and the receiver simulated using the QuaDRiGa platform.

[0097] After that, a binary hypothesis testing model is established. Let D0 represent the state where the transmitting user is a legitimate transmitter, and D1 represent the state where the transmitting user is an illegal attacker. is the CFR information of the m-th sample between Bob and Alice that Bob has stored. is the CFR information of the m-th sample between Bob and an unknown transmitter that Bob has received, which can be expressed as:

[0098] D0:

[0099] D1:

[0100] Then, taking the Euclidean distance between the current CFR information and the historical CFR information as the test statistic L, that is, L(H t ,H A ) = ||H t - H A || 2 ; where ||·|| 2 is the Euclidean distance operator. Then, compare L with the detection threshold k for hypothesis testing. If L < k, the legitimate user Bob accepts the state D0, considers the transmitting user to be the legitimate user Alice, receives the data packet, and updates the historical CFR information; otherwise, accepts the state D1, considers the transmitting user to be the spoofing user Eve, discards the data packet. The above process can be expressed by the formula as:

[0101] Construct the state value with dynamic physical layer fingerprint features. The channel frequency response value obtained by Bob at time slot t is s t , that is, s t = [H t . Therefore, the state space S can be expressed as S = {s1, s2,..., s t}, with a dimension of 30; then input s t into the realistic actor network, and the threshold selection for the current time slot t can be output as k t , and the value range is [0, 0.1]. Therefore, the action space A can be expressed as A = {k1, k2,..., k t}.

[0102] Based on this, taking the Bayesian risk function as the instantaneous benefit function, the instantaneous benefit function r for Bob to receive the current data packet can calculate the Bayesian risk value of spoofing detection according to whether the data packet comes from a legitimate or illegal sending user.

[0103] When the data packet comes from a legitimate sending user, the reward for the receiver is:

[0104] r1 = (G1·(1 - p f ) - C1·pf )·(1-y);

[0105] Among them, G1 is the benefit of Bob receiving legally sent user data packets, C1 is the cost of Bob refusing to send legally sent user data packets, y is the attack probability, p f is the false alarm probability, i.e., the probability that a data packet from a legitimate sender is mistakenly regarded as a data packet from an illegal attacker.

[0106] When a data packet comes from a legitimate sending user, the receiver's response is:

[0107] r2=(G0·(1-p m )-C0·p m )·y;

[0108] Among them, G0 is the benefit of Bob refusing to send illegal user data packets, C0 is the cost of Bob receiving illegal user data packets, p m is the missed detection probability, that is, the probability that the data packet from the illegal attacker is mistaken for the data packet from the legitimate sender.

[0109] The receiver's benefit function can be expressed as the sum of the receiver's rewards for packets from legitimate and illegitimate users, that is:

[0110] r=r1+r2=(G1·(1-p f )-C1·p f )·(1-y)+(G0·(1-p m )-C0·p m )·y;

[0111] Among them, the above parameters can be set to G1=6, G0=9, C0=6, C1=4 respectively.

[0112] Finally, if Figure 2 As shown in Figure 1, based on the deep deterministic policy gradient framework, a detection threshold dynamic selection method is designed to detect physical layer spoofing attacks. Specifically, for the unknown transmitter at the current time t, the receiver obtains the current CFR information based on the received data packet and converts it into state s t , interact with the environment and combine the behavioral noise I to select the detection threshold. The initial value of I can be set to 0.05. Then establish a hypothesis test to detect the data packet and calculate the instantaneous benefit value r of the current state t , store sample data {s j ,a j ,r j ,s j+1} Experience replay memory, the maximum capacity of experience replay memory is 2×10 4During the interaction with the environment, when the number of samples reaches the maximum capacity of the experience replay memory, network learning begins, sampling a batch training size N from the experience replay memory. B A batch of 32 sample data {s j ,a j ,r j ,s j+1} as the training data set, and the behavioral noise gradually decreases with the increase of training times.

[0113] All neural networks in the deep deterministic policy gradient framework are three-layer deep neural networks, where the hidden layer of the network contains 256 neurons. In the training, both the real actor network and the real critic network adopt the small batch training method, the learning rate is set to 0.001, and the Adam optimizer is used for parameter update. The real critic network updates the parameters by minimizing the loss function, and the real actor network updates the parameters by back propagation of the expected return gradient.

[0114] For the real critic network, the loss function used is:

[0115]

[0116]

[0117] For real actor networks, the expected return used is:

[0118]

[0119] Among them, the discount factor λ is 0.9, j is the number of samples, r j is the instantaneous benefit value of the current data sample, s j+1 is the state of the next data sample, θ q is the actual critic network parameter, θ μ is the actual actor network parameter, θ μ' is the target actor network parameter, θ q' is the target critic network parameter.

[0120] After that, the target actor network parameters and the target critic network parameters are soft updated with a soft update coefficient of 0.01 at a certain interval T, that is, the convex combination of the current network parameters and the target network parameters is used for update. The soft update process can be expressed as:

[0121] θ q' =τθ q +(1-τ)θ q' ;

[0122] θμ' =τθ μ +(1-τ)θ μ' ;

[0123] Faced with complex actual communication scenarios and the physical layer channel environment model of unknown senders and receivers, the present invention proposes a method for detecting physical layer deception attacks by utilizing the physical layer channel state information of the communication between the senders and receivers. This method combines the decision-making advantages of reinforcement learning and the perception advantages of deep learning, and establishes a state-behavior-benefit triple based on a deep deterministic policy gradient framework, so that the receiver can select the optimal detection threshold based on the difference in CFR information between different senders and receivers in continuous time by maximizing the expected cumulative benefit, thereby quickly and accurately detecting physical layer deception attacks. At the same time, this method is adaptive to dynamic unknown environments, and can realize dynamic and continuous selection of detection thresholds, making up for the shortcomings of traditional fixed detection threshold methods.

[0124] It should be noted that the above embodiments are not intended to limit the protection scope of the present invention, and equivalent changes or substitutions made on the basis of the above technical solutions all fall within the protection scope of the claims of the present invention.

Claims

1. A physical layer deception detection method based on deep reinforcement learning, characterized in that: The following steps are involved: Step 1: Establish a spoofing attack scenario, where the receiver extracts the physical layer channel information between the sender and the receiver to represent the physical layer fingerprint features; Step 2: Establish a binary hypothesis testing model; Step 3: construct the state value with the dynamic physical layer fingerprint feature, construct the behavior value with the threshold value, use the Bayesian risk function as the instantaneous benefit function, and establish the state-behavior-benefit triple; Step 4: Based on the deep deterministic policy gradient framework in combination with step 3, a detection threshold dynamic selection method is designed to detect physical layer spoofing attacks according to the detection threshold selection result; The deception attack in step 1 is an active attack, that is, the deception attacker and the legitimate transmitter do not transmit signals at the same time. The attack probability that Eve pretends to be Alice and sends a deception signal to Bob is y∈[0,Y MAX ), where the maximum attack probability Y MAX <1; The physical layer channel information between the sender and the receiver extracted by the receiver in step 1 is the channel frequency response information of the signal between the sender and the receiver; The binary hypothesis test model in step 2 is expressed as: D0:H t =H A ; D1:H t ≠H A ; Among them, D0 represents the state where the transmitting user is a legitimate transmitter, D1 represents the state where the transmitting user is an illegal attacker, and H A is the CFR of the signal between Bob and Alice, the channel frequency response information, H t The CFR information of the signal received by Bob from the unknown transmitter; In step 2, the test statistic L is compared with the detection threshold k to perform hypothesis testing, which can be expressed as: L(H t ,H A )=||H t -H A || 2 ; Among them, ||·|| 2 is the Euclidean distance operator, L is the Euclidean distance between the current CFR information and the historical CFR information; In step 3, the state value is constructed by the dynamic physical layer fingerprint feature, and the behavior value is constructed by the threshold value, which can be expressed as: s t =[H t ]∈S; k t ∈A; Among them, s t represents the channel frequency response value obtained by Bob in time slot t, k t represents the detection threshold selected by Bob at time slot t, S is the state space, and A is the behavior space; In step 3, the Bayesian risk function is used as the instantaneous benefit function, which is expressed as: r=(G1·(1-p f )-C1·p f )·(1-y)+(G0·(1-p m )-C0·p m )·y; Among them, r is the instantaneous benefit function of Bob receiving the current data packet, G1 is the benefit of Bob receiving legally sent user data packets, G0 is the benefit of Bob refusing illegally sent user data packets, C0 is the cost of Bob receiving illegally sent user data packets, C1 is the cost of Bob refusing legally sent user data packets, and y is the attack probability; In the step 4, the detection threshold dynamic selection method based on the deep deterministic policy gradient framework is described. Specifically, Bob interacts with the environment to store the four-tuple {s t ,a t ,r t ,s t+1 } and put it into the experience pool as training sample data, where {s t ,a t ,r t ,s t+1 } respectively represent the current state, detection threshold, environmental feedback benefit and the state at the next moment, and in training, the real critic network updates parameters by minimizing the loss function, and the real actor network updates parameters by back propagating the expected return gradient. Both networks use the Adam optimizer; the network update method in step 4 specifically includes: For the real critic network, the loss function used is: For the real actor network, the expected return used is: Among them, θ q is the actual critic network parameter, θ μ is the actual actor network parameter, θ μ' is the target actor network parameter, θ q' is the target critic network parameter, λ is the discount factor, j is the number of samples, and j=1,...,N B , N B is the batch training size, r j is the instantaneous benefit value of the current data sample, s j+1 is the state of the next data sample; The target actor network parameters and the target critic network parameters are soft-updated at the interval pre-review time T, that is, the convex combination of the current network parameters and the target network parameters is used for update. The soft update process can be expressed as: i q' =tθ q +(1-τ)θ q' ; i μ' =tθ μ +(1-τ)θ μ' ; Among them, τ is the soft update coefficient, θ q' and θ μ' They are the target critic network parameters and the target actor network parameters respectively.

2. The physical layer deception detection method based on deep reinforcement learning according to claim 1 is characterized in that: The deception attack scenario established in step 1 is that Alice and Bob are legitimate users, Alice is the transmitter, Bob is the receiver, and Eve is a deceptive user; the distances between Eve and Alice and Bob are all greater than half a wavelength, that is, the illegal channel is unrelated to the legitimate channel.