Active anti-jamming method based on multi-user deep reinforcement learning

Abstract

The application provides an active anti-interference method based on multi-user deep reinforcement learning, and the implementation steps are as follows: 1) obtaining the action space and network environment information of each pair of users; 2) constructing an anti-interference network model; 3) iteratively training the anti-interference network model; and 4) obtaining an anti-interference result of the anti-interference network. The anti-interference network model constructed by the application can attack the interference efficiency evaluation function, so that the interference benefit evaluation is wrong, the interference is actively attacked, the problem that the user is interfered when the interference intelligence is enhanced is avoided, and the anti-interference efficiency is effectively improved. The virtual interference environment subnetwork can restore the real interference to the maximum extent, a virtual interference environment is established, data is sampled in the virtual environment, the training time of the method is effectively reduced, and the anti-interference efficiency is further improved.

Description

Technical Field

[0001] This invention belongs to the field of wireless ad hoc networks and relates to a multi-user anti-interference method, specifically a proactive anti-interference method based on multi-user deep reinforcement learning. Background Technology

[0002] The open and shared nature of wireless communication exposes communication networks to the wireless environment, making them more vulnerable to interference attacks compared to wired networks. Especially with the integration of artificial intelligence and communication countermeasures technologies, jamming devices possess sensing, learning, and decision-making capabilities, posing unprecedented challenges to the development of anti-jamming technologies for wireless communication.

[0003] Traditional anti-jamming approaches focus on multiple dimensions, including frequency, power, link, coding, and spatial domains. They employ techniques such as power adjustment, time-hopping / frequency-hopping, coding design, beamforming, and link optimization, leveraging reinforcement learning and game theory to alter behavior in each domain and reduce the impact of interference. Currently, single-user anti-jamming techniques are relatively mature, while multi-user anti-jamming techniques require further improvement. Mutual interference among multiple users and the unknown strategies employed by interfering users further complicate the multi-user anti-jamming problem, increasing training time, improving convergence, and significantly reducing the effectiveness of anti-jamming methods.

[0004] The development of multi-agent deep reinforcement learning has provided a promising approach to improving the above problems and has achieved breakthrough progress in the past few years. For example, patent application CN112867087A, entitled "An Anti-interference Method Based on Multi-User Random Forest Reinforcement Learning," discloses an anti-interference method based on multi-user random forest reinforcement learning. This method models the anti-interference communication problem in multi-user, multi-relay scenarios as a Markov game. Then, based on the multi-user random forest reinforcement learning algorithm, it considers both external interference and mutual interference between users. After each time slot, users exchange information, and the receiver of each user pair makes the next decision based on the current channel quality. The "decision-feedback-adjustment" process is executed iteratively to obtain the multi-user anti-interference strategy. This method achieves a high data delivery rate by continuously adjusting the multi-user anti-interference strategy through "decision-feedback-adjustment." However, this method still has shortcomings: its anti-interference efficiency is low because it passively adjusts the user's communication channel to resist interference in an interference environment. As the intelligence of interference increases, the delivery rate of this method remains low; furthermore, the training time is long when communication interaction is difficult or costly. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the existing technology and propose an active anti-interference method based on multi-user deep reinforcement learning. This method actively combats interference while ensuring anti-interference capability, thereby improving the efficiency of anti-interference.

[0006] To achieve the above objectives, the technical solution adopted by the present invention includes the following steps:

[0007] (1) Obtain the action space and network environment information for each pair of users:

[0008] The initialization of a wireless ad hoc network includes a jammer and N pairs of independent transceiver users U = {u1, u2, ..., u...}. n ,...u N}, each pair of users u n In the action space of time slot t, And collect data from each pair of users u n Based on the network environment information in the first T time slots, we obtain the network environment information set S = {s1, s2, ..., s...} t ,...s T}, where s t This represents the network environment information of N pairs of users U in the t-th time slot. Indicates each pair of users u n Network environment information in the t-th time slot. They represent user u respectively n The communication channel, received power, collected interference channel, interference power, and signal-to-interference-plus-noise ratio in the t-th time slot;

[0009] (2) Constructing an anti-interference network model H:

[0010] Construct an anti-interference network model H comprising a virtual interference environment subnetwork and a multi-agent cooperative subnetwork connected sequentially, wherein:

[0011] The virtual interference environment subnetwork includes a parallel interference waveform generation subnetwork and an interference behavior prediction subnetwork; the interference waveform generation subnetwork includes a generation network and a discrimination network connected in sequence; the interference behavior prediction subnetwork includes a feature extraction network, an LSTM network, and a softmax output layer connected in sequence.

[0012] The multi-agent cooperative sub-network includes an agent policy network and a hybrid network connected in sequence; the agent policy network includes an input layer, a GRU-gated recurrent neural network, and an output layer connected in sequence; the hybrid network includes an evaluation network and a target network arranged in parallel.

[0013] (3) Iteratively train the adversarial network model H:

[0014] (3a) Initialize the number of iterations to e, the maximum number of iterations to E, E≥100, and the current anti-interference network model H e The network parameters are θ e And let e = 0;

[0015] (3b) Each pair of users u n The network environment information S in the first T time slots is used as input to the anti-interference network model H for forward propagation:

[0016] (3b1) The interference waveform generation subnetwork in the virtual interference environment subnetwork performs feature learning on the network environment information S to obtain the simulated interference waveform; at the same time, the interference behavior prediction subnetwork performs future behavior prediction on the network environment information S to obtain the interference channel with the highest probability value of interference channel A and interference power B of the jammer. Interference power

[0017] (3b2) In the multi-agent cooperative sub-network, the agent policy network considers the simulated interference waveform and the interference channel with the highest probability values ​​for interference channel A and interference power B. Interference power Perform forward propagation to obtain each pair of users u n Action space and return value Hybrid networks, through each pair of users u n Action space and return value Calculate and estimate the overall action-value function Q tot (evaluate) and the overall action-value function Q of the target tot (target);

[0018] (3c) Employ the cross-entropy loss function and estimate the overall action-value function Q. tot (evaluate) and the overall action-value function Q of the target tot The difference between (target) and (target) is used to calculate the loss value L of the anti-interference network model H. e ; through the loss value L e Network parameters θ of the anti-interference network model e The system is updated to obtain the anti-interference network model H after this iteration. e ;

[0019] (3d) Determine whether e≥E holds true. If yes, obtain the trained anti-interference network model H'. Otherwise, let e=e+1 and H=H e and perform step (3b);

[0020] (4) Obtain the anti-interference results of the anti-interference network:

[0021] Each pair of users u n Network environment information in time slot l The data is then forward-propagated as input to the trained anti-interference network model H' to obtain the data for each pair of users u. n Q - Value table, and Q in it - The communication channel and communication power corresponding to the maximum value are used as the anti-interference result for each user.

[0022] Compared with the prior art, the present invention has the following advantages:

[0023] (1) The multi-agent cooperative sub-network in the anti-interference network model built in this invention uses the method of switching the communication state according to probability when the user is not interfered with and not switching the communication mode according to probability when interfered with, to attack the interference effectiveness evaluation function, so as to make the interference benefit evaluation incorrect, thereby destroying the interference learning loop, avoiding the problem that the user's anti-interference strategy is suppressed when the interference intelligence is enhanced, actively attacking the interference, and improving the efficiency of anti-interference.

[0024] (2) The virtual interference environment subnetwork in the anti-interference network model constructed in this invention includes an interference waveform generation subnetwork and an interference behavior prediction subnetwork. The interference waveform generation subnetwork can autonomously learn the distribution of the interference waveform and generate a simulated interference waveform with the same distribution as the interference waveform. The interference behavior prediction subnetwork uses the experience of the LSTM network on historical interference channels and interference power to perform feature learning and predict the future behavior of the interfering party. The virtual interference environment is established through the output of the two subnetworks to restore the real interference to the greatest extent. By sampling data in the virtual environment, the training time of the method is effectively reduced and the anti-interference efficiency is further improved. Attached Figure Description

[0025] Figure 1 This is a flowchart illustrating the implementation of the present invention;

[0026] Figure 2 This is a schematic diagram of the anti-interference network model of the present invention;

[0027] Figure 3 This is the curve showing the change in the average reward value during the training process of this invention.

[0028] Figure 4 This is the curve showing the change in delivery rate after the convergence of this invention. Detailed Implementation

[0029] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0030] Reference Figure 1The present invention includes the following steps:

[0031] Step 1) Obtain the action space and network environment information for each pair of users:

[0032] The initialization of a wireless ad hoc network includes a jammer and N pairs of independent transceiver users U = {u1, u2, ..., u...}. n ,...u N}, each pair of users u n In the action space of time slot t, And collect data from each pair of users u n Based on the network environment information in the first T time slots, we obtain the network environment information set S = {s1, s2, ..., s...} t ,...s T}, where s t This represents the network environment information of N pairs of users U in the t-th time slot. Indicates each pair of users u n Network environment information in the t-th time slot. They represent user u respectively n The communication channel, received power, collected interference channel, interference power, and signal-to-interference-plus-noise ratio in the t-th time slot;

[0033] Step 2) Construct the anti-interference network model H:

[0034] Construct an anti-interference network model H comprising a virtual interference environment subnetwork and a multi-agent cooperative subnetwork connected sequentially, wherein:

[0035] The virtual interference environment subnetwork includes a parallel interference waveform generation subnetwork and an interference behavior prediction subnetwork; the interference waveform generation subnetwork includes a generation network and a discrimination network connected in sequence; the interference behavior prediction subnetwork includes a feature extraction network, an LSTM network, and a softmax output layer connected in sequence.

[0036] The multi-agent cooperative sub-network includes an agent policy network and a hybrid network connected in sequence; the agent policy network includes an input layer, a GRU-gated recurrent neural network, and an output layer connected in sequence; the hybrid network includes an evaluation network and a target network arranged in parallel.

[0037] Step 3) Iteratively train the adversarial network model H:

[0038] Step 3a) Initialize the number of iterations to e, the maximum number of iterations to E, E≥100, and the current anti-interference network model H e The network parameters are θ e And let e = 0;

[0039] Step 3b) For each pair of users u n The network environment information S in the first T time slots is used as input to the anti-interference network model H for forward propagation:

[0040] Step 3b1) The generation network in the interference waveform generation subnetwork of the virtual interference environment subnetwork obtains the interference data D = {d1, d2, ..., d...} of the jammer in T time slots by splitting the network environment information S. t ,...d T The generator network learns the data distribution of interference data D to obtain a simulated interference waveform; the discriminator network compares the data distribution of interference data D with the simulated interference waveform to determine whether it is true or false. If it is false, the generator network relearns the data distribution of interference data D to obtain a simulated interference waveform; if it is true, it outputs the simulated interference waveform, where d t This represents the jamming data of the jammer over t time slots.

[0041] The feature extraction network in the interference behavior prediction subnetwork extracts features from the network environment information S, obtaining a feature set G = {g1, g2, ..., g...} t ,...,g T}; The LSTM network processes each feature g t Time series prediction is performed to obtain the probability values ​​of A interference channels and B interference powers; the softmax output layer outputs the interference channel with the highest probability value among A interference channels and B interference powers through the softmax function. and interference power

[0042] Step 3b2) The GRU-gated recurrent neural network of the agent policy network in the multi-agent cooperative sub-network compares the simulated interference waveform with the interference channels A and B, which have the highest probability values ​​for interference power. Interference power Calculate each pair of users u n The Q-value table; and the judgment based on the Q-value table. Is this true? If so, user u n If time slot T is interfered with, a random value is selected. If the value is greater than Z, the communication state is switched, and the communication channel for the (T+1)th time slot is selected from the Q-value table according to the ε-greedy strategy. and communication power Otherwise, user u n If time slot T is not disturbed, let Get each pair of users u n execution action space and through Calculate the return value

[0043]

[0044]

[0045] Here, the ε-greedy strategy means randomly selecting a Q-value from the Q-value table with probability ε, and selecting the largest Q-value from the Q-value table with probability 1-ε, where α represents the channel handover cost factor. Here, β represents the power cost factor, k represents the number of mutually interfering node pairs, and Th is the SINR value that the receiving node can decode. Indicate whether there is external interference. Indicates whether there is mutual interference;

[0046] Hybrid networks, through each pair of users u n Action space and return value Calculate and estimate the overall action-value function Q tot (evaluate) and the overall action-value function Q of the target tot (target), the calculation formulas are as follows:

[0047]

[0048]

[0049] Where x represents the network parameters of the evaluation network, and y represents the network parameters of the target network;

[0050] Step 3c) employs the cross-entropy loss function and estimates the overall action-value function Q. tot (evaluate) and the overall action-value function Q of the target tot The difference between (target) and (target) is used to calculate the loss value L of the anti-interference network model H. e ; through the loss value L e Network parameters θ of the anti-interference network model e The system is updated to obtain the anti-interference network model H after this iteration. e ;

[0051] The calculation and update formulas are as follows:

[0052]

[0053]

[0054] Where C represents each pair of users u n Total number of communications, θ e 'represents θ eThe update result, where η represents the step size of the parameter shift. Indicates L e Perform gradient descent updates;

[0055] Step 3d) Determine if e ≥ E holds true. If yes, obtain the trained anti-interference network model H'. Otherwise, let e = e + 1 and H = H. e and perform step (3b);

[0056] Step 4) Obtain the anti-interference results of the anti-interference network:

[0057] Each pair of users u n Network environment information in time slot l The data is then forward-propagated as input to the trained anti-interference network model H' to obtain the data for each pair of users u. n Q - Value table, and Q in it - The communication channel and communication power corresponding to the maximum value are used as the anti-interference result for each user.

[0058] The technical effects of the present invention will be explained below with reference to simulation experiments.

[0059] 1. Simulation conditions and content:

[0060] The hardware test platform for the simulation experiment is as follows: CPU is Intel Core i7-9700 with a main frequency of 3.00GHz, memory is 8GB, and GPU is NVIDIA GeForce GTX 1050Ti; the software platform is Windows 10 64-bit operating system and Python 3.7.

[0061] The simulation experiment took place in a 1000m x 1000m area, where five pairs of independent transceivers and one jammer were randomly placed. The jammer employed a reinforcement learning-based jamming strategy. During the simulation training phase, the average reward value of this invention changed with the training step size as shown in the curve below. Figure 3 As shown; the curve of the delivery rate after convergence of this invention versus time is shown below. Figure 4 As shown.

[0062] 2. Simulation Result Analysis:

[0063] Reference Figure 3 , Figure 4 Compared with the prior art, the present invention has shown superiority in training time and delivery rate, and improved anti-interference efficiency.

Claims

1. An active anti-interference method based on multi-user deep reinforcement learning, characterized in that, Includes the following steps: (1) Obtain the action space and network environment information for each pair of users: The initialization of a wireless ad hoc network includes a jammer and N pairs of independent transceiver users U = {u1, u2, ..., u...}. n ,...u N }, each pair of users u n In the action space of time slot t, And collect data from each pair of users u n Based on the network environment information in the first T time slots, we obtain the network environment information set S = {s1, s2, ..., s...} t ,...s T }, where s t This represents the network environment information of N pairs of users U in the t-th time slot. Indicates each pair of users u n Network environment information in the t-th time slot. They represent user u respectively n The communication channel, received power, collected interference channel, interference power, and signal-to-interference-plus-noise ratio in the t-th time slot; (2) Constructing an anti-interference network model H: Construct an anti-interference network model H comprising a virtual interference environment subnetwork and a multi-agent cooperative subnetwork connected sequentially, wherein: The virtual interference environment subnetwork includes a parallel interference waveform generation subnetwork and an interference behavior prediction subnetwork; the interference waveform generation subnetwork includes a generation network and a discrimination network connected in sequence; the interference behavior prediction subnetwork includes a feature extraction network, an LSTM network, and a softmax output layer connected in sequence. The multi-agent cooperative sub-network includes an agent policy network and a hybrid network connected in sequence; the agent policy network includes an input layer, a GRU-gated recurrent neural network, and an output layer connected in sequence; the hybrid network includes an evaluation network and a target network arranged in parallel. (3) Iteratively train the adversarial network model H: (3a) Initialize the number of iterations to e, the maximum number of iterations to E, E≥100, and the current anti-interference network model H e The network parameters are θ e And let e = 0; (3b) Each pair of users u n The network environment information S in the first T time slots is used as input to the anti-interference network model H for forward propagation: (3b1) The interference waveform generation subnetwork in the virtual interference environment subnetwork performs feature learning on the network environment information S to obtain the simulated interference waveform; at the same time, the interference behavior prediction subnetwork performs future behavior prediction on the network environment information S to obtain the interference channel with the highest probability value of interference channel A and interference power B of the jammer. Interference power (3b2) In the multi-agent cooperative sub-network, the agent policy network considers the simulated interference waveform and the interference channel with the highest probability values ​​for interference channel A and interference power B. Interference power Perform forward propagation to obtain each pair of users u n Action space and return value Hybrid networks, through each pair of users u n Action space and return value Calculate and estimate the overall action-value function Q tot (evaluate) and the overall action-value function Q of the target tot (target); (3c) Employ the cross-entropy loss function and estimate the overall action-value function Q. tot (evaluate) and the overall action-value function Q of the target tot The difference between (target) and (target) is used to calculate the loss value L of the anti-interference network model H. e ; through the loss value L e Network parameters θ of the anti-interference network model e The system is updated to obtain the anti-interference network model H after this iteration. e ; (3d) Determine whether e≥E holds true. If yes, obtain the trained anti-interference network model H'. Otherwise, let e=e+1 and H=H e and perform step (3b); (4) Obtain the anti-interference results of the anti-interference network: Each pair of users u n Network environment information in time slot l The data is then forward-propagated as input to the trained anti-interference network model H' to obtain the data for each pair of users u. n Q - Value table, and Q in it - The communication channel and communication power corresponding to the maximum value are used as the anti-interference result for each user.

2. The active anti-interference method based on multi-user deep reinforcement learning according to claim 1, characterized in that, The interference waveform generation subnetwork in the virtual interference environment subnetwork described in step (3b1) performs feature learning on the network environment information S, and the interference behavior prediction subnetwork predicts future behavior of the network environment information S. The implementation steps are as follows: The generator network in the interference waveform generation subnetwork obtains the interference data D = {d1, d2, ..., d...} of the jammer in T time slots by splitting the network environment information S. t ,...d T The generator network learns the data distribution of interference data D to obtain a simulated interference waveform; the discriminator network compares the data distribution of interference data D with the simulated interference waveform to determine whether it is true or false. If it is false, the generator network relearns the data distribution of interference data D to obtain a simulated interference waveform; if it is true, it outputs the simulated interference waveform, where d t This represents the jamming data of the jammer over t time slots. The feature extraction network in the interference behavior prediction subnetwork extracts features from the network environment information S, obtaining a feature set G = {g1, g2, ..., g...} t ,...,g T }; The LSTM network processes each feature g t Time series prediction is performed to obtain the probability values ​​of A interference channels and B interference powers; the softmax output layer outputs the interference channel with the highest probability value among A interference channels and B interference powers through the softmax function. and interference power 3. The active anti-interference method based on multi-user deep reinforcement learning according to claim 1, characterized in that, The agent policy network described in step (3b2) identifies the simulated interference waveform and the interference channel with the highest probability values ​​among A interference channels and B interference power. Interference power Perform forward propagation, and for each pair of users u in the hybrid network. n Action space and return value Calculations are performed to obtain the estimated overall action-value function Q. tot (evaluate) and the overall action-value function Q of the target tot (target), the implementation steps are as follows: The GRU-gated recurrent neural network of the agent policy network compares simulated interference waveforms with A interference channels and B interference channels with the highest probability values ​​of interference power. Interference power Calculate each pair of users u n The Q-value table; and the judgment based on the Q-value table. Is this true? If so, user u n If interference occurs in time slot T, the communication state is switched according to probability, and the communication channel for the (T+1)th time slot is selected from the Q-value table according to the ε-greedy strategy. and communication power Otherwise, user u n If time slot T is not disturbed, let Get each pair of users u n execution action space and through Calculate the return value Here, the ε-greedy strategy means randomly selecting a Q-value from the Q-value table with probability ε, and selecting the largest Q-value from the Q-value table with probability 1-ε, where α represents the channel handover cost factor. Here, β represents the power cost factor, k represents the number of mutually interfering node pairs, and Th is the SINR value that the receiving node can decode. Indicate whether there is external interference. Indicates whether there is mutual interference; Hybrid networks, through each pair of users u n Action space and return value Calculate and estimate the overall action-value function Q tot (evaluate) and the overall action-value function Q of the target tot (target), the calculation formulas are as follows: Where x represents the network parameters of the evaluation network, and y represents the network parameters of the target network.

4. The active anti-interference method based on multi-user deep reinforcement learning according to claim 1, characterized in that, The step (3c) involves calculating the loss value L of the anti-interference network model H. e ; and the network parameters θ of the adversarial interference network model. e The update is performed, and the calculation and update formulas are as follows: Where C represents each pair of users u n Total number of communications, θ e 'represents θ e The update result, where η represents the step size of the parameter shift. Indicates L e Perform gradient descent updates.

Images