Ad hoc network anti-interference system implementation method based on deep reinforcement learning

Abstract

The invention discloses an implementation method of an ad hoc network anti-interference system based on deep reinforcement learning, and the method comprises the steps: 1, constructing a simulation network environment based on a network simulator 3 (NS-3) simulation platform and an OpenAI Gym interface, and enabling the simulation network environment to be used for collecting a channel state in real time and feeding back the channel state to an intelligent agent; and 2, after receiving the network state, the intelligent agent feeds back a decision result obtained by learning to a network node to guide channel switching in real time. And step 3, fusing mechanisms such as a long short-term memory (LSTM) network, a competition network structure, a double deep Q network, a double deep Q network, a prioritized experience replay (PER) and the like, designing a PER-D3RQN algorithm, and carrying out efficient prediction on an interference time sequence rule. According to the method, the accuracy of channel selection can be remarkably improved in multiple interference modes, the network performance and the anti-interference robustness are improved, and the performance superior to that of a traditional deep Q-network (DQN) and a traditional frequency hopping strategy is shown.

Description

TECHNICAL FIELD

[0001] The present application relates to a method for implementing an anti-interference system of an ad hoc network based on deep reinforcement learning, and belongs to the technical field of wireless communication networks. BACKGROUND

[0002] An ad hoc network is an instant communication network composed of a group of nodes with wireless communication capabilities. It has no fixed topology structure, and nodes can freely perform network entry or network exit operations. Compared with traditional networks that rely on infrastructure, an ad hoc network does not need a central management unit such as a base station to establish a multi-hop wireless transmission network, and the nodes in the network cooperate to ensure the stable operation of the network and the reliable transmission of data. Each node can directly interact with neighbor nodes within its communication range. If the data generated by a node needs to be transmitted to a destination node beyond the communication range, it can be forwarded to the destination node step by step by means of relay nodes through multi-hop transmission.

[0003] However, an ad hoc network still faces many challenges when it is in a complex interference environment. First, network nodes often work in parallel using wireless media, which are easily affected by external interference sources (such as hostile interference and a noisy electromagnetic environment) and internal network conflicts (channel competition between adjacent nodes). Second, due to the dynamic changes in node topology, the quality of links and the state of available channels often fluctuate, making it difficult for traditional fixed channels or simple frequency hopping mechanisms to continuously and effectively resist changing interference patterns. In emergency or military scenarios, interference can take many forms such as random, sweeping, or directional, requiring the network to have adaptive and cross-layer coordination capabilities to maintain sufficient communication bandwidth and link communication quality.

[0004] In recent years, machine learning has attracted extensive attention in the field of wireless communication networks. A real-time reinforcement learning method is proposed to learn the optimal dynamic channel allocation strategy through interaction with the mobile communication environment. Based on the probabilistic interference model, a frequency hopping strategy is implemented using the Q-learning algorithm, which has fewer frequency hopping times and average revenue close to the theoretical optimum compared to existing methods. Deep reinforcement learning combines the powerful representation ability of neural networks, effectively solving the problem of large state-action space, and has better environmental perception and strategy learning ability. A deep anti-interference Q network is proposed, which directly uses a convolutional neural network (CNN) to process the spectrum sequence to estimate the Q value of the communication action, without the need for explicit interference pattern modeling, and can cope with unknown interference. A double deep Q network based on a Transformer encoder is designed to simulate the confrontation between a cognitive wireless network and a jammer, estimate the action value from raw spectrum data, and effectively defend against various interference attacks. Finally, a double deep Q network is used to simultaneously learn channel access and transmission power control strategies to cope with various interference scenarios, and the Q function is approximated by a neural network to effectively improve communication performance.

[0005] Although some progress has been made in reinforcement learning-based communication network optimization, existing research is mostly verified in simplified or idealized environments, lacking the support of real protocol stacks and reliable simulation platforms. At the same time, existing algorithms still lack in modeling the timing characteristics and dynamic changes of interference, making it difficult to adapt to actual complex environments. Finally, existing research mostly makes decisions for single channels, with limited spectrum resource utilization, failing to take advantage of the potential of multi-channel parallel access in improving bandwidth and anti-interference capability. SUMMARY

[0006] The present invention aims to address the deficiencies and shortcomings of the prior art by proposing an implementation method for an ad hoc network anti-interference system based on deep reinforcement learning. This method can process the state information of multiple channels simultaneously and improve system performance by intelligently selecting the optimal channel combination. Simulation results show that this method can significantly improve the accuracy of channel selection, network performance, and anti-interference robustness under various interference modes, outperforming traditional deep Q networks (DQN) and traditional frequency hopping strategies.

[0007] The technical solution adopted by the present invention to solve its technical problems is an implementation method for an ad hoc network anti-interference system based on deep reinforcement learning, which includes the following steps:

[0008] Step 1: a simulation network environment based on the interface between a network simulator 3 (NS-3) simulation platform and OpenAI Gym is constructed, and is used for collecting channel states in real time and feeding back to an intelligent agent.

[0009] Step 2: after receiving the network state, the intelligent agent feeds back the decision result learned to network nodes, and guides channel switching in real time.

[0010] Step 3: a PER-D3RQN algorithm is designed by fusing a long short-term memory (LSTM) network, a dueling network structure, a double deep Q-network (Double DQN) and a prioritized experience replay (PER) mechanism, and the algorithm is used for efficient prediction of interference timing rules.

[0011] Beneficial effects:

[0012] 1: the application can process the state information of multiple channels simultaneously, and improves system performance by intelligently selecting an optimal channel combination. Simulation results show that the method can significantly improve the accuracy of channel selection under various interference modes, improve network performance and anti-interference robustness, and has better performance than a traditional deep Q-network (DQN) and a traditional frequency hopping strategy.

[0013] 2: the intelligent anti-interference method based on the PER-D3RQN proposed in the application effectively improves the spectrum scheduling capability of a wireless ad hoc network in a complex interference environment through deep reinforcement learning, and the superiority of the method in multi-channel selection and anti-interference performance is verified through NS-3 simulation results, and the method can explore a distributed architecture and cross-layer optimization, considers the influence of same-frequency transmission to enhance the practicability and expansibility of the algorithm, and provides a more robust solution for intelligent wireless communication. DETAILED DESCRIPTION

[0014] Figure 1 It is a system network model diagram of the application.

[0015] Figure 2 It is an ns3-gym framework diagram of the application.

[0016] Figure 3 It is a single-channel Markov state transition model diagram of the application.

[0017] Figure 4 It is a PER-D3RQN algorithm framework diagram of the application.

[0018] Figure 5 This is a schematic diagram illustrating the performance of PER-D3RQN under different interference conditions according to the present invention.

[0019] Figure 6 This is a schematic diagram comparing the various comb-like interference algorithms of this invention.

[0020] Figure 7 This is a schematic diagram comparing the various algorithms of the dual frequency sweep of this invention.

[0021] Figure 8 This is a schematic diagram comparing the various algorithms of the greedy random perturbation method of this invention.

[0022] Figure 9 This is a schematic diagram comparing the various algorithms of the present invention, which combine random and dual frequency sweeping. Detailed Implementation

[0023] The invention will now be described in further detail with reference to the accompanying drawings.

[0024] like Figure 1 As shown, the network nodes of this invention are divided into three categories: sensing nodes, interference nodes, and communication nodes. Sensing nodes are responsible for real-time sensing of the occupancy status of each channel within the communication frequency band and uploading the observation results to the central agent. Interference nodes interfere with some channels according to a preset mode, simulating external interference sources in a real wireless environment. To optimize spectrum resource allocation and avoid resource contention between subnets, the communication nodes are divided into multiple subnets. The agent, based on global channel state information, centrally decides to select the optimal channel combination and allocates these channels to different subnets.

[0025] This invention models the multi-channel anti-jamming allocation problem as a reinforcement learning task and constructs a co-simulation platform based on the NS-3 network simulator and the OpenAI Gym interface. This platform utilizes the ns3-gym middleware.

[12] To achieve seamless integration of C++ and Python programming environments, such as... Figure 2 As shown. NS-3 is responsible for simulating the physical layer and protocol stack characteristics of the network, including node topology, spectrum propagation model, 802.11 protocol and interference mechanism; while the reinforcement learning algorithm is implemented in the OpenAI Gym environment. The two interact in real time through a socket interface based on ZMQ for state observation and action execution.

[0026] The environment definition for reinforcement learning in this invention includes the following:

[0027] (1) Environment: The present application is based on IEEE 802.11a protocol, which divides the 5GHz band into n non-overlapping channels, each with a bandwidth of 20MHz, and the channel index is 0 to n-1. Compared with the 2.4GHz band, the 802.11a protocol provides more non-overlapping channel resources in the 5GHz band, which helps to reduce co-channel interference and improve spectrum utilization efficiency. Denoted by a set as:

[0028] M = {0, 1, 2,..., n-1} (1)

[0029] Each node can freely select a channel within the above set for communication.

[0030] The interference node simulates the rapidly changing and complex interference mode in reality, which is manifested as part of the channels being blocked due to interference in each time slot. The channel state can be divided into normal (available) or interfered (unavailable), as shown in Figure 3 .

[0031] where P 10 represents the channel in the "interfered" state, and the probability of changing to the "normal" state in the next time slot;

[0032] P 11 represents the probability of the interfered channel continuing to be in the "interfered" state, and P 10 and P 11 are complementary and sum to 1. Correspondingly, P 00 and P 01 represent the probability of the channel in the "normal" state continuing to be normal and interfered in the next time slot, respectively.

[0033] (2) State: In this simulation environment, the power spectral density of the i-th channel at frequency f and time slot t is defined as:

[0034] PSD i (f, t) = N0 + S i (f) (2)

[0035] where N0 is the background noise spectral density, which is taken as 1x10 -15 W / Hz. S i (f) is the flat transmit spectral density of the interference source on the channel, which is calculated as:

[0036]

[0037] where P int,i is the total transmit power of the i-th interference source, and B i represents the channel bandwidth, which is taken as 20MHz according to the IEEE 802.11a protocol. For the bandwidth B iThe total received power of the ith channel at time slot t can be obtained by integrating (or summing up in discrete case) the power spectral density:

[0038]

[0039] where f k is the discrete sampling frequency and Δf is the frequency resolution. The linear power P i (t) is converted to dBW:

[0040] P i,dB (t) = 10log 10 (P i (t)) (5) The CCA (Clear Channel Assessment) threshold θ CCA is set to -92dBW according to IEEE 802.11a protocol (i.e. -62dBm energy detection threshold). The occupation state of the ith channel is:

[0041]

[0042] Finally, all channel states are concatenated into a column vector of length n as the observation input of the agent at time slot t. This state vector reflects whether each channel is available at the current time, and the agent judges which channels are interfered and avoids the occupied channels to select the idle channels for data transmission.

[0043] s(t) = [s1(t), s2(t),..., s n (t)] T (7)

[0044] (2) Action: The action space A is a discrete space, representing the channel combination that the agent can choose at each time slot. The size of the action space is determined by the following formula:

[0045] |A| = C(n, k) (8)

[0046] where n represents the total number of available channels, k represents the number of channels to be selected each time, and C(n, k) represents the number of combinations of selecting k channels from n channels. Each action a e A corresponds to a channel combination, which can be represented as:

[0047] a = {c1, c2,..., c k} (9)

[0048] where c i represents the number of the ith selected channel.

[0049] (3) Reward: The design of the reward function R takes into account the channel occupation state:

[0050]

[0051] where r i denotes the immediate reward of the i-th selected channel:

[0052]

[0053] O(c i ) denotes the occupancy state of channel c i (0 for free, 1 for occupied), a positive reward +1.0 is obtained when a free channel is selected, and a negative reward -1.0 is obtained when an occupied channel is selected.

[0054] Through the above model and definition, the goal of the agent is to obtain the perception information of the state of each channel in the current time slot at the beginning of each time slot, and predict the state change of the next time slot based on the information, so as to maximize the long-term cumulative reward R.

[0055] In view of the characteristics that the interference state has time-varying and incomplete observation in the multi-channel environment, the application combines the Dueling structure and Double Q learning strategy on the basis of DRQN (Deep Recurrent Q-Network), further introduces priority experience replay, and proposes a channel decision algorithm based on PER-D3RQN. The method combines multiple improved technologies in the field of reinforcement learning, aims to solve the problems of time sequence dependence and overestimation at the same time, and improves the training efficiency through priority playback, Figure 4 The PER-D3RQN algorithm framework diagram.

[0056] On the basis of the deep reinforcement learning framework, the application designs a PER-D3RQN algorithm to cope with the characteristics that the interference state has time-varying and incomplete observation in the multi-channel environment. The algorithm integrates the following key technical components:

[0057] (1) LSTM time sequence modeling: In view of the problem that the channel state is partially observable in the ad hoc network environment, LSTM is introduced to capture the time sequence characteristics of interference. The core calculation process is:

[0058] f t =σ(W f ·[h t-1 ,x t ]+b f ) (12)

[0059] In the formula, f t represents the forgetting gate output, which determines how much information of the previous state is retained; sigma is a sigmoid activation function; W f is a forgetting gate weight matrix; h t-1 is the hidden state at the previous moment; xt is the input at the current time; b f is the bias term. LSTM effectively extracts the time correlation of channel interference through the memory mechanism, enhancing the prediction ability of future interference states.

[0060] (2) Dueling network structure: decompose Q value into state value function V(s) and advantage function A(s, a):

[0061]

[0062] In the formula, Q(s, a) is the state-action value function; V(s) measures the intrinsic value of state s; A(s, a) evaluates the relative advantage of selecting action a in state s; |A| is the size of the action space. This decomposition enables the model to learn the value of the state without changing the current policy, improving learning efficiency and stability.

[0063] (3) Double DQN mechanism: separate action selection and evaluation to alleviate the problem of overestimation of Q value:

[0064] y i = r i + γQ(s i+1 , argmax a′ Q(s i+1 , a'; θ); θ - ) (14)

[0065] In the formula, y i is the target Q value; r i is the immediate reward; γ is the discount factor; θ is the current network parameter; θ - is the target network parameter. The current network is used to select actions, while the target network evaluates the value of the action, reducing the bias caused by over-optimistic estimation.

[0066] (4) Prioritized experience replay: Compared with traditional uniform random sampling, prioritized experience replay determines the importance of samples based on the size of their TD error, and preferentially learns conversion samples with large information quantity. Prioritized experience replay determines sample sampling probability according to TD error, accelerating the learning of key conversions:

[0067] p i = |δ i | + ε (15)

[0068]

[0069] where p i is the priority of sample i; δ iis the TD error; ε is a small positive number to prevent priority zero; P(i) is the sampling probability; and α is the control parameter of priority degree. To correct the bias caused by non-uniform sampling, the importance sampling weight w is introduced i :

[0070]

[0071] where N is the buffer size, and β increases from the initial value to 1 as the training proceeds. This mechanism makes the network focus more on samples with large amounts of information, improving learning efficiency and convergence speed.

[0072] Combining the above modules, the PER-D3RQN algorithm forms an end-to-end deep reinforcement learning solution that can effectively deal with the interference prediction and channel allocation problem in a multi-channel environment.

[0073] Table 1 is the anti-interference decision algorithm. In the algorithm design, the simulation environment will start spectrum detection at fixed time, and the observation state of each channel will be sent into the reinforcement learning interface in real time through the callback mechanism in NS-3; after the Python agent receives the complete channel occupation state, it generates the channel selection of the next time slot using the PER-D3RQN network and the ε-greedy strategy, and sends the selection to the simulation environment to complete the channel switching. The simulation environment performs data transmission and statistics packet transmission success rate, compares the agent's predicted channel with the current spectrum state and returns the instant reward; finally, the agent stores the experience composed of the current state, the selected action, the reward and the next state into the priority replay pool, and updates the network parameters based on the TD error priority sampling.

[0074] Table 1 Anti-interference algorithm design

[0075]

[0076] The simulation experiment and performance comparison of the present application include the following:

[0077] The present application uses the NS3-Gym framework to simulate the reinforcement learning algorithm on the Linux platform. The Python version is 3.8, and the PyTorch framework is used to simulate and evaluate the anti-interference method based on PER-D3RQN. The anti-interference performance of the proposed method is tested in different interference environments, and compared with the random frequency hopping method and the anti-interference method based on DQN.

[14]

[0078] The simulation constructs a Figure 1 ​The network system model shown, the simulation environment contains two subnets, each subnet is distributed within several communication nodes, each subnet is randomly generated a UDP data stream. Network nodes using IEEE 802.11a protocol Ad Hoc mode communication, OLSR protocol to realize multi-hop routing. Every 0.1 seconds reinforcement learning algorithm for channel decision and dynamic switching. Data communication, the nodes within each subnet through the UDP application for data packet transmission, packet generation cycle is 0.1 seconds, to keep the smart decision cycle corresponding.

[0079] In the simulation, six different interference patterns are designed and implemented:

[0080] (1) Swept frequency interference: in each interference time slot, the interference signal is switched on the candidate channel in turn, so as to simulate the effect that the center frequency of the interference signal changes gradually with time according to the fixed scanning rate.

[0081] (2) Comb interference: this mode divides the channels into two groups: odd and even. Using the way of alternating switch, it presents the "on-off-on-off" staggered state on one group of channels, maintains the fixed mode in each time slot, and periodically flips in the next time slot, so as to form a comb-like spectrum distribution.

[0082] (3) Greedy random strategy interference: in each time slot, the interference state of each channel remains unchanged with 80% probability, and the original state is reversed with 20% probability, so that the interference signal presents the characteristics of random fluctuation.

[0083] (4) Random interference: this mode depends on the preset interference probability of each channel, and randomly decides whether each channel produces interference in each time slot, that is, whether each channel receives interference at each time according to the preset probability distribution, so as to constitute another random change interference mode.

[0084] (5) Double swept frequency interference: on the basis of traditional swept frequency interference, a bidirectional scanning mechanism is introduced, that is, when the two ends of the channel are in the interference state at the same time, the interference signal will converge to the middle.

[0085] (6) Multiple mixed interference: this mode combines random interference with double swept frequency interference, that is, in each time slot, the interference state of each channel is randomly set first, and then the swept frequency state is forced to be introduced at the predetermined position, so that the interference signal has randomness and periodicity at the same time, and presents multiple interference characteristics.

[0086] All interference patterns are implemented in the same 20MHz bandwidth, and the interference signal power is obtained by dividing the preset 10 normalized units by the bandwidth to get the power spectral density, so as to ensure the relative power of the signal in each mode is consistent, providing a unified reference for subsequent reinforcement learning decision.

[0087] As Figure 5 shown in the figure, the channel selection accuracy of the algorithm under different interference modes shows a trend of continuous improvement with the increase of training rounds when PER-D3RQN selects 2 channels in 6 channels. Regular interference modes such as sweep, comb and double sweep can reach a high accuracy in a short training round, and gradually converge to a level close to 100%. This shows that the agent can quickly learn the characteristics of these interference modes and make corresponding optimal channel selection.

[0088] For random, greedy and multiple mixed interference, the agent needs to face higher uncertainty and must continue to explore in a longer training time to steadily improve the prediction accuracy of channel occupation, resulting in slower rising speed or more fluctuations in the middle and later stages.

[0089] The present application divides the interference type into regular interference (such as comb, double sweep) and random interference (such as greedy interference, mixed interference), and compares and analyzes through PER-D3RQN, DQN and random frequency hopping three algorithms.

[0090] As Figure 6 shown, the performance of PER-D3RQN and DQN is relatively close under comb interference, because the regularity of comb interference is relatively simple, and the interference state of each time slot channel alternates, so that DQN can learn an effective strategy through cumulative experience even without the ability of time modeling and priority experience replay.

[0091] As Figure 7 shown, under the double sweep interference environment, the accuracy of PER-D3RQN is significantly higher than that of DQN, and can reach an accuracy close to 100% in the later training period. The reason is that DQN is memoryless and cannot use past observation information, so its strategy decision-making ability is limited when facing interference modes with time-dependent channel state changes.

[0092] As Figure 8 shown, the accuracy of PER-D3RQN always leads under greedy random interference environment, and finally converges above 75%. The characteristic of greedy random interference is that each channel state has high randomness. In this case, reinforcement learning algorithm needs long-term memory of historical observations to predict future interference patterns. In addition, PER-D3RQN combines priority experience replay, which can more efficiently use key experiences to improve training efficiency and make it converge to a better strategy.

[0093] As Figure 9As shown, in the complex mixed interference environment, the accuracy of PER-D3RQN is stabilized at about 80%, which is much higher than 60% of DQN and 35% of random frequency hopping. The mixed interference mode contains random and timing changes at the same time. The LSTM of PER-D3RQN effectively captures the complex timing relationship, combines Double DQN, Dueling DQN structure and priority experience replay mechanism, reduces the overestimation of Q value and improves the decision quality, so that it shows better robustness and adaptability in complex environment.

[0094] In the NS-3 environment, the present application analyzes the actual network performance of PER-D3RQN, DQN and random frequency hopping algorithms in a complex interference environment. The simulation uses IEEE 802.11a protocol, and the nodes realize multi-hop routing through OLSR protocol. The agent of each subnet performs channel decision and transmits data packet every 0.1 seconds.

[0095] The simulation compares and analyzes the communication network performance of PER-D3RQN, DQN and random frequency hopping algorithms under different interference conditions. As shown in the figure, Figure 9 As shown, the packet sending success rate of each algorithm is more than 75%, which is mainly due to the built-in retransmission mechanism of IEEE 802.11a, so that even if the channel is misjudged for a short time, the data packet still has a chance to be successfully sent in the subsequent switching. From the figure, it can be seen that in the mixed interference environment, PER-D3RQN can more accurately predict the interference state, so as to effectively avoid high interference channels and reduce data packet loss.

[0096] The intelligent anti-interference method based on PER-D3RQN proposed in the present application effectively improves the spectrum scheduling ability of wireless ad hoc network in complex interference environment through deep reinforcement learning. The NS-3 simulation results verify its superiority in multi-channel selection and anti-interference performance, and can explore the distributed architecture and cross-layer optimization, consider the influence of same frequency transmission to enhance the practicality and expansibility of the algorithm, and provide a more robust solution for intelligent wireless communication.

Claims

1. An implementation method of a deep reinforcement learning based anti-jamming system for ad hoc networks, characterized in that, The method comprises the following steps: Step 1: Construct a simulation network environment based on the interface between the Network Simulator 3 (NS-3) simulation platform and OpenAI Gym, which is used for real-time collection of channel states and feedback to the agent; Step 2: After receiving the network state, the agent feeds back the decision result learned to the network node to guide the channel switching in real time; Step 3: Fuse the Long Short-Term Memory (LSTM), Dueling Network, Double Deep Q-Network (Double DQN) and Prioritized Experience Replay (PER) mechanism, and efficiently predict the interference timing law through the PER-D3RQN algorithm.

2. The implementation method of the anti-jamming system for ad hoc networks based on deep reinforcement learning according to claim 1, characterized in that, The step 1 comprises: The multi-channel anti-interference allocation problem is modeled as a reinforcement learning task, and a joint simulation platform based on the NS-3 network simulator and the OpenAI Gym interface is constructed. The platform realizes seamless integration of C++ and python programming environments through the ns3-gym middleware. NS-3 is responsible for simulating the physical layer and protocol stack characteristics of the network, including node topology, spectrum propagation model, 802.11 protocol and interference mechanism. The reinforcement learning algorithm is implemented in the OpenAI Gym environment, and the two are interacted in real time through the socket interface based on ZMQ for state observation and action execution; The environment definition of reinforcement learning comprises: (1) Environment: Based on IEEE 802.11a protocol, divided into n non-overlapping channels in 5GHz frequency band, each channel bandwidth is 20MHz, channel index is 0 to n-1, compared with 2.4GHz frequency band, 802.11a protocol provides more non-overlapping channel resources in 5GHz frequency band, which helps to reduce co-channel interference and improve spectrum utilization efficiency, expressed as a set: M={0,1,2,...,n-1} (1) Each node can freely select a channel for communication within the above set; The interference node simulates the rapid and complex interference mode in reality, which is manifested as that part of the channels will be blocked due to interference in each time slot, and the channel state can be divided into normal (available) or interfered (unavailable); where P 10 represents the probability that a channel in the "disturbed" state will change to the "normal" state in the next time slot. P 11 denotes the probability that a channel in the "disturbed" state continues to be in the "disturbed" state, P 10 and P 11 are complementary and sum to 1, and in correspondence, P 00 and P 01 denote the probabilities that a channel in the "normal" state continues to be normal and disturbed, respectively, in the next time slot. (2) State: In this simulation environment, the power spectral density of the i-th channel at frequency f and time slot t is defined as: PSD i (f,t) = N0+ S i (f) (2) where N0is the background noise spectral density, taken as 1 x 10 -15 W / Hz, S i (f) is the flat transmit spectral density of the interferer on the channel, calculated as: where P int,i is the total transmit power of the i-th interferer, B i denotes the channel bandwidth, 20 MHz according to the IEEE 802.11a protocol, and the total received power of the i-th channel at time slot t is obtained by integrating (or summing in the discrete case) the power spectral density over the bandwidth B i : where f k is the discrete sampling frequency point, Δf is the frequency resolution, and the linear power P i (t) is converted to dBW: P i,dB (t) = 10 log 10 (P i (t)) (5) CCA (Clear Channel Assessment) threshold θ CCA According to the provision of IEEE 802.11a protocol, -92dBW (i.e. -62dBm energy detection threshold), the occupation state of the i-th channel is:

3. The implementation method of the anti-jamming system for ad hoc networks based on deep reinforcement learning according to claim 1, characterized in that, The step 2 comprises: All channel states are spliced into a column vector with a length of n as the observation input of the agent at time slot t, which reflects whether each channel is available at the current moment, and the agent judges which channels are interfered to avoid occupied channels and select idle channels for data transmission: s(t) = [s1(t), s2(t),...,s n (t)] T (7) (2) Action: The action space A is a discrete space, which represents the channel combination that the agent can select at each time slot, and the size of the action space is determined by the following formula: |A|=C(n,k) (8) Where: n represents the total number of available channels, k represents the number of channels to be selected each time, C(n, k) represents the number of combinations of selecting k channels from n channels, and each action a e A corresponds to a channel combination, which can be represented as: a = {c1, c2,..., c k} (9) where c i represents the selected i-th channel number; (3) Reward: The design of the reward function R takes into account the channel occupation state: where r i represents the immediate reward of the ith selected channel: O(c i ) denotes the occupancy status of channel c i (0 for free, 1 for occupied), a positive reward +1.0 is obtained when a free channel is chosen, a negative reward -1.0 is obtained when an occupied channel is chosen; Through the above model and definition, the goal of the agent is to obtain the perception information of the state of each channel in the current time slot at the beginning of each time slot, and to predict the state change of the next time slot based on this information to maximize the long-term cumulative reward R.

4. The implementation method of the anti-jamming system for ad hoc networks based on deep reinforcement learning according to claim 1, characterized in that, The step 3 comprises: The PER-D3RQN algorithm is designed to deal with the characteristics of time-varying interference state and incomplete observation in a multi-channel environment. This algorithm integrates the following key technical components: (1) LSTM time series modeling: To address the problem of partial observability of channel state in ad hoc network environment, LSTM is introduced to capture the time series characteristics of interference. The core calculation process is: f t = σ(W f · [h t-1 , x t ]+ b f )(12) where f t represents the forget gate output, which determines how much information from the previous state to retain; σ is the sigmoid activation function; W f is the forget gate weight matrix; h t-1 is the hidden state at the previous time; x t is the input at the current time; b f is the bias term. LSTM effectively extracts the time correlation of channel interference through the memory mechanism, enhancing the prediction ability of future interference states. (2) Dueling network structure: The Q value is decomposed into state value function V(s) and advantage function A(s, a): Where Q(s, a) is the state-action value function, V(s) measures the intrinsic value of state s, A(s, a) evaluates the relative advantage of selecting action a in state s, and |A| is the size of the action space. This decomposition allows the model to learn the value of the state without changing the current policy, improving learning efficiency and stability. (3) Double DQN mechanism: By separating action selection and evaluation, the problem of Q value overestimation is alleviated: y i = r i + γQ(s i+1 , a'; θ); θ a′ Q(s i+1 , a'; θ); θ - )(14) where y i is the target Q value; r i is the immediate reward; γ is the discount factor; θ is the current network parameters; θ - is the target network parameters, the current network is used to select actions, while the target network evaluates the value of the action, reducing the bias caused by over-optimistic estimation; (4) Prioritized experience replay: Compared with traditional uniform random sampling, prioritized experience replay determines the importance of samples based on their TD error size, prioritizes learning of high-information transition samples, and prioritizes experience replay based on TD error to determine sample sampling probability, accelerating the learning of key transitions: p i = |δ i |+ε (15) where p i is the priority of sample i; δ i is the TD error; ε is a small positive number to prevent priority from being zero; P(i) is the sampling probability; α is a control parameter for the degree of priority; and w i is the importance sampling weight introduced to correct the bias caused by non-uniform sampling. Where N is the buffer size, and β increases from the initial value to 1 as the training progresses. This mechanism makes the network focus more on high-information samples, improving learning efficiency and convergence speed.

Images