Anti-jamming zero-sum and markov game model and max-min deep q-learning method

Abstract

This invention discloses an anti-jamming zero-sum Markov game model and a maximum-minimum deep Q-learning method. The model considers a communication adversarial scenario involving a pair of transceivers, a fixed jammer, and an intelligent jammer. The fixed jammer releases frequency sweeping interference, the intelligent jammer optimizes the jamming channel selection, and the user optimizes channel and power selection to maximize transmission utility. The adversarial interaction process is modeled as an anti-jamming zero-sum Markov game. The method involves: initializing the anti-jamming network parameters and training hyperparameters; constructing a spectrum waterfall plot and inputting it into the anti-jamming network, outputting an adversarial Q-value matrix; calculating and executing anti-jamming actions, calculating the reward value obtained for the current action and saving the adversarial record; sampling the adversarial record, calculating the Q-value estimation error, and updating the anti-jamming network using a backpropagation gradient algorithm; repeating the above interaction process until network training is complete. This invention effectively solves the problem of non-stationary environmental changes caused by jamming strategy updates, obtaining a more robust anti-jamming strategy.

Description

Technical Field

[0001] This invention belongs to the field of wireless communication technology, specifically an anti-interference zero-sum Markov game model and a maximum-minimum depth Q-learning method. Background Technology

[0002] Due to the openness of wireless channels, legitimate users' communications in wireless communication networks are vulnerable to interference attacks from malicious users. Therefore, communication anti-jamming technology plays a crucial role in both civilian and military communications. However, traditional communication anti-jamming methods, such as frequency hopping spread spectrum and direct sequence spread spectrum, have fixed modes and are difficult to effectively cope with dynamic interference. To address this, in recent years, researchers have continuously proposed intelligent anti-jamming technologies based on machine learning. Empowered by artificial intelligence algorithms, legitimate users can learn and uncover patterns in interference changes, thereby adopting efficient and reliable communication methods. Existing research has applied deep reinforcement learning methods to the field of anti-jamming, allowing users to obtain optimal access strategies through interactive learning with the interference environment without needing prior interference information (X. Liu et al., “Anti-jamming communications using spectrum waterfall: A deep reinforcement learning approach,” IEEE Communications Letter, vol. 22, no. 5, pp. 998–1001, 2018.). Similarly, existing literature has applied deep reinforcement learning methods to UAV video transmission, effectively evading interference attacks and improving the intelligence of video transmission by optimizing coding, modulation, power, and channel (L. Xiao et al., “UAV anti-jamming videotransmissions with QoE guarantee: a reinforcement learning-based approach,” IEEE Transactions on Communications, vol. 69, no. 9, pp. 5933-5947, 2021.). Going further, existing literature has considered intelligent anti-jamming methods in multi-user scenarios, using the joint Q-value of all users for multi-user decision-making (Q. Zhou et al., “Intelligent Anti-Jamming Communication for Wireless Sensor Networks: A Multi-Agent Reinforcement Learning Approach,” IEEE Open Journal of the Communications Society, vol. 2, pp. 775-784, 2021.). However, previous studies have mostly considered that interference lacks intelligence and that interference attacks exhibit obvious regularities. With the continuous development of wireless and artificial intelligence technologies, interference capabilities are also constantly being strengthened. They will also have environmental awareness and policy update capabilities, possessing intelligence comparable to that of the communicating party.Therefore, the intelligent anti-interference methods that previously considered fixed-pattern interference will fail or even be completely suppressed when facing intelligent interference.

[0003] Currently, some research has begun to consider anti-intelligent jamming. Existing studies have considered anti-jamming scenarios for UAV-assisted mobile networks, assuming that jamming uses Q-learning to update its strategy, and designed a UAV relay power selection method based on deep reinforcement learning, effectively reducing transmission error rate and energy consumption (X. Lu et al., “UAV-aided cellular communications with deep reinforcement learning against jamming,” IEEE Wireless Communications, vol. 27, no. 4, pp. 48-53, 2020.). In addition, some research has considered jamming UAVs based on deep reinforcement learning to intelligently jam communication UAVs by observing their trajectories, and designed adversarial algorithms based on deep reinforcement learning to evade jamming UAV attacks (N. Gao et al., “Anti-intelligent UAV jamming strategy via deep Q-networks,” IEEE Transactions on Communications, vol. 68, no. 1, pp. 569-581, 2019.). However, previous studies, considering the relatively weak capabilities of the adversary, did not delve into the characteristics of intelligent communication adversarial mechanisms. Instead, they directly treated intelligent interference as the environment, ignoring the non-stationarity of environmental state changes caused by the dynamic updates of interference strategies, thus violating the stationarity assumption of single-user reinforcement learning convergence. Further consideration of the characteristics of intelligent adversarial mechanisms is needed in the design of anti-interference algorithms.

[0004] In summary, existing research on intelligent anti-jamming is insufficient to effectively counter intelligent jamming with strategy update capabilities. The main problems are as follows: 1) Most existing intelligent anti-jamming methods consider relatively fixed and simple jamming patterns and do not take into account the intelligence of the jamming; 2) Existing research on intelligent jamming has not deeply analyzed the characteristics of intelligent countermeasures, but simply treats intelligent jamming as part of the environment, ignoring the non-stationary changes in the environment caused by the update of jamming strategies, making it difficult to gain an advantage in communication confrontation. Summary of the Invention

[0005] The purpose of this invention is to address the problems existing in the prior art by providing an anti-interference Markov game model and a maximum-minimum depth Q-learning method to evade intelligent interference attacks and effectively improve user transmission rates.

[0006] The technical solution to achieve the purpose of this invention is as follows: On the one hand, an anti-interference zero-sum Markov game model is provided. In this model, both the user and the intelligent interference have the ability to observe the environment and update the strategy. Both parties observe the environmental state and make decisions to change the environmental state. The user's goal is to maximize the transmission utility, while the intelligent interference has the completely opposite goal. The sum of the utilities of both parties is zero.

[0007] On the other hand, a method for resisting intelligent interference based on maximum-minimum depth Q-learning is provided, the method comprising the following steps:

[0008] Step 1: Model the anti-interference problem under the threat of intelligent interference as the anti-interference zero-sum Markov game model. The participants in the game are the user and the intelligent interference. The user's optimization objective is to obtain the optimal anti-interference strategy that is the worst interference strategy, which corresponds to the Nash equilibrium strategy.

[0009] Step 2: The user constructs an anti-interference decision network and randomly initializes the network parameters. Simultaneously, the user sets the hyperparameters for network training, including the learning rate α, discount factor γ, exploration probability ε0, and target network update step size N. T Experience replay unit

[0010] Step 3: The user perceives the real-time environmental spectrum status and constructs a spectrum waterfall plot from the historical perception data. t The spectrum waterfall plot is input into the anti-interference decision network to calculate and output the current state Q-value matrix Q(s). t ,·|θ), where θ represents the decision network parameters, and then the current anti-interference action is calculated using an ε-greedy strategy and linear programming. And carry out joint actions; among them, and p t These are the user communication frequency and the transmission power, respectively.

[0011] Step 4: Calculate the reward value r for the current action. t And obtain interference actions through sensing data. t And construct the spectral state s t+1 , will record the interaction t =(s t ,a t ,o t ,r t ,s t+1 Add to the experience replay unit.

[0012] Step 5, from the experience replay unit Randomly select training sample set Where (s) i ,a i,o i ,r i ,s′ i ) represent the current state s in the i-th training sample. i Anti-interference action a i Interference actions i Return value r i And the next state s′ i Then, the Q-value estimation error value L(θ) is calculated, and then the anti-interference decision network is updated using the gradient descent method.

[0013] Step 6: Repeat steps 3 through 5 until the specified number of iterations is reached.

[0014] Furthermore, in step 1, the model is a robust zero-sum Markov game model, specifically:

[0015] Anti-interference zero-sum Markov game using six-tuples Indicates; among which, The environmental state is a set of environmental states, defined as a spectrum waterfall plot, which contains three-dimensional information of time, frequency, and power. This represents a set of user actions, where the user selects a joint channel power decision for anti-interference communication. This represents a set of interference actions; the interference selects a channel to disrupt user transmissions. This represents the state transition function, which indicates the probability of transitioning to the next state under the influence of user actions and disturbances. The reward value function is defined as r. t =C t -ωp t C t ω represents the transmission rate, p represents the cost factor, and ω represents the transmission rate. t Represents user power; γ represents the discount factor;

[0016] Considering the worst-case interference scenario, the user's optimization goal is to obtain the optimal anti-interference strategy π. * To maximize the cumulative discount return in the future:

[0017]

[0018] in Indicates that in state s t Under the following conditions, when the user and the disturbance adopt policies π and μ respectively, the user's future cumulative discount reward value, and the state change of s follows a transition function. This indicates the calculation of the expected value, where γ is the discount factor and r is the value. t+i This represents the reward value at time t+i; in a zero-sum random game, the above objective corresponds to finding the Nash equilibrium strategy.

[0019] Further, in step 2, the anti-interference decision network comprises a two-layer convolutional network and a three-layer fully connected network. The convolutional layers extract useful features from the spectral waterfall plot, and the fully connected layers integrate the feature values ​​and compute the Q-value matrix. The first convolutional layer contains f1 convolutional kernels of size z1×z1 with a stride of d1. The second convolutional layer contains f2 convolutional kernels of size z2×z2 with a stride of d2. The number of neurons in the first and second fully connected layers are n1 and n2, respectively. The number of neurons in the last fully connected layer of the anti-interference decision network is...

[0020] Furthermore, in step 3, the historical sensing data is constructed into a spectral waterfall plot. t Specifically:

[0021] The instantaneous spectrum data obtained by the user at time t is o. t =[o1,o2,…,o L ], where L=(f U -f L ) / Δf is the number of sampling points, f U f is the upper limit of frequency. L The lower limit of frequency is given by Δf, which represents the frequency resolution; the formula for calculating the i-th sample value is: Where S(f) is the power spectral density function of the user's received signal, expressed as:

[0022]

[0023] Where h1, h2, and h3 represent the channel gains from the user transmitter, intelligent jammer, and fixed jammer to the user receiver, respectively. The power spectral density function representing the user signal. Indicates the frequency of the user center. The power spectral density function of the intelligent interference signal is represented by M, which represents the number of channels covered by the interference signal. This represents the center frequency of the m-th channel covered by intelligent interference. This represents the power spectral density function for a fixed interference signal. Let n(f) represent the center frequency of the channel with fixed interference coverage, and n(f) represent the power spectral density function of the ambient noise.

[0024] The spectrum waterfall plot is constructed from historical spectrum data and is represented as s. t =[o t ,o t-1 ,…,o t-Φ+1 ] T , where Φ is the length of historical data.

[0025] Furthermore, step 3 involves using an ε-greedy strategy and linear programming to calculate the current anti-interference action. And perform joint actions, specifically:

[0026] The user uses an ε-greedy strategy to select actions, which uses ε t The probability of randomly selecting an action is 1-ε. t The probability is calculated based on the Q-value matrix to determine the equilibrium strategy. Then, actions are sampled according to the equilibrium strategy. Where ε t The update rule is ε t =ε f +(ε0-ε f )e -t / v Where ε0 is the initial value, ε f The final value is given by v, where v is the fading coefficient.

[0027] After calculating the Q-value matrix Q(s) t After ,·|θ), the anti-interference equilibrium strategy can be calculated through linear programming. Right now:

[0028]

[0029] Where π(a|s) t ) indicates that the user is in state s t The probability of choosing action 'a' is assumed to be... Then for any disturbance action o, the following condition is satisfied: The above formula can then be transformed into:

[0030]

[0031] Further, step 4 involves calculating the reward value r under the current action. t And obtain interference actions through sensing data. t And construct the spectral state s t+1 , will record the interaction t =(s t ,a t ,o t ,r t ,s t+1 Add to the experience replay unit. Interaction record e t =(s t ,a t ,o t ,r t ,s t+1 Add to the experience replay unit Specifically:

[0032] Intelligent jamming inputs the obtained spectral waterfall plot into the jamming DQN network and selects jamming actions according to an ε-greedy approach. t ;

[0033] After the user and the interference each perform their respective actions, the user calculates the reward value r at the current moment. t =C t -ωp t ;

[0034] The next state s is obtained by sensing the spectrum. t+1 ;

[0035] Record this interaction e t =(s t ,a t ,o t ,r t ,s t+1 Stored in the experience replay unit.

[0036] Further, step 5 involves calculating the Q-value estimation error L(θ), followed by updating the anti-interference decision network using the gradient descent method, specifically as follows:

[0037] Randomly sample B training samples {(s i ,a i ,o i ,r i ,s i ′)} i∈[B] Used for network updates;

[0038] For each training sample, the state s i The estimated Q value Q(s) is obtained by inputting it into the anti-interference decision network. i ,a i ,o i |θ);

[0039] The next state s′ i The input is fed into the target value network to obtain the Q-value matrix Q(s). i ′,a i ,o i |θ - ), where θ - The network parameters represent the target value, and the target Q value y is then calculated. i for:

[0040]

[0041] in The Q-value matrix Q(s′) i ,·|θ - User balancing strategies under ( );

[0042] Calculate the estimation error L(θ):

[0043]

[0044] Update network parameters using gradient descent:

[0045]

[0046] Compared with the prior art, the significant advantages of this invention are:

[0047] (1) It can counter intelligent interference with strategy update capability, gain an advantage in intelligent communication confrontation, and effectively improve the user's anti-interference effectiveness.

[0048] (2) No interference or channel prior information is required. Users can continuously optimize their strategies by interacting with the spectrum environment.

[0049] The present invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0050] Figure 1 This is a schematic diagram illustrating the anti-smart interference method based on maximum and minimum depth Q-learning of the present invention.

[0051] Figure 2 This is a framework diagram of the anti-intelligent interference method based on maximum and minimum depth Q-learning of this invention.

[0052] Figure 3 This is a schematic diagram illustrating the convergence performance of the proposed method and the comparison algorithm in the embodiments of the present invention.

[0053] Figure 4 This is a schematic diagram illustrating the performance test of the proposed method and the comparison algorithm in the embodiments of the present invention. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0055] In one embodiment, the present invention proposes an anti-interference zero-sum Markov game model and a maximum-minimum deep Q-learning method to jointly optimize the selection of transmission power and communication channel under the threat of intelligent interference.

[0056] Combination Figure 1In a typical communication anti-jamming scenario consisting of intelligent jammers, fixed jammers, and communication transceiver pairs, the transceiver pairs transmit and receive data through the transmission link. The jammers reduce the received signal-to-noise ratio by releasing wireless interference attacks, thereby disrupting the normal communication of the transceiver pairs. Therefore, the transceiver pairs need to optimize and adjust their own power and channels in real time to avoid interference attacks and ensure reliable data transmission.

[0057] This invention proposes an anti-jamming zero-sum Markov game model, considering an intelligent communication adversarial scenario involving a pair of transceivers (users), a fixed jammer, and an intelligent jammer. The fixed jammer releases frequency sweeping interference, the intelligent jammer uses deep reinforcement learning to optimize the jamming channel selection, and the user optimizes the channel and power selection to maximize transmission utility. The adversarial interaction process is modeled as an anti-jamming zero-sum Markov game model.

[0058] This invention aims to jointly optimize transmission power and communication channels by using intelligent learning algorithms to allow users to continuously interact with the interference environment in order to find the optimal communication parameters.

[0059] The anti-smart interference method based on maximum-minimum depth Q-learning proposed in this invention includes the following steps:

[0060] Step 1: Model the anti-interference problem under the threat of intelligent interference as an anti-interference zero-sum Markov game model. The participants in the game are the user and the intelligent interference. Consider that the intelligent interference is perfectly rational. The user's goal is to obtain the optimal anti-interference strategy with the worst interference strategy, which corresponds to the Nash equilibrium strategy.

[0061] Here, the model is a robust zero-sum Markov game model, specifically:

[0062] Anti-interference zero-sum Markov game using six-tuples It indicates. Among them, The environmental state is a set of environmental states, defined as a spectrum waterfall plot, which contains three-dimensional information of time, frequency, and power. It not only reflects the changes in spectrum state, but also provides sufficient information for anti-interference decision-making. This represents a set of user actions, where the user selects a joint channel power decision for anti-interference communication. This represents a set of interference actions; the interference selects a channel to disrupt user transmissions. This represents the state transition function, which indicates the probability of transitioning to the next state under the influence of user actions and disturbances. The reward value function is defined as r. t =C t -ωp t C t ω represents the transmission rate, p represents the cost factor, and ω represents the transmission rate. t γ represents the user's power; γ represents the discount factor.

[0063] Considering the worst-case scenario of interference, the user's optimization goal is to find the optimal anti-interference strategy π. * To maximize the cumulative discount return in the future:

[0064]

[0065] in Indicates that in state s t Under the following conditions, when the user and the disturbance adopt policies π and μ respectively, the user's future cumulative discount reward value, and the state change of s follows a transition function. This indicates the calculation of the expected value, where γ is the discount factor and r is the value. t+i This represents the reward value at time t+i; in a zero-sum random game, the above objective corresponds to finding the Nash equilibrium strategy.

[0066] Step 2: The user constructs an anti-interference decision network and randomly initializes the network parameters, while setting the hyperparameters for network training, including the learning rate, discount factor, and exploration probability.

[0067] Here, since the input to the anti-interference decision network is a two-dimensional spectral waterfall plot, a neural network model consisting of convolutional layers and fully connected layers is used. The convolutional layers extract useful features from the two-dimensional spectral waterfall plot and reduce computational complexity, while the fully connected layers integrate feature values ​​and compute the Q-value matrix. The designed anti-interference decision network comprises two convolutional layers and three fully connected layers. The first convolutional layer contains f1 convolutional kernels of size z1×z1 with a stride of d1. The second convolutional layer contains f2 convolutional kernels of size z2×z2 with a stride of d2. The number of neurons in the first and second fully connected layers are n1 and n2, respectively. The number of neurons in the last fully connected layer of the anti-interference decision network is...

[0068] Randomly initialize the parameters of the anti-interference network, including the discount factor γ, learning rate α, exploration probability ε0, and target network update step size N. T Experience replay unit

[0069] Step 3: The user perceives the real-time environmental spectrum status and constructs a spectrum waterfall plot from the historical perception data. t The spectral waterfall plot is input into the anti-interference decision network, and the current state Q-value matrix Q(s) is calculated and output. t Then, using an ε-greedy strategy and linear programming, the current anti-interference action is calculated. And carry out joint actions.

[0070] Here, historical sensing data is constructed into a spectrum waterfall plot.t Specifically:

[0071] The instantaneous spectrum data obtained by the user at time t is o. t =[o1,o2,…,o L ], where L=(f U -f L ) / Δf is the number of sampling points, f U f is the upper limit of frequency. L The lower limit of frequency is given by Δf, which represents the frequency resolution; the formula for calculating the i-th sample value is: Where S(f) is the power spectral density function of the user's received signal, expressed as:

[0072]

[0073] Where h1, h2, and h3 represent the channel gains from the user transmitter, intelligent jammer, and fixed jammer to the user receiver, respectively. The power spectral density function representing the user signal. Indicates the frequency of the user center. The power spectral density function of the intelligent interference signal is represented by M, which represents the number of channels covered by the interference signal. This represents the center frequency of the m-th channel covered by intelligent interference. This represents the power spectral density function for a fixed interference signal. Let n(f) represent the center frequency of the channel with fixed interference coverage, and n(f) represent the power spectral density function of the ambient noise.

[0074] The spectrum waterfall plot is constructed from historical spectrum data and is represented as s. t =[o t ,o t-1 ,…,o t-Φ+1 ] T , where Φ is the length of historical data.

[0075] Here, the current anti-interference action is calculated using an ε-greedy strategy and linear programming. And perform joint actions, specifically:

[0076] Users select actions using an ε-greedy approach, which uses ε t The probability of randomly selecting an action is 1-ε. t The probability is calculated based on the Q-value matrix to determine the equilibrium strategy. Then the sampling action To enhance action exploration, this invention considers that random exploration decays with the number of iterations, ε t The update rule is ε t =ε f +(ε0-ε f )e-t / v Where ε0 is the initial value, ε f is the final value, and v is the fading coefficient.

[0077] After calculating the Q-value matrix Q(s) t After ,·|θ), the anti-interference equilibrium strategy can be calculated through linear programming. Right now:

[0078]

[0079] Where π(a|s) t ) indicates that the user is in state s t The probability of choosing action 'a' is assumed to be... Then for any disturbance action o, the following condition is satisfied: The above formula can then be transformed into:

[0080]

[0081] Step 4: Calculate the reward value r for the current action. t And obtain interference actions through sensing data. t And construct the spectral state s t+1 , will record the interaction t =(s t ,a t ,o t ,r t ,s t+1 Add to the experience replay unit Specifically:

[0082] Intelligent jamming inputs the obtained spectral waterfall plot into the jamming DQN network and selects jamming actions according to an ε-greedy approach. t After the user and the interference each perform their respective actions, the user calculates the reward value r at the current moment. t =C t -ωp t Then, the state s of the next time step is obtained by sensing the spectrum. t+1 Finally, record this interaction e t =(s t ,a t ,o t ,r t ,s t+1 Stored in the experience replay unit middle.

[0083] Step 5, from the experience replay unit Randomly select training sample set The Q-value estimation error L(θ) is calculated, and then the robust decision network is updated using the gradient descent method. Specifically:

[0084] Randomly sample B training samples {(s i ,a i ,o i ,r i ,s′ i )} i∈[B] Used for network updates. For each training sample, the state s i The estimated Q-value Q(s) can be calculated by inputting it into the policy network. i ,a i ,o i |θ). The next state s′ i The Q-value matrix Q(s′) can be calculated by inputting it into the target value network. i ,a i ,o i |θ - ), where θ - The network parameters represent the target value, and the target Q value y is then calculated. i for:

[0085]

[0086] in The Q-value matrix Q(s′) i ,·|θ - The user equilibrium strategy is then applied. Finally, the estimation error L(θ) is calculated, and the network parameters are updated using gradient descent. The error is calculated as follows:

[0087]

[0088] Step 6: Repeat steps 3 through 5 until the specified number of iterations is reached.

[0089] As a specific example, the invention will be further described and verified in detail in one embodiment.

[0090] In this embodiment, the system simulation uses the PyTorch deep learning framework, and the network training runs on an RTX 2080Ti. The parameter settings do not affect generality. The simulation communication frequency band is set to 830MHz-850MHz, evenly divided into 10 non-overlapping channels, each with a bandwidth of 2MHz. The user communication time slot length is set to 10ms. The perceptron's frequency sampling interval is 0.1MHz, with 200 sampling points per sampling. The spectral waterfall plot backtracking length is set to 200, resulting in a 200*200 matrix. Interference and user baseband signals are simulated using a raised cosine roll-off filter with a roll-off factor of 0.5. The user's maximum transmit power is 200mW (23dBm), divided into 10 power levels, with a receiver signal-to-noise ratio threshold of 5dB. A fixed jammer releases sweeping interference with a bandwidth of 4MHz, cyclically sweeping from the start to the end of the band at a speed of 500MHz / s, with a transmit power of 50dBm. An intelligent jammer transmits with a power of 50dBm, an interference signal bandwidth of 6MHz, and can cover 3 channels. The background noise power is -90dBm. The deep reinforcement learning algorithm uses a learning rate of 1e-4, a batch size of 32, and an experience pool size of 10000. The first convolutional layer contains 16 4x4 convolutional kernels with a stride of 2, the second convolutional layer contains 32 4x4 convolutional kernels with a stride of 2, and the three fully connected layers contain 512, 256, and 1000 neurons respectively. The following three comparison algorithms are mainly considered:

[0091] 1) DQN-based method: This method directly treats interference as part of the environment, without conducting in-depth modeling and analysis of the intelligent communication countermeasure process, and uses the single-user DQN method to learn user patterns.

[0092] 2) Perception-based access method: This method performs spectrum access according to fixed rules. Based on the current spectrum perception results, the user accesses the channel with the lowest current signal energy value, while the power remains unchanged.

[0093] 3) Random selection method: The user randomly selects an action from the action set for each time slot.

[0094] Figure 2 The framework diagram of the proposed method is given. At each time step, the user perceives the spectrum environment, then constructs a spectrum waterfall plot and inputs it into the decision network, outputting a joint decision on channel and power. Then, a batch of training samples is randomly selected from the memory replay unit, the network loss value is calculated, and the network is updated.

[0095] Figure 3The performance of different algorithms in online adversarial against DQN interference was compared. During online communication adversarial operations, both the user and the interference continuously update their strategies. The average reward value on the vertical axis of the graph is the average of the reward values ​​over 100 time slots. Each curve represents the result of five repeated trials. The shaded area in the graph represents the 95% confidence interval, indicating the range of algorithm performance fluctuations. As shown in the graph, the average reward value of the perceptual access-based method gradually decreases with increasing iterations. This is because the perceptual access method is an anti-interference method based on fixed rules, making it easy for DQN interference to learn its changing patterns and thus suppress it through intelligent interference. Since the random selection-based method has no changing patterns, DQN interference has difficulty learning effective countermeasures, and its anti-interference performance remains stable. However, due to the existence of fixed-pattern interference, the random selection method will randomly select the interference channel, resulting in a lower reward value and no performance improvement. Both the single-user DQN-based method and the proposed method can improve their anti-interference strategies through interaction with the environment; therefore, the average reward of both increases with increasing iterations. However, compared to the DQN method, the proposed method achieves approximately a 15% performance improvement.

[0096] Figure 4 The stability of the anti-interference model was tested. Exploitability is an important indicator for evaluating the stability of a trained model, representing how easily an adversary can learn and exploit it. Higher exploitability indicates that the anti-interference model is more vulnerable to attack, and intelligent interference can more easily learn its changing patterns; conversely, lower exploitability indicates that the model is more robust, and intelligent interference has difficulty learning effective adversarial strategies. This invention compares the performance of different anti-interference models when facing DQN interference with online policy updates. The anti-interference model keeps its network parameters unchanged. As shown in the figure, when facing DQN interference, except for the random selection method which maintains unchanged performance, the performance of the other methods decreases with the number of iterations. This is because the DQN interference improves its interference strategy through interaction. The single-user DQN method eventually drops to the same performance level as the perceptual algorithm, indicating that its obtained anti-interference model has high exploitability and is completely suppressed when facing intelligent interference. However, the proposed method's performance decline rate is significantly slower than the other methods, and it still maintains a high average reward value, indicating that the anti-interference model obtained by the proposed method has lower exploitability.

[0097] Comparative analysis revealed that the intelligent adversarial algorithm proposed in this invention outperforms the traditional DQN anti-interference algorithm, achieving an anti-interference effectiveness improvement of approximately 15%, and its performance is more stable during dynamic adversarial processes.

[0098] In summary, the maximum-minimum depth Q-learning method proposed in this invention optimizes user transmission power and channel. By considering the optimal anti-interference strategy under the worst interference conditions, it converges to the adversarial game equilibrium solution. Users do not need prior information about interference and channel during the decision-making process; they only need to interact with the interference environment to obtain a robust anti-interference strategy.

[0099] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention without departing from its spirit and scope should be included within the protection scope of the present invention.

Claims

1. A method for resisting intelligent interference based on maximum-minimum depth Q-learning, characterized in that, The method is based on an anti-interference zero-sum Markov game model, in which both the user and the intelligent interference have the ability to observe the environment and update the strategy. Both parties observe the environmental state and make decisions to change the environmental state. The user's goal is to maximize the transmission utility, while the intelligent interference has the completely opposite goal. The sum of the utilities of both parties is zero. The method includes the following steps: Step 1: Model the anti-interference problem under the threat of intelligent interference as the anti-interference zero-sum Markov game model. The participants in the game are the user and the intelligent interference. The user's optimization objective is to obtain the optimal anti-interference strategy that is the worst interference strategy, which corresponds to the Nash equilibrium strategy. Step 2: The user constructs an anti-interference decision network and randomly initializes the network parameters, while also setting the hyperparameters for network training, including the learning rate. Discount Factor Exploring Probability Target network update step size Experience replay unit ; Step 3: The user perceives the real-time environmental spectrum status and constructs a spectrum waterfall chart from the historical perception data. The spectrum waterfall plot is input into the anti-interference decision network to calculate and output the current state Q-value matrix. ,in Represent the decision network parameters, and then use - Greedy strategy and linear programming are used to calculate the current anti-interference action. And carry out joint actions; among them, and These are the user communication frequency and the transmission power, respectively. Step 4: Calculate the reward value for the current action. And obtain interference actions through sensing data. And construct the spectrum state Record the interaction Add to the experience replay unit ; Step 5, from the experience replay unit Randomly select training sample set ,in They represent the first The current state in each training sample Anti-interference actions Interference actions Return value and the next state Then calculate the Q-value estimation error. Then, the anti-interference decision network is updated using the gradient descent method; Step 6: Repeat steps 3 through 5 until the specified number of iterations is reached.

2. The anti-smart interference method based on maximum-minimum depth Q-learning according to claim 1, characterized in that, In step 1, the model is a robust zero-sum Markov game model, specifically: Anti-interference zero-sum Markov game using six-tuples Indicates; among which, The environmental state is a set of environmental states, defined as a spectrum waterfall plot, which contains three-dimensional information of time, frequency, and power. This represents a set of user actions, where the user selects a joint channel power decision for anti-interference communication. This represents a set of interference actions; the interference selects a channel to disrupt user transmissions. This represents the state transition function, which indicates the probability of transitioning to the next state under the influence of user actions and disturbances. The reward value function is defined as follows: ,in Indicates the transmission rate. Represents the cost factor. Indicates user power; This represents the discount factor; Considering the worst-case scenario of interference, the user's optimization goal is to obtain the optimal anti-interference strategy. To maximize the cumulative discount return in the future: in Indicates the state Below, strategies are adopted for users and interference respectively. and At that time, the user's future cumulative discount reward value, status The change follows a transfer function , This indicates the calculation of the expected value. As a discount factor, Indicates in The reward value at time; in a zero-sum random game, the above objective corresponds to finding the Nash equilibrium strategy of the game.

3. The anti-smart interference method based on maximum-minimum depth Q-learning according to claim 1, characterized in that, In step 2, the anti-interference decision network consists of two convolutional layers and three fully connected layers. The convolutional layers are used to extract useful features from the spectral waterfall plot and reduce computational complexity, while the fully connected layers are used to integrate feature values ​​and compute the Q-value matrix. The first convolutional layer includes... There are n convolutional kernels, and the kernel size is [value missing]. Step size is The second convolutional layer includes There are n convolutional kernels, and the kernel size is [value missing]. Step size is The number of neurons in the first and second fully connected layers are respectively and The number of neurons in the last fully connected layer of the anti-interference decision network is... .

4. The anti-smart interference method based on maximum-minimum depth Q-learning according to claim 1, characterized in that, In step 3, historical sensing data is used to construct a spectrum waterfall plot. Specifically: User at any time The instantaneous spectrum data obtained by sensing and sampling is ,in The number of sampling points. This is the upper limit of frequency. This is the lower limit of frequency. This represents the frequency resolution; the formula for calculating the i-th sample value is: ,in The power spectral density function of the user's received signal is expressed as: in, , and These represent the channel gains from the user transmitter, intelligent jammer, and fixed jammer to the user receiver, respectively. The power spectral density function representing the user signal. Indicates the frequency of the user center. The power spectral density function representing the intelligent interference signal. This indicates the number of channels covered by the interference signal. The first intelligent interference coverage The center frequency of each channel This represents the power spectral density function for a fixed interference signal. Indicates the center frequency of the fixed interference coverage channel. The power spectral density function representing environmental noise; A spectrum waterfall plot is constructed from historical spectrum data and is represented as follows: ,in The length of the historical data.

5. The anti-smart interference method based on maximum-minimum depth Q-learning according to claim 1, characterized in that, Step 3 describes the use of - Greedy strategy and linear programming are used to calculate the current anti-interference action. And perform joint actions, specifically: User utilization - Greedy strategy selects actions, which are based on The probability of randomly selecting an action, in order to The probability is calculated based on the Q-value matrix to determine the equilibrium strategy. Then, actions are sampled according to the balancing strategy. ;in The update rule is as follows ,in As the initial value, For the final value, The fading coefficient; After calculating the Q-value matrix Then, an anti-interference equilibrium strategy can be calculated using linear programming. ,Right now: in Indicates the user's state Select action The probability, assuming Then for any interference action All satisfy Then the above formula can be transformed into: （4）。 6. The anti-smart interference method based on maximum-minimum depth Q-learning according to claim 1, characterized in that, Step 4 involves calculating the reward value for the current action. And obtain interference actions through sensing data. And construct the spectrum state Record the interaction Add to the experience replay unit Record the interaction Add to the experience replay unit Specifically: Intelligent interference inputs the obtained spectral waterfall plot into the interference DQN network, based on... - Greedy selection of interference actions ; After the user and the interference each perform their respective actions, the user calculates the reward value at the current moment. ; Obtain the state at the next moment by sensing the spectrum. ; Record this interaction Stored in the experience replay unit .

7. The anti-smart interference method based on maximum-minimum depth Q-learning according to claim 1, characterized in that, Step 5 describes calculating the Q-value estimation error. Then, the anti-interference decision network is updated using the gradient descent method, specifically: Random sampling training samples Used for network updates; For each training sample, the state The estimated Q value is obtained by inputting it into the anti-interference decision network. ; Next state The input is fed into the target value network to obtain the Q-value matrix. ,in The network parameters represent the target value, which are then used to calculate the target Q-value. for: （5） in Q-value matrix User balancing strategy; Calculate the estimation error : （6） Update network parameters using gradient descent: （7）。

Images