A Reinforcement Learning-Based Communication Interference Decision-Making Method

Abstract

This invention proposes a communication jamming decision-making method based on reinforcement learning, comprising the following steps: establishing a communication adversarial scenario model and determining indicators based on the scenario to evaluate the jamming effect; abstracting the learning process into a Markov process and using an ε-greedy strategy to determine the Q-value update criterion; specifying the Markov process and setting the reward function according to different communication adversarial backgrounds; designing specific steps of the ε-time-varying Q-Learning communication jamming decision-making method; and conducting experimental simulations of communication jamming implementation in two different scenarios to verify the advantages of the invention from multiple aspects. This invention, through solving the algorithm strategy, can obtain the optimal decision for our side to implement jamming under communication adversarial conditions, completing the tracking and jamming of unknown signals from the opponent, and effectively optimizing the shortcomings of traditional algorithms in several aspects.

Description

Technical Field

[0001] This invention relates to the field of communication technology, and more specifically to a communication interference decision-making method based on reinforcement learning. Background Technology

[0002] Communication countermeasures are an important part of the field of communications, and communication jamming is one of the means of communication countermeasures. It mainly involves transmitting jamming signals with corresponding frequencies, bandwidths, power and time characteristics based on the characteristics of communication signals, so as to prevent them from working properly or cause them to malfunction, thereby achieving the purpose of interfering with communications.

[0003] Traditional communication jamming typically involves high-power suppression jamming, which cannot be targeted at specific targets based on received communications. This method has several drawbacks: jamming parameters determined by operator experience are ill-suited to the ever-changing competitive landscape; high-power suppression jamming easily exposes our positional information, hindering the preservation of our forces and excessively consuming communication jamming resources; high-power suppression jamming interferes with communication equipment on both sides, negatively impacting our own communications; and if the enemy employs advanced anti-jamming measures, we cannot achieve effective tracking jamming. Therefore, given the increasingly complex electromagnetic environment of the battlefield, reinforcement learning algorithms with environmental awareness and autonomous decision-making capabilities have become a research hotspot for solving the jamming decision-making problem.

[0004] Current reinforcement learning algorithms are mainly divided into two different approaches based on the learning policy: value-based methods and policy-based methods. Among them, value-based methods mainly learn the value function of each state and use the estimated value of the state to guide decision-making.

[0005] Q-Learning is the most representative value-based algorithm. This method is a model-free reinforcement learning approach that does not require complete prior information about the environment. It can learn the optimal policy directly from interactions with the environment, reducing dependence on prior information while ensuring the convergence of the decision-making process.

[0006] In Q-Learning, the agent maintains a Q-table to record the Q-value for each state-action, where the value represents the expected long-term reward for taking that action in that state. By continuously exploring the environment, performing actions, and updating the Q-value, the agent can learn the optimal policy.

[0007] When selecting actions, the initial strategy is usually to randomly select actions. However, such exploration is unpredictable and cannot effectively utilize the optimal actions found. Therefore, to avoid local optima, an ε-greedy strategy is typically adopted. The essence of the ε-greedy strategy is: to randomly select an action in the action space with a probability of ε, which represents "exploration"; and to select the optimal action corresponding to the Q-value table in the current state with a probability of 1-ε. The optimal action is the one with the largest Q-value, which represents "utilization".

[0008] However, traditional Q-Learning methods are extremely sensitive to the initial value of the exploration factor ε: if the initial value is set too high, the agent may overconfidently choose certain actions in the early stages, missing better exploration opportunities; if the initial value is set too low, the learning process may be too conservative, requiring more time to discover high-value actions. Considering the context of communication interference decision-making, a larger exploration factor leads to actions that are more inclined towards exploration, resulting in a lower interference hit rate; a smaller exploration factor results in a longer learning step size and learning time, leading to lower efficiency. This invention, by solving the algorithm strategy, can obtain the optimal decision for our side to implement interference under communication adversarial conditions, completing the tracking and interference of the opponent's unknown signals, and effectively optimizing the shortcomings of traditional algorithms in several aspects. Summary of the Invention

[0009] To address the shortcomings of existing technologies, this invention proposes a communication interference decision-making method based on reinforcement learning, which specifically includes the following steps:

[0010] Step 1: Establish a communication countermeasure scenario model, determine indicators based on the scenario to evaluate the interference effect, and establish a learning process;

[0011] Step 2: Abstract the learning process into a Markov process and use the ε-greedy strategy to determine the Q-value update criterion;

[0012] Step 3: Define the Markov process and set the reward function based on two different communication adversarial scenarios;

[0013] Step 4: Specific steps for designing a time-varying ε-based Q-Learning communication interference decision-making method;

[0014] Step 5: Conduct experimental simulations and verifications of communication interference implementation in two different communication countermeasure scenarios.

[0015] The first step is specifically as follows:

[0016] Assuming the communication jamming scenario is fixed-frequency communication, the communicating party, i.e. the other party, communicates through a fixed communication channel or a set of channels according to a certain preset rule. The other party will switch communication channels due to interference. In order to achieve effective communication jamming, the jamming party, i.e. our side, needs to learn the unknown communication channel change rules through online learning.

[0017] We use single-frequency interference as the interference model. Let the amplitude of the single-frequency interference signal be B, the continuous time be t, and the interference frequency be f. j If the initial phase of the signal is φ0 and the pi is pi, then the time-domain expression of the signal is:

[0018] j(t)=Bcos(2×pi×f j t+φ0) (1)

[0019] Complex number expression is

[0020]

[0021] Suppose the other party has n communication channels available for communication, and these n channels are independent and have the same bandwidth. The other party's communication scenario can be divided into two categories:

[0022] Scenario 1: The other party's communication channel is unknown, but the communication channel changes over a long period of time and remains unchanged even if it is interfered with by our side. The other party randomly selects one of the n communication channels to communicate. Our side uses an interference strategy to select one of the n communication channels to interfere until the interfering party learns the communication channel.

[0023] Scenario 2 is that the other party's communication channel is unknown, but after being interfered with, it will switch to the next channel in a certain order to avoid interference. Assuming that the other party's channel switching strategy is a randomly generated set of random sequences, when our interference channel is the same as the other party's communication channel, the other party switches to the next channel according to the channel switching strategy. We continue to use the interference strategy to select the channel for interference until we learn the communication party's channel switching strategy.

[0024] The interference effect is evaluated by the overlap between the interference frequency and the target signal frequency, and the reward function is designed using the frequency overlap in the subsequent process.

[0025] Frequency overlap is defined as follows:

[0026] Let the frequency of the interference signal be f. i The target signal frequency is f c Then the frequency overlap ε t Represented as:

[0027] ε t =|f i -fc | / f c (3)

[0028] By analyzing expression (3), we can conclude that the closer the interference frequency is to the target signal frequency, the closer the frequency overlap is to 0, that is, the higher the frequency overlap, the higher the overlap between the interference frequency and the frequency of the interfered signal, and the better the interference effect.

[0029] The second step is as follows:

[0030] The learning process is abstracted as a Markov Decision Process (MDP), where the policy depends entirely on the current state. The problem to be solved is represented as a six-tuple M = (S, A, P, R, γ, π), where S represents the state space, A represents the action space, P represents the state transition probability matrix, R represents the reward, γ is the reward decay factor and γ∈[0,1], and π represents the policy. If the agent's state in the Q-Learning process is s... t ∈S, in the current state, perform action a t ∈A, change the current state, where the next state s is entered. t+1 The probability of ∈S is p t ∈P;

[0031] Based on the agent's actions, an action a is taken. t Reward r t+1 And so on, the agent continues to choose the next action and obtain a new reward value. The ultimate goal is to learn an optimal policy. The iterative process of state and behavior changes constitutes the main body of reinforcement learning. The sequence description obtained in the MDP process is as follows:

[0032] s0,a0,r1,s1,a1,r2,s2,a2,r3,...,s n-1 ,a n-1 ,r n

[0033] The cumulative reward of the agent after taking action under the optimal policy π is represented as:

[0034]

[0035] The goal of reinforcement learning is to maximize cumulative reward, while the reward received by the agent from the environment is the immediate reward. In reinforcement learning, it is necessary to consider the current immediate reward and the reward that may be obtained in the future, and set the reward function according to scenario one and scenario two.

[0036] On the other hand, for environments with a large decision space, the agent cannot traverse all actions and states. The agent learns better interference strategies by continuously interacting with the environment and utilizing the feedback from the environment to traverse all actions and states as much as possible. The Q-Learning algorithm is used to solve the problem of intelligent interference decision-making.

[0037] The core of the Q-Learning algorithm is the Q-value table, which adopts the ε-greedy strategy. The essence of the ε-greedy strategy is: generate a random number ρ from [0,1]. If ρ is less than ε, then randomly select an action in the action space with a probability of ε, which means exploration; if ρ is greater than or equal to ε, then select the optimal action corresponding to the Q-value table in the current state with a probability of 1-ε. The optimal action is the action with the largest Q-value, which means utilization.

[0038] During the exploration process, the Q-value update criterion is determined based on the exploration factors and reward function:

[0039] Q(s,a)←Q(s,a)+α[r+γmax a' Q(s',a')-Q(s,a)] (5)

[0040] Where ε is the exploration factor, γ is the reward decay factor, and α is the learning step size.

[0041] The third step specifically involves:

[0042] For scenario one, assuming the opponent's communication channel has a long change period and remains unchanged for a certain period of time, that is, the opponent's communication frequency remains unchanged; in this case, we optimize the interference frequency through continuous feedback of the interference effect.

[0043] Let the communication frequency sequence of the other party be {f1, f2, ..., f n Let the corresponding channel sequences be {1, 2, ..., n}, then the state space S is {f1, f2, ..., fn}. n The action space A is {f1, f2, ..., f}. n In the initial state s1, the communication frequency of the other party is f. i We first adopted frequency f j The interference is implemented, and the environmental state at this time is s1={f j The frequency overlap is used to represent the effectiveness of the interference. The closer the frequency overlap is to 0, the more effective the communication interference is; conversely, the effect of the interference is poor. According to equation (3), the frequency overlap is:

[0044] ε1=|f i -f j | / f i (6)

[0045] Compare the relationship between ε1 and 0 to determine the next action and strategy;

[0046] During the learning process, the intelligent interference decision-making algorithm will determine the communication channel to be selected as the interference action in the next moment based on the interference effect feedback from the current state, and execute it. After implementing the interference, the effect of the interference is evaluated, and the reward value obtained by taking the current action is recorded. Based on the reward value, the adopted strategy is optimized. According to the effectiveness of the interference implementation, the reward function is defined as follows:

[0047]

[0048] Since the interference frequency is the same as the target frequency, ε t There exists a case where the value equals 0, in which case the expression is meaningless. Therefore, to avoid this situation, a small number is added to the denominator; here, 0.01 is chosen. Thus, the reward function is corrected to:

[0049]

[0050] When the communication frequency of the communicating party is successfully predicted, i.e., ε t When the value is 0, the maximum reward value is 100.

[0051] According to equation (8), in state s1, the interfering party selects action f. j The reward value is

[0052]

[0053] If ε1≠0 at this time, and the perturbation action selected by the strategy at the next moment is a1={f k Therefore, the next state update is s2 = {f} k};

[0054] For scenario two, assuming the other party switches to the next channel in a certain order after being interfered with, that is, after being interfered with, the other party's communication channel selects the next communication channel in a certain order; the interference frequency is optimized by utilizing the continuous feedback of the interference effect. Let the range of the other party's communication channel sequence be {1, 2, ..., n}, and the corresponding communication frequency sequence for each channel be {f1, f2, ..., f...}. n If f(f1, f2, ..., f(f ... n The action space A is {f1, f2, ..., f}. n The communication channel switching rule is {f}. i ,f j ,...,f k}(i,j,...,k∈{1,2,...,n}∩i≠j≠...≠k), in the initial state s1, the communication frequency of the other party is f. i We first adopted frequency f m The interference is implemented, and the environmental state at this time is s1={f i ,f m Similarly, frequency overlap is used to represent the effectiveness of the interference.

[0055] To learn the opponent's communication channel switching strategy as quickly as possible, the reward function is optimized. Based on the effectiveness of the interference implementation, constraints are added for consecutive successful interference hits. The reward function is defined as follows:

[0056]

[0057] Where k is the consecutive hit factor, k satisfies k=1 if and only if there are two consecutive successful interferences, and k=0 otherwise;

[0058] When the opponent's communication channel is successfully interfered with twice in a row, an additional reward value of 5 is given. In state s1, if ε1 = 0 at this time, it means that the interference was successful in this state, and the opponent switches to the next communication channel f according to the channel switching rules. j The next time-instance disturbance action selected using the strategy is a1 = {f} p}, at this point the state is updated to s2 = {f j ,f p In state s2, if ε2=0, it means that interference was also successful in this state, and the continuous interference condition is met at this time, so an additional reward value is given; when continuous interference is met, the reward value in this state is 105.

[0059] The fourth step is specifically as follows:

[0060] Based on the above detailed learning process and reward function settings, a Q-Learning communication interference decision-making method based on ε time-varying is designed to achieve the optimal solution of the communication interference strategy. Under the premise of ensuring the interference hit rate, the cumulative reward value and hit rate convergence of the learning process are improved, and the number of learning times is reduced.

[0061] Scene 1:

[0062] The specific steps of designing a time-varying Q-Learning communication interference decision-making method based on ε are as follows: A function is designed to make the exploration factor ε decrease linearly with the number of rounds.

[0063] 1. Define the state space S and action space A in the environment;

[0064] 2. Initialize Q(s,a)=0, π(s,a)=current optimal policy;

[0065] 3. Initialize ε, α, γ;

[0066] 4. Initialize state s and accumulated reward value R;

[0067] 5. Generate a random number ρ from [0,1]. If ρ < ε, then randomly generate an interference frequency f from the action space; otherwise, let f = π(s) and select the current optimal action to execute.

[0068] 6. Update the reward value according to the reward function, which is expression (8);

[0069] 7. Update the Q value based on the state-action value function;

[0070] 8. Update status;

[0071] 9. Update total reward value;

[0072] 10. Proceed to step 5 until the target state is reached, then terminate the iteration;

[0073] 11. Update the ε value according to the formula, return to step 4, and repeat until the training round ends;

[0074] Scene 2:

[0075] The specific steps of designing a time-varying Q-Learning communication interference decision-making method based on ε are as follows: A function is designed to make the exploration factor ε decrease linearly with the number of rounds.

[0076] 1. Define the state space S and action space A in the environment;

[0077] 2. Initialize Q(s,a)=0, π(s,a)=current optimal policy;

[0078] 3. Initialize ε, α, γ;

[0079] 4. Initialize state s and accumulated reward value R;

[0080] 5. Generate a random number ρ from [0,1]. If ρ < ε, then randomly generate an interference frequency f from the action space; otherwise, let f = π(s) and select the current optimal action to execute.

[0081] 6. Update the reward value according to the reward function, which is expression (10);

[0082] 7. Update the Q value based on the state-action value function;

[0083] 8. Update status;

[0084] 9. Update total reward value;

[0085] 10. Proceed to step 5 until the target state is reached, then terminate the iteration;

[0086] 11. Update the ε value according to the formula, return to step 4, and continue until the training round ends.

[0087] The fifth step is specifically as follows:

[0088] For Scenario 1, under the condition that the opponent's communication channel remains unchanged for a long time, simulation experiments are conducted according to the above algorithm steps. The interference effects of the communication interference decision-making method based on traditional Q-Learning and the communication interference decision-making method based on ε time-varying Q-Learning in a simple scenario are compared from three aspects: the number of learning attempts, the cumulative reward value, and the convergence and convergence results of the successful interference hit rate.

[0089] The number of learning rounds required to implement interference under the theoretical value, the traditional fixed ε condition, and the time-varying ε condition proposed in this method is depicted using three different numerical curves as the number of training rounds increases. By comparing the above numerical curves, it is found that under the time-varying ε condition, the convergence effect of the learning rounds of this method is better than that of the traditional algorithm.

[0090] Four different numerical curves are used to depict the changes in cumulative reward value and average cumulative reward under the traditional fixed ε condition and the cumulative reward value and average cumulative reward under the time-varying ε condition proposed in this method as the number of training rounds increases. By comparing the above numerical curves, it can be concluded that as the number of training rounds increases, the cumulative reward value and average cumulative reward under the time-varying ε condition will increase rapidly and exceed the value under the fixed ε condition.

[0091] Four different numerical curves depict the changes in hit rate and average hit rate under the traditional fixed ε condition and the hit rate and average hit rate under the time-varying ε condition proposed in this method as the number of training rounds increases. By comparing the above numerical curves, it can be concluded that as the number of training rounds increases, the hit rate and average hit rate under the time-varying ε condition will increase rapidly and exceed the values ​​under the fixed ε condition.

[0092] For scenario two, it is assumed that after being interfered with, the communicating party will switch to the next channel in a certain order. That is, after being interfered with, the communication channels select the next communication channel in a certain order. The algorithm steps described above are designed, and simulation experiments are conducted with different numbers of channels and different channel switching orders for the other party.

[0093] In the first experiment, the number of communication channels to the other party was set to 10, and the channel switching order was randomly assumed.

[0094] The second experiment further increased the difficulty, setting the number of communication channels of the other party to 20, and again randomly assuming the channel switching order of the other party;

[0095] The following were drawn respectively:

[0096] In the first experiment, the number of learning attempts and the average number of learning attempts under the time-varying condition of ε increased with the number of training rounds;

[0097] In the first experiment, the cumulative reward value and the average cumulative reward value under the time-varying condition of ε as the number of training rounds increases;

[0098] Hit rate and average hit rate under time-varying ε conditions as the number of training rounds increases in the first experiment;

[0099] In the second experiment, the number of learning attempts and the average number of learning attempts under the time-varying condition of ε increased with the number of training rounds;

[0100] In the second experiment, the cumulative reward value and the average cumulative reward value under the time-varying condition of ε increase with the number of training rounds;

[0101] In the second experiment, the hit rate and average hit rate under time-varying ε conditions increased with the number of training rounds;

[0102] For two scenarios—one where the opponent's communication channel remains unchanged and the other where the communication channel changes according to a preset pattern—the reward function is optimized based on these two different scenarios: The optimization function is designed using the overlap between the channel frequency used for our interference and the channel frequency used for the opponent's communication as an indicator. When the opponent's communication channel remains unchanged, this method adds a minimum value component to the denominator of the reward function, avoiding meaningless processes and improving the robustness of the algorithm without affecting its operation. When the opponent's communication channel changes according to a preset order, this method further optimizes the reward function by adding a consecutive hit factor to reward two consecutive successful interference attempts, thereby reducing the number of iterations in complex adversarial scenarios and accelerating the convergence speed. Experiments are conducted according to specific steps 1-11 of the fourth step of the communication interference decision-making implementation method based on ε-time-varying Q-Learning.

[0103] Furthermore, in the second step, the following characteristics are established: the exploration factor ε = 0.1, the learning step size α = 0.3, the reward decay factor γ = 0.9, the time-varying exploration factor expression ε = ε - 0.0002·i, and the reward function... Attached Figure Description

[0104] Figure 1 The diagram illustrates the basic principles of reinforcement learning.

[0105] Figure 2 A diagram illustrating the overall implementation steps of a reinforcement learning-based communication interference decision-making method is provided.

[0106] Figure 3The flowchart of the Q-Learning communication interference decision-making procedure based on ε time-varying is shown;

[0107] Figure 4 The comparison of the learning iteration convergence curves of the traditional Q-Learning method and the present invention is shown under adversarial scenario one.

[0108] Figure 5 A comparison of the cumulative reward value convergence curves of the traditional Q-Learning method and the present invention is shown in adversarial scenario one.

[0109] Figure 6 The comparison of the hit rate convergence curves of the traditional Q-Learning method and the present invention is shown in adversarial scenario one.

[0110] Figure 7 The convergence curve of the learning iterations of this invention is shown for the opponent's switching order of 10 channels under the second adversarial scenario.

[0111] Figure 8 The cumulative reward value convergence curve of this invention is shown for the opponent's 10 channel switching order under the second adversarial scenario.

[0112] Figure 9 The hit rate convergence curve of this invention is shown for the switching order of the opponent's 10 channels under the second adversarial scenario.

[0113] Figure 10 The convergence curve of the learning iterations of this invention is shown for the opponent's 20 channel switching order under the second adversarial scenario.

[0114] Figure 11 The cumulative reward value convergence curve of this invention is shown under the second adversarial scenario, considering the opponent's switching order of 20 channels.

[0115] Figure 12 The hit rate convergence curve of this invention is shown for the opponent's switching order of 20 channels under the second adversarial scenario. Detailed Implementation

[0116] The present invention will be further described below with reference to the embodiments and accompanying drawings.

[0117] Figure 2This paper presents the overall implementation steps of a communication jamming decision-making method based on reinforcement learning. First, a communication adversarial scenario is modeled, and evaluation indicators are determined based on the scenario to assess the jamming effect. Second, the reinforcement learning process is abstracted into a Markov process, setting up a state space, action space, state, action, and reward function. The classic Q-Learning method in reinforcement learning is introduced to reduce reliance on prior information, while an ε-greedy strategy is used to determine the Q-value update criterion. Then, the Markov process is instantiated according to different communication adversarial backgrounds, and specific expressions for the reward function are set. Further, specific steps of an ε-time-varying Q-Learning communication jamming decision-making method are designed. Finally, experimental simulations are conducted to verify the implementation of communication jamming in two different scenarios.

[0118] Figure 3 The flowchart of the communication interference decision-making procedure based on ε time-varying Q-Learning is shown. The specific algorithm steps are as follows:

[0119] 1. Define the state space S and action space A in the environment;

[0120] 2. Initialize Q(s,a)=0, π(s,a)=current optimal policy;

[0121] 3. Initialize ε, α, γ;

[0122] 4. Initialize state s, accumulated reward value R

[0123] 5. Generate a random number ρ from [0,1].

[0124] If ρ < ε, then an interference frequency f is randomly generated from the action space;

[0125] Otherwise, let f = π(s), and select the current optimal action to execute;

[0126] 6. Update the reward value according to the reward function;

[0127] 7. Update the Q value based on the state-action value function;

[0128] 8. Update status;

[0129] 9. Update total reward value;

[0130] 10. Proceed to step 5 until the target state is reached, then terminate the iteration;

[0131] 11. Update the ε value according to the formula, return to step 4, and continue until the training round ends.

[0132] Example:

[0133] Let the initial Q(s,a) = 0, π(s,a) = the current optimal policy, the initial exploration factor ε = 0.1, the learning step size α = 0.3, and the reward decay factor γ = 0.9;

[0134] The aforementioned reinforcement learning-based communication interference decision-making method specifically includes:

[0135] Step 1: Establish a communication countermeasure scenario model, determine indicators based on the scenario to evaluate the interference effect, and establish a learning process;

[0136] Two communication adversarial scenarios are set up:

[0137] Scenario 1 assumes that the other party's communication channel has a long change period, and the communication channel remains unchanged for a certain period of time, that is, the other party's communication frequency remains unchanged.

[0138] Scenario 2 assumes that after being interfered with, the other party will switch to the next channel in a certain order. That is, after being interfered with, the communication channel of the communicating party will select the next communication channel in a certain order.

[0139] This invention focuses on how to achieve tracking interference in communication channels without considering the specific form of the communication signal. Therefore, the simplest single-frequency interference is used as the interference model.

[0140] Step 2: Abstract the learning process into a Markov process and use the ε-greedy strategy to determine the Q-value update criterion.

[0141] Reinforcement learning can be abstracted as a Markov Decision Process (MDP), where the policy depends entirely on the current state. The problem to be solved can be represented as a six-tuple M = (S, A, P, R, γ, π), where S represents the state space, A represents the action space, P represents the state transition probability matrix, R represents the reward, γ is the reward decay factor and γ∈[0,1], and π represents the policy.

[0142] In different adversarial scenarios, or under the context of the opponent's different channel switching strategies in the same adversarial scenario, the corresponding state space, action space, and reward function have been instantiated; the state transition probability matrix represents the agent's state s in the current state. t Under ∈S, perform action a t ∈A, change the current state, where the next state s is entered. t+1 The probability of ∈S. The reward decay factor γ is set to 0.9, and the initial policy π(s,a) = the current optimal policy.

[0143] The core of the Q-Learning algorithm is the Q-value table, which records the maximum reward value obtained by taking different actions in each state, facilitating the selection of the optimal action in the corresponding state. When selecting actions, the initial strategy is generally to randomly select actions, but such exploration is unpredictable and cannot effectively utilize the discovered optimal actions. Therefore, to balance the relationship between exploration and utilization, considering both immediate and future rewards, and avoiding local optima, an ε-greedy strategy is typically adopted. This means randomly selecting an action in the action space with a probability of ε, representing "exploration"; and selecting the optimal action in the Q-value table corresponding to the current state with a probability of 1-ε. The optimal action, which is the action with the largest Q-value, represents "utilization."

[0144] During the exploration process, the Q value is updated according to equation (5), where Q(s,a) is initialized to 0, ε is the exploration factor with an instantiation value of 0.1, γ is the reward decay factor with an instantiation value of 0.9, and α is the learning step size with an instantiation value of 0.3.

[0145] Therefore, the initial Q value update formula for this example is:

[0146] Q(s,a)←0+0.3[r+0.9max' a′ Q(s′,a′)] (11)

[0147] Step 3: Specificify the Markov process and set the reward function according to different communication adversarial backgrounds.

[0148] Let the opponent's communication frequency be f = 10 GHz, and the corresponding channel sequences be {1, 2, ..., 5}. Then the state space S is {f1, f2, ..., f5 = 10 GHz}, and the action space A is {f1, f2, ..., f5}. In the initial state s1, we first use frequency f... j The interference is implemented, and the environmental state at this time is s1={f j The frequency overlap is used to represent the effectiveness of the interference. The closer the frequency overlap is to 0, the more effective the communication interference; conversely, the greater the overlap, the less effective the interference. According to equation (6), the frequency overlap is:

[0149] ε1=|f1-f j | / f1 (12)

[0150] Compare the relationship between ε1 and 0 to determine whether effective interference has been implemented.

[0151] According to equation (8), in state s1, we choose action f. j The reward value is:

[0152]

[0153] When there are 10 communication channels, assume that the communicating parties communicate in the following channel switching order: [4,2,6,7,1,0,9,8,3,5]; the other party's communication channel sequence range is {1,2,...,10}, and the corresponding communication frequency sequence for each channel is {f1,f2,...,f...}. 10 If f(f1, f2, ..., f(f ... 10}, the action space A is {f1,f2,...,f1...,f10}0}, and the channel switching rule is: {f i ,f j ,...,f k}(i,j,...,k∈{1,2,...,10}∩i≠j≠...≠k).

[0154] When there are 20 communication channels, assume the communicating parties communicate in the following channel switching order: [7,16,4,5,11,17,0,8,1,13,18,15,9,2,19,10,14,6,12,3]. The other party's communication channel sequence ranges from {1, 2, ...,20}, and the corresponding communication frequency sequence for each channel is {f1,f2,...,f...}. 20 If f(f1, f2, ..., f(f ... 20 The action space A is {f1, f2, ..., f}. 20 The communication channel switching rule is {f}. i ,f j ,...,f k}(i,j,...,k∈{1,2,...,20}∩i≠j≠...≠k).

[0155] To learn the opponent's communication channel switching strategy as quickly as possible, the reward function is optimized. Based on the effectiveness of the jamming implementation, constraints are added for consecutive jamming hits, and the reward function is defined as follows:

[0156]

[0157] Where k is the consecutive hit factor, k satisfies k=1 if and only if there are two consecutive successful interferences, otherwise k=0.

[0158] Step 4: Designing a time-varying ε-based Q-Learning communication interference decision-making method: Specific steps

[0159] Based on the above detailed learning process and reward function settings, we design an ε-time-varying Q-Learning communication interference decision-making method to achieve the optimal solution of the communication interference strategy. While ensuring the interference hit rate, we improve the cumulative reward value and hit rate convergence of the learning process and reduce the number of learning iterations.

[0160] To avoid an imbalance between exploration and exploitation actions during the acquisition of the optimal strategy, and considering both the current immediate reward and the potential future reward, this invention decreases the value of the exploration factor ε as the training rounds increase. Its expression is as follows:

[0161] ε=ε-0.0002·i (15)

[0162] The advantage of this is that a larger initial exploration factor ε allows for a more thorough exploration of the action space, which is beneficial for utilizing the optimal action when the exploration factor ε is smaller later, and thus facilitates the convergence of the algorithm.

[0163] According to equation (15), the exploration factor ε increases continuously with each training round and decreases by 0.0002 per round until it decreases to 0 in the 500th round.

[0164] Therefore, when ε = 0, the final algorithm will become a pure exploitation form, continuously using the existing best strategy to select the best strategy in the current state.

[0165] The specific steps for developing the ε-time-varying Q-Learning communication interference decision-making method are as follows:

[0166]

[0167] Specifically, Scenario 2 continues the above steps, but the reward function has been adjusted and optimized.

[0168] In the specific implementation process, for scenario one, the reward function in step 6 of the interference method uses equation (8); for scenario two, the reward function in step 6 of the interference method uses equation (10); compared with equation (8), the reward function of scenario two, namely equation (10), adds a continuous hit factor to reward two consecutive successful interference implementations, so as to reduce the number of iterations in complex adversarial scenarios and accelerate the convergence speed.

[0169] Step 5: Conduct experimental simulations of communication interference in two different scenarios to verify the advantages of the basic invention from multiple aspects.

[0170] For Scenario 1, under the condition that the opponent's communication channel remains unchanged for a long time, simulation experiments are conducted according to the above algorithm steps. The interference effects of the communication interference decision-making method based on traditional Q-Learning and the communication interference decision-making method based on ε time-varying Q-Learning in a simple scenario are compared from three aspects: the number of learning attempts, the cumulative reward value, and the convergence and convergence results of the successful interference hit rate.

[0171] Three numerical curves of different colors depict the number of learning rounds required to implement interference under the theoretical value, the traditional fixed ε condition, and the time-varying ε condition proposed in this invention, as the number of training rounds increases. Figure 4 It can be concluded that under the time-varying ε condition, the convergence effect of the learning iterations of the present invention is better than that of the traditional algorithm;

[0172] The cumulative reward value and average cumulative reward under the traditional fixed ε condition are depicted using four numerical curves of different colors as the number of training rounds increases; the changes in the cumulative reward value and average cumulative reward under the time-varying ε condition proposed in this invention are also shown. Figure 5 It can be concluded that as the number of training rounds increases, the cumulative reward value and the average cumulative reward under the time-varying ε condition will increase rapidly and exceed the value under the fixed ε condition;

[0173] The accuracy and average accuracy under the traditional fixed ε condition are depicted using four numerical curves of different colors as the number of training rounds increases; the changes in accuracy and average accuracy under the time-varying ε condition proposed in this invention are also shown. Figure 6 The comparison shows that as the number of training rounds increases, the hit rate and average hit rate under the time-varying ε condition will increase rapidly and exceed the value under the fixed ε condition.

[0174] For scenario two, it is assumed that after being interfered with, the communicating party will switch to the next channel in a certain order. That is, after being interfered with, the communication channels select the next communication channel in a certain order. The algorithm steps described above are designed, and simulation experiments are conducted with different numbers of channels and different channel switching orders for the other party.

[0175] In the first experiment, the number of communication channels to the other party was set to 10, and the channel switching order was randomly assumed.

[0176] The second experiment further increased the difficulty, setting the number of communication channels of the other party to 20, and again randomly assuming the channel switching order of the other party;

[0177] The following were drawn respectively:

[0178] In the first experiment, the number of learning iterations and the average number of learning iterations under the time-varying condition of ε were obtained as the number of training rounds increased. Figure 7 Shown;

[0179] In the first experiment, the cumulative reward value and average cumulative reward value under the time-varying condition of ε increased with the number of training rounds. Figure 8 Shown;

[0180] In the first experiment, the hit rate and average hit rate under time-varying ε conditions increased with the number of training rounds. Figure 9 Shown;

[0181] In the second experiment, the number of learning iterations and the average number of learning iterations under the time-varying condition of ε increased with the number of training rounds, through... Figure 10 Shown;

[0182] In the second experiment, the cumulative reward value and average cumulative reward value under the time-varying condition of ε increased with the number of training rounds, through... Figure 11 Shown;

[0183] In the second experiment, the hit rate and average hit rate under time-varying ε conditions increased with the number of training rounds, through... Figure 12 As shown.

[0184] For the two cases where the opponent's communication channel remains unchanged and the communication channel changes according to a preset rule, the reward function is optimized according to the two different scenarios: the overlap between the channel frequency of our interference and the channel frequency of the opponent's communication is used as an indicator to design an optimization function; when the opponent's communication channel remains unchanged, the present invention adds a minimum value part to the denominator of the reward function, which avoids meaningless processes without affecting the operation of the algorithm and improves the robustness of the algorithm operation, as shown in the above formula (8); when the opponent's communication channel changes according to a preset order, the present invention further optimizes the reward function and adds a continuous hit factor to reward two consecutive successful interference implementations, so as to reduce the number of iterations in complex adversarial scenarios and accelerate the convergence speed, as shown in the above formula (10). The experiment is carried out according to the specific steps 1-11 of the communication interference decision implementation method based on ε time-varying Q-Learning.

[0185] For Scenario 1, under the condition that the other party's communication channel remains unchanged for a long time, a simulation experiment is carried out for a fixed frequency f=10GHz according to the above instantiation settings. The interference effect of the communication interference decision method based on traditional Q-Learning and the communication interference decision method based on ε time-varying Q-Learning in a simple scenario is compared from three aspects: the number of learning times, the cumulative reward value, and the convergence and convergence result of the hit rate of successful interference.

[0186] pass Figure 4 The algorithm comparison shows that the exploration factor ε decreases as the training rounds increase, which can enhance the interference effect. The ε-time-varying algorithm has a better convergence effect in the number of training rounds than the traditional algorithm. Figure 5 and Figure 6 The results demonstrate that as the number of training rounds increases, the cumulative reward and hit rate of the ε-time-varying algorithm rapidly increase and quickly stabilize, ultimately reaching their maximum values. Therefore, the ε-time-varying Q-Learning algorithm for learning the frequency of interference from unknown signals can learn the optimal interference strategy.

[0187] For scenario two, it is assumed that after being interfered with, the communicating party will switch to the next channel in a certain order. That is, after being interfered with, the communication channels select the next communication channel in a certain order. According to the above instantiation settings, simulation experiments were carried out for 10 channels and their switching order, and 20 channels and their switching order.

[0188] from Figure 7 and Figure 10 It can be seen that as the number of explorations increases, the number of interferences decreases, allowing the interfering party to find the communication frequency of the communicating party more quickly and effectively, eventually converging to the first frequency, indicating a good learning effect. Analysis Figure 8 , Figure 9 , Figure 11 and Figure 12 The results showed that as the number of explorations increased, both the maximum cumulative reward and the hit rate tended to converge, with the maximum cumulative reward being relatively large and the hit rate reaching 100%, thus achieving effective interference.

[0189] This invention proposes a communication jamming decision-making method based on reinforcement learning, comprising the following steps: establishing a communication adversarial scenario model and determining indicators based on the scenario to evaluate the jamming effect; abstracting the learning process into a Markov process and using an ε-greedy strategy to determine the Q-value update criterion; specifying the Markov process and setting the reward function according to different communication adversarial backgrounds; designing specific steps of the ε-time-varying Q-Learning communication jamming decision-making method; and conducting experimental simulations of communication jamming implementation in two different scenarios to verify the advantages of the invention from multiple aspects. This invention, through solving the algorithm strategy, can obtain the optimal decision for our side to implement jamming under communication adversarial conditions, completing the tracking and jamming of unknown signals from the opponent, and effectively optimizing the shortcomings of traditional algorithms in several aspects.

Claims

1. A communication interference decision-making method based on reinforcement learning, characterized in that, Specifically, the following steps are included: Step 1: Establish a communication countermeasure scenario model, determine indicators based on the scenario to evaluate the interference effect, and establish a learning process; The first step is specifically as follows: The communication jamming scenario is set as fixed-frequency communication. The communicating party, i.e. the other party, communicates through a fixed communication channel or a set of channels according to a certain preset rule. The other party will switch communication channels when it is interfered with. In order to achieve effective communication jamming, the jamming party, i.e. our side, needs to learn the unknown communication channel change rules through online learning. We use single-frequency interference as the interference model. Let the amplitude of the single-frequency interference signal be B, the continuous time be t, and the interference frequency be f. j If the initial phase of the signal is φ0 and the pi is pi, then the time-domain expression of the signal is: j(t)=Bcos(2×pi×f j t+φ0) (1) Complex number expression is Suppose the other party has n communication channels available for communication, and these n channels are independent and have the same bandwidth. The other party's communication scenario can be divided into two categories: Scenario 1: The other party's communication channel is unknown, but the communication channel changes over a long period of time and remains unchanged even if it is interfered with by our side. The other party randomly selects one of the n communication channels to communicate. Our side uses an interference strategy to select one of the n communication channels to interfere until the interfering party learns the communication channel. Scenario 2 is that the other party's communication channel is unknown, but after being interfered with, it will switch to the next channel in a certain order to avoid interference. Assuming that the other party's channel switching strategy is a randomly generated set of random sequences, when our interference channel is the same as the other party's communication channel, the other party switches to the next channel according to the channel switching strategy. We continue to use the interference strategy to select the channel for interference until we learn the communication party's channel switching strategy. The interference effect is evaluated by the overlap between the interference frequency and the target signal frequency, and the reward function is designed using the frequency overlap in the subsequent process. Frequency overlap is defined as follows: Let the frequency of the interference signal be f. i The target signal frequency is f c Then the frequency overlap ε t Represented as: ε t =|f i -f c | / f c (3) By analyzing expression (3), we can conclude that the closer the interference frequency is to the target signal frequency, the closer the frequency overlap is to 0, that is, the higher the frequency overlap is. Step 2: Abstract the learning process into a Markov process and use the ε-greedy strategy to determine the Q-value update criterion; Step 3: Define the Markov process and set the reward function based on two different communication adversarial scenarios; The third step is specifically as follows: For scenario one, assuming the opponent's communication channel has a long change period and remains unchanged for a certain period of time, that is, the opponent's communication frequency remains unchanged; in this case, we optimize the interference frequency through continuous feedback of the interference effect. Let the communication frequency sequence of the other party be {f1, f2, ..., f n Let the corresponding channel sequences be {1, 2, ..., n}, then the state space S is {f1, f2, ..., fn}. n The action space A is {f1, f2, ..., f}. n In the initial state s1, the communication frequency of the other party is f. i We first adopted frequency f j The interference is implemented, and the environmental state at this time is s1={f j The frequency overlap is used to represent the effectiveness of the interference. The closer the frequency overlap is to 0, the more effective the communication interference is; conversely, the effect of the interference is poor. According to equation (3), the frequency overlap is: ε1=|f i -f j | / f i (6) Compare the relationship between ε1 and 0 to determine the next action and strategy; During the learning process, the intelligent interference decision-making algorithm will determine the communication channel to be selected as the interference action in the next moment based on the interference effect feedback from the current state, and execute it. After implementing the interference, the effect of the interference is evaluated, and the reward value obtained by taking the current action is recorded. Based on the reward value, the adopted strategy is optimized. According to the effectiveness of the interference implementation, the reward function is defined as follows: Since the interference frequency is the same as the target frequency, ε t There exists a case where the value equals 0, in which case the expression is meaningless. Therefore, to avoid this situation, a small number is added to the denominator; here, 0.01 is chosen. Thus, the reward function is corrected to: When the communication frequency of the communicating party is successfully predicted, i.e., ε t When the value is 0, the maximum reward value is 100. According to equation (8), in state s1, the interfering party selects action f. j The reward value is If ε1≠0 at this time, and the perturbation action selected by the strategy at the next moment is a1={f k Therefore, the next state update is s2 = {f} k }; For scenario two, assuming the other party switches to the next channel in a certain order after being interfered with, that is, after being interfered with, the other party's communication channel selects the next communication channel in a certain order; the interference frequency is optimized by utilizing the continuous feedback of the interference effect. Let the range of the other party's communication channel sequence be {1, 2, ..., n}, and the corresponding communication frequency sequence for each channel be {f1, f2, ..., f...}. n If f(f1, f2, ..., f(f ... n The action space A is {f1, f2, ..., f}. n The communication channel switching rule is {f}. i ,f j ,...,f k Given that (i,j,...,k∈{1,2,...,n}, and i,j,...,k are all different), in the initial state s1, the communication frequency of the other party is f. i We first adopted frequency f m The interference is implemented, and the environmental state at this time is s1={f i ,f m Similarly, frequency overlap is used to represent the effectiveness of the interference. To learn the opponent's communication channel switching strategy as quickly as possible, the reward function is optimized. Based on the effectiveness of the interference implementation, constraints are added for consecutive successful interference hits. The reward function is defined as follows: Where k is the consecutive hit factor, k satisfies k=1 if and only if there are two consecutive successful interferences, and k=0 otherwise; When the opponent's communication channel is successfully interfered with twice in a row, an additional reward value of 5 is given. In state s1, if ε1 = 0 at this time, it means that the interference was successful in this state, and the opponent switches to the next communication channel f according to the channel switching rules. j The next time-instance disturbance action selected using the strategy is a1 = {f} p }, at this point the state is updated to s2 = {f j ,f p In state s2, if ε2 = 0, it means that interference was also successful in this state, and the continuous interference condition is met at this time, so an additional reward value is given; when continuous interference is met, the reward value in this state is 105. Step 4: Specific steps for designing a time-varying ε-based Q-Learning communication interference decision-making method; Step 5: Conduct experimental simulations and verifications of communication interference implementation in two different communication countermeasure scenarios.

2. The communication interference decision-making method based on reinforcement learning as described in claim 1, characterized in that: The second step is as follows: The learning process is abstracted as a Markov Decision Process (MDP), where the policy depends entirely on the current state. The problem to be solved is represented as a six-tuple M = (S, A, P, R, γ, π), where S represents the state space, A represents the action space, P represents the state transition probability matrix, R represents the reward, γ is the reward decay factor and γ∈[0,1], and π represents the policy. If the agent's state during the Q-Learning process is s at this time t ∈S, in the current state, perform action a t ∈A, change the current state, where the next state s is entered. t+1 The probability of ∈S is p t ∈P; Based on the agent's actions, an action a is taken. t Reward r t+1 And so on, the agent continues to choose the next action and obtain a new reward value. The ultimate goal is to learn an optimal policy. The iterative process of state and behavior changes constitutes the main body of reinforcement learning. The sequence description obtained in the MDP process is as follows: s0,a0,r1,s1,a1,r2,s2,a2,r3,...,s n-1 ,a n-1 ,r n The cumulative reward of the agent after taking action under the optimal policy π is represented as: The goal of reinforcement learning is to maximize cumulative reward, while the reward received by the agent from the environment is the immediate reward. In reinforcement learning, it is necessary to consider the current immediate reward and the reward that may be obtained in the future, and set the reward function according to scenario one and scenario two. On the other hand, for environments with a large decision space, the agent cannot traverse all actions and states. The agent learns better interference strategies by continuously interacting with the environment and utilizing the feedback from the environment to traverse all actions and states as much as possible. The Q-Learning algorithm is used to solve the problem of intelligent interference decision-making. The core of the Q-Learning algorithm is the Q-value table, which adopts the ε-greedy strategy. The essence of the ε-greedy strategy is: generate a random number ρ from [0,1]. If ρ is less than ε, then randomly select an action in the action space with a probability of ε, which means exploration; if ρ is greater than or equal to ε, then select the optimal action corresponding to the Q-value table in the current state with a probability of 1-ε. The optimal action is the action with the largest Q-value, which means utilization. During the exploration process, the Q-value update criterion is determined based on the exploration factors and reward function: Q(s,a)←Q(s,a)+α[r+γmax a' Q(s',a')-Q(s,a)](5) Where ε is the exploration factor, γ is the reward decay factor, and α is the learning step size.

3. The communication interference decision-making method based on reinforcement learning as described in claim 1, characterized in that: The fourth step specifically involves: Based on the above detailed learning process and reward function settings, a Q-Learning communication interference decision-making method based on ε time-varying is designed to achieve the optimal solution of the communication interference strategy. Under the premise of ensuring the interference hit rate, the cumulative reward value and hit rate convergence of the learning process are improved, and the number of learning times is reduced. Scene 1: The specific steps of designing a time-varying Q-Learning communication interference decision-making method based on ε are as follows:

1. Define the state space S and action space A in the environment; 2. Initialize Q(s,a)=0, π(s,a)=current optimal policy; 3. Initialize ε, α, γ; 4. Initialize state s and accumulated reward value R; 5. Generate a random number ρ from [0,1]. If ρ < ε, then randomly generate an interference frequency f from the action space; otherwise, let f = π(s) and select the current optimal action to execute.

6. Update the reward value according to the reward function, which is expression (8); 7. Update the Q value based on the state-action value function; 8. Update status; 9. Update total reward value; 10. Proceed to step 5 until the target state is reached, then terminate the iteration; 11. Update the ε value according to the formula, return to step 4, and repeat until the training round ends; Scene 2: The specific steps of designing a time-varying Q-Learning communication interference decision-making method based on ε are as follows:

1. Define the state space S and action space A in the environment; 2. Initialize Q(s,a)=0, π(s,a)=current optimal policy; 3. Initialize ε, α, γ; 4. Initialize state s and accumulated reward value R; 5. Generate a random number ρ from [0,1]. If ρ < ε, then randomly generate an interference frequency f from the action space; otherwise, let f = π(s) and select the current optimal action to execute.

6. Update the reward value according to the reward function, which is expression (10); 7. Update the Q value based on the state-action value function; 8. Update status; 9. Update total reward value; 10. Proceed to step 5 until the target state is reached, then terminate the iteration; 11. Update the ε value according to the formula, return to step 4, and continue until the training round ends.

4. The communication interference decision-making method based on reinforcement learning as described in claim 1, characterized in that: The fifth step is specifically as follows: For Scenario 1, under the condition that the opponent's communication channel remains unchanged for a long time, the simulation experiment is carried out according to the above steps. The interference effect of the communication interference decision method based on traditional Q-Learning and the communication interference decision method based on ε time-varying Q-Learning in the simple scenario is compared from three aspects: the number of learning times, the cumulative reward value, and the convergence and convergence result of the successful interference hit rate. The number of learning rounds required to implement interference under the theoretical value, the traditional fixed ε condition, and the time-varying ε condition proposed in this method are depicted using three different numerical curves as the number of training rounds increases. The cumulative reward value and average cumulative reward under the traditional fixed ε condition, and the cumulative reward value and average cumulative reward under the time-varying ε condition proposed in this method are depicted by four different numerical curves as the number of training rounds increases. The changes in hit rate and average hit rate under the traditional fixed ε condition and the hit rate and average hit rate under the time-varying ε condition proposed in this method are depicted by four different numerical curves as the number of training rounds increases. For scenario two, it is assumed that after being interfered with, the communicating party will switch to the next channel in a certain order. That is, after being interfered with, the communication channel selects the next communication channel in a certain order. The simulation experiment is carried out according to the above steps, and different numbers of channels and channel switching orders are set for the other party: In the first experiment, the number of communication channels to the other party was set to 10, and the channel switching order was randomly assumed. The second experiment further increased the difficulty, setting the number of communication channels of the other party to 20, and again randomly assuming the channel switching order of the other party; The following were drawn respectively: In the first experiment, the number of learning attempts and the average number of learning attempts under the time-varying condition of ε increased with the number of training rounds; In the first experiment, the cumulative reward value and the average cumulative reward value under the time-varying condition of ε as the number of training rounds increases; Hit rate and average hit rate under time-varying ε conditions as the number of training rounds increases in the first experiment; In the second experiment, the number of learning attempts and the average number of learning attempts under the time-varying condition of ε increased with the number of training rounds; In the second experiment, the cumulative reward value and the average cumulative reward value under the time-varying condition of ε increase with the number of training rounds; In the second experiment, the hit rate and average hit rate under time-varying ε conditions increased with the number of training rounds; For the two scenarios of the other party's communication channel remaining unchanged and the communication channel changing according to a preset rule, the reward function is optimized according to the two different scenarios: the overlap between the channel frequency of our interference and the channel frequency of the other party's communication is used as an indicator to design an optimization function; the experiment is carried out according to the specific steps of the communication interference decision implementation method based on ε time-varying Q-Learning in the fourth step.

5. The communication interference decision-making method based on reinforcement learning as described in claim 2, characterized in that: Exploration factor ε = 0.1, learning step size α = 0.3, reward decay factor γ = 0.9, time-varying exploration factor expression ε = ε - 0.0002·i, reward function

Images