Communication interference decision-making method based on reinforcement learning

Abstract

The invention provides a communication interference decision-making method based on reinforcement learning, which comprises the following steps of: establishing a communication confrontation scene model, and determining an index according to a scene so as to evaluate an interference effect; abstracting a learning process into a Markov process, and determining a Q value updating criterion by adopting an epsilon-greedy strategy; materializing the Markov process according to different communication confrontation backgrounds, and setting a reward function; designing specific steps of a Q-Learning communication interference decision method based on epsilon time varying; experimental simulation is carried out on communication interference implementation in two different scenes, and the advantages of the basic invention are verified from multiple aspects. According to the method, the optimal decision of interference implemented by our party under the communication confrontation condition can be obtained by solving the algorithm strategy, the tracking and interference of unknown signals of the opposite party are completed, and the defects of the traditional algorithm are better optimized in multiple aspects.

Description

Technical Field

[0001] The present invention relates to the field of communication technology, and in particular to a communication interference decision-making method based on reinforcement learning. Background Art

[0002] Communication confrontation is an important part of the communication field, and communication interference is one of the means of communication confrontation. It mainly transmits interference signals with corresponding frequency, bandwidth, power and time characteristics according to the characteristics of communication signals, making them unable to work normally or causing malfunctions, thereby achieving the purpose of interfering with communications.

[0003] Traditional communication jamming is generally high-power suppression jamming, which cannot be targeted according to the received communication. This jamming implementation method has many defects: the jamming parameters determined by the operator's experience are difficult to adapt to the modern and changing competitive situation; high-power suppression jamming easily exposes our position information, which is not conducive to preserving our strength, and consumes excessive communication jamming resources; high-power suppression jamming has a jamming effect on both sides' communication equipment, which is not conducive to our communication; if the other party adopts advanced anti-jamming means, we cannot achieve effective tracking jamming. Therefore, in view of the increasingly complex battlefield electromagnetic environment, reinforcement learning algorithms with environmental perception and autonomous decision-making actions have become a research hotspot for solving jamming decision-making problems.

[0004] The current reinforcement learning algorithms are mainly divided into two different types of methods from the perspective of learning strategies: value-based methods and strategy-based methods. Among them, the value-based method mainly learns the value function of each state and guides decision-making by estimating the value of the state.

[0005] Among them, Q-Learning is the most representative value-based algorithm. This method is a model-free reinforcement learning method that does not require complete prior information about the environment. It can directly learn the optimal strategy from the interaction with the environment, reducing the dependence on prior information; at the same time, it ensures the convergence of the decision-making process.

[0006] In Q-Learning, the agent maintains a Q-table to record the Q value corresponding to each state-action, which represents the expected long-term reward for taking the action in that state. By continuously exploring the environment, executing actions, and updating Q values, the agent can learn the optimal strategy.

[0007] When selecting actions, the initial strategy is generally to randomly select actions, but such exploration is irregular and cannot make good use of the optimal action explored. Therefore, in order to avoid the situation of local optimal solution, the ε-greedy strategy can usually be adopted. The essence of the ε-greedy strategy is: randomly select actions in the action space with a probability of ε, which means "exploration"; select the optimal action corresponding to the Q value table in the current state with a probability of 1-ε. The optimal action is the action with the largest Q value, which means "utilization".

[0008] However, the traditional Q-Learning method is extremely sensitive to the initial value of the exploration factor ε: when the initial value is set too high, it may cause the agent to overconfidently choose certain actions in the early stage and miss better exploration opportunities; if the initial value is set too low, the learning process may be too conservative and more time is needed to discover high-value actions. In view of the implementation background of communication interference decision-making, the larger the exploration factor, the more the action tends to explore, and the lower the interference hit rate; the smaller the exploration factor, the longer the learning step and learning time, and the lower the efficiency. The present invention can obtain the best decision for our interference under the condition of communication confrontation by solving the algorithm strategy, complete the tracking and interference of the opponent's unknown signal, and better optimize the defects of traditional algorithms in many aspects. Summary of the invention

[0009] In view of the deficiencies in the prior art, the present invention proposes a communication interference decision method based on reinforcement learning, which specifically includes the following steps:

[0010] Step 1: Establish a communication adversarial scenario model, determine indicators based on the scenario to evaluate the jamming effect, and establish a learning process;

[0011] Step 2: Abstract the learning process into a Markov process and use the ε-greedy strategy to determine the Q value update criterion;

[0012] Step 3: Specify the Markov process and set the reward function according to two different communication confrontation scenarios;

[0013] Step 4: Design the specific steps of the Q-Learning communication interference decision method based on ε time-varying;

[0014] Step 5: Experimentally simulate and verify the implementation of communication interference in two different communication countermeasure scenarios.

[0015] The first step is specifically:

[0016] Assume that the communication confrontation scenario is fixed-frequency communication. The communicating party, that is, the other party, communicates with a fixed communication channel or a channel set according to a certain preset rule. The other party will switch the communication channel after being interfered. In order to achieve effective communication interference, the interfering party, that is, our party, needs to learn the change rule of this unknown communication channel through online learning.

[0017] We use single-frequency interference as the interference model, record the amplitude of the single-frequency interference signal as B, the continuous time as t, and the interference frequency of the interference signal as f j , the initial phase of the signal is φ0, and the circumference is pi, then the time domain expression of the signal is:

[0018] j(t)=Bcos(2×pi×f j t+φ0) (1)

[0019] The complex expression is

[0020]

[0021] Assume that the other party has n communication channels for communication, and these n channels meet the conditions of mutual independence and the same bandwidth. The other party's communication scenarios are divided into two:

[0022] Scenario 1: The other party's communication channel is unknown, but the communication channel changes over a long period of time. Even if it is interfered by us, it remains unchanged for a long time. The other party randomly selects one of the n communication channels for communication. We use the interference strategy to select one of the n communication channels for interference until the interfering party learns the communication channel.

[0023] Scenario 2: The other party's communication channel is unknown, but after being interfered with, it will switch to the next channel in a certain order to avoid interference. Assuming that the other party's channel switching strategy is a randomly generated set of random sequences, when our interference channel is the same as the other party's communication channel, the other party switches to the next channel according to the channel switching strategy. We continue to use the interference strategy to select channel interference until we learn the communication party's channel switching strategy;

[0024] The interference effect is evaluated by the overlap between the interference frequency and the target signal frequency, and the reward function is designed using the frequency overlap in the subsequent process;

[0025] The frequency coincidence is defined as follows:

[0026] Assume the frequency of the interference signal is f i , the target signal frequency is f c , then the frequency overlap ε t It is expressed as:

[0027] ε t =|f i -fc | / f c (3)

[0028] By analyzing expression (3), we can conclude that the closer the interference frequency is to the target signal frequency, the closer the frequency overlap is to 0, that is, the higher the frequency overlap is, the higher the overlap between the interference frequency and the interfered signal frequency is, and the better the interference effect is.

[0029] The second step is specifically as follows:

[0030] The learning process is abstracted as a Markov decision process (MDP), that is, the strategy depends entirely on the current state. The problem to be solved is represented as a six-tuple M = (S, A, P, R, γ, π), where S represents the state space, A represents the action space, P represents the state transition probability matrix, R represents the reward, γ is the reward attenuation factor and γ∈[0,1], and π represents the strategy; if the state of the agent in the Q-Learning learning process is s t ∈S, in the current state, perform action a t ∈A, change the current state to the next state s t+1 The probability of ∈S is p t ∈P;

[0031] According to the action of the agent, take action a t Rewards t+1 , and so on, the agent continues to choose the next action and obtains a new reward value. The ultimate goal is to learn an optimal strategy. The reciprocating changes in state and behavior constitute the main body of reinforcement learning. The sequence obtained in the MDP process is further described as:

[0032] s0,a0,r1,s1,a1,r2,s2,a2,r3,...,s n-1 ,a n-1 ,r n

[0033] The cumulative reward after the agent takes action under the optimal strategy π is expressed as:

[0034]

[0035] The goal of reinforcement learning is to maximize the cumulative reward, while the reward received by the agent from the environment is an immediate reward. In reinforcement learning, it is necessary to consider the current immediate reward and the possible reward in the future, and set the reward function according to scenario one and scenario two;

[0036] On the other hand, for environments with large decision spaces, the agent cannot traverse all actions and states. The agent continuously interacts with the environment and uses the feedback from the environment to traverse all actions and states as much as possible, thereby learning a better interference strategy and using the Q-Learning algorithm to solve the intelligent interference decision problem.

[0037] The core of the Q-Learning algorithm is the Q-value table, which adopts the ε-greedy strategy. The essence of the ε-greedy strategy is: generate a random number ρ from [0,1]. If ρ is less than ε, randomly select an action in the action space with a probability of ε, which means exploration; if ρ is greater than or equal to ε, select the optimal action corresponding to the Q-value table in the current state with a probability of 1-ε. The optimal action is the action with the largest Q value, which means utilization;

[0038] During the exploration process, the Q value update criterion is determined based on the exploration factor and reward function:

[0039] Q(s,a)←Q(s,a)+α[r+γmax a' Q(s',a')-Q(s,a)] (5)

[0040] Among them, ε is the exploration factor, γ is the reward decay factor, and α is the learning step size.

[0041] The third step is specifically as follows:

[0042] For scenario 1, assume that the communication channel of the other party has a long change cycle and the communication channel remains unchanged within a certain period of time, that is, the communication frequency of the other party remains unchanged. In this case, we optimize the interference frequency through continuous feedback of the interference effect.

[0043] Assume that the other party's communication frequency sequence is {f1,f2,...,f n}, the corresponding channel sequences are {1, 2, …, n}, then the state space S is {f1, f2, …, f n}, the action space A is {f1,f2,...,f n}, in the initial state s1, the other party's communication frequency is f i , we first use the frequency f j Implement interference, the environmental state at this time is s1={f j}, the frequency overlap is used to indicate the effectiveness of the interference implementation. If the frequency overlap is closer to 0, the effectiveness of the implemented communication interference is higher. Otherwise, the interference effect is not good. According to formula (3), the frequency overlap is:

[0044] ε1=|f i -f j | / f i (6)

[0045] Compare the relationship between ε1 and 0 to determine the next action and strategy;

[0046] During the learning process, the intelligent interference decision algorithm will give the communication channel that we should choose as the interference action at the next moment according to the interference effect fed back by the current state and execute it. After the interference is implemented, the effect of the interference implementation is evaluated, and the reward value obtained by taking the current action is recorded. The strategy adopted is optimized according to the reward value. According to the effectiveness of the interference implementation, the reward function is defined as:

[0047]

[0048] Since the interference frequency is the same as the target frequency, ε t There is a case where it is equal to 0, and the expression is meaningless at this time. Therefore, in order to avoid the situation where the formula is meaningless, a smaller number is added to the denominator. Here, 0.01 is chosen, so the reward function is corrected to:

[0049]

[0050] When the communication frequency of the communication party is successfully predicted, that is, ε t =0, the maximum reward value is 100;

[0051] According to equation (8), in state s1, the interferer chooses action f j The reward value is

[0052]

[0053] If ε1≠0 at this time, and the interference action at the next moment selected by the strategy is a1={f k}, so the next state is updated to s2 = {f k};

[0054] For scenario 2, it is assumed that the other party will switch to the next channel in a certain order after being interfered with, that is, after being interfered with, the other party's communication channel selects the communication channel at the next moment in a certain order; the interference frequency is optimized by continuous feedback of the interference effect. The range of the other party's communication channel sequence is set to {1, 2, ..., n}, and the corresponding communication frequency sequence corresponding to each channel is {f1, f2, ..., f n}, then the state space S is {f1,f2,...,f n}, the action space A is {f1,f2,...,f n}, the communication channel switching rule is {f i ,f j ,...,f k}(i,j,...,k∈{1,2,...,n}∩i≠j≠...≠k), in the initial state s1, the other party's communication frequency is f i , we first use the frequency f m Implement interference, the environmental state at this time is s1={f i ,f m}, the frequency overlap is also used to indicate the effectiveness of interference implementation;

[0055] In order to learn the other party's communication channel switching strategy as quickly as possible, the reward function is optimized. According to the effectiveness of the interference implementation, the restriction conditions for continuous interference are added, and the reward function is defined as:

[0056]

[0057] Where k is the continuous hit factor, k satisfies k=1 if and only if there are two consecutive successful interferences, and k=0 in other cases;

[0058] When the other party's communication channel is successfully interfered with twice in a row, an additional reward value of 5 is given. In state s1, if ε1 = 0 at this time, it means that the interference is successful in this state, and the other party switches to the next communication channel f according to the channel switching rule. j ; The next interference action selected by the strategy is a1 = {f p}, the state is updated to s2 = {f j ,f p}, in state s2, if ε2 = 0, it means that interference is also successful in this state, and the continuous interference condition is met at this time, so an additional reward value is given; when the continuous interference condition is met, the reward value in this state is 105.

[0059] The fourth step is specifically as follows:

[0060] According to the above specific learning process and reward function settings, a Q-Learning communication interference decision method based on ε time-varying is designed to achieve the optimal solution of the communication interference strategy. Under the premise of ensuring the interference hit rate, the cumulative reward value and hit rate convergence of the learning process are improved, and the number of learning times is reduced;

[0061] Scenario 1:

[0062] Design a function that makes the exploration factor ε decrease linearly with the number of rounds, and formulate a Q-Learning communication interference decision method based on the time-varying ε. The specific steps are as follows:

[0063] 1. Define the state space S and action space A in the environment;

[0064] 2. Initialize Q(s,a)=0,π(s,a)=current optimal strategy;

[0065] 3. Initialize ε, α, γ;

[0066] 4. Initialize state s, cumulative reward value R;

[0067] 5. Generate a random number ρ from [0,1]. If ρ < ε, randomly generate an interference frequency f from the action space; otherwise, let f = π(s) and select the current optimal action to execute;

[0068] 6. Update the reward value according to the reward function, wherein the reward function is expression (8);

[0069] 7. Update the Q value according to the state-action value function;

[0070] 8. Update status;

[0071] 9. Update the total reward value;

[0072] 10. Go to step 5 until the target state is reached and terminate the iteration;

[0073] 11. Update the ε value according to the formula and return to step 4 until the training round ends;

[0074] Scenario 2:

[0075] Design a function that makes the exploration factor ε decrease linearly with the number of rounds, and formulate a Q-Learning communication interference decision method based on the time-varying ε. The specific steps are as follows:

[0076] 1. Define the state space S and action space A in the environment;

[0077] 2. Initialize Q(s,a)=0,π(s,a)=current optimal strategy;

[0078] 3. Initialize ε, α, γ;

[0079] 4. Initialize state s, cumulative reward value R;

[0080] 5. Generate a random number ρ from [0,1]. If ρ < ε, randomly generate an interference frequency f from the action space; otherwise, let f = π(s) and select the current optimal action to execute;

[0081] 6. Update the reward value according to the reward function, wherein the reward function is expression (10);

[0082] 7. Update the Q value according to the state-action value function;

[0083] 8. Update status;

[0084] 9. Update the total reward value;

[0085] 10. Go to step 5 until the target state is reached and terminate the iteration;

[0086] 11. Update the ε value according to the formula and return to step 4 until the training round ends.

[0087] The fifth step is specifically as follows:

[0088] For scenario 1, when the other party's communication channel remains unchanged for a long time, simulation experiments are carried out according to the above algorithm steps. The interference effects of the communication interference decision method based on traditional Q-Learning and the communication interference decision method based on ε-time-varying Q-Learning in a simple scenario are compared from three aspects: the convergence of the number of learning times, the cumulative reward value, and the hit rate of successful interference:

[0089] Three different numerical curves are used to depict the number of learning rounds required to implement interference under the theoretical value, the traditional ε fixed condition, and the ε time-varying condition proposed by this method as the training rounds increase; by comparing the above numerical curves, it is found that under the ε time-varying condition, the convergence effect of the learning times of this method is better than that of the traditional algorithm;

[0090] Four different numerical curves are used to depict the changes in the cumulative reward value and the cumulative reward average value under the traditional ε fixed condition and the cumulative reward value and the cumulative reward average value under the ε time-varying condition proposed by this method as the number of training rounds increases. By comparing the above numerical curves, it is found that as the number of training rounds increases, the cumulative reward value and the cumulative reward average value under the ε time-varying condition will increase rapidly and exceed the value under the ε fixed condition.

[0091] Four different numerical curves are used to depict the changes in the hit rate and the average hit rate under the traditional ε fixed condition, and the hit rate and the average hit rate under the ε time-varying condition proposed by this method as the number of training rounds increases; by comparing the above numerical curves, it is concluded that as the number of training rounds increases, the hit rate and the average hit rate under the ε time-varying condition will increase rapidly and exceed the values ​​under the ε fixed condition.

[0092] For scenario 2, it is assumed that the communication party will switch to the next channel in a certain order after being interfered with, that is, after being interfered with, the communication channel selects the communication channel at the next moment in a certain order. According to the above algorithm steps, the simulation experiment is carried out by setting different channel numbers and channel switching orders of the other party:

[0093] In the first experiment, the number of communication channels of the other party was set to 10, and the switching order of the channels was randomly assumed;

[0094] The second experiment further increased the difficulty by setting the number of communication channels of the other party to 20 and also randomly assuming the channel switching order of the other party;

[0095] They were drawn:

[0096] In the first experiment, the number of learning times and the average number of learning times under the time-varying condition of ε as the number of training rounds increases;

[0097] In the first experiment, the cumulative reward value and the average cumulative reward value under the time-varying condition of ε as the number of training rounds increases;

[0098] Hit rate and average hit rate under time-varying ε condition as the number of training rounds increases in the first experiment;

[0099] In the second experiment, the number of learning times and the average number of learning times under the time-varying condition of ε as the number of training rounds increases;

[0100] In the second experiment, the cumulative reward value and the average cumulative reward value under the time-varying condition of ε as the number of training rounds increases;

[0101] Hit rate and average hit rate under time-varying ε condition as the number of training rounds increases in the second experiment;

[0102] For the two situations where the opponent's communication channel remains unchanged and the communication channel changes according to a preset rule, the reward function is optimized according to two different scenarios: the overlap between the channel frequency of our interference and the channel frequency of the opponent's communication is used as an indicator to design an optimization function; when the opponent's communication channel remains unchanged, this method adds a minimum value part to the denominator of the reward function to avoid meaningless processes without affecting the operation of the algorithm, thereby improving the robustness of the algorithm operation; when the opponent's communication channel changes according to a preset order, this method further optimizes the reward function and adds a continuous hit factor to reward successful interference for two consecutive times, so as to reduce the number of iterations in complex confrontation scenarios and increase the convergence speed; experiments are carried out according to the specific steps 1-11 of the communication interference decision implementation method based on ε time-varying Q-Learning in the fourth step.

[0103] Furthermore, the second step is characterized by exploration factor ε=0.1, learning step size α=0.3, reward decay factor γ=0.9, time-varying exploration factor expression ε=ε-0.0002·i, and reward function BRIEF DESCRIPTION OF THE DRAWINGS

[0104] Figure 1 The basic principle diagram of reinforcement learning is shown;

[0105] Figure 2 The overall implementation steps of the communication interference decision-making method based on reinforcement learning are shown;

[0106] Figure 3A flow chart of the Q-Learning communication interference decision procedure based on time-varying ε is shown;

[0107] Figure 4 The comparison of the learning times convergence curves of the traditional Q-Learning method and the present invention under the confrontation scenario 1 is shown;

[0108] Figure 5 The comparison of the cumulative reward value convergence curves of the traditional Q-Learning method and the present invention under the confrontation scenario 1 is shown;

[0109] Figure 6 The comparison of the hit rate convergence curves of the traditional Q-Learning method and the present invention under the confrontation scenario 1 is shown;

[0110] Figure 7 The figure shows the convergence curve of the learning times of the present invention for the switching sequence of 10 channels of the other party under the confrontation scenario 2;

[0111] Figure 8 The cumulative reward value convergence curve of the present invention is shown for the switching sequence of 10 channels of the other party under the confrontation scenario 2;

[0112] Fig. 9 The hit rate convergence curve of the present invention is shown for the switching sequence of 10 channels of the other party under the confrontation scenario 2;

[0113] Fig.10 The figure shows the convergence curve of the learning times of the present invention for the switching sequence of 20 channels of the other party under the confrontation scenario 2;

[0114] Fig.11 The figure shows the convergence curve of the cumulative reward value of the present invention for the switching sequence of 20 channels of the other party under the confrontation scenario 2;

[0115] Fig.12 The hit rate convergence curve of the present invention is shown for the switching sequence of 20 channels of the other party under the confrontation scenario 2; DETAILED DESCRIPTION

[0116] The present invention will be further described below in conjunction with embodiments and drawings.

[0117] Figure 2The overall implementation steps of the communication interference decision-making method based on reinforcement learning are shown. First, the communication confrontation scenario is modeled, and the evaluation indicators are determined according to the scenario to evaluate the interference effect; secondly, the reinforcement learning process is abstracted into a Markov process, and the state space, action space, state, action, and reward function are set; and the classic Q-Learning method in reinforcement learning is introduced to reduce the dependence on prior information, and the ε-greedy strategy is used to determine the Q value update criterion; then, the Markov process is instantiated according to different communication confrontation backgrounds and the specific expression of the reward function is set; further, the specific steps of the Q-Learning communication interference decision-making method based on ε time-varying are designed; finally, the communication interference implementation in two different scenarios is verified by experimental simulation.

[0118] Figure 3 The flowchart of the Q-Learning communication interference decision procedure based on time-varying ε is shown. The specific algorithm steps are as follows:

[0119] 1. Define the state space S and action space A in the environment;

[0120] 2. Initialize Q(s,a)=0,π(s,a)=current optimal strategy;

[0121] 3. Initialize ε, α, γ;

[0122] 4. Initialize state s, cumulative reward value R

[0123] 5. Generate a random number ρ from [0,1],

[0124] If ρ<ε, a disturbance frequency f is randomly generated from the action space;

[0125] Otherwise, let f = π(s) and select the current optimal action to execute;

[0126] 6. Update the reward value according to the reward function;

[0127] 7. Update the Q value according to the state-action value function;

[0128] 8. Update status;

[0129] 9. Update the total reward value;

[0130] 10. Go to step 5 until the target state is reached and terminate the iteration;

[0131] 11. Update the ε value according to the formula and return to step 4 until the training round ends.

[0132] Example:

[0133] Assume that initial Q(s,a)=0, π(s,a)=current optimal strategy, initial exploration factor ε=0.1, learning step size α=0.3, reward decay factor γ=0.9;

[0134] The communication interference decision-making method based on reinforcement learning is specifically as follows:

[0135] Step 1: Establish a communication adversarial scenario model, determine indicators based on the scenario to evaluate the jamming effect, and establish a learning process;

[0136] Set up two communication confrontation scenarios:

[0137] Scenario 1 assumes that the change cycle of the other party's communication channel is relatively long, and the communication channel remains unchanged within a certain period of time, that is, the other party's communication frequency remains unchanged.

[0138] Scenario 2 assumes that the other party will switch to the next channel in a certain order after being interfered with, that is, after being interfered with, the communication channel of the communicating party selects the communication channel at the next moment in a certain order.

[0139] The present invention focuses on studying how to implement tracking interference of communication channels without considering too much the specific form of the communication signal, and therefore considers using the simplest single-frequency interference as the interference model.

[0140] Step 2: Abstract the learning process into a Markov process and use the ε-greedy strategy to determine the Q value update criterion

[0141] Reinforcement learning can be abstracted as a Markov decision process (MDP), that is, the strategy depends entirely on the current state. The problem to be solved is represented as a sextuple M = (S, A, P, R, γ, π), where S represents the state space, A represents the action space, P represents the state transition probability matrix, R represents the reward, γ is the reward attenuation factor and γ∈[0,1], and π represents the strategy.

[0142] In different confrontation scenarios or in the context of different channel switching strategies of the opponent in the same confrontation scenario, the corresponding state space, action space, and reward function have been instantiated; the state transition probability matrix represents the agent in the current state s t ∈S, perform action a t ∈A, change the current state to the next state s t+1 ∈S. The reward decay factor γ is set to 0.9, and the initialization strategy π(s,a) = the current optimal strategy.

[0143] The core of the Q-Learning algorithm is the Q-value table, which records the maximum reward value obtained by taking different actions in each state, so as to facilitate the selection of the optimal action in the corresponding state. When selecting an action, the initial strategy is generally to randomly select an action, but such exploration is irregular and cannot make good use of the optimal action explored. Therefore, in order to balance the relationship between exploration and utilization, consider immediate rewards and future rewards, and avoid the situation of local optimal solutions, the ε-greedy strategy is usually adopted. That is, randomly selecting an action in the action space with a probability of ε, which means "exploration"; selecting the corresponding optimal action in the Q-value table in the current state with a probability of 1-ε. The optimal action is the action with the largest Q value, which means "utilization".

[0144] During the exploration process, the Q value is updated according to formula (5), where Q(s, a) is initialized to 0, ε is the exploration factor, and the instantiation value is 0.1; γ is the reward decay factor, and the instantiation value is 0.9; α is the learning step size, and the instantiation value is 0.3.

[0145] Therefore, the initial Q value update formula for this example is:

[0146] Q(s,a)←0+0.3[r+0.9max' a′ Q(s′,a′)] (11)

[0147] Step 3: Specify the Markov process and set the reward function according to different communication adversarial backgrounds

[0148] Assume that the other party's communication frequency is f = 10GHz, and the corresponding channel sequence is {1, 2, ..., 5}, then the state space S is {f1, f2, ..., f5 = 10GHz}, and the action space A is {f1, f2, ..., f5}. In the initial state s1, we first use the frequency f j Implement interference, the environmental state at this time is s1={f j The frequency overlap is used to indicate the effectiveness of interference implementation. If the frequency overlap is closer to 0, the effectiveness of the implemented communication interference is higher. Otherwise, the interference effect is not good. According to formula (6), the frequency overlap is:

[0149] ε1=|f1-f j | / f1 (12)

[0150] Compare the relationship between ε1 and 0 to determine whether effective interference is implemented.

[0151] According to formula (8), in state s1, we choose action f j The reward value is:

[0152]

[0153] When the communication channel is 10 channels, it is assumed that the communicating parties communicate in the following channel switching order: [4, 2, 6, 7, 1, 0, 9, 8, 3, 5]; the communication channel sequence range of the other party is {1, 2, ..., 10}, and the corresponding communication frequency sequence corresponding to each channel is {f1, f2, ..., f 10}, then the state space S is {f1,f2,...,f 10}, the action space A is {f1,f2,...,f1...,f10}0}, and the channel switching rule is: {f i ,f j ,...,f k}(i,j,...,k∈{1,2,...,10}∩i≠j≠...≠k).

[0154] When the communication channel is 20 channels, it is assumed that the communicating parties communicate in the following channel switching order: [7, 16, 4, 5, 11, 17, 0, 8, 1, 13, 18, 15, 9, 2, 19, 10, 14, 6, 12, 3]. The communication channel sequence of the other party ranges from {1, 2, ..., 20}, and the corresponding communication frequency sequence corresponding to each channel is {f1, f2, ..., f 20}, then the state space S is {f1,f2,...,f 20}, the action space A is {f1,f2,...,f 20}. The communication channel switching rule is {f i ,f j ,...,f k}(i,j,...,k∈{1,2,...,20}∩i≠j≠...≠k).

[0155] In order to learn the other party's communication channel switching strategy as quickly as possible, the reward function is optimized. According to the effectiveness of the interference implementation, the restriction condition for continuous interference is added, and the reward function is defined as:

[0156]

[0157] Where k is the continuous hit factor, k satisfies k=1 if and only if there are two consecutive successful interferences, and k=0 in other cases.

[0158] Step 4: Design a Q-Learning communication interference decision method based on ε time-varying Specific steps

[0159] According to the above specific learning process and reward function settings, we design a Q-Learning communication interference decision-making method based on ε time-varying to achieve the optimal solution for the communication interference strategy. On the premise of ensuring the interference hit rate, we improve the cumulative reward value and hit rate convergence of the learning process and reduce the number of learning times.

[0160] In order to avoid the imbalance between exploration and utilization in the process of obtaining the optimal strategy, the present invention comprehensively considers the current immediate reward and the possible reward in the future, and makes the value of the exploration factor ε decrease with the continuous increase of the training rounds. Its expression is:

[0161] ε=ε-0.0002·i (15)

[0162] The advantage of this is that a larger initial exploration factor ε can explore the action space more fully, which is conducive to the use of the optimal action when the subsequent exploration factor ε is smaller, and is conducive to the convergence of the algorithm.

[0163] According to formula (15), the exploration factor ε increases with each training round and decays by 0.0002 each round until it decays to 0 in the 500th round.

[0164] Therefore, when ε=0, the final algorithm will become a pure utilization form, continuously utilizing the existing optimal strategy to select the optimal strategy under the current state.

[0165] The specific steps of the Q-Learning communication interference decision method based on time-varying ε are finally formulated as follows:

[0166]

[0167] Specifically, scenario 2 still continues the above steps, but adjusts and optimizes the reward function.

[0168] In the specific implementation process, for scenario 1, the reward function in the specific step 6 of the interference method uses formula (8); for scenario 2, the reward function in the specific step 6 of the interference method uses formula (10); compared with formula (8), the reward function of scenario 2, i.e., formula (10), adds a continuous hit factor to reward successful interference for two consecutive times, so as to reduce the number of iterations in complex confrontation scenarios and increase the convergence speed.

[0169] Step 5: Conduct experimental simulation on communication interference implementation in two different scenarios to verify the advantages of the basic invention from multiple aspects.

[0170] For scenario 1, when the other party's communication channel remains unchanged for a long time, simulation experiments are carried out according to the above algorithm steps. The interference effects of the communication interference decision method based on traditional Q-Learning and the communication interference decision method based on ε-time-varying Q-Learning in a simple scenario are compared from three aspects: the convergence of the number of learning times, the cumulative reward value, and the hit rate of successful interference:

[0171] Three numerical curves of different colors are used to depict the number of learning rounds required for implementing interference under the theoretical value, the traditional ε fixed condition, and the ε time-varying condition proposed by the present invention as the number of training rounds increases; Figure 4 It can be concluded that under the condition of time-varying ε, the convergence effect of the learning times of the present invention is better than that of the traditional algorithm;

[0172] Four numerical curves of different colors are used to depict the changes of the cumulative reward value and the cumulative reward average value under the traditional ε fixed condition and the cumulative reward value and the cumulative reward average value under the ε time-varying condition proposed by the present invention as the number of training rounds increases. Figure 5 It can be concluded that as the number of training rounds increases, the cumulative reward value and the average value of the cumulative reward under the condition of ε varying will increase rapidly and exceed the value under the condition of ε being fixed;

[0173] Four numerical curves of different colors are used to depict the changes of the hit rate and the average hit rate under the traditional ε fixed condition and the hit rate and the average hit rate under the ε time-varying condition proposed by the present invention as the training rounds increase. Figure 6 It can be concluded from the comparison that with the increase of training rounds, the hit rate and the average hit rate under the condition of time-varying ε will increase rapidly and exceed the values ​​under the condition of fixed ε.

[0174] For scenario 2, it is assumed that the communication party will switch to the next channel in a certain order after being interfered with, that is, after being interfered with, the communication channel selects the communication channel at the next moment in a certain order. According to the above algorithm steps, the simulation experiment is carried out by setting different channel numbers and channel switching orders of the other party:

[0175] In the first experiment, the number of communication channels of the other party was set to 10, and the switching order of the channels was randomly assumed;

[0176] The second experiment further increased the difficulty by setting the number of communication channels of the other party to 20 and also randomly assuming the channel switching order of the other party;

[0177] They were drawn:

[0178] In the first experiment, as the number of training rounds increases, the number of learning times and the average number of learning times under the time-varying condition of ε are obtained. Figure 7 Shown;

[0179] In the first experiment, as the number of training rounds increases, the cumulative reward value and the average cumulative reward value under the time-varying condition of ε are calculated. Figure 8 Shown;

[0180] In the first experiment, the hit rate and the average hit rate under the time-varying condition of ε as the number of training rounds increases are obtained by Fig. 9 Shown;

[0181] In the second experiment, as the number of training rounds increases, the number of learning times and the average number of learning times under the time-varying condition of ε are obtained. Fig.10 Shown;

[0182] In the second experiment, as the number of training rounds increases, the cumulative reward value and the average cumulative reward value under the time-varying condition of ε are obtained. Fig.11 Shown;

[0183] In the second experiment, the hit rate and the average hit rate under the time-varying condition of ε as the number of training rounds increases are obtained through Fig.12 Shown.

[0184] In view of the two situations that the communication channel of the other party remains unchanged and the communication channel changes according to a preset rule, the reward function is optimized according to two different scenarios: the overlap between the channel frequency of the interference implemented by our party and the channel frequency of the communication implemented by the other party is used as an indicator to design an optimization function; when the communication channel of the other party remains unchanged, the present invention adds a minimum value part to the denominator of the reward function to avoid meaningless processes without affecting the operation of the algorithm, thereby improving the robustness of the algorithm operation, which can be referred to in the above formula (8); when the communication channel of the other party changes according to a preset order, the present invention further optimizes the reward function and adds a continuous hit factor to reward successful interference for two consecutive times, so as to reduce the number of iterations in complex confrontation scenarios and increase the convergence speed, which can be referred to in the above formula (10) and carry out experiments according to the specific steps 1-11 of the communication interference decision implementation method based on ε time-varying Q-Learning.

[0185] For scenario one, when the other party’s communication channel remains unchanged for a long time, a simulation experiment is carried out for a fixed frequency of f=10GHz according to the above-mentioned instantiation settings. The interference effects of the communication interference decision method based on traditional Q-Learning in a simple scenario and the communication interference decision method based on ε-time-varying Q-Learning are compared from three aspects: the convergence of the number of learning times, the cumulative reward value, and the hit rate of successful interference.

[0186] pass Figure 4 From the comparison of the algorithms, it can be seen that the exploration factor ε decreases as the number of training rounds increases, which can enhance the interference effect. The convergence effect of the learning number of the algorithm with time-varying ε is better than that of the traditional algorithm. Figure 5 and Figure 6 It shows that as the number of training rounds increases, the cumulative reward value and hit rate of the ε-time-varying algorithm will increase rapidly, and will soon reach stability, eventually reaching the maximum reward value and hit rate. Therefore, the unknown signal interference frequency learning algorithm based on ε-time-varying Q-Learning can learn the optimal interference strategy.

[0187] For scenario 2, it is assumed that the communication party will switch to the next channel in a certain order after being interfered, that is, after being interfered, the communication channel selects the communication channel at the next moment in a certain order. According to the above instantiation settings, simulation experiments are carried out for the other party's 10 channels and their switching order, and 20 channels and their switching order.

[0188] from Figure 7 and Fig.10 It can be seen that as the number of explorations increases, the number of interferences decreases, and the interfering party can find the communication frequency of the communicating party more quickly with better results, and finally converges to 1 time, with better learning effect. Figure 8 , Fig. 9 , Fig.11 and Fig.12 , it is concluded that with the increase of exploration times, the maximum cumulative reward value and hit rate tend to converge, and the maximum cumulative reward value is large, and the hit rate can reach 100%, which achieves effective interference well.

[0189] The present invention proposes a communication interference decision-making method based on reinforcement learning, including the following steps: establishing a communication confrontation scenario model, determining indicators based on the scenario to evaluate the interference effect; abstracting the learning process into a Markov process, and using the ε-greedy strategy to determine the Q value update criterion; concretizing the Markov process and setting the reward function according to different communication confrontation backgrounds; designing the specific steps of the Q-Learning communication interference decision-making method based on ε time-varying; experimentally simulating the implementation of communication interference in two different scenarios, and verifying the advantages of the basic invention from multiple aspects. The present invention can obtain the best decision for our interference implementation under the condition of communication confrontation by solving the algorithm strategy, complete the tracking and interference of the other party's unknown signal, and better optimize the defects of traditional algorithms in multiple aspects.

Claims

1. A communication interference decision method based on reinforcement learning, characterized in that: The specific steps include: Step 1: Establish a communication adversarial scenario model, determine indicators based on the scenario to evaluate the jamming effect, and establish a learning process; Step 2: Abstract the learning process into a Markov process and use the ε-greedy strategy to determine the Q value update criterion; Step 3: Specify the Markov process and set the reward function according to two different communication confrontation scenarios; Step 4: Design the specific steps of the Q-Learning communication interference decision method based on ε time-varying; Step 5: Experimentally simulate and verify the implementation of communication interference in two different communication countermeasure scenarios.

2. The communication interference decision-making method based on reinforcement learning according to claim 1, characterized in that: The first step is specifically: Assume that the communication confrontation scenario is fixed-frequency communication. The communicating party, that is, the other party, communicates with a fixed communication channel or a channel set according to a certain preset rule. The other party will switch the communication channel after being interfered. In order to achieve effective communication interference, the interfering party, that is, our party, needs to learn the change rule of this unknown communication channel through online learning. We use single-frequency interference as the interference model, record the amplitude of the single-frequency interference signal as B, the continuous time as t, and the interference frequency of the interference signal as f j , the initial phase of the signal is φ0, and the circumference is pi, then the time domain expression of the signal is: j(t)=Bcos(2×pi×f j t+φ0) (1) The complex expression is Assume that the other party has n communication channels for communication, and these n channels meet the conditions of mutual independence and the same bandwidth. The other party's communication scenarios are divided into two: Scenario 1: The other party's communication channel is unknown, but the communication channel changes over a long period of time. Even if it is interfered by us, it remains unchanged for a long time. The other party randomly selects one of the n communication channels for communication. We use the interference strategy to select one of the n communication channels for interference until the interfering party learns the communication channel. Scenario 2: The other party's communication channel is unknown, but after being interfered with, it will switch to the next channel in a certain order to avoid interference. Assuming that the other party's channel switching strategy is a randomly generated set of random sequences, when our interference channel is the same as the other party's communication channel, the other party switches to the next channel according to the channel switching strategy. We continue to use the interference strategy to select channel interference until we learn the communication party's channel switching strategy; The interference effect is evaluated by the overlap between the interference frequency and the target signal frequency, and the reward function is designed using the frequency overlap in the subsequent process; The frequency coincidence is defined as follows: Assume the frequency of the interference signal is f i , the target signal frequency is f c , then the frequency overlap ε t It is expressed as: ε t =|f i -f c | / f c (3) By analyzing expression (3), we can conclude that the closer the interference frequency is to the target signal frequency, the closer the frequency overlap is to 0, that is, the higher the frequency overlap is, the higher the overlap between the interference frequency and the interfered signal frequency is, and the better the interference effect is.

3. The communication interference decision-making method based on reinforcement learning according to claim 1, characterized in that: The second step is specifically as follows: The learning process is abstracted into a Markov decision process (MDP), that is, the strategy depends entirely on the current state, and the problem to be solved is represented as a six-tuple M = (S, A, P, R, γ, π), where S represents the state space, A represents the action space, P represents the state transition probability matrix, R represents the reward, γ is the reward attenuation factor and γ∈[0,1], and π represents the strategy; If the state of the agent in the Q-Learning process is s t ∈S, in the current state, perform action a t ∈A, change the current state to the next state s t+1 The probability of ∈S is p t ∈P; According to the action of the agent, take action a t Rewards t+1 , and so on, the agent continues to choose the next action and obtains a new reward value. The ultimate goal is to learn an optimal strategy. The reciprocating changes in state and behavior constitute the main body of reinforcement learning. The sequence obtained in the MDP process is further described as: s0,a0,r1,s1,a1,r2,s2,a2,r3,...,s n-1 ,a n-1 ,r n The cumulative reward after the agent takes action under the optimal strategy π is expressed as: The goal of reinforcement learning is to maximize the cumulative reward, while the reward received by the agent from the environment is an immediate reward. In reinforcement learning, it is necessary to consider the current immediate reward and the possible reward in the future, and set the reward function according to scenario one and scenario two; On the other hand, for environments with large decision spaces, the agent cannot traverse all actions and states. The agent continuously interacts with the environment and uses the feedback from the environment to traverse all actions and states as much as possible, thereby learning a better interference strategy and using the Q-Learning algorithm to solve the intelligent interference decision problem. The core of the Q-Learning algorithm is the Q-value table, which adopts the ε-greedy strategy. The essence of the ε-greedy strategy is: generate a random number ρ from [0,1]. If ρ is less than ε, randomly select an action in the action space with a probability of ε, which means exploration; if ρ is greater than or equal to ε, select the optimal action corresponding to the Q-value table in the current state with a probability of 1-ε. The optimal action is the action with the largest Q value, which means utilization; During the exploration process, the Q value update criterion is determined based on the exploration factor and reward function: Q(s,a)←Q(s,a)+α[r+γmax a' Q(s',a')-Q(s,a)] (5) Among them, ε is the exploration factor, γ is the reward decay factor, and α is the learning step size.

4. The communication interference decision-making method based on reinforcement learning according to claim 1, characterized in that: The third step is specifically as follows: For scenario 1, assume that the communication channel of the other party has a long change cycle and the communication channel remains unchanged within a certain period of time, that is, the communication frequency of the other party remains unchanged. In this case, we optimize the interference frequency through continuous feedback of the interference effect. Assume that the other party's communication frequency sequence is {f1,f2,...,f n }, the corresponding channel sequences are {1, 2, ..., n}, then the state space S is {f1, f2, ..., f n }, the action space A is {f1,f2,...,f n }, in the initial state s1, the other party's communication frequency is f i , we first use the frequency f j Implement interference, the environmental state at this time is s1={f j }, the frequency overlap is used to indicate the effectiveness of the interference implementation. If the frequency overlap is closer to 0, the effectiveness of the implemented communication interference is higher. Otherwise, the interference effect is not good. According to formula (3), the frequency overlap is: ε1=|f i -f j | / f i (6) Compare the relationship between ε1 and 0 to determine the next action and strategy; During the learning process, the intelligent interference decision algorithm will give the communication channel that we should choose as the interference action at the next moment according to the interference effect fed back by the current state and execute it. After the interference is implemented, the effect of the interference implementation is evaluated, and the reward value obtained by taking the current action is recorded. The strategy adopted is optimized according to the reward value. According to the effectiveness of the interference implementation, the reward function is defined as: Since the interference frequency is the same as the target frequency, ε t There is a case where it is equal to 0, and the expression is meaningless at this time. Therefore, in order to avoid the situation where the formula is meaningless, a smaller number is added to the denominator. Here, 0.01 is chosen, so the reward function is corrected to: When the communication frequency of the communication party is successfully predicted, that is, ε t =0, the maximum reward value is 100; According to equation (8), in state s1, the interferer chooses action f j The reward value is If ε1≠0 at this time, and the interference action at the next moment selected by the strategy is a1={f k }, so the next state is updated to s2 = {f k }; For scenario 2, it is assumed that the other party will switch to the next channel in a certain order after being interfered with, that is, after being interfered with, the other party's communication channel selects the communication channel at the next moment in a certain order; the interference frequency is optimized by continuous feedback of the interference effect. The range of the other party's communication channel sequence is set to {1, 2, ..., n}, and the corresponding communication frequency sequence corresponding to each channel is {f1, f2, ..., f n }, then the state space S is {f1,f2,...,f n }, the action space A is {f1,f2,...,f n }, the communication channel switching rule is {f i ,f j ,...,f k }(i,j,...,k∈{1,2,...,n}∩i≠j≠...≠k), in the initial state s1, the other party's communication frequency is f i , we first use the frequency f m Implement interference, the environmental state at this time is s1={f i ,f m }, the frequency overlap is also used to indicate the effectiveness of interference implementation; In order to learn the other party's communication channel switching strategy as quickly as possible, the reward function is optimized. According to the effectiveness of the interference implementation, the restriction conditions for continuous interference are added, and the reward function is defined as: Where k is the continuous hit factor, k satisfies k=1 if and only if there are two consecutive successful interferences, and k=0 in other cases; When the other party's communication channel is successfully interfered with twice in a row, an additional reward value of 5 is given. In state s1, if ε1 = 0 at this time, it means that the interference is successful in this state, and the other party switches to the next communication channel f according to the channel switching rule. j ; The next interference action selected by the strategy is a1 = {f p }, the state is updated to s2 = {f j ,f p }, in state s2, if ε2 = 0, it means that interference is also successful in this state, and the continuous interference condition is met at this time, so an additional reward value is given; when the continuous interference condition is met, the reward value in this state is 105.

5. The communication interference decision-making method based on reinforcement learning according to claim 1, characterized in that: The fourth step is specifically as follows: According to the above specific learning process and reward function settings, a Q-Learning communication interference decision method based on ε time-varying is designed to achieve the optimal solution of the communication interference strategy. Under the premise of ensuring the interference hit rate, the cumulative reward value and hit rate convergence of the learning process are improved, and the number of learning times is reduced; Scenario 1: Design a function that makes the exploration factor ε decrease linearly with the number of rounds, and formulate a Q-Learning communication interference decision method based on the time-varying ε. The specific steps are as follows:

1. Define the state space S and action space A in the environment; 2. Initialize Q(s,a)=0,π(s,a)=current optimal strategy; 3. Initialize ε, α, γ; 4. Initialize state s, cumulative reward value R; 5. Generate a random number ρ from [0,1]. If ρ < ε, randomly generate an interference frequency f from the action space; otherwise, let f = π(s) and select the current optimal action to execute; 6. Update the reward value according to the reward function, wherein the reward function is expression (8); 7. Update the Q value according to the state-action value function; 8. Update status; 9. Update the total reward value; 10. Go to step 5 until the target state is reached and terminate the iteration; 11. Update the ε value according to the formula and return to step 4 until the training round ends; Scenario 2: Design a function that makes the exploration factor ε decrease linearly with the number of rounds, and formulate a Q-Learning communication interference decision method based on the time-varying ε. The specific steps are as follows:

1. Define the state space S and action space A in the environment; 2. Initialize Q(s,a)=0,π(s,a)=current optimal strategy; 3. Initialize ε, α, γ; 4. Initialize state s, cumulative reward value R; 5. Generate a random number ρ from [0,1]. If ρ < ε, randomly generate an interference frequency f from the action space; otherwise, let f = π(s) and select the current optimal action to execute; 6. Update the reward value according to the reward function, wherein the reward function is expression (10); 7. Update the Q value according to the state-action value function; 8. Update status; 9. Update the total reward value; 10. Go to step 5 until the target state is reached and terminate the iteration; 11. Update the ε value according to the formula and return to step 4 until the training round ends.

6. The communication interference decision-making method based on reinforcement learning according to claim 1, characterized in that: The fifth step is specifically as follows: For scenario 1, when the other party's communication channel remains unchanged for a long time, simulation experiments are carried out according to the above algorithm steps. The interference effects of the communication interference decision method based on traditional Q-Learning and the communication interference decision method based on ε-time-varying Q-Learning in a simple scenario are compared from three aspects: the convergence of the number of learning times, the cumulative reward value, and the hit rate of successful interference: Three different numerical curves are used to depict the number of learning rounds required to implement interference under the theoretical value, the traditional ε fixed condition, and the ε time-varying condition proposed by this method as the training rounds increase; by comparing the above numerical curves, it is found that under the ε time-varying condition, the convergence effect of the learning times of this method is better than that of the traditional algorithm; Four different numerical curves are used to depict the changes in the cumulative reward value and the cumulative reward average value under the traditional ε fixed condition and the cumulative reward value and the cumulative reward average value under the ε time-varying condition proposed by this method as the number of training rounds increases. By comparing the above numerical curves, it is found that as the number of training rounds increases, the cumulative reward value and the cumulative reward average value under the ε time-varying condition will increase rapidly and exceed the value under the ε fixed condition. Four different numerical curves are used to depict the changes in the hit rate and the average hit rate under the traditional ε fixed condition, and the hit rate and the average hit rate under the ε time-varying condition proposed by this method as the number of training rounds increases; by comparing the above numerical curves, it is concluded that as the number of training rounds increases, the hit rate and the average hit rate under the ε time-varying condition will increase rapidly and exceed the values ​​under the ε fixed condition. For scenario 2, it is assumed that the communication party will switch to the next channel in a certain order after being interfered with, that is, after being interfered with, the communication channel selects the communication channel at the next moment in a certain order. According to the above algorithm steps, the simulation experiment is carried out by setting different channel numbers and channel switching orders of the other party: In the first experiment, the number of communication channels of the other party was set to 10, and the switching order of the channels was randomly assumed; The second experiment further increased the difficulty by setting the number of communication channels of the other party to 20 and also randomly assuming the channel switching order of the other party; They were drawn: In the first experiment, the number of learning times and the average number of learning times under the time-varying condition of ε as the number of training rounds increases; In the first experiment, the cumulative reward value and the average cumulative reward value under the time-varying condition of ε as the number of training rounds increases; Hit rate and average hit rate under time-varying ε condition as the number of training rounds increases in the first experiment; In the second experiment, the number of learning times and the average number of learning times under the time-varying condition of ε as the number of training rounds increases; In the second experiment, the cumulative reward value and the average cumulative reward value under the time-varying condition of ε as the number of training rounds increases; Hit rate and average hit rate under time-varying ε condition as the number of training rounds increases in the second experiment; For the two situations where the opponent's communication channel remains unchanged and the communication channel changes according to a preset rule, the reward function is optimized according to two different scenarios: the overlap between the channel frequency of our interference and the channel frequency of the opponent's communication is used as an indicator to design an optimization function; when the opponent's communication channel remains unchanged, this method adds a minimum value part to the denominator of the reward function to avoid meaningless processes without affecting the operation of the algorithm, thereby improving the robustness of the algorithm operation; when the opponent's communication channel changes according to a preset order, this method further optimizes the reward function and adds a continuous hit factor to reward successful interference for two consecutive times, so as to reduce the number of iterations in complex confrontation scenarios and increase the convergence speed; experiments are carried out according to the specific steps 1-11 of the communication interference decision implementation method based on ε time-varying Q-Learning in the fourth step.

7. A communication interference decision method based on reinforcement learning as claimed in claim 3, characterized in that: Exploration factor ε = 0.1, learning step size α = 0.3, reward decay factor γ = 0.9, time-varying exploration factor expression ε = ε-0.0002·i, reward function

Images