A reinforcement learning communication anti-interference method based on feature engineering

By building a reinforced learning environment in a communication interference environment, combining feature engineering and greedy algorithms, anti-interference strategies are designed, and problems of large amount of computation and poor anti-interference effect in the existing technology are solved, and efficient communication anti-interference effect is achieved. Reliable technical support to combat malicious interference.

CN114978407BActive Publication Date: 2024-11-26THE 54TH RESEARCH INSTITUTE OF CHINA ELECTRONICS TECHNOLOGY GROUP CORPORATION
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210532452.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-05-12
Publication Date
2024-11-26
Estimated Expiration
2042-05-12

AI Technical Summary

Technical Problem

The existing communication anti-interference method based on reinforcement learning has problems such as large computational volume, complex system structure, poor incremental anti-interference effect and a large amount of manpower participation, and has failed to effectively consider the relationship between pre and post timing behaviors.

Method used

By building a communication interference environment in the time domain and frequency domain, combining feature engineering, building a reinforcement learning environment, designing state-action value table and reward function, using greedy algorithms to select the optimal action, update the value table, and realize the generation and generation of anti-interference strategies and verify.

Benefits of technology

It improves the information transmission effect of the communication network, provides reliable technical support to fight malicious interference, and can effectively avoid random switching interference of intelligent jammers, and the anti-interference success rate reaches more than 99.1%.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114978407B_ABST
    Figure CN114978407B_ABST
Patent Text Reader

Abstract

The present invention provides a reinforcement learning communication anti-interference method based on feature engineering, the method comprising the following steps: constructing a communication interference environment in the time domain and the frequency domain; constructing a reinforcement learning environment based on the interference environment and feature engineering; constructing a reinforcement learning communication interference method model based on the reinforcement learning environment; and verifying the reinforcement learning communication anti-interference method model based on the interference environment. The method of the present invention has a clear structure and strong interpretability, can effectively avoid periodic random interference style switching interference, and has wide industrial applicability.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a reinforcement learning communication anti-interference method based on feature engineering, and belongs to the intersection field of reinforcement learning and intelligent communication. Background Art

[0002] Communication anti-jamming technology is a key factor in protecting our communication security, affecting the transmission effect of real-time command information and the construction of battlefield situation. At present, the shortcomings of the anti-jamming method based on reinforcement learning are mainly reflected in the problem of large amount of calculation. In the article "Pattern-aware intelligent anti-jamming communication: a sequential deep reinforcement learning approach", deep learning is first used to identify the interference pattern of the current jammer, and then different anti-jamming strategies are trained for different interference patterns. This method divides the anti-jamming system into two parts, and the series connection will lead to a long decision time. In the article "Anti-jamming communications using spectrum waterfall: A deep reinforcement learning approach", the method of combining convolutional neural networks (CNN) and recurrent neural networks (RNN) will lead to a complex system structure and a long RNN training time. The patent "Method, communication equipment and storage medium for improving the anti-interference ability of wireless communication" with publication number CN113938239A adopts the method of manually designing anti-interference rules. Although this method proposes a perfect and systematic combination of hardware design, the incremental anti-interference effect is poor and its application requires a lot of manpower participation. The patent "A satellite communication anti-interference method based on zero-sum game theory" with publication number CN113472423A establishes a zero-sum game model with communication capacity and bit error rate as optimization targets, and generates a joint strategy with parameters such as power and waveform transformation. This method statically models the adversarial model in advance, but does not consider the impact of the correlation between the previous and subsequent timing behaviors on the strategy. Summary of the invention

[0003] In view of this, the present invention provides a reinforcement learning communication anti-interference method based on feature engineering, which can be used to combat malicious interference in communication networking, provide new ideas and methods for generating anti-interference strategies, improve the information transmission effect of communication networks, and provide reliable technical support for the construction of command and control situation.

[0004] The technical solution adopted by the present invention is:

[0005] A reinforcement learning communication anti-interference method based on feature engineering, comprising the following steps:

[0006] Step 1: Construct the communication interference environment in the time domain and frequency domain;

[0007] Step 2: Build a reinforcement learning environment based on interference environment and feature engineering;

[0008] Step 3: Construct a reinforcement learning communication anti-interference method model based on the reinforcement learning environment;

[0009] Step 4: Verify the reinforcement learning communication anti-interference method model based on the interference environment.

[0010] Furthermore, the specific method of step 1 is:

[0011] Step 1-1: At time t, the number of communication channels on the entire communication frequency band is N. The entire communication frequency band at time t is represented by matrix C as follows:

[0012] C t =[c t,1 c t,2 …c t,n ]

[0013] Among them, when there is no communication signal and interference signal in the channel, c t,n =0; when there is no communication signal in the channel but only interference signal, c t,n =-1; when there is only communication signal in the channel and no interference signal, c t,n =1; when there are both communication signals and interference signals in the channel, c t,n =2;

[0014] Step 1-2: The time-frequency information of the communication frequency band in the entire T time in the communication interference environment is O t It is expressed as:

[0015]

[0016] Step 1-3: In the game environment, the jammer adopts a random jamming pattern, that is, the jamming pattern is randomly switched with a period of T; the jamming patterns adopted by the jammer are fixed frequency jamming and swept frequency jamming.

[0017] Furthermore, the specific method of step 2 is:

[0018] Step 2-1: Construct a communication anti-interference environment reward function:

[0019]

[0020] Among them, a t,j represents the channel selected by the jammer at time t, at,c represents the channel selected by the communication device at time t, λ is the discount coefficient, which is 0.6;

[0021] Step 2-2: Construct the communication anti-interference environment state space and express the state at time t as:

[0022] s t =[C t ,C t-1 ].

[0023] Furthermore, the specific method of step 3 is:

[0024] Step 3-1: Construct a state-action value table. The value table is represented by the matrix Q as follows:

[0025]

[0026] The dimension of the matrix Q is M×N, the number of rows M is the type of state, the number of columns N is the number of channels, and the value q(s1′,a1) represents the value that can be obtained by selecting a1 when the state type is s1′, that is:

[0027] Q(s′ m ,a n ) = q(s′) m ,a n )

[0028] Step 3-2: Construct a greedy algorithm to select the current state s t The optimal action value a under t ; During the training process, the formula for selecting the optimal action is as follows:

[0029]

[0030] Where random(n) means selecting an integer with uniform probability in the closed interval [1,N]; ε means a random number in the interval [0,1]; η means the threshold; argmaxQ(s t ) means that in the value matrix, choose s t The action with the highest value in the state;

[0031] During the verification process, the formula for selecting the optimal action is as follows:

[0032] a t = argmaxQ(s t )

[0033] Step 3-3: Complete the update of the state-action value table during the game process; after completing a game process, the current state s t , select action a t , Rewardt and game results t+1 A Markov chain is formed, that is, t ,a t ,r t ,s t+1 >, the matrix Q is updated by the following formula:

[0034]

[0035] Among them, γ is the reward decay, and α is the decay coefficient used to control the value update rate.

[0036] Furthermore, the specific method of step 4 is:

[0037] Step 4-1: Perform training with a step size of K per round. After the reward function converges, execute step 4-2.

[0038] Step 4-2: Verify the anti-interference effect. The step length of each round is K′. When the average anti-interference success rate is greater than 90%, the anti-interference is considered successful, and the reinforcement learning communication anti-interference based on feature engineering is completed; the average anti-interference success rate is a t,j ≠a t,c Probability of occurrence.

[0039] The beneficial effects of the present invention are:

[0040] 1. The present invention constructs a state space by combining the historical frequency domain including interference pattern information through feature engineering methods, and considers the impact of channel feature changes caused by channel switching on signal modulation and demodulation, based on which a reward function is designed.

[0041] 2. The present invention combines a reinforcement learning model to generate an anti-interference strategy, which can be used to combat malicious interference in communication networks.

[0042] 3. The present invention provides new ideas and methods for generating anti-interference strategies, which can improve the information transmission effect of the communication network, thereby providing reliable technical support for the construction of command and control situation. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 The figure is a flow chart of a method according to an embodiment of the present invention.

[0044] Figure 2 This is a diagram showing the effect of random switching to counter fixed-frequency interference and swept-frequency interference.

[0045] Figure 3 Graph showing the anti-interference success rate in an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments. ​

[0047] A reinforcement learning communication anti-interference method based on feature engineering, the overall process is as follows Figure 1 As shown, the specific steps include:

[0048] Step 1: Construct the communication interference environment in the time and frequency domains.

[0049] Step 1-1: At time t, the number of communication channels on the entire communication frequency band is N, where N=10. Therefore, at time t, the entire communication frequency band can be represented by matrix C as follows:

[0050] C t =[c t,1 c t,2 …c t,n ]

[0051] Among them, when there is no communication signal and interference signal in the channel, c t,n =0; when there is no communication signal and only interference signal in the channel, c t,n =-1; when there is only communication signal and no interference signal in the channel, c t,n =1; when there are communication signals and interference signals in the channel, c t,n =2.

[0052] Step 1-2: Time-frequency information of the communication frequency band in the entire T time in the communication interference environment t It can be expressed as:

[0053]

[0054] Step 1-3: In the game environment, the jammer uses a random jamming pattern, that is, randomly switches the jamming pattern with a period of T, T = 10. The jamming patterns used by the jammer are fixed frequency jamming and swept frequency jamming.

[0055] Step 2: Build a reinforcement learning environment based on the interference environment and feature engineering.

[0056] Step 2-1: Construct a communication anti-interference environment reward function. This function needs to be considered from two aspects: on the one hand, from the perspective of ensuring effective communication, no interference signals can appear in the channel where the communication signal is located; on the other hand, due to the different channel characteristics between different communication channels, it is necessary to consider the relationship between the channel selected by the current communication device and the channel selected at the previous moment. Therefore, the communication anti-interference environment reward function can be expressed as:

[0057]

[0058] Among them, a t,j represents the channel selected by the jammer at time t, a t,c represents the channel selected by the communication device at time t, and λ is the discount coefficient.

[0059] Step 2-2: Construct the state space of the communication anti-interference environment. In order to effectively perceive the interference style adopted by the current jammer and fit the reward function, the state at time t can be expressed as:

[0060] s t =[C t ,C t-1 ]

[0061] Step 3: Construct a reinforcement learning communication anti-interference method model based on the reinforcement learning environment.

[0062] Step 3-1: Construct a state-action value table. The value table is represented by the matrix Q as shown below:

[0063]

[0064] The dimension of the matrix Q is M×N. The number of rows M is the type of state, which is automatically generated and recorded by the machine. The number of columns N is the number of channels. The value q(s1′,a1) represents the value that can be obtained by choosing a1 when the state type is s1′, that is:

[0065] Q(s′ m ,a n ) = q(s′) m ,a n )

[0066] Step 3-2: Construct a greedy algorithm to select the current state s t The optimal action value a under t During the training process, the optimal action selection formula is as follows:

[0067]

[0068] Where random(n) means selecting an integer with uniform probability in the closed interval [1,N]; ε means a random number in the interval [0,1]; η means the threshold, which is 0.9; argmaxQ(s t ) means that in the value matrix, choose s t The action with the highest value in the state.

[0069] When in the verification process, the formula for selecting the optimal action is as follows:

[0070] a t = argmaxQ(s t )

[0071] Step 3-3: Complete the update of the state-action value table during the game. After completing a game, the current state s t, select action a t , Reward t and game results t+1 A Markov chain is formed, that is, t ,a t ,r t ,s t+1 >. The Q table can be updated by the following formula:

[0072]

[0073] Among them, γ is the reward attenuation used to control the reward, which can be taken as 0.9 here; α is the attenuation coefficient used to control the value update rate, which can be taken as 0.9 here.

[0074] Step 4: Verify the reinforcement learning communication anti-interference method model based on the interference environment.

[0075] Step 4-1: Perform 1000 rounds of training, with a step size of K in each round, for a total of 5000 steps. After the reward function converges, execute step 4-2 to verify the anti-interference effect.

[0076] Step 4-2: Perform 100 rounds of testing, each with a step length of K′, for a total of 5000 steps. When the average anti-interference success rate is greater than 90%, that is, a t,j ≠a t,c When the probability of occurrence is greater than 90%, reinforcement learning communication anti-interference based on feature engineering is implemented.

[0077] Anti-interference effect such as Figure 2 and Figure 3 As shown. Figure 2 It can be seen that the communication signal (white) can effectively avoid the fixed-frequency interference and swept-frequency interference (black) generated by the jammer. Figure 3 It can be seen that the anti-interference success rate is stable at more than 99.1%, and the experiment verifies the effectiveness of the anti-interference method.

[0078] In summary, the method of the present invention has a clear structure and strong interpretability. It can accurately avoid the random switching interference of the intelligent jammer and effectively avoid the switching interference of the periodic random interference pattern, and has wide industrial applicability.​

Claims

1. A reinforcement learning communication anti-interference method based on feature engineering, characterized in that: The following steps are involved: Step 1: Construct the communication interference environment in the time domain and frequency domain; the specific method is: Step 1-1: At time t, the number of communication channels on the entire communication frequency band is N. The entire communication frequency band at time t is represented by matrix C as follows: C t =[[c t,1 c t,2 …c t,n ]] Among them, when there is no communication signal and interference signal in the channel, c t,n =0; when there is no communication signal in the channel but only interference signal, c t,n =-1; when there is only communication signal in the channel and no interference signal, c t,n =1; when there are both communication signals and interference signals in the channel, c t,n =2; Step 1-2: The time-frequency information of the communication frequency band in the entire T time in the communication interference environment is O t It is expressed as: Step 1-3: In the game environment, the jammer uses a random jamming pattern, that is, randomly switches the jamming pattern with a period of T; the jamming patterns used by the jammer are fixed frequency jamming and swept frequency jamming; Step 2: Build a reinforcement learning environment based on interference environment and feature engineering; the specific method is: Step 2-1: Construct a communication anti-interference environment reward function: Among them, a t,j represents the channel selected by the jammer at time t, a t,c represents the channel selected by the communication device at time t, λ is the discount coefficient, which is 0.6; Step 2-2: Construct the communication anti-interference environment state space and express the state at time t as: s t =[C t ,C t-1 ]; Step 3: Construct a reinforcement learning communication anti-interference method model based on the reinforcement learning environment; the specific method is: Step 3-1: Construct a state-action value table. The value table is represented by the matrix Q as follows: The dimension of matrix Q is M×N, the number of rows M is the type of state, the number of columns N is the number of channels, and the value q(s1¢,a1) represents the value that can be obtained by selecting a1 when the state type is s1¢, that is: Q(s¢ m ,a n )=q(s¢ m ,a n ) Step 3-2: Construct a greedy algorithm to select the current state s t The optimal action value a under t ; During the training process, the formula for selecting the optimal action is as follows: Among them, random(n) means selecting an integer with uniform probability in the closed interval [1,N]; e means a random number in the interval [0,1]; h means the threshold; argmaxQ(s t ) means that in the value matrix, choose s t The action with the highest value in the state; During the verification process, the formula for selecting the optimal action is as follows: a t =argmaxQ(s t ) Step 3-3: Complete the update of the state-action value table during the game process; after completing a game process, the current state s t , select action a t , Reward t and game results t+1 A Markov chain is formed, that is, t ,a t ,r t ,s t+1 >, the matrix Q is updated by the following formula:​ Among them, γ is the reward decay, and α is the decay coefficient used to control the value update rate; Step 4: Verify the reinforcement learning communication anti-interference method model based on the interference environment; the specific method is: Step 4-1: Perform multiple rounds of training, and execute step 4-2 after the reward function converges; Step 4-2: Perform multiple rounds of verification on the anti-interference effect. When the average anti-interference success rate is greater than 90%, the anti-interference is considered successful, and the reinforcement learning communication anti-interference based on feature engineering is completed; the average anti-interference success rate is a t,j 1a t,c Probability of occurrence.