Cognitive communication interference method and system based on Transform and deep reinforcement learning

Abstract

The invention discloses a cognitive communication interference method and system based on Transform and deep reinforcement learning, and the method comprises the steps: taking Transform Encoder as a deep reinforcement learning network, processing the state information of enemy communication, and selecting an interference action according to the processed information through DDQN; the action with low power consumption is selected by designing a reward function and establishing an invalid interference list, and the convergence speed of the algorithm is improved. The system comprises a communication interference system model construction module, an interference decision module, a state and action space definition module, a reward function design module, a network construction module, an invalid interference list establishment module, a training module and a communication interference module. According to the method, multiple communication signals can be successfully interfered, the interaction frequency can be reduced, the convergence speed is improved, and the performance is relatively stable.

Description

Technical Field

[0001] The present invention belongs to the field of communication countermeasure technology, and in particular to a cognitive communication interference method and system based on Transformer (full self-attention network) and deep reinforcement learning. Background Art

[0002] Wireless communication plays an important role in intelligence transmission, battlefield command and other aspects in the military field. How to interfere with enemy communications during combat has a decisive impact on battlefield victory. However, with the development of frequency hopping communication, direct spread spectrum communication, adaptive technology and anti-interference communication technology, especially the use of cognitive radio technology, it has become increasingly difficult to interfere with enemy communications.

[0003] In this case, a simple method is to use high-power noise to suppress the frequency band. However, this interference method not only consumes a lot of energy, but also damages the communication of the enemy. In a real environment, the enemy needs to communicate in real time and with high quality, and the energy available for interference is also limited. Therefore, it is urgent to find a low-power, precise, intelligent, and adaptive interference strategy for enemy communications.

[0004] Traditional jamming methods are based on game theory, optimization theory and other theoretical techniques to find the optimal parameters for jamming, but these studies rely on prior information about the communication parties and the environment. In scenarios where prior information is lacking, these theories are not applicable. Currently, more scholars use reinforcement learning to learn communication jamming strategies. For example, some scholars have proposed a jamming bandit algorithm (GB) based on a multi-armed tiger machine framework, which can find the optimal physical layer parameters for attack. Other scholars have proposed using orthogonal decomposition to achieve different jamming styles, and an intelligent jamming algorithm (IB) that continuously optimizes the jamming action selection area during training. However, these jamming methods still have the following defects:

[0005] (1) The research object is limited to modulation modes such as BPSK, QPSK, and 16QAM, so the interference strategy obtained through learning is likely not optimal;

[0006] (2) The power consumption of the interferer is not considered. When the power that the interferer can transmit is small, the interference effect will be affected.

[0007] (3) The jamming method based on reinforcement learning can only be trained for one state. When the enemy changes the modulation signal, it is necessary to find the optimal jamming parameters again. This type of algorithm is not suitable for the rapidly changing environment in practice. Summary of the invention

[0008] The purpose of the present invention is to provide a cognitive communication interference method and system based on Transformer and deep reinforcement learning, so as to solve the problem that the current communication interference technology can only interfere with a few modulation styles and cannot intelligently adjust the interference strategy according to environmental changes.

[0009] The technical solution to achieve the purpose of the present invention is: a cognitive communication interference method based on Transformer and deep reinforcement learning, comprising the following steps:

[0010] Step 1: Establish a communication interference system model, determine the communication interference process, and determine the modulation style of the communication party to be interfered with and the interference modulation style adopted by the interfering party;

[0011] Step 2: Use deep reinforcement learning to implement interference decision-making, select the full self-attention network, i.e. Transformer, as the deep reinforcement learning network, and DDQN as the deep learning algorithm;

[0012] Step 3: Define the state and action space required for deep reinforcement learning based on the characteristics of communication signals and interference signals;

[0013] Step 4: Design a reward function based on the jamming effect and jamming signal power consumption;

[0014] Step 5: Construct a Q network based on Transformer Encoder. Use 1 layer of Transformer Encoder and 2 layers of FC to form the Q network, where Encoder represents the encoder and FC represents the fully connected layer.

[0015] Step 6: Before starting to train the Q network, establish an invalid interference list for each communication modulation pattern, which is used to record the interference modulation patterns with invalid interference;

[0016] Step 7: Train the Q network based on DDQN, randomly selecting a state for training in each round until the specified number of training rounds is reached;

[0017] Step 8: Use the trained Q network to perform cognitive communication interference.

[0018] A cognitive communication jamming system based on Transformer and deep reinforcement learning, the system is used to implement the cognitive communication jamming method based on Transformer and deep reinforcement learning, the system includes a communication jamming system model building module, a jamming decision module, a state and action space definition module, a reward function design module, a network building module, an invalid jamming list establishment module, a training module and a communication jamming module, wherein:

[0019] The communication interference system model building module establishes the communication interference system model, determines the communication interference process, and determines the modulation style of the communication party to be interfered with and the interference modulation style adopted by the interfering party;

[0020] The interference decision module uses deep reinforcement learning to realize interference decision-making. The full self-attention network, i.e. Transformer, is selected as the deep reinforcement learning network, and DDQN is used as the deep learning algorithm.

[0021] The state and action space definition module defines the state and action space required for deep reinforcement learning based on the characteristics of communication signals and interference signals;

[0022] Reward function design module, which designs the reward function based on the interference effect and the power consumption of the interference signal;

[0023] Network construction module, build a Q network based on Transformer Encoder, use 1 layer of Transformer Encoder and 2 layers of FC to form a Q network, where Encoder represents the encoder and FC represents the fully connected layer;

[0024] The invalid interference list establishment module establishes an invalid interference list for each communication modulation pattern before starting to train the Q network, and is used to record the interference modulation patterns with invalid interference;

[0025] The training module trains the Q network based on DDQN, randomly selecting a state for training in each round until the specified number of training rounds is reached;

[0026] The communication jammer module uses the trained Q network to perform cognitive communication jammer.

[0027] A mobile terminal comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the cognitive communication interference method based on Transformer and deep reinforcement learning as described above is implemented.

[0028] Compared with the prior art, the present invention has the following significant advantages:

[0029] (1) The present invention studies the interference strategies for 19 modulation styles. The research scope is wider than previous studies and can effectively learn the interference strategies for each modulation style.

[0030] (2) The reward function designed by the present invention takes the power consumption of the interferer into consideration, so that the interference strategy learned by the algorithm can successfully interfere with the communication of the communicating party with low power consumption.

[0031] (3) The algorithm designed in the present invention can be trained for multiple states, can effectively deal with the situation where the communication party changes the modulation signal, and is suitable for changing environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 Schematic diagram of the communication interference system model.

[0033] Figure 2 This is a schematic diagram of the algorithm network structure.

[0034] Figure 3 Schematic diagram of the comparison network structure.

[0035] Figure 4 It is a schematic diagram of the training effect of the network of the present invention and the comparison network.

[0036] Figure 5 This is a schematic diagram of the training effect of the algorithm of the present invention and the deep reinforcement learning comparison algorithm.

[0037] Figure 6 It is a schematic diagram of the training effect of the interference method of the present invention and other interference methods.

[0038] Figure 7 Schematic diagram of the training effect of the algorithm with and without invalid interference list. DETAILED DESCRIPTION

[0039] It is easy to understand that according to the technical solution of the present invention, without changing the essential spirit of the present invention, a person skilled in the art can imagine various implementation methods of the present invention. Therefore, the following specific implementation methods and drawings are only exemplary descriptions of the technical solution of the present invention, and should not be regarded as the whole of the present invention or as a limitation or restriction of the technical solution of the present invention.

[0040] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention unless otherwise specifically stated.

[0041] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0042] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0043] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0044] The present invention designs a cognitive communication interference method based on Transformer and deep reinforcement learning to try to solve the problems existing in the prior art. First, the real environment is simulated, a communication interference model is established, and a variety of digital signals and analog signals are fully considered. By modeling the interference model, the interference action is designed as a three-dimensional space containing modulation style, interference power and communication power ratio and duty cycle, and the advantage of deep reinforcement learning that can handle multi-state space and multi-action space is utilized to use deep reinforcement learning for the selection of interference actions. In order to realize the algorithm's selection of low-power actions, a reasonable reward function is designed. And a interference strategy based on Transformer and deep reinforcement learning is proposed, which uses the multi-head attention mechanism of Transformer to extract state features and uses it as a network component of deep reinforcement learning. In addition, an invalid interference list is established to reduce the number of interactions between the algorithm and the environment, thereby improving the convergence speed of the algorithm.

[0045] The present invention provides a cognitive communication interference method based on Transformer and deep reinforcement learning, comprising the following steps:

[0046] Step 1: Establish a communication interference system model, determine the communication interference process, and determine the modulation style of the communication party to be interfered with and the interference modulation style adopted by the interfering party;

[0047] Step 2: Use deep reinforcement learning to implement interference decision-making, select the full self-attention network, i.e. Transformer, as the deep reinforcement learning network, and DDQN as the deep learning algorithm;

[0048] Step 3: Define the state and action space required for deep reinforcement learning based on the characteristics of communication signals and interference signals;

[0049] Step 4: Design a reward function based on the jamming effect and jamming signal power consumption;

[0050] Step 5: Construct a Q network based on Transformer Encoder. Use 1 layer of Transformer Encoder and 2 layers of FC to form the Q network, where Encoder represents the encoder and FC represents the fully connected layer.

[0051] Step 6: Before starting to train the Q network, establish an invalid interference list for each communication modulation pattern, which is used to record the interference modulation patterns with invalid interference;

[0052] Step 7: Train the Q network based on DDQN, randomly selecting a state for training in each round until the specified number of training rounds is reached;

[0053] Step 8: Use the trained Q network to perform cognitive communication interference.

[0054] As a specific example, in step 1, a communication interference system model is established as follows:

[0055] The communication interference system model includes a sender, a receiver, an interferer and a channel, wherein the channel is used to simulate the influence of environmental noise on the signal;

[0056] The sender sends a communication signal S T (t):

[0057] S T (t) = I T cos(2π*f c *t)+Q T sin(2π*f c *t)

[0058] Among them, I T and Q T is the modulation information of the sender’s in-phase and quadrature dimensions, f c is the carrier frequency, t represents time; I T and Q T The modulation pattern M of the signal is implied in T and communication power P T information;

[0059] The jammer first obtains the communication signal parameters, including the modulation pattern M, through the reconnaissance system. T , communication power P T , carrier frequency f c Then select the interference strategy and send the interference signal S J (t); The prerequisite for successful interference is that the frequency of the interference signal is aligned with that of the communication signal, so the two frequencies are the same;

[0060] S J (t) = I J cos(2π*f c *t)+Q J sin(2π*f c *t)

[0061] Among them, I J and Q J is the modulation information of the in-phase dimension and the orthogonal dimension of the interfering party;

[0062] The jamming strategy is determined by the jamming modulation pattern M J , interference signal power P J and the communication signal power P T The ratio pr and the duty cycle ρ are composed of the ratio of the interference time to the intermittent interference period;

[0063] The signal S after the communication signal and the interference signal reach the receiver through the channel R (t) is:

[0064] S R (t) = S T (t)+S J (t)+n(t)

[0065] =(I T +I J )cos(2π*f c *t)+(Q T +Q J )sin(2π*f c *t)+n(t)

[0066] Where n(t) is the channel noise;

[0067] The receiver determines whether it is interfered with based on the bit error rate or signal-to-noise ratio of the received signal, and feeds back to the sender; the sender decides whether to change the communication behavior based on the feedback received; then, the interferer evaluates the interference effect based on the changes in communication behavior before and after the interference, and selects an interference strategy based on the current communication behavior of the sender; the above process is repeated until the communicating party stops communicating.

[0068] As a specific example, in step 1, the modulation pattern of the communication party to be interfered and the interference modulation pattern adopted by the interfering party are determined as follows:

[0069] Communication signal modulation mode M T Including 19 modulation modes: BASK, MSK, GMSK, BPSK, 4PSK, 8PSK, 16PSK, 16QAM, 64QAM, 256QAM, 2FSK, 4FSK, 8FSK, 16FSK, DQPSK, OQPSK, AM, FM, SSB;

[0070] Interference signal modulation pattern M J Including BASK, MSK, GMSK, BPSK, 4PSK, 8PSK, 16PSK, 16QAM, 64QAM, 256QAM, 2FSK, 4FSK, 8FSK, 16FSK, DQPSK, OQPSK, AM, FM, SSB, AWGN;

[0071] Interference signal power P J and the communication signal power P T The ratio pr is discretized into 5 levels: 0.2, 0.4, 0.6, 0.8, and 1;

[0072] The duty cycle ρ is discretized into 9 levels: 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, and 0.9.

[0073] As a specific example, in step 3, the state and action space required for deep reinforcement learning are defined based on the characteristics of the communication signal and the interference signal, as follows:

[0074] The action space A is defined as the J , pr and ρ, that is, A=[M J ,pr,ρ];

[0075] The communication strategy of the communicating party is the parameter information of the communication signal, i.e., modulation style, power and frequency;

[0076] During interference, the interference frequency must be aligned with the communication frequency, and the action contains pr, so the communication power has no effect on the result. Therefore, the state S is only determined by the communication modulation pattern M. T One element, that is, S = [M T ].

[0077] As a specific example, in step 4, the reward function is designed according to the interference effect and the power consumption of the interference signal, as follows:

[0078] The design of the reward function r is as follows

[0079]

[0080] Among them, s is the current state, s' is the state at the next moment;

[0081] s'=s represents interference failure, and the reward R1 is obtained;

[0082] s'=None means the communication party stops communicating, the interference is successful, and the reward R2+(0.6-pr)×R is obtained e +(0.6-ρ)×R e ;

[0083] The reward for successful interference is designed as a combination of the basic reward for successful interference and the additional reward, where R2 is the basic reward for successful interference, (0.6-pr)×R e +(0.6-ρ)×R e It is an additional reward given according to the adopted pr and ρ; the smaller the pr and ρ selected by the interference strategy, the lower the power consumption and the higher the reward obtained, so that DDQN tends to choose low-power actions; R e Represents the weight of the extra reward, and the weight means the importance of selecting low-power actions.

[0084] As a specific example, in step 5, a Transformer Encoder-based Q network is constructed, using 1 layer of Transformer Encoder and 2 layers of FC to form the Q network, as follows:

[0085] The Q network first inputs the state information s t After being encoded into a feature vector S by the Embedding layer, it is processed by the multi-head attention mechanism Multi-Head(S):

[0086] Multi-Head(S)=W O [head1,head2,...,head H ]

[0087]

[0088] Among them, W O is the matrix parameter; H is the number of attention heads; Q = W Q i S, K = W K i S, V = W V i S; and W O Same, W Q i , W K i , W V i are matrix parameters, d model is the dimension of S;

[0089] The data after multi-head attention processing is passed to the feed-forward network FFN(x) after residual connection and normalization. FFN(x) consists of two fully connected layers:

[0090] FFN(x)=ReLU(W1x+b1)W2+b2 (8)

[0091] Among them, W1, b1 and W2, b2 are the parameters of the two fully connected layers respectively;

[0092] The result obtained by FFN processing is then passed to FC after residual connection and normalization:

[0093] FC(x)=ReLU(W3x+b3)W4+b4 (9)

[0094] Same as the FFN layer, W3,b3 and W4,b4 are the parameters of the two fully connected layers in FC;

[0095] After FC, we get the state-action value vector V t .

[0096] As a specific example, in step 6, the process of establishing the invalid interference list is as follows:

[0097] Establish an invalid interference list L, L = [l1,l2,…,l i ,…,l 19 ],l i Used to record the interference patterns that are ineffective for interfering with the i-th modulation pattern, so as to obtain the actions that are ineffective for interfering with each state;

[0098] First, we construct an action set A consisting of each modulation style, maximum power ratio, and duty cycle. m , A m ={[M Jm ,pr,ρ]|pr=1,ρ=0.9} and invalid interference list L;

[0099] For any state s i , select A m Each action a m Interference i , will interfere with the invalid action a u The corresponding modulation pattern M Ju Add to i The corresponding l i In the process, all states are traversed and finally an invalid interference list L is formed.

[0100] As a specific example, in step 7, the Q network is trained based on DDQN as follows:

[0101] ① Set the number of training rounds, target network update frequency, initialize network parameters, and generate a list of invalid interferences;

[0102] ② Obtain the state from the environment, use the Q network to process the state information, and obtain the predicted reward for each action;

[0103] ③ Select the action to be performed according to the ε-greedy strategy and the invalid interference list;

[0104] ④After executing the action, get the environmental feedback and the next state, and store the state, action and reward generated by this interaction into the experience replay pool;

[0105] ⑤ Extract a fixed amount of experience from the experience replay pool to update the Q network, so that the action reward output by the Q network is closer and closer to the real reward;

[0106] ⑥ Repeat steps ② to ⑤ until the number of training rounds is reached and the training ends.

[0107] The present invention also provides a cognitive communication jamming system based on Transformer and deep reinforcement learning, which is used to implement the cognitive communication jamming method based on Transformer and deep reinforcement learning. The system includes a communication jamming system model building module, an jamming decision module, a state and action space definition module, a reward function design module, a network building module, an invalid jamming list establishment module, a training module and a communication jamming module, wherein:

[0108] The communication interference system model building module establishes the communication interference system model, determines the communication interference process, and determines the modulation style of the communication party to be interfered with and the interference modulation style adopted by the interfering party;

[0109] The interference decision module uses deep reinforcement learning to realize interference decision-making. The full self-attention network, i.e. Transformer, is selected as the deep reinforcement learning network, and DDQN is used as the deep learning algorithm.

[0110] The state and action space definition module defines the state and action space required for deep reinforcement learning based on the characteristics of communication signals and interference signals;

[0111] Reward function design module, which designs the reward function based on the interference effect and the power consumption of the interference signal;

[0112] Network construction module, build a Q network based on Transformer Encoder, use 1 layer of Transformer Encoder and 2 layers of FC to form a Q network, where Encoder represents the encoder and FC represents the fully connected layer;

[0113] The invalid interference list establishment module establishes an invalid interference list for each communication modulation pattern before starting to train the Q network, and is used to record the interference modulation patterns with invalid interference;

[0114] The training module trains the Q network based on DDQN, randomly selecting a state for training in each round until the specified number of training rounds is reached;

[0115] The communication jammer module uses the trained Q network to perform cognitive communication jammer.

[0116] The present invention also provides a mobile terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the cognitive communication interference method based on Transformer and deep reinforcement learning when executing the program.

[0117] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0118] Example 1

[0119] This embodiment provides a cognitive communication interference method based on Transformer and deep reinforcement learning, including the following steps:

[0120] Step 1: Establish a communication interference system model, determine the communication interference process, and determine the modulation style of the communication party to be interfered with and the interference modulation style adopted by the interfering party; this embodiment includes 19 modulation styles in the research scope.

[0121] Step 2: Based on the characteristics of the communication signal and the interference signal, the state and action space required for deep reinforcement learning are defined. Deep reinforcement learning selects the interference strategy based on the state information of the communication signal.

[0122] Step 3: Design a reward function based on the jamming effect and jamming signal power consumption;

[0123] Step 4: Select the dual deep Q network, or DDQN, as the deep reinforcement learning algorithm for learning the interference strategy.

[0124] Step 5: Build the network structure of DDQN, using 1 layer of Transformer (full self-attention network) Encoder layer and 2 layers of FC (fully connected) as the network part;

[0125] Step 6, establish an invalid interference list: before starting training, establish an invalid interference list for each communication modulation pattern to record the interference modulation patterns with invalid interference; during the training process, avoid selecting actions composed of invalid interference patterns according to the invalid interference list, speed up the training process, and improve the convergence speed;

[0126] Step 7: Pass the constructed invalid interference list into the algorithm and start training, randomly selecting a state for learning each time until the specified number of training rounds is reached.

[0127] The following is a detailed introduction to each part:

[0128] (1) Model establishment

[0129] The communication interference system model established in this embodiment is as follows: Figure 1 As shown:

[0130] The model consists of four parts: sender, receiver, interferer and channel. The channel part is responsible for simulating the impact of environmental noise and other factors on the signal.

[0131] The overall process of the model is as follows. The sender sends a communication signal S T .

[0132] S T (t) = I T cos(2π*fc *t)+Q T sin(2π*f c *t)

[0133] Among them, I and Q are the modulation information of the in-phase dimension and the orthogonal dimension, f c is the carrier frequency. I and Q imply the modulation mode M of the signal. T and power P T information.

[0134] The jammer first obtains the communication signal parameters through the reconnaissance system, the modulation pattern M T , power P T , frequency f c Then select the interference strategy and send the interference signal S J The prerequisite for successful interference is that the frequency of the interference signal is aligned with that of the communication signal, so that both have the same frequency.

[0135] S J (t) = I J cos(2π*f c *t)+Q J sin(2π*f c *t)

[0136] The jamming strategy is determined by the jamming modulation pattern M J , interference power P J With communication power P T The ratio pr and the duty cycle ρ.

[0137] The signal S after the communication signal and the interference signal reach the receiver through the channel R , where n(t) is the channel noise.

[0138] S R (t) = S T (t)+S J (t)+n(t)

[0139] =(I T +I J )cos(2π*f c *t)+(Q T +Q J )sin(2π*f c *t)+n(t)

[0140] The receiver determines whether it is interfered with based on the bit error rate (BER) or signal-noise ratio (SNR) of the received signal, and feeds back to the sender. The sender decides whether to change the communication behavior based on the feedback received to reduce the impact of interference on communication. After that, the interferer evaluates the interference effect based on the changes in communication behavior before and after the interference, and selects an interference strategy based on the current communication behavior of the sender. The above process is repeated until the communicating party stops communicating.

[0141] In order to be able to interfere with multiple modulation signals, the communication signal modulation pattern M T It consists of a variety of digital modulation styles and analog modulation styles, including BASK, MSK, GMSK, MPSK, MQAM, MFSK, DQPSK, OQPSK, AM, FM, SSB and other 19 modulation styles. Interference signal modulation style M J It is composed of the above modulation patterns and (Additive White Gaussian Noise, AWGN). In order to conveniently represent the action space of deep reinforcement learning, this embodiment discretizes ρ and pr into 9 and 5 levels respectively. The optional values ​​of communication parameters and interference parameters are shown in Table 1.

[0142] Table 1 shows the communication parameters and interference parameters

[0143]

[0144] (2) Deep reinforcement learning state and action definition

[0145] This embodiment uses a deep reinforcement learning algorithm to achieve the target task. First, the state and action need to be defined. This embodiment defines the action space A as a function of M J , pr and ρ, that is, A=[M J ,pr,ρ]. The communication strategy of the communication party is the parameter information of the communication signal, namely the modulation style, power and frequency. When interfering, the interference frequency should be aligned with the communication frequency, and the action contains pr, so the communication power has no effect on the result. Therefore, the state S is only determined by the communication modulation style M T One element, that is, S = [M T ].

[0146] (3) Reward function design

[0147] The design of the reward function r in this embodiment is as follows

[0148]

[0149] Where s is the current state and s' is the state at the next moment. s' = s means the interference fails and the reward R1 is obtained; s' = None means the communication party stops communicating, the interference is successful, and the reward R2 + (0.6-pr) × R is obtained. e +(0.6-ρ)×R e .. In order to make the action selected by the algorithm to interfere successfully with lower power consumption, it is necessary to give additional rewards according to the pr and ρ of the interference action. Therefore, the present invention designs the reward for successful interference as a combination of the basic reward for successful interference and the additional reward. Among them, R2 is the basic reward for successful interference, (0.6-pr)×R e +(0.6-ρ)×R e . is the additional reward given according to the adopted pr and ρ. The smaller the pr and ρ selected by the interference strategy, the lower the power consumption and the higher the reward obtained, so that DDQN tends to choose low-power actions. R e Represents the weight of the extra reward, and the weight means the importance of selecting low-power actions.

[0150] (4) Selection of Deep Reinforcement Learning Algorithm

[0151] This embodiment selects DDQN as the implementation algorithm of the interference method. DDQN uses a deep learning network as a Q network to process state information and can handle multi-state multi-action space tasks. The model constructed in this embodiment has 19×900=17100 state-action pairs, and DDQN is suitable for processing the tasks of the present invention. The parameter settings for DDQN in this embodiment are shown in Table 2.

[0152] Table 2 shows the DDQN parameter settings.

[0153] Parameter name Numeric learning rate 0.001 γ 0.02 εmin 0.1 εmax 0.9 Batch size 128 Replay memory size 60000 episodes 7500 target network update frequency 200 <![CDATA[R1]]> -200 <![CDATA[R2]]> 200 <![CDATA[R e ]]> 50

[0154] (5) Constructing the network structure of DDQN

[0155] This embodiment uses 1 layer of Transformer Encoder and 2 layers of FC to form a Q network. The structure is as follows Figure 2 shown.

[0156] The Q network first inputs the state information s t It is encoded into a feature vector S by the Embedding layer and then processed by the multi-head attention mechanism.

[0157] Multi-Head(S)=W O [head1,head2,...,head H ]

[0158]

[0159] Among them, W O . is the matrix parameter; H is the number of attention heads; Q = W Q i S.,K=W K i S, V = W V i S; and W O Same, W Q i , W K i , W V i are matrix parameters, d model is the dimension of S. The data after multi-head attention processing is passed into FFN (feedforward network) after residual connection and normalization. FFN consists of two fully connected layers.

[0160] FFN(x)=ReLU(W1x+b1)W2+b2 (8)

[0161] W1, b1 and W2, b2 are the parameters of the two fully connected layers respectively. The result obtained by FFN is then passed to FC after residual connection and normalization.

[0162] FC(x)=ReLU(W3x+b3)W4+b4 (9)

[0163] Similar to the FFN layer, W3, b3 and W4, b4 are also the parameters of the two fully connected layers in FC. After FC, the state-action value vector V is obtained t .

[0164] The network parameter settings of this embodiment are shown in Table 3.

[0165] Table 3 shows the network parameter settings of this embodiment.

[0166] Parameter name Numeric num embedding 40 embedding dim 256 H 32 <![CDATA[d model ]]> 256 Hidden layer dim of FC 1024 feedforward dim 256

[0167] (6) Establishment of invalid interference list

[0168] The model constructed in this embodiment has a total of 17,100 state-action pairs. i, to find an action with better interference effect from 900 actions, the model needs to interact with the environment at least 900 times. This requires that different actions can be selected each time, and the model can learn each interaction experience well. This is the ideal situation, but in practical applications, each action selection is random, and model learning also requires multiple interactions. This results in more than 900 interactions required to learn the interference strategy for a state. However, when the number of states increases, not only does the number of interactions with the environment need to increase, but the model also needs to adjust parameters to balance the processing of each state. Therefore, the number of interactions required to achieve the goal of learning interference strategies for multiple modulation styles is far more than 17,100 times, which takes a lot of time. For a certain modulation style M Ti The state of the composition i , there are some interference patterns M Jk The action of a r , even with the maximum power ratio and duty cycle interference s i , and cannot interfere with success, then M Jk Other actions that constitute a u Nor can it successfully interfere with s i , mathematically expressed as follows:

[0169]

[0170] If the algorithm can avoid choosing A corresponding to the current state in its interaction with the environment u If the actions in the set are large, the number of interactions between the model and the environment will be greatly reduced, and the convergence speed will be rapidly improved. Therefore, this embodiment establishes an invalid interference list L, L = [l1, l2, ..., l i ,…,l 19 ],l i It is used to record the interference patterns that are ineffective for interfering with the i-th modulation pattern, so as to obtain the actions that are ineffective for interfering with each state.

[0171] First, we construct an action set A consisting of each modulation style, maximum power ratio, and duty cycle. m , A m ={[M Jm ,pr,ρ]|pr=1,ρ=0.9} and invalid interference list L. For any state s i , select A m Each action a m Interference i , will interfere with the invalid action a u The corresponding modulation pattern M Ju Add to i The corresponding l i In. Traverse all states and finally form an invalid interference list L.

[0172] (7) Start training

[0173] Before starting training, first establish a corresponding invalid interference list for each communication modulation style based on the principle of establishing an invalid interference list. Pass the established invalid interference list into the action selection module of the algorithm. In each round, a state is randomly selected for training, and the state information is first passed into the network, and then the action is selected based on the invalid interference list and the ε-greedy strategy. If an action in the invalid interference list is selected, reselect it. After the interference is successful, this round ends and the next round of training begins until the specified number of training rounds is reached.

[0174] Example 2

[0175] The Adam optimizer is used for optimization. The experiment of this embodiment is based on simulation, and the channel noise is set to AWGN with a mean of 0 and a variance of 1, and the signal-to-noise ratio is 20dB. The communication party sends 8192 code elements in one communication, and the signal data used in this embodiment is generated by the program. The receiving party judges the degree of interference of the digital modulation signal and the analog modulation signal according to BER and SNR respectively. GJB 6741-2009 Digital Communication Interference Effect Evaluation Criteria describes the interference effect when BER exceeds 0.1 as strong interference, and the communication quality is seriously damaged. Therefore, the condition for judging that the digital signal is successfully interfered with in this embodiment is that BER is greater than 0.1. When SNR is less than 4.5dB, the power of the communication signal is less than 3 times the power of the interference signal, and the communication quality is affected. The condition for judging that the analog signal is successfully interfered with in this embodiment is that SNR is less than 4.5dB. The learning rate is set to 0.001, ε gradually decreases from 0.9 to 0.1, the size of the experience replay pool is set to 6000, and 128 experiences are extracted each time to update the network. The rewards for interference failure and success are -200 and 200 respectively, and the additional reward weight is 50.

[0176] This embodiment sets up four experiments, which are comparative experiments on network structure, deep reinforcement learning algorithm, interference algorithm, and ablation experiment of invalid interference list. The goal of the present invention is to find accurate and low-power interference strategies for multiple states. The interference success rate is the ratio of the number of states with successful interference to the total number of states, which can effectively reflect the accuracy of the interference strategy. The reward function takes into account the influence of interference power consumption. The lower the power consumption of the interference action, the greater the reward obtained. Therefore, the total interference reward for all states can reflect whether the model can find actions with lower power consumption. In addition, the more times the model interacts with the environment, the more resources are consumed in actual application, so the convergence speed of the model is equally important. Loss, as the difference between the predicted value and the target value of the Q network, can reflect the convergence speed of the algorithm in real time and effectively. After the above analysis, the present invention selects loss, interference success rate and total interference reward as indicators for evaluating model performance. The present invention records the loss in the training phase, tests the current model once after 500 rounds of training, and calculates its interference success rate and total interference reward for all states.

[0177] Example 3

[0178] This example compares the network structures:

[0179] This embodiment uses 1 layer of Transformer Encoder and 2 layers of FC to implement the network part of the algorithm. In order to prove that it can better process state information and make the algorithm successfully interfere with multiple states, this embodiment compares it with other neural networks. The network structure of the present invention and the comparative network structure are as follows Figure 3 As shown, for ease of representation, Trans1 is used to represent the network used in the present invention, and Trans2, LSTM, GRU, FC1024, and FC4096 are used to represent the comparison network. Trans2 uses a 2-layer Transformer Encoder and a fully connected layer. LSTM and GRU use 4 layers of LSTM and GRU and a fully connected layer to form a network structure. Trans1, Trans2, LSTM, and GRU use the same fully connected layer. FC1024 and FC4096 both use 5 layers of full connection, with 1024 and 4096 hidden layers, respectively.

[0180] The learning effect of interference algorithms using different network structures is as follows: Figure 4 As shown in Table 4, Figure 4 The real-time change results of loss during training and the interference success rate of each interference test are shown respectively. Table 4 shows the highest rewards obtained by different network structures. Figure 4As shown in the figure, the lengths of the loss curves of different networks are different, and the length of the curve represents the number of times the network is updated. The longer the curve, the more interactions there are and the worse the convergence ability. The loss value of the Trans1 network decreases the fastest, has the lowest value, the shortest length, and the test curve is relatively stable. The loss of Trans2 is higher than that of Trans1, and it is difficult to maintain the interference success rate at 100%. It is difficult for LSTM and GRU to reach a convergence state. When the training time of the fully connected network is long enough, good results can be achieved, but its network performance is poor compared with Transformer. Trans1 uses Embedding to expand the state information obtained from the environment, and its multi-head attention mechanism and normalization operation make it easier to extract useful information from it, thereby enhancing the performance of the algorithm.

[0181] Table 4 shows the highest rewards obtained during the training process of the network of the present invention and the comparison network.

[0182] Network structure Maximum total reward Trans1 3855 Trans2 3890 LSTM 2485 GRU 1785 FC4096 3865 FC1024 3700

[0183] Table 5 shows the training time and computational effort required for the network of the present invention and the comparative network.

[0184]

[0185]

[0186] Table 5 shows the training time and required computation of different networks. Trans1 has the smallest computation and the shortest running time. Although LSTM and GRU have small computation, they cannot converge and the running time is too long.

[0187] After the above analysis, Transformer is more suitable for processing the state information of the present invention than other networks. The increase in the number of parameters can enable the network to better distinguish the differences between different actions. However, the increase in the number of parameters will also make the algorithm more difficult to converge and the amount of calculation will be greater. Therefore, the network of the present invention has the advantages of smaller network parameters, faster convergence speed and better stability.

[0188] Example 4

[0189] This example compares deep reinforcement learning algorithms:

[0190] In order to show the advantages of DDQN and enable the algorithm to converge faster and more stably, this paper compares other deep reinforcement learning algorithms with DDQN, including DQN (Deep Q Network), Dueling DQN (Dueling Deep Q Network), and DQNPer (Deep Q Network with Prioritized Experience Replay). These are all value-based reinforcement learning algorithms, which belong to the same type of algorithms as DDQN and are also the most commonly used algorithms in interference methods.

[0191] The learning effect of interference algorithm using different deep reinforcement learning algorithms is as follows Figure 5 As shown in Table 6, Figure 5 The real-time change results of loss during training and the interference accuracy during each interference test are shown respectively. Table 6 shows the highest rewards obtained by different algorithms. The loss values ​​and decline rates of DDQN and Dueling DQN are almost the same. The loss curve of Dueling DQN is shorter, but the test results are unstable and the reward is the smallest. DQN has an over-estimation problem, and the loss value is higher than DDQN and Dueling DQN. The algorithm is more unstable than DDQN. The loss change trend of DQN Per is completely opposite to that of other algorithms, and the interference success rate is difficult to maintain at 100%. The reason for this phenomenon is that DQN Per always prioritizes data with large loss for learning, which makes it easier to choose low-power actions and obtain the highest reward, but increases the risk of interference failure, so the test effect is poor.

[0192] Table 6 shows the highest rewards obtained during the training process of the algorithm of the present invention and the comparative algorithm.

[0193] algorithm Maximum total reward DDQ 3855 Dueling DQN 3805 DQN Per 3875 DQN 3845

[0194] The DDQN algorithm used in the present invention has the advantage of better stability compared to other deep reinforcement learning algorithms and is more suitable for implementing interference tasks.

[0195] Example 5

[0196] This embodiment compares the interference algorithms:

[0197] The interference method of the present invention is compared with the JB and IB algorithms proposed in previous studies. Both methods are based on reinforcement learning, do not use neural networks to process state information, and can only learn interference strategies for a single state. In order to compare the method of the present invention with the two methods fairly, only one state is trained. All three algorithms are trained 1000 times. The training process is as follows: Figure 6 As shown. The training curve of JB is different from that of the other two algorithms, and more interactions are required to find a suitable interference strategy. Both the algorithm of the present invention and the IB algorithm can remove certain invalid interference actions. The difference is that the algorithm of the present invention removes some invalid interference actions in advance and then does not change. The IB algorithm continuously adds invalid interference actions during the training process, and can remove more invalid interference actions. Therefore, the IB algorithm converges faster than the algorithm of the present invention. The highest rewards obtained by different interference algorithms are shown in Table 7. The rewards obtained by the algorithm of the present invention and the JB algorithm are the same, and the reward obtained by the IB algorithm is lower.

[0198] Table 7 shows the rewards obtained by the interference method of the present invention and the comparative interference method.

[0199] algorithm award Our 205 IB 200 JB 205

[0200] Based on the above analysis, compared with the previous interference algorithm, the method of the present invention can achieve a comparable learning effect when training for a single state. In addition, the method of the present invention can train for multiple states, which other interference methods cannot do.

[0201] Example 6

[0202] This embodiment compares the presence or absence of invalid interference lists:

[0203] The present invention constructs an invalid interference list before training, and then no longer selects actions in the list during the training process, thereby reducing the number of interactions and improving the convergence speed of the algorithm. The invalid interference list finally constructed removes a total of 6930 invalid state-action pairs, accounting for about 40% of the total state-action space. These invalid state-action pairs will not be selected during the training process, which can greatly reduce the number of interactions between the algorithm and the environment.

[0204] Table 8 shows the number of interactions required and the maximum reward obtained by the algorithm with and without invalid interference lists

[0205]

[0206] Figure 7 Table 8 shows the impact of the invalid interference list on the algorithm training effect. Figure 7 The interference test success rate curves of the algorithms with and without invalid interference lists are shown in Table 8. The number of interactions in the first 500 rounds of the algorithms with and without invalid interference lists and when the interference test reaches 100% for the first time, as well as the highest reward obtained after 7500 rounds of training, are listed. Figure 7 It can be seen that the algorithm without adding the invalid interference list has poor stability and cannot maintain a stable accuracy of 100%. As can be seen from Table 8, the number of interferences required for the first 500 rounds of the algorithm without adding the invalid interference list is about twice that of the algorithm with the invalid interference list. When the interference success rate reaches 100% for the first time, the number of interactions of the former is about 1.5 times that of the latter. By comparing the number of interactions between the two and the environment, it can be seen that adding an invalid interference list can significantly reduce the number of interactions between the algorithm and the environment. However, after adding the invalid interference list, the reward obtained is smaller, indicating that this method has a certain impact on the performance of the algorithm.

[0207] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

[0208] It should be understood that in order to simplify the present invention and help those skilled in the art understand the various aspects of the present invention, in the above description of the exemplary embodiments of the present invention, the various features of the present invention are sometimes described in a single embodiment or described with reference to a single figure. However, the present invention should not be interpreted as the features included in the exemplary embodiments are all necessary technical features of the patent claims.

Claims

1. A cognitive communication jamming method based on Transformer and deep reinforcement learning, characterized in that: The following steps are involved: Step 1: Establish a communication interference system model, determine the communication interference process, and determine the modulation style of the communication party to be interfered with and the interference modulation style adopted by the interfering party; Step 2: Use deep reinforcement learning to implement interference decision-making, select the full self-attention network, i.e. Transformer, as the deep reinforcement learning network, and DDQN as the deep learning algorithm; Step 3: Define the state and action space required for deep reinforcement learning based on the characteristics of communication signals and interference signals; Step 4: Design a reward function based on the jamming effect and jamming signal power consumption; Step 5: Construct a Q network based on Transformer Encoder. Use 1 layer of Transformer Encoder and 2 layers of FC to form the Q network, where Encoder represents the encoder and FC represents the fully connected layer. Step 6: Before starting to train the Q network, establish an invalid interference list for each communication modulation pattern, which is used to record the interference modulation patterns with invalid interference; Step 7: Train the Q network based on DDQN, randomly selecting a state for training in each round until the specified number of training rounds is reached; Step 8: Use the trained Q network to perform cognitive communication interference.

2. The cognitive communication jamming method based on Transformer and deep reinforcement learning according to claim 1, characterized in that: In step 1, a communication interference system model is established as follows: The communication interference system model includes a sender, a receiver, an interferer and a channel, wherein the channel is used to simulate the influence of environmental noise on the signal; The sender sends a communication signal S T (t): S T (t)=I T cos(2π*f c *t)+Q T sin(2π*f c *t) Among them, I T and Q T is the modulation information of the sender’s in-phase and quadrature dimensions, f c is the carrier frequency, t represents time; I T and Q T The modulation pattern M of the signal is implied in T and communication power P T information; The jammer first obtains the communication signal parameters, including the modulation pattern M, through the reconnaissance system. T , communication power P T , carrier frequency f c Then select the interference strategy and send the interference signal S J (t); The prerequisite for successful interference is that the frequency of the interference signal is aligned with that of the communication signal, so the two frequencies are the same; S J (t)=I J cos(2π*f c *t)+Q J sin(2π*f c *t) Among them, I J and Q J is the modulation information of the in-phase dimension and the orthogonal dimension of the interfering party; The jamming strategy is determined by the jamming modulation pattern M J , interference signal power P J and the communication signal power P T The ratio pr and the duty cycle ρ are composed of the ratio of the interference time to the intermittent interference period; The signal S after the communication signal and the interference signal reach the receiver through the channel R (t) is: S R (t)=S T (t)+S J (t)+n(t) =(I T +I J )cos(2π*f c *t)+(Q T +Q J )sin(2π*f c *t)+n(t) Where n(t) is the channel noise; The receiver determines whether it is interfered with based on the bit error rate or signal-to-noise ratio of the received signal, and feeds back to the sender; the sender decides whether to change the communication behavior based on the feedback received; then, the interferer evaluates the interference effect based on the changes in communication behavior before and after the interference, and selects an interference strategy based on the current communication behavior of the sender; the above process is repeated until the communicating party stops communicating.

3. The cognitive communication jamming method based on Transformer and deep reinforcement learning according to claim 2 is characterized in that: In step 1, the modulation style of the communication party to be interfered and the interference modulation style adopted by the interfering party are determined as follows: Communication signal modulation mode M T Including 19 modulation modes: BASK, MSK, GMSK, BPSK, 4PSK, 8PSK, 16PSK, 16QAM, 64QAM, 256QAM, 2FSK, 4FSK, 8FSK, 16FSK, DQPSK, OQPSK, AM, FM, SSB; Interference signal modulation pattern M J Including BASK, MSK, GMSK, BPSK, 4PSK, 8PSK, 16PSK, 16QAM, 64QAM, 256QAM, 2FSK, 4FSK, 8FSK, 16FSK, DQPSK, OQPSK, AM, FM, SSB, AWGN; Interference signal power P J and the communication signal power P T The ratio pr is discretized into 5 levels: 0.2, 0.4, 0.6, 0.8, and 1; The duty cycle ρ is discretized into 9 levels: 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, and 0.

9.

4. The cognitive communication jamming method based on Transformer and deep reinforcement learning according to claim 3 is characterized in that: In step 3, based on the characteristics of communication signals and interference signals, the state and action space required for deep reinforcement learning are defined as follows: The action space A is defined as the J , pr and ρ, that is, A=[M J ,pr,ρ]; The communication strategy of the communicating party is the parameter information of the communication signal, i.e., modulation style, power and frequency; During interference, the interference frequency must be aligned with the communication frequency, and the action contains pr, so the communication power has no effect on the result. Therefore, the state S is only determined by the communication modulation pattern M. T One element, that is, S = [M T ].

5. The cognitive communication jamming method based on Transformer and deep reinforcement learning according to claim 4 is characterized in that: In step 4, the reward function is designed based on the interference effect and the power consumption of the interference signal, as follows: The design of the reward function r is as follows Among them, s is the current state, s' is the state at the next moment; s'=s represents interference failure, and the reward R1 is obtained; s'=None means the communication party stops communicating, the interference is successful, and the reward R2+(0.6-pr)×R is obtained e +(0.6-ρ)×R e ; The reward for successful interference is designed as a combination of the basic reward for successful interference and the additional reward, where R2 is the basic reward for successful interference, (0.6-pr)×R e +(0.6-ρ)×R e It is an additional reward given according to the adopted pr and ρ; the smaller the pr and ρ selected by the interference strategy, the lower the power consumption and the higher the reward obtained, so that DDQN tends to choose low-power actions; R e Represents the weight of the extra reward, and the weight means the importance of selecting low-power actions.

6. The cognitive communication jamming method based on Transformer and deep reinforcement learning according to claim 5 is characterized in that: In step 5, a Transformer Encoder-based Q network is constructed, using 1 layer of Transformer Encoder and 2 layers of FC to form the Q network, as follows: The Q network first inputs the state information s t After being encoded into a feature vector S by the Embedding layer, it is processed by the multi-head attention mechanism Multi-Head(S): Multi-Head(S)=W O [head1,head2,...,head H ] Among them, W O is the matrix parameter; H is the number of attention heads; Q = W Q i S, K = W K i S, V = W V i S; and W O Same, W Q i , W K i , W V i are matrix parameters, d model is the dimension of S; The data after multi-head attention processing is passed to the feed-forward network FFN(x) after residual connection and normalization. FFN(x) consists of two fully connected layers: FFN(x)=ReLU(W1x+b1)W2+b2 (8) Among them, W1, b1 and W2, b2 are the parameters of the two fully connected layers respectively; The result obtained by FFN processing is then passed to FC after residual connection and normalization: FC(x)=ReLU(W3x+b3)W4+b4 (9) Same as the FFN layer, W3,b3 and W4,b4 are the parameters of the two fully connected layers in FC; After FC, we get the state-action value vector V t .

7. The cognitive communication jamming method based on Transformer and deep reinforcement learning according to claim 6 is characterized in that: In step 6, the process of establishing the invalid interference list is as follows: Establish an invalid interference list L, L = [l1,l2,…,l i ,…,l 19 ],l i Used to record the interference patterns that are ineffective for interfering with the i-th modulation pattern, so as to obtain the actions that are ineffective for interfering with each state; First, we construct an action set A consisting of each modulation style, maximum power ratio, and duty cycle. m , A m ={[M Jm ,pr,ρ]|pr=1,ρ=0.9} and invalid interference list L; For any state s i , select A m Each action a m Interference i , will interfere with the invalid action a u The corresponding modulation pattern M Ju Add to i The corresponding l i In the process, all states are traversed and finally an invalid interference list L is formed.

8. The cognitive communication jamming method based on Transformer and deep reinforcement learning according to claim 7 is characterized in that: In step 7, the Q network is trained based on DDQN as follows: ① Set the number of training rounds, the target network update frequency, initialize the network parameters, and generate an invalid interference list; ② Obtain the state from the environment, use the Q network to process the state information, and obtain the predicted reward for each action; ③ Select the action to be performed according to the ε-greedy strategy and the invalid interference list; ④After executing the action, get the environmental feedback and the next state, and store the state, action and reward generated by this interaction into the experience replay pool; ⑤ Extract a fixed amount of experience from the experience replay pool to update the Q network, so that the action reward output by the Q network is closer and closer to the real reward; ⑥ Repeat steps ② to ⑤ until the number of training rounds is reached and the training ends.

9. A cognitive communication jamming system based on Transformer and deep reinforcement learning, characterized in that: The system is used to implement the cognitive communication interference method based on Transformer and deep reinforcement learning according to any one of claims 1 to 8, and the system includes a communication interference system model construction module, an interference decision module, a state and action space definition module, a reward function design module, a network construction module, an invalid interference list establishment module, a training module and a communication interference module, wherein: The communication interference system model building module establishes the communication interference system model, determines the communication interference process, and determines the modulation style of the communication party to be interfered with and the interference modulation style adopted by the interfering party; The interference decision module uses deep reinforcement learning to achieve interference decision-making. The full self-attention network, i.e. Transformer, is selected as the deep reinforcement learning network, and DDQN is used as the deep learning algorithm. The state and action space definition module defines the state and action space required for deep reinforcement learning based on the characteristics of communication signals and interference signals; Reward function design module, which designs the reward function based on the interference effect and the power consumption of the interference signal; Network construction module, build a Q network based on Transformer Encoder, use 1 layer of Transformer Encoder and 2 layers of FC to form a Q network, where Encoder represents the encoder and FC represents the fully connected layer; The invalid interference list establishment module establishes an invalid interference list for each communication modulation pattern before starting to train the Q network, and is used to record the interference modulation patterns with invalid interference; The training module trains the Q network based on DDQN, randomly selecting a state for training in each round until the specified number of training rounds is reached; The communication jammer module uses the trained Q network to perform cognitive communication jammer.

10. A mobile terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the cognitive communication interference method based on Transformer and deep reinforcement learning as described in any one of claims 1 to 8 is implemented.

Images