Cognitive communication interference method and system based on transformer and deep reinforcement learning

Abstract

The application discloses a cognitive communication interference method and system based on a Transformer and deep reinforcement learning, and the method is as follows: a Transformer Encoder is used as a network of deep reinforcement learning to process state information of enemy communication, and then a DDQN is used to select interference actions according to the processed information; a reward function is designed and an invalid interference list is established to select actions with low power consumption and improve the convergence speed of the algorithm. The system comprises a communication interference system model construction module, an interference decision module, a state and action space definition module, a reward function design module, a network construction module, an invalid interference list establishment module, a training module and a communication interference module. The application can interfere with a plurality of communication signals successfully, can reduce the number of interactions, can improve the convergence speed, and has relatively stable performance.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of communication countermeasures, and particularly relates to a cognitive communication interference method and system based on a Transformer (full self-attention network) and deep reinforcement learning. BACKGROUND

[0002] Wireless communication plays an important role in intelligence transmission and battlefield command in the military field, and how to interfere with the enemy's communication during the war has a crucial influence on winning the battle. However, with the development of frequency hopping communication, direct spread spectrum communication, adaptive technology and anti-jamming communication technology, especially the use of cognitive radio technology, it is increasingly difficult to interfere with the enemy's communication.

[0003] For this situation, a simple method is to use high-power noise to suppress the frequency band. However, this interference method not only consumes a large amount of energy, but also damages the communication of the own side. In the real environment, the own side needs to carry out real-time and high-quality communication, and the energy available for interference is limited. Therefore, it is urgent to find a low-power, accurate, intelligent and adaptive interference strategy for the enemy's communication.

[0004] Traditional interference methods are based on game theory, optimization theory and other theoretical technologies to find the optimal parameters for interference, but these studies rely on prior information of the communication parties and the environment. In the absence of prior information, these theories are not applicable. At present, more scholars use reinforcement learning to learn communication interference strategies. For example, some scholars have proposed a jamming bandit algorithm (GB) based on a multi-armed bandit framework, which can find the optimal physical layer parameters for attack. Some scholars have proposed using orthogonal decomposition to realize different interference styles, and an intelligent interference algorithm (IB) that continuously optimizes the interference action selection area during the training process. However, these interference methods still have the following defects:

[0005] (1) The research object is limited to BPSK and QPSK, 16QAM and other modulation styles, so the interference strategy learned through learning may not be optimal;

[0006] (2) The power consumption of the interference party is not considered, and when the interference party can emit a small amount of power, it will affect the interference effect.

[0007] (3) The interference method based on reinforcement learning can only train for one state, and when the enemy changes the modulation signal, it needs to find the best interference parameters again. This type of algorithm is not suitable for the rapidly changing environment in practice. SUMMARY

[0008] The application aims to provide a cognitive communication interference method and system based on a Transformer and deep reinforcement learning, and solve the problem that current communication interference technology can only interfere with a few modulation patterns and cannot intelligently adjust interference strategies according to environmental changes.

[0009] The technical solution for achieving the object of the application is a cognitive communication interference method based on a Transformer and deep reinforcement learning, comprising the following steps:

[0010] Step 1: establishing a communication interference system model, determining a communication interference process, and determining the modulation pattern of the communication party to be interfered and the interference modulation pattern adopted by the interference party;

[0011] Step 2: realizing interference decision-making by using deep reinforcement learning, selecting a full self-attention network, i.e., a Transformer, as a deep reinforcement learning network, and DDQN as a deep learning algorithm;

[0012] Step 3: defining the state and action space required by deep reinforcement learning according to the characteristics of communication signals and interference signals;

[0013] Step 4: designing a reward function according to the interference effect and interference signal power consumption;

[0014] Step 5: constructing a Q network based on a Transformer Encoder, using a 1-layer Transformer Encoder and a 2-layer FC to constitute the Q network, wherein the Encoder represents an encoder, and the FC represents a fully connected layer;

[0015] Step 6: before starting to train the Q network, establishing an invalid interference list for each communication modulation pattern, which is used to record the interference modulation pattern that is invalid for interference;

[0016] Step 7: training the Q network based on DDQN, and randomly selecting a state for training in each round until a specified number of training rounds is reached;

[0017] Step 8: realizing cognitive communication interference by using the trained Q network.

[0018] A cognitive communication interference system based on a Transformer and deep reinforcement learning, which is used to realize the cognitive communication interference method based on a Transformer and deep reinforcement learning, and comprises a communication interference system model construction module, an interference decision-making module, a state and action space definition module, a reward function design module, a network construction module, an invalid interference list establishment module, a training module, and a communication interference module, wherein:

[0019] A communication interference system model construction module establishes a communication interference system model, determines a communication interference process, and determines a communication party modulation pattern to be interfered and an interference modulation pattern adopted by an interference party;

[0020] An interference decision module adopts deep reinforcement learning to realize interference decision, selects a full self-attention network, i.e., a Transformer, as a deep reinforcement learning network, and selects DDQN as a deep learning algorithm;

[0021] A state and action space definition module defines a state and an action space required by deep reinforcement learning according to communication signal and interference signal characteristics;

[0022] A reward function design module designs a reward function according to interference effect and interference signal power consumption;

[0023] A network construction module constructs a Q network based on a Transformer Encoder, and uses a 1-layer Transformer Encoder and a 2-layer FC to constitute the Q network, wherein the Encoder represents an encoder, and the FC represents a fully connected layer;

[0024] An invalid interference list establishment module establishes an invalid interference list for each communication modulation pattern before starting to train the Q network, and the invalid interference list is used to record interference modulation patterns that are invalid for interference;

[0025] A training module trains the Q network based on DDQN, and randomly selects a state for training in each round until a specified training round number is reached;

[0026] A communication interference module adopts the trained Q network to perform cognitive communication interference.

[0027] A mobile terminal includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor realizes the cognitive communication interference method based on the Transformer and deep reinforcement learning when executing the program.

[0028] Compared with the prior art, the present application has the following advantages:

[0029] (1) The present application studies interference strategies for 19 modulation patterns, and the research range is wider than before, and the interference strategies for each modulation pattern can be effectively learned.

[0030] (2) The reward function designed in the present application takes the power consumption of the interference party into account, so that the interference strategies learned by the algorithm can successfully interfere with the communication of the communication party with low power consumption.

[0031] (3) The algorithm designed in the present application can train multiple states, effectively cope with the case that the communication party changes the modulation signal, and is suitable for changing environments. Attached Figure Description

[0032] Figure 1 This is a schematic diagram of a communication interference system model.

[0033] Figure 2 This is a schematic diagram of the algorithm's network structure.

[0034] Figure 3 For comparison, see the network structure diagram.

[0035] Figure 4 This is a schematic diagram illustrating the training effects of the network of this invention and the comparison network.

[0036] Figure 5 This is a schematic diagram illustrating the training effects of the algorithm of this invention and a comparison algorithm of deep reinforcement learning.

[0037] Figure 6 This is a schematic diagram illustrating the training effect of the interference method of the present invention and other interference methods.

[0038] Figure 7 This is a schematic diagram illustrating the training effect of the algorithm with and without invalid interference lists. Detailed Implementation

[0039] It is readily understood that, based on the technical solution of this invention, those skilled in the art can conceive of various embodiments of this invention without altering its essential spirit. Therefore, the following specific embodiments and accompanying drawings are merely illustrative examples of the technical solution of this invention and should not be considered as the entirety of this invention or as limitations or restrictions on its technical solution.

[0040] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the invention.

[0041] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the invention or its application or use.

[0042] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0043] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0044] The application designs a cognitive communication interference method based on Transformer and deep reinforcement learning to try to solve the problems existing in the prior art. First, the real environment is simulated, a communication interference model is established, and various digital signals and analog signals are fully considered. By modeling the interference model, the interference action is designed as a three-dimensional space containing modulation style, interference power, communication power ratio and duty cycle, and the advantages of deep reinforcement learning in handling multi-state space and multi-action space are used to select the interference action by deep reinforcement learning. In order to realize the selection of low-power action by the algorithm, a reasonable reward function is designed. And an interference strategy based on Transformer and deep reinforcement learning is proposed, which extracts state features by means of the multi-head attention mechanism of Transformer and uses them as network components of deep reinforcement learning. In addition, an invalid interference list is established to reduce the number of algorithm and environment interactions, thereby improving the convergence speed of the algorithm.

[0045] The application is a cognitive communication interference method based on Transformer and deep reinforcement learning, which comprises the following steps:

[0046] Step 1, establish a communication interference system model, determine the communication interference process, and determine the modulation style of the communication party to be interfered and the interference modulation style adopted by the interference party;

[0047] Step 2, realize interference decision by deep reinforcement learning, select a full self-attention network, i.e. Transformer, as the deep reinforcement learning network, and DDQN as the deep learning algorithm;

[0048] Step 3, define the state and action space required by deep reinforcement learning according to the characteristics of communication signals and interference signals;

[0049] Step 4, design a reward function according to the interference effect and the power consumption of the interference signal;

[0050] Step 5, construct a Q network based on Transformer Encoder, which is composed of 1 layer Transformer Encoder and 2 layers FC, wherein Encoder represents an encoder, and FC represents a fully connected layer;

[0051] Step 6, before starting to train the Q network, an invalid interference list is established for each communication modulation style to record the interference modulation style that is invalid for interference;

[0052] Step 7, train the Q network based on DDQN, randomly select a state for training in each round until the specified number of training rounds is reached;

[0053] Step 8, use the trained Q network to perform cognitive communication interference.

[0054] As a specific example, in step 1, a communication interference system model is established, specifically as follows:

[0055] The communication interference system model includes a sender, a receiver, an interferer and a channel, wherein the channel is used to simulate the influence of environmental noise on the signal;

[0056] The sender sends a communication signal S T (t) :

[0057] S T (t) = I T cos(2π*f c *t) + Q T sin(2π*f c *t)

[0058] Where I T and Q T are the modulation information of the in-phase and quadrature components of the sender, f c is the carrier frequency, and t represents time; I T and Q T implicitly contain the modulation pattern M T and the communication power P T information of the signal;

[0059] The interferer first obtains the communication signal parameters, including the modulation pattern M T , the communication power P T , and the carrier frequency f c , through a reconnaissance system, then selects an interference strategy, and sends an interference signal S J (t) ; The prerequisite for successful interference is that the frequency of the interference signal is aligned with that of the communication signal, so the frequencies of the two are the same;

[0060] S J (t) = I J cos(2π*f c *t) + Q J sin(2π*f c *t)

[0061] Where I J and Q J are the modulation information of the in-phase and quadrature components of the interferer;

[0062] The interference strategy consists of the interference modulation pattern M J , the ratio pr of the interference signal power P J to the communication signal power P T , and the duty cycle ρ, which is the ratio of the interference time to the intermittent interference period;

[0063] S R (t) is:

[0064] S R (t) = S T (t) + S J (t) + n(t)

[0065] = (I T + I J ) cos (2π*f c *t) + (Q T + Q J ) sin (2π*f c *t) + n(t)

[0066] where n(t) is channel noise;

[0067] The receiver determines whether it is interfered according to the bit error rate or signal-to-noise ratio of the received signal, and feeds back to the sender; the sender decides whether to change the communication behavior according to the received feedback; then, the interferer evaluates the interference effect according to the change of the communication behavior before and after interference, and selects the interference strategy according to the current communication behavior of the sender; repeat the above process until the communication party stops communication.

[0068] As a specific example, in step 1, the modulation pattern of the communication party to be interfered and the interference modulation pattern of the interferer are determined, as follows:

[0069] Communication signal modulation pattern M T including BASK, MSK, GMSK, BPSK, 4PSK, 8PSK, 16PSK, 16QAM, 64QAM, 256QAM, 2FSK, 4FSK, 8FSK, 16FSK, DQPSK, OQPSK, AM, FM, SSB, 19 modulation patterns;

[0070] Interference signal modulation pattern M J including BASK, MSK, GMSK, BPSK, 4PSK, 8PSK, 16PSK, 16QAM, 64QAM, 256QAM, 2FSK, 4FSK, 8FSK, 16FSK, DQPSK, OQPSK, AM, FM, SSB, AWGN;

[0071] Interference signal power P J The ratio pr of the communication signal power P T is discretized into 5 levels: 0.2, 0.4, 0.6, 0.8, 1;

[0072] The duty cycle p is discretized into 9 levels: 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9.

[0073] As a specific example, in step 3, the state and action space required by deep reinforcement learning is defined according to the characteristics of the communication signal and the interference signal, as follows:

[0074] The action space A is defined as a three-tuple consisting of M J , pr and p, i.e., A = [M J , pr, p];

[0075] The communication strategy of the communication party is the parameter information of the communication signal, i.e., the modulation pattern, power and frequency;

[0076] When interfering, the interference frequency is aligned with the communication frequency, and the action contains pr, so the communication power has no effect on the result, and therefore the state S consists of only the communication modulation pattern M T , i.e., S = [M T ].

[0077] As a specific example, in step 4, the reward function is designed according to the interference effect and the power consumption of the interference signal, as follows:

[0078] The design of the reward function r is as follows

[0079]

[0080] where s is the current state and s' is the next state;

[0081] s' = s represents that the interference fails, and a reward R1 is obtained;

[0082] s' = None represents that the communication party stops communication, the interference succeeds, and a reward R2 + (0.6 - pr) x R e + (0.6 - p) x R e is obtained;

[0083] The reward obtained by successful interference is designed as a combination of the basic reward for successful interference and the additional reward, where R2 is the basic reward for successful interference, (0.6 - pr) x R e + (0.6 - p) x R e is the additional reward given according to the pr and p adopted; the smaller the pr and p selected by the interference strategy, the lower the power consumption, and the higher the reward obtained, so that the DDQN is biased towards selecting low-power actions; R e represents the weight of the additional reward, and the weight size means the importance of selecting low-power actions.

[0084] As a specific example, in step 5, a Q network based on a Transformer Encoder is constructed, using 1 layer of Transformer Encoder and 2 layers of FC to constitute the Q network, as follows:

[0085] The Q network first encodes the input state information s t into a feature vector S through an Embedding layer, and then processes it through a multi-head attention mechanism Multi-Head(S):

[0086] Multi-Head(S)=W O [head1,head2,...,head H ]

[0087]

[0088] where W O is a matrix parameter; H is the number of attention heads; Q=W Q i S, K=W K i S, V=W V i S; W O , W Q i , W K i , W V i are all matrix parameters, and d model is the dimension of S;

[0089] After the multi-head attention processing, the data is transmitted to the feedforward network FFN(x) through residual connection and normalization, and FFN(x) is composed of two fully connected layers:

[0090] FFN(x)=Re L U(W1x+b1)W2+b2 (8)

[0091] where W1, b1 and W2, b2 are the parameters of the two fully connected layers, respectively;

[0092] The result obtained after FFN processing is transmitted to FC through residual connection and normalization:

[0093] FC(x)=ReLU(W3x+b3)W4+b4 (9)

[0094] The same as the FFN layer, W3, b3 and W4, b4 are the parameters of the two fully connected layers in FC;

[0095] After FC, the state-action value vector V t is obtained.

[0096] As a specific example, the process of creating the invalid interference list in step 6 is as follows:

[0097] Create an invalid interference list L, L = [l1, l2, ..., l i ,…,l 19 ], l i Used to record interference patterns that are ineffective against the i-th modulation pattern, thereby obtaining actions that are ineffective against interference in each state;

[0098] First, construct the motion set A, which consists of each modulation style, maximum power ratio, and duty cycle. m A m ={[M Jm ,pr,ρ]|pr=1,ρ=0.9} and invalid interference list L;

[0099] For any state s i Choose A m Each action a m Interference s i The action a that interferes with is invalid u Corresponding modulation style M Ju Add to s i The corresponding l i In the middle; traverse all states and finally form an invalid interference list L.

[0100] As a specific example, in step 7, the Q-network is trained based on DDQN, as follows:

[0101] ① Set the number of training rounds, the target network update frequency, initialize the network parameters, and generate an invalid interference list;

[0102] ② Obtain the state from the environment, process the state information using the Q network, and obtain the predicted reward for each action;

[0103] ③ Select the action to be performed based on the ε-greedy strategy and the list of invalid interferences;

[0104] ④ After performing an action, obtain environmental feedback and the next state, and store the state, action, and reward generated by this interaction into the experience replay pool;

[0105] ⑤ A fixed amount of experience is drawn from the experience replay pool to update the Q network, so that the action reward output by the Q network becomes closer and closer to the real reward;

[0106] ⑥ Repeat steps ② to ⑤ until the required number of training rounds is reached, at which point the training ends.

[0107] The application further provides a cognitive communication interference system based on a Transformer and deep reinforcement learning, which is used for implementing the cognitive communication interference method based on the Transformer and deep reinforcement learning.

[0108] The communication interference system model construction module establishes a communication interference system model, determines a communication interference process, and determines a communication modulation mode of a communication party to be interfered and an interference modulation mode of an interference party;

[0109] The interference decision module realizes interference decision by using deep reinforcement learning, selects a full self-attention network, i.e., a Transformer, as a deep reinforcement learning network, and selects DDQN as a deep learning algorithm;

[0110] The state and action space definition module defines a state and an action space required by deep reinforcement learning according to characteristics of a communication signal and an interference signal;

[0111] The reward function design module designs a reward function according to an interference effect and an interference signal power consumption;

[0112] The network construction module constructs a Q network based on a Transformer Encoder, and uses one layer of the Transformer Encoder and two layers of FC to constitute the Q network, wherein the Encoder represents an encoder, and the FC represents a fully connected layer;

[0113] The invalid interference list establishment module establishes an invalid interference list for each communication modulation mode before starting to train the Q network, and the invalid interference list is used for recording interference modulation modes invalid for interference;

[0114] The training module trains the Q network based on DDQN, and randomly selects a state for training in each round until a specified training round number is reached;

[0115] The communication interference module realizes cognitive communication interference by using the trained Q network.

[0116] The application further provides a mobile terminal, which comprises a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and the processor realizes the cognitive communication interference method based on the Transformer and deep reinforcement learning when executing the program.

[0117] The application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0118] Embodiment 1

[0119] The embodiment provides a cognitive communication interference method based on a Transformer and deep reinforcement learning, and comprises the following steps.

[0120] Step 1, a communication interference system model is established, a communication interference process is determined, and a communication modulation style to be interfered and an interference modulation style adopted by an interference party are determined; in the embodiment, 19 modulation styles are included in the research range.

[0121] Step 2, according to the characteristics of the communication signal and the interference signal, the state and action space required by deep reinforcement learning are defined, and deep reinforcement learning selects an interference strategy according to the state information of the communication signal;

[0122] Step 3, a reward function is designed according to the interference effect and the interference signal power consumption;

[0123] Step 4, a double deep Q network (DDQN) is selected as a deep reinforcement learning algorithm for learning an interference strategy;

[0124] Step 5, the network structure of the DDQN is constructed, and an Encoder layer of 1 layer of a Transformer (full self-attention network) and 2 layers of FC (full connection) are used as network parts;

[0125] Step 6, an invalid interference list is established: before starting training, an invalid interference list is established for each communication modulation style, which is used to record invalid interference modulation styles; during the training process, the invalid interference list is used to avoid selecting actions composed of invalid interference styles, so as to accelerate the training process and improve the convergence speed;

[0126] Step 7, the constructed invalid interference list is input into the algorithm, and training is started, a state is randomly selected each time for learning until a specified number of training rounds is reached.

[0127] The following will introduce the contents in detail:

[0128] (1) Model establishment

[0129] The communication interference system model established in the embodiment is as shown in the following figure: Figure 1

[0130] The model comprises four parts, a sender, a receiver, an interference party and a channel, wherein the channel part is responsible for simulating the influence of environmental noise and other factors on the signal.

[0131] The overall process of the model is as follows. The sender sends a communication signal S T .

[0132] S T (t)=I T cos(2π*f​c t) + Q T sin(2π*f c t)

[0133] where I and Q are the in-phase and quadrature modulation information, f c is the carrier frequency. I and Q imply the modulation type M T and the power P T of the signal.

[0134] The jammer first gets the communication signal parameters, modulation type M T , power P T , frequency f c , etc. through reconnaissance system, then selects the jamming strategy and sends the jamming signal S J . The prerequisite for successful jamming is that the frequency of the jamming signal is aligned with that of the communication signal, so they have the same frequency.

[0135] S J (t) = I J cos(2π*f c t) + Q J sin(2π*f c t)

[0136] The jamming strategy consists of the jamming modulation type M J , the ratio pr of the jamming power P J to the communication power P T , and the duty cycle p.

[0137] The signal S R after the communication signal and the jamming signal pass through the channel to the receiving party, where n(t) is the channel noise.

[0138] S R (t) = S T (t) + S J (t) + n(t)

[0139] = (I T + I J )cos(2π*f c t) + (Q T + Q J )sin(2π*f c t) + n(t)

[0140] The receiving party judges whether it is interfered according to the bit error rate (BER) or signal-to-noise ratio (SNR) of the received signal, and feeds back to the sending party. The sending party decides whether to change the communication behavior according to the received feedback, and reduces the influence of interference on communication. Then, the interference party evaluates the interference effect according to the change of communication behavior before and after interference, and selects the interference strategy according to the current communication behavior of the sending party. Repeat the above process until the communication party stops communication.

[0141] In order to interfere with a variety of modulation signals, the communication signal modulation pattern M T is composed of a variety of digital modulation patterns and analog modulation patterns, including 19 kinds of modulation patterns such as BASK, MSK, GMSK, MPSK, MQAM, MFSK, DQPSK, OQPSK, AM, FM, SSB, etc. The interference signal modulation pattern M J is composed of the above modulation patterns and (Additive White Gaussian Noise, AWGN). In order to facilitate the representation of the action space of deep reinforcement learning, the embodiment discretizes ρ and pr into 9 and 5 levels respectively. The optional values of communication parameters and interference parameters are shown in Table 1.

[0142] Table 1 for communication parameters and interference parameters

[0143]

[0144] (2) Definition of deep reinforcement learning state and action

[0145] The embodiment uses a deep reinforcement learning algorithm to achieve the target task, which first needs to define the state and action. The embodiment defines the action space A as a three-tuple composed of M J , pr and ρ, that is, A = [M J , pr, ρ]. The communication strategy of the communication party is the parameter information of the communication signal, that is, the modulation pattern, power and frequency. When interfering, the interference frequency is aligned with the communication frequency, and pr is contained in the action, so the communication power has no effect on the result. Therefore, the state S is only composed of one element of the communication modulation pattern M T , that is, S = [M T ].

[0146] (3) Reward function design

[0147] The design of the reward function r of the embodiment is as follows

[0148]

[0149] Wherein, s is the current state, s' is the next state. s' = s represents interference failure, and a reward R1 is obtained; s' = None represents that the communication party stops communication, and interference is successful, and a reward R2 + (0.6-pr) * R e +(0.6-ρ)×R e . To make the action selected by the algorithm interfere successfully with lower power consumption, an additional reward needs to be given according to the pr and p of the interference action. Therefore, the reward obtained by successful interference is designed as a combination of the basic reward of successful interference and the additional reward. Wherein, R2 is the basic reward of successful interference, (0.6-pr) * R e +(0.6-ρ)×R e . is the additional reward given according to the pr and p adopted. The smaller the pr and p selected by the interference strategy, the lower the power consumption, and the higher the reward obtained, so as to make the DDQN tend to select low-power actions. R e represents the weight of the additional reward, and the weight size means the importance of selecting low-power actions.

[0150] (4) Deep reinforcement learning algorithm selection

[0151] DDQN is selected as the implementation algorithm of the interference method in this embodiment. DDQN uses a deep learning network as a Q network to process state information, and can process multi-state and multi-action space tasks. The model constructed in this embodiment has 19 * 900 = 17100 state-action pairs, and DDQN is suitable for processing the task of the application. The parameter settings of DDQN in this embodiment are shown in Table 2.

[0152] Table 2 is the parameter setting of DDQN

[0153] Parameter name Value learning rate 0.001 γ 0.02 εmin 0.1 εmax 0.9 Batch size 128 Replay memory size 60000 episodes 7500 target network update frequency 200 [R1] -200 [R2] 200 [R e ]]> 50

[0154] (5) Network structure of DDQN

[0155] This embodiment uses 1 layer of Transformer Encoder and 2 layers of FC to constitute the Q network, and the structure is as shown in Figure 2 .

[0156] The Q network first encodes the input state information s t into a feature vector S through an Embedding layer, and then processes it through a multi-head attention mechanism.

[0157] Multi-Head(S)=W O [head1,head2,...,head H ]

[0158]

[0159] where W O is a matrix parameter; H is the number of attention heads; Q = W Q i S., K = W K i S, V = W V i S; and W O is the same as W Q i , W K i , W V i are matrix parameters, d model is the dimension of S. After multi-head attention processing, the data is transmitted to the FFN (feedforward network) through residual connection and normalization, and the FFN consists of two fully connected layers.

[0160] FFN(x) = ReLU(W1x + b1)W2 + b2 (8)

[0161] W1, b1 and W2, b2 are parameters of the two fully connected layers, respectively. The result obtained after FFN processing is transmitted to the FC through residual connection and normalization.

[0162] FC(x) = ReLU(W3x + b3)W4 + b4 (9)

[0163] W3, b3 and W4, b4 are also parameters of the two fully connected layers in the FC, the same as the FFN layer. After FC, the state-action value vector V t is obtained.

[0164] The parameter settings of the network of the embodiment are shown in Table 3.

[0165] Table 3 is the network parameter settings of the embodiment

[0166] Parameter name Value num embedding 40 embedding dim 256 H 32 d model ]]> 256 hidden layer dim of FC 1024 feedforward dim 256

[0167] (6) Invalid interference list establishment

[0168] The model constructed in the embodiment has a total of 17100 state-action pairs. For any state s iTo find a single action with a good perturbation effect from 900 actions, the model needs to interact with the environment at least 900 times. This requires selecting a different action each time and for the model to learn well from each interaction. This is the ideal situation, but in practical applications, the action selection is random, and the model's learning also requires multiple interactions. This results in the number of interactions required to learn a perturbation strategy for a single state exceeding 900. However, as the number of states increases, not only does the number of interactions with the environment need to increase, but the model also needs to adjust its parameters to balance the treatment of each state. Therefore, the number of interactions required to learn perturbation strategies for multiple modulation styles far exceeds 17,100, which consumes a significant amount of time. For a given modulation style M... Ti The state s constituted i There are certain interference patterns M Jk The action a constitutes r Even with the highest power ratio and duty cycle interference s i And it cannot interfere with success, so M Jk Other actions constituted a u It also failed to successfully interfere with s i The mathematical representation is as follows:

[0169]

[0170] If the algorithm can avoid selecting A corresponding to the current state during its interaction with the environment... u If the actions are in the set, the number of interactions between the model and the environment will be greatly reduced, and the convergence speed will be improved rapidly. Therefore, this embodiment establishes an invalid interference list L, L = [l1, l2, ..., l...]. i ,…,l 19 ], l i This is used to record the interference patterns that are ineffective against the i-th modulation pattern, thereby obtaining the action that is ineffective against interference in each state.

[0171] First, construct the motion set A, which consists of each modulation style, maximum power ratio, and duty cycle. m A m ={[M Jm ,pr,ρ]|pr=1,ρ=0.9} and invalid interference list L. For any state s i Choose A m Each action a m Interference s i The action a that interferes with is invalid u Corresponding modulation style M Ju Add to s i The corresponding l i In the middle, traverse all states to form an invalid interference list L.

[0172] (7) Start training

[0173] Before starting training, an invalid interference list is established for each communication modulation mode according to the principle of establishing an invalid interference list. The established invalid interference list is transmitted to the action selection module of the algorithm. In each round, a state is randomly selected for training. The state information is first transmitted to the network, and then the action is selected according to the invalid interference list and the ε-greedy strategy. If the selected action is in the invalid interference list, the action is reselected. After the interference is successful, this round is over, and the next round of training begins until the specified number of training rounds is reached.

[0174] Embodiment 2

[0175] Optimized with Adam optimizer. The experiment in this embodiment is based on simulation, and the channel noise is set to AWGN with a mean of 0 and a variance of 1, and the signal-to-noise ratio is 20 dB. The communication party transmits 8192 symbols at a time, and the signal data used in this embodiment is generated by the program. The receiving party judges the degree of interference of digital and analog modulation signals according to BER and SNR. In the GJB 6741-2009 Digital Communication Interference Effect Evaluation Criteria, the interference effect when the BER exceeds 0.1 is described as very strong interference, and the communication quality is severely damaged. Therefore, the condition for judging that the digital signal is successfully interfered in this embodiment is that the BER is greater than 0.1. When the SNR is less than 4.5 dB, the communication signal power is less than 3 times the interference signal power, and the communication quality is affected. The condition for judging that the analog signal is successfully interfered in this embodiment is that the SNR is less than 4.5 dB. The learning rate is set to 0.001, and ε gradually decreases from 0.9 to 0.1. The size of the experience replay pool is set to 6000, and 128-sized experiences are extracted each time to update the network. The rewards for interference failure and success are -200 and 200 respectively, and the additional reward weight is 50.

[0176] Four experiments are set in this embodiment, which are comparative experiments on network structure, deep reinforcement learning algorithm, interference algorithm, and ablation experiment of invalid interference list. The goal of the present application is to find an accurate and low-power interference strategy for multiple states. The success rate of interference, as the ratio of the number of successful states to the total number of states, can effectively reflect the accuracy of the interference strategy. The reward function takes the impact of interference power consumption into account, and the lower the power consumption of the interference action, the greater the reward obtained. Therefore, the total reward of interference for all states can reflect whether the model can find lower power consumption actions. In addition, the more times the model interacts with the environment, the more resources it consumes in practical applications, so the convergence speed of the model is also important. Loss, as the difference between the predicted value of the Q network and the target value, can effectively reflect the convergence speed of the algorithm in real time. After the above analysis, the present application selects loss, interference success rate and total interference reward as indicators to evaluate the performance of the model. The present application records the loss during the training phase, and every 500 rounds of training, the current model is tested to calculate the interference success rate and total interference reward for all states.

[0177] Embodiment 3

[0178] This embodiment compares the network structure:

[0179] This embodiment uses a 1-layer Transformer Encoder and a 2-layer FC to implement the network part of the algorithm. In order to prove that it can better process state information and make the algorithm successful for multiple state interference, this embodiment compares it with other neural networks. The network structure of the present application and the comparison network structure are shown in Figure 3 In order to facilitate representation, Trans1 is used to represent the network used in the present application, and Trans2, LSTM, GRU, FC1024, and FC4096 are used to represent the comparison networks respectively. Trans2 uses a 2-layer Transformer Encoder and a fully connected layer. LSTM and GRU use 4-layer LSTM and GRU respectively with a fully connected layer to form the network structure. Trans1, Trans2, LSTM and GRU use the same fully connected layer. FC1024 and FC4096 both use 5-layer fully connected, with hidden layers of 1024 and 4096 respectively.

[0180] The learning effect of interference algorithms using different network structures is shown in Figure 4 and Table 4, Figure 4 which respectively show the real-time change of loss during training, and the interference success rate of each interference test. Table 4 shows the highest reward obtained by different network structures. As Figure 4As shown, the length of the loss value curve of different networks is different, and the length of the curve represents the number of network updates. The longer the curve, the more interactions and the worse the convergence ability. The loss value of Transl network decreases the fastest, has the lowest value, and has the shortest length. The test curve is also relatively stable. The loss of Trans2 is higher than that of Transl, and the interference success rate is difficult to maintain at 100%. LSTM and GRU are difficult to achieve convergence state. The fully connected network can achieve good results when the training time is long enough, but its network performance is poorer than that of the Transformer. Transl uses Embedding to expand the state information obtained from the environment, and the multi-head attention mechanism and normalization operation used by it make it easier to extract useful information from it, thereby enhancing the algorithm performance.

[0181] Table 4 is the highest reward obtained during the training process of the network of the present application and the comparative network

[0182] Network structure Highest total reward Trans1 3855 Trans2 3890 LSTM 2485 GRU 1785 FC4096 3865 FC1024 3700

[0183] Table 5 is the training time and computational amount required by the network of the present application and the comparative network

[0184]

[0185]

[0186] Table 5 shows the training time and computational amount required by different networks. Transl has the smallest computational amount and the shortest running time. Although LSTM and GRU have small computational amount, they cannot converge, resulting in long running time.

[0187] Through the above analysis, the Transformer is more suitable for processing the state information of the present application than other networks. The increase in the number of parameters can make the network better distinguish the difference between different actions. However, the increase in the number of parameters also makes the algorithm more difficult to converge and the computational amount larger. Therefore, the network of the present application has the advantages of smaller number of network parameters, faster convergence speed, and better stability.

[0188] Example 4

[0189] This example compares deep reinforcement learning algorithms:

[0190] In order to show the advantages of DDQN, the algorithm can converge faster and more stably. The present application compares DDQN with other deep reinforcement learning algorithms, such as DQN (deep Q network), Dueling DQN (dueling deep Q network), and DQNPer (priority experience replay deep Q network). These are value-based reinforcement learning algorithms, which belong to the same type of algorithm as DDQN and are the most commonly used algorithms in the interference method.

[0191] The interference algorithm learning effects using different deep reinforcement learning algorithms are as shown in Figure 5 and Table 6. Figure 5 The real-time change results of loss during the training process and the interference accuracy during each interference test are respectively shown. Table 6 shows the highest rewards obtained by different algorithms. The loss values and the falling speeds of DDQN and Dueling DQN are almost equivalent. The loss curve of Dueling DQN is shorter, but the test result is unstable, and the obtained reward is the smallest. DQN has overestimation problem, and the loss value is higher than that of DDQN and Dueling DQN. The algorithm is more unstable than DDQN. The loss change trend of DQN Per is completely opposite to other algorithms, and the interference success rate is difficult to maintain at 100%. The reason for this phenomenon is that DQN Per always preferentially selects data with large loss for learning, is more likely to select low-power actions, can obtain the highest reward, but increases the risk of interference failure, and therefore the test effect is poor.

[0192] Table 6 is the highest reward obtained by the algorithm of the present application and the comparative algorithm during the training process

[0193] Algorithm Highest total reward DDQN 3855 Dueling DQN 3805 DQN Per 3875 DQN 3845

[0194] The DDQN algorithm used in the present application has the advantage of better stability compared to other deep reinforcement learning algorithms, and is more suitable for implementing the interference task.

[0195] Embodiment 5

[0196] This embodiment compares interference algorithms:

[0197] The interference method of the present application is compared with the JB and IB algorithms proposed in previous studies. Both of these two methods are based on reinforcement learning and do not use neural networks to process state information, but can only learn the interference strategy of a single state. In order to fairly compare the method of the present application with the two, only one state is trained. 1000 times of training are performed on the three algorithms, and the training process is as shown in Figure 6 The training curve of JB is different from the other two algorithms, and more interactions are needed to find a suitable interference strategy. The algorithm of the present application and the IB algorithm can remove some invalid interference actions. The difference is that the algorithm of the present application removes some invalid interference actions in advance and does not change thereafter. The IB algorithm continuously adds invalid interference actions during the training process, and can remove more invalid interference actions. Therefore, the IB algorithm converges faster than the algorithm of the present application. The highest rewards obtained by different interference algorithms are shown in Table 7. The rewards obtained by the algorithm of the present application and the JB algorithm are the same, and the reward obtained by the IB algorithm is lower.

[0198] Table 7 is the reward obtained by the interference method of the present application and the comparative interference method

[0199] Algorithm Reward Our 205 IB 200 Figure 7 205

[0200] In summary, the method of the present application can achieve comparable learning effect when training for a single state compared to previous interference algorithms. In addition, the method of the present application can train for multiple states, which other interference methods cannot do.

[0201] Example 6

[0202] This example compares the presence and absence of invalid interference list:

[0203] The present application constructs an invalid interference list before training, and then no longer selects the actions in the list during the training process, thereby reducing the number of interactions and improving the convergence speed of the algorithm. The finally constructed invalid interference list removes a total of 6930 invalid state-action pairs, accounting for about 40% of the total state-action space. These invalid state-action pairs will not be selected during the training process, and the number of interactions between the algorithm and the environment can be greatly reduced.

[0204] Table 8 is the number of interactions required by the algorithm with and without the invalid interference list and the highest reward obtained

[0205]

[0206] Figure 7 And Table 8 shows the effect of the presence and absence of the invalid interference list on the training effect of the algorithm. Figure 7 is the interference test success rate curve of the algorithm with and without the invalid interference list. Table 8 lists the number of interactions of the algorithm with and without the invalid interference list before 500 rounds and when the interference test first reaches 100%, and the highest reward obtained after 7500 rounds of training. From ​ it can be seen that the algorithm without adding the invalid interference list has poor stability and cannot always be stable at 100% accuracy. From Table 8, it can be seen that the number of disturbances required by the algorithm without adding the invalid interference list before 500 rounds is about twice that of the algorithm with the invalid interference list. When the interference success rate first reaches 100%, the number of interactions of the former is about 1.5 times that of the latter. By comparing the number of interactions between the two and the environment, it can be seen that adding the invalid interference list can significantly reduce the number of interactions between the algorithm and the environment. However, after adding the invalid interference list, the reward obtained is smaller, indicating that this method has a certain impact on the performance of the algorithm.

[0207] The above description is only a preferred specific embodiment of the present application, but the protection scope of the present application is not limited thereto, and any changes or replacements within the technical scope disclosed by the present application can be easily thought of by those skilled in the art, which should be covered within the protection scope of the present application.

[0208] It is to be understood that, in describing the examples of the application above, various features of the application are sometimes grouped together in a single embodiment, or described in a single figure, for the purpose of streamlining the disclosure and aiding in the understanding of various aspects of the application. This is for illustrative purposes only and is not intended to limit the application as claimed.

Claims

1. A cognitive communication jamming method based on Transformer and deep reinforcement learning, characterized in that, The method comprises the following steps: Step 1, a communication interference system model is established, a communication interference process is determined, and a modulation pattern of a communication party to be interfered and a modulation pattern of an interference party are determined; Step 2, deep reinforcement learning is used to realize interference decision, a full self-attention network, namely, a Transformer, is selected as a deep reinforcement learning network, and DDQN is selected as a deep learning algorithm; Step 3, according to characteristics of communication signals and interference signals, a state space and an action space required by deep reinforcement learning are defined, and the state space and the action space are specifically as follows: The action space A is defined by the interference modulation pattern M. J Interference signal power P J With communication signal power P T The triple consisting of the ratio pr and the duty cycle ρ, i.e. ; A duty cycle ρ is a ratio of interference time to an intermittent interference period; A communication strategy of the communication party is parameter information of the communication signal, namely, a modulation pattern, power and frequency; In the interference time, the interference frequency is aligned with the communication frequency, and the action contains pr, so the communication power has no effect on the result, and therefore the state S is only determined by the communication modulation pattern M T One element consists of, namely ; Step 4, a reward function is designed according to an interference effect and interference signal power consumption; Step 5, a Q network based on a Transformer Encoder is constructed, and the Q network is composed of one layer of the Transformer Encoder and two layers of FC, wherein the Encoder represents an encoder, and the FC represents a fully connected layer; Step 6, before starting to train the Q network, an invalid interference list is established for each communication modulation pattern, and the invalid interference list is used to record invalid interference modulation patterns; Step 7, the Q network is trained based on DDQN, a state is randomly selected for training in each round, and the training is performed until a specified number of training rounds is reached; Step 8, cognitive communication interference is performed by using the trained Q network.

2. The cognitive communication jamming method based on Transformer and deep reinforcement learning according to claim 1, characterized in that, In step 1, the communication interference system model is established, and the communication interference system model specifically comprises the following: The communication interference system model comprises a sending party, a receiving party, an interference party and a channel, wherein the channel is used to simulate an influence of environmental noise on a signal; The sender transmits the communication signal : ； wherein, and are the modulation information of the in-phase and quadrature components of the transmitter, is the carrier frequency, and t represents time; and implicit in the signal modulation pattern M T and the communication power P T information; The jammer first gets the communication signal parameters through the reconnaissance system, including modulation pattern M T , communication power P T , carrier frequency f c , and then selects a jamming strategy to send a jamming signal ; the prerequisite for successful jamming is that the frequency of the jamming signal is aligned with that of the communication signal, so they are the same. ； wherein, and are the modulation information of the in-phase and quadrature components of the interfering party. The interference strategy is composed of an interference modulation pattern M J , a ratio pr of an interference signal power P J to a communication signal power P T , and a duty cycle ρ. Signal after communication signal and interference signal pass through channel to reach receiver To: ； Wherein n(t) is channel noise; The receiving party judges whether to be interfered according to a bit error rate or a signal-to-noise ratio of the received signal, and feeds back to the sending party; the sending party decides whether to change a communication behavior according to the received feedback; then, the interference party evaluates an interference effect according to changes of communication behaviors before and after interference, and selects an interference strategy according to a current communication behavior of the sending party; the above process is repeated until the communication party stops communication.

3. The cognitive communication jamming method based on Transformer and deep reinforcement learning according to claim 2, characterized in that, In step 1, the modulation pattern of the communication party to be interfered and the interference modulation pattern of the interference party are determined, and the modulation pattern and the interference modulation pattern are specifically as follows: Modulation pattern M of the communication signal T The 19 modulation patterns include BASK, MSK, GMSK, BPSK, 4PSK, 8PSK, 16PSK, 16QAM, 64QAM, 256QAM, 2FSK, 4FSK, 8FSK, 16FSK, DQPSK, OQPSK, AM, FM, SSB. Interfering signal modulation pattern M J including BASK, MSK, GMSK, BPSK, 4PSK, 8PSK, 16PSK, 16QAM, 64QAM, 256QAM, 2FSK, 4FSK, 8FSK, 16FSK, DQPSK, OQPSK, AM, FM, SSB, AWGN; The ratio pr of the interference signal power P J to the communication signal power P T is discretized into 5 levels: 0.2, 0.4, 0.6, 0.8, 1. The duty cycle ρ is discretized into nine levels: 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8 and 0.

9.

4. The cognitive communication jamming method based on Transformer and deep reinforcement learning according to claim 3, characterized in that, In step 4, the reward function is designed according to the interference effect and the interference signal power consumption, and the reward function is specifically as follows: The reward function r is designed as follows ； Wherein s is a current state, and s' is a next time state; s' = s represents that interference fails, and a reward R1 is obtained; s' = None represents that the communication party stops communication, the interference is successful, and a reward is obtained ; The reward for successful interference is designed as a combination of a basic reward for successful interference and an additional reward, where R2 is the basic reward for successful interference, is the additional reward given according to the adopted pr and p; the smaller the pr and p selected by the interference strategy, the smaller the power consumption and the higher the reward obtained, thereby causing the DDQN to prefer to select a low-power action; R e represents the weight of the additional reward, and the weight size means the importance of selecting a low-power action.

5. The cognitive communication jamming method based on Transformer and deep reinforcement learning according to claim 4, characterized in that, In step 5, the Q network based on the Transformer Encoder is constructed, and the Q network is composed of one layer of the Transformer Encoder and two layers of FC, and the Q network is specifically as follows: The Q network first encodes the input state information s t is encoded into a feature vector S by an Embedding layer, and then processed by a multi-head attention mechanism Process: ； ； wherein is a matrix parameter; H is the number of attention heads; , , ; and are the same as in , , are matrix parameters, is the dimension of S; The data after multi-head attention processing is transmitted into the feedforward network after residual connection and normalization , consists of two fully connected layers: ； wherein, , and , are parameters of two fully connected layers, respectively; A result obtained through FFN processing is transmitted to FC through residual connection and normalization: ； The same as FFN layers, , and , are parameters of two fully connected layers in FC. The state-action value vector V is obtained after the FC t .

6. The cognitive communication jamming method based on Transformer and deep reinforcement learning according to claim 5, characterized in that, In step 6, a process of establishing the invalid interference list is specifically as follows: Create an invalid interference list L, L=[l1, l2,…, l i ,…, l 19 ], l i Used to record interference patterns that are ineffective against the i-th modulation pattern, thereby obtaining actions that are ineffective against interference in each state; First, an action set A consisting of each modulation pattern and maximum power ratio, duty cycle is constructed m , and an invalid interference list L; For any state s i , select A m each action a m interferes with s i , the action a u corresponding modulation pattern M Ju is added to s i corresponding l i ; all states are traversed, and finally form the invalid interference list L.

7. The cognitive communication jamming method based on Transformer and deep reinforcement learning according to claim 6, characterized in that, In step 7, the Q network is trained based on DDQN, and the Q network is trained as follows: ① Set the number of training rounds, target network update frequency, and initialize network parameters, and generate invalid interference list; ② Obtain the state from the environment, process the state information using the Q network, and get the predicted reward of each action; ③ Select the action to be executed according to the ε-greedy strategy and the invalid interference list; ④ After executing the action, get the environment feedback and the next state, and store the state, action and reward generated by this interaction into the experience replay pool; ⑤ Extract a fixed size of experience from the experience replay pool for updating the Q network, so that the action reward output by the Q network is closer and closer to the true reward; ⑥ Repeat steps ②~⑤ until the number of training rounds is reached, and the training is completed.

8. A cognitive communication jamming system based on Transformer and deep reinforcement learning, characterized in that, The system is used to implement the cognitive communication interference method based on Transformer and deep reinforcement learning according to any one of claims 1~7, and the system comprises a communication interference system model construction module, an interference decision module, a state and action space definition module, a reward function design module, a network construction module, an invalid interference list establishment module, a training module and a communication interference module, wherein: The communication interference system model construction module establishes a communication interference system model, determines the communication interference process, and determines the communication modulation style to be interfered and the interference modulation style adopted by the interference party; The interference decision module uses deep reinforcement learning to realize interference decision, selects a full self-attention network, i.e., Transformer, as a deep reinforcement learning network, and DDQN as a deep learning algorithm; The state and action space definition module defines the state and action space required by deep reinforcement learning according to the characteristics of communication signals and interference signals; The reward function design module designs a reward function according to the interference effect and interference signal power consumption; The network construction module constructs a Q network based on Transformer Encoder, which uses 1 layer Transformer Encoder and 2 layers FC to constitute the Q network, wherein Encoder represents an encoder and FC represents a fully connected layer; The invalid interference list establishment module establishes an invalid interference list for each communication modulation style before starting to train the Q network, which is used to record the interference modulation style that is invalid for interference; The training module trains the Q network based on DDQN, and randomly selects a state for training each round until the specified number of training rounds is reached; The communication interference module uses the trained Q network to perform cognitive communication interference.

9. A mobile terminal comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the cognitive communication interference method based on Transformer and deep reinforcement learning according to any one of claims 1~7 when executing the program.

Images