An intelligent decision-making method for multi-user communication anti-interference based on deep reinforcement learning

Abstract

The present invention discloses a multi-user communication anti-interference intelligent decision-making method based on deep reinforcement learning, comprising the steps of: constructing a multi-user wireless communication anti-interference system model, firstly, the base station uses the current spectrum information of the perceived multiple users and jammers as the input of the strategy neural network of deep reinforcement learning, then selects a joint action according to a dynamic greedy algorithm, and helps users intelligently select the communication frequency band through base station feedback; at the same time, calculates the immediate reward generated by the joint action in the current time slot, and stores the experience in an experience replay pool. When the number of experiences in the experience replay pool reaches a given value, a certain number of experiences are randomly drawn from the experience pool to update the parameters of the strategy neural network, and the parameters of the target neural network are updated once every fixed time slot; repeating the training process to complete the multi-user communication anti-interference intelligent decision-making method. The present invention can realize multi-user communication anti-interference and effectively avoid communication interference caused by external jammers and internal users.

Description

Technical Field

[0001] The present invention relates to a multi-user communication anti-interference intelligent decision-making method, and in particular to a multi-user communication anti-interference intelligent decision-making method based on deep reinforcement learning. Background Art

[0002] Interference attacks are a difficult problem to address in wireless communication networks. Typically, malicious jamming signals disrupt the normal data reception of legitimate transmission links, posing a serious threat to communication security. Furthermore, in multi-user scenarios with multiple transmission links, communication performance degrades even more severely due to external malicious interference and internal co-channel interference. Therefore, the issue of collaborative anti-interference in communications requires further research.

[0003] Spread spectrum technology is a mainstream anti-interference technology for communications, with frequency-hopping spread spectrum and direct sequence spread spectrum being widely used. These technologies offer significant anti-interference effectiveness against conventional jammers, such as sweep frequency jammers, pulse jammers, and broadband blocking jammers. However, traditional anti-interference methods have limitations. For example, frequency-hopping spread spectrum relies on a predetermined frequency-hopping pattern, while direct sequence spread spectrum relies on a local pseudo-random code. Furthermore, the development of artificial intelligence and software-defined radio (SDR) technologies, such as the diversity, dynamism, and intelligence of jammers, are placing higher demands on communication anti-interference technology. Summary of the Invention

[0004] Purpose of the invention: The purpose of the present invention is to provide a multi-user communication anti-interference intelligent decision-making method based on deep reinforcement learning that can effectively deal with external malicious interference and avoid mutual interference between users.

[0005] Technical Solution: The multi-user communication anti-interference intelligent decision-making method of the present invention first constructs a wireless communication anti-interference system model for multiple users, and then helps each user intelligently select the optimal communication frequency band through feedback from the base station. The method includes the following steps:

[0006] S1, builds a multi-user wireless communication anti-interference system model consisting of multiple users, a base station and a jammer. Users, base stations and jammers are randomly distributed in an open area and share a spectrum space;

[0007] S2, the base station obtains the current spectrum information of multiple users and jammers;

[0008] S3 builds two convolutional neural network models, uses the current spectrum information as input, selects joint actions based on a dynamic greedy algorithm, and uses base station feedback to help users intelligently select communication frequency bands.

[0009] S4, calculate the immediate reward generated by the joint action in the current time slot and store the experience in the experience replay pool; the experience includes the current spectrum selection state, joint action, immediate reward and next spectrum selection information;

[0010] S5: When the number of experiences in the experience replay pool reaches a given value, a certain number of experiences are randomly drawn from the experience pool to update the parameters of the policy neural network, and the parameters of the target neural network are updated once every fixed time interval; the iteration stops when the set number of iterations is reached.

[0011] Furthermore, in step S2, the base station receives the signal to interference and noise ratio SINR of user u. u Determine whether user u has successfully communicated. If the communication is successful, the normalized threshold r u (f) is 1, otherwise it is 0;

[0012] The signal-to-interference-and-noise ratio (SINR) of user u received by the base station u for:

[0013]

[0014] Among them, G u represents the channel gain from user u to the base station, G j represents the channel gain from the jammer to the base station, U j (f) represents the power spectrum density of the jammer, f represents the signal frequency, f k Indicates the center frequency of channel k selected by user u, f l represents the interference frequency of a user-selected channel l, n(f) represents the power spectrum density of the noise, represents the co-channel interference from other users in the user set when user u selects channel k; δ(ε) is an indicator function, which is 1 if ε is true and 0 otherwise; Represents a user set;

[0015] Defining β th is the threshold of signal-to-noise ratio transmission. When the SINR of user u is received, u Greater than β th When the signal to interference and noise ratio SINR of user u is received, it indicates that the transmission is successful; u Less than or equal to β th When the transmission fails, the normalized threshold g u (f) is:

[0016]

[0017] Furthermore, in step S3, the two convolutional neural networks are a strategy neural network with a weight parameter of θ and a policy neural network with a weight parameter of θ. -The target neural network is randomly initialized with weight parameters; the two-dimensional spectrum waterfall O t As the input of the neural network, it passes through four convolutional layers, and then the output of the convolution is flattened into one-dimensional data through the expansion layer, and then passes through four fully connected layers to obtain the final output value;

[0018] The dynamic ε-greedy algorithm is used to select the joint action a(t) as follows:

[0019] The probability of randomly selecting action a(t) in each iteration is ε, and the policy network Q is selected policy The maximum action a′=argmax a Q policy (O t ,a;θ i ) has a probability of 1-ε, where ε0 is the initial greedy probability, decay is the decay coefficient, i is the number of iterations, ε decreases exponentially as the number of iterations i increases, and e is a natural constant.

[0020] Furthermore, the spectrum waterfall O t The solution process is as follows:

[0021] The discrete spectrum sampling values ​​are defined as follows:

[0022]

[0023] Where S(f) represents the user power spectrum density received by the base station, Δf is the resolution of spectrum analysis, and i is a positive integer representing the number of samples.

[0024] The spectrum status results observed by the base station each time are:

[0025] o' t =[o' 1,t ,o' 2,t …o' L,t ]

[0026] The spectrum waterfall O t Defined as:

[0027] O t =[o' t ,o' t+1 …o' t+W-1 ]

[0028] Among them, W represents the number of historical states to be traced back, O t It is a two-dimensional matrix of size W×L, containing information in the frequency domain and time domain.

[0029] Furthermore, in step S4, the process of calculating the immediate reward generated by the action in the current time slot is as follows:

[0030] The action space is represented as:

[0031] A={a1,a2,…,a n×m}

[0032] Where n is the number of users, m is the number of channels, and a q represents the joint action taken by each user at time t, q = 1, 2, ..., n × m, and there are a total of n × m joint actions a(t);

[0033] Then the joint action a(t) at time t is:

[0034] a(t)=[f1(t),f1(t),…,f n (t)]

[0035] Among them, f n (t) represents the center frequency of the channel selected by user n at time t;

[0036] The state transition probability refers to the probability that the user set is in state O. t After executing the joint action a(t), the state is transferred to O. t+1 The probability of is expressed as:

[0037] P:(O t ,a)→O t+1

[0038] For the immediate reward r(t), it is defined as:

[0039]

[0040] Where c is the frequency hopping cost.

[0041] Furthermore, in step S5, when the number of elements in the experience replay pool D is greater than one batch, Num samples e are randomly selected from the experience replay pool D. batch ={e k ,e k ~U(D),k=1,2,...,Num}, the parameters θ of the policy network are adjusted by the gradient descent algorithm i Iterative update; for the parameters of the target network The parameters of the policy network are copied regularly to achieve parameter updates;

[0042] After the training is completed, the environment state O t Input strategy network calculation, get output Q(O t ,a;θ), where a represents the joint action taken by the user, θ represents the weight of the strategy network, and the action corresponding to the maximum Q value is selected. The feedback from the base station helps each user select the optimal communication frequency band to resist interference.

[0043] Compared with the prior art, the present invention has the following significant effects:

[0044] 1. In the multi-user communication anti-interference scenario, the present invention uses deep reinforcement learning, which is no longer limited to traditional reinforcement learning. At the same time, it adopts a dynamic ε-greedy strategy to improve the learning rate and accelerate the convergence speed of the algorithm;

[0045] 2. The present invention constructs a wireless communication anti-interference system model for multiple users. It is no longer limited to randomly selecting communication frequency bands through frequency hopping technology. Instead, it helps each user intelligently select the optimal communication frequency band based on the current spectrum status through feedback from the base station, that is, selects the frequency band with the least possibility of interference. It can effectively deal with external malicious interference and avoid internal interference between users. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is a system flow chart of the present invention;

[0047] Figure 2 This is a model diagram of the multi-user wireless communication anti-interference system of the present invention;

[0048] Figure 3 This is a diagram of the neural network structure of deep reinforcement learning in the present invention;

[0049] FIG4( a ) is a spectrum diagram of a swept frequency interference mode in an embodiment of the present invention;

[0050] FIG4( b ) is a spectrum diagram of a comb interference pattern in an embodiment of the present invention;

[0051] FIG4( c ) is a spectrum diagram of a dynamic interference pattern in an embodiment of the present invention;

[0052] FIG4( d ) is a spectrum diagram of the intelligent jamming mode in an embodiment of the present invention;

[0053] FIG4( e ) is a spectrum diagram of a random interference pattern in an embodiment of the present invention;

[0054] FIG5( a ) is a spectrum diagram of an anti-interference model in a swept frequency interference mode according to an embodiment of the present invention;

[0055] FIG5( b ) is a spectrum diagram of the anti-interference model under the comb interference mode in an embodiment of the present invention;

[0056] FIG5( c ) is a spectrum diagram of the anti-interference model in the dynamic interference mode according to an embodiment of the present invention;

[0057] FIG5( d ) is a spectrum diagram of the anti-interference model in the intelligent interference mode according to an embodiment of the present invention;

[0058] Figure 5(e) is the spectrogram of the anti-jamming model under the random interference pattern in the embodiment of the present invention;

[0059] Figure 6 It is a comparison diagram of the normalized throughput of each user under the dynamic interference pattern in the embodiment of the present invention. Specific implementation manners

[0060] The multi-user communication anti-jamming intelligent decision-making method of the present invention will be further described in detail below in conjunction with the specification drawings and specific implementation manners.

[0061] For the multi-user communication anti-jamming intelligent decision-making method of the present invention, first, a multi-user wireless communication anti-jamming system model is constructed. First, the base station uses the currently perceived spectrum information of multiple users and jammers as the input of the policy neural network of deep reinforcement learning (to implement the selection of channel policy actions), and then selects joint actions (i.e., multi-user communication channel selection) according to the dynamic greedy algorithm, and helps users intelligently select communication frequency bands through the base station feedback. At the same time, calculate the immediate reward generated by the joint action in the current time slot, and store the experience (the current spectrum selection state, action, immediate reward, next spectrum selection information) in the experience replay pool. When the number of experiences in the experience replay pool reaches a given value, randomly extract a certain number of experiences from the experience pool to update the parameters of the policy neural network, and update the parameters of the target neural network every fixed time slot. Repeat the above process. Finally, each user can intelligently select the optimal communication frequency band, effectively cope with external malicious interference, and avoid internal mutual interference between users. The overall flow chart is as Figure 1 shown, and specifically includes the following steps:

[0062] Step 1: Build a multi-user wireless communication anti-jamming system model

[0063] As Figure 2 shown, the multi-user wireless communication anti-jamming system model of the present invention consists of multiple users, a base station, and a jammer. The users, the base station, and the jammer are randomly distributed in an open area and share a spectrum space. The users communicate with the base station respectively, and the user set is expressed as:

[0064]

[0065] The spectrum space is divided into multiple channels, and the channel set is expressed as:

[0066]

[0067] The number of available channels is m (n < m), the bandwidth of each channel is b, and the transmission power p of the kth user k is:

[0068]

[0069] Among them, U k (f) is the power spectral density of the signal transmitted by the kth user, n is the number of users, and m is the number of channels.

[0070] The set of channels interfered by the jammer is:

[0071] J = {1, ..., j} (4)

[0072] Where j is the number of interference channels.

[0073] When the interfering channel is the same as the user's communication channel, the interference is successful. If two or more users select the same frequency band, mutual interference will occur. To achieve reliable transmission, both external malicious interference and mutual interference caused by competition between users must be considered. The base station carries an intelligent agent with spectrum sensing, learning, and decision-making capabilities. After each communication time slot, it defines the received spectrum information of the current time slot as the environmental state, uses a deep reinforcement learning algorithm to make anti-interference decisions, and notifies each user of the communication channel for the current time slot.

[0074] Step 2: SINR of user u received by the base station u for:

[0075]

[0076] Among them, G u represents the channel gain from user u to the base station, G j represents the channel gain from the jammer to the base station, U j (f) represents the power spectrum density of the jammer, f k represents the center frequency of channel k selected by user u, n(f) represents the power spectral density of noise, f represents the signal frequency, and f l represents the interference frequency of a user's selected channel l, represents the co-channel interference from other users in the user set when user u selects channel k; δ(ε) is an indicator function, which is 1 if ε is true and 0 otherwise.

[0077] Defining β th is the threshold of signal-to-noise ratio transmission. When the SINR of user u is received, u Greater than β th When the signal to interference and noise ratio SINR of user u is received, it indicates that the transmission is successful; u Less than or equal to β th When the transmission fails, the normalized threshold g u (f) is:

[0078]

[0079] Step 3: Define the discrete spectrum sampling value as

[0080]

[0081] Where S(f)=∑ k∈N G k U k (ff k )+G j U j (ff k )+n(f) represents the user power spectrum density received by the base station, Δf is the resolution of spectrum analysis, and i is a positive integer and is used to represent the number of samples.

[0082] The spectrum status results observed by the base station each time are:

[0083] o' t =[o' 1,t ,o' 2,t …o' L,t ] (8)

[0084] Define the state of the environment as:

[0085] O t =[o' t ,o' t+1 …o' t+W-1 ] (9)

[0086] Among them, W represents the number of historical states to be traced back, O t is a two-dimensional matrix of size W×L, and O t The thermodynamic diagram is called a spectral waterfall, which contains information in both the frequency domain and the time domain.

[0087] Step 4: Use deep reinforcement learning algorithm to solve the anti-interference problem and convert the spectrum waterfall into t Set to the current state of the environment at time t;

[0088] The action space is:

[0089] A={a1,a2,…,a n×m} (10)

[0090] Where n is the number of users, m is the number of channels, and a q represents the joint actions that each user can take at time t, q = 1, 2, ..., n × m, and there are a total of n × m joint actions a(t).

[0091] Then the joint action a(t) at time t is:

[0092] a(t)=[f1(t),f2(t),…,f n (t)] (11)

[0093] Among them, f n (t) represents the center frequency of the channel selected by user n at time t;

[0094] The state probability is expressed as:

[0095] P:(O t ,a)→O t+1 (12)

[0096] It means that the user set is in state O t After executing the joint action a(t), the state is transferred to O. t+1 The transition probability.

[0097] The immediate reward r(t) is defined as:

[0098]

[0099] Where c is the frequency hopping cost.

[0100] Step 5: Due to the environmental state O t It is based on the unknown probability P(O t+1 |O t , a(t)) changes dynamically, and the state-action space is very large in the anti-interference decision process. A deep convolutional neural network (CNN) is used to approximate the Q function of each state-action pair (state O t and the expected discounted long-term reward of action a(t), namely:

[0101]

[0102] Among them, r(t) is the immediate reward at the current time t, a′ is the joint action taken by the user set when the Q value is the largest, O t+1 If the user set is in state O t is the next state where the joint action a(t) is taken, and γ is the discount factor.

[0103] like Figure 3 As shown, two convolutional neural networks are established, one is a policy neural network with a weight parameter of θ, and the other is a policy neural network with a weight parameter of θ - The target neural network is randomly initialized with weight parameters. t As the input of the neural network, it passes through four convolutional layers, and then the convolution output is flattened into one-dimensional data through the expansion layer, and then passes through four fully connected layers to obtain the final output value. In this way, the neural network can be used to implement nonlinear functions to represent the Q function.

[0104] The experience e at each time step t t =(Ot ,a(t),r(t),O t+1 ) is stored in the dataset D t , and by randomly selecting elements in the uniform distribution e~U(D), the target value η of machine learning can be obtained i :

[0105]

[0106] e t =(O t ,a(t),r(t),O t+1 ) (16)

[0107] D t =(e1,...,e t ) (17)

[0108] in, is the parameter of the target Q network at the i-th iteration, and a′ is the parameter of the policy network Q policy Maximum action. When input is O t When the target Q network output is η i Assume that the parameters of the policy Q network at the i-th iteration are θ i , the mean square error between the target value and the actual output of the policy Q network can be used as the loss function:

[0109]

[0110] The gradient of the loss function is:

[0111]

[0112] Step 6: During the training phase, according to state O t , the dynamic ε-greedy algorithm is used to select the joint action a(t), that is, the probability of randomly selecting action a(t) in each iteration is ε, and the policy network Q is selected policy The maximum action a′=argmax a Q policy (O t ,a;θ i ) has a probability of 1-ε, where is the initial greedy probability, decay is the decay coefficient, i is the number of iterations, and ε decreases exponentially with the increase of the number of iterations i. In order to ensure the exploratory nature of the algorithm, ε will not decay to 0. t =(O t ,a(t),r(t),O t+1 ) is stored in the experience replay pool D. After the experience replay pool D is full, the experience replay pool is updated with new samples according to the first-in-first-out principle.

[0113] After the number of elements in the experience replay pool D is greater than one batch, randomly select Num samples e in the experience replay pool D batch ={e k ,e k ~U(D),k=1,2,...,Num}, the parameters θ of the policy network are adjusted by the gradient descent algorithm i Iterative update, for the parameters of the target network The parameters of the policy network are copied periodically (every iteration C times) to achieve parameter update, and the training process is repeated until the maximum number of iterations is reached.

[0114] After the training is completed, the environment state O t Input the policy network to calculate the output Q(O t ,a;θ), where a represents the joint action taken by the user, θ represents the weight of the policy network, and the action corresponding to the maximum Q value is selected. The base station feedback helps each user select the optimal communication frequency band for anti-interference, without the need to continue iteratively updating the network parameters.

[0115] The specific algorithm implementation process is as follows:

[0116]

[0117] Example 1

[0118] The embodiment of the present invention is described in detail as follows: The system simulation uses the Python Pytorch framework. The system model includes two users, a base station, and a jammer. The two users and the jammer fight each other in the frequency band B = 20 MHz. The base station performs full-band sensing at Δf = 100 KHz every 1 ms and retains the spectrum data within 200 ms. Therefore, the matrix O t The size is 200 × 200. The bandwidth of the user signal is 4 MHz, and the center frequency changes every 10 ms in steps of 4 MHz. The bandwidth of the interference signal is also 4 MHz. Both the user signal and the interference signal are raised cosine waveforms with a roll-off factor σ = 0.5. The signal power of user 1 and user 2 is 0 dBm, and the signal power of the jammer is 30 dBm. The demodulation threshold β at all frequencies th Set to 10dB, and the channel gain is set to G n =G j =1, and the frequency hopping cost of each user is set to c=0.2.

[0119] In this embodiment, five interference modes are selected, as follows:

[0120] (1) Sweep frequency interference:

[0121] The center frequency of the interference signal is determined by the sweep rate v and time t:

[0122]

[0123] Wherein, “%” is the remainder operator; the frequency sweep rate v is 0.6 GHz / s.

[0124] (2) Comb interference: The center frequencies of the interference signals are fixed at 6 MHz and 14 MHz.

[0125] (3) Dynamic interference: Randomly select comb interference or sweep interference mode and keep it unchanged for a certain period of time (100ms).

[0126] (4) Intelligent interference: By counting the user's action probabilities in the past 100ms, the first two action probabilities with the highest probability are selected as the center frequencies of the comb interference.

[0127] (5) Random interference: Randomly select two frequencies from 2MHz, 6MHz, 10MHz, 14MHz, and 18MHz as the center frequencies of the comb interference and keep them unchanged for a certain period of time (50ms).

[0128] Figure 4(a) 、 4(b) 4(c), 4(d), and 4(e) are spectrum waterfall diagrams of five interference modes according to the embodiment of the present invention, where the horizontal axis represents the frequency (unit is 10 5 Hz), the vertical axis represents time (in ms), and the depth of the color in the figure represents the power (in dBm). In the spectrum waterfall, the light-colored squares represent the communication signals transmitted by the user, the dark diagonal lines and squares represent the interference signals emitted by the jammer, and the black background represents noise.

[0129] Figure 5(a) 、 5(b) 5(c), 5(d), and 5(e) are the frequency spectra of the anti-interference model under the five interference modes in the embodiment of the present invention. It can be seen that after iterative training, the intelligent agent can learn the interference strategy of the jammer and help the user avoid the interference signal of the jammer. At the same time, it takes into account the mutual interference caused by competition between users, thereby achieving effective anti-interference.

[0130] Figure 6This is a comparison chart of the normalized throughput of each user in the dynamic interference mode in an embodiment of the present invention. As can be seen from the figure, as the number of iterations increases, the normalized throughput of each user (the probability of successful communication per unit time) gradually increases and then tends to converge. Compared with the deep reinforcement learning algorithm with random frequency hopping and a fixed ε value, the normalized throughput of each user after convergence of the present invention is better than the other two algorithms. After convergence after 4000 iterations, the normalized throughput can reach above 0.94, which once again proves that the algorithm proposed in the present invention has a good anti-interference effect.

[0131] In summary, the multi-user communication anti-interference intelligent decision-making method proposed in the present invention adopts a dynamic ε-greedy strategy, improves the learning rate, and can effectively deal with external malicious interference and mutual interference caused by competition between users. It is no longer limited to randomly selecting communication frequency bands through frequency hopping technology, but helps users automatically select the optimal communication frequency band according to the current spectrum status, that is, the frequency band with the least possibility of interference.

[0132] The above description is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a technician familiar with this technical field within the technical scope disclosed by the present invention should be covered by the scope of protection of the present invention.

Claims

1. A multi-user communication anti-interference intelligent decision-making method based on deep reinforcement learning. First, a wireless communication anti-interference system model for multiple users is constructed. Then, feedback from the base station is used to help each user intelligently select the optimal communication frequency band. The method is characterized by: The steps are as follows: S1, builds a multi-user wireless communication anti-interference system model consisting of multiple users, a base station and a jammer. Users, base stations and jammers are randomly distributed in an open area and share a spectrum space; S2, the base station obtains the current spectrum information of multiple users and jammers; S3 builds two convolutional neural network models, uses the current spectrum information as input, selects joint actions based on a dynamic greedy algorithm, and uses base station feedback to help users intelligently select communication frequency bands. S4, calculate the immediate reward generated by the joint action in the current time slot and store the experience in the experience replay pool; the experience includes the current spectrum selection state, joint action, immediate reward and next spectrum selection information; S5, when the number of experiences in the experience replay pool reaches a given value, a certain number of experiences are randomly drawn from the experience pool to update the parameters of the policy neural network, and the parameters of the target neural network are updated once every fixed time interval; the iteration stops when the set number of iterations is reached; In step S2, the base station receives the signal to interference and noise ratio SINR of user u. u Determine whether user u has successfully communicated. If the communication is successful, the normalized threshold r u (f) is 1, otherwise it is 0; SINR of user u received by the base station u for: Among them, G u represents the channel gain from user u to the base station, G j represents the channel gain from the jammer to the base station, U j (f) represents the power spectrum density of the jammer, f represents the signal frequency, f k Indicates the center frequency of channel k selected by user u, f l represents the interference frequency of a user-selected channel l, n(f) represents the power spectrum density of the noise, represents the co-channel interference from other users in the user set when user u selects channel k; δ() is an indicator function, which is 1 if · is true and 0 otherwise; Represents a user set; Defining β th is the threshold of signal-to-noise ratio transmission. When the signal-to-interference-and-noise ratio SINR of user u is received, u Greater than β th When the signal to interference and noise ratio SINR of user u is received, it indicates that the transmission is successful; u Less than or equal to β th When the transmission fails, the normalized threshold g u (f) is: In step S3, the two convolutional neural networks are a strategy neural network with a weight parameter of θ and a policy neural network with a weight parameter of θ. - The target neural network is randomly initialized with weight parameters; the two-dimensional spectrum waterfall O t As the input of the neural network, it passes through four convolutional layers, and then the output of the convolution is flattened into one-dimensional data through the expansion layer, and then passes through four fully connected layers to obtain the final output value; The dynamic ε-greedy algorithm is used to select the joint action a(t) as follows: The probability of randomly selecting action a(t) in each iteration is ε, and the policy network Q is selected policy The maximum action a′=argmax a Q policy (O t ,a;θ i ) has a probability of 1-ε, where ε0 is the initial greedy probability, decay is the decay coefficient, i is the number of iterations, ε decreases exponentially as the number of iterations i increases, and e is a natural constant.

2. The multi-user communication anti-interference intelligent decision-making method based on deep reinforcement learning according to claim 1 is characterized in that The spectrum waterfall O t The solution process is as follows: The discrete spectrum sampling values ​​are defined as follows: Where S(f) represents the user power spectrum density received by the base station, Δf is the resolution of spectrum analysis, and i is a positive integer representing the number of samples. The spectrum status results observed by the base station each time are: oh t =[o' 1,t ,o' 2,t ...oh' L,t ] The spectrum waterfall O t Defined as: The t =[o' t ,o' r+1 ...oh' r+W-1 ] Among them, W represents the number of historical states to be traced back, O t It is a two-dimensional matrix of size W×L, containing information in the frequency domain and time domain.

3. The multi-user communication anti-interference intelligent decision-making method based on deep reinforcement learning according to claim 1 is characterized in that In step S4, the process of calculating the immediate reward generated by the action in the current time slot is as follows: The action space is represented as: <h2 style=";text-align:left;direction:ltr">A={a1,a2,…,a<h2 style=";text-align:left;direction:ltr"> n×m <h2 style=";text-align:left;direction:ltr">} Where n is the number of users, m is the number of channels, and a q represents the joint action taken by each user at time t, q = 1, 2, ..., n × m, and there are a total of n × m joint actions a(t); Then the joint action a(t) at time t is: a(t)=[f1(t),f1(t),…,f n (t)] Among them, f n (t) represents the center frequency of the channel selected by user n at time t; The state transition probability refers to the probability that the user set is in state O. t After executing the joint action a(t), the state is transferred to O. t+1 The probability of is expressed as: DUST t ,a)→The t+1 For the immediate reward r(t), it is defined as: Where c is the frequency hopping cost.

4. The multi-user communication anti-interference intelligent decision-making method based on deep reinforcement learning according to claim 1 is characterized in that In step S5, after the number of elements in the experience replay pool D is greater than one batch, Num samples e are randomly selected from the experience replay pool D. batch ={e k ,e k ~U(D),k=1,2,...,Num}, the parameters θ of the policy network are adjusted by the gradient descent algorithm i Iterative update; for the parameters of the target network The parameters of the policy network are copied regularly to achieve parameter updates; After the training is completed, the environment state O t Input strategy network calculation, get output Q(O t ,a;θ), where a represents the joint action taken by the user, θ represents the weight of the strategy network, and the action corresponding to the maximum Q value is selected. The feedback from the base station helps each user select the optimal communication frequency band to resist interference.

Images