Multi-user cooperative anti-interference strategy based on multi-agent discrete flexible actor-commentator algorithm

Abstract

The invention discloses a multi-user cooperative anti-interference strategy based on a multi-agent discrete flexible actor-commentator algorithm (MA-DSAC), and belongs to the technical field of wireless communication. The invention provides a multi-user cooperative anti-interference strategy based on a multi-agent discrete flexible actor-commentator algorithm in combination with a frequency domain, a power domain and a modulation coding domain in order to simultaneously solve external malicious interference and same-frequency interference between internal communication receiving and transmitting pairs in a multi-user scene. In the strategy, firstly, a Markov game is introduced to carry out modeling and analysis on a multi-user anti-interference problem; secondly, discretizing a flexible actor-commentator algorithm to cope with a multi-dimensional discrete action space; and finally, through information interaction between communication receiving and transmitting pairs, based on a framework of centralized training and distributed execution, the algorithm is expanded into a multi-agent deep reinforcement learning algorithm, so that efficient multi-user cooperative anti-interference is realized.

Description

Technical Field

[0001] The present invention relates to the field of wireless communication technology, and mainly to a multi-user collaborative anti-interference strategy based on a multi-agent discrete flexible actor-critic algorithm. Background Art

[0002] With the application of software defined radio (SDR) in electronic warfare, wireless communications are more vulnerable to malicious interference attacks from non-cooperative parties; compared with ordinary interference, malicious interference is more targeted, adaptable and destructive. In the early stages, the working mode of malicious interference was relatively fixed, and the communication party could easily dig out the changing rules of interference and then formulate corresponding anti-interference strategies. In recent years, with the development of artificial intelligence (AI) and its application in electronic warfare, intelligent jammers equipped with AI technology have the ability to perceive complex electromagnetic environments, learn the behavior of communication parties and autonomously adjust interference strategies, which has brought unprecedented challenges to the development of wireless communication anti-interference technology. Therefore, there is an urgent need to improve the ability and level of intelligent anti-interference of communication parties.

[0003] Traditional anti-interference technologies, such as frequency hopping, spread spectrum, power control, and rate adaptation, are widely used to combat malicious interference. However, these technologies usually have to sacrifice frequency band utilization in order to improve transmission reliability. At the same time, with the development of AI and SDR technologies, interference is showing development trends such as diversity, dynamism, and intelligence, while traditional anti-interference technologies rely on fixed or preset working modes, which are usually only effective for specific confrontation scenarios and lack the ability to make intelligent decisions. Therefore, it is particularly important to study intelligent anti-interference methods to combat various types of interference attacks, especially in dynamic and intelligent interference scenarios that will be prevalent in the future.

[0004] In recent years, reinforcement learning, as a mainstream machine learning paradigm, does not require pre-labeled training samples. Instead, it performs different actions and continuously adjusts its actions in the process of interacting with the environment to seek more favorable strategies. It has the characteristics of autonomous exploration and optimization, and is therefore widely used in communication anti-interference decision-making. However, traditional reinforcement learning algorithms, such as Q-learning, are only applicable to low-dimensional discrete action spaces and require the extraction of limited state spaces from the original spectrum. Otherwise, when the state or action space is large, it will face the problem of dimensionality disaster.

[0005] At present, some inventions have used multi-agent reinforcement learning to solve the multi-user anti-interference problem. After searching, the application publication number CN115454141A is a multi-agent multi-domain anti-interference method for drone clusters based on partial observable information. It uses partial observed environmental information of each agent, retains historical experience data through a long short-term memory network, inputs the deep recurrent q network of each agent to fit the action value function, and uses the ε-greedy algorithm to select the channel and power corresponding to the maximum output q value. After continuously and independently training the deep recurrent q network of each agent, the q value distribution is updated, and finally the optimal decision of the drone channel and transmission power that can adapt to the unknown interference scenario to minimize the communication transmission energy consumption is learned. However, the deep reinforcement learning algorithm it uses is a typical reinforcement learning method based on value function approximation. When the action dimension is high and the action space is large, it will face many problems, such as sensitivity to hyperparameters, insufficient exploration and training difficulties. Application Publication No. CN116634575A, a multi-user anti-interference dynamic spectrum access control method based on multi-agent reinforcement learning, uses a policy gradient-based reinforcement learning method, does not estimate the value function, but directly estimates the strategy of the agent, and the network output is the probability distribution of the action, so it is very suitable for problems with large action space dimensions. However, its actions are only defined as the selected access channel, and the technical means of anti-interference are relatively simple. In order to improve system performance, it is necessary to integrate multi-domain technologies to achieve multi-user intelligent collaborative anti-interference. Summary of the invention

[0006] The present invention aims to solve the above problems of the prior art, and proposes a multi-user collaborative anti-interference strategy based on the Multi-Agent Discrete Soft Actor-Critic (MA-DSAC) algorithm by combining the frequency domain, power domain and modulation coding domain. The soft actor-critic (SAC) algorithm is extended to a multi-agent deep reinforcement learning algorithm by using a centralized training and distributed execution framework. The technical solution of the present invention is as follows:

[0007] Step 1. Enter strategy parameters;

[0008] Step 2: Randomly initialize the actor network parameters, critic network parameters, critic target network parameters and temperature parameters of all communication agents.

[0009] Step 3: Each communication transceiver pair executes the action decided in the previous time slot, senses the spectrum environment while communicating, and obtains their own private state;

[0010] Step 4: Each communication receiving and sending pair calculates the achievable rate of the communication process and obtains the corresponding environmental reward value;

[0011] Step 5, each communication transceiver pair broadcasts the action and private status performed in the current time slot to other communication transceiver pairs;

[0012] Step 6: Each communication transceiver pair receives the private state and action broadcasted by other communication transceiver pairs to form a global state;

[0013] Step 7, each communication transceiver pair stores the global state of the previous time slot, the action of the previous time slot, the reward obtained in the previous time slot, and the global state of the current time slot as experience samples in their respective experience replay pools;

[0014] Step 8: If the number of experience samples stored in the experience replay pool of each communication transceiver pair is sufficient, a batch of historical experience samples are randomly sampled from it and the network parameters are updated using the gradient descent method;

[0015] Step 9: The agent of each communication pair decides the action;

[0016] Step 10: Repeat steps 3 to 9 until the maximum number of iterations, i.e., the number of time slots T, is reached.

[0017] Furthermore, the strategy parameters in step 1 include the number of time slots T, the discount factor γ, and the learning rate μ Q , μ π , μ α (where the subscripts Q, π, and α represent the Critic network, the Actor network, and the temperature parameter, respectively), the soft update coefficient κ, the capacity of the experience replay pool D, and the number of batch sampling samples Switching overhead factor k c , k p , k l (wherein the subscripts c, p, and l represent channel, power, and transmission rate, respectively).

[0018] Furthermore, in step 2, the Actor network parameters, Critic network parameters, Critic target network parameters and temperature parameters of all communication agents are randomly initialized, as follows:

[0019] The action space of the communication transceiver pair covers the frequency domain, power domain and modulation coding domain, resulting in a huge action space. In order to enhance the exploratory nature of the anti-interference network model, a random strategy needs to be determined for the agent of the communication transceiver pair n, which maximizes the expected cumulative reward while maximizing the strategy entropy, that is,

[0020]

[0021] Where π is a random strategy, π * is the optimal strategy, argmax represents the strategy with the largest mean, o n，trepresents the private state, the subscript n represents the nth communication transceiver pair, and t represents the tth time slot; π(·|o n，t ) indicates that it is in private state o n，t The probability distribution of taking any action under , the corresponding entropy is expressed as represents the mean value; α n is the temperature parameter of the nth communication transceiver pair, which determines the relative importance of the entropy term and the reward term.

[0022] Then, the soft action value function is defined according to the Bellman equation:

[0023]

[0024] where s t represents the global state, r n，t represents the reward for communication pair n at time slot t, a n，t represents the action of the communication transceiver pair n at time slot t. According to the above formula, the current strategy π can be determined old Soft Q value The subscript old represents the current one, and then the new strategy π is determined by solving the following optimization problem new , the subscript new indicates new:

[0025]

[0026] Among them, D KL represents the Kullback-Leible divergence, and the subscript KL is the abbreviation of Kullback-Leible. is the normalization factor, and exp represents the exponential function. The solution to the above equation is to find the t Action distribution and The strategy that minimizes the KL divergence of the distribution.

[0027] Since there are multiple agents in the multi-user collaborative anti-interference environment, a parameterized action value function is considered and strategies where θ n，i and φ n Represents the deep neural network parameters of the agent deployed at the receiver of the communication transceiver pair n, namely the parameters of the Critic network and the Actor network. In order to alleviate the overfitting problem during training, truncated double Q learning is adopted, so each agent will learn two Critic networks, whose parameters are θ n，1 and θ n，2 , that is, i∈{1,2}. In addition, in order to ensure the stability of training, the parameters will be maintained by soft copy (Son Copy) as and Two target Critic networks.

[0028] Furthermore, the action space of the communication transceiver pair covers the frequency domain, power domain and modulation coding domain, as follows:

[0029] The communication user set is composed of N pairs of communication transceivers, denoted as They share the same frequency band [f Lower , f Upper ], f Lower and f Upper The lowest frequency and the highest frequency are respectively, and the frequency band is divided into equal intervals A channel with bandwidth B; let The center frequency of each channel is Corresponding to The frequency band occupied by the channels is in Considering the joint frequency domain, power domain and modulation coding domain to realize the multi-domain anti-interference strategy design, the available channel index set, power level set and transmission rate set for communication transceiver are defined as and The subscripts P and L represent the power level and transmission rate level, respectively. The element 0 in indicates that the communication transceiver pair chooses to sleep. In order to facilitate the modeling of multi-user cooperative anti-interference, the entire transmission time is divided into T time slots. Assume that in the tth time slot, the communication transceiver pair The selected channel, power level and transmission rate are respectively and Indicates that here l n，t By the modulation order m n，t and coding efficiency η n，t Based on this, the joint decision of the communication transceiver pair n in time slot t is further defined as [c n，t , p n，t , m n，t , η n，t ].

[0030] Furthermore, the private state, global state, action, and reward are as follows:

[0031] First, considering that each time slot is very short during the confrontation between the communicating party and the interfering party, the block fading channel model is considered, that is, the channel state information remains unchanged in each time slot. Therefore, the channel link gain between the receiver and the transmitter of the communication transceiver pair n is:

[0032]

[0033] Among them, dn is the distance between the receiver and the transmitter, the subscript n represents the nth communication pair, α u is the path fading factor, and the subscript u represents the first letter of the English word user. It reflects the large-scale fading effect, including path loss and shadow fading; ν(t) represents the small-scale fading coefficient, which is a complex Gaussian variable with zero mean and unit variance.

[0034] Therefore, at time slot t, for the communication transceiver pair n, its received signal-to-interference-to-noise ratio is:

[0035]

[0036] where p n，t Indicates the transmission power of the transmitter of the communication pair n in time slot t (Note: c n，t =0, the above formula is 0); It represents the external interference power borne by the receiver of the communication transceiver pair n. The external interference comes from various malicious jammers and environmental noise, which can be expressed as:

[0037]

[0038] in is the channel where the jammer is located at time slot t, the subscript j represents the jammer, n(f) and J(·) are the power spectral density functions of the complex Gaussian background white noise and the jammer, respectively, δ(·) is the indicator function, when c n，t =c j，t , that is, when the communication transceiver pair n and the jammer are in the same channel, it is 1, otherwise it is 0; g n，j (t) is the channel gain between the jammer and the receiver of the communication transceiver pair n:

[0039]

[0040] Among them, d n，j is the distance between the receiver and the jammer of the communication transceiver pair n, α j is the path fading factor, ε(t) represents the small-scale fading coefficient, which is a complex Gaussian variable with zero mean and unit variance.

[0041] Internal interference is the co-frequency interference between the communication transceiver pairs, expressed as:

[0042]

[0043] in is the transmission power of the transmitter of the communication pair m in time slot t. m，n (t) is the channel gain between the receiver of communication transceiver pair n and the transmitter of communication transceiver pair m:

[0044]

[0045] where d m，n represents the distance between the receiver of communication transceiver pair n and the transmitter of communication transceiver pair m (m≠n), Represents the small-scale fading coefficient, which is a complex Gaussian variable with zero mean and unit variance.

[0046] The communication achievable rate is defined by the received signal-to-interference-to-noise ratio:

[0047]

[0048] where λ n，t is the demodulation threshold of the communication transceiver pair n in time slot t, and its size is related to the transmission rate. Different transmission rates correspond to different signal-to-interference-noise ratio demodulation thresholds; The corresponding signal-to-interference-noise ratio demodulation threshold set can be defined as but When SINR n，t (·)≥λ n，t When , it indicates that the communication between the sender and the receiver is successful, otherwise the communication fails.

[0049] The multi-user cooperative anti-interference problem is modeled as a Markov game, as follows:

[0050] 1) Action: The action of the communication sender and receiver in time slot t on n is defined as the joint decision of time slot t+1, that is, a n，t =[c n，t+1 , p n，t+1 , m n，t+1 , η n，t+1 ], where c n，t+1 , p n，t+1 , m n，t+1 and η n，t+1 They represent the channel index, power, modulation order and coding efficiency that the communication transceiver pair n will select in time slot t+1 respectively; the transmission rate is determined by the modulation order and coding efficiency. The modulation order set is defined as The coding efficiency set is L1 and L2 represent the number of modulation orders and the number of coding efficiencies, respectively. The transmission rate of the communication transceiver pair n in time slot t is l n，t =η n，t log2m n，t , where m n，t ∈Mod,η n，t ∈Cod. Let the action space A consist of all candidate actions of the communication sending and receiving pairs n. n , then a n，t ∈A n , whose size is C×P×L1×L2; a t =[a 1，t, a 2，t ..., a n，t , ..., a N，t ] represents the action set of all communicating transceiver pairs in time slot t.

[0051] 2) State: The state is divided into global state and private state. Define the private state o of the communication pair n in time slot t n，t =[SINR n，t (F1), SINR n，t (F2), ..., SINR n，t (F C ), l n，t ], where the first C elements correspond to the SINR of different channels, l n，t is the transmission rate of the current time slot communication pair n. The private state space O is composed of all private states. n , that is o n，t ∈O n ;Define o t =[o 1，t , o 2，t， ..., o n，t , ..., o N，t ] represents the private state of all communication pairs in time slot t. Correspondingly, the global state s that encompasses all user private states and actions is defined t =[o t , a t-1 ]; All global states form the global state space S, where s t ∈S.

[0052] 3) Reward: In state s t The following communication sends and receives action a n，t After that, the environment will give the agent n the corresponding reward r n，t , effective rewards help the agent find the optimal communication strategy. Since anti-interference communication requires frequent channel switching, power adjustment and transmission rate adjustment, it will bring certain overhead; for this reason, the channel, power and transmission rate switching weight factors are introduced, and they are expressed as k c , k p and k l Therefore, according to the expression of the achievable communication rate, the utility function of the communication transmission and reception of time slot t to n is defined as:

[0053] u n，t (c n，t , c n，t-1 , p n，t , p n，t-1 , l n，t , l n，t-1 )=V n，t (·)-k cδ c (c n，t , c n，t-1 )-k p δ p (p n，t , p n，t-1 )-k l δ l (l n，t , l n，t-1 ) where δ(·) is the indicator function. c (c n，t , c n，t-1 )=δ p (p n，t , p n，t-1 )=δ l (l n，t , l n，t-1 )=0, it means that the communication transceiver pair n selects the same action in the t-1 time slot and the t-th time slot; on the contrary, when the communication transceiver pair n selects different actions in the t-1 time slot and the t-th time slot, δ c (c n，t , c n，t-1 )=δ p (p n，t , p n，t-1 )=δ l (l n，t , l n，t-1 )=1.

[0054] Furthermore, the reward for communication transmission and reception pair n in time slot t is defined as:

[0055]

[0056] Where C1 and C2 are constants, u n，t+1 (·) represents the communication transceiver pair n giving a joint strategy action a in time slot t n，t After that, the utility function value obtained in the next time slot t+1.

[0057] Furthermore, the structure of the deep neural network is as follows:

[0058] Both the Actor and Critic networks are composed of fully connected neural networks. Each communication pair of senders and receivers has an Actor network, two Critic networks, and two Critic target networks deployed on the agent. The input of the Actor network is the private state o of the communication pair n. n，t , the input parameter is C+1, after being processed by the neural network, the probability value of all discrete actions is output, and the output parameter is (C+1)×P×L1×L2. The input of the Critic network is the global state s t , the input parameter is (C+1)·N+diman，t N, dim represents the dimension of user n’s actions. After being processed by FCNN, the estimated value of all discrete actions is output, and the output parameter volume is (C+1)×P×L1×L2.

[0059] Furthermore, the step 10 is specifically as follows:

[0060] The entire transmission time is divided into T time slots. Communication is carried out in time slots as the basic unit. Each time slot includes five sub-time slots: transmission, spectrum sensing, information exchange, learning and decision-making, and feedback. After each time slot begins, each communication transceiver transmits information according to the joint decision of the current time slot. The duration of the sub-time slot is τ c At the same time, in this sub-time slot, the communication transceiver pair performs broadband spectrum sensing, corresponding to step 3. After completing spectrum sensing, each communication transceiver pair can obtain an environmental reward value and e The sub-time slot of τ completes the information exchange, including broadcasting its current private state and action, and receiving broadcast information from other communication transceiver pairs, corresponding to steps 4 to 6. Then, in the sub-time slot of length τ l In the sub-time slot of , each communication transceiver pair will use the centralized training and distributed execution framework, combine all private states and actions, and then apply the deep reinforcement learning algorithm to train the agent and make a joint decision for the next time slot, corresponding to steps 7 to 9. Finally, in the time slot of τ d In the feedback sub-time slot, the receiver of each communication transceiver pair transmits the joint decision of the next time slot to the corresponding transmitter.

[0061] Furthermore, the centralized training and distributed execution framework is as follows:

[0062] During training, the Critic network incorporates the private states and actions of other communication transceiver pairs as additional information; during policy execution, only the Actor network needs to make decisions based on its own private state. The Actor network can quickly output an action that enables the agent to obtain the maximum expected benefit based on the private state of the agent, and the Critic network is used to evaluate the actions output by the agent Actor network during the model training phase, and to assist in the training of the parameters of the agent Actor network. Specifically, the Actor network deployed on the agent of the communication transceiver pair n uses the private state o n，t As input, the output is the probability value of all discrete actions, and then randomly sample action a n，t When the actions of the agents deployed on the receivers of all communication pairs interact with the environment, the environment will feedback to each agent the immediate reward r n，tAt the same time, the communication transceiver pair n will broadcast its own private state and action to other communication transceiver pairs through the information exchange link, and receive the private state and action broadcast by other users. Finally, the experience sample ξ is formed in the form of a tuple t , and stored in the experience replay pool according to the first-in-first-out principle When enough experience is accumulated in the experience replay pool, the communication sender and receiver pair n will extract the experience containing Batches of samples are used to update network parameters.

[0063] Furthermore, the network parameters are updated as follows:

[0064] The parameter θ n，i , that is, the Critic network parameters, can be determined by minimizing the soft Bellman residual:

[0065]

[0066] Where L Q represents the loss value of the Critic network, the subscript Q represents the Critic network, (s t , a t , r n，t ,s t+1 ) is the experience playback pool of the communication transceiver n The experience sample in y n，t is the time difference target value, which represents the global state s t+1 Next, implement action a n，t The soft Q-value estimation of

[0067]

[0068] in The parameter φ n , that is, the Actor network parameter, log means finding the logarithm, and min means finding the minimum value. It can be determined by minimizing the following loss function using the soft strategy improvement method:

[0069]

[0070] In the above formula, the temperature parameter α n Determines the importance of entropy to the algorithm. n If α is too small, the algorithm may degenerate into a common Actor-Critic algorithm. n This will cause the algorithm to focus too much on maximizing entropy and ignore the impact of environmental rewards. To this end, it is hoped that when facing an environment that has been explored, α will be reduced. n To reduce entropy; when facing an unknown environment, increase α nTo enhance the randomness of strategy exploration. To this end, a target entropy And update α by minimizing the following loss function n :

[0071]

[0072] For the parameters of the target network and Update in soft copy mode, i.e.

[0073]

[0074] The Actor network, Critic network and temperature parameters are updated using the stochastic gradient descent method, namely

[0075]

[0076] where μ Q , μ π and μ α are the learning rates of the Critic network, Actor network and temperature parameters respectively, It means to find the gradient.

[0077] The advantages and beneficial effects of the present invention are as follows:

[0078] The present invention provides a multi-user collaborative anti-interference strategy based on a multi-agent discrete flexible actor-critic algorithm, with the goal of maximizing the system and rate, and jointly realizing effective anti-interference in the frequency domain, power domain and modulation coding domain. First, the present invention models multi-user collaborative anti-interference as a Markov game model; then, the flexible actor-critic algorithm is used to introduce an entropy regularization term to enhance the exploratory nature of the model, and at the same time, the flexible actor-critic algorithm is discretized to cope with multi-dimensional discrete action space; then, through the information interaction between each communication transceiver pair, the flexible actor-critic algorithm is expanded into a multi-agent deep reinforcement learning algorithm with the help of centralized training and distributed execution framework to solve the problem of multi-user collaborative anti-interference. The present invention does not require any prior information about the communication system or interference. After the training is completed, each communication transceiver pair can independently realize anti-interference based on its own perception of the electromagnetic environment; compared with a single action dimension, the multi-domain action space of the present invention provides richer anti-interference means, which is more conducive to solving the anti-interference problem in a complex electromagnetic environment; in addition, compared with the traditional reinforcement learning algorithm, the present invention is more suitable for anti-interference scenarios with larger state and action spaces. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Figure 1 This is a multi-user collaborative anti-interference scene diagram provided by the present invention

[0080] Figure 2This is a multi-user collaborative anti-interference strategy framework diagram based on the multi-agent discrete flexible actor-critic algorithm provided by the present invention

[0081] Figure 3 This is the Actor network structure diagram provided by the present invention

[0082] Figure 4 This is the Critic network structure diagram provided by the present invention

[0083] Figure 5 This is the training framework diagram of the multi-agent discrete flexible actor-critic algorithm provided by the present invention

[0084] Figure 6 The communication time slot structure diagram provided by the present invention is DETAILED DESCRIPTION

[0085] The following will describe the technical solutions in the embodiments of the present invention in detail in conjunction with the accompanying drawings in the embodiments of the present invention. The described embodiments are only part of the embodiments of the present invention.

[0086] like Figure 1 As shown, in the multi-user cooperative anti-interference system model of the embodiment of the present invention, there is a communication user set consisting of N pairs of communication transceivers, denoted as The communication user receiver is deployed with an intelligent agent with training and decision-making capabilities. The communication transceiver transmits data through the communication link and transmits feedback information through the control link. The communication transceiver uses the information exchange link to exchange training data. It should be pointed out that since the control link and the information exchange link have less information to be transmitted, the spread spectrum communication technology with high spread spectrum gain is usually used to transmit information. Therefore, these two links can be regarded as completely reliable links. All users share the same frequency band [f Lower , f Upper ], f Lower and f Upper The lowest frequency and the highest frequency are respectively, and the frequency band is divided into equal intervals A channel with bandwidth B; let The center frequency of each channel is Corresponding to The frequency band occupied by the channels is in Considering the joint frequency domain, power domain and modulation coding domain to realize the multi-domain anti-interference strategy design, the available channel index set, power level set and transmission rate set for communication transceiver are defined as and The subscripts P and L represent the power level and transmission rate level, respectively. The element 0 in indicates that the communication transceiver pair chooses to sleep. In order to facilitate the modeling of multi-user cooperative anti-interference, the entire transmission time is divided into T time slots. Assume that in the tth time slot, the communication transceiver pair The selected channel, power level and transmission rate are respectively and Indicates that here l n，t By the modulation order m n，t and coding efficiency η n，t Based on this, the joint decision of the communication transceiver pair n in time slot t is further defined as [c n，t , p n，t , m n，t , η n，t ].

[0087] The communication transceiver will face external interference and internal interference during the communication process. External interference comes from various malicious jammers and environmental noise, while internal interference refers to the same-frequency interference between the communication transceivers. Figure 1 Model, in view of the fact that each time slot is very short during the confrontation between the communicating party and the interfering party, the block fading channel model is considered, that is, the channel state information remains unchanged in each time slot. Therefore, the channel gain between the jammer and the receiver of the communication transceiver pair n is:

[0088]

[0089] Among them, d n，j is the distance between the receiver and the jammer of the communication transceiver pair n, α j is the path fading factor, It reflects the large-scale fading effect, including path loss and shadow fading; ε(t) represents the small-scale fading coefficient, which is a complex Gaussian variable with zero mean and variance η.

[0090] Let c j，t ∈C is the channel where the jammer is located at time slot t, n(f) and J(·) are the power spectral density functions of the complex Gaussian background white noise and the jammer respectively, then the external interference power borne by the communication transceiver to the receiver n can be expressed as:

[0091]

[0092] Where δ(·) is the indicator function, when c n，t =c j，t , that is, it is 1 when the communication transceiver pair n is in the same channel as the jammer, otherwise it is 0.

[0093] The channel gain between the receiver of communication transceiver pair n and the transmitter of communication transceiver pair m is modeled as:

[0094]

[0095] where d m，n represents the distance between the receiver of communication transceiver pair n and the transmitter of communication transceiver pair m (m≠n), α u is the path fading factor; represents the small-scale fading coefficient, which is a complex Gaussian variable with zero mean and unit variance. Therefore, the internal interference borne by the receiver of communication transceiver pair n is the sum of the interferences exerted on it by the transmitters of other communication transceiver pairs, that is,

[0096]

[0097] in is the transmission power of the transmitter of communication transceiver pair m in time slot t.

[0098] Furthermore, the channel link gain between the communication transceiver pair n receiver and transmitter is:

[0099]

[0100] Among them, d n is the distance between the receiver and the transmitter, the subscript n represents the nth communication pair, α u is the path fading factor, the subscript u represents the first letter of user, ν(t) represents the small-scale fading coefficient, and is a complex Gaussian variable with zero mean and unit variance.

[0101] Furthermore, at time slot t, for the communication transceiver pair n, its received signal-to-interference-noise ratio is:

[0102]

[0103] where p n，t represents the transmission power of the communication transceiver to the transmitter n in time slot t (Note: c n，t =0, the above formula is 0). Based on this, the communication achievable rate is defined as:

[0104]

[0105] where λ n，t is the demodulation threshold of user n in time slot t, which is related to the transmission rate. Different transmission rates correspond to different SINR demodulation thresholds. The corresponding SINR demodulation threshold set can be defined as but When SINR n，t (·)≥λ n，t When , it indicates that the communication transceiver successfully communicates with n, otherwise the communication fails.

[0106] Based on the above system model, the multi-user collaborative anti-interference strategy of the present invention is based on the multi-agent discrete flexible actor-critic algorithm. The main steps are as follows:

[0107] Step 1. Enter strategy parameters;

[0108] The strategy parameters include the number of time slots T, the discount factor γ, and the learning rate μ Q , μ π , μ α , soft update coefficient κ, experience replay pool capacity D, batch sampling sample number Switching overhead factor k c , k p , k l .

[0109] Step 2: Randomly initialize the Actor network parameters, Critic network parameters, Critic target network parameters, and temperature parameters of all communication agents.

[0110] In order to enhance the exploratory nature of the anti-interference network model, a random strategy needs to be determined for the communication sender and receiver pairs n, which maximizes the expected cumulative reward while maximizing the strategy entropy, that is,

[0111]

[0112] Where π is a random strategy, π * is the optimal strategy, o n，t represents the private state, the subscript n represents the nth communication transceiver pair, and t represents the tth time slot; π(·|o n，t ) indicates that it is in private state o n，t The probability distribution of taking any action under the condition is expressed as α n is the temperature parameter of the nth communication transceiver pair, which determines the relative importance of the entropy term and the reward term.

[0113] Then, the soft action value function is defined according to the Bellman equation:

[0114]

[0115] where s t represents the global state, r n，t represents the reward for communication pair n at time slot t, a n，t represents the action of the communication transceiver pair n at time slot t. According to the above formula, the current strategy π can be determined old Soft Q value Then the new strategy π is determined by solving the following optimization problem new :

[0116]

[0117] Among them, D KL represents the Kullback-Leibler divergence, is the normalization factor. The solution to the above equation is to find t Action distribution and The strategy that minimizes the KL divergence of the distribution.

[0118] Since there are multiple agents in the multi-user collaborative anti-interference environment, a parameterized action value function is considered and strategies where θ n，i and φ n Represents the deep neural network parameters of the agent deployed at the receiver of the communication transceiver pair n, namely the parameters of the Critic network and the Actor network. In order to alleviate the overfitting problem during training, truncated double Q learning is adopted, so each agent will learn two Critic networks, whose parameters are θ n，1 and θ n，2 , that is, i∈{1,2}. In addition, in order to ensure the stability of training, the parameters will be maintained by soft copy. and Two target Critic networks.

[0119] like Figure 2 As shown in the figure, during training, the Critic network incorporates the private states and actions of other communication transceiver pairs as additional information; during policy execution, only the Actor network needs to make decisions based on its own private state. The Actor network can quickly output an action that can maximize the expected benefit of the agent based on the private state of the agent, and the Critic network is used to evaluate the actions output by the agent Actor network during the model training phase, and to assist in the training of the parameters of the agent Actor network. Specifically, the Actor network deployed on the agent of the communication transceiver pair n uses the private state o n，t As input, the output is the probability value of all discrete actions, and then randomly sample action a n，t When the actions of the agents deployed on the receivers of all communication pairs interact with the environment, the environment will feedback to each agent the immediate reward r n，t At the same time, the communication transceiver pair n will broadcast its own private state and action to other communication transceiver pairs through the information exchange link, and receive the private state and action broadcast by other users. Finally, the experience sample ξ is formed in the form of a tuple t , and stored in the experience replay pool according to the first-in-first-out principle When enough experience is accumulated in the experience replay pool, the communication sender and receiver pair n will extract the experience containing Batches of samples are used to update network parameters.

[0120] Step 3: Each communication transceiver pair executes the action decided in the previous time slot, senses the spectrum environment while communicating, and obtains their own private state;

[0121] Step 4: Each communication receiving and sending pair calculates the achievable rate of the communication process and obtains the corresponding environmental reward value;

[0122] Step 5, each communication transceiver pair broadcasts the action and private status performed in the current time slot to other communication transceiver pairs;

[0123] Step 6: Each communication transceiver pair receives the private state and action broadcasted by other communication transceiver pairs to form a global state;

[0124] Step 7, each communication transceiver pair stores the global state of the previous time slot, the action of the previous time slot, the reward obtained in the previous time slot, and the global state of the current time slot as experience samples in their respective experience replay pools;

[0125] The private state, global state, actions, and rewards in the above steps are as follows:

[0126] 1) Action: The action of the communication sender and receiver in time slot t on n is defined as the joint decision of time slot t+1, that is, a n，t =[c n，t+1， p n，t+1 , m n，t+1 , η n，t+1 ], where c n，t+1 , p n，t+1 , m n，t+1 and η n，t+1 They represent the channel index, power, modulation order and coding efficiency that the communication transceiver pair n will select in time slot t+1 respectively; the transmission rate is determined by the modulation order and coding efficiency. The modulation order set is defined as The coding efficiency set is Then the transmission rate of the communication pair n in time slot t is l n，t =η n，t log2m n，t , where m n，t ∈Mod,η n，t ∈Cod. Let the action space A consist of all candidate actions of the communication sending and receiving pairs n. n , then a n，t ∈A n , whose size is C×P×L1×L2; a t =[a 1，t , a 2，t , ..., an，t , ..., a N，t ] represents the action set of all communicating transceiver pairs in time slot t.

[0127] 2) State: The state is divided into global state and private state. Define the private state o of the communication pair n in time slot t n，t =[SINR n，t (F1), SINR n，t (F2), ..., SINR n，t (F C ), l n，t ], where the first C elements correspond to the SINR of different channels, l n，t is the transmission rate of the current time slot communication pair n. The private state space O is composed of all private states. n , that is o n，t ∈O n ;Define o t =[o 1，t , o 2，t , ..., o n，t , ..., o N，t ] represents the private state of all communication pairs in time slot t. Correspondingly, the global state s that encompasses all user private states and actions is defined t =[o t , a t-1 ]; All global states form the global state space S, where s t ∈S.

[0128] 3) Reward: In state s t The following communication sends and receives action a n，t After that, the environment will give the agent n the corresponding reward r n，t , effective rewards help the agent find the optimal communication strategy. Since anti-interference communication requires frequent channel switching, power adjustment and transmission rate adjustment, it will bring certain overhead; for this reason, the channel, power and transmission rate switching weight factors are introduced, and they are expressed as k c , k p and k l Therefore, according to the expression of the achievable communication rate, the utility function of the communication transmission and reception of time slot t to n is defined as:

[0129] u n，t (c n，t , c n，t-1 , p n，t , p n，t-1 , l n，t , l n，t-1 )=V n，t (·)-k c δ c (cn，t , c n，t-1 )-k p δ p (p n，t , p n，t-1 )-k l δ l (l n，t , l n，t-1 )

[0130] Where δ(·) is the indicator function. c (c n，t , c n，t-1 )=δ p (p n，t , p n，t-1 )=δ l (l n，t , l n，t-1 )=0, it means that the communication transceiver pair n selects the same action in the t-1 time slot and the t-th time slot; on the contrary, when the communication transceiver pair n selects different actions in the t-1 time slot and the t-th time slot, δ c (c n，t , c n，t-1 )=δ p (p n，t , p n，t-1 )=δx(l n，t , l n，t-1 )=1.

[0131] Furthermore, the reward for communication transmission and reception pair n in time slot t is defined as:

[0132]

[0133] Where C1 and C2 are constants, u n，t+1 (·) represents the communication transceiver pair n giving a joint strategy action a in time slot t n，t After that, the utility function value obtained in the next time slot t+1.

[0134] Step 8: If the number of experience samples stored in the experience replay pool of each communication transceiver pair is sufficient, a batch of historical experience samples are randomly sampled from it and the network parameters are updated using the gradient descent method;

[0135] like Figure 3 and Figure 4 As shown in Figure 1, both the Actor and Critic networks are composed of fully connected neural networks. Each communication pair of senders and receivers has an Actor network, two Critic networks, and two Critic target networks deployed on the agent. The input of the Actor network is the private state o of the communication pair n. n，t, the input parameter is C+1, after being processed by the neural network, the probability value of all discrete actions is output, and the output parameter is (C+1)×P×L1×L2. The input of the Critic network is the global state s t , the input parameter is (C+1)·N+dima n，t N, after being processed by FCNN, outputs the estimated value of all discrete actions, and the output parameter volume is (C+1)×P×L1×L2.

[0136] like Figure 5 As shown, the intelligent agents deployed at the receiver of the communication transceiver pair n set up a separate experience replay pool To store historical experience And use historical experience to train the above parameters {θ n，i ,φ n}, for example ξ t =(s t , a t , r n，t ,s t+1 ). Further, The parameter θ n，i , that is, the Critic network parameters, can be determined by minimizing the soft Bellman residual:

[0137]

[0138] Among them (s t , a t , r n，t ,s t+1 ) is the experience playback pool of the communication transceiver n The experience sample in y n，t is the time difference target value, which represents the global state s t+1 Next, implement action a n，t The soft Q-value estimation of

[0139]

[0140] in The parameter φ n , i.e., the Actor network parameters, can be determined using the soft policy improvement method by minimizing the following loss function:

[0141]

[0142] In the above formula, the temperature parameter α n Determines the importance of entropy to the algorithm. n If α is too small, the algorithm may degenerate into a common Actor-Critic algorithm. nThis will cause the algorithm to focus too much on maximizing entropy and ignore the impact of environmental rewards. To this end, it is hoped that when facing an environment that has been explored, α will be reduced. n To reduce entropy; when facing an unknown environment, increase α n To enhance the randomness of strategy exploration. To this end, a target entropy And update α by minimizing the following loss function n :

[0143]

[0144] For the parameters of the target network and Update in soft copy mode, i.e.

[0145]

[0146] The Actor network, Critic network and temperature parameters are updated using the stochastic gradient descent method, namely

[0147]

[0148] where μ Q , μ π and μ α They are the learning rates of the Critic network, Actor network and temperature parameters respectively.

[0149] Step 9: The agent of each communication pair decides the action;

[0150] Actor network in private state o n，t As input, output the probability values ​​of all discrete actions, and then randomly sample action a n，t =[c n，t+1 , p n，t+1 , m n，t+1 , η n，t+1 ], that is, the joint decision for the next time slot.

[0151] Step 10: Repeat steps 3 to 9 until the maximum number of iterations, i.e., the number of time slots T, is reached.

[0152] like Figure 6 As shown in Figure 1, the entire transmission time is divided into T time slots. Communication is carried out in time slots as the basic unit. Each time slot includes five sub-time slots: transmission, spectrum sensing, information exchange, learning and decision-making, and feedback. After each time slot begins, each communication transceiver transmits information according to the joint decision of the current time slot. The duration of the sub-time slot is τ. c At the same time, in this sub-time slot, the communication transceiver pair performs broadband spectrum sensing, corresponding to step 3. After completing spectrum sensing, each communication transceiver pair can obtain an environmental reward value ande The sub-time slot of τ completes the information exchange, including broadcasting its current private state and action, and receiving broadcast information from other communication transceiver pairs, corresponding to steps 4 to 6. Then, in the sub-time slot of length τ l In the sub-time slot of , each communication transceiver pair will use the centralized training and distributed execution framework, combine all private states and actions, and then apply the deep reinforcement learning algorithm to train the agent and make a joint decision for the next time slot, corresponding to steps 7 to 9. Finally, in the time slot of τ d In the feedback sub-time slot, the receiver of each communication transceiver pair transmits the joint decision of the next time slot to the corresponding transmitter.

[0153] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0154] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0155] The above embodiments should be understood to be only used to illustrate the present invention and not to limit the protection scope of the present invention. After reading the contents of the present invention, technicians can make various changes or modifications to the present invention, and these equivalent changes and modifications also fall within the scope defined by the claims of the present invention.

Claims

1. A multi-user collaborative anti-interference strategy based on a multi-agent discrete flexible actor-critic algorithm, characterized in that: The following steps are involved: Step 1. Enter strategy parameters; Step 2: Randomly initialize the actor network parameters, critic network parameters, critic target network parameters and temperature parameters of all communication agents. Step 3: Each communication transceiver pair executes the action decided in the previous time slot, senses the spectrum environment while communicating, and obtains their own private state; Step 4: Each communication receiving and sending pair calculates the achievable rate of the communication process and obtains the corresponding environmental reward value; Step 5, each communication transceiver pair broadcasts the action and private status performed in the current time slot to other communication transceiver pairs; Step 6: Each communication transceiver pair receives the private state and action broadcasted by other communication transceiver pairs to form a global state; Step 7, each communication transceiver pair stores the global state of the previous time slot, the action of the previous time slot, the reward obtained in the previous time slot, and the global state of the current time slot as experience samples in their respective experience replay pools; Step 8: If the number of experience samples stored in the experience replay pool of each communication transceiver pair is sufficient, a batch of historical experience samples are randomly sampled from it and the network parameters are updated using the gradient descent method; Step 9: The agent of each communication pair decides the action; Step 10: Repeat steps 3 to 9 until the maximum number of iterations, i.e., the number of time slots T, is reached.

2. According to claim 1, a multi-user collaborative anti-interference strategy based on a multi-agent discrete flexible actor-critic algorithm is characterized in that: The strategy parameters in step 1 include the number of time slots T, the discount factor γ, and the learning rate μ Q , μ π , μ α (where the subscripts Q, π, and α represent the Critic network, the Actor network, and the temperature parameter, respectively), the soft update coefficient κ, the capacity of the experience replay pool D, and the number of batch sampling samples Switching overhead factor k c , k p , k l (wherein the subscripts c, p, and l represent channel, power, and transmission rate, respectively).

3. The multi-user collaborative anti-interference strategy based on the multi-agent discrete flexible actor-critic algorithm according to claim 1 is characterized in that: In step 2, the Actor network parameters, Critic network parameters, Critic target network parameters and temperature parameters of all communication agents are randomly initialized as follows: The action space of the communication transceiver pair covers the frequency domain, power domain and modulation coding domain, resulting in a huge action space. In order to enhance the exploratory nature of the anti-interference network model, a random strategy needs to be determined for the agent of the communication transceiver pair n, which maximizes the expected cumulative reward while maximizing the strategy entropy, that is, Where π is a random strategy, π * is the optimal strategy, argmax represents the strategy with the largest mean, o n，t represents the private state, the subscript n represents the nth communication transceiver pair, and t represents the tth time slot; π(·|o n，t ) indicates that it is in private state o n，t The probability distribution of taking any action under the condition is expressed as represents the mean value; α n is the temperature parameter of the nth communication transceiver pair, which determines the relative importance of the entropy term and the reward term. Then, the soft action value function is defined according to the Bellman equation: where s t represents the global state, r n，t represents the reward for communication pair n at time slot t, a n，t represents the action of the communication transceiver pair n at time slot t, and γ represents the discount factor. According to the above formula, the current strategy π can be determined old Soft Q value The subscript old represents the current one, and then the new strategy π is determined by solving the following optimization problem new , the subscript new indicates new: Among them, D KL represents the Kullback-Leible divergence, and the subscript KL is the abbreviation of Kullback-Leible. is the normalization factor, and exp represents the exponential function. The solution to the above equation is to find the t Action distribution and The strategy that minimizes the KL divergence of the distribution. Since there are multiple agents in the multi-user collaborative anti-interference environment, a parameterized action value function is considered and strategies where θ n，i and φ n Represents the deep neural network parameters of the agent deployed at the receiver of the communication transceiver pair n, namely the parameters of the Critic network and the Actor network. In order to alleviate the overfitting problem during training, truncated double Q learning is adopted, so each agent will learn two Critic networks, whose parameters are θ n，1 and θ n，2 , that is, i∈{1,2}. In addition, in order to ensure the stability of training, the parameters will be maintained by soft copy. and Two target Critic networks.

4. The multi-user collaborative anti-interference strategy based on the multi-agent discrete flexible actor-critic algorithm according to claim 3 is characterized in that: The action space of the communication transceiver pair covers the frequency domain, power domain and modulation coding domain, as follows: The communication user set is composed of N pairs of communication transceivers, denoted as They share the same frequency band [f Lower , f Upper ], f Lower and f Upper The lowest frequency and the highest frequency are respectively, and the frequency band is divided into equal intervals A channel with bandwidth B; let The center frequency of each channel is Corresponding to The frequency band occupied by the channels is in Considering the joint frequency domain, power domain and modulation coding domain to realize the multi-domain anti-interference strategy design, the available channel index set, power level set and transmission rate set for communication transceiver are defined as and The subscripts P and L represent the power level and transmission rate level, respectively. The element 0 in indicates that the communication transceiver pair chooses to sleep. In order to facilitate the modeling of multi-user cooperative anti-interference, the entire transmission time is divided into T time slots. Assume that in the tth time slot, the communication transceiver pair The selected channel, power level and transmission rate are respectively and Indicates that here l n，t By the modulation order m n，t and coding efficiency η n，t Based on this, the joint decision of the communication transceiver pair n in time slot t is further defined as [c n，t , p n，t , m n，t , η n，t ].

5. The multi-user collaborative anti-interference strategy based on the multi-agent discrete flexible actor-critic algorithm according to claim 3 is characterized in that: The private state, global state, actions, and rewards are as follows: First, considering that each time slot is very short during the confrontation between the communicating party and the interfering party, the block fading channel model is considered, that is, the channel state information remains unchanged in each time slot. Therefore, the channel link gain between the receiver and the transmitter of the communication transceiver pair n is: Among them, d n is the distance between the receiver and the transmitter, the subscript n represents the nth communication transceiver pair, α u is the path fading factor, and the subscript u represents the first letter of the English word user. It reflects the large-scale fading effect, including path loss and shadow fading; ν(t) represents the small-scale fading coefficient, which is a complex Gaussian variable with zero mean and unit variance. Therefore, at time slot t, for the communication transceiver pair n, its received signal-to-interference-and-noise ratio (SINR) is: where p n，t Indicates the transmission power of the transmitter of the communication pair n in time slot t (Note: c n，t =0, the above formula is 0); It represents the external interference power borne by the receiver of the communication transceiver pair n. The external interference comes from various malicious jammers and environmental noise, which can be expressed as: in is the channel where the jammer is located at time slot t, the subscript j represents the jammer, n(f) and J(·) are the power spectral density functions of the complex Gaussian background white noise and the jammer, respectively, δ(·) is the indicator function, when c n，t =c j，t , that is, when the communication transceiver pair n and the jammer are in the same channel, it is 1, otherwise it is 0; g n，j (t) is the channel gain between the jammer and the receiver of the communication transceiver pair n: Among them, d n，j is the distance between the receiver and the jammer of the communication transceiver pair n, α j is the path fading factor, ε(t) represents the small-scale fading coefficient, which is a complex Gaussian variable with zero mean and unit variance. Internal interference is the co-frequency interference between the communication transceiver pairs, expressed as: in is the transmission power of the transmitter of the communication pair m in time slot t. m，n (t) is the channel gain between the receiver of communication transceiver pair n and the transmitter of communication transceiver pair m: where d m，n represents the distance between the receiver of communication transceiver pair n and the transmitter of communication transceiver pair m (m≠n), Represents the small-scale fading coefficient, which is a complex Gaussian variable with zero mean and unit variance. The communication achievable rate is defined by the received signal-to-interference-to-noise ratio: where λ n，t is the demodulation threshold of the communication transceiver pair n in time slot t, and its size is related to the transmission rate. Different transmission rates correspond to different signal-to-interference-noise ratio demodulation thresholds; The corresponding signal-to-interference-noise ratio demodulation threshold set can be defined as but When SINR n，t (·)≥λ n，t When , it indicates that the communication between the sender and the receiver is successful, otherwise the communication fails. The multi-user cooperative anti-interference problem is modeled as a Markov game, as follows: 1) Action: The action of the communication sender and receiver in time slot t on n is defined as the joint decision of time slot t+1, that is, a n，t =[c n，t+1 , p n，t+1 , m n，t+1， η n，t+1 ], where c n，t+1 , p n，t+1 , m n，t+1 and η n，t+1 They represent the channel index, power, modulation order and coding efficiency that the communication transceiver pair n will select in time slot t+1 respectively; the transmission rate is determined by the modulation order and coding efficiency. Define the modulation order set Coding efficiency set L1 and L2 represent the number of modulation orders and the number of coding efficiencies, respectively. The transmission rate of the communication transceiver pair n in time slot t is l n，t =η n，t log2m n，t , log means finding the logarithm, where m n，t ∈Mod,η n，t ∈Cod. Let the action space A consist of all candidate actions of the communication sending and receiving pairs n. n , then a n，t ∈A n , whose size is C×P×L1×L2; a t =[a 1，t , a 2，t , ..., a n，t , ..., a N，t ] represents the action set of all communicating transceiver pairs in time slot t. 2) State: The state is divided into global state and private state. Define the private state o of the communication pair n in time slot t n，t =[SINR n，t (F1), SINR n，t (F2), ..., SINR n，t (F C ), l n，t ], where the first C elements correspond to the SINR of different channels, l n，t is the transmission rate of the current time slot communication pair n. The private state space O is composed of all private states. n , that is o n，t ∈O n ;Define o t =[o 1，t , o 2，t , ..., o n，t , ..., o N，t ] represents the private state of all communication pairs in time slot t. Correspondingly, the global state s that encompasses all user private states and actions is defined t =[o t , a t-1 ]; All global states form the global state space S, where s t ∈S. 3) Reward: In state s t The following communication sends and receives action a n，t After that, the environment will give the agent n the corresponding reward r n，t , effective rewards help the agent find the optimal communication strategy. Since anti-interference communication requires frequent channel switching, power adjustment and transmission rate adjustment, it will bring certain overhead; for this reason, the channel, power and transmission rate switching weight factors are introduced, and they are expressed as k c , k p and k l Therefore, according to the expression of the achievable communication rate, the utility function of the communication transmission and reception of time slot t to n is defined as: u n，t (c n，t ,c n，t-1 ,p n，t ,p n，t-1 ,l n，t ,l n，t-1 )=V n，t (·)-k c δ c (c n，t ,c n，t-1 )-k p δ p (p n，t ,p n，t-1 )-k l δ l (l n，t ,l n，t-1 ) Where δ(·) is the indicator function. c (c n，t , c n，t-1 )=δ p (p n，t , p n，t-1 )=δ l (l n，t , l n，t-1 )=0, it means that the communication transceiver pair n selects the same action in the t-1 time slot and the t-th time slot; on the contrary, when the communication transceiver pair n selects different actions in the t-1 time slot and the t-th time slot, δ c (c n，t , c n，t-1 )=δ p (p n，t , p n，t-1 )=δ l (l n，t , l n，t-1 )=1. Furthermore, the reward for communication transmission and reception pair n in time slot t is defined as: Where C1 and C2 are constants, u n，t+1 (·) represents the communication transceiver pair n giving a joint strategy action a in time slot t n，t After that, the utility function value obtained in the next time slot t+1.

6. The multi-user collaborative anti-interference strategy based on the multi-agent discrete flexible actor-critic algorithm according to claim 3 is characterized in that: The structure of the deep neural network is as follows: Both the Actor and Critic networks are composed of fully connected neural networks. Each communication pair of senders and receivers has an Actor network, two Critic networks, and two Critic target networks deployed on the agent. The input of the Actor network is the private state o of the communication pair n. n，t , the input parameter is C+1, after being processed by the neural network, the probability value of all discrete actions is output, and the output parameter is (C+1)×P×L1×L2. The input of the Critic network is the global state s t , the input parameter is (C+1)·N+dima n，t N, dim represents the dimension of user n’s actions. After being processed by FCNN, the estimated value of all discrete actions is output, and the output parameter volume is (C+1)×P×L1×L2.

7. The multi-user collaborative anti-interference strategy based on the multi-agent discrete flexible actor-critic algorithm according to claim 1 is characterized in that: The step 10 is specifically as follows: The entire transmission time is divided into T time slots. Communication is carried out in time slots as the basic unit. Each time slot includes five sub-time slots: transmission, spectrum sensing, information exchange, learning and decision-making, and feedback. After each time slot begins, each communication transceiver transmits information according to the joint decision of the current time slot. The duration of the sub-time slot is τ c At the same time, in this sub-time slot, the communication transceiver pair performs broadband spectrum sensing, corresponding to step 3 in claim 1. After completing spectrum sensing, each communication transceiver pair can obtain an environmental reward value and e The sub-time slot of the communication pair completes information exchange, including broadcasting its own current private state and action, and receiving broadcast information from other communication transceiver pairs, corresponding to steps 4 to 6 in claim 1. Then, in the sub-time slot of length τ l In the sub-time slot of , each communication transceiver pair will use the centralized training and distributed execution framework, combine all private states and actions, and then apply the deep reinforcement learning algorithm to train the agent and make a joint decision for the next time slot, corresponding to steps 7 to 9 in claim 1. Finally, in the time slot of τ d In the feedback sub-time slot, the receiver of each communication transceiver pair transmits the joint decision of the next time slot to the corresponding transmitter.

8. The multi-user collaborative anti-interference strategy based on the multi-agent discrete flexible actor-critic algorithm according to claim 7 is characterized in that: The centralized training and distributed execution framework is as follows: During training, the Critic network incorporates the private states and actions of other communication transceiver pairs as additional information; during policy execution, only the Actor network needs to make decisions based on its own private state. The Actor network can quickly output an action that enables the agent to obtain the maximum expected benefit based on the private state of the agent, and the Critic network is used to evaluate the actions output by the agent Actor network during the model training phase, and to assist in the training of the parameters of the agent Actor network. Specifically, the Actor network deployed on the agent of the communication transceiver pair n uses the private state o n，t As input, the output is the probability value of all discrete actions, and then randomly sample action a n，t When the actions of the agents deployed on the receivers of all communication pairs interact with the environment, the environment will feedback to each agent the immediate reward r n，t At the same time, the communication transceiver pair n will broadcast its own private state and action to other communication transceiver pairs through the information exchange link, and receive the private state and action broadcast by other users. Finally, the experience sample ξ is formed in the form of a tuple t , and stored in the experience replay pool according to the first-in-first-out principle When enough experience is accumulated in the experience replay pool, the communication sender and receiver pair n will extract the experience containing Batches of samples are used to update network parameters.

9. The multi-user collaborative anti-interference strategy based on the multi-agent discrete flexible actor-critic algorithm according to claim 8 is characterized in that: The network parameters are updated as follows: The parameter θ n，i , that is, the Critic network parameters, can be determined by minimizing the soft Bellman residual: Where L Q represents the loss value of the Critic network, and the subscript Q represents the Critic network. t , a t , r n，t ,s t+1 ) is the experience playback pool of the communication transceiver n The experience sample in y n，t is the time difference target value, which represents the global state s t+1 Next, implement action a n，t The soft Q-value estimation of in The parameter φ n , that is, the Actor network parameter, log means finding the logarithm, and min means finding the minimum value. It can be determined by minimizing the following loss function using the soft strategy improvement method: In the above formula, min means to find the minimum value, superscript T means to find the transpose, and temperature parameter α n Determines the importance of entropy to the algorithm. n If α is too small, the algorithm may degenerate into a common Actor-Critic algorithm. n This will cause the algorithm to focus too much on maximizing entropy and ignore the impact of environmental rewards. To this end, it is hoped that when facing an environment that has been explored, α will be reduced. n To reduce entropy; when facing an unknown environment, increase α n To enhance the randomness of strategy exploration. To this end, a target entropy And update α by minimizing the following loss function n : For the parameters of the target network and Update in soft copy mode, i.e. The Actor network, Critic network and temperature parameters are updated using the stochastic gradient descent method, namely where μ Q , μ π and μ α are the learning rates of the Critic network, Actor network and temperature parameters respectively, It means to find the gradient.

Images