Multi-domain joint anti-interference decision-making method based on improved SAC

Abstract

The invention discloses a multi-domain joint anti-interference decision-making method based on improved SAC, and belongs to the technical field of wireless communication. The method comprises the following steps: constructing a wireless communication system consisting of a pair of communication users and a jammer; modeling a communication parameter decision process as a Markov decision process; initializing agent parameters and system parameters; the intelligent agent generates an anti-interference action according to the environment state and sends the action to the transmitter; the transmitter executes an anti-interference action, and the receiver sends feedback information to the intelligent agent; the samples (st, at, rt, st + 1) are stored in an experience playback pool, and the environment state is updated; the intelligent agent extracts a sample from the experience playback pool to update the value network; updating regularization coefficients of the strategy network, the target network and the entropy; and repeating the interaction and updating steps until the requirements are met. According to the method provided by the invention, the anti-interference decision can be realized under the condition that the interference pattern is unknown, the anti-interference capability and the self-adaptive capability of a communication system are improved, and the communication success rate of two communication parties is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of wireless communications and relates to a multi-domain joint anti-interference decision method based on improved SAC. Background Art

[0002] With the continuous development of information technology and the rapid popularization of wireless devices, wireless communication plays an increasingly important role in daily life. While wireless communication technology is developing rapidly, the wireless electromagnetic environment it faces is becoming more and more complex. Due to its own open nature, wireless communication is easily attacked by external interference during the transmission of information, resulting in a decrease in communication quality or even failure of normal communication. Traditional frequency hopping and spread spectrum technologies are difficult to resist increasingly intelligent malicious interference attacks due to their fixed anti-interference strategies, and the anti-interference performance achieved is difficult to meet the requirements of anti-interference communication.

[0003] With the rapid development of artificial intelligence technology with machine learning as the core, machine learning algorithms are increasingly used in the field of communications. In particular, reinforcement learning algorithms are widely used in the field of anti-interference decision-making because they can learn optimal strategies and make real-time dynamic decisions by constantly interacting with the environment in an unknown environment. Most of the existing anti-interference decision-making studies consider the construction of a single-domain or two-domain joint anti-interference decision-making model from the time domain, frequency domain, power domain and spatial domain, and use relevant reinforcement learning algorithms to intelligently decide communication parameters to achieve anti-interference effects and achieve the purpose of continuous normal communication between the sender and receiver. When considering a single-domain anti-interference decision-making model, the system model is simple to construct, but when the interference power of the interferer is large or the interference frequency band is wide, the communication party cannot effectively resist the interference. In addition, when considering the multi-domain joint decision-making problem, the decision space becomes larger due to the increase in decision dimensions, and the algorithm has problems of slower decision-making speed and poor decision stability. Summary of the invention

[0004] In view of the above technical problems, the present invention proposes a multi-domain joint anti-interference decision-making method based on improved SAC. In view of malicious interference in wireless communication environments, the present invention considers that communication users jointly counter interference from multiple domains including frequency domain, power domain, modulation domain and coding rate domain, and uses an improved SAC (Soft Actor Critic) algorithm to solve the problems of slow decision-making speed and poor decision-making stability in the algorithm during multi-domain joint decision-making. The proposed algorithm can realize intelligent anti-interference decision-making by interacting with the environment in the case of unknown interference patterns, thereby improving the anti-interference performance and adaptive capability of the communication system and the communication success rate of both parties.

[0005] 1. A multi-domain joint anti-interference decision-making method based on improved SAC, characterized in that the method comprises the following steps:

[0006] S1: Construct a wireless communication system consisting of a pair of communication users and a jammer. The communication users and the jammer use time slots as the minimum communication unit for data transmission. The jammer selects channels and interference power in each time slot to interfere with the normal communication of the communication user. The communication user interacts with the jammer in each time slot to decide the communication parameters of the next time slot to counter the interference.

[0007] S2: The process of interactive decision-making of communication parameters between communication users and jammers in wireless communication systems is modeled as a Markov decision process;

[0008] S3: Construct the network structure based on the improved SAC multi-domain joint anti-interference decision-making method, initialize the network parameters and system parameters, and initialize the environment state;

[0009] S4: The agent at the sending end inputs the received environmental state into the policy network, generates an anti-interference action, and sends the anti-interference action to the sender;

[0010] S5: The transmitter executes the received anti-interference action, the receiver obtains interference information through spectrum sensing, generates a reward value according to the reward function, obtains the environmental state of the next time slot, and sends feedback information including interference information and reward value to the agent;

[0011] S6: The agent stores the current time slot environment state, action, reward, and next time slot environment state into the experience replay pool and updates the environment state;

[0012] S7: When the number of elements in the experience replay pool reaches the specified requirement, the agent updates the network parameters while interacting with the environment, and uses the priority experience replay mechanism to extract a certain number of samples from the experience replay pool to update the value network parameters;

[0013] S8: When the number of value network parameter updates reaches the requirement, update the strategy network and target network, and update the entropy regularization coefficient;

[0014] S9: Repeat steps S4 to S8 until the specified requirements are met.

[0015] 2. The method according to claim 1 is characterized in that, in step S1, within a time slot, the interference generated by the jammer does not change; within a communication time slot, the transmitter transmits information through a data link according to the received anti-interference action; the receiver obtains interference channel and interference power information through spectrum sensing, and feeds back the perceived interference information, signal to noise ratio, and reward to the intelligent agent through a feedback link; the intelligent agent learns according to the received feedback information and the perceived environmental information, and outputs the communication parameters of the communication party in the next time slot; the signal to noise ratio calculation formula of the receiving end is expressed as:

[0016]

[0017] Among them, σ 2 represents the noise power, h u represents the channel gain from transmitter to receiver, h j represents the channel gain from the jammer to the receiver; δ(x) is an indicator function, and represents the communication channel and communication power selected by the communication user in the k+1 time slot, and It represents the interference channel and interference power selected by the jammer in the k+1 time slot.

[0018] 3. The method according to claim 1, characterized in that in step S2, the constructed Markov decision process consists of four tuples The specific elements are defined as follows:

[0019] represents the state space, where and v k They represent the communication channel, transmission power and transmission rate selected by the communication party in time slot k respectively. and They represent the interference channel and interference power selected by the k-time slot jammer respectively;

[0020] represents the action space, where a k represents the action chosen by the communicating party in time slot k, and v k+1 They represent the communication channel, communication power and transmission rate selected by the communication party in the k+1 time slot respectively;

[0021] The state transition probability P is defined as Indicates the current state of the agent Next select action Transfer to next state probability;

[0022] r represents the reward, and the reward function is set as follows:

[0023]

[0024] Among them, θ(x) is an indicator function, c u is the power transfer cost, c f is the frequency switching cost, v max represents the maximum information transmission rate that the transmitter can choose, p u_maxIndicates the maximum communication power that the transmitter can choose; SINR th It indicates the signal-to-interference-noise ratio threshold that the receiving end needs to reach for normal demodulation when the communicating parties transmit data according to a certain modulation mode and coding rate.

[0025] 4. The method according to claim 1, characterized in that in step S3, the six network parameters initialized are: strategy network π θ Network parameters θ, target policy network Network parameters Two value networks Network parameters α1 and α2, two target value networks Network parameters and The initialized system parameters include the number of training rounds, the total number of training time slots, the capacity of the experience replay pool, the number of sampling samples, the learning rate of the policy network and the learning rate of the value network; the environment state is initialized.

[0026] 5. The method according to claim 1, characterized in that, in step S7, the value network parameter updating method is as follows:

[0027] The samples (s) drawn from the experience replay pool i ,a i ,r i ,s i+1 ) in the next time slot state s i+1 Input the target policy network to get action a i ' +1 Distribution, calculation target:

[0028]

[0029] where r i represents the reward obtained in time slot i; γ represents the discount factor; It means that according to the action value obtained by the target value network, the one with the smaller output action value is selected from the two target value networks as the target Q value to alleviate the problem of overestimation of Q value; α is the regularization coefficient of entropy, represents the action probability output by the target policy network;

[0030] Get the action value from the value network Calculate the loss function of the two value networks Update the value network parameters to minimize the loss function.

[0031] 6. The method according to claim 1, characterized in that in step S8, the regularization coefficient update method of the strategy network parameters, the target network parameters and the entropy is:

[0032] The action probability distribution π is obtained by the policy network according to the state θ (a i |s i ), calculate the strategy network loss function based on the action entropy value and action value Update the policy network to minimize the loss function according to Update the entropy regularization coefficient α, and soft-update the target value network parameters and target strategy network parameters.

[0033] Beneficial Effects

[0034] A multi-domain joint anti-interference decision-making method based on improved SAC is different from the traditional method of considering single-domain or two-domain joint decision-making from the time domain, frequency domain and power domain. The present invention considers the joint frequency domain, power domain, modulation domain and coding rate domain for anti-interference decision-making. In the case of unknown interference prior knowledge, the intelligent body can learn anti-interference strategies through continuous interaction with the environment, effectively deal with interference, improve the overall anti-interference performance and adaptability of the communication system, and improve the communication success rate of both communicating parties. The present invention improves the original SAC algorithm, further considers and utilizes sample information, thereby accelerating the convergence speed of the algorithm and improving the anti-interference performance of the algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to make the purpose, technical solution and beneficial effects of the present invention clearer, the present invention provides the following drawings for illustration:

[0036] Figure 1 A schematic diagram of a wireless communication system model considered by the present invention;

[0037] Figure 2 A schematic diagram of time slot division of a wireless communication system considered by the present invention;

[0038] Figure 3 The algorithm processing framework diagram considered for the present invention; DETAILED DESCRIPTION

[0039] In order to make the steps of the present invention more detailed and clear, the present invention is further described in detail below with reference to the accompanying drawings and implementation examples.

[0040] like Figure 1As shown in the figure, the communication party in the wireless communication system consists of a transmitter and a receiver, and the interference party consists of a jammer. The transmitter and the receiver transmit data through a data link, and the receiver feeds back feedback information to the intelligent agent at the transmitting end through a feedback link. The intelligent agent learns and makes decisions based on the received feedback information and the perceived environmental state information, and sends the communication parameters of the decision to the transmitter through the decision link. It is assumed that both the transmitter and the receiver have a certain perception ability and can perceive the interference situation of each channel and the interference power size on the interference channel. The jammer can only interfere with the receiver and cannot interfere with the feedback link.

[0041] Assume that the available bandwidth of the communication system is B MHz, the channel bandwidth is b MHz, and the center frequency of the i-th channel is f i , the communication frequency range is The communication parameters that can be selected by the communication parties include communication channel, communication transmission power, modulation mode and coding rate. The total number of available channels in the communication system is m, and the channel set is M = {0, 1, 2, ... m-1}. The user's available power set is The available modulation modes are: The encoding rate set is Assuming that the coding method used by the communication party is fixed, under the condition that other factors are the same, the communication transmission rate of the communication party is determined by the modulation method and the coding rate. The communication transmission rate set that the communication party can choose is Where N3=N1×N2.

[0042] The set of interference channels that the interferer can select is consistent with the set of communication channels, and the set of interference power that the interferer can select is

[0043] Assume that the wireless communication system is a time slot communication system, that is, the communication users and jammers in the wireless communication system use time slots as the minimum communication unit for data transmission. The system time slot division is as follows: Figure 2 As shown. The interference generated by the jammer will not change within a time slot. The communication party can only choose one channel for communication in each time slot. In a communication time slot, the transmitter transmits information through the data link according to the received anti-interference action. The receiver obtains the interference channel and interference power selected by the jammer through spectrum sensing, and feeds back the perceived interference information, signal-to-interference-noise ratio, and the reward obtained according to the reward function to the intelligent agent through the feedback link. The intelligent agent updates the network parameters in the learning stage according to the received feedback information and the perceived environmental information, and decides to output the communication parameters of the communication party in the next time slot.

[0044] The signal-to-interference-noise ratio calculation formula at the receiving end is as follows:

[0045]

[0046] Among them, σ 2 represents the noise power, h u represents the channel gain from transmitter to receiver, h j represents the channel gain from the jammer to the receiver; δ(x) is an indicator function, and represents the communication channel and communication power selected by the communication party in the k+1 time slot, and It represents the interference channel and interference power selected by the interferer in the k+1 time slot.

[0047] The process of interactive decision-making of communication parameters between communication users and jammers in wireless communication systems is modeled as a Markov decision process. The Markov decision process consists of four tuples. The specific elements are defined as follows:

[0048] represents the state space, where and v k They represent the communication channel, transmission power and transmission rate selected by the communication party in time slot k respectively. and They represent the interference channel and interference power selected by the interferer in time slot k respectively.

[0049] represents the action space, where a k represents the action chosen by the communicating party in time slot k, and v k+1 They respectively represent the communication channel, communication power and transmission rate selected by the communicating party in the k+1 time slot.

[0050] P represents the state transition probability, which is defined as Indicates the current state of the agent Next select action Transfer to next state probability.

[0051] r represents the reward, and the reward function is set as follows:

[0052]

[0053] Among them, θ(x) is an indicator function, c u is the power transfer cost, c f is the frequency switching cost, v max represents the maximum information transmission rate that the transmitter can choose, pu_max Indicates the maximum communication power that the transmitter can select. SINR th It indicates the signal-to-interference-noise ratio threshold that the receiving end needs to reach for normal demodulation when the communicating parties transmit data according to a certain modulation mode and coding rate.

[0054] In the anti-interference decision model of the present invention, the action space of the communicating party is a discrete action space. The original SAC method is for continuous action space. To make it applicable to discrete action space, it is necessary to modify it on the basis of the original algorithm. The output of the policy network of the SAC algorithm for continuous action is the mean and variance of the Gaussian distribution, and the action value is selected according to the Gaussian distribution. The value network inputs the state and action, and outputs a real value to represent the action value function. The policy network of the SAC algorithm for discrete action outputs the softmax distribution of the discrete action space, and the discrete action value is obtained by sampling according to the softmax distribution. The value network inputs the state and outputs the action value function of all actions in the state.

[0055] In the anti-interference decision model of the present invention, corresponding improvements are made on the basis of the SAC algorithm in view of the shortcomings of the SAC algorithm. First, the original one policy network is expanded into two policy networks. The behavior policy network is responsible for interacting with the environment, and the target policy network is used to generate target values, reducing the target uncertainty caused by the frequent updates of the policy network. Secondly, the instant update mechanism of the SAC algorithm is converted into a delayed update mechanism, so that the update frequency of the policy network is lower than that of the value network, reducing the network propagation error in the instant update. Finally, the random sampling of samples from the experience replay pool in the SAC algorithm is converted to sampling using a priority experience replay mechanism to improve the utilization rate of samples.

[0056] The processing framework of the multi-domain joint anti-interference decision-making method based on improved SAC is shown in the figure Figure 3 As shown in the figure. In the initial stage of learning, the agent randomly initializes its own network weights. Then, according to the acquired environment state, the policy network generates an anti-interference action, sends the anti-interference action to the transmitter, and the transmitter executes the action to transmit information. The receiver generates feedback information and sends the feedback information to the agent. The sample information consisting of the current state, action, reward, and next state is stored in the experience replay pool.

[0057] When the number of samples in the experience replay pool does not meet the requirement, the agent continues to interact with the environment to generate more samples and store them in the experience replay pool without updating the network parameters.

[0058] When the number of samples in the experience replay pool reaches the requirement, the agent interacts with the environment while updating the network parameters. Specifically, a batch of samples is extracted from the experience replay pool using the priority experience replay mechanism, the target value is calculated using the target network, and a value with a smaller output action value is selected from the two target value networks as the target action value to alleviate the problem of overestimation of Q value. According to the action values ​​output by the two value networks, the value network loss function is calculated, and the value network parameters are updated to minimize the value network loss function. When the number of value network updates reaches the requirement, the action probability distribution is obtained by the policy network, and the action value is obtained by the value network based on the state and action distribution. The policy network loss function is calculated based on the action probability distribution and the action value, and the policy network parameters are updated to minimize the policy network loss function. The entropy regularization coefficient is updated according to the loss function of the entropy regularization coefficient, and the target value network and target policy network are soft-updated.

[0059] The specific pseudo code of the algorithm in the multi-domain joint anti-interference decision-making method based on improved SAC is as follows:

[0060]

[0061]

[0062] Finally, it should be noted that the above preferred embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail through the above preferred embodiments, those skilled in the art should understand that various changes can be made thereto in form and details without departing from the scope defined by the claims of the present invention.

Claims

1. A multi-domain joint anti-interference decision-making method based on improved SAC, characterized in that: The method comprises the following steps: S1: Construct a wireless communication system consisting of a pair of communication users and a jammer. The communication users and the jammer use time slots as the minimum communication unit for data transmission. The jammer selects channels and interference power in each time slot to interfere with the normal communication of the communication user. The communication user interacts with the jammer in each time slot to decide the communication parameters of the next time slot to counter the interference. S2: The process of interactive decision-making of communication parameters between communication users and jammers in wireless communication systems is modeled as a Markov decision process; S3: Build a network structure based on the improved SAC (Soft Actor Critic) multi-domain joint anti-interference decision-making method, initialize network parameters and system parameters, and initialize the environment state; S4: The agent at the sending end inputs the received environmental state into the policy network, generates an anti-interference action, and sends the anti-interference action to the sender; S5: The transmitter executes the received anti-interference action, the receiver obtains interference information through spectrum sensing, generates a reward value according to the reward function, obtains the environmental state of the next time slot, and sends feedback information including interference information and reward value to the agent; S6: The agent stores the current time slot environment state, action, reward, and next time slot environment state into the experience replay pool and updates the environment state; S7: When the number of elements in the experience replay pool reaches the specified requirement, the agent updates the network parameters while interacting with the environment, and uses the priority experience replay mechanism to extract a certain number of samples from the experience replay pool to update the value network parameters; S8: When the number of value network parameter updates reaches the requirement, update the strategy network and target network, and update the entropy regularization coefficient; S9: Repeat steps S4 to S8 until the specified requirements are met.

2. The method according to claim 1, characterized in that: In step S1, within a time slot, the interference generated by the jammer will not change; within a communication time slot, the transmitter transmits information through the data link according to the received anti-interference action; the receiver obtains the interference channel and interference power information through spectrum sensing, and feeds back the perceived interference information, signal to noise ratio, and reward to the intelligent agent through the feedback link; the intelligent agent learns according to the received feedback information and the perceived environmental information, and outputs the communication parameters of the communication party in the next time slot; the signal to noise ratio calculation formula of the receiving end is expressed as: Among them, σ 2 represents the noise power, h u represents the channel gain from transmitter to receiver, h j represents the channel gain from the jammer to the receiver; δ(x) is an indicator function, and represents the communication channel and communication power selected by the communication user in the k+1 time slot, and It represents the interference channel and interference power selected by the jammer in the k+1 time slot.

3. The method according to claim 1, characterized in that: In step S2, the constructed Markov decision process consists of the four-tuple The specific elements are defined as follows: represents the state space, where and v k They represent the communication channel, transmission power and transmission rate selected by the communication party in time slot k respectively. and They represent the interference channel and interference power selected by the k-time slot jammer respectively; represents the action space, where a k represents the action chosen by the communicating party in time slot k, and v k+1 They represent the communication channel, communication power and transmission rate selected by the communication party in the k+1 time slot respectively; The state transition probability P is defined as Indicates the current state of the agent Next select action Transfer to next state probability; r represents the reward, and the reward function is set as follows: Among them, θ(x) is an indicator function, c u is the power transfer cost, c f is the frequency switching cost, v max represents the maximum information transmission rate that the transmitter can choose, p u_max Indicates the maximum communication power that the transmitter can choose; SINR th It indicates the signal-to-interference-noise ratio threshold that the receiving end needs to reach for normal demodulation when the communicating parties transmit data according to a certain modulation mode and coding rate.

4. The method according to claim 1, characterized in that: In step S3, the six network parameters initialized are: strategy network π θ Network parameters θ, target policy network Network parameters Two value networks Network parameters α1 and α2, two target value networks Network parameters and The initialized system parameters include the number of training rounds, the total number of training time slots, the capacity of the experience replay pool, the number of sampling samples, the learning rate of the policy network and the learning rate of the value network; the environment state is initialized.

5. The method according to claim 1, characterized in that: In step S7, the value network parameter updating method is as follows: The samples (s) drawn from the experience replay pool i ,a i ,r i ,s i+1 ) in the next time slot state s i+1 Input the target policy network to get action a i ' +1 Distribution, calculation target: where r i represents the reward obtained in time slot i; γ represents the discount factor; It means that according to the action value obtained by the target value network, a value with a smaller output action value is selected from the two target value networks as the target Q value to alleviate the problem of overestimation of Q value; α is the regularization coefficient of entropy, represents the action probability output by the target policy network; Get the action value from the value network Calculate the loss function of the two value networks Update the value network parameters to minimize the loss function.

6. The method according to claim 1, characterized in that: In step S8, the policy network parameters, target network parameters and entropy regularization coefficients are updated as follows: The action probability distribution π is obtained from the policy network θ (a i |s i ), calculate the strategy network loss function based on the action entropy value and action value Update the policy network parameters to minimize the loss function according to Update the entropy regularization coefficient α, and soft-update the target value network parameters and target strategy network parameters.

Images