Wireless communication multi-dimensional joint anti-interference method and system

Abstract

The invention discloses a multi-dimensional joint anti-interference method and system for wireless communication, and the method comprises the steps: modeling an anti-interference decision required by a communication party into a Markov decision process in an interference environment; determining a state and an action of a Markov decision process, obtaining a state space, an action space and a corresponding state-action matrix, namely, a Q table, and initializing the state space, the action space and the corresponding state-action matrix; determining an instant reward function of a Markov decision process, calculating an instant reward value corresponding to the action, and taking the instant reward value as a key basis for updating a corresponding Q value in a Q table; on the basis of a greedy strategy algorithm, selecting an action in the current state, obtaining a corresponding reward value after the action is executed, recording and updating a Q value in the Q table, and transferring to a next moment state; and iterating the action-value function of the transmitter of the communication party to be converged or iterated to a specified number of times to obtain a final Q table, and outputting an optimal anti-interference result of the transmitter. The method can dynamically adapt to different interference scenes and make an intelligent anti-interference decision.

Description

Technical Field

[0001] The present invention relates to the field of wireless communication technologies, and in particular to a multi-dimensional joint anti-interference method and system for wireless communication. Background Art

[0002] Wireless communication is carried out by electromagnetic waves, which are open and susceptible to interference from other electromagnetic waves. This has led to the rapid development of various interference and anti-interference technologies, and the interference threats faced by communication systems have become increasingly complex and varied. Traditional frequency control technologies such as spread spectrum and frequency hopping, and time slot control technologies such as interference avoidance are not adequate in terms of anti-interference capabilities, and are unable to cope with highly dynamic, diversified, and intelligent interference.

[0003] In order to counter enemy intelligent interference and ensure the reliability of communication transmission, communication systems urgently need to introduce more intelligent methods to improve the adaptability and robustness of the system. Summary of the invention

[0004] In order to solve the above problems, the present invention proposes a multi-dimensional joint anti-interference method and system for wireless communication, which is based on interference monitoring, analyzing channel interference, obtaining the mode and power of interference signals, and providing an anti-interference model based on the Markov decision process, and using the Q-Learning algorithm for implementation, so that the Q-Learning algorithm can dynamically adapt to different interference scenarios and make intelligent anti-interference decisions. Among them, Q-Learning, as a method of adapting to environmental changes and learning through trial and error, shows great potential in communication systems. It enables communication systems to learn to adopt appropriate strategies in different interference scenarios and realize dynamic configuration of anti-interference performance.

[0005] The technical solution adopted by the present invention is as follows:

[0006] A multi-dimensional joint anti-interference method for wireless communication, comprising:

[0007] In an interference environment, the anti-interference decision required by the communication party is modeled as a Markov decision process, and interference countermeasures are performed based on the Q-Learning algorithm;

[0008] Determine the state and action of the Markov decision process, obtain the state space S and action space a, and the corresponding state-action matrix, i.e., Q table, and initialize them;

[0009] Determine the immediate reward function of the Markov decision process, calculate the immediate reward value corresponding to the action, and use it as the key basis for updating the corresponding Q value in the Q table;

[0010] Input the interference pattern and parameters perceived in the interference environment and the initial state of the current communication party, select the action in the current state based on the greedy strategy algorithm, and obtain the corresponding reward value R after executing the action k , record and update the Q value in the Q table, and transfer to the next state s k+1 ;

[0011] The action-value function Q(s k ,a k ) Iterate until convergence or iterate to a specified number of times to obtain the final Q table and output the best anti-interference result of the transmitter

[0012] Furthermore, in determining the state and action of the Markov decision process, the state to be determined includes the communication mode of the communicating party and the interference mode of the interfering party. in is the communication channel, transmission power and modulation mode in the current k time slot, is the channel, power and interference type of the interference signal in the current k time slot.

[0013] Furthermore, in the state and action of determining the Markov decision process, the communication party makes an anti-interference decision. In the current k time slots, the action selection can only change the communication mode. That is, changing the communication channel, transmission power and modulation method.

[0014] Furthermore, determining the instantaneous reward function of the Markov decision process and calculating the instantaneous reward value corresponding to the action includes:

[0015] When the selected action reaches the transmission rate required by the communication system, the corresponding transmission rate reward will be obtained;

[0016] When the selected action reaches the bit error rate required by the communication system, the corresponding bit error rate reward is obtained;

[0017] When the selected action satisfies the above two conditions at the same time, the corresponding channel selection reward, power selection reward and modulation mode selection reward are obtained.

[0018] Furthermore, the instantaneous reward function of the Markov decision process includes:

[0019] R=ω1f ber δ(x1)+ω2f rate δ(x2)+(ω3f power +ω4f modulation +ω5f channels )δ(x1)δ(x2)

[0020] Among them, R is the immediate reward function, fber is the normalized bit error rate evaluation index, f rate is the normalized transmission rate evaluation index, f power is the normalized power index, f modulation is the normalized modulation index, f channels is the normalized channel combination index; ω1~ω5 are weight coefficients; δ(x1) is the bit error rate indicator function, and δ(x2) is the transmission rate indicator function.

[0021] Furthermore, the greedy strategy algorithm is used to select the action in the current state, including: using the greedy strategy, the transmitter selects the action with probability 1-ε And other actions are randomly selected with probability ε, so the probability of selecting an action is expressed as:

[0022]

[0023] Furthermore, the interference pattern and parameters perceived in the input interference environment and the initial state of the current communicating party are used to select an action in the current state based on a greedy strategy algorithm, including: using time division multiple access, at the beginning of each time slot, the communicating party perceives the interference parameters, and using a Q-Learning algorithm to select an action based on the perceived interference parameters.

[0024] Furthermore, the greedy strategy algorithm selects an action in the current state and obtains a corresponding reward value R after executing the action. k , record and update the Q value in the Q table, including:

[0025] The communication party makes online decisions based on historical information and current status, selects execution actions based on the greedy strategy, obtains the benefits of different actions, and then updates the Q table; the communication party's goal is to maximize the cumulative discounted reward at the current moment:

[0026]

[0027] Where γ∈(0,1] is a discount factor used to weigh the impact of potential future rewards on the present.

[0028] Furthermore, the function Q(s) corresponding to the Q value k ,a k ) means in state s k Next, perform action a k The value of:

[0029]

[0030] Among them, γ i r (k+i+1)is a series of discounted future reward values, s and a are the state space set and action space set respectively;

[0031] Combined with the Bellman equation, the expression of the action-value function is obtained as follows:

[0032]

[0033] Among them, s k+1 Indicates the state of the next time slot, a k+1 Indicates the action that will be selected in the next time slot.

[0034] A wireless communication multi-dimensional joint anti-interference system, comprising:

[0035] An anti-interference decision conversion module is configured to model the anti-interference decision required by the communication party in the interference environment as a Markov decision process and perform interference countermeasure based on a Q-Learning algorithm;

[0036] The state and action determination module is configured to determine the state and action of the Markov decision process, obtain the state space S and the action space a, and the corresponding state-action matrix, i.e., the Q table, and initialize them;

[0037] A reward function determination module is configured to determine the instantaneous reward function of the Markov decision process, calculate the instantaneous reward value corresponding to the action, and use it as a key basis for updating the corresponding Q value in the Q table;

[0038] The anti-interference decision optimization module is configured to input the interference mode and parameters perceived in the interference environment and the initial state of the current communication party, and select the action in the current state based on the greedy strategy algorithm, and obtain the corresponding reward value R after executing the action. k , record and update the Q value in the Q table, and transfer to the next state s k+1 ;

[0039] The anti-interference result output module is configured to convert the action-value function Q(s k ,a k ) Iterate until convergence or iterate to a specified number of times to obtain the final Q table and output the best anti-interference result of the transmitter

[0040] The beneficial effects of the present invention are:

[0041] (1) By intelligently adjusting the communication channel, transmission power and modulation mode, it is beneficial to change the communication strategy in real time when interference occurs, and quickly build reliable wireless communication. At the same time, due to the multi-dimensional joint anti-interference, in an environment with complex interference patterns and changing interference types, various factors can be comprehensively considered to flexibly select the anti-interference communication mode.

[0042] (2) The selected anti-interference communication mode can always guarantee the transmission rate and bit error rate requirements in the current time slot, and it has a certain resistance to interference.

[0043] (3) The Q-table is iteratively generated through the Q-Learning algorithm, and the anti-interference result is finally output through table lookup. The Q-table is simple in structure, the calculation complexity is relatively low, and it has the ability to be updated in real time. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is a flow chart of a multi-dimensional joint anti-interference method for wireless communication according to embodiment 1 of the present invention.

[0045] Figure 2 This is a flow chart of a multi-dimensional joint anti-interference method for wireless communication according to embodiment 2 of the present invention.

[0046] Figure 3 It is a schematic diagram of the anti-interference action strategy for the intelligent agent to interact with the environment and update the anti-interference action strategy according to Example 2 of the present invention.

[0047] Figure 4 This is the 16-QAM constellation mapping diagram of Example 2 of the present invention.

[0048] Figure 5 It is a schematic diagram of the anti-interference effect against narrowband noise interference in Example 2 of the present invention.

[0049] Figure 6 It is a schematic diagram of the anti-interference effect against broadband noise interference in Example 2 of the present invention.

[0050] Figure 7 It is a schematic diagram of the anti-interference effect against single-tone interference in Example 2 of the present invention.

[0051] Figure 8 It is a schematic diagram of the anti-interference effect against multi-tone interference in Example 2 of the present invention. DETAILED DESCRIPTION

[0052] In order to have a clearer understanding of the technical features, purposes and effects of the present invention, the specific implementation methods of the present invention are now described. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention, that is, the embodiments described are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present invention.

[0053] Example 1

[0054] like Figure 1As shown, this embodiment provides a multi-dimensional joint anti-interference method for wireless communication, including:

[0055] In an interference environment, the anti-interference decision required by the communication party is modeled as a Markov decision process, an anti-interference model based on the Markov decision process is obtained, and interference countermeasures are performed based on the Q-Learning algorithm;

[0056] Determine the state and action of the Markov decision process, obtain the state space S and action space a, and the corresponding state-action matrix, i.e., Q table, and initialize them;

[0057] Determine the immediate reward function of the Markov decision process, calculate the immediate reward value corresponding to the action, and use it as the key basis for updating the corresponding Q value in the Q table;

[0058] Input the interference pattern and parameters perceived in the interference environment and the initial state of the current communication party, select the action in the current state based on the greedy strategy algorithm, and obtain the corresponding reward value R after executing the action k , record and update the Q value in the Q table, and transfer to the next state s k+1 ;

[0059] The action-value function Q(s k ,a k ) Iterate until convergence or iterate to a specified number of times to obtain the final Q table and output the best anti-interference result of the transmitter

[0060] In this embodiment, the anti-interference model based on the Markov decision process includes the following features:

[0061] Feature 1, the definition of state S, including the communication mode of the communicating party and the interference mode of the interfering party in is the communication channel, transmitter power, and modulation mode in the current k time slot; It is the channel, power and interference type of the interference signal in the current time slot.

[0062] Feature 2: The communication party makes an anti-interference decision. In the current time slot, the selection of action α can only change the communication mode. That is, change the communication channel, transmission power, and modulation method.

[0063] Feature 3: Q-Learning algorithm is used to implement the Markov decision process. If the transmission rate required by the communication system is achieved under a specific interference mode, the corresponding transmission rate reward will be obtained; if the bit error rate required by the communication system is met, the corresponding bit error rate reward will be obtained. Only when the selected action meets the above two basic requirements at the same time can the corresponding channel selection reward, power selection reward and modulation mode selection reward be obtained.

[0064] Feature 4, the optimization of transition probability, that is, strategy, which represents the probability distribution of mapping from the current state S to an optional action α, that is, π(s k ,a k ):s k →a k In the Q-Learning algorithm, the action for the next time slot is selected using a greedy strategy algorithm.

[0065] Specifically, in feature 1, for the Markov decision process, the state space set is defined as: in is the interfered channel detected in the current time slot; is the power of the interference signal in the current time slot; The type of interference signal in the current time slot includes {single-tone interference, multi-tone interference, narrowband noise interference, broadband noise interference}; the channels selected for the communication system; is the transmission power of the communication signal; The modulation methods selected for communication signals include {BPSK, QPSK, 16QAM, 64QAM}. N is the combination of all available channels. The transmitter power set is P U = {P U1 ,P U2 ,…,P Ui ,P UL}, where P Ui Indicates the available transmit power, there are L types of transmit power. The transmit power selected in the kth time slot is The interference power set is P J = {P J1 ,P J2 ,…,P Ji ,P JW}, where P Ji Indicates the detected interference power, there are W types of interference power. The interference power in the kth time slot is

[0066] In feature 2, the action space set is defined as: in The communication channel selected for the k-th time slot transmitter The transmission power adopted for the k-th time slot The modulation method adopted for the k-th time slot. Therefore, the size of the action space is N×L×4.

[0067] In Feature 3, the definition of the reward function is as follows:

[0068] R = ω1f ber δ(x1) + ω2f rate δ(x2) + (ω3f power + ω4f modulation + ω5f channels )δ(x1)δ(x2)

[0069] The reward function R consists of five basic elements and two indicator functions, where:

[0070] f ber is the normalized bit error rate evaluation index, indicating the normalized system bit error rate calculated after taking a certain action in the k-th time slot; f rate is the normalized transmission rate evaluation index, indicating the normalized system transmission rate calculated after taking a certain action in the k-th time slot. f power is the normalized power index, f modulation is the normalized modulation method index, f channels is the normalized representation of the selected channel combination index.

[0071] ω is the weight coefficient. Since the five basic elements play different roles in the anti-jamming communication system, their weight values are also different.

[0072] δ(x) is the indicator function. When x is true, the value of δ(x) is 1, otherwise the value is 0. δ(x1) represents the bit error rate indicator function, that is, δ(x1) = δ(p e <p e0 )), where P e is the actual system bit error rate after selecting the action, P e0 is the bit error rate threshold required by the system. δ(x2) represents the transmission rate indicator function, that is, δ(x2) = δ(R0 < R < C), where R is the actual transmission rate, R0 is the transmission rate required by the system, and C is the channel capacity. Only when the selected action meets the bit error rate requirement can the first reward be obtained; only when the transmission rate requirement is met can the second reward be obtained; only when both the bit error rate and transmission rate requirements are met can the last three rewards be obtained.

[0073] In Feature 4, the greedy strategy is adopted, and the transmitter selects the action with probability 1 - ε where satisfies And other actions are randomly selected with a probability of ε, so the probability of selecting an action is expressed as:

[0074]

[0075] This embodiment adopts a time division multiple access method. At the beginning of each time slot, the communication party senses the interference parameter and selects an action using Q-Learning based on the sensed interference parameter.

[0076] Specifically, the communication party makes online decisions based on historical information and current status, selects actions to be executed by the greedy strategy, obtains the benefits of different actions, and then updates the Q table. The goal of the communication anti-interference agent is to maximize the cumulative discounted reward at the current moment, that is:

[0077]

[0078] Where γ∈(0,1] is a discount factor used to weigh the impact of potential future rewards on the present. k It can also be written as:

[0079]

[0080] Q-Learning is to store Q values ​​by transforming states and actions into a two-dimensional Q table. k ,a k ) means in state s k Next, perform action a k The value of can be expressed as:

[0081]

[0082] Here is the gamma i r (k+i+1) is a series of discounted future reward values, s and a are the state space set and action space set respectively. Combined with the Bellman equation, the expression of the state-action Q value function is:

[0083]

[0084] Among them, s k+1 Indicates the state of the next time slot, a k+1 Indicates the action that will be selected in the next time slot. Finally, the agent uses the iteratively converged Q table to obtain the strategy that maximizes the reward.

[0085] Example 2

[0086] This embodiment is based on embodiment 1:

[0087] This embodiment provides a multi-dimensional joint anti-interference method for wireless communication, which is applied to anti-interference communication. The process is as follows: Figure 2 As shown in Figure 2, time division multiple access is used. At the beginning of each time slot, the communication party senses the interference parameter and selects an action using Q-Learning based on the sensed interference parameter.

[0088] like Figure 3 As shown in Figure 1, the communication party makes online decisions based on historical information and current status, selects actions to be executed by the greedy strategy, obtains the benefits of different actions, and then updates the Q table and strategy. The goal of the communication anti-interference agent is to maximize the cumulative discounted reward at the current moment, that is, Where: γ∈(0,1] is a discount factor, which is used to weigh the impact of potential future rewards on the present. Q-Learning is to store Q values ​​by stretching states and actions into a two-dimensional Q table. The Q value function Q(s k ,a k ) means in state s k Next, perform action a k The value of can be expressed as:

[0089]

[0090] Combined with the Bellman equation, the expression of the state-action Q value function is:

[0091]

[0092] Finally, the agent uses the iteratively converged Q table to obtain the strategy with the maximum reward.

[0093] Specifically, the wireless communication multi-dimensional joint anti-interference method of this embodiment can be implemented by the following steps:

[0094] Step 1: When the transmitter of the communication party sends a signal to the target user, there is a malicious jammer in the communication process, which electromagnetically interferes with the channel access of the target user. Obtain the channel occupied by the interference through interference monitoring Interference power Interference Type and the noise power σ 2 Based on this information, the transmitter relies on Q-Learning to update and iterate, and selects the appropriate channel, modulation mode and transmission power to enable the received signal to meet the transmission rate and bit error rate requirements. Set the discount factor γ, learning rate α, noise power spectrum density N0, system total bandwidth B, required transmission rate R0, and required bit error rate Pe0.

[0095] Step 2, using Q-Learning, the state is defined as This includes the channel, power and modulation selected by the communication system, as well as the type of interference, interference channel and interference power detected. The action is defined as Including the channels, power and modulation methods that can be changed by the communication system. However, since there are only four interference modes, it would be very lengthy if it were performed in the same Q table, so four different Q tables can be constructed for the four different types of interference.

[0096] Step 3: Define the immediate reward.

[0097] When the agent completes the selected action in the current state, it will receive the corresponding immediate reward value R k , which can directly affect the update of Q value, thus guiding the agent to choose actions that can obtain high reward value in future decision-making, and then choose a good anti-interference communication method. Based on this, the design of the reward mechanism follows the following principles: 1. Encourage the selection of interference-free channels; 2. Encourage communication at a higher rate to reduce the possibility of interference; 3. If interfered, the rate needs to be reduced to ensure the bit error rate; 4. Considering the complexity of modulation and the fact that high power is easy to be discovered, it is encouraged to use low-order modulation and low power while ensuring the needs.

[0098] The immediate reward value of choosing an action is composed of the following five parts:

[0099] R=ω1f ber δ(x1)+ω2f rate δ(x2)+(ω3f power +ω4f modulation +ω5f channels )δ(x1)δ(x2)

[0100] Among them, f ber is the normalized bit error rate evaluation index, with a weight value of ω1 of 0.1 and f rate is the normalized transmission rate evaluation index, and its weight value ω2 is 0.1, f power is the normalized transmission power evaluation index, and its weight value ω3 is 0.2, f modulation is the normalized representation of the modulation mode evaluation index, and its weight value ω4 is 0.2, f channels It is the normalized representation of the channel selection evaluation index, and its weight value ω5 is 0.4.

[0101] δ(x) is the indicator function. When x is true, δ(x) is 1, otherwise it is 0. δ(x1) represents the bit error rate indicator function, that is, δ(x1) = δ(p e <p e0 ), where P e is the actual system bit error rate after selecting the action, P e0is the bit error rate threshold required by the system. δ(x2) represents the transmission rate indication function, that is, δ(x2) = δ(R0 < R < C), where R is the actual transmission rate, R0 is the transmission rate required by the system, and C is the channel capacity. Only when the selected action meets the bit error rate requirement can the first reward be obtained; only when the transmission rate requirement is met can the second reward be obtained; only when both the bit error rate and the transmission rate requirements are met simultaneously can the last three rewards be obtained.

[0102] Since the immediate reward uses the method of weighted summation, it is necessary to unify the dimensions of the above five items, and it is necessary to unify the evaluation index value interval to [0,1].

[0103] The result of bit error rate normalization is:

[0104]

[0105] In the formula, Pe max is the maximum bit error rate, that is, the lowest power is selected, and 64QAM is selected (if there is interference on the selected channel, the interference power is also the largest), Pe is the actually detected BER, Pe min is the case of the minimum bit error rate, that is, the highest power is selected, and the modulation method is BPSK.

[0106] The transmission rate normalization expression is as follows:

[0107]

[0108] Since R = baud rate (Baud) × modulation efficiency (bits / symbol). So R max is the rate when the selected modulation method is 64QAM; R min is the rate when the selected modulation method is BPSK.

[0109] The result of transmit power normalization is:

[0110]

[0111] In the formula, P is the current power. P min is the minimum power level, and its value is 1; P max is the maximum power level, and its value is 6.

[0112] The result of modulation method complexity normalization is:

[0113]

[0114] In the formula, M is the complexity of the current modulation method. The complexity values of the 4 modulation methods of BPSK, QPSK, 16QAM, and 64QAM are 1, 2, 4, and 6 in sequence.

[0115] The specific expression of channel selection normalization is:

[0116]

[0117] Where N is the total number of channel selections in the action, N j is the number of channels that are interfered with, which is to punish the selection of interference channels; max The maximum interval value for selecting channels, the numerator is the interval between channels, this is to encourage considering the continuity of frequency when selecting channels.

[0118] The corresponding bit error rate expressions for different modulation modes and interference modes are as follows.

[0119] In the non-interference mode, the bit error rate expression for BPSK and QPSK is:

[0120]

[0121] In the non-interference mode, the bit error rate expression of M-QAM is:

[0122]

[0123] In the narrowband noise interference and broadband noise interference modes, the corresponding bit error rate expression of the M-PSK modulation method is:

[0124]

[0125] In the narrowband noise interference and broadband noise interference modes, the corresponding bit error rate expression of M-QAM modulation is:

[0126]

[0127] Where erfc(x) is the complementary error probability function, E b is the unit bit energy, M is the modulation order, N0 is the noise power spectrum density, J0 is the narrowband and broadband noise interference power spectrum density, P j is the power of the interference signal, B j Interference signal bandwidth.

[0128] In single-tone interference mode, BPSK modulation mode, the corresponding bit error rate expression is:

[0129]

[0130] In single-tone interference mode, QPSK modulation mode, the corresponding bit error rate expression is:

[0131]

[0132] In single-tone interference mode, 16-QAM modulation mode, the corresponding bit error rate expression is:

[0133]

[0134] The expression of K is:

[0135]

[0136] like Figure 4 As shown in the constellation diagram of 16-QAM in the first quadrant, k1 is the projection in the positive direction of the imaginary axis, with 2 different amplitude values, so k1=2; similarly, k2 is the projection in the positive direction of the real axis, with 2 different amplitude values, so k2=2.

[0137] Where E b is the unit bit energy, M is the modulation order, N0 is the noise power spectrum density, J is the interference signal power, θ j0 is the initial phase of the single-tone interference, and θ j0 It follows a uniform distribution on [0,2π].

[0138] In single-tone interference mode, the 64-QAM modulation mode has a bit error rate expression similar to that of 16-QAM:

[0139]

[0140] However, k1=4, k2=4, and the modulation order is M=64.

[0141] Multi-tone interference can be regarded as a combination of several single-tone interferences on multiple channels.

[0142] In BSPK multi-tone interference mode, the bit error rate expression is as follows:

[0143]

[0144] In the QSPK multi-tone interference mode, the bit error rate expression is as follows:

[0145]

[0146] The bit error rate expression of M-QAM in multi-tone interference mode is as follows:

[0147]

[0148]

[0149] Where Δf i Indicates the frequency interval between each interference signal and the carrier signal in multi-tone interference.

[0150] Regarding the judgment of the transmission rate, from the formula of transmission rate (bps) = baud rate (Bauds) × modulation efficiency (bits / symbol), it can be seen that in the communication system, the transmission rate of the sender is related to the baud rate and modulation efficiency, and the baud rate is mainly affected by the bandwidth. The expression of modulation efficiency is: modulation efficiency (bits / symbol = log2M). The first condition for judging the transmission rate is that the product of the modulation efficiency determined by the modulation method and the baud rate should be greater than the required transmission rate R0. Secondly, in the actual communication process, in order to ensure the bit error rate, the actual transmission rate R0 needs to be less than the Shannon capacity C, In order to achieve reliable transmission under this bit error rate requirement.

[0151] After completing the calculation of the indicator function and the corresponding five basic reward values, the actual instant reward value R can be obtained.

[0152] Step 4: Update the Q value according to the following formula:

[0153]

[0154] where s k ,a k are the specific state in the current time slot and the action selected in the current time slot, respectively, and R k is the real-time reward value calculated in step 3; α and γ are the learning rate of Q-Learning and the discount rate for future rewards respectively. After completing the calculation of the current Q value and updating the corresponding Q value in the Q table, the interference in the environment is evaluated again, and it is combined with the action selected in the current time slot to form the state s at the next moment k+1 , in order to complete the iteration process of the Q table, until the iteration converges or reaches a specified number of times.

[0155] The anti-interference effect of this embodiment is as follows Figures 5 to 8 shown.

[0156] Figure 5 In the figure, the interference type is narrowband noise interference, the vertical axis is channel, and the horizontal axis is time. The anti-interference action includes the newly selected channel, modulation mode and power. Figure 5 In the case of anti-interference failure, that is, failure to meet the bit error rate and transmission rate requirements, the modulation mode and power level in the selected channel will be displayed in black characters + special form, and this embodiment uses a bold box to distinguish; and when anti-interference is successful, that is, when the communication requirements are met, the modulation mode and power level in the selected channel will be displayed only in black characters. Figure 5 It can be seen that when the communication equipment is interfered with, adjustments can be made quickly to avoid interference.

[0157] Figure 6In, the interference type is broadband noise interference; Figure 7 In, the interference type is single-tone interference; Figure 8 In the example, the interference type is multi-tone interference.

[0158] Example 3

[0159] This embodiment provides a wireless communication multi-dimensional joint anti-interference system, including:

[0160] An anti-interference decision conversion module is configured to model the anti-interference decision required by the communication party in the interference environment as a Markov decision process and perform interference countermeasure based on a Q-Learning algorithm;

[0161] The state and action determination module is configured to determine the state and action of the Markov decision process, obtain the state space S and the action space a, and the corresponding state-action matrix, i.e., the Q table, and initialize them;

[0162] A reward function determination module is configured to determine the instantaneous reward function of the Markov decision process, calculate the instantaneous reward value corresponding to the action, and use it as a key basis for updating the corresponding Q value in the Q table;

[0163] The anti-interference decision optimization module is configured to input the interference mode and parameters perceived in the interference environment and the initial state of the current communication party, and select the action in the current state based on the greedy strategy algorithm, and obtain the corresponding reward value R after executing the action. k , record and update the Q value in the Q table, and transfer to the next state s k+1 ;

[0164] The anti-interference result output module is configured to convert the action-value function Q(s k ,a k ) Iterate until convergence or iterate to a specified number of times to obtain the final Q table and output the best anti-interference result of the transmitter

[0165] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity of description, they are expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

Claims

1. A multi-dimensional joint anti-interference method for wireless communication, characterized in that: include: In an interference environment, the anti-interference decision required by the communication party is modeled as a Markov decision process, and interference countermeasures are performed based on the Q-Learning algorithm; Determine the state and action of the Markov decision process, obtain the state space S and action space a, and the corresponding state-action matrix, i.e., Q table, and initialize them; Determine the immediate reward function of the Markov decision process, calculate the immediate reward value corresponding to the action, and use it as the key basis for updating the corresponding Q value in the Q table; Input the interference pattern and parameters perceived in the interference environment and the initial state of the current communication party, select the action in the current state based on the greedy strategy algorithm, and obtain the corresponding reward value R after executing the action k , record and update the Q value in the Q table, and transfer to the next state s k+1 ; The action-value function Q(s k ,a k ) Iterate until convergence or iterate to a specified number of times to obtain the final Q table and output the best anti-interference result of the transmitter 2. A wireless communication multi-dimensional joint anti-interference method according to claim 1, characterized in that: In determining the state and action of the Markov decision process, the state to be determined includes the communication mode of the communication party and the interference mode of the interference party. in is the communication channel, transmission power and modulation mode in the current k time slot, is the channel, power and interference type of the interference signal in the current k time slot.

3. A wireless communication multi-dimensional joint anti-interference method according to claim 2, characterized in that: In determining the state and action of the Markov decision process, the communication party makes an anti-interference decision. In the current k time slots, the action selection can only change the communication mode. That is, changing the communication channel, transmission power and modulation method.

4. A wireless communication multi-dimensional joint anti-interference method according to claim 1, characterized in that: Determining the instantaneous reward function of the Markov decision process and calculating the instantaneous reward value corresponding to the action include: When the selected action reaches the transmission rate required by the communication system, the corresponding transmission rate reward will be obtained; When the selected action reaches the bit error rate required by the communication system, the corresponding bit error rate reward is obtained; When the selected action satisfies the above two conditions at the same time, the corresponding channel selection reward, power selection reward and modulation mode selection reward are obtained.

5. The multi-dimensional joint anti-interference method for wireless communication according to claim 1, characterized in that: The instantaneous reward function of the Markov decision process includes: R=ω1f ber δ(x1)+ω2f rate δ(x2)+(ω3f power +ω4f modulation +ω5f channels )δ(x1)δ(x2) Among them, R is the immediate reward function, f ber is the normalized bit error rate evaluation index, f rate is the normalized transmission rate evaluation index, f power is the normalized power index, f modulation is the normalized modulation index, f channels is the normalized channel combination index; ω1~ω5 are weight coefficients; δ(x1) is the bit error rate indicator function, and δ(x2) is the transmission rate indicator function.

6. A wireless communication multi-dimensional joint anti-interference method according to claim 1, characterized in that: The greedy strategy algorithm is used to select the action in the current state, including: Using a greedy strategy, the transmitter selects an action with probability 1-ε And other actions are randomly selected with probability ε, so the probability of selecting an action is expressed as:

7. A wireless communication multi-dimensional joint anti-interference method according to claim 1, characterized in that: The interference pattern and parameters perceived in the input interference environment and the initial state of the current communication party are used to select an action in the current state based on a greedy strategy algorithm, including: using a time division multiple access method, at the beginning of each time slot, the communication party perceives the interference parameters, and using a Q-Learning algorithm to select an action based on the perceived interference parameters.

8. The wireless communication multi-dimensional joint anti-interference method according to claim 1, characterized in that: The greedy strategy algorithm selects the action in the current state and obtains the corresponding reward value R after executing the action. k , record and update the Q value in the Q table, including: The communication party makes online decisions based on historical information and current status, selects execution actions based on the greedy strategy, obtains the benefits of different actions, and then updates the Q table; the communication party's goal is to maximize the cumulative discounted reward at the current moment: Where γ∈(0,1] is a discount factor used to weigh the impact of potential future rewards on the present.

9. A wireless communication multi-dimensional joint anti-interference method according to claim 8, characterized in that: The function Q(s) corresponding to the Q value k ,a k ) means in state s k Next, perform action a k The value of: Among them, γ i r (k+i+1) is a series of discounted future reward values, s and a are the state space set and action space set respectively; Combined with the Bellman equation, the expression of the action-value function is obtained as follows: Among them, s k+1 Indicates the state of the next time slot, a k+1 Indicates the action that will be selected in the next time slot.

10. A wireless communication multi-dimensional joint anti-interference system, characterized in that: include: An anti-interference decision conversion module is configured to model the anti-interference decision required by the communication party in the interference environment as a Markov decision process and perform interference countermeasure based on a Q-Learning algorithm; The state and action determination module is configured to determine the state and action of the Markov decision process, obtain the state space S and the action space a, and the corresponding state-action matrix, i.e., the Q table, and initialize them; A reward function determination module is configured to determine the instantaneous reward function of the Markov decision process, calculate the instantaneous reward value corresponding to the action, and use it as a key basis for updating the corresponding Q value in the Q table; The anti-interference decision optimization module is configured to input the interference mode and parameters perceived in the interference environment and the initial state of the current communication party, and select the action in the current state based on the greedy strategy algorithm, and obtain the corresponding reward value R after executing the action. k , record and update the Q value in the Q table, and transfer to the next state s k+1 ; The anti-interference result output module is configured to convert the action-value function Q(s k ,a k ) Iterate until convergence or iterate to a specified number of times to obtain the final Q table and output the best anti-interference result of the transmitter

Images