A dynamic spreading factor anti-interference method based on annealing Q-learning

Abstract

The present application provides a dynamic spreading factor anti-interference method based on annealing Q learning, the method comprising: determining the information transmission rate according to the spreading factor; determining the system throughput and optimization target according to the signal-to-noise ratio of the receiving end; determining the state space, action space, state transition probability matrix and reward in the decision-making process; calculating the reward obtained by selecting the action according to the broadband spectrum sensing result; obtaining the interference situation at the current moment through broadband spectrum sensing; deciding the spreading factor of the next time slot according to the Q value table; returning the decision information according to ACK: the user selects the action of the next time slot according to the decision information and updates the state: calculating the reward obtained by selecting the corresponding action according to the broadband spectrum sensing result; selecting the spreading factor of the next time slot according to the Q value table, and updating the Q value of the action selected in the state; the user makes the most anti-interference strategy in a dynamic interference environment. The present application maximizes the system throughput and avoids the delay and overhead caused by channel switching.

Description

Technical Field

[0001] The present application relates to the field of wireless communication technology, and in particular to a dynamic spreading factor anti-interference method based on annealing Q learning. Background Art

[0002] Due to the open sharing of wireless communication networks and the fact that the communication medium is a free electromagnetic space, it is very easy to be suppressed by interference or even have communication interrupted. At present, the anti-interference technology of wireless communication networks is mainly divided into interference avoidance technology, interference elimination technology and hard interference resistance technology, such as switching channels, direct sequence spread spectrum, adaptive filtering, etc. to achieve anti-interference. The above methods can achieve the effect of anti-interference in specific scenarios. Among them, direct sequence spread spectrum technology, as one of the hard anti-interference technologies, can achieve anti-interference without switching channels, saving the delay and overhead caused by re-establishing channel links, and is widely used in various communication scenarios. However, the parameters of traditional direct sequence spread spectrum communication technology are mostly set in advance, the anti-interference mode considered is relatively fixed, and the intelligence in communication decision-making is not strong. There is a lack of mining and research on the available spectrum in the outside world, and it is difficult to deal with dynamic interference in complex electromagnetic spectrum environments.

[0003] Therefore, studying a dynamic spreading factor anti-interference strategy that can dynamically adjust the spreading factor according to the interference intensity when channel resources are tight to optimize the anti-interference strategy, select a larger spreading factor when the interference intensity is large to reduce the information transmission rate; select a smaller spreading factor when the interference intensity is small to increase the information transmission rate, has become a current research hotspot. Summary of the invention

[0004] The present application provides an anti-interference method for a dynamic spreading factor based on annealing Q learning, which can be used to solve the technical problem that the anti-interference strategy cannot be flexibly changed when channel resources are tight.

[0005] Step 1: Determine the information transmission rate according to the spreading factor.

[0006] Each transmission time slot Communication time , spectrum sensing time , Q value table update time and confirmation signal ACK transmission time ; In the first communication slot The user selects the spreading factor of the first time slot to transmit data as , the information transmission rate of the first time slot is ; When the chip rate When a certain time is given, the information transmission rate of the kth time slot is Chip rate The relationship between them is:

[0007] (1);

[0008] Where k is the time slot number, which does not exceed the total number of time slots K; is the spreading factor of the kth time slot.

[0009] Step 2: Determine the system throughput and optimization target based on the signal-to-noise ratio at the receiving end.

[0010] In the current communication scenario, the communication user's channel is fixed, the bandwidth is unchanged, the chip rate is unchanged, and the power is constant. Different interferences are dealt with by adjusting M kinds of spreading factors; the signal-to-interference-to-noise ratio at the receiver input is used. To determine the degree of interference, It is expressed as follows:

[0011] (2);

[0012] in, is the transmitter power, is the noise power spectral density at the receiver, is the channel bandwidth, is the interference power of the jammer. Assuming that the channel experiences Rayleigh fading, the channel model of the communication user is expressed as ,in, is the transmission link distance between communication users, is the path loss coefficient of the link, is the instantaneous random component of the user link; the channel model of the jammer is expressed as ,in, is the transmission link distance from the jammer to the communication user, is the path loss coefficient of the link, is the instantaneous random component of the jammer-to-user link;

[0013] In order to ensure the normal operation of the receiver, without considering the antenna loss, the signal-to-interference-to-noise ratio at the receiver input must be greater than the threshold value. :

[0014] (3);

[0015] For fixed channel communication, the interference effect is equivalent to an interference model whose intensity changes over time. Interference intensity Unchanged, interference intensity at different time slots Change occurs;

[0016] If the input signal-to-interference-to-noise ratio caused by interference is lower than , the receiving end fails to receive the communication data. Define the mth spreading factor selected for the current k time slot The effective interference function is:

[0017] (4);

[0018] Then the system throughput of the kth time slot is defined as:

[0019] (5);

[0020] in, To select the spreading factor The probability of M is the total number of spreading factors that can be implemented;

[0021] From the perspective of Q learning, the set of all feasible strategies is , given k time slots, the spreading factor selection strategy ,but The strategy is selected by the history before the kth time slot and the throughput history value before the kth time slot Determine and optimize your goals The goal is to find the optimal spreading factor selection strategy to maximize the cumulative expected throughput:

[0022] (6).

[0023] Step 3: Determine the state space, action space, state transition probability matrix, and reward in the decision-making process.

[0024] In spectrum sensing time The user obtains the channel quality of the first time slot through broadband spectrum sensing and obtains the interference intensity , the corresponding user's complete status is ;

[0025] The anti-interference decision process is modeled as a Markov decision process MDP, and the optimal anti-interference strategy is found by solving the MDP; the MDP is defined as a four-tuple , where S represents the state space, A represents the action space, P represents the state transition probability matrix, and r represents the reward, which is defined as follows:

[0026] Defining the state space :

[0027] : , , indicating the The state of the user in each time slot, represents the spreading factor of the kth time slot, represents the interference intensity of the kth time slot;

[0028] Defining the action space :

[0029] , among which The decision action made in the time slot is , i.e. the spreading factor of the next time slot;

[0030] Define the state transition probability matrix :

[0031] , ,in Indicates in status Next select action After transfer to state probability;

[0032] Defining Rewards :

[0033] Indicates that the kth time slot is in state Next select action Rewards received;

[0034] (7);

[0035] in, As the maximum rate among all available information transmission rates, introduce It is to normalize the information transmission rate of the current time slot. is the reward-penalty factor, defined as follows:

[0036] (8);

[0037] That is, if the action selected in the kth time slot can successfully resist interference, a positive reward factor is obtained, otherwise a negative penalty factor is obtained.

[0038] Step 4: Calculate the reward for the selected action based on the broadband spectrum sensing result; the user selects the spreading factor for the first time slot based on the sensing result. Rewards .

[0039] Step 5: Obtain the interference situation of the next time slot through broadband spectrum sensing .

[0040] Step 6, obtain the spreading factor of the next time slot according to the Q value table; Q value table update time The user selects the spreading factor of the next time slot according to the Q value table. .

[0041] In the process of selecting actions, the agent selects actions from the Boltzmann distribution corresponding to the learned Q value. The probability of selecting the mth action in the kth time slot is:

[0042] (9);

[0043] Where, M is the number of selectable spreading factors; is the Boltzmann temperature constant;

[0044] The exploration mechanism of the agent choosing action a when the state is S is Defined as:

[0045] (10);

[0046] Where r is a randomly generated number between 0 and 1; They are the 1st, 2nd…Mth actions respectively;

[0047] is the Boltzmann temperature constant, which controls the heat of the action selection probability; when When it approaches 0, the agent tends to greedily choose the strategy, that is, always choose the action with the highest Q value; when As it approaches infinity, the agent tends to a completely random strategy, that is, it explores the environment completely randomly; in order to enable the agent to discover better strategies more in the early stages and make more use of the known best strategies in the later stages of learning, It should decrease gradually with the iteration of learning, so the Boltzmann temperature constant is adjusted by linear annealing. To update:

[0048] (11);

[0049] in, is the initial temperature parameter, is the minimum temperature parameter, is the number of learning times in the kth time slot, is the total number of learning times.

[0050] Step 7: Decision information based on ACK return: Last ACK transmission time Within the time, the receiver transmits the decision information back to the transmitter in the form of ACK to coordinate the spreading factor of the next time slot.

[0051] Step 8: The user selects the action for the next time slot based on the decision information and updates the status: In the second time slot, the user selects the action for the next time slot based on the decision information sent back. To transfer data, the user status is updated to .

[0052] Step 9: Calculate the reward for selecting the corresponding action based on the broadband spectrum sensing result; the user obtains the spreading factor for the second time slot based on the sensing result. Rewards .

[0053] Step 10: Select the spreading factor for the next time slot according to the Q value table and update the status Next select action Q value.

[0054] Continue to select the spreading factor through Q learning and update the state as follows Next select action Q value:

[0055] (12);

[0056] in, Represent the state and action of the kth time slot respectively, is the learning rate, represents the discount factor, Represents the state Execute actions when Rewards received, is the Q value corresponding to the state and action of the kth time slot, is the updated Q value of the k+1th time slot, represents the maximum value of all Q values ​​of the state in the k+1th time slot, that is , The action corresponding to the maximum value of all Q values ​​of the state in the k+1th time slot.

[0057] Step 11, update the Q value in different states according to Q learning iteration. The communication user repeats steps 1 to 10 to update the Q value in different states. The intelligent agent's cognition of the environment is strengthened through cyclic execution. Finally, the Q value table tends to be stable, and the user makes the most anti-interference strategy in a dynamic interference environment.

[0058] The implementation process of this application is as follows:

[0059] Step A1, initialization, includes:

[0060] Step A11, set parameters: learning rate , discount rate , initial temperature parameter , minimum temperature parameter ;

[0061] Step A12, the total number of simulation time slots is K, , set the initial time slot ,in ;

[0062] Step A13, initializing the Q value table to an all-zero matrix;

[0063] Step A14, the initial working state is ,in The spreading factor selected for the first time slot, The channel interference situation obtained by broadband spectrum sensing in the first time slot;

[0064] Step A2, when k≤K, loop;

[0065] Step A21: The kth time slot uses a spreading factor Transmit data and obtain the current information transmission rate ;

[0066] Step A22, obtaining the interference intensity of the kth time slot through broadband spectrum sensing , calculated using Rewards ;

[0067] Step A23, updating the spreading factor selection probability vector according to formula (10) , and according to Explore the selection action, i.e. the spreading factor of the next time slot ;

[0068] Step A24, send back ACK information and coordinate the transmission spreading factor of the next time slot ;

[0069] Step A25, updating the Q value table; .

[0070] The present invention solves the problem of insufficient dynamic parameter adjustment capability of traditional direct sequence spread spectrum technology in unknown and dynamically changing interference scenarios. Different from most strategies that avoid interference, the proposed method can solve dynamic anti-interference strategies through annealing Q learning when channel resources are tight. Under the premise of ensuring reliable communication of fixed channels, the system throughput is maximized, and the delay and overhead caused by channel switching are avoided. The simulation verifies that the throughput performance of this method is better than the random spread factor selection strategy based on spectrum perception and the fixed spread factor strategy. And compared with traditional Q learning, the annealing Q learning algorithm proposed in the present invention introduces a linear annealing operation to the Boltzmann temperature parameter, ensuring that the intelligent agent can fully interact with the environment in the early stage of learning, and is more inclined to use learning experience in the later stage of learning, which better balances the relationship between environmental exploration and experience utilization, avoids Q learning from falling into local optimality, and accelerates the convergence speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 A system scenario diagram provided for an embodiment of the present application;

[0072] Figure 2 A diagram of the Q learning time slot structure provided in an embodiment of the present application;

[0073] Figure 3 Anti-interference performance diagram of different spreading factors provided in the embodiment of the present application;

[0074] Figure 4 The Q value variation curves at different intensities provided in the embodiments of the present application;

[0075] Figure 5 The throughput variation curves of different strategies provided in the embodiments of the present application. DETAILED DESCRIPTION

[0076] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0077] The following first introduces the embodiments of the present application in conjunction with the accompanying drawings.

[0078] Figure 1 This is the system scenario diagram of the present invention. There is a pair of communication users and a jammer in the system. The jammer can generate N=3 interference signals of different strengths. The interference time slot length is , the communication user's channel is fixed, the bandwidth is unchanged, the power is constant, and different interferences are dealt with by adjusting M=3 kinds of spreading factors. The communicating parties will transmit time slots As the smallest transmission time unit, the spreading factor is adjusted at the beginning of each time slot, and then a length.

[0079] Figure 2 This is the Q learning time slot structure diagram of the algorithm proposed in the present invention. In this scenario, the jammer can generate a narrowband interference signal with a strength that changes over time. The Q learning implementation process within each time slot is as follows: Figure 2 The user transmission time slot is ,The receiver’s internal agent executes the Q-learning algorithm, and the communicating user in each transmission time slot ,selects and adjusts the spreading factor according to the Q-learning result.

[0080] Figure 3 The anti-interference performance curves of three spreading factors SF={16, 32, 64} obtained by simulation are given. Under the premise of , the corresponding anti-interference tolerance is set according to the simulation results. ={1dB,7dB,11dB}.

[0081] Figure 4 The Q value change curves under different intensities are given. In the initial stage of learning, the user's Q value for each spreading factor is 0. With continuous interactive learning with the environment, the Q value is constantly updated. The user will eventually select a spreading factor of SF=16 when the interference intensity is 9dBm; a spreading factor of SF=32 when the interference intensity is 15dBm; and a spreading factor of SF=64 when the interference intensity is 19dBm.

[0082] Figure 5 The throughput performance change curves of the proposed Q-learning-based spreading factor selection algorithm, the perception-based random spreading factor selection algorithm, the fixed strategy based on the spreading factor of 64, the fixed strategy based on the spreading factor of 32, and the fixed strategy based on the spreading factor of 16 are given. In order to more conveniently and intuitively reflect the throughput performance differences of different strategies, Figure 5 The throughput value of each point in is calculated by the average value of every 100 consecutive time slots. From the simulation diagram, it can be seen that the system throughput based on the proposed Q learning algorithm is about 70kbps, which is about 28kbps higher than the random spreading factor selection algorithm based on perception, and about 32kbps, 48kbps, and 68kbps higher than the throughput of the three fixed spreading factor strategies. Among them, compared with the random spreading factor selection algorithm based on perception, the performance of this method is improved by about 66.7%.

[0083] The above-described embodiments of the present application do not constitute a limitation on the protection scope of the present application.

Claims

1. A dynamic spreading factor anti-interference method based on annealing Q learning, characterized in that: The method comprises: Step 1, determining the information transmission rate according to the spreading factor; Step 2, determining the system throughput and optimization target according to the signal-to-noise ratio at the receiving end; Step 3, determine the state space, action space, state transition probability matrix and reward in the decision-making process; Step 4: Calculate the reward for the selected action based on the broadband spectrum sensing result; the user selects the spreading factor for the first time slot based on the sensing result. Rewards ; Step 5: Obtain the interference situation at the current moment through broadband spectrum sensing ; Step 6, obtain the spreading factor of the next time slot according to the Q value table; Q value table update time The user selects the spreading factor of the next time slot according to the Q value table. ; Step 7: Decision information based on ACK return: Last ACK transmission time Within the time, the receiver transmits the decision information back to the transmitter in the form of ACK to coordinate the spreading factor of the next time slot; Step 8: The user selects the action for the next time slot based on the decision information and updates the status: In the second time slot, the user selects the action for the next time slot based on the decision information sent back. To transfer data, the user status is updated to ; Step 9: Calculate the reward for selecting the corresponding action based on the broadband spectrum sensing result; the user obtains the spreading factor for the second time slot based on the sensing result. Rewards ; Step 10: Select the spreading factor for the next time slot according to the Q value table and update the status Next select action Q value; Step 11, iteratively update the Q value in different states according to Q learning; the communication user repeats steps 1 to 10 to update the Q value in different states, and strengthens the agent's cognition of the environment through cyclic execution. Finally, the Q value table tends to be stable, and the user makes the most anti-interference strategy in a dynamic interference environment.

2. The method according to claim 1, characterized in that Step 1, determining the information transmission rate according to the spreading factor, includes: Each transmission time slot Communication time , spectrum sensing time , Q value table update time And the confirmation signal ACK transmission time ; In the first communication slot The user selects the spreading factor of the first time slot to transmit data as , the information transmission rate of the first time slot is ; When the chip rate When a certain time is given, the information transmission rate of the kth time slot is Chip rate The relationship between them is: （1）； Where k is the time slot number, which does not exceed the total number of time slots K; is the spreading factor of the kth time slot.

3. The method according to claim 2, characterized in that Step 2, determining the system throughput and optimization target according to the receiving end signal-to-noise ratio, including: In the current communication scenario, the communication user's channel is fixed, the bandwidth is unchanged, the chip rate is unchanged, and the power is constant. Different interferences are dealt with by adjusting M kinds of spreading factors; the signal-to-interference-to-noise ratio at the receiver input is used. To determine the degree of interference, It is expressed as follows: （2）； in, is the transmitter power, is the noise power spectral density at the receiver, is the channel bandwidth, is the interference power of the jammer. Assuming that the channel experiences Rayleigh fading, the channel model of the communication user is expressed as ,in, is the transmission link distance between communication users, is the path loss coefficient of the link, is the instantaneous random component of the user link; the channel model of the jammer is expressed as ,in, is the transmission link distance from the jammer to the communication user, is the path loss coefficient of the link, is the instantaneous random component of the jammer-to-user link; In order to ensure the normal operation of the receiver, without considering the antenna loss, the signal-to-interference-to-noise ratio at the receiver input must be greater than the threshold value. : （3）； For fixed channel communication, the interference effect is equivalent to an interference model whose intensity changes over time. Interference intensity Unchanged, interference intensity in different time slots Change occurs; Define the mth spreading factor selected for the current k time slot The effective interference function is: （4）； Then the system throughput of the kth time slot is defined as: （5）； in, To select the spreading factor The probability of M is the total number of spreading factors that can be implemented; From the perspective of Q learning, the set of all feasible strategies is , given k time slots, the spreading factor selection strategy ,but The strategy is selected by the history before the kth time slot and the throughput history value before the kth time slot Determine and optimize your goals The goal is to find the optimal spreading factor selection strategy to maximize the cumulative expected throughput: （6）。 4. The method according to claim 3, characterized in that Step 3: Determine the state space, action space, state transition probability matrix and reward in the decision-making process, including: In spectrum sensing time The user obtains the channel quality of the first time slot through broadband spectrum sensing and obtains the interference intensity , the corresponding user's complete status is ; The anti-interference decision process is modeled as a Markov decision process MDP, and the optimal anti-interference strategy is found by solving the MDP; the MDP is defined as a four-tuple , where S represents the state space, A represents the action space, P represents the state transition probability matrix, and r represents the reward, which is defined as follows: Defining the state space : : , , indicating the The state of the user in each time slot, represents the spreading factor of the kth time slot, represents the interference intensity of the kth time slot; Defining the action space : , among which The decision action made in the time slot is , i.e. the spreading factor of the next time slot; Define the state transition probability matrix : , ,in Indicates in status Next select action After transfer to state probability; Defining Rewards : Indicates that the kth time slot is in state Next select action Rewards received; （7）； in, As the maximum rate among all available information transmission rates, introduce It is to normalize the information transmission rate of the current time slot. is the reward-penalty factor, defined as follows: （8）； That is, if the action selected in the kth time slot can successfully resist interference, a positive reward factor is obtained, otherwise a negative penalty factor is obtained.

5. The method according to claim 4, characterized in that Step 6, obtain the spreading factor of the next time slot according to the Q value table; Q value table update time The user selects the spreading factor of the next time slot according to the Q value table. ,include: In the process of selecting actions, the agent selects actions from the Boltzmann distribution corresponding to the learned Q value. The probability of selecting the mth action in the kth time slot is: （9）； Where, M is the number of selectable spreading factors; is the Boltzmann temperature constant; The exploration mechanism of the agent choosing action a when the state is S is Defined as: （10）； Where r is a randomly generated number between 0 and 1; They are the 1st, 2nd…Mth actions respectively; is the Boltzmann temperature constant, which controls the heat of the action selection probability; when When it approaches 0, the agent tends to greedily choose the strategy, that is, always choose the action with the highest Q value; when When it tends to infinity, the agent tends to a completely random strategy, that is, it explores the environment completely randomly; the Boltzmann temperature constant is adjusted by linear annealing. To update: （11）； in, is the initial temperature parameter, is the minimum temperature parameter, is the number of learning times in the kth time slot, is the total number of learning times.

6. The method according to claim 5, characterized in that Step 10: Select the spreading factor for the next time slot according to the Q value table and update the status Next select action Q values ​​include: Continue to select the spreading factor through Q learning and update the state as follows Next select action Q value: （12）； in, Represent the state and action of the kth time slot respectively, is the learning rate, represents the discount factor, Represents the state Execute action when Rewards received, is the Q value corresponding to the state and action of the kth time slot, is the updated Q value of the k+1th time slot, represents the maximum value of all Q values ​​of the state in the k+1th time slot, that is , The action corresponding to the maximum value of all Q values ​​of the state in the k+1th time slot.

Images