Adaptive modulation coding method based on LSTM-DQN

Abstract

The invention discloses an adaptive modulation and coding method based on LSTM-DQN, and belongs to the field of meteor trail communication. Aiming at the problem of insufficient timeliness of channel state information caused by long transmission time delay of a meteoric trail communication channel and the problem that performance reduction of a traditional adaptive modulation coding algorithm in a fast time-varying channel is limited, the invention provides a method for carrying out strategy selection by combining an LSTM (Long Short Term Memory) and a DQN (Digital Quality Network) network so as to improve the throughput of a system. The method comprises the following steps that firstly, a network framework of LSTM and DQN is initialized, the transmission condition of each step is obtained through a signal-to-noise ratio estimation algorithm and error detection codes to serve as neural network training data, and then the LSTM is responsible for conducting time sequence prediction to improve the problem that channel state information is expired; the DQN network uses empirical playback and a target network to train an adaptive modulation coding policy network. Through continuous iteration of network parameters, selection of modulation and coding scheme strategies under different channel states is optimized, and then a modulation and coding scheme which is more suitable for the current state of a channel is obtained.

Description

Technical Field

[0001] The present invention relates to the field of meteor trail communication, and in particular to a method for selecting a coding modulation scheme by using an LSTM-DQN network in a meteor trail communication process. Background Art

[0002] In scenarios such as meteor trail communication (MBC), satellite communication, and underwater communication, the transmitter needs to understand the current channel status through feedback signals from the receiver, and further optimize its own coding and modulation method to adapt to the channel and maximize throughput. This strategy is called adaptive coding and modulation (AMC).

[0003] However, due to the long communication distance and other reasons, the feedback signals in the above communication process need to go through a certain delay to reach the transmitter. Since the channel of the above communication system changes rapidly, the delay is often greater than the channel coherence time, which means that the feedback signal obtained by the transmitter is likely to be outdated, which will lead to wrong judgments. Therefore, it is necessary to process the outdated signal in terms of timing to get a closer to the real state of the channel at the current moment.

[0004] At the same time, the traditional adaptive coding modulation algorithm obtains the modulation and coding scheme to be adopted next by looking up the modulation and coding strategy (MCS) table. However, this fixed table lookup method has limitations in fast time-varying channels. Taking meteor trail communication as an example, the channel bursts appear, the scattered signal reaches a peak in a very short time and then decays exponentially, and the signal-to-noise ratio fluctuates greatly. At this time, if the MCS obtained under ideal conditions is still used for table lookup, the strategy obtained will be a suboptimal solution. Therefore, a modulation and coding strategy selection algorithm that can better adapt to fast time-varying channels is needed. Summary of the invention

[0005] The present invention provides an adaptive modulation and coding method of LSTM-DQN for meteor trail communication. In view of the fast time-varying characteristics of MBC channels, the present invention proposes to first use a long short-term memory model (LSTM) to perform time series processing on the signal-to-noise ratio information in the feedback signal, and then use deep reinforcement learning (DRL) to train the modulation and coding strategy. The model obtained by this method can adapt to the MBC channel well, and further improves the system throughput on the basis of traditional AMC.

[0006] The technical solution of the present invention to solve the above problems is as follows:

[0007] An adaptive modulation coding method based on LSTM-DQN includes the following steps:

[0008] Step 1. Create LSTM and deep Q learning (DQN) network framework on the sending end, including input layer, hidden layer, output layer, and initialize network parameters;

[0009] Step 2. Initial stage signal transmission: Since the training of the neural network requires a certain number of input parameters, before the number of feedback signals reaches the threshold, the transmitter processes the pre-transmitted bit stream data according to the initial coding and modulation scheme and sends the generated signal to the receiver;

[0010] Step 3. After receiving the signal, the receiving end demodulates and decodes the signal in a corresponding manner according to the identifier in the frame header information, selects to feedback ACK or NAK according to the verification result of the CRC code, and estimates the bit error rate (BER) according to the pilot signal, and transmits it to the sending end as a feedback signal;

[0011] Step 4. After receiving the feedback signal, the transmitter first estimates the signal-to-noise ratio, obtains the current signal-to-noise ratio as the basis for selecting the next action, and determines whether the data in the experience pool is sufficient. If the data is insufficient, return to step 2 and continue to send repeatedly until the data reaches the threshold requirement, otherwise proceed to the next step;

[0012] Step 5. After the amount of data in the experience pool at the transmitter reaches the preset amount requirement, start training the network model. First, train LSTM to reduce the impact of transmission delay, input n-step SNR data, predict the SNR at the next moment, and after obtaining the SNR of the actual received signal at the next moment, use the gradient descent method to optimize the network;

[0013] Step 6. After passing through the LSTM network, the corrected SNR is obtained as the input state s of reinforcement learning. According to s, the modulation coding mode, i.e., action a, is selected. The reward r is obtained according to the feedback signal. The SNR of the next received signal is taken as s'. Done is set according to whether the model converges. The above data are combined into the experience tuple (s, a, r, s', done).

[0014] Step 7. Add the experience element to the experience pool. When training DQN, use the experience replay algorithm to randomly sample data from the experience pool as the input data of the network. After obtaining the estimated values ​​of the Q network and the target network, calculate the mean square error (MSE), and minimize the loss function through gradient descent to optimize the DQN network.

[0015] Step 8. Whenever the transmitter receives a feedback signal, it adds the experience to the experience pool and updates the network parameters. Repeat the above network training steps until the model converges, and the obtained model can better adapt to fast time-varying channels.

[0016] The design principle of the present invention is to first determine the number of modulation and coding schemes according to the transmission requirements, and after the transmitting end receives the feedback signal from the receiving end, the signal-to-noise ratio of the channel at the current moment and the transmission status at the previous moment are obtained according to the feedback signal in the meteor trail communication system, which is used as the training data of the LSTM and DQN networks, and then train a coding and modulation strategy that can better adapt to fast time-varying channels. This method comprehensively considers the impact of the channel state information (CSI) delay and the channel characteristics of fast time-varying channels that are different from general channels. The obtained system throughput is higher than the traditional AMC algorithm based on table lookup, and the performance is better.

[0017] The specific operation method for determining the number of modulation and coding schemes is as follows: comprehensively consider the block error rate and throughput of the system, and select different coding and modulation schemes to be used:

[0018] {(M1,R1),(M2,R2),…,(M n-1 ,R n-1 )}

[0019] where M1,…,M n-1 For different modulation modes, R1,…,R n-1 Different schemes have different requirements for channel conditions for different code rates. For example, the following modulation and coding scheme set is pre-set as the expected output strategy of the DQN network:

[0020] {(BPSK,1 / 2),(BPSK,3 / 4),…,(16QAM,1 / 2),(16QAM,3 / 4)}

[0021] The specific operation method of measuring the signal-to-noise ratio at the transmitter is: add a pilot signal to the head of the feedback signal, use the pilot-assisted least squares (LS) signal-to-noise ratio estimation algorithm, and estimate the signal-to-noise ratio The signal-to-noise ratio information with delay is obtained as the input for the next LSTM training.

[0022] Y k is the received signal in the frequency domain, X k is the pilot signal in the frequency domain, is the ensemble mean, (·) * is the complex conjugate operation, is the cross-correlation function,

[0023] The specific operation method of obtaining the transmission status of the previous moment according to the feedback signal is as follows: the receiving end first estimates the bit error rate according to the pilot signal, where the bit error rate calculation formula is: The receiver selects ACK or NAK based on the verification result of the CRC code, and transmits the above information to the transmitter as a feedback signal. The transmitter obtains the transmission status of the previous moment based on the feedback signal as the reward parameter r of reinforcement learning, where r is calculated as:

[0024]

[0025] Among them, R and M are the code rate and modulation order selected for this transmission respectively, and the reward r obtained is regarded as the transmission situation at the previous moment, which serves as the input data of reinforcement learning.

[0026] The specific operation method of using LSTM for time series prediction is as follows: first preprocess the data, split the input sequence into multiple sliding windows, and predict the signal-to-noise ratio of the next time based on the past n time steps each time the model is trained. Then, use the memory unit and gating mechanism to build the network framework, and calculate the MSE of the output value and the true target to obtain the loss function L, calculate the gradient through the back propagation algorithm and update the network weight. The calculation formula of the loss function is:

[0027]

[0028] The specific operation method of using the DQN network to select the modulation and coding strategy is as follows: the signal-to-noise ratio obtained by the LSTM network is used as the input state s of the deep reinforcement learning network, and after passing through DQN, the modulation and coding method to be taken next is obtained, that is, the action a in reinforcement learning, thereby realizing the adaptive selection of the modulation and coding strategy. During the training process of the DQN network, (s, a, r, s', done) is used as a set of experience data, and a set of experience data is added to the experience pool each time it is transmitted. Through experience playback, a batch of data is randomly sampled from the experience pool for training at each training step, making full use of the data and improving the training effect. The optimization of the Q network uses the Bellman optimal equation, and the update formula of DQN is:

[0029]

[0030] Where r is the immediate reward, s' is the next state, γ is the discount factor, and θ - are the parameters of the target network. Then the loss function is calculated, the gradient descent algorithm is used to minimize the loss function, and the weights of the network are updated. The loss function is:

[0031]

[0032] Where Q(s t ,a t ; θ) is the current Q value estimated by the Q network. At the same time, the weights of the Q network are periodically copied to the target network.

[0033] The advantage of the present invention is that for fast time-varying channels such as meteor trails, taking into account the hysteresis of CSI in channel conditions and the fast time-varying characteristics of the channel, an adaptive modulation coding algorithm based on LSTM-DQN is designed. The method performs timing prediction through an LSTM network, which greatly reduces the impact of CSI delay; further, deep reinforcement learning DRL is used to replace the traditional AMC based on MCS table lookup, which solves the problem that the MCS scheme is applicable to ideal channels and cannot adapt well to fast time-varying channels. The DQN network continuously learns and optimizes its own network parameters, and continuously approaches the optimal Q network, thereby maximizing throughput. This method can not only adapt well to the characteristics of meteor trail channels and obtain more suitable coding and modulation schemes, but also has low computational complexity and is easy to implement in engineering. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a flowchart of the adaptive modulation and coding method based on LSTM and DQN network in the present invention;

[0035] Figure 2 This is the framework flow chart of the DQN network module;

[0036] Figure 3 This is the time-signal-to-noise ratio simulation diagram of the meteor trail fast time-varying channel;

[0037] Figure 4 The throughput change trend of the LSTM-DQN algorithm in multiple trainings;

[0038] Figure 5 The throughput simulation comparison chart of different coding modulation schemes and the AMC algorithm based on LSTM-DQN proposed in this paper. Specific implementation methods

[0040] Below, we will explain the present invention in more detail with reference to the embodiments and drawings. It should be noted that the embodiments of the present invention are not limited to this example. Example 1

[0041] Refer to the appendix of the manual Figure 1 The specific implementation of Example 1 is divided into the following steps:

[0042] Step 1. Create an LSTM network framework at the sending end. LSTM is composed of memory cells and gating mechanisms, and the gate mechanism includes forget gate, input gate, and output gate, which are used to decide whether to discard the cell state information of the previous time step, whether to store the current input information in the cell state, and how the current cell state affects the hidden state of the output;

[0043] Step 2. Create a DQN network framework on the sending end. The DQN network framework is as follows: Figure 2 As shown, two neural networks are used: a Q network for estimating the current Q value, and a target network for calculating the target Q value, and the parameters of the Q network need to be copied to the target network periodically. At the same time, an experience pool is used to store the agent's experience in different states. These experiences are stored in the form of experience tuples, including the current state, action, reward, next state, and a flag indicating whether it has terminated. Corresponding to the task of adaptive modulation and coding, the experience tuples are the current signal-to-noise ratio, the modulation and coding strategy adopted, the corresponding transmission situation in the feedback information, the signal-to-noise ratio at the next moment, and whether the model has converged;

[0044] Step 3. Initial stage signal transmission: Since the training of the neural network requires a certain number of input parameters, before the number of feedback signals reaches the threshold, the transmitter processes the pre-transmitted bit stream data according to the initial coding and modulation scheme and sends the generated signal to the receiver;

[0045] Step 4. After receiving the signal, the receiving end demodulates and decodes the signal in a corresponding manner according to the identifier in the frame header information, selects to feedback ACK or NAK according to the verification result of the CRC code, and estimates the bit error rate (BER) according to the pilot signal, and transmits it to the sending end as a feedback signal;

[0046] Step 5. After receiving the feedback signal, the transmitter first estimates the signal-to-noise ratio, obtains the current signal-to-noise ratio as the basis for selecting the next action, and determines whether the data in the experience pool is sufficient. If the data is insufficient, return to step 3 and continue to send repeatedly until the data reaches the threshold requirement, otherwise proceed to the next step;

[0047] Step 6. After the number of data in the experience pool at the sending end reaches the preset number requirement, start training the network model. Input n-step SNR data into LSTM, predict the SNR at the next moment, compare the generated output value with the true target value, calculate the mean square error MSE, and then calculate the gradient and update the network weights through the back propagation algorithm;

[0048] Step 7. During the training process of DQN, by maintaining the experience pool, a batch of data is randomly sampled from the experience pool for training each time to improve the training effect. DQN uses the Bellman optimal equation as the basis for updating the Q function, calculates the mean square error between the predicted Q value and the target Q value as the loss function, uses the gradient descent algorithm to minimize the loss function, and continuously updates the weights of the Q network. At the same time, DQN also needs to use the ∈ greedy strategy to balance the relationship between exploration and utilization during learning, so that the intelligent agent can fully learn various strategies;

[0049] Step 8. Whenever the sender receives a feedback signal, it adds the experience to the experience pool and updates the network parameters. When the amount of data in the experience pool exceeds the preset experience pool size, the experience tuples with a longer existence will be discarded according to the queue-based first-in-first-out principle, making full use of the data while ensuring the timeliness of the experience pool data;

[0050] Step 9. Repeat the above network training steps until the model converges, and the obtained model can better adapt to fast time-varying channels.

[0051] In the process of meteor trail communication, since meteors decay rapidly and become unusable after a short appearance, the meteor trail channel has the characteristics of burst and fast time variation. The signal-to-noise ratio reaches a peak in a short time and then decays rapidly. A typical meteor trail channel consists of under-dense meteor trails and over-dense meteor trails. The channel simulation diagram is shown in the appendix. Figure 3 shown.

[0052] Furthermore, the transmitted data is processed with noise based on the simulated channel to simulate the actual meteor trail communication process, and the LSTM-DQN network training is carried out based on this process. The experimental results are shown in the appendix of the manual. Figure 4 As shown in the figure, after 100 rounds of training, the total throughput of the system continues to increase and finally stabilizes.

[0053] Finally, various modulation and coding schemes were simulated, including: AMC scheme based on LSTM-DQN, AMC based on traditional MCS lookup table, scheme with fixed modulation mode of 16QAM and code rate of 3 / 4, scheme with fixed modulation mode of BPSK and code rate of 1 / 2. The simulation results show that the adaptive modulation and coding scheme based on LSTM-DQN achieves the maximum throughput under the same channel conditions, exceeding the AMC scheme based on traditional MCS lookup table.

[0054] The basic principle of the present invention is that the transmitting end of the meteor trail communication system performs LSTM timing prediction and DQN deep reinforcement learning according to the feedback signal, and adaptively changes the coding modulation scheme through the learned strategy. The method proposed in the present invention comprehensively considers the impact of channel state information (CSI) delay, and uses LSTM timing prediction to predict and correct CSI to make it closer to the actual situation of the channel; at the same time, considering that the AMC algorithm based on the traditional MCS table lookup is not the optimal solution in the fast time-varying channel, the DQN network is used to re-learn the modulation and coding strategy for the meteor trail channel with large link change fluctuations. The simulation experimental results show that the method proposed in the present invention achieves higher throughput and comprehensive performance than the traditional scheme.

[0055] The above is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any simple modification or equivalent change made to the above embodiment based on the technical essence of the present invention shall fall within the protection scope of the present invention.

Claims

1. An adaptive modulation coding method based on LSTM-DQN, characterized in that: The steps include: Step 1. Create LSTM and deep Q learning (DQN) network framework on the sending end, including input layer, hidden layer, output layer, and initialize network parameters; Step 2. Initial stage signal transmission: Since the training of the neural network requires a certain number of input parameters, before the number of feedback signals reaches the threshold, the transmitter processes the pre-transmitted bit stream data according to the initial coding and modulation scheme and sends the generated signal to the receiver; Step 3. After receiving the signal, the receiving end demodulates and decodes the signal in a corresponding manner according to the identifier in the frame header information, selects to feedback ACK or NAK according to the verification result of the CRC code, and estimates the bit error rate (BER) according to the pilot signal, and transmits it to the sending end as a feedback signal; Step 4. After receiving the feedback signal, the transmitter first estimates the signal-to-noise ratio, obtains the current signal-to-noise ratio as the basis for selecting the next action, and determines whether the data in the experience pool is sufficient. If the data is insufficient, return to step 2 and continue to send repeatedly until the data reaches the threshold requirement, otherwise proceed to the next step; Step 5. After the amount of data in the experience pool at the transmitter reaches the preset amount requirement, start training the network model. First, train LSTM to reduce the impact of transmission delay, input n-step SNR data, predict the SNR at the next moment, and after obtaining the SNR of the actual received signal at the next moment, use the gradient descent method to optimize the network; Step 6. After passing through the LSTM network, the corrected SNR is obtained as the input state s of reinforcement learning. According to s, the modulation coding mode, i.e., action a, is selected. The reward r is obtained according to the feedback signal. The SNR of the next received signal is taken as s'. Done is set according to whether the model converges. The above data are combined into the experience tuple (s, a, r, s', done). Step 7. Add the experience element to the experience pool. When training DQN, use the experience replay algorithm to randomly sample data from the experience pool as the input data of the network. After obtaining the estimated values ​​of the Q network and the target network, calculate the mean square error (MSE), and minimize the loss function through gradient descent to optimize the DQN network. Step 8. Whenever the transmitter receives a feedback signal, it adds the experience to the experience pool and updates the network parameters. Repeat the above network training steps until the model converges, and the obtained model can better adapt to fast time-varying channels.

2. The adaptive modulation and coding method based on LSTM-DQN according to claim 1, characterized in that: The specific operation method for determining the number of modulation and coding schemes is: select different coding and modulation schemes through the DQN network: {(M1,R1),(M2,R2),…,(M n-1 ,R n-1 )} where M1,…,M n-1 For different modulation modes, R1,…,R n-1 For different code rates.

3. The adaptive modulation and coding method based on LSTM-DQN according to claim 1, characterized in that: The specific operation method of measuring the signal-to-noise ratio at the transmitter is: add a pilot signal to the head of the feedback signal, use the pilot-assisted least squares (LS) signal-to-noise ratio estimation algorithm, and estimate the signal-to-noise ratio Get a delayed signal-to-noise ratio information as the input for the next LSTM training. k is the received signal in the frequency domain, X k is the pilot signal in the frequency domain, is the ensemble mean, (·) * is the complex conjugate operation, is the cross-correlation function, 4. The adaptive modulation and coding method based on LSTM-DQN according to claim 1, characterized in that: The specific operation method of obtaining the transmission status of the previous moment according to the feedback signal is as follows: the receiving end first estimates the bit error rate according to the pilot signal, and chooses to feedback ACK or NAK according to the verification result of the CRC code, and transmits the above information to the sending end as a feedback signal. The sending end obtains the transmission status of the previous moment as the reward parameter r of reinforcement learning, where r is calculated as: Among them, R and M are the code rate and modulation order selected for this transmission respectively, and the reward r obtained is regarded as the transmission situation at the previous moment, which serves as the input data of reinforcement learning.

5. The adaptive modulation and coding method based on LSTM-DQN according to claim 1, characterized in that: The specific operation method of using LSTM for time series prediction is as follows: first preprocess the data, split the input sequence into multiple sliding windows, and predict the signal-to-noise ratio of the next time based on the past n time steps each time the model is trained. Then, use the memory unit and gating mechanism to build the network framework, and calculate the MSE of the output value and the true target to obtain the loss function L, calculate the gradient through the back propagation algorithm and update the network weight. The calculation formula of the loss function is:

6. The adaptive modulation and coding method based on LSTM-DQN according to claim 1, characterized in that: The specific operation method of using the DQN network to select the modulation and coding strategy is as follows: the signal-to-noise ratio obtained by the LSTM network is used as the input state s of the deep reinforcement learning network, and after passing through DQN, the modulation and coding method to be taken next is obtained, that is, the action a in reinforcement learning, thereby realizing the adaptive selection of the modulation and coding strategy. During the training process of the DQN network, (s, a, r, s', done) is used as a set of experience data, and a set of experience data is added to the experience pool each time it is transmitted. Through experience playback, a batch of data is randomly sampled from the experience pool for training at each training step, making full use of the data and improving the training effect. The optimization of the Q network uses the Bellman optimal equation, and the update formula of DQN is: Where r is the immediate reward, s' is the next state, γ is the discount factor, and θ - are the parameters of the target network. Then the loss function is calculated, the gradient descent algorithm is used to minimize the loss function, and the weights of the network are updated. The loss function is: Where Q(s t ,a t ; θ) is the current Q value estimated by the Q network. At the same time, the weights of the Q network are periodically copied to the target network.

Images