Satellite-ground adaptive transmission method under multi-channel probability distribution

Abstract

The invention discloses a satellite-ground adaptive transmission method under multi-channel probability distribution. According to the invention, based on the deep Q network, an optimal satellite-ground adaptive transmission scheme is learned and screened; in the learning screening process, the influence of transmission parameters such as a symbol rate, a coding rate and a modulation mode on the channel transmission state is fully considered, the optimal switching boundary between different transmission modes can be obtained through DQN reinforcement learning under the condition of different channel state probability distribution, and on the premise of ensuring the reliability of data transmission, the transmission efficiency is improved. And the average transmission rate of the whole process is maximized.

Description

Technical Field

[0001] The present invention relates to the technical field of satellite data transmission, and in particular to a satellite-to-ground adaptive transmission method under multi-channel probability distribution. Background Art

[0002] As the satellite moves rapidly, the satellite data transmission link will experience significant changes in the quality of the communication link under the influence of various environmental factors. The channel state takes different values ​​and is unevenly distributed during the communication process. In this case, if a single information transmission scheme is used, it will not be able to cope with the channel conditions that are sometimes good and sometimes bad, and the transmission data performance will be sometimes high and sometimes low. Adaptive transmission technology has been proposed to cope with various channel conditions. It can select appropriate transmission parameters such as power, modulation mode, and coding rate based on the channel state.

[0003] In recent years, reinforcement learning algorithms have been used to make decisions and design satellite-to-ground adaptive transmission schemes under different channels. An adaptive modulation scheme based on a deep Q network (DQN) divides the signal-to-noise ratio range into rate regions, and uses the rate region boundaries as states for reinforcement learning, thereby meeting the requirements of transmission reliability and further improving spectrum efficiency. The optimal state memory is introduced to reduce the number of training times, allowing reinforcement learning to converge quickly. However, the algorithm only considers the parameter selection of the modulation method, and has not yet selected parameters such as the symbol rate and the coding rate. In addition, the existing satellite-to-ground adaptive transmission schemes are mainly designed based on the assumption that the channel state is uniformly distributed, and cannot adapt to a variety of channel states. Summary of the invention

[0004] In view of this, the present invention provides a satellite-to-ground adaptive transmission method under multi-channel probability distribution, which can select appropriate transmission parameters such as power, modulation mode, and coding rate according to the real-time channel state when the channels are unevenly distributed. The satellite-to-ground adaptive transmission scheme can obtain the optimal switching boundary between different transmission modes through DQN reinforcement learning under different channel state probability distribution conditions, and maximize the average transmission rate of the entire process while ensuring the reliability of data transmission.

[0005] The satellite-to-ground adaptive transmission method under multi-channel probability distribution of the present invention comprises:

[0006] Step 1: Build a DQN reinforcement learning model;

[0007] The state in DQN reinforcement learning is the switching boundary S of the satellite-to-ground adaptive transmission scheme, S = {s1, s2, ..., s N-1}, where N is the number of transmission combinations consisting of m modulation modes, n coding rates and k symbol rates; the initial state is an evenly divided switching boundary;

[0008] The action in DQN reinforcement learning is the adjustment action A of each switching boundary of the satellite-to-ground adaptive transmission combination. Each switching boundary has two adjustment methods: increase or decrease the boundary value; if the boundary value is increased, the action value of the boundary is 0; if the boundary value is decreased, the boundary value is 1;

[0009] The reward in DQN reinforcement learning is Among them, R ci is the symbol rate of the i-th transmission combination; q i is the modulation order corresponding to the i-th transmission combination; w i is the weight of the carrier-to-noise ratio range of the i-th transmission combination in the overall carrier-to-noise ratio change range; is the average bit error rate of the i-th transmission combination, is the theoretical bit error rate under the current carrier-to-noise ratio; is the carrier-to-noise ratio probability distribution; is the carrier-to-noise ratio switching boundary between the i-th transmission combination and the i+1-th transmission combination, CNR max is the maximum carrier-to-noise ratio;

[0010] Step 2: Train the DQN reinforcement learning model constructed in step 1:

[0011] Step 2.1, randomly select an action with probability ε, that is, adjust the switching boundary and execute it, to obtain the satellite-to-ground adaptive transmission scheme after the switching boundary is adjusted;

[0012] Determine whether the state of the adjusted satellite-to-ground adaptive transmission combination is valid; if the state satisfies:

[0013] (1) The carrier-to-noise ratio value corresponding to this state does not exceed the carrier-to-noise ratio value of the transmission combination with a larger state number;

[0014] (2) The state is within the carrier-to-noise ratio range;

[0015] (3) E before encoding corresponding to this state b / N0 is greater than the lowest signal-to-noise ratio after demodulation;

[0016] If the state is invalid, the training will continue. If the state is invalid, the training will end and a new training will start. The best state in the best state memory in DQN reinforcement learning will be set as the state for the new training.

[0017] Step 2.2: Calculate the reward value r corresponding to the satellite-to-ground adaptive transmission scheme after the switching boundary is adjusted t ; and determine whether the spacing of the adjusted carrier-to-noise ratio boundaries is less than the minimum spacing. If so, reduce the reward value r t, if it is greater than or equal to, the reward value remains unchanged; at the same time, if the bit error rate of the satellite-to-ground adaptive transmission scheme after the switching boundary adjustment is less than or equal to the BER constraint, the reward value r is increased. t , if the bit error rate is greater than or equal to the BER constraint, the reward value r is greatly reduced t ;

[0018] Step 2.3, extract samples from the DQN experience replay pool to update the DQN value neural network and store the current best boundary adjustment strategy;

[0019] Step 2.4: Check whether the DQN reinforcement learning has gradually converged. If so, stop the reinforcement learning training after reaching the number of iterations to obtain the best satellite-to-ground adaptive transmission combination under the current channel state probability distribution; otherwise, return to step 2.1;

[0020] Step 3: Perform data transmission according to the best satellite-to-ground adaptive transmission combination obtained in step 2.4.

[0021] Preferably, in step 2.2, if the spacing of the adjusted CNR boundaries is less than the minimum spacing, then r t ←r t -5r;

[0022] If the bit error rate of the satellite-to-ground adaptive transmission scheme after the switching boundary adjustment is less than the BER constraint, then r t ←r t +r, if the bit error rate is greater than or equal to the BER constraint, then r t ←r t -10r.

[0023] Beneficial effects:

[0024] The present invention is based on a deep Q network to learn and screen out the best satellite-to-ground adaptive transmission scheme; in the learning and screening process, the influence of transmission parameters such as symbol rate, coding rate, modulation mode, etc. on the channel transmission state is fully considered, while ensuring the reliability of data transmission, the data transmission rate is effectively improved; the present invention is based on real-time channel state The learning and screening is applicable to situations where the channel state is unstable. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is a block diagram of the satellite-to-ground adaptive transmission method of the present invention. DETAILED DESCRIPTION

[0026] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0027] The present invention provides a satellite-to-ground adaptive transmission method under multi-channel probability distribution, the structure framework of the method is as follows: Figure 1 As shown, the following steps are included:

[0028] Step 1: Initialize and configure the specific encoding rate R of each transmission mode c , modulation order q and symbol rate R s The switching boundaries between different transmission modes are initialized with the same transmission parameters, the current channel probability distribution is obtained, an evaluation model for evaluating the current scheme is constructed, the switching boundaries, boundary adjustments, and evaluation models in the satellite-to-ground adaptive transmission scheme are established, and the element mapping between states, actions, and rewards in DQN reinforcement learning is constructed, and the neural network structure of reinforcement learning is constructed.

[0029] Specifically,

[0030] S11, setting a set of satellite-ground adaptive transmission schemes: N transmission mode combinations consisting of m modulation modes, n coding rates, k symbol rates and other transmission parameters.

[0031] S12, switching boundary between initialization schemes: The channel state is usually expressed by the carrier-to-noise ratio, and the initialization switching boundary evenly distributes the entire carrier-to-noise ratio variation range, expressed as

[0032]

[0033] in, represents the carrier-to-noise ratio switching boundary between the i-th transmission scheme and the i+1-th transmission scheme, CNR max Indicates the maximum carrier-to-noise ratio. Gets the probability distribution of the current channel carrier-to-noise ratio.

[0034] S13, define an evaluation model for evaluating the current scheme: evaluate the transmission scheme under the current channel probability from the two aspects of reliability and transmission capacity. Reliability can be evaluated by bit error rate, set BER constraints, and calculate the average bit error rate of each transmission combination The calculation method is

[0035]

[0036] in, represents the probability distribution of the carrier-to-noise ratio, Indicates the theoretical bit error rate under the current carrier-to-noise ratio.

[0037] The transmission capacity is based on the average data rate of the entire channel change process Evaluation, calculated as

[0038]

[0039] Among them, R bi represents the data rate corresponding to the i-th transmission combination, R bi =R si R ci qi , where R si represents the symbol rate corresponding to the i-th transmission combination, R ci represents the encoding rate corresponding to the i-th transmission combination, q i represents the modulation order corresponding to the i-th transmission combination. The evaluation model comprehensively considers the modulation mode, coding rate and symbol rate.

[0040] S14,To adjust the switching boundary of the satellite-to-ground adaptive transmission scheme through DQN reinforcement learning, it is necessary to convert the learning task into the corresponding elements in reinforcement learning, namely, state, action and reward.

[0041] The state in DQN reinforcement learning is defined as the switching boundary of the satellite-to-ground adaptive transmission scheme, written as S = {s1, s2, …, s N-1}, N is the number of transmission combinations in the scheme.

[0042] The action in DQN reinforcement learning is defined as the adjustment action of each switching boundary of the satellite-to-ground adaptive transmission scheme, expressed as There are two adjustment methods for each switching boundary. If the action value corresponding to a boundary is 0, it means increasing the boundary value; if the value is 1, it means decreasing the boundary value.

[0043] Define the reward in DQN reinforcement learning as R ci refers to the encoding rate of the i-th transmission combination, w i Refers to the weight of the carrier-to-noise ratio range of the i-th transmission combination in the overall carrier-to-noise ratio range. This reward takes into account the modulation method and the coding rate.

[0044] S15, initialize the DQN neural network structure: target neural network, value neural network, experience replay pool, optimal state memory and evaluation model. The value neural network Q is iteratively updated in each round for network training, and the network parameter is θ. Target neural network Keep the current optimal state-action value (Q value) function, the network parameter is θ - The experience replay pool D stores the current state of the evaluation model, the selected action, the corresponding reward value, and the state after the transfer. The optimal state memory B stores the state with the maximum average transfer rate.

[0045] Step 2: Apply the channel state probability distribution and the current satellite-to-ground adaptive transmission mode to the evaluation model, obtain a new switching boundary adjustment strategy through DQN reinforcement learning, and calculate the reward value and the adjusted new switching boundary under the current evaluation model. The best state under the current evaluation model, i.e., the best switching boundary, is stored in the DQN best state memory.

[0046] Specifically,

[0047] S21, DQN reinforcement learning is performed on the satellite-to-ground adaptive transmission scheme under the current channel probability distribution. The initial state of the DQN reinforcement learning is an evenly divided switching boundary. First, an action is randomly selected with probability ε, that is, adjusting the switching boundary and executing it, to obtain the satellite-to-ground adaptive transmission scheme after the switching boundary is adjusted.

[0048] S22, judging whether the adjusted state is valid. If the state does not meet the requirements, the training is judged to be over, a new round of training is started, and the best state in the best state memory is set as the state of the new round of training.

[0049] The criteria for determining whether a state is valid are:

[0050] (1) The carrier-to-noise ratio value corresponding to each state cannot exceed the carrier-to-noise ratio value of the transmission scheme with a larger state number, expressed as

[0051] (2) Each state needs to be within the carrier-to-noise ratio range, expressed as

[0052] (3) E before encoding corresponding to each state b / N0 must be greater than the minimum demodulated signal-to-noise ratio, expressed as E b / N0>10log 10 (R si q i )dB.

[0053] S23, calculating the reward value r corresponding to the satellite-to-ground adaptive transmission scheme after the execution of the action at the current moment t , severe penalties will be imposed on the adjusted transmission schemes that do not meet the requirements. Determine the spacing of the adjusted carrier-to-noise ratio boundaries. If it is less than the minimum spacing, r t ←r t -5r, if it is greater than or equal to the minimum interval, the reward value remains unchanged. In addition, adjusting the CNR may cause the BER within this range to fail to meet the constraint requirements, resulting in state failure. Therefore, when the bit error rate is less than or equal to the BER constraint, the reward value r t ←r t +r; if it is greater than the BER constraint, it means that the current solution cannot meet the basic communication requirements, r t ←r t -10r.

[0054] Step 3: Extract samples from the DQN experience replay pool to update the DQN value neural network and store the current best boundary adjustment strategy.

[0055] Specifically,

[0056] S31, randomly select a batch of sample data (s k ,a k ,r k ,s′ k ). Where s k is the state value of the kth round, a k is the action value of the kth round, r k is the reward value of the kth round, s′ k is the state value after adjustment for the k-th round action.

[0057] S32, calculate the target Q value

[0058] S33, for the loss function y k -Q(s k ,a k ;θ) performs gradient descent to optimize the Q network parameters θ.

[0059] S34, after each fixed number of steps C, the target Q network Update with Q.

[0060] Step 4, determine whether the iterative update of the reinforcement learning task makes the network gradually converge. If so, after reaching the number of iterations, stop the reinforcement learning training and obtain the best satellite-to-ground adaptive transmission scheme under the current channel state probability distribution; otherwise, return to step 2 to continue learning.

[0061] Step 5: Perform data transmission according to the best satellite-to-ground adaptive transmission scheme obtained in step 4.

[0062] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A satellite-to-ground adaptive transmission method under multi-channel probability distribution, characterized in that: include: Step 1: Build a DQN reinforcement learning model; The state in DQN reinforcement learning is the switching boundary S of the satellite-to-ground adaptive transmission scheme, S = {s1, s2, ..., s N-1 }, where N is the number of transmission combinations consisting of m modulation modes, n coding rates and k symbol rates; the initial state is an evenly divided switching boundary; The action in DQN reinforcement learning is the adjustment action A of each switching boundary of the satellite-to-ground adaptive transmission combination. Each switching boundary has two adjustment methods: increase or decrease the boundary value; if the boundary value is increased, the action value of the boundary is 0; if the boundary value is decreased, the boundary value is 1; The reward in DQN reinforcement learning is Among them, R ci is the symbol rate of the i-th transmission combination; q i is the modulation order corresponding to the i-th transmission combination; w i is the weight of the carrier-to-noise ratio range of the i-th transmission combination in the overall carrier-to-noise ratio change range; is the average bit error rate of the i-th transmission combination, is the theoretical bit error rate under the current carrier-to-noise ratio; is the carrier-to-noise ratio probability distribution; is the carrier-to-noise ratio switching boundary between the i-th transmission combination and the i+1-th transmission combination, CNR max is the maximum carrier-to-noise ratio; Step 2: Train the DQN reinforcement learning model constructed in step 1: Step 2.1, randomly select an action with probability ε, that is, adjust the switching boundary and execute it, to obtain the satellite-to-ground adaptive transmission scheme after the switching boundary is adjusted; Determine whether the state of the adjusted satellite-to-ground adaptive transmission combination is valid; if the state satisfies: (1) The carrier-to-noise ratio value corresponding to this state does not exceed the carrier-to-noise ratio value of the transmission combination with a larger state number; (2) The state is within the carrier-to-noise ratio range; (3) E before encoding corresponding to this state b / N0 is greater than the lowest signal-to-noise ratio after demodulation; If the state is invalid, the training will continue. If the state is invalid, the training will end and a new training will start. The best state in the best state memory in DQN reinforcement learning will be set as the state for the new training. Step 2.2: Calculate the reward value r corresponding to the satellite-to-ground adaptive transmission scheme after the switching boundary is adjusted t ; and determine whether the spacing of the adjusted carrier-to-noise ratio boundaries is less than the minimum spacing. If so, reduce the reward value r t , if it is greater than or equal to, the reward value remains unchanged; at the same time, if the bit error rate of the satellite-to-ground adaptive transmission scheme after the switching boundary adjustment is less than or equal to the BER constraint, the reward value r is increased. t , if the bit error rate is greater than or equal to the BER constraint, the reward value r is greatly reduced t ; Step 2.3, extract samples from the DQN experience replay pool to update the DQN value neural network and store the current best boundary adjustment strategy; Step 2.4: Check whether the DQN reinforcement learning has gradually converged. If so, stop the reinforcement learning training after reaching the number of iterations to obtain the best satellite-to-ground adaptive transmission combination under the current channel state probability distribution; otherwise, return to step 2.1; Step 3: Perform data transmission according to the best satellite-ground adaptive transmission combination obtained in step 2.

4.

2. The method according to claim 1, characterized in that In step 2.2, if the spacing between the adjusted CNR boundaries is less than the minimum spacing, then r t ←r t -5r; If the bit error rate of the satellite-to-ground adaptive transmission scheme after the switching boundary adjustment is less than the BER constraint, then r t ←r t +r, if the bit error rate is greater than or equal to the BER constraint, then r t ←r t -10r.

Images