NB-IoT uplink coverage enhancement optimization method based on deep reinforcement learning
Through deep reinforcement learning, NB-IoT uplink coverage is optimized, combined with uplink power spectral density, repeated sending times, and modulation and coding strategies, the problem of insufficient optimization of coverage enhancement in the existing technology is solved, achieving higher reliability and wider coverage NB-IoT uplink communication.
Patent Information
- Application Number
- CN202210846565.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-07-19
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2042-07-19
AI Technical Summary
The existing NB-IoT protocol has a coarse granularity in the uplink coverage enhancement level, resulting in large differences in device path losses at the same level, and some terminal resources are wasted. The existing optimization research has failed to fully consider the uplink power spectrum density and the number of repeated sendings. Three factors such as modulation and coding strategies cannot achieve optimal coverage enhancement.
Using the NB-IoT uplink coverage enhancement optimization method (DDQN-CE) based on deep reinforcement learning, the reliable transmission of uplink data is used as a constraint, and the interaction between the agent and the NB-IoT communication environment is performed through deep neural networks and experience playback pools, and the optimization is performed. The state transfer sample of the user terminal is trained to minimize the uplink communication delay and realize the optimized configuration of multi-dimensional parameters.
It realizes higher reliability, lower latency and wider coverage of NB-IoT uplink communication, overcomes the problem of resource waste, and personalized parameter optimization configuration for different devices, improving coverage enhancement effect.
Smart Images

Figure CN115396902B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the fields of NB-IoT communication technology and deep reinforcement learning, and specifically relates to an NB-IoT uplink coverage enhancement optimization method based on deep reinforcement learning. Background Art
[0002] As a low-power wide-area Internet of Things technology, one of the design goals of NB-IoT is to have a larger coverage than GSM and LTE. The uplink coverage enhancement of NB-IoT is mainly affected by three factors: uplink power spectral density, number of repeated transmissions, and modulation and coding scheme (MCS). The existing NB-IoT protocol has a relatively coarse granularity in the division of coverage enhancement levels, which are divided into three coverage enhancement levels, namely: conventional coverage, MCL < 144 dB; extended coverage, 144 dB < MCL < 154 dB; extreme coverage, MCL > 154 dB. The range of the maximum coupling loss interval divided is relatively wide, which will cause relatively large differences in the path loss of different devices under the same coverage enhancement level. And for devices under the same coverage enhancement level, the same parameter configuration will be adopted, resulting in waste of resources for some terminals. Currently, for the related research on optimizing the NB-IoT coverage enhancement mechanism, some only consider one or two of the three factors related to NB-IoT coverage enhancement, without comprehensively incorporating all influencing factors and cannot achieve the optimal coverage enhancement; there are also some that consider all three dimensions, but their complexity is relatively high or they fail to achieve the optimal performance.
[0003] Deep learning uses deep neural networks to abstract high-dimensional data into low-dimensional representations (features) to achieve effective perception of things. In reinforcement learning, the agent interacts with the environment with the goal of maximizing the cumulative reward and continuously optimizes the behavioral decision-making, having a powerful decision-making ability. Deep reinforcement learning combines the perception expression ability of deep learning and the decision-making ability of reinforcement learning, can make better decisions based on the directly obtained original information, and deep reinforcement learning uses deep neural networks to fit the value function, which can solve the curse of dimensionality problem of traditional reinforcement learning.凭借其深度神经网络的泛化能力和特征提取能力以及强化学习的决策能力,在许多领域获得成功,已有学者将深度强化学习成功应用于无线网络研究,并取得较好的效果。 Summary of the Invention
[0004] In view of the shortcomings of the prior art, the present invention proposes a NB-IoT uplink coverage enhancement optimization method based on deep reinforcement learning (DDQN-CE, Double DQN based Coverage Enhancement). This method constructs the NB-IoT uplink coverage resource allocation problem as a deep reinforcement learning process, takes the reliable transmission of uplink data (BLER≤10%) as a constraint, and takes minimizing the uplink communication delay as the optimization goal. The deep reinforcement learning agent interacts with the NB-IoT communication environment, generates state transition samples of the user terminal, and stores them in the experience replay pool. The agent extracts batch samples from the experience replay pool for training and learning, and uses gradient back propagation to update the neural network parameters so that the neural network parameters of the agent converge, and finally realizes the optimization configuration of the multi-dimensional parameters of the NB-IoT uplink coverage. In this way, the reliability of NB-IoT uplink communication is enhanced, the uplink communication delay is reduced, the uplink coverage range is expanded, and the uplink coverage of NB-IoT is effectively enhanced.
[0005] To achieve the above-mentioned purpose of the invention, the present invention adopts the following technical solution: a NB-IoT uplink coverage enhancement optimization method based on deep reinforcement learning, comprising the following steps:
[0006] Step 1: Construct two neural networks with the same structure in DDQN, the estimated Q network and the target Q network; initialize the network parameters θ of the estimated Q network and the target Q network parameters θ - , let θ - =θ; initialize the experience replay pool R; initialize the algorithm parameter learning step α, target Q network parameter update time step C, number of training steps per round T, discount factor γ, maximum exploration rate ε max and the minimum value ε min , exploration rate reduction Δ, exploration rate ε = ε max , the number of samples m for batch gradient descent;
[0007] Step 2, initialize the uplink communication parameters of the user terminal in the NB-IoT communication environment: the number of repeated transmissions, the transmission bandwidth, the MCS level and the number of transmission block resource units, and the initial time step t = 0;
[0008] Step 3: The agent (decision maker and learner in deep reinforcement learning) obtains the state information of the user terminal from the NB-IoT communication environment and obtains the state s t , the state s t Input estimation Q network, select action a according to ∈-greedy strategy t ;
[0009] Step 4: tConvert to the parameter allocation form of NB-IoT uplink coverage enhancement, configure the NB-IoT user terminal, change the user terminal status, perform uplink data transmission, collect data such as SNR (signal-to-noise ratio) and transmission delay, and calculate the reward value;
[0010] Step 5: The agent obtains the reward r after executing the action from the NB-IoT communication environment t+1 and status t+1 , the state transfer sample (s t ,a t ,r t+1 ,s t+1 ) is stored in the experience replay pool;
[0011] Step 6: Determine whether the number of samples in the experience pool is less than the batch sampling number m. If so, go to step 3. Otherwise, randomly select a batch of m samples (s j,n ,a j,n ,r j+1,n ,s j+1,n ), n∈[1,m], train the estimated Q network, and update the parameters θ of the estimated Q network by gradient descent through neural network gradient back propagation;
[0012] Step 7: Update the exploration rate ε, let ε max =ε max -Δ,ε=max(ε max ,ε min );
[0013] Step 8, let t = t + 1, if t%C = 0, update the parameter θ of the target Q network - , let θ - =θ;
[0014] Step 9, determine whether t is less than the number of training steps per round T. If so, go to step 3. Otherwise, calculate the cumulative reward for this round and determine whether the estimated Q network parameter θ converges. If so, go to step 10. Otherwise, go to step 2.
[0015] Step 10: Input the state of the NB-IoT user terminal into the estimated Q network, adopt a greedy strategy to select and execute the action with the greatest value, and achieve the optimal configuration of the NB-IoT uplink coverage enhancement parameters;
[0016] Step 11: The NB-IoT uplink coverage enhancement optimization process ends.
[0017] Furthermore, in the above step 2, the uplink communication parameters of the user terminal in the NB-IoT communication system are initialized as follows: the number of repeated transmissions is 1; the transmission bandwidth is 180kHz (the subcarrier spacing is 15kHz, and the number of subcarriers is 12); the MSC level is 0; and the number of transmission block resource units is 1.
[0018] Furthermore, in the above step 3, the state space S = {I Rep ,I BW ,I MCS ,I RU ,delay,rel},state s t ∈S.
[0019] in:
[0020] I Rep The index value of the number of repeated transmissions, the value range is: 0-7;
[0021] I BW The transmission bandwidth index value ranges from 0 to 4.
[0022] I MCS MCS level index value, value range: single subcarrier transmission, 0-10; multi-subcarrier transmission, 0-12;
[0023] I RU The index value of the number of transport block resource units, the value range is: 0-7;
[0024] delay is the uplink data transmission delay, which is a real number;
[0025] rel is the transmission reliability and has two possible values: 0 when BLER is greater than 0.1; and 1 when BLER is less than or equal to 0.1.
[0026] The ∈-greedy strategy formula is as follows:
[0027]
[0028] Among them, π(a|s) is the probability of selecting action a in state s, and |A(s)| is the number of actions that can be selected in state s.
[0029] The action space A is defined as I Rep ,I BW ,I MCS ,I RU There are three possible changes in each value: plus one, unchanged, and minus one. Action a t ∈A.
[0030] Furthermore, in the above step 4, when constructing the NB-IoT uplink coverage enhancement optimization problem as a deep reinforcement learning process, reliable uplink data transmission (BLER ≤ 10%) is used as a constraint condition, and minimizing the uplink communication delay is used as the optimization goal. The specific expression formula is as follows:
[0031] min delay
[0032] stBLER≤0.1
[0033] Wherein, delay is the uplink transmission delay of the user terminal, and BLER is the uplink transmission block error rate of the user terminal.
[0034] The calculation formula for defining the reward r is as follows:
[0035]
[0036] Among them, delay max is the maximum delay of uplink communication; whether BLER is greater than 0.1, based on the SNR and SNR of the current uplink channel Th By comparison, when SNR ≥ SNR Th When BLER≤0.1, otherwise BLER>0.1. SNR Th It is the signal-to-noise ratio corresponding to a BLER of 0.1 under the current uplink transmission parameter configuration.
[0037] Furthermore, in step 6 above, the estimated Q network is trained and the following steps are performed to update the parameter θ:
[0038] 6.1 Perform forward propagation on the estimated Q network and obtain the sample estimated action value as follows:
[0039]
[0040] in, is a state-action pair (s j,n ,a j,n )’s estimated action value.
[0041] 6.2 Selections j+1,n Actions:
[0042]
[0043] Where a′ is the state s j+1,n The action corresponding to the maximum estimated action value.
[0044] 6.3 Finding s using the target Q network j+1,n And the estimated action value of a′:
[0045]
[0046] in, is a state-action pair (s j+1,n ,a′)’s target action value.
[0047] 6.4 Calculate TD (Temporal Difference) target and TD error:
[0048]
[0049] Among them, y j,n For TD goal.
[0050]
[0051] Among them, δ j,n is the TD error.
[0052] 6.5 Back propagate the estimated Q network and find Q(s j,n ,a j,n ;θ) with respect to the gradient of parameter θ:
[0053]
[0054] 6.6 Perform gradient descent to update the estimated parameters of the Q network:
[0055]
[0056] Furthermore, in the above step 10, a greedy strategy is used to select actions to achieve optimal configuration of NB-IoT uplink coverage enhancement parameters. The greedy strategy formula is as follows:
[0057]
[0058] Among them, π * (a|s) represents the optimal strategy, q * (s,a) is the optimal action-value function.
[0059] Beneficial effects of the present invention:
[0060] The NB-IoT uplink coverage method based on deep reinforcement learning in the present invention takes into account three factors related to NB-IoT uplink coverage enhancement, and can optimize NB-IoT uplink coverage more comprehensively, so that NB-IoT uplink communication has higher reliability, lower uplink communication delay, and wider uplink coverage range; according to the actual uplink channel quality of the NB-IoT terminal, its uplink coverage parameters are optimized and configured in an individualized manner, which can overcome the problem of waste of some terminal resources due to the coarse granularity of NB-IoT coverage level division. BRIEF DESCRIPTION OF THE DRAWINGS
[0061] Figure 1 It is the overall flow chart of the method proposed in the present invention.
[0062] Figure 2 It is the overall framework of the method proposed in this invention. DETAILED DESCRIPTION
[0063] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be pointed out that the specific embodiments described herein are only used to explain the present invention and do not constitute a limitation to the present invention.
[0064] like Figure 1 , Figure 2 As shown, the present invention relates to a NB-IoT uplink coverage enhancement optimization method based on deep reinforcement learning, comprising the following steps:
[0065] Step 1: Construct two neural networks with the same structure in DDQN, the estimated Q network and the target Q network; initialize the network parameters θ of the estimated Q network and the target Q network parameters θ - , let θ - =θ; initialize the experience replay pool R; initialize the algorithm parameter learning step α, target Q network parameter update time step C, number of training steps per round T, discount factor γ, maximum exploration rate ε max and the minimum value ε min , exploration rate reduction Δ, exploration rate ε = ε max , the number of samples m for batch gradient descent.
[0066] As a preferred embodiment of the present invention, the estimated Q network parameter θ is randomly initialized, and the target Q network parameter θ is - =θ; the capacity of the experience replay pool is initialized to 1000; the learning step α is initialized to 0.001; the target Q network parameter update time step C is initialized to 100; the number of training steps per round T is initialized to 5000; the discount factor γ is initialized to 0.95; the maximum exploration rate ε max and the minimum value ε min They are initialized to 1 and 0.1 respectively; the exploration rate reduction Δ is initialized to 0.004; and the number of samples m for batch gradient descent is initialized to 128.
[0067] Step 2, initialize the uplink communication parameters of the user terminal in the NB-IoT communication environment: number of repeated transmissions, transmission bandwidth, MCS level and number of transmission block resource units, and the initial time step is t=0.
[0068] As a preferred embodiment of the present invention, the uplink communication parameters of the user terminal in the NB-IoT communication system are initialized as follows: the number of repeated transmissions is 1; the transmission bandwidth is 180kHz (the subcarrier spacing is 15kHz, and the number of subcarriers is 12); the MSC level is 0; and the number of transmission block resource units is 1.
[0069] Step 3: The agent (decision maker and learner in deep reinforcement learning) obtains the state information of the user terminal from the NB-IoT communication environment and obtains the state s t , the state s t Input estimation Q network, select action a according to ∈-greedy strategy t .
[0070] As a preferred embodiment of the present invention, the state space S={I Rep ,I BW ,I MCS ,I RU ,delay,rel},state s t ∈S. Among them:
[0071] I Rep The index value of the number of repeated transmissions, the value range is: 0-7, and the corresponding relationship with the number of repeated transmissions is as follows:
[0072] Table 1 Values of the number of repetitions
[0073]
[0074] I BW The transmission bandwidth index value has a value range of 0-4. The corresponding relationship between the subcarrier spacing and the number of subcarriers is as follows:
[0075] Table 2 Values of subcarrier spacing, number of subcarriers, RU duration and frequency factor
[0076]
[0077]
[0078] I MCS MCS level index value, the value range is: 0-10 for single subcarrier transmission, corresponding to MCS0-MCS10; 0-12 for multi-subcarrier transmission, corresponding to MCS0-MCS12.
[0079] I RU The index value of the number of transport block resource units has a value range of 0-7. The corresponding relationship with the number of single transport block resource units is as follows:
[0080] Table 3 Number of resource units in a single transport block
[0081]
[0082] delay is the uplink data transmission delay, which is a real number;
[0083] rel is the transmission reliability and has two possible values: 0 when BLER is greater than 0.1; and 1 when BLER is less than or equal to 0.1.
[0084] As a preferred embodiment of the present invention, the ε-greedy strategy formula is as follows:
[0085]
[0086] Among them, π(a|s) is the probability of selecting action a in state s, and |A(s)| is the number of actions that can be selected in state s.
[0087] The action space A is defined as I Rep ,I BW ,I MCS ,I RU There are three possible changes in each value: plus one, unchanged, and minus one. Action a t ∈A.
[0088] Step 4: t Convert to the parameter allocation form of NB-IoT uplink coverage enhancement, configure the NB-IoT user terminal, change the user terminal status, perform uplink data transmission, collect SNR (signal-to-noise ratio), transmission delay and other data, and calculate the reward value.
[0089] As a preferred embodiment of the present invention, when the NB-IoT uplink coverage enhancement optimization problem is constructed as a deep reinforcement learning process, reliable uplink data transmission (BLER ≤ 10%) is used as a constraint condition, and minimizing the uplink communication delay is used as the optimization goal. The specific optimization model is expressed as the following formula:
[0090] min delay
[0091] stBLER≤0.1
[0092] Wherein, delay is the uplink transmission delay of the user terminal, and BLER is the uplink transmission block error rate of the user terminal.
[0093] The calculation formula for defining the reward r is as follows:
[0094]
[0095] Among them, delay maxis the maximum delay of uplink communication; whether BLER is greater than 0.1, based on the SNR and SNR of the current uplink channel Th By comparison, when SNR ≥ SNR Th When BLER≤0.1, otherwise BLER>0.1. SNR Th It is the signal-to-noise ratio corresponding to a BLER of 0.1 under the current uplink transmission parameter configuration.
[0096] Step 5: The agent obtains the reward r after executing the action from the NB-IoT communication environment t+1 and status t+1 , the state transfer sample (s t ,a t ,r t+1 ,s t+1 ) is stored in the experience replay pool.
[0097] Step 6: Determine whether the number of samples in the experience pool is less than the batch sampling number m. If so, go to step 3. Otherwise, randomly select a batch of m samples (s j,n ,a j,n ,r j+1,n ,s j+1,n ), n∈[1,m], train the estimated Q network, and update the parameters θ of the estimated Q network by gradient descent through neural network gradient back propagation.
[0098] As a preferred embodiment of the present invention, the estimated Q network is trained and the following steps are performed to update the parameter θ:
[0099] 6.1 Perform forward propagation on the estimated Q network and obtain the sample estimated action value as follows:
[0100]
[0101] in, is a state-action pair (s j,n ,a j,n )’s estimated action value.
[0102] 6.2 Selections j+1,n Actions:
[0103]
[0104] Where a′ is the state s j+1,n The action corresponding to the maximum estimated action value.
[0105] 6.3 Finding s using the target Q network j+1,n And the estimated action value of a′:
[0106]
[0107] in, is a state-action pair (s j+1,n ,a′)’s target action value.
[0108] 6.4 Calculate TD (Temporal Difference) target and TD error:
[0109]
[0110] Among them, y j,n For TD goal.
[0111]
[0112] Among them, δ j,n is the TD error.
[0113] 6.5 Back propagate the estimated Q network and find Q(s j,n ,a j,n ;θ) with respect to the gradient of parameter θ:
[0114]
[0115] 6.6 Perform gradient descent to update the estimated parameters of the Q network:
[0116]
[0117] Step 7: Update the exploration rate ε, let ε max =ε max -Δ,ε=max(ε max ,ε min ).
[0118] Step 8, let t = t + 1, if t%C = 0, update the parameter θ of the target Q network - , let θ - =θ.
[0119] Step 9, determine whether t is less than the number of training steps per round T. If so, go to step 3. Otherwise, calculate the cumulative reward for this round and determine whether the estimated Q network parameter θ converges. If so, go to step 10. Otherwise, go to step 2.
[0120] Step 10: Input the state of the NB-IoT user terminal into the estimated Q network, adopt a greedy strategy to select and execute the action with the greatest value, and achieve the optimal configuration of the NB-IoT uplink coverage enhancement parameters.
[0121] As a preferred embodiment of the present invention, the greedy strategy formula is as follows:
[0122]
[0123] Among them, π * (a|s) represents the optimal strategy, q * (s,a) is the optimal action-value function.
[0124] Step 11: The NB-IoT uplink coverage enhancement optimization process ends.
[0125] The above embodiments are merely specific descriptions of feasible implementation methods of the present invention, which are used to enable those skilled in the art to understand the contents of the present invention and implement them accordingly. The present invention is not limited to the above embodiments, and equivalent implementation methods and modifications made by those skilled in the art based on the principles and design ideas of the present invention are all within the protection scope of the present invention.
Claims
1. A NB-IoT uplink coverage enhancement optimization method based on deep reinforcement learning, characterized in that: The steps include: Step 1: Construct two neural networks with the same structure in DDQN: the estimated Q network and the target Q network; initialize the network parameters θ of the estimated Q network and the target Q network parameters θ - , let θ - =θ; initialize the experience replay pool R; initialize the algorithm parameters learning step α, target Q network parameter update time step C, number of training steps per round T, discount factor γ, maximum exploration rate ε max and the minimum value ε min , exploration rate reduction Δ, exploration rate ε = ε max , the number of samples m for batch gradient descent; Step 2, initialize the uplink communication parameters of the user terminal in the NB-IoT communication environment: the number of repeated transmissions, the transmission bandwidth, the MCS level and the number of transmission block resource units, and the initial time step is t=0; Step 3: The agent obtains the status information of the user terminal from the NB-IoT communication environment and obtains the status s t , the state s t Input estimation Q network, select action a according to ∈-greedy strategy t ; Step 4: t Convert to the parameter allocation form of NB-IoT uplink coverage enhancement, configure the NB-IoT user terminal, change the user terminal status, perform uplink data transmission, collect SNR (signal-to-noise ratio) and transmission delay data, and calculate the reward value; Step 5: The agent obtains the reward r after executing the action from the NB-IoT communication environment t+1 and status t+1 , the state transfer sample (s t ,a t ,r t+1 ,s t+1 ) is stored in the experience replay pool; Step 6: Determine whether the number of samples in the experience pool is less than the batch sampling number m. If so, go to step 3. Otherwise, randomly select a batch of m samples (s j,n ,a j,n ,r j+1,n ,s j+1,n ), n∈[1,m], train the estimated Q network, and update the parameters θ of the estimated Q network by gradient descent through neural network gradient back propagation; Step 7: Update the exploration rate ε, let ε max =ε max -Δ,ε=max(ε max ,ε min ); Step 8, let t = t + 1, if t%C = 0, update the parameter θ of the target Q network - , let θ - =θ; Step 9, determine whether t is less than the number of training steps per round T. If so, go to step 3. Otherwise, calculate the cumulative reward for this round and determine whether the estimated Q network parameter θ converges. If so, go to step 10. Otherwise, go to step 2. Step 10: Input the state of the NB-IoT user terminal into the estimated Q network, adopt a greedy strategy to select and execute the action with the greatest value, and achieve the optimal configuration of the NB-IoT uplink coverage enhancement parameters; Step 11: The NB-IoT uplink coverage enhancement optimization process ends.
2. According to claim 1, a method for optimizing NB-IoT uplink coverage enhancement based on deep reinforcement learning is characterized in that: In step 2, the uplink communication parameters of the user terminal in the NB-IoT communication system are initialized as follows: the number of repeated transmissions is 1; the transmission bandwidth is 180kHz, the subcarrier spacing is 15kHz, the number of subcarriers is 12; the MSC level is 0; and the number of transport block resource units is 1.
3. According to claim 1, a method for optimizing NB-IoT uplink coverage enhancement based on deep reinforcement learning is characterized in that: In step 3, the state space S = {I Rep ,I BW ,I MCS ,I RU ,delay,rel},state s t ∈S; where: I Rep The index value of the number of repeated transmissions, the value range is: 0-7; I BW The transmission bandwidth index value ranges from 0 to 4. I MCS MCS level index value, value range: 0-10 for single subcarrier transmission; 0-12 for multi-subcarrier transmission; I RU The index value of the number of transport block resource units, the value range is: 0-7; delay is the uplink data transmission delay, which is a real number; rel is the transmission reliability, which has two values: 0 when BLER is greater than 0.1; 1 when BLER is less than or equal to 0.1; The ∈-greedy strategy formula is as follows: Where π(a|s) is the probability of selecting action a in state s, and |A(s)| is the number of actions that can be selected in state s; The action space A is defined as I Rep ,I BW ,I MCS ,I RU There are three possible changes in each value: plus one, unchanged, and minus one. Action a t ∈A.
4. According to claim 1, a method for optimizing NB-IoT uplink coverage enhancement based on deep reinforcement learning is characterized in that: In step 4, when the NB-IoT uplink coverage enhancement optimization problem is constructed as a deep reinforcement learning process, reliable uplink data transmission (BLER ≤ 10%) is used as a constraint condition, and minimizing the uplink communication delay is used as the optimization goal. The specific expression formula is as follows: min delay stBLER≤0.1 Wherein, delay is the uplink transmission delay of the user terminal, and BLER is the uplink transmission block error rate of the user terminal; The calculation formula for defining the reward r is as follows: Among them, delay max is the maximum delay of uplink communication; whether BLER is greater than 0.1, based on the SNR and SNR of the current uplink channel Th By comparison, when SNR ≥ SNR Th When BLER≤0.1; otherwise BLER>0.1; SNR Th It is the signal-to-noise ratio corresponding to a BLER of 0.1 under the current uplink transmission parameter configuration.
5. According to claim 1, a method for optimizing NB-IoT uplink coverage enhancement based on deep reinforcement learning, characterized in that: In step 6, the estimated Q network is trained and the following steps are performed to update the parameter θ: 6.1 Perform forward propagation on the estimated Q network and obtain the sample estimated action value as follows: in, is a state-action pair (s j,n ,a j,n )’s estimated action value; 6.2 Selections j+1,n Actions: Where a′ is the state s j+1,n The action corresponding to the maximum estimated action value; 6.3 Finding s using the target Q network j+1,n And the estimated action value of a′: in, is a state-action pair (s j+1,n ,a′)’s target action value; 6.4 Calculate TD (Temporal Difference) target and TD error: Among them, y j,n For TD target; Among them, δ j,n is the TD error; 6.5 Back propagate the estimated Q network and find Q(s j,n ,a j,n ;θ) with respect to the gradient of parameter θ: 6.6 Perform gradient descent to update the estimated parameters of the Q network:
6. The NB-IoT uplink coverage enhancement optimization method based on deep reinforcement learning according to claim 1, characterized in that: In step 10, a greedy strategy is used to select actions to achieve optimal configuration of NB-IoT uplink coverage enhancement parameters. The greedy strategy formula is as follows: Among them, π * (a|s) represents the optimal strategy, q * (s,a) is the optimal action-value function.
Citation Information
Patent Citations
Deep reinforcement learning-based heterogeneous cellular network joint optimization method
CN108848561A
Improved deep reinforcement learning method and system based on Double DQN
CN111461321A
