DRQN-based nr-u and wifi coexistence network channel access initial contention window optimization method

Abstract

The application discloses an initial contention window optimization algorithm for an NR-U and WiFi coexisting network channel access based on a DRQN, and the algorithm comprises the following steps: (1) initializing a current value network, a target value network and an experience replay pool; (2) an NR-U user and a WiFi node compete for access to a channel; and (3) optimizing the initial contention window size of the NR-U user. The application can effectively improve the throughput of the NR-U system under the premise of guaranteeing the fairness of the coexisting network.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of wireless communication, and in particular to a DRQN-based initial contention window optimization method for NR-U and WiFi coexistence network channel access. BACKGROUND

[0002] With the popularity of smart mobile devices and various emerging mobile applications, mobile communication data volume is growing explosively, which brings heavy burden to current mobile cellular networks. Due to the very limited spectrum resources, it is difficult to expand network capacity by obtaining new licensed frequency bands. In order to cope with the increasing demand for spectrum resources of cellular networks, 3GPP evolved the traditional LTE LAA into 5G NR-U in Release 16, so that the 5G network can run on unlicensed frequency bands. When the NR-U system shares the same unlicensed channel with the WiFi system, it is necessary to prevent the deployment of the NR-U system from causing a serious decline in the performance of the WiFi system. According to the requirements of 3GPP, the deployment of the NR-U system should not have a greater impact on the WiFi system than adding a WiFi system on the same channel. In order to solve this problem, 3GPP has proposed two NR-U and WiFi coexistence network channel access mechanisms, namely LBT mechanism and CSAT mechanism. In the 5G NR-U system, 3GPP TR 38.899 suggests that NR-U users use CAT4 LBT mechanism to compete for access to unlicensed channels.

[0003] In the process of NR-U users competing for access to the channel, if the NR-U system can adaptively optimize and adjust the initial contention window size according to the current coexistence network state, it can effectively improve the access probability of NR-U users and the throughput of the NR-U system. However, the randomness of the CAT4 LBT mechanism makes it unsuitable for the optimization of the initial contention window size of NR-U users using traditional optimization algorithms. With the development of artificial intelligence technology, machine learning is being used more and more widely to solve performance bottlenecks in mobile cellular networks. Reinforcement learning is a typical machine learning method that can obtain an optimized strategy for solving problems through interaction with the environment, and is suitable for dynamically changing network environments. Therefore, the present application applies machine learning methods to the optimization of the initial contention window size of NR-U users in the CAT4 LBT mechanism. SUMMARY

[0004] The technical problem to be solved by the present application is to provide a DRQN-based initial contention window optimization method for NR-U and WiFi coexistence network channel access, which is used to solve the interference problem of the NR-U system to the WiFi system in the NR-U and WiFi coexistence network, while improving the throughput of the NR-U system.

[0005] To solve the above technical problems, the application provides a DRQN-based NR-U and WiFi coexistence network channel access initial contention window optimization method, which comprises the following steps:

[0006] (1) initialize the current value network, target value network and experience replay pool;

[0007] (2) NR-U users and WiFi nodes compete for access to the channel. WiFi nodes use the CSMA / CA mechanism to compete for access to the channel, and NR-U users use the CAT4 LBT mechanism to compete for access to the channel;

[0008] (3) NR-U user initial contention window size optimization. When a new update period arrives, the base station optimizes the initial contention window size of the current update period according to the throughput of the WiFi system and the NR-U system in the previous update period, the Jain fairness index of the coexistence network, the number of remaining NR-U users in the user waiting queue and other parameters. At the same time, update the DRQN parameters according to the latest coexistence network state.

[0009] In the method of the application, in step (1), the current value network, the target value network and the experience replay pool are initialized, and the steps include:

[0010] (1.1) randomly initialize the current value network and target value network parameters;

[0011] (1.2) when the number of experience data samples does not reach the capacity of the experience replay pool, select the action a t according to the current state s t and the output of the current value network. Here, the state is defined as a five-tuple, represented as:

[0012]

[0013] Where, W l represents the initial contention window size of the NR-U user in the previous period, R l represents the throughput of the WiFi system in the previous period, R w represents the throughput of the NR-U system in the previous period, B is a binary parameter representing whether the Jain fairness index of the coexistence network in the previous period satisfies the constraint condition, represents the number of remaining NR-U users in the user waiting queue, represents a set of state values.

[0014] The Jain fairness index J(R w , R l ) is defined as:

[0015]

[0016] wherein N represents the number of WiFi nodes in the coexistence network. In order to guarantee the data transmission requirement of the WiFi system, the Jain fairness index of the coexistence network should satisfy certain constraint condition, that is:

[0017] J(R w , R l ) > Θ,

[0018] wherein Θ represents the constraint threshold of the Jain fairness index of the coexistence network. If the Jain fairness index of the coexistence network satisfies the constraint condition, the corresponding parameter B is 1; otherwise, B is 0.

[0019] The action is defined as: selecting an initial contention window size for the NR-U user, that is:

[0020]

[0021] wherein W1, …, W L represent the selectable initial contention window sizes, L represents the number of selectable initial contention window sizes, and the set of action values is denoted as W.

[0022] (1.3) performing the action a t , that is, selecting an initial contention window size for the NR-U user;

[0023] (1.4) calculating the reward and obtaining the next update cycle state s t+1 .

[0024] Herein, the reward function is defined as: when the NR-U system throughput is less than the WiFi system throughput, the reward function is the NR-U system throughput; when the NR-U system throughput exceeds the WiFi system throughput, if the Jain fairness index of the coexistence network satisfies the constraint condition, the reward function is the NR-U system throughput; if the constraint condition is not satisfied, the reward function is the negative value of the throughput, that is:

[0025]

[0026] (1.5) storing the current experience sample data into the experience replay pool, and jumping to step (1.1) until the number of experience data samples reaches the capacity of the experience replay pool.

[0027] In the present application, in step (2), the NR-U user and the WiFi node compete for access to the channel, and the steps thereof include:

[0028] (2.1) when a new NR-U user arrives at the coexistence network, the user first enters the user waiting queue to wait. If the queue is full, the user is rejected;

[0029] (2.2) WiFi node adopts CSMA / CA mechanism to compete for access to the channel, and the NR-U user adopts CAT4 LBT mechanism to compete for access to the channel.

[0030] In the present application, in step (3), the initial contention window size of the NR-U user is optimized, and the steps thereof include:

[0031] (3.1) Obtain the throughput of the WiFi system and the NR-U system in the previous update period, the number of remaining NR-U users in the user waiting queue, and calculate the Jain fairness index to determine whether it meets the constraint condition;

[0032] (3.2) Calculate the reward of the previous update period

[0033] (3.3) Store the current experience sample data in the experience replay pool;

[0034] (3.4) Select the action a t according to the current state s t and the output of the current value network.

[0035] (3.5) Perform the action a t , that is, select an initial contention window size of an NR-U user;

[0036] (3.6) Randomly extract a batch of samples from the experience replay pool, and update the current value network parameters by minimizing the gradient of the loss function . Here, the loss function is defined as the mean square error between the current Q value and the target Q value, that is:

[0037]

[0038]

[0039] In the formula, θ represents the current value network parameters, θ' represents the target value network parameters, represents the output of the target value network, Q(s t , a t | θ) represents the output of the current value network, represents the current reward function, and γ represents the discount factor.

[0040] (3.7) Every M update period, assign the current value network parameters to the target value network.

[0041] The present application has the following beneficial effects:

[0042] While ensuring fairness in coexisting networks, it can effectively improve the throughput of NR-U systems. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of the NR-U and WiFi coexistence network system model in this invention.

[0044] Figure 2 This is a schematic diagram of the process of initializing the current value network, the target value network, and the experience replay pool in this invention.

[0045] Figure 3 This is a schematic diagram illustrating the process of NR-U users and WiFi nodes competing for access to the channel in this invention.

[0046] Figure 4 This is a schematic diagram of the steps for updating the initial contention window size of NR-U users in this invention.

[0047] Figure 5 This is a schematic diagram of the method flow of the present invention. Detailed Implementation

[0048] This invention discloses an initial contention window optimization method for channel access in a DRQN-based NR-U and WiFi coexistence network. The coexistence network consists of a WiFi system and an NR-U system. The WiFi system comprises one WiFi access point and N WiFi nodes, and is in a saturated state. The NR-U system consists of one base station and dynamically arriving NR-U users. The arrival of NR-U users follows a Poisson distribution, and upon arrival, users enter a user waiting queue of length K. If the queue is full when a user arrives, the user is rejected. WiFi nodes and NR-U users share an unlicensed channel. WiFi nodes compete for channel access using the CSMA / CA mechanism, while NR-U users compete for channel access using the CAT4 LBT mechanism. The CAT4 LBT access process consists of two time periods: a delay period (ICCA phase) and a contention period (ECCA phase). If the channel remains idle during the ICCA phase, the NR-U user will immediately transmit data; otherwise, the NR-U user will enter the ECCA phase and compete with the WiFi node for the unlicensed channel.

[0049] An embodiment of the present invention provides an initial contention window optimization algorithm for NR-U and WiFi coexistence network channel access based on DRQN, which mainly includes three steps: (1) initializing the neural network and the experience replay pool; (2) NR-U users and WiFi nodes compete for access to the channel; and (3) optimizing the initial contention window size of NR-U users.

[0050] First, initialize the current value network, the target value network, and the experience replay pool;

[0051] Secondly, NR-U users and WiFi nodes compete for access to the channel. WiFi nodes use the CSMA / CA mechanism to compete for access to the channel, and NR-U users use the CAT4 LBT mechanism to compete for access to the channel;

[0052] Finally, the initial contention window size optimization of NR-U users. After reaching a new update period, the base station optimizes the initial contention window size of the current update period according to the throughput of the WiFi system and the NR-U system in the previous update period, the Jain fairness index of the coexistence network, the number of remaining NR-U users in the user waiting queue and other parameters. At the same time, update the DRQN parameters according to the latest coexistence network state.

[0053] As shown in Figure 5 , the implementation steps of the NR-U and WiFi coexistence network channel access initial contention window optimization algorithm based on DRQN disclosed by the embodiment of the application are as follows:

[0054] (1) Initialize the neural network and the experience replay pool:

[0055] Initialize the current value network, the target value network and the experience replay pool, and the steps include:

[0056] 1) Randomly initialize the parameters of the current value network and the target value network;

[0057] 2) When the number of experience data samples does not reach the capacity of the experience replay pool, select the action a t according to the current state s t and the output of the current value network. Here, the state is defined as a five-tuple, which is expressed as:

[0058]

[0059] Where, W l represents the initial contention window size of the NR-U user in the previous period, R l represents the throughput of the WiFi system in the previous period, R w represents the throughput of the NR-U system in the previous period, B is a binary parameter indicating whether the Jain fairness index of the coexistence network in the previous period satisfies the constraint condition, represents the number of remaining NR-U users in the user waiting queue, represents a set composed of state values.

[0060] The Jain fairness index J(R w , R l ) is defined as:

[0061]

[0062] wherein N denotes the number of WiFi nodes in the coexistence network. In order to guarantee the data transmission requirement of the WiFi system, the Jain fairness index of the coexistence network should satisfy certain constraint condition, i.e.

[0063] J(R w , R l ) > Θ,

[0064] wherein Θ denotes the constraint threshold of the Jain fairness index of the coexistence network. If the Jain fairness index of the coexistence network satisfies the constraint condition, the corresponding parameter B is 1; otherwise, B is 0.

[0065] The action is defined as: selecting an initial contention window size for the NR-U user, i.e.

[0066]

[0067] wherein W1, …, W L denote the selectable initial contention window sizes, L denotes the number of selectable initial contention window sizes, denotes the set of values of the action.

[0068] 3) performing the action a t , i.e. selecting an initial contention window size for the NR-U user;

[0069] 4) calculating the reward obtaining the next updated period state s t+1 .

[0070] Herein, the reward function is defined as: when the NR-U system throughput is less than the WiFi system throughput, the reward function is the NR-U system throughput; when the NR-U system throughput exceeds the WiFi system throughput, if the Jain fairness index of the coexistence network satisfies the constraint condition, the reward function is the NR-U system throughput; if the constraint condition is not satisfied, the reward function is the negative value of the throughput, i.e.

[0071]

[0072] 5) storing the current experience sample data into the experience replay pool, and jumping to step 1) until the number of experience data samples reaches the capacity of the experience replay pool.

[0073] The specific operation can be seen from the flowchart of Figure 2 .

[0074] (2) NR-U user and WiFi node compete for access channel:

[0075] The NR-U user and the WiFi node compete for the access channel, and the steps include:

[0076] 1) When a new NR-U user arrives in the coexistence network, it first enters the user waiting queue. If the queue is full, the user is rejected;

[0077] 2) WiFi nodes use the CSMA / CA mechanism to compete for access to the channel, and NR-U users use the CAT4 LBT mechanism to compete for access to the channel.

[0078] The specific operation can be seen from the flowchart of Figure 3 .

[0079] (3) NR-U user initial contention window size optimization:

[0080] The steps of NR-U user initial contention window size optimization include:

[0081] 1) Obtain the throughput of the WiFi system and the NR-U system in the previous update period, the number of remaining NR-U users in the user waiting queue, and calculate the Jain fairness index to determine whether it meets the constraint condition;

[0082] 2) Calculate the reward in the previous update period

[0083] 3) Store the current experience sample data in the experience replay pool;

[0084] 4) According to the current state s t and the output of the current value network, select the action a t .

[0085] 5) Perform action a t , that is, select an initial contention window size of an NR-U user;

[0086] 6) Randomly extract a batch of samples from the experience replay pool, and update the current value network parameters by minimizing the gradient of the loss function . Here, the loss function is defined as the mean square error between the current Q value and the target Q value, that is:

[0087]

[0088]

[0089] Where θ represents the current value network parameters, θ' represents the target value network parameters, represents the output of the target value network, and Q(s t , a t | θ) represents the output of the current value network, represents the current reward function, and γ represents the discount factor;

[0090] 7) every M update cycles, assign the current value network parameters to the target value network.

[0091] The specific operation can be seen from the flowchart of Figure 4 .

[0092] Although the present application is illustrated and described with reference to the embodiments, it is understood that various changes and modifications can be made to the application without departing from the scope of the claims.

Claims

1. A method for DRQN-based NR-U and WiFi coexistence network channel access initial contention window optimization, characterized in that, Comprising the following steps: (1) initializing the current value network, the target value network and the experience replay pool; (2) NR-U users and WiFi nodes compete for access to the channel; (3) after reaching a new update cycle, the base station optimizes the initial contention window size of the NR-U user in the current update cycle according to the throughput of the WiFi system and the NR-U system, the Jain fairness index of the coexistence network and the number of remaining NR-U users in the user waiting queue in the previous update cycle, and updates the DRQN parameters according to the latest coexistence network state; In step (1), the current value network, the target value network and the experience replay pool are initialized, and the steps include: (1.1) randomly initialize the current value network and the target value network parameters; (1.2) When the number of experience data samples does not reach the experience replay pool capacity, select an action according to the current state and the output of the current value network ; (1.3) performing an action i.e. selecting an initial contention window size for one NR-U user; (1.4) computing a reward , obtaining a next update period state ; (1.5) store the current experience sample data into the experience replay pool, and go to step (1.1) until the number of experience data samples reaches the capacity of the experience replay pool; In step (1.2), the state is defined as a five-tuple, represented as: ； wherein, denotes the initial contention window size of the NR-U users in the previous period, denotes the throughput of the WiFi system in the previous period, denotes the throughput of the NR-U system in the previous period, is a binary variable indicating whether the Jain's fairness index of the coexistence network in the previous period satisfies the constraint condition or not, denotes the number of NR-U users remaining in the user waiting queue, denotes a set of state values; The action is defined as: selecting an initial contention window size for the NR-U user, that is: ； wherein, represents a selectable initial contention window size, L represents a number of selectable initial contention window sizes, represents a set of action values; Jain fairness index is defined as: ； Where N represents the number of WiFi nodes in the coexistence network; the constraint condition that the Jain fairness index of the coexistence network should satisfy is: ； wherein, represents a coexistence network Jain fairness index constraint condition threshold; if the coexistence network Jain fairness index satisfies the constraint condition, the corresponding parameter is 1; otherwise, is 0. 2.The method of claim 1, wherein, In step (1.4), the reward function is defined as: when the NR-U system throughput is less than the WiFi system throughput, the reward function is the NR-U system throughput; when the NR-U system throughput exceeds the WiFi system throughput, if the Jain fairness index of the coexistence network satisfies the constraint condition, the reward function is the NR-U system throughput; if the constraint condition is not satisfied, the reward function is the negative value of the throughput. 3.The method of claim 1, wherein, In step (2), the NR-U users and WiFi nodes compete for access to the channel, and the steps include: (2.1) when a new NR-U user arrives at the coexistence network, it first enters the user waiting queue to wait, and if the queue is full, the user is refused; (2.2) WiFi nodes compete for access to the channel using the CSMA / CA mechanism, and NR-U users compete for access to the channel using the CAT4 LBT mechanism.

4. The method of claim 1, wherein the DRQN-based NR-U and WiFi coexistence network channel access initial contention window optimization method is characterized by, In step (3), the initial contention window size of the NR-U user is optimized, and the steps include: (3.1) obtain the throughput of the WiFi system and the NR-U system, the number of remaining NR-U users in the user waiting queue, and calculate the Jain fairness index to determine whether it satisfies the constraint condition; (3.2) Calculate the reward of the previous update period ; (3.3) storing the current experience sample data into the experience replay pool; (3.4) selecting an action according to the current state and the output of the current value network ; (3.5) performing an action i.e. selecting an initial contention window size for one NR-U user; (3.6) Randomly sample a batch of samples from the experience replay pool, minimize the gradient of the loss function updating the current value network parameters; the loss function is defined as the mean squared error between the current Q value and the target Q value; (3.7) every M update cycles, assign the current value network parameters to the target value network.

Images