A coexistence method based on Q-learning in non-cooperative environment

Abstract

The present invention relates to a coexistence method based on Q learning under non-cooperation, and belongs to the field of communication technology. The present invention comprises the following steps: S1: setting a D2D duty cycle action set and a state set of a base station intelligent body, initializing a matrix as a zero-order matrix, and the base station detects the initial state information of a channel; S2: the base station selects an action according to an ε-greedy selection strategy; S3: the base station detects the channel condition in the Wi-Fi channel use phase by executing the selected action, thereby calculating the throughput and spectrum utilization of the system under the current action, and obtaining the reward of the currently selected action; S4: updating the Q table according to the Q table formula of Q learning, and the base station enters the next state; S5: repeatedly executing S2 to S4 until the selected action reaches the target state, and an iteration is terminated; S6: setting t←t+1, repeatedly executing steps S2 to S5, until the Q matrix converges, and selecting the optimal D2D duty cycle allocation sequence according to the Q matrix. The present invention enables the base station to learn and adapt to the dynamic behavior of Wi-Fi random traffic. Without knowing the Wi-Fi traffic information, the base station detects the channel usage information and selects the appropriate D2D duty cycle in the face of real-time Wi-Fi traffic through Q learning, so that the entire system can achieve maximum throughput and spectrum utilization, thereby improving spectrum efficiency.

Description

Technical Field

[0001] The invention belongs to the technical field of communications and relates to a coexistence method based on Q learning in a non-cooperative environment. Background Art

[0002] In recent years, with the rapid development of mobile terminals and wireless communications, the mobile traffic generated worldwide has exploded. Faced with the huge data traffic demand, even though some researchers have proposed some key technologies to alleviate the pressure on the licensed spectrum, such as orthogonal multiplexing technology, Multiple-input–multiple-output (MIMO) technology, Device-to-device (D2D) technology, etc., the licensed spectrum is still in a state of resource shortage. Given that the licensed spectrum has been largely utilized, it is difficult to optimize resource allocation on the licensed spectrum.

[0003] When two communication devices are relatively close, D2D communication technology can be used to improve spectrum utilization by sharing the spectrum band of mobile users under the centralized management of mobile base stations. By shortening the communication distance of D2D pairs and controlling the power of their transmitters, D2D communication can effectively improve system throughput, reduce energy consumption and reduce latency. Due to the limited authorized spectrum resources, D2D communication and cellular user communication reuse channels in the authorized spectrum, which will cause interference to cellular users. Therefore, D2D-Unlicensed (D2D-U) technology is proposed to study the scenario of D2D working on unlicensed spectrum.

[0004] In most existing studies, it is assumed that the D2D-U system has perfect knowledge of the Wi-Fi system information. Specifically, only after receiving the required information from the Wi-Fi system, such as Wi-Fi traffic and the number of Wi-Fi connections, can the D2D-U system optimize the parameters within the system to protect the Wi-Fi system in terms of Wi-Fi throughput or outage probability. However, in reality, since it is difficult to establish a dedicated channel between two independent systems and Wi-Fi traffic is usually time-varying, this information about the Wi-Fi system may not be applicable to the D2D-U system.

[0005] The Q-learning algorithm is a reinforcement learning algorithm that can determine the optimal decision-making strategy. It is also an asynchronous dynamic programming method that does not require an environment model. Q-learning is applied to communication technology. By interacting with the environment, the strategy is continuously updated through trial and error learning under the reward or penalty feedback of the environment until the optimal strategy that meets expectations is generated.

[0006] Unlike other model strategies, the D2D-U and Wi-Fi coexistence strategy based on Q learning in non-cooperative conditions does not need to know Wi-Fi traffic information in advance, but makes real-time decisions based on the detected channel information, that is, based on the time when the base station detects the channel used by the Wi-Fi system, the channel is busy, that is, the time when Wi-Fi occupies the channel during transmission and collision, so as to quickly and effectively allocate spectrum resources in real time. Therefore, in non-cooperative conditions, it is of great research value to introduce Q learning to solve the coexistence problem of D2D-U and Wi-Fi in unlicensed spectrum. Summary of the invention

[0007] In view of this, the present invention provides a coexistence method based on Q learning in a non-cooperative environment to maximize the throughput during non-cooperative coexistence in an unlicensed spectrum.

[0008] In order to achieve the above object, the present invention provides the following technical solutions:

[0009] A non-cooperative coexistence method based on Q-learning includes the following steps:

[0010] S1: Set the action set A of D2D duty cycle = {T Dmin , T Dmin +X,T Dmin +2X,...,T Dmax}; and state set S = {S1, S2, S3}, the initialization matrix is ​​a zero-order matrix, and the base station detects the initial state information of the channel;

[0011] S2: The base station selects an action A according to the ε-greedy selection strategy t , A t ∈A;

[0012] S3: The base station performs action A t At the same time, the channel usage is detected during the time period used by the Wi-Fi system, that is, the time when the channel is busy when the Wi-Fi system uses the channel within a time period, so as to calculate the system throughput and spectrum utilization under the current action, and obtain the currently selected action A t The reward r(S t ,A t );

[0013] S4: Update the Q table according to the Q table formula of Q learning, and the base station enters the next state;

[0014] S5: Repeat S2 to S4 until the selected action reaches the target state, ending one iteration;

[0015] S6: Let t←t+1, and repeat steps S2 to S5 until the Q matrix converges, and select the optimal D2D duty cycle allocation sequence according to the Q matrix.

[0016] Further, in step S1, for the action set A={T Dmin , T Dmin +X,T Dmin +2X,...,T Dmax}, each action element in the set represents a different D2D duty cycle, where T Dmin and T Dmax They represent the minimum threshold and maximum threshold of the D2D duty cycle, respectively. X represents the step size. The larger the X, the larger the span between the elements in the action set A, and the fewer the elements in the action set A. On the contrary, the smaller the X, the smaller the span between the elements in the action set A, and the more the elements in the action set A. Here, the value of X is related to the cycle length and the D2D duty cycle threshold. X∈(0,T Dmax -T Dmin ], and should also meet Where Z is an integer.

[0017] For a state set S = {S1, S2, S3}, each state S t The throughput R t and spectrum utilization F t Composition, for state S t The throughput R t , represents the sum of the throughput of the D2D system and the Wi-Fi system, which is defined as:

[0018] R t =R D2D +R WiFi

[0019]

[0020]

[0021] Among them, R D2D and R WiFi Denote the throughput of D2D and Wi-Fi respectively, T D Indicates that in a cycle time resource T F The time occupied by D2D, T WB Represents the time resource T used in the Wi-Fi system W In the busy time of the channel, T W Indicates the time resources allocated by the intelligent base station to the Wi-Fi system.

[0022] For state St The spectrum utilization F t , the spectrum utilization function is defined as:

[0023]

[0024] Where T W Indicates the time resources allocated by the intelligent base station to the Wi-Fi system.

[0025] According to the predefined throughput and spectrum utilization thresholds, the obtained states are divided into three types: low throughput and low spectrum utilization, high throughput and low spectrum utilization, and high throughput and high spectrum utilization. Among them, high throughput and high spectrum utilization is the state target to be achieved by the system, and the definition states are as follows:

[0026]

[0027] Among them, R min and F min They represent the minimum thresholds of system throughput and spectrum utilization respectively.

[0028] Furthermore, in step S2, since the random selection strategy may generate multiple iterations due to repeated selection strategies, the greedy selection strategy may also have a local optimum and fail to reach the global optimum. Therefore, in the present invention, the base station selects an action through the ε-greedy action selection strategy, and the two are combined to better select an action. The ε-greedy strategy is defined as:

[0029]

[0030] When selecting an action, the action is randomly selected with a probability of ε, and the action corresponding to the maximum Q value in the Q table is selected with a probability of 1-ε, that is, the D2D duty cycle corresponding to the maximum Q value.

[0031] Further, in step S3, the ε-greedy strategy is used to select action A t , the base station detects the channel usage when the Wi-Fi system uses the channel, that is, the time when the channel is busy, so as to calculate action A t The corresponding throughput R t and spectrum utilization F t , confirm the status corresponding to the current action.

[0032] In addition, in step S3, when action A is selected t After completion, the reward r(S t ,A t ). The reward function is defined as:

[0033]

[0034] Rewards are only given when the system throughput and spectrum utilization meet the requirements, that is, when high throughput and high spectrum utilization are achieved.

[0035] Further, in step 4, the Q table is updated according to the Q table update formula of Q learning, and the update formula is:

[0036]

[0037] Where α represents the learning rate and α∈(0,1), γ represents the discount factor and 0≤γ<1.

[0038] Further, in step S5, for the Q learning in this article, only when the current state reaches the target state, that is, the current state reaches high throughput and high spectrum utilization, is the iteration process completed.

[0039] Further, in step S6, steps S2 to S5 are repeatedly executed to make the Q expression converge.

[0040] The beneficial effect of the present invention is that by using the Q learning algorithm in a non-cooperative situation to dynamically optimize the resource allocation problem of the D2D system and the Wi-Fi system on the unlicensed spectrum, this greatly utilizes the idle unlicensed spectrum resources, thereby alleviating the problem of crowded and tight licensed spectrum resources, which has a reference significance for the harmonious coexistence of other heterogeneous networks.

[0041] Advantages and beneficial effects of the present invention

[0042] With the rapid development of mobile terminals and wireless communications, the mobile traffic generated globally has exploded, leading to an extreme shortage of existing licensed spectrum resources. Since D2D has the characteristics of short communication distance and low transmission power, it is possible to consider deploying D2D communication in unlicensed frequency bands with abundant spectrum resources to further alleviate the pressure on spectrum demand.

[0043] Most existing studies assume that the D2D-U system can perfectly understand the Wi-Fi system information. In those studies, only after receiving the required information from the Wi-Fi system can the D2D-U system optimize the system parameters to protect the Wi-Fi system in terms of Wi-Fi throughput or interruption probability. However, in reality, since it is difficult to establish a dedicated channel between two independent systems, and the Wi-Fi traffic is usually time-varying, such information about the Wi-Fi system may not be applicable to the D2D-U system. Therefore, the present invention considers how the D2D-U system shares the same unlicensed spectrum with Wi-Fi in the absence of signaling exchange. Although the Wi-Fi traffic is unknown to the D2D-U system, the base station can detect the activity of Wi-Fi in the channel in real time, such as the time when the channel is idle or busy when the Wi-Fi system uses the channel, and then make a reasonable allocation of channel resources through Q learning based on the detected channel information, so as to maximize the system throughput and spectrum utilization.

[0044] The design goal of the present invention is to maximize the throughput of the entire system while ensuring the spectrum utilization of the D2D-U system and the Wi-Fi system. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to make the purpose, technical solution and beneficial effects of the present invention clearer, the present invention provides the following drawings for illustration:

[0046] Figure 1 Schematic diagram of the algorithm flow

[0047] Figure 2 This is the Q-learning framework diagram;

[0048] Figure 3 A network model diagram showing the coexistence of the D2D system and the Wi-Fi system;

[0049] Figure 4 The framework diagram for allocating channel resources to the D2D system and the Wi-Fi system access. DETAILED DESCRIPTION

[0050] The present invention will be described in detail below in conjunction with the accompanying drawings.

[0051] The present invention mainly aims at the coexistence problem of D2D and Wi-Fi in the unlicensed frequency band, and proposes a coexistence method based on Q learning in a non-cooperative environment. Referring to the harmonious coexistence schemes of D2D and Wi-Fi in many studies, the present invention can dynamically optimize the channel resource allocation of the base station based on the Q learning algorithm in a non-cooperative environment without knowing the actual traffic of the Wi-Fi system. The base station can dynamically adjust the duty cycle of D2D according to the real-time network environment. The Q learning framework is as follows Figure 1As shown in the figure, first, the base station acts as an intelligent agent. In a certain state, it selects an action according to the ε-greedy action selection strategy. This action can obtain rewards in a certain environment, and the Q table is updated according to the Q learning update formula. Repeat the above actions until the Q table converges.

[0052] In the coexistence scenario, there are base stations, D2D users and Wi-Fi users. The network model is as follows: Figure 2 As shown in the figure, it is assumed that the D2D system is always in a saturated state and the Wi-Fi is in an unsaturated state. Since the real-time information of Wi-Fi is unknown, the base station, as an intelligent agent, detects the channel state when the Wi-Fi system uses the channel, that is, the busy or idle state. According to the detected channel information, the base station allocates channel resources to D2D and Wi-Fi.

[0053] In Q-learning, we treat base stations in heterogeneous networks as intelligent agents. At a specific moment, the agent observes the state from its environment and takes action. At each decision moment t, the agent will take appropriate actions to maximize the reward at the next moment t+1. In Q-learning, the learned Q value is updated using the immediate reward and the discounted reward and saved in a two-dimensional Q table, and finally the Q table converges.

[0054] In heterogeneous networks, D2D devices and Wi-Fi users coexist in unlicensed frequency bands. Based on the working principle of Q learning, the action set A is defined as {T Dmin , T Dmin +X,T Dmin +2X,...,T Dmax}; and state set S = {S1, S2, S3}.

[0055] For an action set A = {T Dmin , T Dmin +X,T Dmin +2X,...,T Dmax}, each action element in the set represents a different D2D duty cycle, where T Dmin and T Dmax They represent the minimum threshold and maximum threshold of the D2D duty cycle, respectively. X represents the step size. The larger the X, the larger the span between the elements in the action set A. At the same time, the fewer the elements in the action set A are. On the contrary, the smaller the X, the smaller the span between the elements in the action set A. At the same time, the more the elements in the action set A are. Here, the value of X is related to the cycle length and the D2D duty cycle threshold. X∈(0,T Dmax -T Dmin ], and should also meet Where Z is an integer.

[0056] For a state set S = {S1, S2, S3}, each state S in the set t The throughput R t and spectrum utilization F t Composition, for state S t The throughput R t , represents the sum of the throughput of the D2D system and the Wi-Fi system, which is defined as:

[0057] R t =R D2D +R WiFi

[0058]

[0059]

[0060] Among them, R D2D and R WiFi Denote the throughput of D2D and Wi-Fi respectively, T D Indicates that in a cycle time resource T F The time occupied by D2D, T WB Represents the time resource T used in the Wi-Fi system W In the busy time of the channel, T W Indicates the time resources allocated by the intelligent base station to the Wi-Fi system.

[0061] For state S t The spectrum utilization F t It represents the spectrum utilization function in the average sense, and the spectrum utilization function is defined as:

[0062]

[0063] Where T W Indicates the time resources allocated by the intelligent base station to the Wi-Fi system.

[0064] According to the predefined throughput and spectrum utilization thresholds, the obtained states are divided into three types: low throughput and low spectrum utilization, high throughput and low spectrum utilization, and high throughput and high spectrum utilization. Among them, high throughput and high spectrum utilization is the state target to be achieved by the system, and the definition states are as follows:

[0065]

[0066] Among them, R min and F min They represent the minimum thresholds of system throughput and spectrum utilization respectively.

[0067] The algorithm uses the ε-greedy strategy to select actions. Since the random selection strategy may produce multiple iterations due to repeated selection strategies, and the greedy selection strategy may also have local optimal situations and fail to achieve global optimality, a combination of the two can efficiently and accurately select actions. The ε-greedy strategy is defined as:

[0068]

[0069] When selecting an action, the action is randomly selected with a probability of ε, and the action corresponding to the maximum Q value in the Q table is selected with a probability of 1-ε, that is, the D2D duty cycle corresponding to the maximum Q value.

[0070] After selecting an action according to the ε-greedy strategy, the base station performs action A t At the same time, the channel usage is detected to calculate the throughput and spectrum utilization under the current action to obtain the reward r(S t ,A t ).

[0071] Its reward function is defined as:

[0072]

[0073] R min and F min Respectively represent the minimum throughput and minimum spectrum utilization, only when action A t The selected action will be rewarded only after the corresponding throughput and spectrum utilization meet certain conditions. Update the Q value according to the Q learning update formula:

[0074]

[0075] Where α represents the learning rate and α∈(0,1), γ represents the discount factor and 0≤γ<1. The larger the learning rate α, the less the Q value update is based on past experience. On the contrary, the smaller α is, the more the Q value update is based on past experience. γ is the discount factor, which indicates the degree of influence of possible future returns on current returns. The larger its value is, the more important future returns are, and the algorithm pays more attention to long-term benefits. On the contrary, the smaller γ is, the more the algorithm pays attention to short-term benefits.

[0076] Finally, for the non-cooperative Q-learning algorithm proposed in the present invention, the algorithm completes an iteration process only when the current state reaches the target state, that is, the current state of the base station agent is high throughput and high spectrum utilization.

[0077] Finally, it should be noted that the above preferred embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail through the above preferred embodiments, those skilled in the art should understand that various changes can be made in form and details without departing from the scope defined by the claims of the present invention.

Claims

1. A non-cooperative coexistence method based on Q-learning, characterized by: The specific steps are as follows: S1: Set the action set A of D2D duty cycle = {T Dmin , T Dmin +X,T Dmin +2X,...,T Dmax }, and the state set S = {S1, S2, S3}, the initialization matrix is ​​a zero-order matrix, and the base station detects the initial state information of the channel; each action element in the action set represents a different D2D duty cycle, T Dmin and T Dmax They represent the minimum and maximum thresholds of the D2D duty cycle, respectively. X represents the step size. The larger the X, the larger the span between the elements in the action set A, and the fewer the elements in the action set A. On the contrary, the smaller the X, the smaller the span between the elements in the action set A, and the more the elements in the action set A. The value of X is related to the cycle length and the D2D duty cycle threshold. X∈(0,T Dmax -T Dmin ], and should also meet Where Z is an integer; each state S in the state set t The throughput R t and spectrum utilization F t Composition, R t =R D2D +R WiFi , where R D2D and R WiFi Represent the throughput of D2D and Wi-Fi respectively, T D Indicates that in a cycle time resource T F The time occupied by D2D, T WB Represents the time resource T used in the Wi-Fi system W The channel is busy during the time, T W Indicates the time resources allocated by the intelligent base station to the Wi-Fi system; spectrum utilization F t Defined as Where T W Indicates the time resources allocated by the intelligent base station to the Wi-Fi system; according to the predefined throughput and spectrum utilization thresholds, the obtained states are divided into three types: low throughput and low spectrum utilization, high throughput and low spectrum utilization, and high throughput and high spectrum utilization. Among them, high throughput and high spectrum utilization is the state target to be achieved by the system. The definition states are as follows: Among them, R min and F min They represent the minimum thresholds of system throughput and spectrum utilization respectively; S2: The base station selects an action A according to the ε-greedy selection strategy t , A t ∈A; Since the random selection strategy may generate multiple iterations due to repeated selection strategies, the greedy selection strategy may also appear in the local optimum and fail to reach the global optimum. Therefore, the base station of the present invention selects the action through the ε-greedy action selection strategy. The ε-greedy strategy is defined as: When selecting an action, the action is randomly selected with a probability of ε, and the action corresponding to the maximum Q value in the Q table is selected with a probability of 1-ε, that is, the D2D duty cycle corresponding to the maximum Q value; S3: The base station performs action A t , and detect the channel usage at the same time, calculate the system throughput and spectrum utilization under the current action, and obtain the currently selected action A t The reward r(St,At); when action A is selected t After completion, the reward r(St,At) is obtained according to the currently selected action, which is defined as: Rewards will be given only when the system throughput and spectrum utilization meet the requirements, that is, when high throughput and high spectrum utilization are achieved; S4: Update the Q table according to the Q table formula of Q learning, and the base station enters the next state. The update formula is: Where α represents the learning rate and α∈(0,1), γ represents the discount factor and 0≤γ<1; S5: Repeat steps S2 to S4 until the selected action reaches the target state, and one iteration ends. Only when the current state reaches the target state, that is, the current state achieves high throughput and high spectrum utilization, is the iteration process completed; S6: Let t←t+1, and repeat steps S2 to S5 until the Q matrix converges, and select the optimal D2D duty cycle allocation sequence according to the Q matrix.

2. The non-cooperative Q-learning based coexistence method according to claim 1, characterized in that: By repeatedly executing steps S2 to S5, the Q expression in the Q learning algorithm converges, and the optimal D2D duty cycle allocation sequence is selected according to the Q matrix when facing the randomness of Wi-Fi traffic.

Images