Downlink dynamic spectrum allocation method based on flow prediction and DDPG model

Abstract

The invention discloses a dynamic spectrum allocation method based on user flow demand prediction and a depth deterministic policy gradient, which aims at performing spectrum allocation on wireless network users in the same base station range in a wireless network communicating through orthogonal frequency division multiple access, and solves the problem of insufficient spectrum utilization rate of the wireless network. And an artificial intelligence technology is introduced to predict user traffic information to help to optimize a spectrum allocation problem. In the prediction part, a transform model architecture is adopted for prediction, so that the accuracy under a long-time sequence is ensured; in a network optimization algorithm part, a depth deterministic strategy gradient model is adopted, and the actual total throughput is used as an optimization target by utilizing the advantages of reinforcement learning in a global optimization problem, so that a better spectrum allocation effect is realized, and the throughput and communication benefits are improved.

Description

Technical Field

[0001] The present invention relates to the field of wireless network spectrum allocation, and in particular proposes a downlink-oriented wireless network dynamic spectrum allocation method based on traffic prediction. Background Art

[0002] From the perspective of wireless network service providers, licensed spectrum has always been the most expensive and scarce resource. With the dramatic growth of wireless network capacity demand, spectrum has become more precious. Traditional resource management methods are usually based on user buffer status reports (BSRs). This static resource allocation strategy may not be able to effectively cope with dynamic traffic demand changes. In previous generations of cellular networks, fixed spectrum allocation (FSA) has been a popular method for allocating BW among different APs. When the solution is implemented, an initial capacity plan is performed and BW is allocated to APs based on the maximum demand for services expected during peak hours and available resources. After that, the BW allocation remains static and is independent of the dynamic capacity requirements of different APs in the network. Compared with FSA, DSA (Dynamic Spectrum Allocation) allocates spectrum based on available resources according to the instantaneous demand of APs. DSA is triggered after a specific time interval, estimates the load of the next time interval, calculates the spectrum demand, and detects the instantaneous demand of users by establishing a sensing window, thereby performing dynamic perceptual spectrum allocation.

[0003] DSA technology has been very popular since its inception because it improves the spectral efficiency of the network by allowing underutilized frequency bands to be used in an efficient manner. With the development of wireless technology, DSA technology has also been developing. For example, different methods have emerged in the cognitive radio (CR) environment, where DSA is used to allocate idle channels to Secondary Users (SU), which are usually unlicensed and have lower priority in the network, based on fuzzy logic, Q-learning, random rounding algorithms, etc. to learn, estimate and allocate the required spectrum. For example, a DSA technology based on reinforcement learning (RL) is used to allocate spectrum to IoT users in cellular networks. In this method, the authors successfully demonstrated that DSA technology can be used to identify underutilized spectrum in the network for reuse in sensor-assisted IoT networks, thereby improving the reusability of spectrum. There are also works that study dynamic frequency and bandwidth allocation (DFBA) based on machine learning, which focuses on spectrum allocation for small cells in cellular networks. The authors proposed a technique to learn and predict Long Term Evolution (LTE) KPIs (e.g., SINR per resource block, medium access control (MAC) level throughput, latency, etc.) and allocate / rearrange spectrum allocation for LTE-based SCs in the network. However, this work also does not take into account that the DSA algorithm based on such traffic prediction is prone to fall into local optimality when facing a global optimization problem, and cannot globally optimize resources to maximize resource utilization efficiency. For example, this paper takes spectrum utilization as the optimization objective, but if throughput is used as the optimization objective, the DSA algorithm is no longer suitable. The opportunity to improve spectrum efficiency and system fairness by allocating scarce spectrum to the current load of the unit. Summary of the invention

[0004] Purpose of the invention: The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide a method for dynamic spectrum allocation in a wireless network based on traffic prediction for downlink, which can not only effectively meet the traffic needs of users, but also improve spectrum utilization and throughput, and achieve higher economic benefits.

[0005] Technical solution, in order to solve the above technical problems, the present invention proposes a downlink dynamic spectrum allocation method based on traffic prediction and DDPG model, the method comprising the following steps:

[0006] Step 1) Establish a wireless network user traffic demand data set and form a feature set;

[0007] Step 2) Using the feature set constructed in step 1, the transformer architecture is used to predict the user data packet request result for the next spectrum allocation window;

[0008] Step 3) constructing a base station distribution and scenario model for the wireless network downlink, as well as user distribution and access mode;

[0009] Step 4) Use the network model and access method set in step 3 to build a DDPG reinforcement learning model, set the state space, action space and reward function, input the number of user traffic packets predicted in step 2, use reinforcement learning to optimize the global problem, and simulate the best result of wireless network spectrum allocation.

[0010] Further, the specific method of step 1) is as follows:

[0011] By using a packet capture tool on the user side to complete data collection, the complete dataset includes six features, as shown below:

[0012] F1: the time when the data packet arrives / leaves;

[0013] F2: number of data packets;

[0014] F3: Whether the packet is uplink or downlink;

[0015] F4: source IP address;

[0016] F5: Target IP address;

[0017] F6: Communication protocol used during transmission;

[0018] F3: whether the data packet is uplink or downlink is used to determine whether it is uplink or downlink. F2: the number of data packets represents the traffic demand of users in the wireless network. F6: the communication protocol used during transmission is used to determine the size of the data packet.

[0019] Further, the specific method of step 2) is as follows:

[0020] Step 2.1) normalize the numerical features of the feature set;

[0021] Step 2.2) Define the layer structure of the model, including multi-head self-attention mechanism, feedforward neural network layer, and position encoding;

[0022] Step 2.3) Using the feature set obtained in step 2.1), the transformer architecture is used for prediction. The user UE predicted by the transformer architecture is k The number of demand data packets generated in the next time window τ is L k , the size of the data packet is np, then the user UE k The expected demand rate is:

[0023]

[0024] Assume that user UE k When entering the range of base station BSm, the actual rate is higher than that of user UE k The time of the demand rate is D(B k m ), does not meet user requirements, that is, the actual rate is lower than the user UE k The time of the demand rate L((B k m )for:

[0025]

[0026] Actual user UE k The traffic throughput is expressed as:

[0027] T m,k =R d *D(B k m )+R m,k j,t *L(B k m ) (3).

[0028] is the achievable rate of user UEk associated with base station BSm in time slot t of frame j.

[0029] Further, the specific method of step 3) is as follows:

[0030] In the wireless network downlink, there are S base stations, each distributed in the center of a circular area with a radius of D. All base stations use orthogonal frequency division multiple access technology to connect users. U users are randomly distributed within the base station coverage radius of D, and each sends a data packet request to the base station. UE k Only the nearest base station is visited, and the UE must be within the coverage of the relevant base station; each base station serves multiple associated UEs through OFDMA, performs spectrum allocation scheduling after a fixed time interval, and transmits to each subcarrier at the maximum transmission ratio;

[0031] The BSs set is represented by M = {M|M = 1, 2, ..., S}, the user UE set is represented by K = {K|K = 1, 2, ... U}, and UE k The distance from the serving BS in frame j is The large-scale channel gain is Where β is the path loss exponent. In time slot t of frame j, the user UE associated with base station BSm k The achievable rate is:

[0032]

[0033] Where W0 is the subcarrier spacing;

[0034] p k j,t,n It is allocated to the user UE k The transmitted power on subcarrier n in time slot t of frame j;

[0035] is allocated to user UE in time slot t of frame j k A set of subcarriers;

[0036] N0 is the noise power spectral density;

[0037] It is a user UE k The small-scale channel is the fading factor, on subcarrier n in time slot t of frame j;

[0038] The total transmission power of each BS is expressed as P max , when the transmit power is evenly distributed among the subcarriers, that is, W max is the number of subcarriers, then the instantaneous achievable rate in formula (4) is:

[0039]

[0040] Among them, σ 2 =N0W max is the variance of random Gaussian noise.

[0041] Further, the specific method of step 4) is as follows:

[0042] Step 4.1) State space: The state space of the agent is determined by the observation of the XR device. The state at the tth time slot can be expressed as:

[0043] s t ={n',p i (t),h t ,F t l ,f t e}

[0044] Where n' represents the relevant information of a specific user, which comes from the feature set in step 1); p i (t) represents the number of traffic packets predicted by user i at time t, which comes from the result predicted by the transformer model in step 2); h t Represents the gain of the communication channel between the user and the base station; F l tIndicates the current maximum idle computing resources available for the base station; f t e Indicates the computing resources currently allocated to the user by the server;

[0045] Step 4.2) Action Space, the action space is the spectrum resources allocated to each user, which is further expanded to consider the allocation of computing resources:

[0046] a t ={α i (t),r i (t)}

[0047] Among them, α i (t) is the amount of spectrum resources suitable for user i at t; r i (t) is the amount of computing resources that user i is suitable to be allocated at t;

[0048] Step 4.3) Reward function, transform the action space a t Acting in the reinforcement learning environment, the agent obtains corresponding rewards. Considering that the goal of reinforcement learning is to maximize the expectation of cumulative rewards, the reward function is set as:

[0049]

[0050] Among them, the first term T i (t) is used to measure the network throughput. The calculation formula is the same as the actual user throughput in step 2. M is the maximum number of training iterations, and λ is the penalty factor, which is used to calculate the penalty term for spectrum resource allocation.

[0051] Beneficial effects: Compared with the prior art, the technical solution of the present invention has the following beneficial technical effects:

[0052] Compared with the traditional dynamic spectrum allocation (DSA) method, this paper uses a method combined with traffic prediction to better meet user needs. At the same time, by adopting the DDPG model and taking advantage of reinforcement learning in global problems, it solves the shortcomings of DSA in the face of global optimization problems and improves the actual throughput of users. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is the specific implementation flow chart of this research method.

[0054] Figure 2 It is the wireless network topology scenario diagram studied in this paper.

[0055] Figure 3 This is a comparison chart between the spectrum allocation result obtained by this method based on prediction and the DSA method based on dynamic perception. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0057] like Figure 1 As shown, the present invention proposes a downlink dynamic spectrum allocation method based on traffic prediction and DDPG model, the method comprising the following steps:

[0058] Step 1) Establish a wireless network user traffic demand data set and form a feature set;

[0059] Step 2) Using the feature set constructed in step 1, the transformer architecture is used to predict the user data packet request result for the next spectrum allocation window;

[0060] Step 3) constructing a base station distribution and scenario model for the wireless network downlink, as well as user distribution and access mode;

[0061] Step 4) Use the network model and access method set in step 3 to build a DDPG reinforcement learning model, set the state space, action space and reward function, input the number of user traffic packets predicted in step 2, use reinforcement learning to optimize the global problem, and simulate the best result of wireless network spectrum allocation.

[0062] Further, the specific method of step 1) is as follows:

[0063] By using a packet capture tool on the user side to complete data collection, the complete dataset includes six features, as shown below:

[0064] F1: the time when the data packet arrives / leaves;

[0065] F2: number of data packets;

[0066] F3: Whether the packet is uplink or downlink;

[0067] F4: source IP address;

[0068] F5: Target IP address;

[0069] F6: Communication protocol used during transmission;

[0070] F3: whether the data packet is uplink or downlink is used to determine whether it is uplink or downlink. F2: the number of data packets represents the traffic demand of users in the wireless network. F6: the communication protocol used during transmission is used to determine the size of the data packet.

[0071] Further, the specific method of step 2) is as follows:

[0072] Step 2.1) normalize the numerical features of the feature set;

[0073] Step 2.2) Define the layer structure of the model, including multi-head self-attention mechanism, feedforward neural network layer, and position encoding;

[0074] Step 2.3) Using the feature set obtained in step 2.1), the transformer architecture is used for prediction. The user UE predicted by the transformer architecture is k The number of demand data packets generated in the next time window τ is L k , the size of the data packet is np, then the user UE k The expected demand rate is:

[0075]

[0076] Assume that user UE k When entering the range of base station BSm, the actual rate is higher than that of user UE k The time of the demand rate is D(B k m ), does not meet user requirements, that is, the actual rate is lower than the user UE k The time of the demand rate L((B k m )for:

[0077]

[0078] Actual user UE k The traffic throughput is expressed as:

[0079] T m,k =R d *D(B k m )+R m,k j,t *L(B k m ) (3).

[0080] is the achievable rate of user UEk associated with base station BSm in time slot t of frame j.

[0081] Further, the specific method of step 3) is as follows:

[0082] In the wireless network downlink, there are S base stations, each distributed in the center of a circular area with a radius of D. All base stations use orthogonal frequency division multiple access technology to connect users. U users are randomly distributed within the base station coverage radius of D, and each sends a data packet request to the base station. UE kOnly the nearest base station is visited, and the UE must be within the coverage of the relevant base station; each base station serves multiple associated UEs through OFDMA, performs spectrum allocation scheduling after a fixed time interval, and transmits to each subcarrier at the maximum transmission ratio;

[0083] The BSs set is represented by M = {M|M = 1, 2, ..., S}, the user UE set is represented by K = {K|K = 1, 2, ... U}, and UE k The distance from the serving BS in frame j is The large-scale channel gain is Where β is the path loss exponent. In time slot t of frame j, the user UE associated with base station BSm k The achievable rate is:

[0084]

[0085] Where W0 is the subcarrier spacing;

[0086] It is allocated to the user UE k The transmitted power on subcarrier n in time slot t of frame j;

[0087] is allocated to user UE in time slot t of frame j k A set of subcarriers;

[0088] N0 is the noise power spectral density;

[0089] It is a user UE k The small-scale channel is the fading factor, on subcarrier n in time slot t of frame j;

[0090] The total transmission power of each BS is expressed as P max , when the transmit power is evenly distributed among the subcarriers, that is, W max is the number of subcarriers, then the instantaneous achievable rate in formula (4) is:

[0091]

[0092] Among them, σ 2 =N0W max is the variance of random Gaussian noise.

[0093] Further, the specific method of step 4) is as follows:

[0094] Step 4.1) State space: The state space of the agent is determined by the observation of the XR device. The state at the tth time slot can be expressed as:

[0095] s t ={n',p i (t),h t ,F t l ,f t e}

[0096] Where n' represents the relevant information of a specific user, which comes from the feature set in step 1); p i (t) represents the number of traffic packets predicted by user i at time t, which comes from the result predicted by the transformer model in step 2); h t Represents the gain of the communication channel between the user and the base station; F l t Indicates the current maximum idle computing resources available for the base station; f t e Indicates the computing resources currently allocated to the user by the server;

[0097] Step 4.2) Action Space, the action space is the spectrum resources allocated to each user, which is further expanded to consider the allocation of computing resources:

[0098] a t ={α i (t),r i (t)}

[0099] Among them, α i (t) is the amount of spectrum resources suitable for user i at t; r i (t) is the amount of computing resources that user i is suitable to be allocated at t;

[0100] Step 4.3) Reward function, transform the action space a t Acting in the reinforcement learning environment, the agent obtains corresponding rewards. Considering that the goal of reinforcement learning is to maximize the expectation of cumulative rewards, the reward function is set as:

[0101]

[0102] Among them, the first term T i (t) is used to measure the network throughput. The calculation formula is the same as the actual user throughput in step 2. M is the maximum number of training iterations, and λ is the penalty factor, which is used to calculate the penalty term for spectrum resource allocation.

[0103] Simulation experiments are conducted to compare the effects of prediction-based spectrum allocation and DSA methods at different time intervals, analyze the results, and make a summary.

[0104] like Figure 2As shown in the figure, consider multiple central base stations BSs with a coverage radius of D. There are a large number of user UEs distributed within the base station range. These users obey random distribution, and all base stations adopt orthogonal frequency division multiple access (OFDMA) technology. The UE only visits the nearest base station, and the UE must be within the coverage of the relevant base station. Each base station serves multiple associated UEs through OFDMA, performs spectrum allocation scheduling after a fixed time interval, and transmits to each subcarrier with the maximum transmission ratio. The large-scale channel gain that depends on the UE position remains unchanged within each frame and may change between frames. The small-scale channel gain remains constant within each time slot and is independent and identically distributed between time slots. By predicting the future UE traffic demand, the server knows who will compete for resources on each base station and performs predictive resource allocation to adapt to the UE traffic demand and wireless channel conditions, while the wireless transmission resources (such as the frequency bandwidth of the base station) are limited. Since each BS within the prediction window uses the same resource allocation method, we select one BS for research. The goal of resource allocation is to use the prediction information on the user side and the network side to achieve the maximum throughput of users within each base station within the prediction window.

[0105] This method studies the proposed dynamic spectrum allocation system based on traffic prediction and DDPG by experimentally simulating a single-cell wireless downlink communication network. Users in the network are randomly distributed within the range of the base station, and each generates different traffic data packet requirements. In the dynamic spectrum allocation system based on traffic prediction and DDPG, the Actor network consists of an input layer, two hidden layers, and an output layer, with the number of neurons being 41, 400, 300, and 20 respectively; the Critic network consists of an input layer and two hidden layers, with the number of neurons being 61, 400, and 300 respectively; the activation function of the hidden layer is ReLU. In addition, the Actor learning rate is 1e-4, the Critic learning rate is 1e-3, the discount factor (gamma): 0.99, the soft update parameter (tau): 0.005, the exploration noise (exploration_noise): 0.1, the batch size (batch_size): 64, and the experience replay cache size: 1,000,000. The control delay of this experiment is set to the common DSA model based on dynamic perception, and the actual throughput after the two resource allocations are compared. The experimental results are shown in Figure 2. Figure 3 As shown, the reinforcement learning allocation algorithm based on user traffic demand prediction of the present invention is shown as curve 1 in the figure, and curve 2 is the dynamic spectrum allocation algorithm based on perception. When the total throughput is the optimization target, the effect is better than the traditional DSA algorithm based on perception.

Claims

1. A downlink dynamic spectrum allocation method based on traffic prediction and DDPG model, characterized in that: The method comprises the following steps: Step 1) establishing a wireless network user traffic demand data set to form a feature set; Step 2) Using the feature set constructed in step 1, the transformer architecture is used to predict the user data packet request result for the next spectrum allocation window; Step 3) constructing a base station distribution and scenario model for the wireless network downlink, as well as user distribution and access mode; Step 4) Use the network model and access method set in step 3 to build a DDPG reinforcement learning model, set the state space, action space and reward function, input the number of user traffic packets predicted in step 2, use reinforcement learning to optimize the global problem, and simulate the best result of wireless network spectrum allocation.

2. According to claim 1, a downlink dynamic spectrum allocation method based on traffic prediction and DDPG model is characterized in that: The specific method of step 1) is as follows: By using a packet capture tool on the user side to complete data collection, the complete data set includes six features, as follows: F1: the time when the packet arrives and leaves; F2: the number of packets; F3: whether the packet is uplink or downlink; F4: source IP address; F5: destination IP address; F6: the communication protocol used during transmission.

3. According to claim 1, a downlink dynamic spectrum allocation method based on traffic prediction and DDPG model is characterized in that: The specific method of step 2) is as follows: Step 2.1) normalize the numerical features of the feature set; Step 2.2) Define the layer structure of the model, including multi-head self-attention mechanism, feedforward neural network layer, and position encoding; Step 2.3) Using the feature set obtained in step 2.1), the transformer architecture is used for prediction. The user UE predicted by the transformer architecture is k The number of demand data packets generated in the next time window τ is L k , the size of the data packet is np, then the user UE k The expected demand rate is: Assume that user UE k When entering the range of base station BSm, the actual rate is higher than that of user UE k The time of the demand rate is Does not meet user requirements, that is, the actual rate is lower than the user UE k Time of demand rate for: Actual user UE k The traffic throughput is expressed as: in, is the achievable rate for user UEk associated with base station BSm in time slot t of frame j.

4. The method for downlink dynamic spectrum allocation based on traffic prediction and DDPG model according to claim 1, characterized in that: The specific method of step 3) is as follows: In the wireless network downlink, there are S base stations, each distributed in the center of a circular area with a radius of D. All base stations use orthogonal frequency division multiple access technology to connect users. U users are randomly distributed within the base station coverage radius of D, and each sends a data packet request to the base station. UE k Only the nearest base station is visited, and the UE k Must be within the coverage of the relevant base station; each base station serves multiple associated UEs through OFDMA k , spectrum allocation scheduling is performed after a fixed time interval, and transmitted to each subcarrier at the maximum transmission ratio; The BSs set is represented by M = {M|M = 1, 2, ..., S}, the user UE set is represented by K = {K|K = 1, 2, ... U}, and UE k The distance from the serving BS in frame j is The large-scale channel gain is Where β is the path loss exponent. In time slot t of frame j, the user UE associated with base station BSm k The achievable rate is: Where W0 is the subcarrier spacing; It is allocated to the user UE k The transmitted power on subcarrier n in time slot t of frame j; is allocated to user UE in time slot t of frame j k The set of subcarriers; N0 is the noise power spectrum density; It is a user UE k The small-scale channel is the fading factor, on subcarrier n in time slot t of frame j; The total transmission power of each BS is expressed as P max , when the transmit power is evenly distributed among the subcarriers, that is, W max is the number of subcarriers, then the instantaneous achievable rate in formula (4) is: Among them, σ 2 =N0W max is the variance of random Gaussian noise.

5. The method for downlink dynamic spectrum allocation based on traffic prediction and DDPG model according to claim 1, characterized in that: The specific method of step 4) is as follows: Step 4.1) State space: The state space of the agent is determined by the observation of the XR device. The state at the tth time slot can be expressed as: s t ={n’,p i (t),h t ,F t l ,f t e } Where n' represents the relevant information of a specific user, which comes from the feature set in step 1); p i (t) represents the number of traffic packets predicted by user i at time t, which comes from the result predicted by the transformer model in step 2); h t Represents the gain of the communication channel between the user and the base station; F l t Indicates the current maximum idle computing resources available for the base station; f t e Indicates the computing resources currently allocated to the user by the server; Step 4.2) Action Space, the action space is the spectrum resources allocated to each user, which is further expanded to consider the allocation of computing resources: a t ={α i (t),r i (t)} Among them, α i (t) is the amount of spectrum resources suitable for user i at t; r i (t) is the amount of computing resources that user i is suitable to be allocated at t; Step 4.3) Reward function, transform the action space a t Acting in the reinforcement learning environment, the agent obtains corresponding rewards. Considering that the goal of reinforcement learning is to maximize the expectation of cumulative rewards, the reward function is set as: Among them, the first term T i (t) is used to measure the network throughput. The calculation formula is the same as the actual user throughput in step 2. M is the maximum number of training iterations, and λ is the penalty factor, which is used to calculate the penalty term for spectrum resource allocation.

Images