A downlink dynamic spectrum allocation method based on traffic prediction and DDPG model

Abstract

This invention discloses a dynamic spectrum allocation method based on user traffic demand prediction and deep deterministic policy gradient. In wireless networks using orthogonal frequency division multiple access (OFDM), this invention allocates spectrum for wireless network users within the same base station range. Addressing the problem of insufficient spectrum utilization in wireless networks, it introduces artificial intelligence technology to predict user traffic information, thereby helping to optimize spectrum allocation. In the prediction part, a transformer model architecture is used to ensure accuracy over long time series. In the network optimization algorithm part, a deep deterministic policy gradient model is employed. By leveraging the advantages of reinforcement learning in global optimization problems, the actual total throughput is used as the optimization objective to achieve better spectrum allocation results and improve throughput and communication efficiency.

Description

Technical Field

[0001] This invention relates to the field of wireless network spectrum allocation, and in particular proposes a dynamic spectrum allocation method for wireless networks based on traffic prediction oriented towards the downlink. Background Technology

[0002] From the perspective of wireless network service providers, licensed spectrum has always been the most expensive and scarcest resource. With the dramatic increase in wireless network capacity demands, spectrum has become even more valuable. Traditional resource management methods are typically based on User Buffer State Reports (BSRs), and this static resource allocation strategy may not be effective in responding to dynamic changes in traffic demand. In previous generations of cellular networks, Fixed Spectrum Allocation (FSA) has been a popular method for allocating spectrum resources (BWs) among different access points (APs). The solution involves performing an initial capacity plan and allocating BWs to APs based on the expected maximum service demand during peak periods and available resources. Afterward, BW allocation remains static, independent of the dynamic capacity demands of different APs in the network. In contrast to FSA, Dynamic Spectrum Allocation (DSA) allocates spectrum based on the instantaneous demand of APs and the available resources. DSA refers to dynamically sensing spectrum allocation by triggering a specific time interval, estimating the load for the next time interval, calculating spectrum demand, and establishing a sensing window to detect the instantaneous demand of users.

[0003] DSA technology has been popular since its inception because it improves network spectral efficiency by allowing the efficient use of underutilized frequency bands. With the development of wireless technology, DSA technology has also evolved. For example, different approaches have emerged in cognitive radio (CR) environments, where DSA is used to allocate idle channels to Secondary Users (SUs), who are typically unlicensed and have low priority in the network. These approaches learn, estimate, and allocate the required spectrum based on fuzzy logic, Q-learning, random rounding algorithms, etc. For instance, a reinforcement learning (RL)-based DSA technique is used to allocate spectrum to IoT users in cellular networks. In this method, the authors successfully demonstrated that DSA technology can be used to identify underutilized spectrum in the network for reuse in sensor-assisted IoT networks, thereby improving spectrum reusability. Work has also investigated machine learning-based Dynamic Frequency and Bandwidth Allocation (DFBA), which focuses on spectrum allocation for small cells in cellular networks. The authors propose a technique to learn and predict Long Term Evolution (LTE) key performance indicators (KPIs) such as SINR per resource block, Medium Access Control (MAC) level throughput, latency, etc., and to allocate / rearrange spectrum allocations for LTE-based SCs in the network. However, this work does not consider that the DSA algorithm based on traffic prediction is prone to getting trapped in local optima when facing global optimization problems, failing to globally optimize resources and thus maximizing resource utilization efficiency. For example, this paper uses spectrum utilization as the optimization objective, but if throughput is used as the optimization objective, the DSA algorithm is no longer suitable. This highlights the opportunity to allocate scarce spectrum to cells with current load to improve spectrum efficiency and system fairness. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a dynamic spectrum allocation method for wireless networks based on traffic prediction for downlink, which can not only effectively meet users' traffic needs, but also improve spectrum utilization and throughput, thereby achieving higher economic benefits.

[0005] Technical Solution: To address the aforementioned technical problems, this invention proposes a downlink dynamic spectrum allocation method based on traffic prediction and the DDPG model. This method includes the following steps:

[0006] Step 1) Establish a dataset of wireless network user traffic demand and construct a feature set;

[0007] Step 2) Using the feature set constructed in Step 1, predict the user data packet request results for the next spectrum allocation window through the transformer architecture;

[0008] Step 3) Construct a base station distribution and scenario model for the wireless network downlink, as well as user distribution and access methods;

[0009] Step 4) Construct a DDPG reinforcement learning model using the network model and access method set in Step 3. Set the state space, action space and reward function, input the number of user traffic packets predicted in Step 2, and use reinforcement learning to optimize the global problem and simulate the best result of wireless network spectrum allocation.

[0010] Furthermore, the specific method for step 1) is as follows:

[0011] Data was collected using a packet capture tool on the user's end. The complete dataset includes six features, as shown below:

[0012] F1: Data packet arrival / departure time;

[0013] F2: Number of data packets;

[0014] F3: Is the data packet an uplink or a downlink?

[0015] F4: Source IP address;

[0016] F5: Target IP address;

[0017] F6: The communication protocol used during transmission;

[0018] Use F3: whether the data packet is an uplink or downlink to determine whether it is an uplink or downlink; use F2: the number of data packets to represent the user's traffic demand in the wireless network; and use F6: the communication protocol used during transmission to determine the size of the data packets.

[0019] Furthermore, the specific method for step 2) is as follows:

[0020] Step 2.1) Normalize the numerical features of the feature set;

[0021] Step 2.2) Define the layer structure of the model, including the multi-head self-attention mechanism, feedforward neural network layer, and position encoding;

[0022] Step 2.3) Using the feature set obtained in Step 2.1), perform prediction using the transformer architecture. The user UE predicted by the transformer architecture is then obtained. k The number of demand data packets generated within the next time window τ is L. k If the data packet size is np, then the user UE k The expected demand rate is:

[0023]

[0024] Assuming user UE k When entering the range of base station BSm, the actual data rate is higher than that of user UE. k The time for the demand rate is D(B) k m The actual data rate is lower than the user's (UE) data rate, which does not meet user needs. k The time L((B) of the demand rate k m )for:

[0025]

[0026] Actual User UE k Throughput is expressed as:

[0027] T m,k =R d *D(B k m )+R m,k j,t *L(B k m (3).

[0028] Let be the achievable rate of user UEk associated with base station BSM in time slot t of frame j.

[0029] Furthermore, the specific method for step 3) is as follows:

[0030] In the downlink of a wireless network, there are S base stations, each located at the center of a circular area with radius D. All base stations use Orthogonal Frequency Division Multiple Access (OFDMA) to connect users. U users are randomly distributed within the coverage area of ​​the base stations (radius D) and each sends data packet requests to the base stations. k Only the nearest base station is accessed, and the UE must be within the coverage area of ​​the relevant base station; each base station serves multiple associated UEs via OFDMA, performs spectrum allocation scheduling after fixed time intervals, and transmits to each subcarrier at the maximum transmission ratio;

[0031] Wherein, the BSs set is represented as M = {M | M = 1, 2, ..., S}, and the user UE set is represented as K = {K | K = 1, 2, ..., U}. k The distance between the service BS in frame j and the service BS in frame j is Large-scale channel gain is Where β is the path loss exponent, representing the user UE associated with base station BSM in time slot t of frame j. k The achievable speed is:

[0032]

[0033] Where W0 is the subcarrier spacing;

[0034] p k j,t,n It is assigned to the user UE k Transmit power on subcarrier n in time slot t of frame j;

[0035] It is allocated to the user UE in time slot t of frame j. k The set of subcarriers;

[0036] N0 is the noise power spectral density;

[0037] User UE k The small-scale channel is the fading factor on subcarrier n in time slot t of frame j;

[0038] The total transmit power of each BS is represented as P. max When the transmit power is evenly distributed among the subcarriers, that is W max Let be the number of subcarriers, then the instantaneous achievable rate in formula (4) is:

[0039]

[0040] Where, σ 2 =N0W max The variance is the random Gaussian noise.

[0041] Furthermore, the specific method for step 4) is as follows:

[0042] Step 4.1) State Space: The state space of the agent is determined by the observations of the XR device. The state in the t-th time slot can be represented as:

[0043] s t ={n',p i (t),h t ,F t l ,f t e}

[0044] Where n' represents the relevant information of a specific user, which comes from the feature set in step 1); p i (t) represents the number of traffic packets predicted for user i at time t, derived from the result predicted by the transformer model in step 2); h t F represents the gain of the communication channel between the user and the base station; l tThis represents the maximum currently available idle computing resources at the base station; f t e This indicates the computing resources currently allocated to the user by the server;

[0045] Step 4.2) Action Space: The action space is the spectrum resources allocated to each user, further expanded to consider the allocation of computing resources.

[0046] a t ={α i (t),r i (t)}

[0047] Where, α i (t) represents the amount of spectrum resources that user i can be allocated at time t; r i (t) is the amount of computing resources that user i is suitable to be allocated at time t;

[0048] Step 4.3) Reward function, which applies to the action space a t In a reinforcement learning environment, the agent receives a corresponding reward. Considering that the goal of reinforcement learning is to maximize the expected cumulative reward, the reward function is set as follows:

[0049]

[0050] Among them, the first item T i (t) is used to measure network throughput. The calculation formula is the same as the actual user throughput in step 2. M is the maximum number of training iterations, and λ represents the penalty factor, which is the penalty term for calculating spectrum resource allocation.

[0051] Beneficial effects: Compared with the prior art, the technical solution of the present invention has the following beneficial technical effects:

[0052] Compared with the traditional dynamic spectrum allocation DSA method, this paper uses a method that combines traffic prediction to better meet user needs. At the same time, by adopting the DDPG model, the advantages of reinforcement learning in global problems are utilized to solve the shortcomings of DSA in global optimization problems and improve the actual throughput of users. Attached Figure Description

[0053] Figure 1 This is a flowchart illustrating the specific implementation of the research methodology.

[0054] Figure 2 This is a diagram of the wireless network topology scenario studied in this paper.

[0055] Figure 3 This is a comparison chart of the spectrum allocation results obtained by our method based on prediction and the DSA method based on dynamic sensing. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0057] like Figure 1 As shown, this invention proposes a downlink dynamic spectrum allocation method based on traffic prediction and the DDPG model. The method includes the following steps:

[0058] Step 1) Establish a dataset of wireless network user traffic demand and construct a feature set;

[0059] Step 2) Using the feature set constructed in Step 1, predict the user data packet request results for the next spectrum allocation window through the transformer architecture;

[0060] Step 3) Construct a base station distribution and scenario model for the wireless network downlink, as well as user distribution and access methods;

[0061] Step 4) Construct a DDPG reinforcement learning model using the network model and access method set in Step 3. Set the state space, action space and reward function, input the number of user traffic packets predicted in Step 2, and use reinforcement learning to optimize the global problem and simulate the best result of wireless network spectrum allocation.

[0062] Furthermore, the specific method for step 1) is as follows:

[0063] Data was collected using a packet capture tool on the user's end. The complete dataset includes six features, as shown below:

[0064] F1: Data packet arrival / departure time;

[0065] F2: Number of data packets;

[0066] F3: Is the data packet an uplink or a downlink?

[0067] F4: Source IP address;

[0068] F5: Target IP address;

[0069] F6: The communication protocol used during transmission;

[0070] Use F3: whether the data packet is an uplink or downlink to determine whether it is an uplink or downlink; use F2: the number of data packets to represent the user's traffic demand in the wireless network; and use F6: the communication protocol used during transmission to determine the size of the data packets.

[0071] Furthermore, the specific method for step 2) is as follows:

[0072] Step 2.1) Normalize the numerical features of the feature set;

[0073] Step 2.2) Define the layer structure of the model, including the multi-head self-attention mechanism, feedforward neural network layer, and position encoding;

[0074] Step 2.3) Using the feature set obtained in Step 2.1), perform prediction using the transformer architecture. The user UE predicted by the transformer architecture is then obtained. k The number of demand data packets generated within the next time window τ is L. k If the data packet size is np, then the user UE k The expected demand rate is:

[0075]

[0076] Assuming user UE k When entering the range of base station BSm, the actual data rate is higher than that of user UE. k The time for the demand rate is D(B) k m The actual data rate is lower than the user's (UE) data rate, which does not meet user needs. k The time L((B) of the demand rate k m )for:

[0077]

[0078] Actual User UE k Throughput is expressed as:

[0079] T m,k =R d *D(B k m )+R m,k j,t *L(B k m (3).

[0080] Let be the achievable rate of user UEk associated with base station BSM in time slot t of frame j.

[0081] Furthermore, the specific method for step 3) is as follows:

[0082] In the downlink of a wireless network, there are S base stations, each located at the center of a circular area with radius D. All base stations use Orthogonal Frequency Division Multiple Access (OFDMA) to connect users. U users are randomly distributed within the coverage area of ​​the base stations (radius D) and each sends data packet requests to the base stations. kOnly the nearest base station is accessed, and the UE must be within the coverage area of ​​the relevant base station; each base station serves multiple associated UEs via OFDMA, performs spectrum allocation scheduling after fixed time intervals, and transmits to each subcarrier at the maximum transmission ratio;

[0083] Wherein, the BSs set is represented as M = {M | M = 1, 2, ..., S}, and the user UE set is represented as K = {K | K = 1, 2, ..., U}. k The distance between the service BS in frame j and the service BS in frame j is Large-scale channel gain is Where β is the path loss exponent, representing the user UE associated with base station BSM in time slot t of frame j. k The achievable speed is:

[0084]

[0085] Where W0 is the subcarrier spacing;

[0086] It is assigned to the user UE k Transmit power on subcarrier n in time slot t of frame j;

[0087] It is allocated to the user UE in time slot t of frame j. k The set of subcarriers;

[0088] N0 is the noise power spectral density;

[0089] User UE k The small-scale channel is the fading factor on subcarrier n in time slot t of frame j;

[0090] The total transmit power of each BS is represented as P. max When the transmit power is evenly distributed among the subcarriers, that is... W max Let be the number of subcarriers, then the instantaneous achievable rate in formula (4) is:

[0091]

[0092] Where, σ 2 =N0W max The variance is the random Gaussian noise.

[0093] Furthermore, the specific method for step 4) is as follows:

[0094] Step 4.1) State Space: The state space of the agent is determined by the observations of the XR device. The state in the t-th time slot can be represented as:

[0095] s t ={n',p i (t),h t ,F t l ,f t e}

[0096] Where n' represents the relevant information of a specific user, which comes from the feature set in step 1); p i (t) represents the number of traffic packets predicted for user i at time t, derived from the result predicted by the transformer model in step 2); h t F represents the gain of the communication channel between the user and the base station; l t This represents the maximum currently available idle computing resources at the base station; f t e This indicates the computing resources currently allocated to the user by the server;

[0097] Step 4.2) Action Space: The action space is the spectrum resources allocated to each user, further expanded to consider the allocation of computing resources.

[0098] a t ={α i (t),r i (t)}

[0099] Where, α i (t) represents the amount of spectrum resources that user i can be allocated at time t; r i (t) is the amount of computing resources that user i is suitable to be allocated at time t;

[0100] Step 4.3) Reward function, which applies to the action space a t In a reinforcement learning environment, the agent receives a corresponding reward. Considering that the goal of reinforcement learning is to maximize the expected cumulative reward, the reward function is set as follows:

[0101]

[0102] Among them, the first item T i (t) is used to measure network throughput. The calculation formula is the same as the actual user throughput in step 2. M is the maximum number of training iterations, and λ represents the penalty factor, which is the penalty term for calculating spectrum resource allocation.

[0103] Simulation experiments were conducted to compare the performance of prediction-based spectrum allocation and DSA methods under different time intervals, and the results were analyzed and summarized.

[0104] like Figure 2As shown, consider multiple central base stations (BSs) with a coverage radius of D. A large number of users (ue) are distributed within the range of each base station, following a random distribution. All base stations employ Orthogonal Frequency Division Multiple Access (OFDMA) technology. Each ue only accesses the nearest base station, and the ue must be within the coverage area of ​​the relevant base station. Each base station serves multiple associated ue via OFDMA, performing spectrum allocation scheduling after fixed time intervals and transmitting to each subcarrier with the maximum transmission ratio. Large-scale channel gain, dependent on ue location, remains constant within each frame and may vary between frames. Small-scale channel gain remains constant within each time slot and is independent and identically distributed between time slots. By predicting future ue traffic demands, the server knows who will compete for resources on each base station and performs predictive resource allocation to accommodate ue traffic demands and radio channel conditions, while radio transmission resources (such as base station frequency bandwidth) are limited. Since each BS uses the same resource allocation method within the prediction window, we select one BS for our study. The goal of resource allocation is to achieve the maximum throughput for users within each base station within the prediction window, utilizing prediction information from both the user side and the network side.

[0105] This method studies the proposed dynamic spectrum allocation system based on traffic prediction and DDPG by simulating a single-cell wireless downlink communication network. Users are randomly distributed within the base station's range, each generating different traffic data packet demands. In the dynamic spectrum allocation system based on traffic prediction and DDPG, the Actor network consists of an input layer, two hidden layers, and an output layer with 41, 400, 300, and 20 neurons respectively; the Critic network consists of an input layer and two hidden layers with 61, 400, and 300 neurons respectively. The activation function of the hidden layers is ReLU. Additionally, the learning rate is 1e-4 for Actor and 1e-3 for Critic, with a discount factor (gamma) of 0.99, a soft update parameter (tau) of 0.005, exploration noise of 0.1, a batch size of 64, and an experience replay buffer size of 1,000,000. The control latency in this experiment is set to a common dynamic sensing-based DSA model, comparing the actual throughput after the two resource allocations. The experimental results are as follows: Figure 3 As shown in the figure, the reinforcement learning allocation algorithm based on user traffic demand prediction of this invention is shown by curve 1 in the figure. Curve 2 is the dynamic spectrum allocation algorithm based on perception. When the total throughput is the optimization target, the effect is better than the traditional perception-based DSA algorithm.

Claims

1. A downlink dynamic spectrum allocation method based on traffic prediction and DDPG model, characterized in that, The method includes the following steps: Step 1) Establish a wireless network user traffic demand dataset to build a feature set; Step 2) Using the feature set constructed in Step 1, predict the user data packet request results for the next spectrum allocation window through the transformer architecture; Step 3) Construct a base station distribution and scenario model for the wireless network downlink, as well as user distribution and access methods; Step 4) Construct a DDPG reinforcement learning model using the network model and access method set in Step 3. Set the state space, action space and reward function, input the number of user traffic packets predicted in Step 2, and use reinforcement learning to optimize the global problem and simulate the best result of wireless network spectrum allocation. The specific method for step 2) is as follows: Step 2.1) Normalize the numerical features of the feature set; Step 2.2) Define the layer structure of the model, including the multi-head self-attention mechanism, feedforward neural network layer, and position encoding; Step 2.3) Using the feature set obtained in Step 2.1), perform prediction using the transformer architecture. The user UE predicted by the transformer architecture is then obtained. k The number of demand data packets generated within the next time window τ is L. k If the data packet size is np, then the user UE k The expected demand rate is: Assuming user UE k When entering the range of base station BSm, the actual data rate is higher than that of user UE. k The time for the demand rate is D(B) k m This does not meet user needs, meaning the actual speed is lower than the user's UE (User Equipment) speed. k The time L((B) of the demand rate k m )for: Actual User UE k Throughput is expressed as: T m,k =R d *D(B k m )+R m,k j,t *L(B k m ) (3) in, Let be the achievable rate of user UEk associated with base station BSM in time slot t of frame j; The specific method for step 4) is as follows: Step 4.1) State Space: The state space of the agent is determined by the observations of the XR device. The state in the t-th time slot can be represented as: s t ={n',p i (t),h t ,F t l ,f t e } Where n' represents the relevant information of a specific user, which comes from the feature set in step 1); p i (t) represents the number of traffic packets predicted for user i at time t, derived from the result predicted by the transformer model in step 2); h t F represents the gain of the communication channel between the user and the base station; l t This represents the maximum currently available idle computing resources at the base station; f t e This indicates the computing resources currently allocated to the user by the server; Step 4.2) Action Space: The action space is the spectrum resources allocated to each user, further expanded to consider the allocation of computing resources. a t ={α i (t),r i (t)} Where, α i (t) represents the amount of spectrum resources that user i can be allocated at time t; r i (t) is the amount of computing resources that user i is suitable to be allocated at time t; Step 4.3) Reward function, which applies to the action space a t In a reinforcement learning environment, the agent receives a corresponding reward. Considering that the goal of reinforcement learning is to maximize the expected cumulative reward, the reward function is set as follows: Among them, the first item T i (t) is used to measure network throughput. The calculation formula is the same as the actual user throughput in step 2. M is the maximum number of training iterations, and λ represents the penalty factor, which is the penalty term for calculating spectrum resource allocation.

2. The downlink dynamic spectrum allocation method based on traffic prediction and DDPG model according to claim 1, characterized in that, The specific method for step 1) is as follows: Data was collected by using a packet capture tool on the user's end. The complete dataset includes six features, as shown below: F1: arrival and departure times of packets; F2: number of packets; F3: whether the packets are uplink or downlink; F4: source IP address; F5: destination IP address; F6: communication protocol used during transmission.

3. The downlink dynamic spectrum allocation method based on traffic prediction and DDPG model according to claim 1, characterized in that, The specific method for step 3) is as follows: In the downlink of a wireless network, there are S base stations, each located at the center of a circular area with radius D. All base stations use Orthogonal Frequency Division Multiple Access (OFDMA) to connect users. U users are randomly distributed within the coverage area of ​​the base stations (radius D) and each sends data packet requests to the base stations. k Only access the nearest base station, and the UE k It must be within the coverage area of ​​the relevant base station; each base station serves multiple associated UEs via OFDMA. k Spectrum allocation is scheduled after fixed time intervals and transmitted to each subcarrier at the maximum transmission ratio; Wherein, the BSs set is represented as M = {M | M = 1, 2, ..., S}, and the user UE set is represented as K = {K | K = 1, 2, ..., U}. k The distance d between the service BS in frame j and the service BS in frame j is j k The large-scale channel gain is α j k =(d j k ) β Where β is the path loss exponent, representing the number of user UEs associated with base station BSM in time slot t of frame j. k The achievable speed is: Where W0 is the subcarrier spacing; p k j,t,n It is assigned to the user UE k The transmit power on subcarrier n in time slot t of frame j; N m,k j,t, It is allocated to the user UE in time slot t of frame j. k The set of subcarriers; N0 is the noise power spectral density; h k j,t,n User UE k The small-scale channel is the fading factor on subcarrier n in time slot t of frame j; The total transmit power of each BS is represented as P. max When the transmit power is evenly distributed among the subcarriers, i.e., p k j,t,n =P max / W max W max Let be the number of subcarriers, then the instantaneous achievable rate in formula (4) is: Where, σ 2 =N0W max The variance is the random Gaussian noise.

Images