Method for real-time resource allocation of IEEE 802.11be WiFi based on deep deterministic policy gradient

Abstract

A real-time resource allocation method for IEEE 802.11be WiFi based on deep deterministic policy gradient includes the following steps: 1) establishing an IEEE 802.11be WiFi network model; 2) determining the mobility model, path loss model, and interference model adopted by the network; 3) deriving the network throughput expression; 4) proposing an optimization problem with maximizing the minimum throughput as the objective function. This optimization problem aims to optimize the allocation of power, channels, and resource units in real time, thereby improving the network's minimum throughput; 5) designing a real-time resource allocation algorithm based on deep deterministic policy gradient to solve the optimization problem, realizing real-time resource allocation for IEEE 802.11be WiFi. This invention can effectively improve the network's minimum throughput.

Description

Technical Field

[0001] This invention belongs to the field of network resource allocation and relates to an IEEE 802.11be WiFi real-time resource allocation method. Background Technology

[0002] WiFi based on the IEEE 802.11be standard can be used for network access at indoor nodes. The IEEE 802.11be standard is the next-generation WiFi standard and is currently in the development and refinement stage. From the technical draft of the IEEE 802.11be standard, it can be seen that compared to existing WiFi standards such as IEEE 802.11ac or IEEE 802.11ax, the key features of the IEEE 802.11be standard are as follows: support for enhanced MU-MIMO technology; support for enhanced OFDMA technology; support for multi-RU aggregation technology; and support for multi-link transmission and multi-AP coordination technology. These new features can effectively improve network performance. Therefore, deploying WiFi networks based on the IEEE 802.11be standard is crucial for improving the user experience.

[0003] In WiFi networks based on the IEEE 802.11be standard, dynamically allocating network resources to access points (APs) and nodes in real time based on current network conditions, such as node locations and network interference intensity, is an effective way to improve network throughput. Generally, there are two common methods for resource allocation in WiFi networks. The first is a fixed resource allocation method. In this method, the power and channel used by the AP are relatively fixed. Although network administrators can configure the AP's power and channel, and there are multiple options available, the configured power and channel remain fixed and will not dynamically adjust due to changes in network conditions. This makes it difficult to efficiently utilize network resources, easily leading to low throughput for some nodes in the network. The second is a dynamic resource allocation method based on heuristic algorithms. In recent years, to improve network resource utilization and further increase network throughput, dynamic resource allocation technology based on heuristic algorithms has been proposed. In this type of technology, the AP has parameter adjustment capabilities and can calculate the optimal resource allocation scheme based on the current network conditions. However, this type of technology has a major drawback: the algorithm's time complexity is high. When the AP calculates the optimal resource allocation scheme for a certain network state, the network state may have already changed during the AP's calculation process. Therefore, the real-time performance of the resource allocation schemes obtained by this type of algorithm is relatively poor. Summary of the Invention

[0004] To overcome the shortcomings of existing technologies, this invention provides a real-time resource allocation method for IEEE 802.11be WiFi based on deep deterministic policy gradient, which can effectively improve the minimum throughput of the network.

[0005] The technical solution adopted by this invention to solve its technical problem is:

[0006] A real-time resource allocation method for IEEE 802.11be WiFi based on deep deterministic policy gradient includes the following steps:

[0007] 1) Establish an IEEE 802.11be WiFi network model;

[0008] 2) Determine the mobility model, path loss model, and interference model used by the network;

[0009] 3) Derive the network throughput expression;

[0010] 4) An optimization problem is proposed with the objective function of maximizing the minimum throughput. This optimization problem aims to optimize the allocation of power, channels and resource units in real time, thereby improving the minimum throughput of the network.

[0011] 5) Design a real-time resource allocation algorithm based on deep deterministic policy gradient to solve the optimization problem and realize real-time resource allocation for IEEE 802.11be WiFi.

[0012] The main beneficial effects of this invention are: it can effectively improve the minimum throughput of the network. Attached Figure Description

[0013] Figure 1 It is a network frequency band and channel diagram.

[0014] Figure 2 This is a diagram of the data transmission process in an IEEE 802.11be WiFi network.

[0015] Figure 3 It is link l i,x and link l j,y Interference range diagram.

[0016] Figure 4 It is a timeline diagram of network operation.

[0017] Figure 5 This is a diagram of a reinforcement learning model.

[0018] Figure 6 This is a schematic diagram of a RU in a 40MHz channel.

[0019] Figure 7 It is a planar diagram of candidate solutions for power and channel. Detailed Implementation

[0020] The present invention will now be further described with reference to the accompanying drawings.

[0021] Reference Figures 1 to 7 A real-time resource allocation method for IEEE 802.11be WiFi based on deep deterministic policy gradient includes the following steps:

[0022] 1) Establish an IEEE 802.11be WiFi network model, the process is as follows:

[0023] An IEEE 802.11be WiFi network consists of a network controller, multiple access points (APs), and multiple stations. The network controller is responsible for allocating resources to the APs and coordinating the working status between the APs to reduce interference between adjacent basic service sets (BSSs).

[0024] Let A represent the set of Access Points (APs) and S represent the set of nodes. AP j is equipped with N antennas, and node i is equipped with 1 antenna, j∈A, i∈S. The IEEE 802.11be WiFi network uses spatial multiplexing technology. An AP equipped with N antennas has N spatial streams. Each node can communicate with the AP through one of the spatial streams. The IEEE 802.11be WiFi network uses four frequency bands: 2.4G, 5G-I, 5G-II, and 6G. Each frequency band contains multiple channels. Let w represent the channel width in MHz, w∈W={20,40,80,160,320}MHz, where W is the set of channel bandwidths. One channel can be allocated to one AP. For ease of description, we introduce the set B={2.4G,5G-I,5G-II,6G} to represent the four frequency bands used by the AP, and let C=∪ b∈B C b This represents the set of all channels in an IEEE 802.11be WiFi network, where C b Let c represent the set of channels in frequency band b, and let c be the channel used by AP j in frequency band b. j,b From a given set of channels C b Selected from; the channel used by the network, such as Figure 1 As shown, channels with a bandwidth of 20MHz are called "basic channels," such as channels 1 to 3, channels 5 to 12, channels 20 to 24, and channels 28 to 43, all of which are basic channels with a bandwidth of 20MHz. Several basic channels can be combined to form channels with wider bandwidths; for example, channels 2 and 3 can be combined to form channel 4. Furthermore, [the text abruptly ends here, likely due to an incomplete sentence or missing information]. Figure 1It can be seen that channels with different bandwidths may overlap. For example, channel 4 partially overlaps with channel 2, and channel 13 partially overlaps with channel 5. Therefore, it is necessary to define overlapping channel sets to facilitate channel allocation. For example, in the 2.4 GHz band, two overlapping channel sets can be defined, denoted as Γ1 = {2,4} and Γ2 = {3,4}. Similarly, according to... Figure 1 Overlapping channel sets for frequency bands such as 5G-I, 5G-II, and 6G can be defined. When adjacent BSSs use channels belonging to the same overlapping channel set, they will interfere with each other. This necessitates reasonable resource allocation and coordination through the network controller to minimize interference between BSSs.

[0025] Each channel can be divided into multiple mutually orthogonal Resource Units (RUs). Each RU consists of k data subcarriers, where k ∈ K = {26, 52, 106, 242, 484, 996 × 2, 996 × 4}. An Access Point (AP) can allocate one or more RUs to node i, and the RUs allocated to node i are grouped into a set of RUs. i This means that different nodes can use different sets of RUs to transmit data concurrently; when the RUs in a channel with a bandwidth of w MHz are divided into m... w With a set of N RUs, an AP with N spatial flows can support up to N×m w Each node can send or receive data simultaneously;

[0026] The transmit power p used by AP j in frequency band b j,b From a given power set P b The data transmission process of the IEEE 802.11be WiFi network, as selected, is as follows: Figure 2 As shown, TXOP, PIFS, SIFS, M-BA, and OFDMA-BA represent transmission opportunity, PCF (Point Coordination Function) inter-frame interval, short inter-frame interval, multi-node block acknowledgment, and OFDMA block acknowledgment, respectively. Figure 2 The data transmission process shown combines MU-MIMO and OFDMA. Figure 2 In this system, the channel is divided into x RU sets, and each RU set can be shared by y spatial streams. That is, each RU set can be simultaneously allocated to y nodes located in y different spatial streams. Each data transmission includes uplink PPDU transmission and downlink PPDU transmission. Uplink PPDU transmission requires a TF (Trigger Frame), which carries information such as RU set allocation, uplink PPDU duration, and uplink target received power.

[0027] In a network, the location of the access point (AP) is known. There are two types of nodes in the network: fixed nodes and mobile nodes. Fixed nodes refer to smart devices installed in the target area, such as cameras and smart terminals; mobile nodes refer to smart devices carried by users, such as smartphones and tablets. The location of fixed nodes is known, while the location of mobile nodes changes over time, meaning mobile nodes can move randomly within the target area. Let S... F and S M Let S represent the set of fixed nodes and the set of moving nodes, respectively. Then S = S F ∪S M .

[0028] 2) Determine the mobility model, path loss model, and interference model used by the network. The process is as follows:

[0029] 2.1) Determine the movement model

[0030] Since the movement of mobile nodes is random, the Random Waypoint (RWP) model is used to describe their movement patterns. The movement process of a node using the RWP model is as follows:

[0031] 2.1.1) Set the starting position of node i to (x i ,y i The target location is The initial velocity is v i The starting and target positions are randomly selected within the target area, and the initial velocity is within the range [v]. min ,v max Randomly selected from [ ].

[0032] 2.1.2) Node i travels at a speed v i Along the straight line from position (x i ,y i Move to position

[0033] 2.1.3) Node arrival location After that, stay for a period of time. p , t p In the interval [t] min ,t max Randomly selected from [ ].

[0034] 2.1.4) The user randomly selects a location (x′, y′) within the target area as the new target location, and within the interval [v... min ,v max A velocity v′ is randomly selected from the data as the new velocity.

[0035] Set the starting position Target location and speed v i =v′, return 2.1.2).

[0036] 2.2) Determine the path loss model

[0037] To describe the signal propagation characteristics between the AP and the node, the following path loss model is used:

[0038]

[0039] Where d is the distance between the sender and receiver, in meters. FS (d) is the path loss in free space, measured in dB. BP This is the distance from the sender to the dividing point, in meters. Generally, in large indoor venues, if the distance between the sender and receiver is less than or equal to d... BP If the distance between the sender and receiver is greater than d, then the path between them is a Loss-Oriented (LoS) path, meaning there are no obstacles between them; if the distance between the sender and receiver is greater than d... BP If the path between the sender and receiver is an NLoS path, then there is an obstacle between them. FS The expression for (d) is:

[0040] L FS (d)=20log 10 (d)+20log 10 (f)-147.5 (2)

[0041] Where f is the frequency, measured in Hz. Typically, f can be set to the center frequency of the frequency band. SF is the shadowing fading, measured in dB. SF follows a log-normal distribution with a mean of 0, and its distribution function is:

[0042]

[0043] Where σSF is the standard deviation of shadow fading.

[0044] In d BP Given the given conditions, the path loss between the sender and receiver is mainly determined by d and f. Therefore, L(d) can be rewritten as L(d,f). b ), f b This indicates the center frequency of frequency band b.

[0045] 2.3) Determine the network interference model, the process is as follows:

[0046] 2.3.1) Interference between adjacent basic service sets

[0047] According to the path loss model, PR =P T –L(d,f b ), where P R It is the receiver's RSS (Received Signal Strength), P T This is the transmitter's transmission power. According to equation (4), the distance between the transmitter and receiver is:

[0048]

[0049] With r j and γ j Let P represent the communication distance and interference distance of AP j or node j, respectively. R =θ D Then d = r j If P R =θ I Then d = γ j Here, θ D and θ I These represent the threshold for data frame decoding and the threshold for interference signal strength, respectively. θ D It can be obtained through formula β SINR =θ D / (P I +P noise )get.

[0050] Let STA x represent node x, and STA y represent node y. Let d i,x The distance between APi and STAx is represented by d. j,y Let γ represent the distance between APj and STAy. x and γ y Let l represent the interference distances of STA x and STA y, respectively. i,x Indicates the link between AP i and STA x; using l j,y This represents the link between AP j and STA y. Figure 3 This explains link l i,x Interference range and link l j,y The range of interference.

[0051] With S b (i) represents the set of nodes associated with AP i in frequency band b; S b (j) represents the set of nodes associated with APj in frequency band b. Figure 3 Define the interference range of AP i and AP j in frequency band b. for:

[0052]

[0053] If the distance between APi and APj is less than or equal to If i≠j, and the channels used by these two APs overlap, then link l i,x and l j,y Mutual interference. Under the coordination of the network controller, these two links need to take turns using the channel to avoid mutual interference.

[0054] 2.3.2) Interference from nearby signal sources

[0055] WiFi networks may be affected by interference from other nearby signal sources, such as other WiFi devices or IoT devices. The SINR (Signal to Interference plus Noise Ratio) model can be used to model the level of interference from nearby signal sources. The definition of SINR is as follows:

[0056] SINR = P R / (P I + P noise (6)

[0057] Among them, P R Indicates the receiver's RSS; P I P represents the total interference signal strength from nearby signal sources. noise This represents the thermal noise power. At a given data rate, for a data packet to be successfully received, the receiver's SINR must be greater than or equal to a given threshold β. SINR In β SINR P I and P noise Given the situation, let P R =θ D , through formula β SINR =θ D / (P I +P noise The threshold θ for correct decoding of the receiver's data frame can then be obtained. D .

[0058] 3) Derive the network throughput expression as follows:

[0059] 3.1) Downlink rate of the node

[0060] Assume channel c in frequency band b j,b ∈C b and power p j,b ∈P b It is assigned to AP j. According to the path loss model, the RSS of node i is p. j,b -L(di,j ,f b ), where d i,j It is the distance between node i and AP j. Let K c Indicates channel c j,b Given the total number of data subcarriers, the average RSS of node i in the k-tone RU is k(p j,b -L(d i,j ,f b )) / K c .by Let the SINR of node i in the k-tone RU be:

[0061]

[0062] Wherein, k(P) I +P noise ) / K c It is the average of the total interference power plus the thermal noise power of node i in the k-tone RU. The bit rate of node i in the k-tone RU can be obtained by equation (8), in bps / Hz.

[0063]

[0064] In the above formula, and (x = 1, 2, ..., X) represents the bit rate and the corresponding SINR. (x=1,

[0065] 2,...,X) are known. Assume the set RU is RU. i If a node is assigned to node i, then the downlink rate of node i is:

[0066]

[0067] Among them, W sc This indicates the bandwidth of the subcarrier, measured in Hz.

[0068] 3.2) Node uplink rate

[0069] according to Figure 2 Uplink PPDU transmission is scheduled by trigger frames. Through these trigger frames, the AP can specify the target RSS for uplink transmission of node i. (The RSS is used as the target RSS.) i Indicates that node i is in RU i If the target RSS for uplink transmission is given, then the uplink target RSS of node i in the k-tone RU is: Therefore, the transmission power of node i in the k-tone RU is:

[0070]

[0071] Among them, f k This represents the center frequency of the k-tone RU. Therefore, node i in the RU... i The total transmission power in is by Let the SINR of APj in the k-tone RU be:

[0072]

[0073] Similar to equation (9), it can be based on and The value of (x = 1, 2, ..., X) determines the uplink data rate of the node.

[0074] 3.3) Node throughput

[0075] With S b (j) represents the set of nodes associated with AP j in frequency band b, then the number of nodes associated with AP j in frequency band b is |S b (j)|。 With N b This represents the number of spatial streams in frequency band b for AP j. Assume the channel width of AP j is w and the channel is divided into m... w There are N RU sets, each RU set is divided by N b Each space stream is shared. According to Figure 2 The maximum number of nodes supported for each data transmission is N. b ×m w Therefore, AP j requires Y. j,b The data is transmitted in several steps to complete one round of transmission, that is, in each round of transmission, set S is used. b In (j), each node completes one uplink transmission and one downlink reception. Therefore, we can obtain:

[0076]

[0077] With t TF , t PIFS , t SIFS , t UL_PPDU , t M_MA , t DL_PPDU and t OFDMA_BA This indicates the duration of the trigger frame, PIFS, SIFS, uplink PPDU, M-BA, downlink PPDU, and OFDMA-BA. T represents the duration of these parameters. UL Indicates the uplink transmission time, expressed in T. DL Indicates the downlink transmission time. According to Figure 2 T can be obtained UL =t PIFS +tTF +2t SIFS +t UL_PPDU +t M_BA and T DL =t PIFS +t DL_PPDU +t SIFS +t OFDMA_MA With T j,b Let represent the period of one transmission cycle of AP j in frequency band b, then:

[0078] T j,b = (T) UL + T DL )Y j,b (13)

[0079] Therefore, the throughput of node i in frequency band b of AP j is:

[0080]

[0081] In equation (14), the molecule The denominator represents the total number of bits transmitted by node i in one round of transmission; This represents the total time required for one round of transmission. Here, an indicator variable CCI (Channel Conflict Indicator) is introduced to represent the level of interference at the AP. Using CCI... j,b This indicates the degree of interference that AP j experiences from its neighboring BSS in frequency band b. (CCI) j,b Defined as: the number of adjacent BSSs in frequency band b that belong to the same overlapping channel set as the channel of AP j. Therefore, CCI in equation (14) j,b +1 indicates that AP j and its CCI j,b Two adjacent BSSs interfere with each other in frequency band b, this CCI j,b +1 BSS must be transmitted in turn.

[0082] 4) An optimization problem is proposed with the objective function of maximizing the minimum throughput. This optimization problem aims to optimize the allocation of power, channels and resource units in real time, thereby improving the minimum throughput of the network.

[0083] During network operation, the network state changes over time; for example, the location of mobile nodes changes over time. At this time, some nodes in the network may experience a decrease in throughput due to the change in location. To improve the experience of users with the lowest throughput, the resource allocation of APs and nodes in each time slot can be optimized to maximize the minimum throughput. From the derivation of equation (14), it can be seen that the throughput of node i is related to the power p. j,b Channel c j,b and the RU set RU iThe function. Therefore, by optimizing p j,b c j,b and RU i The value of is chosen to maximize the minimum throughput. The above problem can be described by the following optimization problem.

[0084]

[0085] Where, δ i (t) represents the throughput of node i in time slot t, where t = 1, 2, 3, ..., T, and T is the total number of time slots, T→∞. j,b and c j,b It refers to the power and channel used by AP j in frequency band b, RU i It is the set of RUs that AP j assigns to node i. Constraints C1 to C3 ensure that p j,b c j,b and RU i It takes values ​​within a given range. In constraint C3, RU w,m This indicates that the RUs in a channel with bandwidth w can be combined into m. w There are RU sets, m∈{1,2,...,m} w}, RU i It is a subset of one of the RU sets.

[0086] Network operation timeline as follows Figure 4 As shown. In Figure 4 In this context, the network runtime is discretized into T time slots, where T → ∞. At the beginning of the t-th time slot, the throughput δ of |S| users is... i (t) constitutes the current state of the network, i = 1, 2, ..., |S|, that is: at the beginning of time slot 1, the state of the network is STATE 1 = {δ i (1) |i=1,2,3,...,|S|};At the beginning of time slot 2, the network state is STATE 2={δ i (2) |i=1,2,3,...,|S|},…,At the beginning of time slot x, the network state is STATE x={δ i (x)|i=1,2,3,...,|S|}. At the beginning of time slot t, the network controller needs to obtain a feasible solution to optimization problem (15) to maximize the minimum throughput. Using D t ={p j,b ,c j,b ,RU i Let} represent the resource allocation decision obtained by the network controller at the beginning of time slot t, where j∈A, b∈B, i∈S. Subsequently, the network controller makes the decision D. tAllocate power, channels, and RUs to APs and nodes to maximize minimum throughput. Figure 4 It can be seen that the optimization problem (15) needs to be solved in real time at the beginning of time slot t.

[0087] 5) Design a real-time resource allocation algorithm based on deep deterministic policy gradient to solve the optimization problem and realize real-time resource allocation for IEEE 802.11be WiFi.

[0088] 5.1) Define the reinforcement learning model

[0089] Deep Deterministic Policy Gradient (DDPG) is a major algorithm in reinforcement learning. It uses a deep neural network to express a deterministic policy and updates the network parameters using the deterministic policy gradient. DDPG is effectively applied to reinforcement learning tasks with large-scale or continuous action spaces. For example, in optimization problem (15), a trained and converged neural network can output a reasonable resource allocation strategy in real time based on the current state of the WiFi network.

[0090] In order to design an algorithm based on DDPG to solve the optimization problem (15), the optimization problem (15) needs to be transformed into a reinforcement learning model. This model includes four quantities: state space E, action space H, reward function g(·), and discount coefficient α. The definitions of these four quantities are as follows:

[0091] 5.1.1) State space E

[0092] Due to δ i (t) can reflect the location of the node, the SINR of the AP and the node, and the number of nodes associated with each AP in frequency band b. Therefore, the AP needs to record δ. i (t) and use it as the current state of the network.

[0093] 5.1.2) Action Space H

[0094] p j,b ∈P b ,c j,b ∈C b ,b∈B,j=

[0095] 1,2,…,|A|}. At the beginning of time slot t, the network controller generates an action h(t)∈H, which is essentially the network controller allocating power and channels to each AP. After the AP's power and channels are determined, the AP will generate a corresponding set of RUs based on the channel bandwidth and allocate an RU set to each node associated with it. Since the power and channel values ​​output by the DDPG-based algorithm are values ​​in a continuous real number interval, the power and channel values ​​need to be discretized and mapped to satisfy constraints C1 and C2.

[0096] 5.1.3) Reward function g(t)

[0097] Based on the network state e(t)∈E, we can obtain... Therefore, the reward function can be defined as follows: The goal of optimization problem (15) is to maximize This is equivalent to maximizing the value of the reward function g(t) in a reinforcement learning model.

[0098] 5.1.4) Discount factor α

[0099] To determine the importance of the current reward value obtained after performing action h(t) and the potential future reward value derived from it, reinforcement learning models introduce a discount factor α (0 ≤ α ≤ 1). The value of this factor is adjustable.

[0100] The reinforcement learning model for solving the optimization problem (15) is as follows: Figure 5 As shown.

[0101] 5.2) The designed algorithm uses two deep neural networks: a value network and a policy network. Since each neural network has its own parameters, these parameters are updated separately during iterative training to iteratively find a better resource allocation strategy. It's worth noting that the value network and policy network are copied during training to generate target value networks and target policy networks. The structures of the target value networks and target policy networks are the same as the value networks and policy networks, but their parameter updates are relatively slower, thus making the training process more stable.

[0102] Let M represent the total number of training rounds for the neural network, and let M represent the total number of training rounds for the neural network. Let M′ represent the number of time slots in each round. Let M′ represent the number of rounds the agent has undergone in its initial training, M′ << M. The algorithm steps based on the deep deterministic policy gradient are as follows:

[0103] 5.2.1) Randomly initialize the value network Q and policy network μ, with weights θ respectively. Q and θ μGenerate the target networks Q′ and μ′;

[0104] 5.2.2) Randomly assign power p to AP j j,b and channel c j,b The Association algorithm is used to establish node-AP associations, and the RU assignment algorithm is used to assign a set of RUs to node i. i ;

[0105] 5.2.3) Set episode=1;

[0106] 5.2.4) If episode > M, then proceed to step 5.2.21);

[0107] 5.2.5) Sample the current user's position according to the RWP mobility model; calculate the throughput of each node using formula (14) and obtain the initial state e(0) of the network in the first episode.

[0108] 5.2.6) Set t = 1;

[0109] 5.2.7) If Then set episode = episode + 1, and return to 5.2.4);

[0110] 5.2.8) If episode ≤ M′, then randomly assign power p to AP j. j,b and channel c j,b The Association algorithm is used to establish node-AP associations, and the RU assignment algorithm is used to assign a set of RUs to node i. i ;

[0111] 5.2.9) If episode > M′, then the output action h′(t) = μ(e(t)|θ μ );

[0112] 5.2.10) Call the Discretize algorithm to map h′(t) to a feasible solution h(t)∈H;

[0113] 5.2.11) Execute h(t) to allocate power p to AP j. j,b and channel c j,b The Association algorithm is used for node-AP association; the RU assignment algorithm is used to assign a set of RUs to node i. i ;

[0114] 5.2.12) Sample the current user's location according to the RWP mobility model; calculate the throughput δ of each node using formula (14). i(t), and obtain the reward value. And obtain the new network state e(t+1);

[0115] 5.2.13) Store the data (e(t), h(t), g(t), e(t+1)) into the buffer;

[0116] 5.2.14) Randomly extract from the buffer Data (e) i ,h i ,g i ,e i+1 ), i = 1, 2, ...,

[0117] 5.2.15) Set y i =g i +αQ′(e i+1 ,μ′(e i+1 θ μ′ )θ Q′ ),

[0118] 5.2.16) Update the value network Q by minimizing the loss. The loss is defined as:

[0119]

[0120] 5.2.17) The following strategy is used to update the policy network μ using gradients:

[0121]

[0122] 5.2.18) Update the target value network: θ Q′ ←τθ Q +(1-τ)θ Q′ ;

[0123] 5.2.19) Update the target policy network: θ μ′ ←τθ μ +(1-τ)θ μ′ ;

[0124] 5.2.20) Set t = t + 1, then return to 5.2.7);

[0125] 5.2.21) End and exit.

[0126] 5.3) Association Algorithm

[0127] The Association algorithm is used to establish associations between nodes and access points (APs). The steps are as follows:

[0128] 5.3.1) Node i detects the signal strength from surrounding APs and sends an association request to the AP with the strongest signal;

[0129] 5.3.2) AP j counts the total number of nodes associated with it and calculates the number of nodes allocated to frequency band b according to equations (16) and (17);

[0130]

[0131]

[0132] Among them, s j,b This indicates the number of nodes associated with frequency band b of AP j; the round(·) function represents rounding. C represents the set of nodes that are requested to be associated with AP j; b ′ represents the number of basic channels in frequency band b.

[0133] 5.3.3) AP j in set Select s evenly in the middle j,b Each node is associated with frequency band b of AP j. j,b The set consisting of nodes S b (j) means that |S b (j)|=s j,b .

[0134] 5.4) RUassignment Algorithm

[0135] AP j uses the RUassignment algorithm to assign a set of RUs to its associated node i. i The steps are as follows:

[0136] 5.4.1) Define m w A set of RUs w,m w∈W, m=1,2,...,m w m w It is the maximum number of 106-tone RUs in a channel with a bandwidth of w MHz.

[0137] by Figure 6 For example, w = 40MHz, m w =4. When the total number of spatial flows in frequency band b is N b When, define set RU 40,1 ={484} divided by 1×N b Shared by all nodes; define the set RU40,2 = {242, 242} as 2 × N b Shared by all nodes; Define RU 40,3={{106+26+26},{26+52+52+26},{26+26+106}} divided by 3×N b Shared by all nodes; Define RU 40,4 ={{106+26},106,{106+26},106} divided by 4×N b Each node shares this. It can be seen that the subset RU... i Let i = 1, 2, ..., m be the set RU w,m A subset of RU. For example, RU 40,3 =RU1∪RU2∪RU3, where RU1 = {106+26+26}, RU2 = {26+52+52+26}, and RU3 = {26+26+106}. Since a 106-tone RU is the smallest RU capable of supporting MU-MIMO, the sum of the number of subcarriers in RUs RU1, RU2, and RU3 must be greater than or equal to 106. Similarly, sets of RUs for 20, 80, 160, and 320MHz channels can be defined.

[0138] 5.4.2) Determine the maximum number of nodes supported in each parallel transmission. Since there are N nodes in frequency band b of AP j... b A spatial stream, channel c j,b The maximum number of divisions is m w There are N RU subsets, therefore, the maximum number of nodes supported by each parallel transmission is N. b ×m w .

[0139] 5.4.3) Group the nodes. |S b (j)|divided by N b ×m w , obtained as The remainder is rem. If rem is not equal to 0, |S b (j)| nodes are divided into Group. Group x Contains N b m w The node; the The group contains rem nodes. If rem equals 0, then |S b (j)| nodes are divided into|S b (j)| / (N b m w Groups, each containing N b m w Each node.

[0140] 5.4.4) RU allocation. If rem is not equal to 0, then... The RU subset in the data is assigned to the x-th group of nodes. For the The nodes in the group, in N b A spatial flow is uniformly selected from the spatial flows and assigned to each node. The required set of RUs is then determined based on the number of nodes served by each spatial flow. If rem equals 0, then... The RU subsets are allocated to nodes in each group. When allocating RU sets to nodes, nodes farther from the AP are assigned larger RU sets (i.e., RU sets with a larger total number of data subcarriers), while nodes closer to the AP are assigned smaller RU sets (i.e., RU sets with a smaller total number of data subcarriers). This approach balances the data rates of the nodes.

[0141] 5.5) Discretize Algorithm

[0142] The Discretize algorithm maps the actions generated by the policy network μ to a space of valid candidate solutions, thereby outputting a feasible solution. The steps are as follows:

[0143] 5.5.1) Tile the candidate power values ​​and candidate channel numbers in frequency band b along the x-axis and y-axis of the planar coordinate system, respectively, to generate a continuous bounded candidate solution plane. For example... Figure 7 As shown.

[0144] 5.5.2) Assign the action h′(t) output by the policy network μ to the corresponding point in the candidate solution plane, h′(t)={(H1,H2,…,H |A| )|H j =(p j,b ,c j,b ),p j,b ∈[1,|P b |],c j,b ∈[1,|C b |], b∈B, j=1,2,…,|A|}, where the interval [1,|P|] is a subset of B, j=1,2,…,|A|}. b |] and the interval [1,|C b |] represents a real number interval. j,b and c j,b Values ​​such as Figure 7 As shown by the black dot in the middle.

[0145] 5.5.3) Calculate the coordinates of the nearest integers to these points, and obtain h(t) = {(H1,H2,…,H...} |A| )|H j =(p j,b ,c j,b ),p j,b ∈P b ,c j,b ∈C b,b∈B,j=1,2,…,|A|}, where p j,b and c j,b All are integers that belong to the specified set of integers. For example... Figure 7 The coordinates of the black circle in the middle are shown.

[0146] 5.5.4) Output h(t).

[0147] The embodiments described in this specification are merely examples of implementations of the inventive concept and are for illustrative purposes only. The scope of protection of this invention should not be considered limited to the specific forms described in these embodiments; rather, it extends to equivalent technical means conceived by those skilled in the art based on the inventive concept.

Claims

1. A real-time resource allocation method for IEEE 802.11be WiFi based on deep deterministic policy gradient, characterized in that, The method includes the following steps: 1) Establish an IEEE 802.11be WiFi network model; 2) Determine the mobility model, path loss model, and interference model used by the network; 3) Derive the network throughput expression; 4) Propose an optimization problem with maximizing minimum throughput as the objective function. This optimization problem aims to optimize the allocation of power, channels, and resource units in real time, thereby improving the minimum throughput of the network. 5) Design a real-time resource allocation algorithm based on deep deterministic policy gradient to solve the optimization problem and realize real-time resource allocation for IEEE 802.11be WiFi.

2. The IEEE 802.11be WiFi real-time resource allocation method based on deep deterministic policy gradient as described in claim 1, characterized in that, In step 1), the IEEE 802.11be WiFi network consists of a network controller, multiple APs and multiple nodes. The network controller is responsible for allocating resources to the APs and coordinating the working status between the APs to reduce interference between adjacent Basic Service Sets (BSS). by A Denotes the AP set, in S AP represents a set of nodes. j Equipped N Root antenna, node i Equipped with 1 antenna, j A , i S The IEEE 802.11be WiFi network uses spatial multiplexing technology and is equipped with... N The AP with root antenna has N Each node communicates with the access point (AP) through one of the spatial streams. The IEEE 802.11be WiFi network uses four frequency bands: 2.4 GHz, 5 GHz-I, 5 GHz-II, and 6 GHz, with each band containing multiple channels. w Indicates the channel width, in MHz. w W = {20, 40, 80, 160, 320} MHz, where, W It is a set of channel bandwidths; one channel is assigned to one AP, hence the term "set". B ={2.4G, 5G-I, 5G-II, 6G} represents the four frequency bands used by the AP. This represents the set of all channels in an IEEE 802.11be WiFi network, where... C b Indicates frequency band b The channel set in the AP j In frequency band b The channel used in From a given set of channels C b Selected from; Each channel is divided into multiple mutually orthogonal resource units (RUs), and each RU consists of... k It consists of 1 data subcarrier, k K = {26,52, 106, 242, 484, 996 2,996 4}, AP is a node i Allocate one or more RUs to the nodes. i RU as a set RU i This indicates that different nodes use different sets of RUs to transmit data concurrently; when the bandwidth is... w The RUs in the MHz channel are divided into m w When there are RU sets, it has N AP supports up to one spatial stream N m w Each node can send or receive data simultaneously; AP j In frequency band b The transmission power used in From a given set of power P b The selection is as follows: TXOP, PIFS, SIFS, M-BA, and OFDMA-BA represent transmission opportunity, PCF inter-frame interval, short inter-frame interval, multi-node block acknowledgment, and OFDMA block acknowledgment, respectively.

3. The IEEE 802.11be WiFi real-time resource allocation method based on deep deterministic policy gradient as described in claim 1 or 2, characterized in that, In step 2), the RWP mobility model is used to describe the motion pattern of the mobile node. The motion process of the node using the RWP model is as follows: 2.1.1) Setting up nodes i The starting position is ( x i , y i The target location is ( , ), initial velocity is v i The starting and target positions are randomly selected within the target area, and the initial velocity is within the range []. v min , v max Randomly selected from [ ]; 2.1.2) Node i With speed v i Along a straight line from position ( x i , y i Move to position ( , ); 2.1.3) Node arrival location ( , After that, stay for a while. t p , t p In the interval [ t min , t max Randomly selected from [ ]; 2.1.4) The user randomly selects a location within the target area. , ) as the new target location, and in the interval [ v min , v max Randomly select a speed from ] As a new speed, set the starting position ( x i , y i ) = ( , ), target location ( , ) = ( , and speed v i = (Return to step 2.1.2).

4. The IEEE 802.11be WiFi real-time resource allocation method based on deep deterministic policy gradient as described in claim 1 or 2, characterized in that, In step 2), the following path loss model is used to describe the signal propagation characteristics between the AP and the node: （1）； in, d It is the distance between the sender and receiver, in meters. L FS ( d The path loss in free space is expressed in dB. d BP It is the distance from the sender to the boundary point, in meters; if the distance between the sender and receiver is less than or equal to... d BP If the distance between the sender and receiver is greater than a certain value, then the path between them is a Loss-of-Stake (LoS) path, meaning there are no obstacles between them; if the distance between the sender and receiver is greater than a certain value... d BP In this case, the path between the sender and receiver is an NLoS path, meaning there are obstacles between them. L FS ( d The expression for ) is: （2）； in, f It is frequency, measured in Hz. f Set to the center frequency of the frequency band. SF This is shadow fading, measured in dB. SF It follows a log-normal distribution with a mean of 0, and its distribution function is: （3）； in, It is the standard deviation of shadow fading; exist d BP Given the given conditions, the path loss between the sender and receiver is determined by... d and f Therefore, L ( d Rewritten as L ( d , f b ), f b Indicates frequency band b The center frequency.

5. The IEEE 802.11be WiFi real-time resource allocation method based on deep deterministic policy gradient as described in claim 1 or 2, characterized in that, In step 2), the interference model of the network is determined as follows: 2.3.1) Interference between adjacent basic service sets According to the path loss model, we have P R = P T – L ( d , f b ),in, P R It is the receiver's received signal strength RSS. P T The transmitter's transmission power is given by equation (4). The distance between the transmitter and receiver is: （4）； by r j and j They represent AP respectively j or node j The communication distance and interference distance, if P R = D ,but d = r j ;if P R = I ,but d = j Here, D and I These represent the threshold for data frame decoding and the threshold for interference signal strength, respectively. D Through SINR = D / ( P I + P noise )get, P I This indicates the total interference signal strength from nearby signal sources; P noise Indicates thermal noise power; With STA x Represents a node x , with STA y Represents a node y ,by d i,x Indicates AP i and STA x The distance between them, d j,y Indicates AP j and STA y The distance between them, x and y They represent STA respectively x and STA y The interference distance, in order to l i,x Indicates AP i and STA x The link between; l j,y Indicates AP j and STA y Links between; by S b ( i ) indicates in frequency band b China and AP i A set of associated nodes; S b ( j ) indicates in frequency band b China and AP j The set of associated nodes, defined as AP i and AP j In frequency band b Interference range for: （5）； If AP i and AP j The distance between them is less than or equal to , i j Furthermore, if the channels used by these two APs overlap, then the link... l i,x and l j,y To avoid mutual interference, these two links need to take turns using the channel under the coordination of the network controller. 2.3.2) Interference from nearby signal sources The SINR model is used to model the interference level from nearby signal sources. The definition of SINR is as follows: SINR = P R / ( P I + P noise ) （6）； in, P R This represents the receiver's SINR; at a set data rate, for a data packet to be successfully received, the receiver's SINR must be greater than or equal to a given threshold. SINR ,exist SINR , P I and P noise Given the circumstances, let P R = D , through SINR = D / ( P I + P noise That is, to find the threshold for correctly decoding the data frame at the receiver. D .

6. The IEEE 802.11be WiFi real-time resource allocation method based on deep deterministic policy gradient as described in claim 5, characterized in that, In step 3), the network throughput expression is derived as follows: 3.1) Downlink rate of the node Assuming in frequency band b Channel in and power Assigned to AP j According to the path loss model, nodes i The RSS is ,in, d i,j It is a node i and AP j The distance, in K c Indicates channel The total number of data subcarriers in the middle, then, the node i exist k The average RSS in -tone RU is ,by Represents a node i exist k If SINR is set in -tone RU, then: （7）； in, It is a node i exist k The average value of the total interference power plus the thermal noise power in the -tone RU is obtained by equation (8) for the node. i exist k -tone RU bit rate, in bps / Hz; （8）； in, and x ( x = 1, 2, ..., X () represents the bit rate and the corresponding SINR. x ( x = 1, 2, ..., X ) is known, assuming the RU set is known. RU i Assigned to node i Then the node i The downlink rate is: （9）； in, W sc This indicates the bandwidth of the subcarrier, measured in Hz. 3.2) Node uplink rate Uplink PPDU transmission is scheduled by trigger frames. Through the trigger frames, the AP specifies the node. i The target RSS of the uplink transmission, RSS i Represents a node i exist RU i The target RSS for uplink transmission, then the node i exist k The uplink target RSS in -tone RU is So, node i exist k The transmission power in -tone RU is: （10）； in, f k express k -tone RU's center frequency, therefore, node i exist RU i The total transmission power in is ,by Indicates AP j exist k -tone RU In SINR, then: （11）； according to and x ( x = 1, 2, ..., X The value of ) determines the uplink data rate of the node. ; 3.3) Node throughput by S b ( j ) indicates in frequency band b China and AP j The set of associated nodes is in the frequency band. b China and AP j The number of associated nodes is | S b ( j )|, with N b Indicates AP j In frequency band b The number of spatial flows in AP, assuming AP j The channel width is w And the channel is divided into m w indivual RU Set, each RU The set was N b Each spatial stream is shared, and the maximum number of nodes supported for each data transmission is [number missing]. N b m w Therefore, AP j need Y j,b The data is transmitted in multiple data transfers to complete one round of transmission, that is, in each round of transmission, the data is collected. S b ( j In this process, each node completes one uplink transmission and one downlink reception. Therefore, we have: （12）； by t TF , t PIFS , t SIFS , t UL_PPDU , t M_MA , t DL_PPDU and t OFDMA_BA This indicates the duration of the trigger frame, PIFS, SIFS, uplink PPDU, M-BA, downlink PPDU, and OFDMA-BA. T UL Indicates the uplink transmission time, in T DL This indicates the downlink transmission time, and we obtain... and ,by T j,b Indicates AP j In frequency band b If the cycle of one round of transmission is: T j,b = ( T UL + T DL ) Y j,b （13）； Therefore, node i In AP j frequency band b The throughput rate is: （14）； In equation (14), the molecule Indicates a node in a round of transmission i Total number of bits transmitted; denominator The total time required for one round of transmission is represented by an indicator variable, Channel Collision Indicator (CCI), which indicates the level of interference to the AP. CCI j,b Indicates in frequency band b AP j The degree of interference from neighboring BSS, CCI j,b Defined as: in the frequency band b China and AP j The number of adjacent BSSs belonging to the same overlapping channel set, therefore, in equation (14) CCI j,b +1 indicates AP j and CCI j,b Adjacent BSSs in frequency band b Mutual interference, this CCI j,b + 1 BSS must be transmitted in turn.

7. The IEEE 802.11be WiFi real-time resource allocation method based on deep deterministic policy gradient as described in claim 6, characterized in that, In step 4), during network operation, the network state changes over time. By optimizing the resource allocation of APs and nodes in each time slot to maximize the minimum throughput, as can be seen from the derivation of equation (14), the node... i Throughput is related to power Channel and RU set RU i The function, therefore, through optimization , and RU i The goal of maximizing the minimum throughput is achieved by choosing the value of . The above problem can be described by the following optimization problem: （15）； in, i ( t ) indicates in time slot t Middle node i throughput t = 1, 2, 3, ..., T , T It is the total number of time slots. T , and It is AP j In frequency band b The power and channel used in it, RU i It is AP j Assigned to node i The set of RUs, constraints C 1 to C 3. Ensure , and RU i Take values ​​within a given range, under constraints C 3 in, RU w,m Indicates bandwidth is w RU combination in the channel m w A set of RUs, m {1, 2, ..., m w }, RU i It is a subset of some RU set; The network runtime is discretized into T Each time slot In the t The start time of each time slot, | S | throughput per user i ( t This constitutes the current state of the network. i = 1, 2, ..., | S | That is, at the beginning of time slot 1, the network state is STATE 1 = { i (1) | i = 1, 2, 3, ..., | S |};At the beginning of time slot 2, the network state is STATE 2 = { i (2) | i = 1, 2, 3, ..., | S |},…,in time slots x At the initial moment, the network state is STATE. x = { i ( x ) | i = 1, 2, 3, ..., | S |}, in time slots t At the initial moment, the network controller needs to obtain a feasible solution to the optimization problem (15) to maximize the minimum throughput. D t = { , , RU i } indicates that the network controller is in the time slot t The resource allocation decision obtained at the initial moment, j A , b B , i S Subsequently, the network controller makes a decision. D t To allocate power, channels, and RUs to APs and nodes to maximize minimum throughput, optimization problem (15) needs to be solved in time slots. t The starting time is solved in real time.

8. The IEEE 802.11be WiFi real-time resource allocation method based on deep deterministic policy gradient as described in claim 7, characterized in that, The process of step 5) is as follows: 5.1) Define a reinforcement learning model. For the optimization problem (15), the trained and converged neural network can output a reasonable resource allocation strategy in real time according to the current state of the WiFi network. In order to design an algorithm based on DDPG to solve the optimization problem (15), the optimization problem (15) needs to be transformed into a reinforcement learning model, which includes a state space. E Action space H reward function g ( ) and discount factor The four quantities are defined as follows: 5.1.1) State Space E E { i ( t ) | i = 1, 2, ..., | S |}, due to i ( t This reflects the location of the node, the SINR of the AP and the node, and the frequency band of each AP. b The number of nodes associated with it, therefore, AP needs to record. i ( t This is taken as the current state of the network; 5.1.2) Action Space H H {( H 1, H 2, …, H |A| ) | H j = ( p j,b , c j,b ), p j,b P b , c j,b C b , b B , j = 1, 2,…, | A |}, the network controller in the time slot t An action will occur at the start time. h ( t ) H This action essentially involves the network controller allocating power and channels to each access point (AP). After the AP's power and channels are determined, the AP generates a corresponding set of RUs (Resource Requirement Units) based on the channel bandwidth and assigns an RU set to each node associated with it. Since the power and channel values ​​output by the DDPG-based algorithm are values ​​in a continuous real number range, discretization mapping of the power and channel is necessary to satisfy the constraints. C 1 and C 2; 5.1.3) Reward Function g ( t ) Based on network status e ( t ) E get Therefore, the reward function is defined as follows: g ( t ) = The goal of optimization problem (15) is to maximize This is equivalent to maximizing the reward function in a reinforcement learning model. g ( t The value of ); 5.1.4) Discount Factor 9. In order to determine the action to be performed h ( t The reinforcement learning model introduces a discount factor to determine the importance of the current reward value obtained afterward and the resulting future reward value. 0 1; The value of this coefficient is adjustable. 5.2) The designed algorithm has two deep neural networks: a value network and a policy network. Since these two neural networks have their own parameters, the parameters of the two neural networks are updated separately during iterative training. The purpose is to iteratively find a better resource allocation strategy. It is worth noting that the value network and policy network are copied during training to generate a target value network and a target policy network. The structure of the target value network and the target policy network is the same as that of the value network and policy network. The difference is that the parameter updates of the target value network and the target policy network are relatively slow, which makes the training process more stable. by M Represents the total number of rounds of training for the neural network, in terms of Indicates the number of time slots per round, in This indicates the number of rounds the agent underwent during its initial training. The algorithm steps based on the gradient of a deep deterministic policy are as follows: 5.2.1) Randomly initialize the value network Q and policy network The weights are respectively Q and Generate the target network Q and ; 5.2.2) Random for AP j Power distribution and channel The Association algorithm is used for node-AP association, and the RUassignment algorithm is used for node assignment. i Allocate RU set RU i ; 5.2.3) Set episode = 1; 5.2.4) If episode > M Then proceed to step 5.2.21). 5.2.5) Sample the current user's location based on the RWP mobility model; The throughput of each node is calculated using formula (14), and the initial state of the network in the episode is obtained. e (0); 5.2.6) Settings t = 1; 5.2.7) If t > If so, set episode = episode + 1, and return to 5.2.4). 5.2.8) If episode Then the random is AP j Power distribution and channel The Association algorithm is used for node-AP association, and the RUassignment algorithm is used for node assignment. i Allocate RU set RU i ; 5.2.9) If episode > Then output action ; 5.2.10) Call the Discretize algorithm to... Mapping to a feasible solution h ( t ) H ; 5.2.11) Execution h (t) is AP j Power distribution and channel The Association algorithm is used for node-AP association; the RUassignment algorithm is used for node assignment. i Allocate RU set RU i ; 5.2.12) Sample the current user's location based on the RWP mobility model; The throughput of each node is calculated using formula (14). i ( t ), to obtain reward points g ( t ) = And obtain the new state of the network. e ( t + 1); 5.2.13) The data ( e ( t ), h ( t ), g ( t ), e ( t + 1)) Stored in the buffer; 5.2.14) Randomly extract from the buffer Data ( e i , h i , g i , e i+1 ), i = 1, 2, ..., ; 5.2.15) Settings , i = 1, 2, ..., ; 5.2.16) Update the value network by minimizing the loss. Q The loss is defined as: ； 5.2.17) The following strategy is used to update the policy network using gradients. : ； 5.2.18) Update the target value network: ; 5.2.19) Update the target policy network: ; 5.2.20) Settings t = t + 1, return to 5.2.7); 5.2.21) End and exit.

10. The IEEE 802.11be WiFi real-time resource allocation method based on deep deterministic policy gradient as described in claim 8, characterized in that, Step 5) further includes the following: 5.3) Association algorithm. The role of the Association algorithm is to establish the association between nodes and APs. The steps are as follows: 5.3.1) Node i Detect the signal strength from surrounding APs and send an association request to the AP with the strongest signal; 5.3.2) AP j The total number of nodes associated with the statistical request is counted, and the frequency bands allocated to them are calculated according to equations (16) and (17). b The number of nodes; （16）； （17）； in, Indicates AP j frequency band b The number of associated nodes; round( The function represents rounding. Indicates a request to AP j A set of associated nodes; Indicates frequency band b The number of basic channels; 5.3.3) AP j In the set Select evenly in the middle Each node and AP j frequency band b Related, The set consisting of nodes S b ( j ) indicates that ; 5.4) RUassignment algorithm, AP j The RUassignment algorithm is used to assign the associated nodes. i Allocate RU set RU i The steps are as follows: 5.4.1) Definition m w RU set RU w,m , w W , m = 1, 2, ..., m w , m w It is bandwidth w The maximum number of 106-tone RUs in a MHz channel; 5.4.2) Determine the maximum number of nodes supported for each parallel transmission, since in AP j frequency band b There is N b A spatial stream, channel c j,b At most, it can be divided into m w There are RU subsets, therefore, the maximum number of nodes supported by each parallel transmission is N b m w ; 5.4.3) Group the nodes, and | S b ( j )| Divided by N b m w , obtained as The remainder is rem ,if rem Not equal to 0, | S b ( j )| Nodes are divided into +1 group, number x Group( x = 1,2, ..., )Include N b m w The node; the +1 group contains rem If a node, rem If it equals 0, then | S b ( j )| Nodes are divided into Groups, each group contains N b m w One node; 5.4.4) RU allocation, if rem Not equal to 0, will The RU subset in the middle is assigned to the first x Group nodes, x = 1, 2,..., For the first The nodes in group +1, in N b One spatial flow is uniformly selected from the spatial flows and assigned to each node. Then, the required set of RUs is determined based on the number of nodes served by each spatial flow. rem Equal to 0, will The RU subsets are allocated to nodes in each group. When allocating RU sets to nodes, nodes farther from the AP are allocated larger RU sets, i.e., RU sets with a larger total number of data subcarriers; nodes closer to the AP are allocated smaller RU sets, i.e. RU sets with a smaller total number of data subcarriers. This approach can balance the data rates of the nodes.

11. The IEEE 802.11be WiFi real-time resource allocation method based on deep deterministic policy gradient as described in claim 9, characterized in that, The process in step 5) also includes the following: 5.5) Discretize algorithm, the function of which is to optimize the policy network. The resulting actions are mapped to the space of valid candidate solutions, thus outputting a feasible solution. The steps are as follows: 5.5.1) Frequency band b The candidate power values ​​and candidate channel numbers are respectively arranged in the plane coordinate system. x shaft and y The axes are tiled to generate a continuous bounded candidate solution plane; 5.5.2) Policy Network Output action Corresponding to the corresponding points in the candidate solution plane ={( H 1, H 2, …, H |A| ) | H j = ( p j,b , c j,b ), p j,b [1, | P b |], c j,b [1, | C b |], b B , j = 1, 2, …, | A |}, where the interval [1, | P b |] and the interval [1, | C b |] represents a real number interval; 5.5.3) Calculate the coordinates of the nearest integers to these points, and obtain... h ( t ) = {( H 1, H 2, …, H |A| ) | H j = ( p j,b , c j,b ), p j,b P b , c j,b C b , b B , j = 1, 2, …, | A |}, where, p j,b and c j,b All are integers that belong to the specified set of integers; 5.5.4) Output h ( t ).

Images