Reinforcement learning scheduling methods, devices, equipment, and media for multi-access point wireless networks
By constructing complete joint actions and filtering candidate action subsets, the problem of low sample efficiency in multi-access point wireless networks is solved, achieving efficient wireless network scheduling and improving network performance.
Patent Information
- Application Number
- CN202611576624.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-10-13
- Publication Date
- 2026-11-10
AI Technical Summary
Existing reinforcement learning scheduling methods for multi-access point wireless networks suffer from low sample efficiency, failing to effectively utilize communication and queuing mechanisms to eliminate deterministic inferior candidate actions, resulting in high computational complexity and difficulty in adapting to dynamic changes in terminal access, service load, and wireless channels.
By constructing complete joint actions, a subset of candidate actions is selected and constrained within the decision space of the reinforcement learning policy. Action selection is performed using resource acquisition opportunities and network state parameters, thereby reducing the exploration and reasoning complexity of reinforcement learning.
It improves sample efficiency, reduces computational complexity, enables efficient scheduling of multi-access point wireless networks, and enhances throughput, latency, and packet loss rate performance.
Smart Images

Figure CN123075709A_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of wireless network resource management technology, and specifically relates to a reinforcement learning scheduling method, apparatus, device and medium for multi-access point wireless networks. Background Technology
[0002] Fiber to the Radio (FTTR) networks connect multiple wireless access points (APs) via fiber optic cables to eliminate dead zones and signal attenuation throughout homes, offices, businesses, and campuses. (See reference...) Figure 2 The primary FTTR gateway or primary optical network unit (ONU) is connected to multiple secondary optical network units or wireless access nodes via optical fiber. Multiple wireless terminals need to dynamically select between different access points and wireless resources. The optical side has high capacity and low transmission latency, but the wireless side suffers from problems such as channel fluctuations, wall obstruction, co-channel interference, random access, and queue contention. Within the same resource pool at the same access point, multiple terminals compete for limited scheduling opportunities. The terminal's queue length, service arrival intensity, and service duration jointly determine its throughput, latency, and packet loss rate. The access point-resource pool actions of a terminal not only affect its own link transmission quality but also influence the competition status and queuing service process of other terminals by changing the composition of the resource contention group. Traditional combinatorial optimization methods require solving high-dimensional discrete correlation and resource allocation problems in each control cycle, resulting in excessive computational complexity and difficulty in adapting to the continuous dynamic changes in terminal access, service load, and wireless channels.
[0003] Existing technologies employ Deep Q-Networks (DQN), Deep Deterministic Policy Gradient (DDPG), Proximal Policy Optimization (PPO), or multi-agent reinforcement learning methods. These methods treat link measurements, queue states, service arrivals, and access point load as states, and access point numbers and resource numbers as actions. A weighted sum of throughput, latency, reliability, and energy consumption is used as the reward function, and the mapping from states to actions is learned through environmental interaction. To reduce the dimensionality of the joint action space, some methods generate actions sequentially by terminal in an autoregressive manner.
[0004] However, in existing technical solutions, policy networks mainly rely on trial and error to identify superior and inferior actions, failing to eliminate deterministic inferior candidates using communication and queuing mechanisms before policy sampling, resulting in low sample efficiency. Although autoregressive policies reduce the output dimensionality, the performance of the current candidate action depends on the composition of the entire resource group. If the action is evaluated solely based on the current local channel state of the terminal or the access point load at the previous time step, it cannot reflect the impact of the candidate action on competition group reconstruction, scheduling opportunity allocation, and deadline delivery probability. Summary of the Invention
[0005] This application provides a method, apparatus, device, and medium for reinforcement learning scheduling in multi-access point wireless networks, which can solve the technical problem of low sample efficiency in the prior art.
[0006] To achieve the above objectives, this application provides the following technical solution: A reinforcement learning scheduling method for multi-access point wireless networks, the method comprising: Obtain the current status information of multiple terminals in a wireless communication network; For each candidate action of the current terminal, construct a complete joint action corresponding to the candidate action, wherein an action of the current terminal represents a radio resource contention group selected by the current terminal; Under the complete joint action, the resource acquisition opportunities of the current terminal under the complete joint action are calculated based on the current state information; Based on the resource acquisition opportunities and the network state parameters extracted from the current state information, a subset of candidate actions is selected from all candidate actions of the current terminal. The current state information is input into the reinforcement learning policy, and the decision space of the reinforcement learning policy is constrained according to the subset of candidate actions, so that the reinforcement learning policy outputs the current terminal's execution action within the range defined by the subset of candidate actions.
[0007] Based on the same inventive concept, embodiments of this application also provide a multi-access point wireless network reinforcement learning scheduling device, the device comprising: The information acquisition module is configured to acquire the current status information of multiple terminals in the wireless communication network; A complete joint action construction module is configured to construct a complete joint action corresponding to each candidate action of the current terminal, wherein an action of the current terminal represents a radio resource contention group selected by the current terminal. The calculation module is configured to calculate the resource acquisition opportunities of the current terminal under the complete joint action based on the current state information. The filtering module is configured to filter a subset of candidate actions from all candidate actions of the current terminal based on the resource acquisition opportunity and the network state parameters extracted from the current state information. The action acquisition module is configured to input the current state information into the reinforcement learning policy and constrain the decision space of the reinforcement learning policy according to the subset of candidate actions, so that the reinforcement learning policy outputs the current terminal's action within the range defined by the subset of candidate actions.
[0008] Based on the same inventive concept, this application also provides an electronic device, including: a memory and a processor; the processor is used to read and execute a computer program stored in the memory to implement the steps of the aforementioned multi-access point wireless network reinforcement learning scheduling method.
[0009] Based on the same inventive concept, embodiments of this application also provide a computer storage medium storing computer-executable instructions, which, when executed, implement the steps of the aforementioned multi-access point wireless network reinforcement learning scheduling method.
[0010] Compared with the prior art, this application has the following advantages: By constructing complete joint actions, the analytical performance of competitive combinations can be determined, thereby avoiding the problem of local action evaluation lacking the context of complete joint actions; By compressing the entire set of candidate actions into a subset of candidate actions, the problem of low sample efficiency caused by existing technologies relying on trial and error to identify superior and inferior actions and failing to eliminate deterministic inferior candidates before policy sampling is solved. Constrained reinforcement learning only requires sampling from an average number of non-dominated candidate actions, which is usually significantly smaller than the total number of legitimate candidates, thus reducing the complexity of reinforcement learning exploration and reasoning.
[0011] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures pointed out in the description, claims and drawings. Attached Figure Description
[0012] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0013] Figure 1 A flowchart illustrating the method provided in the embodiments of this application; Figure 2 This is a schematic diagram of an embodiment of the FTTR multi-access point wireless network and centralized scheduling architecture of this application; Figure 3 This is a schematic diagram of the closed-loop process of online scheduling and training of a multi-access point wireless network according to an embodiment of this application; Figure 4 This is a schematic diagram of the terminal-by-terminal counterfactual joint action completion process according to an embodiment of this application; Figure 5 This is a schematic diagram illustrating the consistency between finite total parsed throughput prediction and packet-level simulation according to an embodiment of this application; Figure 6 This is a schematic diagram illustrating the consistency between finite total analytical delay prediction and packet-level simulation according to an embodiment of this application; Figure 7 This is a schematic diagram illustrating the consistency between finite total parsing packet loss rate prediction and data packet level simulation according to an embodiment of this application; Figure 8 This is a schematic diagram illustrating the consistency between finite population analytical average queue prediction and packet-level simulation according to an embodiment of this application; Figure 9 This is a schematic diagram illustrating the performance improvement of the counterfactual Pareto screening PPO relative to the baseline according to an embodiment of this application; Figure 10 This is a schematic diagram of the Pareto screening and PPO front refinement gain source decomposition according to an embodiment of this application; Figure 11 This is a schematic diagram of the functional modules of an embodiment of the multi-access point wireless network reinforcement learning scheduling device of this application; Figure 12 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation
[0014] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0015] To address the shortcomings of existing technologies, refer to Figure 3 , Figure 3 This is a schematic diagram of the closed-loop process of online scheduling and training of a multi-access point wireless network according to an embodiment of this application, as follows: Figure 3As shown, S1: State acquisition, i.e., input: link, queue, arrival rate; S2: Valid candidate generation, i.e., input: access point-resource pool set; S3: Counterfactual completion, input: prefix + candidate + common suffix; S4: Parsing evaluation, output: link, scheduling, queue, deadline; S5: Pareto selection, output: non-dominated action mask; S6: Policy selection, method: in-mask sampling or selection; S7: Radio-side execution, method: forming a real limited contention group and completing transmission; S8: Performance feedback, feedback indicators: throughput, latency, packet loss rate, and queue; S9: Update reinforcement learning policy using real rewards, key principle: the parsing model is determined by observable states and protocol rules.
[0016] Reference Figure 1 This application discloses a reinforcement learning scheduling method for multi-access point wireless networks, the method comprising: Step S10: Obtain the current status information of multiple terminals in the wireless communication network; In this embodiment, an FTTR (Fiber to the Radio) multi-access point wireless network is used as the application scenario. The system includes a main FTTR gateway (or main optical network unit), N wireless terminals, M wireless access points connected by optical fibers, and R_m resource pools under access point m. For an FTTR system, a wireless access point can be a slave optical network unit or a wireless access node connected to the main gateway by optical fiber; a resource pool can represent OFDMA resource units, orthogonal time-frequency resource groups, scheduling queues, frequency bands, beam groups, spatial streams, or protocol-defined contention domains.
[0017] The main FTTR gateway or centralized controller collects terminal-side and network-side status data.
[0018] Terminal-side status data collection includes: location or service area identifier. Optional movement state Current queue length Business reach intensity Link quality measurement and the actions of the previous control cycle .
[0019] Network-side status collection includes: the available resource pool status of each access point and the legality constraints of candidate access points and resource pools.
[0020] The observable state of the i-th terminal at control time t Represented as:
[0021] in, For terminal location or service area identification, This is an optional movement state. This is the current queue length. For business reach intensity, To measure link quality such as RSSI, Reference Received Power (RSRP), Signal-to-Interference-plus-Noise Ratio (SINR), CSI, or obstruction status for each candidate access point, This is the action from the previous control cycle.
[0022] Furthermore, in the formula, Let represent the observable state vector of the i-th terminal at control time t, where i∈{1,…,N}, N represents the total number of terminals participating in the current control cycle scheduling, and t represents the control time. Indicates location or service area identification. Indicates the optional movement state. Indicates the current queue length. Indicates the intensity of business arrival. This represents the link quality measurement vector obtained for each candidate access point. This indicates the action performed by terminal i in the previous control cycle.
[0023] Step S20: For each candidate action of the current terminal, construct a complete joint action corresponding to the candidate action, wherein an action of the current terminal represents a radio resource contention group selected by the current terminal. In some specific embodiments, step S20 includes: According to the predetermined decision-making order, actions are generated for each terminal in turn, wherein an action of the current terminal represents a radio resource contention group selected by the current terminal; For each candidate action of the current terminal, a complete joint action corresponding to the candidate action is constructed based on the selected actions of the processed terminals, the candidate action, and the completion action of the terminal to be processed.
[0024] In this embodiment, a set of legal access points—resource pools—is constructed for each terminal based on coverage, terminal frequency band capabilities, access point availability, resource budget, and protocol constraints.
[0025] The action of the i-th terminal is the access point and resource pool tuple:
[0026]
[0027] In the formula, This represents the action of the i-th terminal. This represents the access point number selected by the i-th terminal. This indicates the resource pool number selected by the i-th terminal under this access point.
[0028] Furthermore, in the formula, A(t) represents the joint action of all N terminals at control time t; This represents the action of terminal i at time t. Indicates the access point number selected. This indicates the resource pool number selected under the corresponding access point; N represents the total number of terminals.
[0029] At the start of each control cycle, the main FTTR gateway or centralized controller determines the decision order of N terminals according to predetermined rules. The predetermined order can be determined by: fixed order of terminal number, order of queue priority, order of service duration, or random replacement.
[0030] Reference Figure 4 The controller generates actions for each terminal sequentially according to a predetermined decision order. During the action generation process of the current terminal i: Processed terminals: Terminal 1 to Terminal i The action for step 1 has been determined, forming the selected action prefix: ; Current terminal: Terminal i is currently making a decision on its action and needs to select from its set of legal candidate actions. Choose one from the options; Terminals to be processed: The actions of terminals i+1 to N have not yet been determined.
[0031] For each legal candidate action of the current terminal i The controller performs the following operations: (1) Get the prefix of the selected action.
[0032] The controller obtains the decisions made by terminals 1 to i. The selected action for 1 is denoted as: These actions have been determined and recorded by the policy network before the current terminal makes its decision; (2) Determine the current candidate action.
[0033] Current candidate action This indicates that the current terminal i will be assigned to the access point. The resource pool below The candidate action is selected from the set of legal candidate actions for the current terminal. This set is pre-constructed by the controller based on coverage, terminal frequency band capabilities, access point availability, resource budget, and protocol constraints. Each candidate action represents a feasible access point and resource pool allocation scheme.
[0034] In the formula, Represents the set of legal candidate actions for terminal i; Indicates the candidate action currently being evaluated; and These represent the access point number and resource pool number corresponding to the candidate action, respectively.
[0035] (3) Determine the completion action for the terminal to be processed.
[0036] For terminals i+1 to N that have not yet made a decision, the controller determines their completion actions according to preset completion rules. The preset completion rules can be implemented in any of the following ways: Method 1: Completion rules based on signal quality.
[0037] Each terminal to be processed is assigned to the legitimate access point with the highest measured signal quality (e.g., highest RSSI or highest SINR) for that terminal, and the resource pool is determined based on the terminal number or a deterministic hash map. This rule remains identical when comparing different current candidate actions.
[0038] Method 2: Completion rules based on load balancing.
[0039] Each terminal to be processed is assigned to the legal access point with the lowest current load, and the resource pool is determined based on the terminal number or deterministic hash mapping.
[0040] Method 3: Completion rules based on historical actions.
[0041] Set the completion action of each terminal to be processed as the action of its previous control cycle. In this embodiment, we will take Method 1 as an example: The controller assigns each terminal to be processed to the access point with the highest RSSI, and determines the resource pool number according to the parity of the terminal number (odd-numbered terminals are assigned to resource pool 0, and even-numbered terminals are assigned to resource pool 1).
[0042] (4) Assemble the complete joint action. The controller will prefix the selected action. The completion actions for the current candidate action c and the terminal to be processed. Concatenate the terminals in sequence to construct the complete joint action corresponding to the current candidate action c:
[0043] In the formula, This represents the complete counterfactual joint action constructed for the candidate action c of the current terminal i; Indicates terminals 1 to i The selected action prefix of 1; Indicates the current candidate action, where and These are the access point number and resource pool number selected by the current terminal, respectively; This represents the common reference suffix obtained by terminals i+1 to N according to the same preset completion rule; N represents the total number of terminals participating in the decision-making of the current control cycle; In this context, the pre part of A represents the already determined action prefix, c represents the current candidate, and the ref part of A represents the common reference action of terminals that have not yet made a decision.
[0044] This complete joint action includes the action configurations of all N terminals, and is used to simulate a complete action configuration of multiple terminals when the current candidate action c is adopted by the current terminal i.
[0045] The controller iterates through the set of legal candidate actions for the current terminal i. For each candidate action, repeat steps (1) to (4) above to construct the corresponding complete joint action for each candidate action.
[0046] Preferably, the reference action assigns the remaining terminals to the legitimate access point with the highest measured signal quality or the lowest load, and maps the resource pool according to the terminal number or deterministic hash. The reference suffix does not need to be optimal, but it must satisfy the following requirements: legitimate, deterministic, repeatable, not reading future information, and maintaining the same rules when comparing different current candidates.
[0047] By using a complete counterfactual action, all members of the resource group to which the current candidate belongs can be determined, and a four-dimensional parsing prior can be calculated. A common suffix ensures that the differences between different candidates arise solely from the current action, avoiding additional confounding for competition groups and performance comparisons caused by candidate-related completion rules.
[0048] The aforementioned counterfactual joint action completion mechanism resolves the technical issue in autoregressive strategies where "the resource competition group formed by the current candidate is uncertain, leading to a lack of complete joint action context in local action evaluation." The common reference suffix ensures that the distinction between different candidates stems solely from the current action, preventing additional contamination of competition groups and performance comparisons by candidate-related completion rules. Consequently, each candidate action can be fairly compared within the complete context of "all terminals have completed action configuration," providing an accurate data foundation for subsequent Pareto safety screening.
[0049] Step S30: Under the complete joint action, calculate the resource acquisition opportunity of the current terminal under the complete joint action based on the current state information; In some specific embodiments, calculating the resource acquisition opportunity of the current terminal under the complete joint action based on the current state information includes: The activity probability of each terminal in the same resource contention group as the current terminal is determined based on the current status information. Obtain the total number of terminals K in the resource contention group to which the current terminal belongs, and the activity probability of the other K-1 terminals besides the current terminal; A probability distribution is constructed based on the activity probabilities of the remaining K-1 terminals. This probability distribution is used to describe the distribution of the number of active terminals among the remaining K-1 terminals. Using the probabilities corresponding to different numbers of active competitors in the probability distribution as weights, the corresponding values of the current terminal obtaining scheduling opportunities under various active competitor numbers are weighted and summed to obtain the expected value of the current terminal obtaining scheduling opportunities. The expected value is used as the resource acquisition opportunity of the current terminal under the complete joint action.
[0050] In this embodiment, in the complete joint action Next, the controller determines other terminals within the same resource contention group as the current terminal i, based on the access point and resource pool allocation for each terminal in this complete joint action. Given a complete joint action A, terminals with the same access point m and the same resource pool r constitute a limited contention group for the action condition:
[0051] in, This indicates the access point assigned to terminal i in the complete joint action. This indicates the resource pool that has been allocated.
[0052] Based on the current state information, the controller determines the activity probability of each of the K-1 terminals in the competition group, excluding the current terminal. The activity probability represents the probability that a terminal will participate in resource contention within the current time slot.
[0053] (1) Terminals whose current queue is not empty.
[0054] If the terminal's current queue length This indicates that the terminal currently has data waiting to be sent and will inevitably participate in resource contention. Its activity probability is: .
[0055] (2) Terminals whose current queue is empty.
[0056] If the terminal's current queue length The terminal has no data to send at the start of the current time slot. However, under the Poisson arrival assumption, the terminal may generate new data packets within a time slot, thus participating in the contention.
[0057] Assume the service reach strength of this terminal is (Number of data packets arriving per unit time), one time slot length is The probability that the terminal generates at least one data packet within a time slot is: .
[0058] In the formula, This represents the probability that terminal i within the competition group is active in resource competition in the current time slot; This represents the queue length of terminal i at time t; Δt represents the service arrival intensity of terminal i; Δt represents the duration of a transmission slot; exp(·) represents the natural exponential function.
[0059] This application preserves the actual identity and heterogeneous activity probability of each member in the group, avoiding the average approximation that conflates inactive terminals with highly active terminals.
[0060] Determine the activity probability of each of the remaining K-1 terminals within the competition group. Then, the controller constructs a probability distribution to describe the distribution of the number of active competitors among the remaining K-1 terminals.
[0061] Due to the activity probability of each terminal Due to heterogeneity, the number of active competitors N follows a Poisson binomial distribution.
[0062] (1) Probability generating function.
[0063] The probability generating function of the Poisson binomial distribution is: .
[0064] (2) Expand into a polynomial.
[0065] Expand the probability generating function into a polynomial: .
[0066] Among them, polynomial coefficients This represents the probability that exactly n terminals are active out of the remaining K-1 terminals.
[0067] In the formula, Let z represent the probability generation function for the number of active competitors in the limited competitive group to which terminal i belongs; z represents the auxiliary variable of the probability generation function. This indicates the resource contention group that terminal i currently belongs to. =| | indicates the total number of terminals in this competition group; This represents the probability that member j of the competing group is active; Indicates anything other than the current terminal i The probability that exactly n terminals are active out of n terminals is that n∈{0,…, }
[0068] (3) Calculate the coefficients of recursive convolution.
[0069] The controller uses recursive convolution to calculate the coefficients. When adding a Bernoulli competitor, the existing probability vector is multiplied by the inactive probability and the active probability respectively, and the latter part is shifted one position to the right.
[0070] The computational complexity is O(K²), and Monte Carlo sampling is not required. The integral can also be calculated numerically directly, or a prefix-suffix polynomial can be used to reduce redundant computation across multiple nodes.
[0071] The controller utilizes the coefficients of the Poisson binomial distribution. Calculate the expected value of the current terminal obtaining a scheduling opportunity.
[0072] (1) Conditional scheduling probability.
[0073] Under the equal-probability random round-robin scheduling rule, when there are n other active competitors, the competition group, including the current terminal, has a total of n+1 active terminals. Under equal-probability round-robin, the conditional probability of the current terminal obtaining a scheduling opportunity is 1 / (n+1).
[0074] (2) Calculation of expected value.
[0075] Current terminal scheduling opportunities for:
[0076] in, is the coefficient of the Poisson binomial distribution (i.e., the probability that the number of active competitors is n), and 1 / (n+1) is the conditional probability that the current terminal will be scheduled in this scenario.
[0077] Formula meaning: Utilizing the probability corresponding to the number of different active competitors in a probability distribution. As a weight, the conditional probability 1 / (n+1) of the current terminal obtaining a scheduling opportunity under various active competitor conditions is summed.
[0078] In another implementation, scheduling opportunities can also be calculated using an equivalent integral expression:
[0079] In the formula, This represents the expected value of terminal i obtaining a resource scheduling opportunity under the condition of complete joint action A; Let x represent the probability generating function corresponding to the competition group determined by the joint action; x∈[0,1] is the integral auxiliary variable.
[0080] This integral form reveals that scheduling opportunities are a multilinear function of the total active probabilities of the competing groups, which facilitates the analysis of monotonicity and the construction of numerical algorithms.
[0081] (3) Closed expressions in special circumstances.
[0082] When all nodes in the competition group have the same probability of being active (all are equal) Furthermore, when the number of participants in the competing group is K, a closed-form expression can be obtained:
[0083] In the formula, β represents the common active probability of each member of the competitive group under the special case of homogeneity, and K represents the total number of terminals in the competitive group; when β=0, there are no other active competitors, so the resource acquisition opportunity of terminal i is 1.
[0084] Theoretical properties of scheduling opportunities: Theory 1 (Scheduling Opportunity Bound): For any group of contested events and any... The finite total scheduling opportunities satisfy:
[0085] Proof: When all other competitors are inactive, the terminal monopolizes resources, and the scheduling opportunity is 1; when the remaining K-1 competitors are all active, K terminals poll with equal probability, and the scheduling opportunity is... Any combination of activity probabilities lies between two extremes.
[0086] Theory 2 (Competition Monotonicity): With other parameters fixed, increasing the active probability of any other member will not improve the scheduling opportunity of terminal i. Its partial derivative is:
[0087] In the formula, This represents the partial derivative of the change in the activity probability of terminal j with respect to the resource acquisition opportunity of terminal i. This represents the product of probability generating factors for the remaining competing members after excluding terminals i and j from the competition group containing terminal i; x is an integral auxiliary variable.
[0088] This property indicates that when a candidate action leads to an increase in highly active nodes in the competing group, the chance of parsing the scheduling will inevitably decrease, providing physical consistency for subsequent Pareto dominance determination.
[0089] This application does not use average load to replace the actual competing groups, but instead retains the real resource group members determined by candidate joint actions. The active probability of each competitor is calculated independently based on its own queue state and service arrival intensity, fully reflecting the heterogeneity of terminals. The Poisson binomial distribution accurately describes the probability distribution of the number of active competitors in a finite population, and the recursive convolution algorithm calculates it accurately in O(K²) complexity without the need for Monte Carlo sampling. The resulting scheduling opportunity is a precise mathematical representation of the scheduling process of finite resource competition groups in wireless communication systems. This scheduling opportunity, as the second dimension of the four-dimensional analytical performance vector, provides an accurate numerical basis for subsequent Pareto safety screening.
[0090] Step S40: Based on the resource acquisition opportunity and the network state parameters extracted from the current state information, a subset of candidate actions is selected from all candidate actions of the current terminal. In some specific embodiments, step S40 includes: Based on the resource acquisition opportunities and the network state parameters extracted from the current state information, a multidimensional performance metric for the current terminal under the candidate actions is constructed. For any first candidate action and second candidate action among all candidate actions, if the multidimensional performance metric of the second candidate action is not lower than the multidimensional performance metric of the first candidate action in every dimension, and is higher than the multidimensional performance metric of the first candidate action in at least one dimension, then it is determined that the second candidate action dominates the first candidate action. Delete the candidate actions that are dominated by at least one other candidate action from all the candidate actions, and the remaining candidate actions constitute the subset of the candidate actions; Among them, network status parameters include at least one of the following: link transmission quality parameters, queue status parameters, and service timeliness parameters; The multidimensional performance metrics include: the probability of successful link transmission of the current terminal, the resource acquisition opportunity, the predicted queue quality index, and the probability of service delivery within the specified time.
[0091] In this embodiment, for the complete joint action A of the current terminal i, the controller constructs a four-dimensional analytical performance metric based on the following information. :
[0092] In the formula, This represents the four-dimensional analytical performance vector of terminal i under the complete joint action A; Indicates the probability of successful transmission of data via the link. Indicates opportunities to acquire resources. This indicates the quality metric for the predicted cohort. It represents the probability of delivery within the business deadline; the superscript T indicates vector transpose.
[0093] The four dimensions are: probability of successful link transmission, resource acquisition opportunity (scheduling opportunity), predicted queue quality index, and probability of service delivery within the specified time. For each dimension, a higher numerical value indicates better performance.
[0094] First dimension: Probability of successful transmission via link .
[0095] The probability of successful transmission on the link represents the probability that the current terminal i will successfully complete a transmission on the access point determined by the candidate action c.
[0096] Calculation basis: The controller calculates the link quality based on the current status information. Calculate the probability of successful transmission of the link, including but not limited to: Wireless measurements such as RSSI, RSRP, SINR, and CSI; Path loss and wall obstruction information; Transmit power and antenna gain; Modulation coding scheme (MCS); Co-channel interference level.
[0097] Calculation method: .
[0098] in, This indicates that terminal i is at the candidate access point. Signal-to-interference-to-noise ratio (PER) The value represents the packet error rate under different signal-to-interference-plus-noise ratios and modulation / coding schemes. This value can be obtained through methods such as packet error rate / block error rate (PER / BLER) lookup tables, analytical integration, historical statistics, or reliability lower bounds.
[0099] In the formula, This represents the probability of terminal i successfully transmitting on the link determined by the candidate action; This indicates that terminal i is connecting to a candidate access point. Signal-to-interference-to-noise ratio (SIR) at that time; Indicates the modulation and coding scheme used by terminal i; PER( , The value represents the packet error rate under the corresponding signal-to-interference-plus-noise ratio and modulation / coding scheme.
[0100] Second dimension: Resource acquisition opportunities (scheduling opportunities) .
[0101] The resource acquisition opportunity is the scheduling opportunity value calculated in the previous embodiment:
[0102] Third dimension: Predicting queue quality metrics .
[0103] The predicted queue quality index reflects the expected queue state of the current terminal after one control cycle, under the current candidate action.
[0104] (1) Calculate the probability of successful transmission in a single time slot.
[0105] The probability that terminal i simultaneously obtains scheduling opportunities and successfully transmits within a time slot is:
[0106] In the formula, This represents the probability that terminal i can simultaneously obtain scheduling opportunities and successfully complete transmission within a time slot under a complete joint action A; This indicates the probability of successful transmission of the link under this action; This indicates the opportunity to acquire resources under this action.
[0107] That is, the product of the probability of successful transmission of the link and the scheduling opportunity.
[0108] (2) Calculate the expected departure volume.
[0109] Suppose a control cycle contains H transmission time slots. Using local prediction with the current state frozen, the expected departure amount within this cycle is:
[0110] in, This represents the current queue length (the number of data packets waiting to be sent). This represents the expected number of new data packets arriving within this period. This represents the number of data packets expected to be successfully transmitted within the period. This formula indicates that the actual number of packets sent is limited by the smaller of the total amount of data that can be sent and the transmission capacity.
[0111] In the formula, This represents the expected amount of traffic leaving terminal i within a control cycle; Indicates the current queue length; H represents the expected number of new data packets arriving within the control cycle; H represents the number of transmission time slots contained in a control cycle. This represents the probability of terminal i being active; Δt represents the number of data packets expected to be successfully transmitted within the period; Δt represents the length of a single transmission slot.
[0112] (3) Calculate the predicted queue length.
[0113]
[0114] That is, the current backlog plus new arrivals, minus the expected departures.
[0115] (4) Truncation and normalization.
[0116] Limit the prediction queue to [0, ... Within the range:
[0117] Then normalize to a queue quality metric where larger values are preferred:
[0118] In the formula, This indicates the predicted queue length of terminal i at the end of the current control cycle; This indicates that the prediction queue will be truncated to [0, ... The value after ]; Indicates the preset maximum queue capacity; This is the normalized prediction queue quality index, and a larger value indicates a smaller prediction queue backlog.
[0119] in, This is the preset maximum queue capacity.
[0120] Fourth dimension: Probability of delivery within the business deadline .
[0121] The probability of delivery within the specified service period represents the probability that a data packet in the current terminal queue will be successfully delivered within the specified service period.
[0122] (1) The first arrival time distribution of the marked data packets.
[0123] For a marked data packet with k-1 pending data packets ahead, the probability of it successfully completing the k-th service in the nth time slot follows a negative binomial distribution:
[0124] in, This represents the probability of successful transmission in a single time slot.
[0125] In the formula, This represents the probability that the tagged data packet of terminal i completes the k-th successful service in the nth time slot; n represents the time slot number where the service is completed, and k represents the number of successful services required to complete the delivery of the tagged data packet; Let n represent the probability of successful transmission in a single time slot for terminal i, where n ≥ k.
[0126] (2) Probability of delivery within the time limit.
[0127] The probability of delivery being completed within D time slots is:
[0128] In the formula, This indicates that there is a k ahead of terminal i. The probability that a tagged data packet of a pending data packet will be delivered within D time slots; D represents the maximum allowed number of time slots corresponding to the service period; This represents the probability quality function of the marked data packet completing the k-th successful service in the nth time slot; n represents the time slot number counting from the current time slot; and k represents the number of successful services required before the marked data packet is delivered.
[0129] (3) Comprehensive handling of multiple packages in the same team.
[0130] The current terminal's queue may contain multiple data packets, and these packets may enter the queue along with other randomly arriving services within the same time slot. Let J represent the number of data packets that can still be accepted. Let represent the probability that exactly j data packets arrive in the same time slot. Then, the overall delivery probability within the deadline is:
[0131] In the formula, This represents the overall delivery probability within the specified timeframe for terminal i; Indicates the maximum queue capacity The number of data packets that the current queue can still accept under the constraint; This represents the probability that exactly j data packets enter the terminal i queue before the data packet to be evaluated within the same arrival time slot; This indicates that the number of packets arriving first is not less than The tail probability; Indicates the current queue length; This indicates that there are a total of [number] packets ahead of the data packet to be evaluated. Probability of delivery within a specified timeframe for a pending data packet; This represents the probability of delivery within a given timeframe when the queue reaches its capacity limit.
[0132] Furthermore, the above four dimensions are combined into a four-dimensional performance vector for the complete joint action A:
[0133] The four components of this four-dimensional vector represent the performance of the link layer, medium access layer, queuing layer, and service duration layer, respectively, and are all uniformly aligned so that the larger the value, the better the performance.
[0134] Furthermore, after constructing all legal candidate actions for the current terminal i... After the corresponding four-dimensional performance measurement, the controller performs Pareto safety screening and deletes candidate actions that are strictly dominated.
[0135] The determination of Pareto dominance is as follows: Suppose the current terminal has two candidate actions c and c. The corresponding four-dimensional performance vectors are as follows: and .
[0136] A candidate action is defined as one that meets the following two conditions. Strict Pareto Domination Candidate Action c: ① Each dimension is no less than: For each of the four dimensions k∈{1,2,3,4}, we have ; ② At least one dimension is strictly higher than: There exists at least one dimension k such that .
[0137] Right now: .
[0138] like If c is dominated, then candidate action c is... Strict control means that under all reasonable business objectives, c cannot be better than .
[0139] Furthermore, the non-dominated candidate actions are screened as follows: set up Let the set of actions strictly dominated by at least one legal candidate action be defined. Then the Pareto non-dominated set is: .
[0140] In the formula, This represents the set of actions among all legal candidate actions of terminal i that are strictly dominated by at least one other candidate action; This represents the complete set of legal candidate actions for terminal i. This represents the set of Pareto non-dominated candidate actions.
[0141] The controller removes candidate actions that are dominated by at least one other candidate action from all candidate actions, and the remaining candidate actions constitute a subset of candidate actions (i.e., the Pareto non-dominated candidate action set).
[0142] In this embodiment, no weights are set between throughput, latency, link success, queue, and deadline metrics. An action mask is constructed using Pareto dominance. For a business utility function with no monotonically decreasing components, deleting a dominated candidate does not change the optimal value.
[0143] That is, if the optimal candidate If strictly dominated by another candidate c′, then by the monotonically non-decreasing property of the components of U, we have U(z(c′)) ≥ U(z( Therefore, there exists a value that is no worse than The candidate is replaced along the dominance relation; eventually, a non-dominated candidate is reached in the finite candidate set, whose utility is no less than that of the original optimal candidate. Therefore, deleting a dominated candidate does not change the optimal value.
[0144] In this embodiment, instead of using a set of manual weights to select actions, candidates that are unnecessary to retain for any monotonic business objective are removed. For throughput priority, reliability priority, latency priority, and weighted utility with different weights, as long as the four quality components remain monotonically constant, at least one of the optimal actions will be retained in the non-dominated set.
[0145] A comprehensive four-dimensional performance profile: the probability of successful link transmission characterizes the physical layer quality, scheduling opportunities characterize the media access layer's competitive capability, predicted queue quality characterizes the queuing layer's backlog risk, and the probability of timely delivery characterizes the guarantee of service timeliness. Decoupling from real rewards: The analytical model only serves as an action selection tool, while the reinforcement learning rewards and value objectives are still generated by the real grouping and queuing environment, preventing prior knowledge from replacing real objectives and causing system bias; Physical consistency: The four-dimensional prior has a consistent direction of advantage and disadvantage in physical terms—an increase in the probability of link success or scheduling opportunities will improve the service probability, reduce the prediction queue, and ensure that the probability of delivery within the deadline is not reduced.
[0146] Furthermore, in one embodiment, the multidimensional performance metric is a predictive performance metric, and the method further includes: Obtain the prediction error range for each dimension of the prediction performance metric; The lower and upper bounds of the performance of each candidate action in each dimension are determined based on the prediction performance metric and the prediction error range, respectively. For any first candidate action and second candidate action, if the lower bound of the performance of the second candidate action in each dimension is not lower than the upper bound of the performance of the first candidate action in each dimension, and is higher than the upper bound of the performance of the first candidate action in at least one dimension, then it is determined that the second candidate action dominates the first candidate action within the error range. Candidate actions are selected based on the relationships that govern within the error range.
[0147] In this embodiment, the controller constructs a four-dimensional analytical performance metric. At the same time, the error range corresponding to the predicted values of each dimension is obtained.
[0148] (1) Sources of error.
[0149] The possible sources of prediction error for the four-dimensional performance metric are shown in Table 1.
[0150] Table 1
[0151] (2) Methods for determining the error bound.
[0152] For the k-th performance dimension (k=1,2,3,4) of candidate action c, the controller determines its prediction error bound. , making .
[0153] in, For predicted values, This is the actual value.
[0154] Error bound It can be determined through empirical statistics, theoretical derivation, expert setting, or online estimation.
[0155] Furthermore, for each candidate action c, the controller determines the action based on the predicted value. and error bound Calculate the lower and upper bounds of performance for each dimension.
[0156] Performance lower bound: .
[0157] The performance lower bound represents the minimum value that the actual performance can achieve after taking into account prediction error (worst-case estimate).
[0158] Upper limit of performance: .
[0159] In the formula, This represents the predicted value of candidate action c in the k-th performance dimension. This represents the corresponding actual performance value. Indicates the prediction error bound; and represents the lower and upper bounds of performance after considering the error bound, respectively, and k∈{1,2,3,4} corresponds to the probability of successful transmission of the link, the opportunity to acquire resources, the predicted queue quality index, and the probability of delivery within the service deadline, respectively.
[0160] The performance upper bound represents the maximum possible value of the actual performance after taking into account prediction error (best-case estimate).
[0161] Furthermore, based on the point estimation Pareto dominance (using direct comparison of predicted values), this embodiment introduces a robust dominance relationship to ensure the reliability of the dominance determination when prediction errors exist.
[0162] For any first candidate action c and second candidate action If the following two conditions are met: ① The lower bound of each dimension is not lower than the upper bound: For each of the four dimensions k∈{1,2,3,4}, we have:
[0163] That is, candidate actions The performance in the worst case (lower bound) is no less than the performance of candidate action c in the best case (upper bound). ② At least one dimension is strictly above the upper bound: There exists at least one dimension k such that:
[0164] This is called a candidate action. Robustly dominates candidate action c within the error range.
[0165] Furthermore, when When robustly dominating c, even considering all possible prediction errors, candidate actions The actual performance of the algorithm remains no worse than that of the candidate action c across all dimensions, and is strictly better in at least one dimension. Therefore, it can be guaranteed that the algorithm will outperform the actual performance of the candidate action c in all dimensions. Dominate c.
[0166] Therefore, when model error is not negligible, confidence interval dominance can be used instead of point estimation dominance to avoid erroneously removing true non-dominated actions. The current point estimation implementation sets the error bound to zero, thus degenerating into a strict Pareto screening.
[0167] The controller iterates through the set of all legal candidate actions for the current terminal i. The selection is based on robust dominance relationships.
[0168] The filtering rules are as follows: If candidate action If a candidate action c is robustly dominated, then the candidate action c is marked as robustly dominated; a candidate action that is robustly dominated by at least one other candidate action is deleted; candidate actions that are not robustly dominated by any other candidate action constitute a robust Pareto non-dominated set.
[0169] When the prediction model is accurate enough, the error bounds of each dimension are all zero:
[0170] but, .
[0171] Robust dominance degenerates into strict Pareto dominance: .
[0172] At this point, robust screening is equivalent to point estimation Pareto screening.
[0173] The robust Pareto safe screening based on the prediction error bound described above avoids the erroneous deletion of true non-dominated candidates by constructing robust dominance relations using upper and lower confidence bounds when there is a known error bound in the analytical prior. When candidate actions When the lower bound of the prediction is not lower than the upper bound of the prediction of candidate action c, it can be guaranteed that under real-world performance... The control of c ensures the reliability of the screening; Error bounds can be obtained through various methods such as empirical statistics, theoretical derivation, expert setting, or online estimation, to adapt to the accuracy requirements of different deployment scenarios; The point estimation method can be naturally extended. When the error bound is zero, the robust screening degenerates into the strict Pareto screening, which is seamlessly compatible with the point estimation method.
[0174] Step S50: Input the current state information into the reinforcement learning policy, and constrain the decision space of the reinforcement learning policy according to the candidate action subset, so that the reinforcement learning policy outputs the current terminal's execution action within the range defined by the candidate action subset.
[0175] In some specific embodiments, constraining the decision space of the reinforcement learning policy based on the subset of candidate actions includes: Obtain the original probability distribution of the reinforcement learning policy on all candidate actions of the current terminal; Set the probabilities in the original probability distribution that correspond to actions outside the subset of candidate actions to zero; The probability distribution after being set to zero is renormalized so that the sum of the probabilities within the subset of candidate actions is one.
[0176] In this embodiment, the controller will display the current state information of terminal i. Input reinforcement learning policy network:
[0177] In the formula, This represents the observable state vector of the i-th terminal at control time t; , , , , and The meanings are consistent with the aforementioned state definitions, representing location or service area identifier, optional mobility state, current queue length, service arrival strength, link quality measurement, and actions from the previous control cycle, respectively.
[0178] Reinforcement learning strategies can be implemented using algorithms such as PPO (Proximal Policy Optimization), Actor-Critic, SAC (Flexible Actor-Critic), DQN (Deep Q-Network), or Policy Gradient.
[0179] In this embodiment, the policy network (Actor network) of the PPO algorithm is used as an example for explanation. Policy network reception state. As input, its output layer generates all legal candidate actions for the current terminal. The original probability distribution on.
[0180] The policy network receives the current state. Then, through forward propagation calculations in the neural network, a dimension equal to the number of legal candidate actions is generated in the output layer. The original probability distribution of |.
[0181] Let the set of legal candidate actions for the current terminal i be:
[0182] Where L=∣ | represents the total number of valid candidate actions.
[0183] The original probability distribution output by the policy network is: The distribution satisfies: .
[0184] In the formula, Let θ represent the original probability of a reinforcement learning policy with parameter θ selecting candidate action c in state s_i; θ represents the policy network parameters. Let L represent the set of legal candidate actions for terminal i; L represents the total number of legal candidate actions.
[0185] The controller obtains the set of non-dominated candidate actions based on the Pareto safety filter. (Candidate action subset), construct action mask :
[0186] That is: if candidate action c belongs to the non-dominated set If the mask value is 1, then the mask value is 1; otherwise, the mask value is 0.
[0187] Action mask Its function is to indicate which candidate actions are allowed to be selected by the policy network.
[0188] Furthermore, the controller utilizes motion masks. Constrain the original probability distribution. (This refers to) the original probability distribution. The middle corresponds to the dominated candidate action (i.e. The probability of (=0) is set to zero.
[0189] The controller renormalizes the probability distribution after setting it to zero, making the set of non-dominated candidate actions more uniform. The sum of the probabilities within is 1.
[0190] Normalization constant (partition function) for:
[0191] The normalized constraint policy distribution is as follows:
[0192] In the formula, Indicates the candidate action mask. =1 indicates that candidate action c belongs to the non-dominated set. And it can be selected. =0 indicates that the candidate action is masked; 1{·} represents the indicator function; This represents the unnormalized probability of actions after masking. This represents the probability of the constraint policy after renormalization. This represents the normalization constant.
[0193] This distribution satisfies: hour, .
[0194] That is, the policy network only applies to the set of non-dominated candidate actions. Internal sampling or selection action.
[0195] The controller is distributed according to the normalized constraint policy. Select the action to perform for the current terminal.
[0196] The selection method can be either random sampling or greedy selection.
[0197] Random sampling: Random sampling is performed according to the normalized probability distribution, with probability... Select candidate action c; Greedy selection: Select the candidate action with the highest probability after normalization as the action to be executed.
[0198] Finally, assume that the policy network samples and selects candidate action B as the action to be executed by the current terminal i:
[0199] In the formula, B represents the candidate action ultimately selected by the policy network; in the example, AP 1 indicates that the number is AP Access point 1, RB 1 indicates that the access point is numbered RB A resource pool of 1; This indicates the action that is ultimately determined and recorded by the current terminal i.
[0200] The action is recorded as the selected action and used for subsequent terminal decisions.
[0201] By using the above action mask-based reinforcement learning decision space constraint method, reinforcement learning only needs to sample from an average number of non-dominated candidate actions, which is usually significantly smaller than the total number of legal candidate actions L, greatly reducing the number of explorations of obviously inferior actions and reducing exploration complexity. Based on Theory 1, under any business utility function that is monotonically non-decreasing for any four-dimensional component, the candidate actions excluded by the action mask cannot become optimal. Therefore, constraining the decision space does not change the optimal value and can maintain optimality. The analytical model is responsible for eliminating deterministic bad actions (constructing a mask), while the real environment is responsible for evaluating the long-term effects between non-dominated actions, thus avoiding the direct propagation of analytical model errors to policy updates and achieving separation of responsibilities. Versatility: This method is applicable to various reinforcement learning algorithms such as PPO, actor-critic, SAC, DQN, and policy gradient, and has strong versatility.
[0202] Furthermore, in one embodiment, the method further includes: Actions are generated for all terminals according to the predetermined decision order, and a complete set of execution actions is formed based on the actions finally determined by each terminal. The complete set of execution actions is sent to each access point in the wireless communication network, so that each access point can perform data transmission according to the complete set of execution actions; Collect actual network performance feedback after the data transmission is executed, and generate a reward signal based on the actual network performance feedback; The parameters of the reinforcement learning policy are updated using the reward signal.
[0203] In this embodiment, after generating actions for all terminals according to a predetermined decision order, the actions finally determined by each terminal constitute a complete set of execution actions:
[0204] in, The action ultimately determined by terminal i For access point numbering, Number the resource pool.
[0205] The controller will distribute the complete set of actions to each access point in the wireless communication network. Specific distribution methods include: FTTR scenario: The main FTTR gateway or main optical network unit associates the access points of each terminal and configures the resources, and then sends the information to each slave optical network unit (wireless access point) through optical fiber connection. Centralized controller scenario: Data is distributed from a centralized network controller or edge computing node to each access point via a wired or wireless backhaul network; Cellular network scenario: Data is sent from the cellular base station controller to each base station.
[0206] The distributed information includes a list of terminals that each access point needs to serve and their corresponding resource pool configurations.
[0207] After each access point receives and parses the complete set of execution actions issued by the controller, it executes the actual data transmission according to the terminal association and resource configuration specified therein.
[0208] Formation of Access Point-Resource Pool Groups: Terminals that select the same access point and the same resource pool form a real wireless resource contention group.
[0209] In the formula, This represents the actual resource contention group consisting of terminals that select the same access point m and the same resource pool r under the complete set of execution actions A; and These represent the access point number and resource pool number in the final action executed by terminal i, respectively.
[0210] For each non-empty access point-resource pool group, the access point performs packet transmission according to the predetermined scheduling rules.
[0211] In this embodiment, an equal-probability round-robin scheduling rule is adopted: Multiple terminals within the same resource contention group will be given scheduling opportunities in turn according to the polling order; Each terminal that gets a scheduling opportunity attempts to transmit data based on its current channel state (link success probability); If the terminal's current queue is not empty and the link transmission is successful, the terminal will complete the data transmission in that time slot; if the queue is empty or the link transmission fails, the time slot resource will be idle or occupied by retransmission.
[0212] During data transmission, the system performs statistical analysis on the actual network performance of each terminal.
[0213] The actual network performance metrics included in the statistics are: throughput, end-to-end latency, packet loss rate, and average queue size.
[0214] The access point summarizes and reports the actual network performance metrics to the controller. The controller receives the actual network performance feedback and generates reward signals based on this feedback.
[0215] Real-world rewards are uniformly expressed as environmental utility functions:
[0216] in, For throughput, For time delay, For packet loss rate, This represents the average queue length.
[0217] In the formula, U_env(·) represents the reward signal generated by the real network environment at control time t; U_env(·) represents the environment utility function. , , and These represent the throughput, end-to-end latency, packet loss rate, and average queue length obtained at time t, respectively.
[0218] Environmental utility function It can take many forms, such as weighted linear combination, product form, and threshold reward.
[0219] The controller utilizes environmental reward signals Update the parameters of the reinforcement learning strategy.
[0220] In this embodiment, the principle of separation of duties is strictly followed: Analytical model: Only responsible for action filtering (constructing action masks), excluding deterministic bad actions; Real-world environment: responsible for evaluating the long-term effects between non-dominated actions and generating environmental rewards; Reinforcement learning: Update policy parameters using real-world rewards.
[0221] This reward is only used for strategy learning and does not participate in the analysis of Pareto screening.
[0222] In this embodiment, the PPO (Proximity Policy Optimization) algorithm is used to update the reinforcement learning policy parameters.
[0223] This application does not rely on a specific neural network architecture. Besides PPO, reinforcement learning strategies can also be implemented using the following algorithms: Actor-Critic: Simultaneously learns the policy network and value network, and updates them using temporal difference error; SAC (Soft Actor-Critic): A reinforcement learning framework based on maximum entropy that maximizes policy entropy while maximizing reward. DQN (Deep Q-Network): Learning State-Action Value Function Choose the action with the highest value; Policy Gradient: Directly parameterizes the policy and optimizes the expected cumulative reward through gradient ascent.
[0224] Through the complete execution process of action distribution, real-world execution, and policy update described above, the parsing model only plays the role of action selection, while the rewards generated by the real-world environment are used for policy updates, thus avoiding the direct propagation of parsing model errors into the policy network. The reinforcement learning rewards and value objectives are generated based on real grouping and queuing environments, without relying on any approximate assumptions of analytical models; Policy updates can employ various reinforcement learning algorithms such as PPO, actor-critic, SAC, DQN, or policy gradient, without being limited by a specific network structure; In this embodiment, the current state information of multiple terminals in the wireless communication network is obtained; for each candidate action of the current terminal, a complete joint action corresponding to the candidate action is constructed, wherein an action of the current terminal represents a wireless resource contention group selected by the current terminal; by constructing a complete joint action, the contention group and resolution performance can be determined, thereby avoiding the problem of lack of complete joint action context in local action evaluation. Under the complete joint action, the resource acquisition opportunity of the current terminal under the complete joint action is calculated based on the current state information; based on the resource acquisition opportunity and the network state parameters extracted from the current state information, a subset of candidate actions is selected from all candidate actions of the current terminal; the set of all candidate actions is compressed into a subset of candidate actions, which solves the problem that the existing technology relies on trial and error to identify superior and inferior actions, and does not use communication and queuing mechanisms to eliminate deterministic inferior candidates before policy sampling, resulting in low sample efficiency; The current state information is input into the reinforcement learning policy, and the decision space of the policy is constrained based on the subset of candidate actions, so that the policy outputs the current terminal's execution action within the range defined by the subset of candidate actions. The constrained reinforcement learning only needs to sample from an average number of non-dominated candidate actions, which is typically significantly smaller than the total number of legitimate candidates, reducing the complexity of reinforcement learning exploration and reasoning.
[0225] Furthermore, to verify the accuracy of the finite-total analytical performance model as a priori for action selection, this embodiment compares the analytical prediction results with the packet-level simulation results on a scenario-by-scenario basis under the same network conditions and action configurations. (Refer to...) Figure 5 The finite total analytical throughput prediction and the packet-level simulated throughput are basically distributed around the ideal concordance line, with a correlation coefficient of 1.000 and a normalized mean absolute error (NMAE) of 0.64%, indicating that the analytical model can accurately predict throughput changes under different competition groups and resource configurations.
[0226] Reference Figure 6The correlation coefficient between the finite population analytical average latency prediction and the packet-level simulation results is 0.987, and the NMAE is 1.47%. Although the latency index is affected by random arrival, contention scheduling, and queuing processes, the analytical results still maintain a high degree of consistency with the packet-level simulation, indicating that the finite population model can effectively characterize the impact of resource contention on average latency.
[0227] Reference Figure 7 The correlation coefficient between the finite population analytical end-to-end packet loss rate prediction and the data packet level simulation results is 0.997, and the NMAE is 3.18%. The sample populations under different packet loss levels are distributed along an ideal congruent line, indicating that the analytical model can reflect the changes in packet loss risk caused by link failure, resource contention, and queue backlog.
[0228] Reference Figure 8 The correlation coefficient between the finite population analytical average queue prediction and the packet-level simulation results is 0.999, and the NMAE is 3.00%. The analytical prediction value can change synchronously with the actual average queue length, indicating that the queue evolution relationship formed by link success probability, resource acquisition opportunities, and service arrival can be accurately characterized by the analytical prior. Figures 5 to 8 It can be seen that the analytical model has high predictive consistency across different levels of metrics such as throughput, latency, packet loss rate, and average queue size. Therefore, it is suitable as a basis for screening candidate actions before reinforcement learning sampling, without having to replace real-world rewards with analytical predictions.
[0229] Reference Figure 9 In a paired evaluation of 100 identical scenarios, compared to the baseline scheme, the scheduling method combining counterfactual joint action completion, Pareto selection, and reinforcement learning described in this application achieved relative improvements of approximately 10.9%, 22.8%, 14.3%, 60.0%, and 37.2% in terms of overall score, throughput, average latency, end-to-end packet loss rate, and average queue, respectively. For average latency, end-to-end packet loss rate, and average queue, the relative improvement represents a reduction in the corresponding metrics. The 95% confidence interval in the figure indicates that the above improvements have good stability across multiple random network states.
[0230] Reference Figure 10 After normalizing the overall framework screening gain to 100%, the relative contribution of PPO further refinement within the Pareto front is generally close to zero, showing only small positive or negative changes on different indicators. Therefore, the main performance gain of this application comes from the early elimination of deterministic inferior actions based on finite population analytical priors and Pareto dominance, while reinforcement learning is mainly used to learn long-term utility and multi-indicator trade-offs among the retained non-dominated candidate actions; the small negative refinement contributions of individual indicators reflect the trade-offs between indicators under the long-term comprehensive objective.
[0231] Based on the same inventive concept, this application also provides a multi-access point wireless network reinforcement learning scheduling device.
[0232] In one embodiment, reference is made to Figure 11 , Figure 11 This is a functional module diagram of an embodiment of the multi-access point wireless network reinforcement learning scheduling device of this application. Figure 11 As shown, the multi-access point wireless network reinforcement learning scheduling device includes: The information acquisition module 10 is configured to acquire the current status information of multiple terminals in the wireless communication network; The complete joint action construction module 20 is configured to construct a complete joint action corresponding to each candidate action of the current terminal, wherein an action of the current terminal represents a radio resource contention group selected by the current terminal. The calculation module 30 is configured to calculate the resource acquisition opportunities of the current terminal under the complete joint action based on the current state information. The filtering module 40 is configured to filter a subset of candidate actions from all candidate actions of the current terminal based on the resource acquisition opportunity and the network state parameters extracted from the current state information. The action acquisition module 50 is configured to input the current state information into the reinforcement learning policy and constrain the decision space of the reinforcement learning policy according to the subset of candidate actions, so that the reinforcement learning policy outputs the current terminal's action within the range defined by the subset of candidate actions.
[0233] Furthermore, in one embodiment, the complete joint action construction module 20 is configured to: According to the predetermined decision-making order, actions are generated for each terminal in turn, wherein an action of the current terminal represents a radio resource contention group selected by the current terminal; For each candidate action of the current terminal, a complete joint action corresponding to the candidate action is constructed based on the selected actions of the processed terminals, the candidate action, and the completion action of the terminal to be processed.
[0234] Furthermore, in one embodiment, the computing module 30 is configured to: The activity probability of each terminal in the same resource contention group as the current terminal is determined based on the current status information. Obtain the total number of terminals K in the resource contention group to which the current terminal belongs, and the activity probability of the other K-1 terminals besides the current terminal; A probability distribution is constructed based on the activity probabilities of the remaining K-1 terminals. This probability distribution is used to describe the distribution of the number of active terminals among the remaining K-1 terminals. Using the probabilities corresponding to different numbers of active competitors in the probability distribution as weights, the corresponding values of the current terminal obtaining scheduling opportunities under various active competitor numbers are weighted and summed to obtain the expected value of the current terminal obtaining scheduling opportunities. The expected value is used as the resource acquisition opportunity of the current terminal under the complete joint action.
[0235] Furthermore, in one embodiment, the filtering module 40 is configured to: Based on the resource acquisition opportunities and the network state parameters extracted from the current state information, a multidimensional performance metric for the current terminal under the candidate actions is constructed. For any first candidate action and second candidate action among all candidate actions, if the multidimensional performance metric of the second candidate action is not lower than the multidimensional performance metric of the first candidate action in every dimension, and is higher than the multidimensional performance metric of the first candidate action in at least one dimension, then it is determined that the second candidate action dominates the first candidate action. Delete the candidate actions that are dominated by at least one other candidate action from all the candidate actions, and the remaining candidate actions constitute the subset of the candidate actions; Among them, network status parameters include at least one of the following: link transmission quality parameters, queue status parameters, and service timeliness parameters; The multidimensional performance metrics include: the probability of successful link transmission of the current terminal, the resource acquisition opportunity, the predicted queue quality index, and the probability of service delivery within the specified time.
[0236] Furthermore, in one embodiment, the multidimensional performance metric is a predictive performance metric, and the screening module 40 is further configured to: Obtain the prediction error range for each dimension of the prediction performance metric; The lower and upper bounds of the performance of each candidate action in each dimension are determined based on the prediction performance metric and the prediction error range, respectively. For any first candidate action and second candidate action, if the lower bound of the performance of the second candidate action in each dimension is not lower than the upper bound of the performance of the first candidate action in each dimension, and is higher than the upper bound of the performance of the first candidate action in at least one dimension, then it is determined that the second candidate action dominates the first candidate action within the error range. Candidate actions are selected based on the relationships that govern within the error range.
[0237] Furthermore, in one embodiment, the action acquisition module 50 is configured to: Obtain the original probability distribution of the reinforcement learning policy on all candidate actions of the current terminal; Set the probabilities in the original probability distribution that correspond to actions outside the subset of candidate actions to zero; The probability distribution after being set to zero is renormalized so that the sum of the probabilities within the subset of candidate actions is one.
[0238] Furthermore, in one embodiment, the device further includes a parameter update module configured to: Actions are generated for all terminals according to the predetermined decision order, and a complete set of execution actions is formed based on the actions finally determined by each terminal. The complete set of execution actions is sent to each access point in the wireless communication network, so that each access point can perform data transmission according to the complete set of execution actions; Collect actual network performance feedback after the data transmission is executed, and generate a reward signal based on the actual network performance feedback; The parameters of the reinforcement learning policy are updated using the reward signal.
[0239] The functions of each module in the multi-access point wireless network reinforcement learning scheduling device correspond to the steps in the multi-access point wireless network reinforcement learning scheduling method embodiment, and their functions and implementation processes will not be described in detail here.
[0240] Based on the same inventive concept, embodiments of this application also provide an electronic device, the structure of which is as follows: Figure 12 As shown, it includes: a memory and a processor, wherein the processor is used to read and execute the computer program stored in the memory to implement the aforementioned reinforcement learning scheduling method for a multi-access point wireless network.
[0241] Based on the same inventive concept, embodiments of this application also provide a computer storage medium storing computer-executable instructions, which, when executed, implement the aforementioned multi-access point wireless network reinforcement learning scheduling method.
[0242] Finally, it should be noted that while some processes described in the embodiments of this application include multiple operations or steps that appear in a specific order, it should be understood that these operations or steps may not be executed in the order they appear in the embodiments of this application, or may be executed in parallel. The sequence number of the operation is only used to distinguish different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed sequentially or in parallel, and these operations or steps may be combined.
[0243] Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.
Claims
1. A reinforcement learning scheduling method for multi-access point wireless networks, characterized in that, The method includes: Obtain the current status information of multiple terminals in a wireless communication network; For each candidate action of the current terminal, construct a complete joint action corresponding to the candidate action, wherein an action of the current terminal represents a radio resource contention group selected by the current terminal; Under the complete joint action, the resource acquisition opportunities of the current terminal under the complete joint action are calculated based on the current state information; Based on the resource acquisition opportunities and the network state parameters extracted from the current state information, a subset of candidate actions is selected from all candidate actions of the current terminal. The current state information is input into the reinforcement learning policy, and the decision space of the reinforcement learning policy is constrained according to the subset of candidate actions, so that the reinforcement learning policy outputs the current terminal's execution action within the range defined by the subset of candidate actions.
2. The reinforcement learning scheduling method for multi-access point wireless networks according to claim 1, characterized in that, For each candidate action of the current terminal, constructing a complete joint action corresponding to the candidate action includes: According to the predetermined decision-making order, actions are generated for each terminal in turn, wherein an action of the current terminal represents a radio resource contention group selected by the current terminal; For each candidate action of the current terminal, a complete joint action corresponding to the candidate action is constructed based on the selected actions of the processed terminals, the candidate action, and the completion action of the terminal to be processed.
3. The reinforcement learning scheduling method for multi-access point wireless networks according to claim 1, characterized in that, The step of calculating the resource acquisition opportunity of the current terminal under the complete joint action based on the current state information includes: The activity probability of each terminal in the same resource contention group as the current terminal is determined based on the current status information. Obtain the total number of terminals K in the resource contention group to which the current terminal belongs, and the activity probability of the other K-1 terminals besides the current terminal; A probability distribution is constructed based on the activity probabilities of the remaining K-1 terminals. This probability distribution is used to describe the distribution of the number of active terminals among the remaining K-1 terminals. Using the probabilities corresponding to different numbers of active competitors in the probability distribution as weights, the corresponding values of the current terminal obtaining scheduling opportunities under various active competitor numbers are weighted and summed to obtain the expected value of the current terminal obtaining scheduling opportunities. The expected value is used as the resource acquisition opportunity of the current terminal under the complete joint action.
4. The reinforcement learning scheduling method for multi-access point wireless networks according to claim 1, characterized in that, The step of selecting a subset of candidate actions from all candidate actions of the current terminal based on the resource acquisition opportunity and the network state parameters extracted from the current state information includes: Based on the resource acquisition opportunities and the network state parameters extracted from the current state information, a multidimensional performance metric for the current terminal under the candidate actions is constructed. For any first candidate action and second candidate action among all candidate actions, if the multidimensional performance metric of the second candidate action is not lower than the multidimensional performance metric of the first candidate action in every dimension, and is higher than the multidimensional performance metric of the first candidate action in at least one dimension, then it is determined that the second candidate action dominates the first candidate action. Delete the candidate actions that are dominated by at least one other candidate action from all the candidate actions, and the remaining candidate actions constitute the subset of the candidate actions; Among them, network status parameters include at least one of the following: link transmission quality parameters, queue status parameters, and service timeliness parameters; The multidimensional performance metrics include: the probability of successful link transmission of the current terminal, the resource acquisition opportunity, the predicted queue quality index, and the probability of service delivery within the specified time.
5. The reinforcement learning scheduling method for multi-access point wireless networks according to claim 4, characterized in that, The multidimensional performance metric is a predictive performance metric, and the method further includes: Obtain the prediction error range for each dimension of the prediction performance metric; The lower and upper bounds of the performance of each candidate action in each dimension are determined based on the prediction performance metric and the prediction error range, respectively. For any first candidate action and second candidate action, if the lower bound of the performance of the second candidate action in each dimension is not lower than the upper bound of the performance of the first candidate action in each dimension, and is higher than the upper bound of the performance of the first candidate action in at least one dimension, then it is determined that the second candidate action dominates the first candidate action within the error range. Candidate actions are selected based on the relationships that govern within the error range.
6. The reinforcement learning scheduling method for multi-access point wireless networks according to claim 1, characterized in that, The constraint on the decision space of the reinforcement learning policy based on the subset of candidate actions includes: Obtain the original probability distribution of the reinforcement learning policy on all candidate actions of the current terminal; Set the probabilities in the original probability distribution that correspond to actions outside the subset of candidate actions to zero; The probability distribution after being set to zero is renormalized so that the sum of the probabilities within the subset of candidate actions is one.
7. The reinforcement learning scheduling method for multi-access point wireless networks according to claim 1, characterized in that, The method further includes: Generate actions for all terminals according to the predetermined decision order, and form a complete set of execution actions based on the final actions determined by each terminal; The complete set of execution actions is sent to each access point in the wireless communication network, so that each access point can perform data transmission according to the complete set of execution actions; Collect actual network performance feedback after the data transmission is executed, and generate a reward signal based on the actual network performance feedback; The parameters of the reinforcement learning policy are updated using the reward signal.
8. A reinforcement learning scheduling device for multi-access point wireless networks, characterized in that, The device includes: The information acquisition module is configured to acquire the current status information of multiple terminals in the wireless communication network; A complete joint action construction module is configured to construct a complete joint action corresponding to each candidate action of the current terminal, wherein an action of the current terminal represents a radio resource contention group selected by the current terminal. The calculation module is configured to calculate the resource acquisition opportunities of the current terminal under the complete joint action based on the current state information. The filtering module is configured to filter a subset of candidate actions from all candidate actions of the current terminal based on the resource acquisition opportunity and the network state parameters extracted from the current state information. The action acquisition module is configured to input the current state information into the reinforcement learning policy and constrain the decision space of the reinforcement learning policy according to the subset of candidate actions, so that the reinforcement learning policy outputs the current terminal's action within the range defined by the subset of candidate actions.
9. An electronic device, characterized in that, include: Memory, processor; The processor is configured to read and execute the computer program stored in the memory to implement the steps of the reinforcement learning scheduling method for a multi-access point wireless network as described in any one of claims 1-7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed, implement the steps of the reinforcement learning scheduling method for a multi-access point wireless network as described in any one of claims 1-7.
