Joint distributed learning for signaling and radio resource allocation policies

Abstract

The invention relates to a wireless resource allocation method and a distributed learning method thereof. A network device includes a policy processor and a training processor. The policy processor determines an initial local state based on a current state of an associated terminal device, generates a first message according to the initial local state, exchanges the first message with an adjacent network device, determines a radio resource allocation configuration based on the initial local state and the first message, and allocates resources accordingly. The training processor generates a second message indicating a local reward from allocating the resource, exchanges the second message with a neighboring network device, calculates a global reward based on the second message, generates a third message indicating a gradient based on a radio resource allocation configuration, a local state change, and the global reward from an earlier time slot, and transmits the generated third message to the neighboring network device. A third message is exchanged with the neighboring network device, and the policy processor is updated based on the third message.

Description

Technical Field

[0001] This application relates to the allocation of radio resources in networks such as wireless networks. This application proposes a joint distributed learning method for signaling and radio resource allocation strategies. This application describes a network device and method for radio resource allocation and distributed learning. Background Art

[0002] It is expected that there will be a large number of terminal devices connected to network devices in the next-generation wireless network, for example, access points (APs). In this case, radio resource allocation, which is a key aspect of wireless networks, will become a very challenging task. In practical applications, each AP needs to collect the status information of the terminal devices connected to it, and then solve an optimization problem to decide which terminal devices to schedule and / or how to allocate radio resources to the terminal devices, such as transmission power, subcarrier frequency, beam selection, etc. Interference between terminal devices belonging to nearby (adjacent) APs may have a significant impact on the overall performance, and thus should be considered when making these decisions.

[0003] According to the methods similarly used in existing wireless networks, each AP can allocate radio resources relying only on the local information from the terminal devices connected to it. Since no communication is required between the APs of the network, this is a scalable method. However, this method does not consider the status of adjacent APs and may result in poor performance. Alternatively, the problem can be handled centrally, for example, by a single master entity that needs to collect a large amount of information from all APs of the network, then solve a more challenging large-scale optimization problem, and then send the corresponding radio resource allocation decisions to each AP. Due to the huge communication overhead and the possible significant increase in latency, this method is not practical. Summary of the Invention

[0004] This application predicts that communication will rely on distributed resource management, where the AP only exchanges necessary information with a small amount of transmitted data. Although this method is extremely attractive, it is also extremely challenging. Most of the proposed ideas are either impractical or have stability and convergence problems, resulting in poor system performance.

[0005] With the recent success in artificial intelligence (AI), especially the successful application of deep neural networks (DNNs) in very complex optimization problems, distributed wireless resource allocation has become more attractive. The need for a decision-making framework for distributed resource management that can adapt to dynamic environments naturally leads to a solution that combines the representational power of DNNs with reinforcement learning (RL). RL is a field of machine learning in which an agent (e.g., an AP) learns the optimal behavior based on its trial-and-error interactions with the environment in order to ultimately maximize a long-term goal. The original RL methods have dimensionality problems that limit their application in models with a small number of states. This limitation can be addressed with DNNs, which can be used to approximate the functions of RL algorithms.

[0006] In addition, there is another challenge to be addressed in RL-based distributed methods for wireless resource allocation. In addition to the dynamic nature of the environment, each AP must also consider the behavior of other agents. For example, one AP must learn to predict the decisions of other APs and make corresponding strategic decisions. In this case, convergence can be a problem, and thus it is usually assumed that centralized training is required, but this is not practical for resource allocation problems. Another approach is to perform independent distributed training on each AP, but this usually has stability problems and leads to unfair solutions, which is a poor performance metric for wireless networks.

[0007] In summary, the purpose of this application is to provide an efficient decentralized wireless resource allocation scheme in a wireless network. Another purpose is to provide a configuration of network devices that enables the network devices to participate in decentralized wireless resource allocation.

[0008] These and other purposes are achieved by the technical solutions described in the independent claims. Advantageous implementations are further defined in the dependent claims.

[0009] A first aspect of the present application provides a network device for wireless resource allocation. The network device includes a policy processor and a training processor. Wherein, the policy processor is configured to, in a current time slot: determine an initial local state of the network device, where the initial local state indicates a current state of one or more terminal devices connected to the network device; generate a first message according to a first message generation model and based on the initial local state; cause the network device to send the first message to one or more neighboring network devices and obtain one or more first messages from the one or more neighboring network devices; determine a wireless resource allocation configuration based on the following: an allocation policy model; the initial local state; at least one of the generated first message and the one or more first messages of the one or more neighboring network devices; allocate wireless resources to the one or more terminal devices connected to the network device according to the wireless resource allocation configuration; the training processor is configured to, in the current time slot: generate a second message, where the second message indicates a local reward obtained by allocating the wireless resources according to the wireless resource allocation configuration; cause the network device to send the second message to the one or more neighboring network devices and obtain one or more second messages from the one or more neighboring network devices, where each second message indicates a corresponding local reward of one of the network devices; calculate a global reward based on all local rewards; generate a third message based on the following: historical records of one or more wireless resource allocation configurations, initial local states, changed local states, and global rewards of one or more consecutive time slots before the current time slot, where the third message indicates gradients for respectively updating the allocation policy model and the first message generation model, and the changed local state is obtained by allocating the wireless resources to the one or more terminal devices connected to the network device; cause the network device to send the third message to the one or more neighboring network devices and obtain one or more third messages from the one or more neighboring network devices; update the first message generation model and the allocation policy model based on the third message.

[0010] The network device of the first aspect can participate in decentralized wireless resource allocation in a wireless network together with other network devices. Therefore, the network devices of the wireless network can implement decentralized wireless resource allocation in the wireless network. By exchanging the first message, the second message, and the third message, and by calculating and considering the global reward and the local reward, an efficient joint distributed learning method for signaling (using and training the first message generation model) and a learning method for wireless resource allocation policies (using and training the allocation policy model) are achieved.

[0011] In one implementation of the first aspect, the policy processor is configured to: determine an initial local state, generate a first message, cause the network device to send the first message, calculate a radio resource allocation configuration, and repeat the radio resource allocation in each of a plurality of consecutive time slots; the training processor is configured to: generate a second message, cause the network device to send the second message, and repeat the calculation of the global reward in each of the plurality of consecutive time slots.

[0012] This can also be done at other network devices in the wireless network, thus achieving distributed learning.

[0013] In one implementation of the first aspect, the training processor is configured to: generate a third message, cause the network device to send the third message, and repeat the update of the first message generation model and the allocation policy model based on the third message every several time slots in the plurality of consecutive time slots.

[0014] This allows sufficient information about the rewards obtained in the network to be considered, which is necessary to determine the gradients in the third message exchanged with one or more neighboring network devices, thereby improving training and radio resource allocation.

[0015] In one implementation of the first aspect, the training processor is further configured to: calculate the local reward based on the changed local state, where the changed local state is caused by the allocation of the radio resources to one or more terminal devices connected to the network device according to the radio resource allocation configuration.

[0016] In one implementation of the first aspect, the state of the one or more terminal devices includes the queue length of each terminal device connected to the network device.

[0017] In one implementation of the first aspect, the training processor is configured to calculate the global reward as the sum of all local rewards.

[0018] In one implementation of the first aspect, the training processor is configured to: use a gradient algorithm to update the first message generation model and the allocation policy model based on the third message, where the third message respectively indicates the gradients for updating the allocation policy model and the first message generation model.

[0019] In one implementation of the first aspect, the policy processor is configured to: determine the initial local state based on the observation of one or more performance parameters of the one or more terminal devices connected to the network device.

[0020] In an implementation of the first aspect, the policy processor is configured to: input at least one of the initial local state, the generated first message, and the one or more first messages of the one or more adjacent network devices into the allocation policy model to receive the radio resource allocation configuration as an output of the allocation policy model.

[0021] In an implementation of the first aspect, the policy processor is configured to: aggregate the generated first message and the one or more first messages of the one or more adjacent network devices, and calculate the radio resource allocation configuration based on the aggregation result that is an input to the allocation policy model.

[0022] In an implementation of the first aspect, the radio resources allocated to the one or more terminal devices include at least one of the following: transmission power, scheduling grant, and precoding options and beamforming options.

[0023] In an implementation of the first aspect, the policy processor is configured to: directly calculate the radio resource allocation configuration; or calculate a probability distribution for different radio resource allocation configurations and randomly select the radio resource allocation configuration based on the probability distribution.

[0024] In an implementation of the first aspect, each first message includes a vector of real numbers, where the dimension of the vector depends on the bandwidth allocated for exchanging the first message between the network device and the one or more adjacent network devices.

[0025] In an implementation of the first aspect, the policy processor is configured to: if the time slot is an initial time slot, generate the first message according to an initial first message generation model.

[0026] In an implementation of the first aspect, both the policy processor and / or the training processor are configured to run at least one neural network to perform the corresponding operations of the policy processor and / or the training processor.

[0027] In an implementation of the first aspect, at least one of the first message generation model, the allocation policy model, and the aggregator for aggregating the generated first message and the one or more first messages of the one or more adjacent network devices is implemented by a neural network.

[0028] The second aspect of the present application provides a method for wireless resource allocation by a network device. The method includes, in a current time slot: determining an initial local state of the network device, where the initial local state indicates the current states of one or more terminal devices connected to the network device; generating a first message according to a first message generation model and based on the initial local state; causing the network device to send the first message to one or more neighboring network devices and obtaining one or more first messages from the one or more neighboring network devices; determining a wireless resource allocation configuration based on the following: an allocation policy model; the initial local state; at least one of the generated first message and the one or more first messages of the one or more neighboring network devices; allocating wireless resources to the one or more terminal devices according to the wireless resource allocation configuration; generating a second message, where the second message indicates a local reward obtained by allocating the wireless resources according to the wireless resource allocation configuration; causing the network device to send the second message to the one or more neighboring network devices and obtaining one or more second messages from the one or more neighboring network devices, where each second message indicates the corresponding local reward of one of the network devices; calculating a global reward based on all local rewards; generating a third message based on the following: one or more wireless resource allocation configurations of one or more consecutive time slots before the current time slot, the initial local state, a changed local state, and a history of global rewards, where the third message indicates gradients for respectively updating the allocation policy model and the first message generation model, and the changed local state is obtained by allocating the wireless resources to the one or more terminal devices connected to the network device; causing the network device to send the third message to the one or more neighboring network devices and obtaining one or more third messages from the one or more neighboring network devices; updating the first message generation model and the allocation policy model based on the third message.

[0029] The method of the second aspect can be executed by the network device of the first aspect. The implementation manner of the method of the second aspect can correspond to the implementation manner of the network device of the first aspect. The method of the second aspect and its implementation manner achieve the same advantages as those described above for the network device of the first aspect.

[0030] The third aspect of the present application provides a computer program including instructions, which, when executed by at least one processor, cause the at least one processor to execute the method described in the second aspect or any of its implementation manners.

[0031] The fourth aspect of the present application provides a non-transitory storage medium storing executable program code, which, when executed by a processor, executes the method according to any one of the second aspect or its implementation manners.

[0032] As a summary of the above aspects and implementations, the present application proposes a system for efficient decentralized wireless resource allocation in a wireless network. The system may include a policy processor and a training processor at each participating network device (e.g., each AP or equivalent base station (BS)), as described for the network device in the first aspect.

[0033] In each time slot, the policy processor of each AP: can observe its local state (i.e., the state of the terminal devices associated with the AP, e.g., the channel state, latency experience, and traffic information of these terminal devices); can generate a first message and send it to adjacent APs; can receive similar first messages from other APs; can output a wireless resource allocation configuration and decision based on its current allocation policy model, according to its local state and the first messages received from its adjacent APs; can observe a new local state as a result of the wireless resource allocation decision (action, allocate wireless resources).

[0034] The training module of each AP: can exchange second messages with other APs so that a global reward (or equivalent cost) can be calculated; can store tuples including actions, local state transitions, and global rewards in a replay buffer; after the AP has used its current policy model for a specified period of time, it can run a policy gradient algorithm to update the policy module, where, by exchanging a third message indicating the relevant gradient, it can update the way of generating the first message and update its own wireless resource allocation policy model.

[0035] It should be noted that all devices, elements, units, and modules described in the present application can be implemented in software or hardware elements or any combination thereof. All steps performed by various entities described in the present application and the functions described to be performed by various entities are intended to indicate that the corresponding entities are adapted or used to perform the corresponding steps and functions. Although in the description of the following specific embodiments, the specific functions or steps to be performed by external entities are not reflected in the description of the specific detailed elements of the entity performing the specific steps or functions, those skilled in the art should be clear that these methods and functions can be implemented in the corresponding software or hardware elements or any combination thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In combination with the accompanying drawings, the description of the following specific embodiments will elaborate on the above aspects and their implementations, in which:

[0037] Figure 1 A network device for wireless resource allocation according to the present application is shown.

[0038] Figure 2Shows blocks of a network device for decentralized resource allocation according to the present application.

[0039] Figure 3 Shows a flowchart of a network device for decentralized resource allocation according to the present application.

[0040] Figure 4 Shows an exemplary implementation of a policy processor.

[0041] Figure 5 Shows an exemplary implementation of a policy processor without an aggregator.

[0042] Figure 6 Shows an exemplary implementation of a policy processor with different types of encoders.

[0043] Figure 7 Shows an exemplary policy module update when using the policy gradient algorithm.

[0044] Figure 8 Shows an exemplary policy module update when updating the policy module by minimizing the training loss function.

[0045] Figure 9 Shows the global reward for training of the solution according to the present application and a comparison with a baseline.

[0046] Figure 10 A method for wireless resource allocation according to the present application. Detailed implementation

[0047] Figure 1 Shows network device 100 according to the present application. Network device 100 is used for wireless resource allocation and can participate in a joint distributed learning method for wireless resource allocation together with other network devices in a wireless network. Figure 1 Shows adjacent network device 120 (e.g., located near network device 100), which can also participate in a joint distributed learning method for wireless resource allocation. There can be multiple such adjacent network devices 120 participating. Figure 1 Also shows terminal device 110 connected to network device 100. Other terminal devices (not shown) can be respectively connected to one or more adjacent network devices 120. Figure 1 The shown devices 100, 110, 120 can belong to or be in the same wireless network.

[0048] The network device 100 includes a policy processor 101 and a training processor 102. In addition, each adjacent network device 120 may include such a policy processor 101 and a training processor 102, where these processors 101, 102 may work in a similar manner in all network devices 100, 120. In this application, the policy processor 101 may also be referred to as a policy module, and the training processor 102 may also be referred to as a training module. Both the policy processor 101 and the training processor 102 may be implemented by the processing circuit of the network device 100. Certain actions of the policy processor 101 and the training processor 102 may be repeatedly executed in each of multiple consecutive time slots.

[0049] In the current time slot, the policy processor 101 is used to determine the initial local state of the network device 100. The initial local state indicates the current state of one or more terminal devices 110 connected to the network device 100. For example, the state of one or more terminal devices 110 may include the queue length of each terminal device 110 connected to the network device 100, or one or more other performance parameters of the terminal device 110. The state of the terminal device 110 may include the channel state, delay experience, and traffic information at these terminal devices 110. For example, the initial local state may be determined based on one or more performance parameters of one or more terminal devices 110 connected to the network device 100.

[0050] The policy processor 101 is also used to generate a model according to the first message and generate the first message 103 based on the initial local state. The policy processor 101 is further used to: cause the network device 100 to send the first message 103 to one or more adjacent network devices 120, and obtain one or more first messages 103 from one or more adjacent network devices 120.

[0051] The policy processor 101 is also used to determine the radio resource allocation configuration based on the following items: the allocation policy model; the initial local state; at least one of the generated first message 103 and one or more first messages 103 of one or more adjacent network devices 120. The policy processor 101 may also aggregate the generated first message 103 and one or more first messages 103 received from one or more adjacent network devices 120, and then determine the radio resource allocation configuration based on the aggregation result that can be used as the input of the allocation policy model.

[0052] The policy processor 101 is also used to allocate radio resources 104 to one or more terminal devices 110 connected to the network device 100. The radio resources 104 allocated to one or more terminal devices 110 may include at least one of the following: transmission power, scheduling authorization, and precoding options and beamforming options.

[0053] In the current time slot, the training processor 102 is used to generate a second message 105 indicating a local reward. The training processor 102 is further used to: cause the network device 100 to send the second message 105 to one or more neighboring network devices 120, and obtain one or more second messages 105 from one or more neighboring network devices 120. The local reward indicated by the second message generated by the network device 100 is obtained based on the allocation of wireless resources 104 according to the wireless resource allocation configuration. Each second message 105 obtained from the neighboring network devices 120 indicates the corresponding local rewards of these neighboring network devices 120. The training processor 102 is further used to calculate the global reward based on all the local rewards.

[0054] The training processor 102 is further used to generate a third message 107, which indicates gradients for updating the allocation policy model and the first message generation model respectively. The third message 107 is generated by the training processor 102 based on one or more wireless resource allocation configurations, the initial local state, the changed local state, and the history of the global rewards in one or more consecutive time slots before the current time slot, where the changed local state is obtained due to the allocation of wireless resources 104 to one or more terminal devices 110 connected to the network device 100. The training processor 102 is further used to: cause the network device 100 to send the third message 107 to one or more neighboring network devices 120, and obtain one or more third messages 107 from one or more neighboring network devices 120. Each third message 107 obtained from the corresponding neighboring network device 120 also indicates the gradients for updating the allocation policy model and the first message generation model respectively, as determined by the corresponding neighboring network device 120.

[0055] Then, the training processor 102 is used to: update 106 the first message generation model and the allocation policy model (usually updating the policy processor 101) based on the third message 107 (the generated third message and the obtained third message), especially based on the gradients indicated in the third message 107. Sending one or more third messages 107 to one or more neighboring network devices 120 and receiving one or more third messages 107 from one or more neighboring network devices 120, as well as updating 106 the first message generation model and the allocation policy model, can be repeated every few time slots in multiple consecutive time slots, while other actions of the processors 101 and 102 (i.e., determining the initial local state, sending and receiving one or more first messages 103, allocating wireless resources 104 according to the calculated wireless resource allocation, sending and receiving one or more second messages 105, and calculating the global reward) can be repeatedly executed in each time slot of multiple consecutive time slots.

[0056] The network device 100 may include a processing circuit (not shown) for performing, conducting, or initiating the various operations of the network device 100 described in this application, in particular for implementing the policy processor 101 and the training processor 102 respectively. The processing circuit may include hardware, and / or the processing circuit may be controlled by software. The hardware may include analog circuits or digital circuits, or both analog and digital circuits. The digital circuit may include components such as an application-specific integrated circuit (ASIC), a field-programmable array (FPGA), a digital signal processor (DSP), or a general-purpose processor. The network device 100 may also include a memory circuit for storing one or more instructions that can be executed by the processor or the processing circuit (specifically, executed under the control of software). For example, the memory circuit may include a non-transitory storage medium storing executable software code, and when the executable software code is executed by the processor or the processing circuit, it causes the various operations of the network device 100 to be performed. In one embodiment, the processing circuit includes one or more processors and a non-transitory memory connected to the one or more processors. The non-transitory memory may carry executable program code, and when the executable program code is executed by the one or more processors, it causes the network device 100, in particular the policy processor 101 and the training processor 102, to perform, conduct, or initiate the operations or methods described herein.

[0057] Figure 2 FIG. 1 shows a network device 100 according to the present application. Specifically, an AP 100 is shown as an example of the network device 100. The AP 100 is based on Figure 1 the network device 100 shown in FIG. 1. Specifically, Figure 2 FIG. 2 shows exemplary blocks of the AP 100. An adjacent network device 120, in particular an adjacent AP 120, is also shown in the figure. The blocks of the AP 100 will be described below. Figure 3 FIG. 3 shows an exemplary flowchart of the proposed functions of these blocks.

[0058] The policy module (policy processor 101) receives local observations of the AP 100 (e.g., channel state and queue length) from the terminal device 110 it is connected to as input at the start of each time slot. Based on this local information, the functions of the policy module are mainly twofold: first, calculate and broadcast a first message 103 (also referred to as an intermediate message in Figure 2 FIG. 3) to the adjacent AP 120, and second, take an action for this time slot based on the first message 103 received from the adjacent AP 120 and the local state.

[0059] The action can be the allocation of any radio resource 104, e.g., transmission power, device scheduling, precoding, and / or beamforming (if multiple antennas are used), allocation of resource blocks to the terminal device 110, etc. The resource allocation action can be implemented by the policy processor 101 in the following ways: if the different radio resource allocation configurations are discrete, output a probability distribution for the different radio resource allocation configurations (e.g., the probability distribution of which device to schedule or which power level to use), and then sample the configuration to be used from this distribution; or directly specify the action (e.g., which beamforming to use). For example, the policy processor 101 can directly calculate the radio resource allocation configuration; or can calculate the probability distribution for the different radio resource allocation configurations and randomly select the radio resource allocation configuration based on this probability distribution.

[0060] In all cases, the probability of the specific radio resource allocation action to be selected at time slot t must be recorded for further use in training. Then, this radio resource allocation 104 is implemented by the corresponding physical layer module and MAC layer module of the AP 100. The first message 103 can be a vector of real numbers, where the dimension of the vector depends on the bandwidth allocated for the signaling between the APs 100, 120 on the control interface.

[0061] Figure 3 Shows Figure 2Exemplary flowchart of network device 100. At 301, network device 100 observes the channel state and traffic state of associated terminal device 110. At 302, network device 100 sends first message 103 to neighboring AP 120. At 303, network device 100 obtains first message 103 from neighboring AP 120 and uses first message 103 and local observation 201 to determine a radio resource allocation configuration, e.g., to schedule terminal device 110. At 304, network device 100 may then exchange the sum of the queue lengths and / or delays of terminal device 110 (local cost or local reward) with other APs 120. Then, at 305, network device 100 calculates a global cost or global reward based on the local reward, e.g., as the sum of the local rewards. At 307, network device determines whether it is time to update the policy model. If it is not time to update the policy model, then at 306, network device 100 stores observation 201 in a buffer and proceeds to 301. If it is time to update the policy model, then at 310, network device 100 exchanges third message 107 indicating the gradient (and potentially first message 103 from the forward propagation on the trajectory for training) with other APs 120. Then, at 309, network device 100 updates the allocation policy model. To do so, network device 100 may run iterations of a policy gradient-based algorithm that uses the gradient indicated by third message 107 based on the calculated global reward. At 308, network device 100 clears the buffer and proceeds to 301.

[0062] In practical applications, policy processor 101 can be parameterized as a modular function, where each module is a neural network or the like. As Figure 4 shown, two basic modules for each APi can be:

[0063] a. State encoder 403, which can be a function that takes local observation 201 as input and outputs a first message that can be a vector of a specified dimension (e.g., according to the capacity of the control interface as described above).

[0064] b. Policy function 402, which can output a policy (radio resource allocation configuration) to follow after receiving inputs from state encoder 403, local observation 201, and first message 103 from neighboring AP 120.

[0065] To process first message 103 from neighboring AP 120, AP 100 can use aggregator 401, which can be a block that takes input from its own state encoder 403 but also from the state encoder 403 of neighboring AP 120.

[0066] Figure 5Shows an alternative implementation of the policy module (policy processor 101), which may include directly using the first message 103 from the adjacent AP 120 without using the aggregator 401.

[0067] Figure 6 Shows another alternative implementation of the policy module (policy processor 101), which may include propagating the local observations 201 of each AP 100 from different state encoders 403 (self-state encoders) before being fed into the policy module. This architecture can be combined with using the aggregator 401 (as Figure 4 shown) for the output of the state encoder 403.

[0068] The parameters θ of the policy module of APi i (e.g., the weights and biases of the neural network that constitutes Figures 4 to 6 the architecture block shown) can be periodically updated by means of a training module (training processor 102), which will be described in detail below.

[0069] The functions of the training processor 102 of the AP 100 (e.g., generally any APi) are, first, to exchange a specific second message 105 of local rewards (or costs) with other APs 120 so that the global reward (or cost) can be calculated. Second, at each time slot, save the transitions for updating the policy module 101 (parameters) in a suitable form for future processing. Third, periodically update the parameters of the policy module 101 of the AP 100.

[0070] The first two parts of the functions of the training processor 102 can be implemented by using a distributed cooperative multi-agent learning method with minimal traffic. Specifically, at the end of each time slot, the AP 100 can:

[0071] a. Calculate the local reward, which may include the negative value of the sum of the queue lengths of the terminal devices 110 associated with this AP 100 denoted as K(i), and only exchange this local reward as the second message 105 with the corresponding one or more adjacent APs 120 to calculate the global reward, which is the sum of all local rewards, i.e., Then, store the global reward as the reward for this state transition.

[0072] b. Store the state transition and action of the AP 100, as well as the global reward (or equivalent cost) accumulated by the radio resource allocation policy, as calculated in the previous step. More specifically, the AP 100 stores the tuple in the replay buffer, which includes the state at the beginning of this time slot t the resource allocation decision taken (i.e., the action of allocating radio resources 104 according to the radio resource allocation configuration), the global reward R calculated based on the exchange of the second message 105 described in the previous point and the next state This is the result of the resource allocation decision of the adjacent AP 120 t and the interference of the resource allocation decision. The state here includes the queue length and may also include the channel state of its K connected devices i.e., when k ∈ K(i),

[0073] To update the parameters of the policy processor 101, this application stipulates that the third message 107 between APs 100 and 120 is for further exchange of information. Specifically, for the last step of training update, the training processor 101 at AP 100 (e.g., usually any APi) works as follows (hereinafter, N(i) represents the adjacent AP 120 of APi):

[0074] a. After a specified period of time when APs 100 and 120 have used their current policies (e.g., M rounds of length T time slots), the training processors of each AP 100 and 120 calculate the future total reward from each time slot t until the end of the trajectory m for M trajectories based on the previously calculated global reward R t The term "trajectory" represents a series of observations, actions, and rewards obtained when taking actions according to a given policy and can be sampled from the replay buffer.

[0075] b. To reduce the variance of the estimate, the training processor 102 can optionally train the baseline function of the local state This baseline function can be used to calculate the advantage function This advantage function quantifies the degree to which a particular action is a good decision or a bad decision under a given state

[0076] c. Then, the training processor 102 updates the policy parameters, e.g., by using the calculated advantage function and appropriate signaling to perform a Monte-Carlo estimate of the future total reward. It should be noted here that due to the first message 103, the policy of AP 100 depends on the local state of the adjacent APs 120 in the network (always depends on the local state of its adjacent APs 120; if n-level aggregation is used, it may also depend on the parameters of the n-hop neighbors), as well as the parameters of the neighbor's state encoder and possibly the aggregator.

[0077] The policy gradient algorithm can be used to estimate the gradient in the direction of the assignment policy model to be updated and the parameters of the block responsible for the first message generation, i.e., the first message generation model. In this case, the APi can use the following training loss:

[0078]

[0079] And the parameters of the policy processor 101 can be updated by performing gradient descent on the above loss and the corresponding loss of its neighboring AP 120. This can be done by exchanging the third message 107 (indicating the gradient) between the APs 100, 120 so that the gradient can be backpropagated subsequently, as Figure 7 shown. The process of the alternative policy module (as Figure 5 and Figure 6 shown) is completely analogous. Thus, Figure 7 shows the policy module update when using the policy gradient algorithm, i.e., a single update in the direction of the gradient of equation (1). The arrows indicate the direction of backpropagation and exchange of the gradient between the APs 100, 120 through the indication in the third message 107.

[0080] Alternatively, as Figure 8 shown, more advanced policy gradient algorithms such as Trust Region Policy Optimization (TRPO) or Proximal Policy Optimization (PPO) can be used. These methods include the existing

[0081] where, ∈ is a parameter, is the assignment policy model used in the first M rounds. In this case, the training is done in multiple rounds, each round including backpropagation and forward propagation: in backpropagation, the parameters of the policy module 101 of the APs 100, 120 are updated in the direction of the gradient of equation (2); for this, backpropagation between the modules is required, as well as signaling of the third message 107 for exchanging the relevant gradient between the neighboring APs 100, 120. In forward propagation, the local observation 201 of each transition in the trajectory used for training is fed into the updated policy module 101, and the corresponding probability of the resource allocation decision is calculated; for this, the first message 103 is exchanged between the APs for each transition. Thus, Figure 8 shows the policy module update, where the policy module is updated by minimizing the training loss function (e.g., PPO, TRPO). The arrows indicate the direction of backpropagation and forward propagation.

[0082] As before, the process of training the alternative policy module in this way (Figure 5 and Figure 6 in) are exactly similar. This method results in more signaling to perform each policy update, but this method makes each policy update more efficient and may make the algorithm learn faster.

[0083] The above process means that if the global reward at each time slot t is known for all APs 100, 120, the factored policy applied in the system will be updated in the direction of the stochastic gradient of the global average reward in a fully distributed manner. Therefore, theoretically, it can be guaranteed that the wireless resource allocation policy and the communication between APs (through the state encoder module) are updated in a performance-improving manner, while the communication overhead required to implement this process is much lower than that of the centralized training method.

[0084] A representative implementation of the above-described system of network devices 100, 120 is in a cellular network environment, where the APs 100, 120 (or BSs) must perform uplink scheduling on the terminal devices 110 they are connected to. Therefore, each AP 100, 120 must decide which of its associated devices 110 to schedule at each time slot. For the traffic model, it can be assumed that the traffic (in bits) arrives at device k at each time slot t according to a stochastic process. Specifically, for APi and its K connected devices, there is:

[0085] · State: when k ∈ K(i), where:

[0086] ■ The number of data bits of device k at time slot t that have not been transmitted (queue length).

[0087] ■ The channel state between device k and APi at time slot t.

[0088] · Action: The action is the selected device to be scheduled at each time slot t. Selecting k = 0 represents the decision not to schedule any device. Given the actions and channel states of all APs at time slot t, the number of transmission bits sent by device i at time slot t is calculated using the Shannon formula:

[0089]

[0090] where W is the bandwidth used for transmission, T s is the duration of each time slot, and P is the uplink transmission power normalized by the receiver noise power. The number of bits remaining in the queue of device k at the start of the next time slot is given by as follows.

[0091] · Reward: The local reward for each AP 100, 120 is defined as the negative value of the sum of the queue lengths of its devices, i.e.,

[0092] The policy processor 101 used in this implementation manner can be a deep neural network (DNN) with K + 1 outputs of parameter θ i . The training processor 102 can use the Generalized Advantage Estimation (GAE) method for advantage estimation, and can use policy gradients to update the parameters of the policy module 101 (see Figure 7 ).

[0093] For the performance evaluation of the solution of this application (especially the above implementation manner), a system of N = 4 APs 100, 120 in a square topology was simulated. The traffic follows a Poisson distribution, and all devices have the same mean. K = 20 terminal devices 110 are randomly placed within the coverage area, and each device is associated with the nearest AP 100, 120. The allocation policy model is updated once every round (4 episodes), and each episode includes 1000 time slots.

[0094] To evaluate the performance of the proposed solution, it was compared with the standard scheduling algorithms of Proportional Fairness (PF) and Max-Weight. Figure 9 For the results averaged over 5 random implementations, the global reward is represented as a function of the training rounds. We can see that after a short training time, during the execution phase and the training phase, the information exchange between APs 100, 120 enables them to quickly learn a method to coordinate and outperform the two baselines.

[0095] Figure 10The method 1000 for a network device 100 to perform wireless resource allocation is shown. The method 1000 includes, in the current time slot: Step 1001: Determine the initial local state of the network device 100, where the initial local state indicates the current states of one or more terminal devices 110 connected to the network device 100; Step 1002: Generate a first message 103 according to the first message generation model and based on the initial local state; Step 1003: Cause the network device 100 to send the first message 103 to one or more neighboring network devices 120 and obtain one or more first messages 103 from the one or more neighboring network devices 120; Step 1004: Determine the wireless resource allocation configuration based on the following: the allocation policy model; the initial local state; at least one of the generated first message 103 and the one or more first messages 103 of the one or more neighboring network devices 120; Step 1005: Allocate wireless resources 104 to one or more terminal devices 110 according to the wireless resource allocation configuration. Steps 1001 to 1005 may be executed by the policy processor 101 of the network device 100.

[0096] The method 1000 further includes, in the current time slot: Step 1006: Generate a second message 105, where the second message 105 indicates the local reward obtained by allocating the wireless resources 104 according to the wireless resource allocation configuration; Step 1007: Cause the network device 100 to send the second message 105 to one or more neighboring network devices 120 and obtain one or more second messages 105 from the one or more neighboring network devices 120, where each second message 105 indicates the corresponding local reward of one of the network devices 100, 120; Step 1008: Calculate the global reward based on all the local rewards; Step 1009: Generate a third message 107 based on the following: the one or more wireless resource allocation configurations of one or more consecutive time slots before the current time slot, the initial local state, the changed local state, and the historical records of the global rewards, where the third message 107 indicates the gradients for respectively updating the allocation policy model and the first message generation model, and the changed local state is obtained by allocating the wireless resources 104 to one or more terminal devices 110 connected to the network device 100; Step 1010: Cause the network device 100 to send the third message 107 to one or more neighboring network devices 120 and obtain one or more third messages 107 from the one or more neighboring network devices 120; Step 1011: Update the first message generation model and the allocation policy model based on the third message 107.

[0097] The advantages of the solution of this application can be summarized as follows:

[0098] · This solution enables APs 100, 120 to learn the wireless resource allocation strategy and the signaling between APs 100, 120, and this signaling can be adjusted according to the communication capabilities of the control interface: Train to adjust to the possible dimensions of the first message 103.

[0099] · The learned first message generation model may be different for each AP 100, 120 to adapt to the traffic characteristics and channel characteristics of its associated devices.

[0100] · Update the wireless resource allocation strategy model and the first message generation model to ensure that the cumulative total reward is improved statistically.

[0101] · This solution is distributed and can therefore be executed simultaneously at each network device 100, 120 (e.g., AP or BS).

[0102] The present application has been described in connection with various embodiments as examples and implementations. However, upon study of the drawings, the present application, and the independent claims, those skilled in the art will be able to understand and implement other variations when implementing the claimed subject matter. In the claims as well as in the specification, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" does not exclude a plurality. A single element or other unit may fulfill the functions of several entities or items described in the claims. Listing certain measures in mutually different dependent claims does not mean that a combination of these measures cannot be used efficiently in an advantageous implementation.

Claims

1. A network device (100) for wireless resource allocation, characterized in that, The network device (100) includes a policy processor (101) and a training processor (102), where, The policy processor (101) is configured to, in a current time slot: Determine an initial local state of the network device (100), where the initial local state indicates a current state of one or more terminal devices (110) connected to the network device (100); Generate a first message (103) according to a first message generation model and based on the initial local state; Cause the network device (100) to send the first message (103) to one or more neighboring network devices (120), and obtain one or more first messages (103) from the one or more neighboring network devices (120); Determine a radio resource allocation configuration based on the following: an allocation policy model; the initial local state; at least one of the generated first message (103) and the one or more first messages (103) of the one or more neighboring network devices (120); Allocate radio resources (104) to the one or more terminal devices (110) connected to the network device (100) according to the radio resource allocation configuration; The training processor (102) is configured to, in the current time slot: Generate a second message (105), where the second message (105) indicates a local reward obtained by allocating the radio resources (104) according to the radio resource allocation configuration; Cause the network device (100) to send the second message (105) to the one or more neighboring network devices (120), and obtain one or more second messages (105) from the one or more neighboring network devices (120), where each second message (105) indicates a corresponding local reward of one of the network devices (100, 120); Calculate a global reward based on all local rewards; Generate a third message (107) based on the following: historical records of one or more radio resource allocation configurations, initial local states, changed local states, and global rewards of one or more consecutive time slots before the current time slot, where the third message (107) indicates gradients for respectively updating the allocation policy model and the first message generation model, and the changed local state is obtained by allocating the radio resources (104) to the one or more terminal devices (110) connected to the network device (100); Cause the network device (100) to send the third message (107) to the one or more neighboring network devices (120), and obtain one or more third messages (107) from the one or more neighboring network devices (120); Update (106) the first message generation model and the allocation policy model based on the third message (107).

2. The network device (100) according to claim 1, wherein, The policy processor (101) is configured to: determine an initial local state, generate a first message (103), cause the network device (100) to send the first message (103), calculate a radio resource allocation configuration, and repeat the radio resource allocation (104) in each of a plurality of consecutive time slots; The training processor (102) is configured to: generate a second message (105), cause the network device (100) to send the second message (105), and repeat calculating a global reward in each of the plurality of consecutive time slots.

3. The network device (100) according to claim 1 or 2, characterized in that, The training processor (102) is configured to: generate a third message (107), cause the network device (100) to send the third message (107), and repeat updating (106) the first message generation model and the allocation policy model based on the third message (107) in every several time slots among the plurality of consecutive time slots.

4. The network device (100) according to any one of claims 1 to 3, characterized in that, The training processor (102) is further configured to: calculate the local reward based on the changed local state, where the changed local state is caused by allocating the radio resources (104) to the one or more terminal devices (110) connected to the network device (100) according to the radio resource allocation configuration.

5. The network device (100) according to any one of claims 1 to 4, characterized in that, The state of the one or more terminal devices (110) includes the queue length of each terminal device (110) connected to the network device (100).

6. The network device (100) according to any one of claims 1 to 5, characterized in that, The training processor (102) is configured to calculate the global reward as the sum of all local rewards.

7. The network device (100) according to any one of claims 1 to 6, characterized in that, The training processor (102) is configured to: use a gradient algorithm to update the first message generation model and the allocation policy model based on the third message, where the third message respectively indicates gradients for updating the allocation policy model and the first message generation model.

8. The network device (100) according to any one of claims 1 to 7, characterized in that, The policy processor (102) is configured to: determine the initial local state based on observations (201) of one or more performance parameters of the one or more terminal devices (110) connected to the network device (100).

9. The network device (100) according to any one of claims 1 to 8, characterized in that, The policy processor (101) is configured to: input the following into the allocation policy model: the initial local state; at least one of the generated first message (103) and the one or more first messages (103) of the one or more neighboring network devices (120), to receive the radio resource allocation configuration as an output of the allocation policy model.

10. The network device (100) according to any one of claims 1 to 9, characterized in that, The policy processor (101) is configured to: aggregate the generated first message (103) and the one or more first messages (103) of the one or more neighboring network devices (120), and calculate the radio resource allocation configuration based on the aggregation result as an input to the allocation policy model.

11. The network device (100) according to any one of claims 1 to 10, characterized in that, The radio resources (104) allocated to the one or more terminal devices (110) include at least one of the following: transmission power, scheduling grant, and precoding options and beamforming options.

12. The network device (100) according to any one of claims 1 to 11, characterized in that, The policy processor (101) is configured to: directly calculate the radio resource allocation configuration; or Calculate the probability distribution for different wireless resource allocation configurations, and randomly select the wireless resource allocation configuration based on the probability distribution.

13. The network device (100) according to any one of claims 1 to 12, characterized in that, Each first message (103) includes a vector of real numbers, where the dimension of the vector depends on the bandwidth allocated for exchanging the first message (103) between the network device (100) and the one or more neighboring network devices (120).

14. The network device (100) according to any one of claims 1 to 13, characterized in that, The policy processor (101) is configured to: if the time slot is an initial time slot, generate the first message (103) according to an initial first message generation model.

15. The network device (100) according to any one of claims 1 to 14, characterized in that, Both the policy processor (101) and / or the training processor (102) are configured to run at least one neural network to perform the corresponding operations of the policy processor (101) and / or the training processor (102).

16. The network device (100) according to any one of claims 1 to 15, characterized in that, At least one of the first message generation model, the allocation policy model, and the aggregator (401) for aggregating the generated first message (103) and the one or more first messages (103) of the one or more neighboring network devices (120) is implemented by a neural network.

17. A method (1000) for wireless resource allocation by a network device (100), characterized in that, The method (1000) includes, in a current time slot: Determine (1001) an initial local state of the network device (100), where the initial local state indicates the current state of one or more terminal devices (110) connected to the network device (100); Generate (1002) a first message (103) according to a first message generation model and based on the initial local state; Cause (1003) the network device (100) to send the first message (103) to one or more neighboring network devices (120), and obtain one or more first messages (103) from the one or more neighboring network devices (120); Determine (1004) a wireless resource allocation configuration based on: an allocation policy model; the initial local state; at least one of the generated first message (103) and the one or more first messages (103) of the one or more neighboring network devices (120); Allocate (1005) wireless resources (104) to the one or more terminal devices (110) according to the wireless resource allocation configuration; Generate (1006) a second message (105), where the second message (105) indicates a local reward obtained by allocating the wireless resources (104) according to the wireless resource allocation configuration; Cause (1007) the network device (100) to send the second message (105) to the one or more neighboring network devices (120), and obtain one or more second messages (105) from the one or more neighboring network devices (120), where each second message (105) indicates the corresponding local reward of one of the network devices (100, 120); Calculate (1008) a global reward based on all local rewards; Generate (1009) a third message (107) based on: one or more radio resource allocation configurations, an initial local state, a changed local state, and a history of global rewards of one or more consecutive time slots before the current time slot, wherein the third message (107) indicates gradients for updating the allocation policy model and the first message generation model respectively, and the changed local state is obtained due to allocating the radio resources (104) to the one or more terminal devices (110) connected to the network device (100); Cause (1010) the network device (100) to send the third message (107) to the one or more neighboring network devices (120), and obtain one or more third messages (107) from the one or more neighboring network devices (120); Update (1011) the first message generation model and the allocation policy model based on the third message (107).

18. A computer program comprising instructions, characterized in that, When the program is executed by at least one processor, the instructions cause the at least one processor to perform the method (1000) according to claim 17.

Images