Reinforcement learning based load balancing among radio access technologies under multi-carrier dynamic spectrum sharing

Abstract

A method may include: receiving a load metric related to available resources of a carrier at a reinforcement learning load balancer. The method may also include: calculating an update to a carrier assignment parameter for a user based on the load metric. The method may also include: calculating an update to an uplink / downlink split ratio and a resource pool split ratio based on the load metric. In addition, the method may include: predicting the impact of updates to carrier assignment parameters, updates to uplink / downlink split ratios, and updates to resource pool split ratios on the load metric. Further, the method may include: transmitting updates to carrier assignment parameters, updates to uplink / downlink split ratios, and updates to resource pool split ratios to one or more radio resource managers of corresponding one or more radio access technologies for application to a communication network.

Description

Technical Field

[0001] Some example embodiments may generally relate to mobile or wireless telecommunication systems, such as long term evolution (LTE) or fifth generation (5G) radio access technology or new radio (NR) access technology or other communication systems. For example, certain embodiments may relate to apparatus, systems and / or methods for reinforcement learning-based inter-radio access technology load balancing in the context of multi-carrier dynamic spectrum sharing. Background Art

[0002] Examples of mobile or wireless telecommunication systems may include Universal Mobile Telecommunications System (UMTS), Terrestrial Radio Access Network (UTRAN), Long Term Evolution (LTE) Evolved UTRAN (E-UTRAN), LTE Advanced (LTE-A), MulteFire, LTE-A Pro and / or fifth generation (5G) radio access technology or new radio (NR) access technology. The fifth generation (5G) wireless system refers to the next generation (NG) radio system and network architecture. 5G is mainly built on the new radio (NR), but 5G (or NG) networks can also be built on the E-UTRAN radio. It is estimated that NR will provide a bit rate of about 10 to 20Gbit / s or higher, and will support at least enhanced mobile broadband (eMBB) and ultra-reliable low latency communication (URLLC) and massive machine type communication (mMTC). NR is expected to deliver extreme broadband and ultra-robust, low latency connectivity and large-scale networking to support the Internet of Things (IoT). As IoT and machine-to-machine (M2M) communications become more and more popular, the demand for networks that meet the needs of lower power, low data rates and long battery life will continue to grow. It should be noted that in 5G, a node that can provide radio access functionality to user equipment (i.e., similar to Node B in UTRAN or eNB in ​​LTE) is named gNB when built on NR radio, and is named NG-eNB when built on E-UTRAN radio. Summary of the invention

[0003] One embodiment may relate to a method. The method may include: receiving a load metric related to available resources of a carrier at a reinforcement learning load balancer. The method may also include: calculating an update to a carrier assignment parameter for a user based on the load metric. The method may also include: calculating an update to an uplink / downlink split ratio and a resource pool split ratio based on the load metric. In addition, the method may include: predicting the impact of updates to carrier assignment parameters, updates to uplink / downlink split ratios, and updates to resource pool split ratios on the load metric. Further, the method may include: transmitting updates to carrier assignment parameters, updates to uplink / downlink split ratios, and updates to resource pool split ratios to one or more radio resource managers of one or more radio access technologies for application to a communication network.

[0004] Another example embodiment may relate to an apparatus. The apparatus may include a component for receiving a load metric related to available resources of a carrier at a reinforcement learning load balancer. The apparatus may also include a component for calculating an update to a carrier assignment parameter for a user based on the load metric. The apparatus may also include a component for calculating an update to an uplink / downlink split ratio and a resource pool split ratio based on the load metric. In addition, the apparatus may include a component for predicting the impact of updates to carrier assignment parameters, updates to uplink / downlink split ratios, and updates to resource pool split ratios on the load metric. Further, the apparatus may include a component for transmitting updates to carrier assignment parameters, updates to uplink / downlink split ratios, and updates to resource pool split ratios to one or more radio resource managers of corresponding one or more radio access technologies for application to a communication network.

[0005] Another example embodiment may be directed to an apparatus that may include at least one processor and at least one memory, the at least one memory including computer program code. The at least one memory and the computer program code may be configured to use at least one processor to cause the apparatus to at least receive a load metric related to available resources of a carrier from a radio resource manager. The apparatus may also be caused to calculate an update to a carrier assignment parameter for a user based on the load metric. In addition, the apparatus may be caused to calculate an update to an uplink / downlink split ratio and a resource pool split ratio based on the load metric. The apparatus may also be caused to predict the impact of the update to the carrier assignment parameter, the update to the uplink / downlink split ratio, and the update to the resource pool split ratio on the load metric. In addition, the apparatus may be caused to transmit the update to the carrier assignment parameter, the update to the uplink / downlink split ratio, and the update to the resource pool split ratio to one or more radio resource managers of the corresponding one or more radio access technologies for application to a communication network.

[0006] According to some example embodiments, a non-transitory computer-readable medium may be encoded with instructions that may perform a method when executed in hardware. The method may include: receiving a load metric related to available resources of a carrier at a reinforcement learning load balancer. The method may also include: calculating an update to a carrier assignment parameter for a user based on the load metric. The method may also include: calculating an update to an uplink / downlink split ratio and a resource pool split ratio based on the load metric. In addition, the method may include: predicting the impact of the update to the carrier assignment parameter, the update to the uplink / downlink split ratio, and the update to the resource pool split ratio on the load metric. Further, the method may include: transmitting the update to the carrier assignment parameter, the update to the uplink / downlink split ratio, and the update to the resource pool split ratio to one or more radio resource managers of the corresponding one or more radio access technologies for application to the communication network.

[0007] According to some example embodiments, a computer program product may perform a method. The method may include: receiving a load metric related to available resources of a carrier at a reinforcement learning load balancer. The method may also include: calculating an update to a carrier assignment parameter for a user based on the load metric. The method may also include: calculating an update to an uplink / downlink split ratio and a resource pool split ratio based on the load metric. In addition, the method may include: predicting the impact of updates to carrier assignment parameters, updates to uplink / downlink split ratios, and updates to resource pool split ratios on the load metric. Further, the method may include: transmitting updates to carrier assignment parameters, updates to uplink / downlink split ratios, and updates to resource pool split ratios to one or more radio resource managers of corresponding one or more radio access technologies for application to a communication network.

[0008] According to some example embodiments, a device may include a circuit system configured to receive a load metric related to available resources of a carrier from a radio resource manager. The device may also include a circuit system configured to calculate an update to a carrier assignment parameter for a user based on the load metric. The device may also include a circuit system configured to calculate an update to an uplink / downlink split ratio and a resource pool split ratio based on the load metric. The device may also include a circuit system configured to predict the impact of updates to carrier assignment parameters, updates to uplink / downlink split ratios, and updates to resource pool split ratios on the load metric. In addition, the device may include a circuit system configured to transmit updates to carrier assignment parameters, updates to uplink / downlink split ratios, and updates to resource pool split ratios to one or more radio resource managers of corresponding one or more radio access technologies for application to a communication network.

[0009] According to some example embodiments, a method may include: sending a load metric related to available resources of a carrier from a radio resource manager. The method may also include: receiving an updated carrier assignment parameter, an updated uplink / downlink split ratio, and an updated resource pool split ratio. The method may also include: applying the updated carrier assignment parameter, the updated uplink / downlink split ratio, and the updated resource pool split ratio to a communication network.

[0010] According to some example embodiments, an apparatus may include means for sending a load metric related to available resources of a carrier from a radio resource manager. The apparatus may also include means for receiving updated carrier assignment parameters, updated uplink / downlink split ratios, and updated resource pool split ratios. In addition, the method may include means for applying the updated carrier assignment parameters, updated uplink / downlink split ratios, and updated resource pool split ratios to a communication network.

[0011] According to some example embodiments, an apparatus may include at least one processor and at least one memory, the at least one memory including computer program code. The at least one memory and the computer program code may be configured to, together with the at least one processor, cause the apparatus to at least send a load metric related to available resources of a carrier to a reinforcement learning load balancer. The apparatus may also be caused to receive updated carrier assignment parameters, updated uplink / downlink split ratios, and updated resource pool split ratios. The apparatus may also be caused to apply the updated carrier assignment parameters, updated uplink / downlink split ratios, and updated resource pool split ratios to a communication network.

[0012] According to some example embodiments, a non-transitory computer-readable medium may be encoded with instructions that, when executed in hardware, perform a method. The method may include: sending a load metric related to available resources of a carrier from a radio resource manager. The method may also include: receiving an updated carrier assignment parameter, an updated uplink / downlink split ratio, and an updated resource pool split ratio. The method may also include: applying the updated carrier assignment parameter, the updated uplink / downlink split ratio, and the updated resource pool split ratio to a communication network.

[0013] According to some example embodiments, a computer program product may perform a method. The method may include: sending a load metric related to available resources of a carrier from a radio resource manager. The method may also include: receiving an updated carrier assignment parameter, an updated uplink / downlink split ratio, and an updated resource pool split ratio. The method may also include: applying the updated carrier assignment parameter, the updated uplink / downlink split ratio, and the updated resource pool split ratio to a communication network.

[0014] According to some embodiments, an apparatus may include circuitry configured to send a load metric related to available resources of a carrier to a reinforcement learning load balancer. The apparatus may also include circuitry configured to receive updated carrier assignment parameters, updated uplink / downlink split ratios, and updated resource pool split ratios. The apparatus may also include circuitry configured to apply the updated carrier assignment parameters, updated uplink / downlink split ratios, and updated resource pool split ratios to a communication network. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] For a proper understanding of the example embodiments, reference should be made to the accompanying drawings, in which:

[0016] Figure 1 The diagram illustrates the coexistence of Long Term Evolution and New Radio in multi-carrier spectrum sharing.

[0017] Figure 2 The inter-radio access technology (RAT) load balancer input / output relationship to the RAT radio resource manager (RRM) is illustrated.

[0018] Figure 3 An example iterative multi-carrier dynamic spectrum sharing (MC-DSS) load balancing (LB) operation is illustrated.

[0019] Figure 4 An example of a reinforcement learning (RL) inter-RAT MC-DSS load balancer operation according to an example embodiment is illustrated.

[0020] Figure 5 An example of RL learning load balancer phase operation is illustrated according to an example embodiment.

[0021] Figure 6 Illustrated are examples of RL load balancer optimality deep Q network (DQN) inputs and outputs according to example embodiments.

[0022] Figure 7 An example of a reinforcement learning load balancer load metric updating deep neural network (DNN) inputs and outputs according to an example embodiment is illustrated.

[0023] Figure 8 An example of a reinforcement learning load balancer signal to interference and noise ratio prediction DNN input and output is illustrated according to an example embodiment.

[0024] Fig. 9 An example of an online training and search process for an RL load balancer according to an example embodiment is illustrated.

[0025] Fig.10 An example flow diagram of a method according to an example embodiment is illustrated.

[0026] Fig.11 An example flow diagram of another method according to an example embodiment is illustrated.

[0027] Fig.12a An apparatus according to an example embodiment is illustrated.

[0028] Figure 12b Another apparatus according to an example embodiment is illustrated. DETAILED DESCRIPTION

[0029] It should be readily appreciated that, as generally described herein and illustrated in the figures herein, the components of certain example embodiments may be arranged and designed in a variety of different configurations. Accordingly, the following is a detailed description of some example embodiments of systems, methods, apparatuses, and computer program products for reinforcement learning (RL) based inter-radio access technology (inter-RAT) load balancing in the context of multi-carrier dynamic spectrum sharing (SS).

[0030] The features, structures, or characteristics of the example embodiments described throughout this specification may be combined in any suitable manner in one or more example embodiments. For example, the phrases "certain embodiments," "example embodiments," "some embodiments," or other similar language used throughout this specification refers to the fact that a particular feature, structure, or characteristic described in conjunction with an embodiment may be included in at least one embodiment. Therefore, the phrases "certain embodiments," "in example embodiments," "in some embodiments," "in other embodiments," or other similar language appearing throughout this specification do not necessarily all refer to the same set of embodiments, and the described features, structures, or characteristics may be combined in any suitable manner in one or more example embodiments.

[0031] In addition, the different functions or steps discussed below may be performed in different orders and / or simultaneously with each other, as desired. In addition, one or more of the functions or steps described may be optional or may be combined, as desired. Thus, the following description should be considered to be merely illustrative of the principles and teachings of certain example embodiments, rather than limiting on certain example embodiments.

[0032] 5G is expected to have multiple radio interfaces, namely, below 6GHz, cmWave and mmWave and can also be integrated with existing legacy radio access technologies (such as LTE). At least in the early stages, the integration with LTE can be implemented as a system in which macro coverage is provided by LTE and 5G radio interface access comes from small cells by aggregation to LTE. In other words, 5G plans to support both inter-RAT operability (such as LTE-5G) and inter-RI operability (inter-radio interface operability, such as below 6GHz–cmWave, below 6GHz–cmWave–mmWave). One of the concepts considered to be used in 5G networks is network slicing, in which multiple independent and dedicated virtual subnets (network instances) can be created within the same infrastructure to run services with different requirements for latency, reliability, throughput and mobility.

[0033] The current architecture in LTE networks is completely distributed in the radio and completely concentrated in the core network. Low-latency applications and services in 5G require content to be close to the radio, which will lead to local grooming and multi-access edge computing (MEC). 5G enables analysis and knowledge generation to occur at the data source. This approach requires the use of resources that may not be continuously connected to the network (such as laptops, smart phones, tablets, and sensors). MEC provides a distributed computing environment for application and service hosting. It can also store and process content in close proximity to cellular users to speed up response time. Edge computing covers a wide range of technologies that can also be classified as local cloud / fog computing and grid / mesh computing, dew computing, mobile edge computing, Duoyun, distributed data storage and retrieval, autonomous self-healing networks, remote cloud services, augmented and virtual reality, data caching, the Internet of Things (large-scale connectivity and / or latency critical) and critical communications (automatic vehicles, traffic safety, real-time analysis, time-critical control, healthcare applications), such as wireless sensor networks, mobile data acquisition, mobile signature analysis, collaborative distributed peer-to-peer ad hoc networking and processing.

[0034] 5G-NR has been specified in the 3rd Generation Partnership Project (3GPP) to be able to coexist with 4G-LTE in the same spectrum. Radio access network (RAN) operators may require the ability to coexist (i.e., spectrum sharing) in the same spectrum to save valuable spectrum resources while supporting multiple radio access technologies (RATs) under the same cellular coverage. In addition, coexistence can occur on multiple carriers in multiple bands.

[0035] For example, Figure 1 The diagram shows the coexistence of LTE and NR in multi-carrier spectrum sharing. Since the data load provided by the two RATs varies over time, the wireless system operators of the two RATs must use the shared radio resources efficiently. This can be achieved through an inter-RAT load balancer (LB) or through an inter-RAT common scheduler (CS), where both LB and CS can use the same throughput maximization or proportional fairness metric maximization framework, but on different time scales. In addition, inter-RAT CS can provide performance benefits over LB because it can exploit subframe or time slot level fluctuations in the load, but is under complexity penalties because CS requires synchronized multi-RAT operations. CS not only increases complexity, but also requires low-latency information exchange and is therefore only applicable to "single RAN" solutions. Therefore, LB can be a viable solution for dynamic spectrum sharing (DSS) at the beginning of coexistence NR-LTE deployment.

[0036] One advantage of 5G-NR over 4G-LTE is that only NR can support extremely wide single-carrier bandwidth (e.g. 100MHz) carriers, allowing much greater throughput. Typically, high-bandwidth carriers are located at high-frequency channels / bands (HFBs) (e.g. frequency region 2 (FR2): above 6GHz, or frequency region 1 (FR1): between 3 and 6GHz) compared to low-frequency bands (LFBs) that typically have low bandwidths (e.g. FR1: below 6GHz, or low FR1: below 3GHz). However, at HFBs, cell coverage is smaller due to high penetration losses at such high frequencies.

[0037] Compared with LFB, both (wide bandwidth and small cell coverage) will make HFB frequency resources severely underutilized. Typically, due to the low UL transmission (Tx) power (relative to DL), there is a large (up to 13dB) downlink (DL) and uplink (UL) coverage gap. In many cases, multi-carrier operation (coupling) is performed between LFB and HFB carriers to alleviate this imbalance between HFB and LFB, and support improvements in coverage, mobility and reliability of HFB.

[0038] It may be beneficial to perform multi-carrier (MC) operation for NR, where at least one of the aggregated carriers is located at the HFB (for improved throughput) and at least one of the aggregated carriers is located at the LFB (for improved coverage). However, in some cases, when NR is deployed, the LFBs may already be full of LTE carriers. With MC-SS, LFB resources can be used with HFB carriers for new 5G users while maintaining the same LFB resources to support pre-existing LTE users. Certain example embodiments may provide solutions to fairly share both spectrum and time resources to UEs of two RATs and over multiple carriers.

[0039] Further, certain example embodiments may define a reinforcement learning (RL) algorithm for unified load balancing (LB) that handles spectrum sharing (between LTE and NR) and LFB-HFB coupling (within NR). This is in addition to the existing LB between carriers within LTE and within NR, and uplink (UL) / downlink (DL) split if a time division duplex (TDD) frame configuration is used. It can also be applied to networks with frequency division duplex (FDD) frame configurations.

[0040] Previously, solutions for inter-RAT load balancing for MC-DSS (MC-DSS LB) have been provided in an effort to maximize user throughput and improve reliability by iteratively and fairly adjusting 1) received signal strength (RSS) or power headroom (PHR) thresholds and target assignment ratios for primary carrier assignments, 2) partitioning of the shared resource pool, and 3) TDD frame configuration. All three can be dynamically updated based on the load metric (LM) of each scheduler to balance the two links. Figure 2 Its input / output relationship with the RAT Radio Resource Manager (RRM) is illustrated in FIG.

[0041] In the past, shared RF resources have been divided into orthogonal (i.e., non-co-interfering) resource pools, with each RAT being assigned a resource pool. A resource pool is an aggregation of time and frequency segments that are dedicated to the RAT and not to any other dedicated use. The LB adjusts the proportion of each resource pool based on the relative provided data load of each RAT at each carrier. After the resource pool proportions are determined by the LB, scheduling of the UE can be performed by each RAT network independently for its resource pool. In addition, the LB assigns the UE to its appropriate primary carrier taking into account the intra-RAT and inter-RAT loads on these carriers. Moreover, for a system with one or more carriers configured using a TDD frame, the LB calculates the TDD UL / DL split ratio taking into account the utilization of the resource pools of multiple RATs at each carrier.

[0042] The RATs may agree on how to divide the shared resource pool between themselves so that when the RAT receives its corresponding resource pool split from the LB, orthogonality between the RP splits is ensured. In addition, the RATs may agree on how to divide the TDD frame configuration so that UL and DL transmissions across the RATs will be synchronized when the UL / DL split ratio is determined by the LB. The splitting rules for RP and TDD may be determined by one of the RRM or by another entity, such as the LB.

[0043] MC-DSS LB operation can be characterized with respect to a three-ring architecture. For example, the LB scheme can iteratively optimize the resource partitioning and user assignment using a common load metric across multiple RATs, links (i.e., UL / DL), bands, and carriers. The scheme can also generate a maximized total log throughput or balanced utilization of resources across all RATs, links, bands, and carriers.

[0044] Figure 3 An example iterative multi-carrier dynamic spectrum sharing (MC-DSS) load balancing (LB) operation is illustrated. Figure 3 As illustrated in , the iterative operation includes an outer loop of TDD frame configuration, a middle loop of resource pool configuration, and an inner loop of primary carrier assignment of UE.

[0045] In the outer loop, after a certain scheduling period or after being triggered, the scheduler of each cell can measure the load metric and pass the load metric to the LB. Using the load metric, the LB can then calculate the UL / DL slicing ratio based on the combined reported load metric of one or more cells. The LB can also pass the slicing ratio to the TDD frame configuration of each cell, which then applies the slicing ratio. Once the slicing ratio is applied by each cell, the middle loop can be executed, and the process of the outer loop can be repeated indefinitely.

[0046] In the middle loop, after a certain scheduling period or after being triggered, the scheduler can measure the load metric and pass the load metric to the LB. Using the load metric, the LB can calculate the resource pool split ratio for each RAT for one or more cells. In addition, the LB can pass the split ratio to the resource pool configuration, which then applies the split ratio. Once the split ratio is applied for each cell, the inner loop can be executed, and the process of the middle loop can be repeated for several iterations, such as, for example, one or more iterations. One iteration can include the measurement of the load metric, the calculation of the configuration parameters, and the application of the configuration parameters.

[0047] In the inner loop, after a certain scheduling period or after being triggered, the scheduler can measure the load metric and pass the load metric to the LB. For one or more cells, the LB can calculate the target number of UE ratios per carrier and receive signal strength (RSS) thresholds for each band. In an example embodiment, the number of UE ratios per carrier can be based on the size of the resource pool for each carrier. In addition, the RSS threshold can be intended to make UEs fit in different frequency bands fairly while ensuring that the coverage of each band (based on the type of MC operation) is not exceeded.

[0048] In the inner loop, the calculated target UE ratio and RSS threshold can be passed to the primary carrier assignment (PCA) after the LB calculates the target number of UE ratios for each carrier and the RSS threshold for each band. The PCA can then assign (apportion) UEs based on the target ratio per carrier and the RSS threshold per band. After the assignment, scheduling can be performed by the cell. Like the outer and middle loops, the process of the inner loop can be repeated for several iterations, and the process can return to the middle loop.

[0049] like Figure 3As illustrated in , the process is an iterative closed form process in which frame configuration, resource pool configuration, and primary carrier assignment are performed sequentially. As such, its convergence to optimal load balancing is slow. Moreover, since the process is reactive to historical loads, it does not provide the ability to load balance for predicted future loads, which may be different from historical loads. Therefore, according to certain example embodiments, an RL algorithm may be provided that jointly calculates and applies optimized frame configurations, resource pool configurations, and primary carrier assignments based on both historical loads and predicted future loads. In addition, the algorithm may include signal-to-interference-plus-noise ratio (SINR), RSS, and PHR distributions into the RL model. These can help perform per-user carrier assignment decisions that affect future loads. Further, due to the ability to perform optimized single-trigger joint updates of multiple LB configuration parameters through RL, certain example embodiments may achieve optimized load balancing faster and more accurately than the above-described previous processes.

[0050] Figure 4 FIG. 4 illustrates an example of RL inter-RAT MC-DSS load balancer operation according to an example embodiment. According to an example embodiment, the RL load balancer may consist of several stages, such as Figure 4 As shown in FIG. Further, Figure 5 An example of the operation of an RL learning load balancer stage according to an example embodiment is illustrated. Here, each stage can calculate potential updates to carrier assignment parameters and scale and predict the impact of these changes on load metrics and overall load balancing. The positive impact of a change in the carrier assignment configuration of a stage on the overall load balance is indicated by a positive Bellman equation Q function value, which is calculated by a deep Q network (DQN). Further, the RL load balancer can use a Q function value that, when maximized, can result in a maximized geometric mean of user throughput across all carriers, bands, RATs, and links.

[0051] In addition, the changes in load metrics caused by changes in carrier assignment configuration can be predicted by a deep neural network (DNN). Figure 5 As shown in , the UL / DL split ratio and the resource pool split ratio can be calculated based on the updated load metric using closed-form equations. If a positive maximum Q-function value is predicted, the predicted load metric and configuration parameters can be passed to the subsequent stage. Otherwise, the RL process can be terminated, and the calculated configuration parameters of the previous stage can be passed and applied to its corresponding RRM.

[0052] Descriptions of load metric formulas are described herein. A load balancing scheme may be formulated based on the load at each RAT / carrier / link combination. For example, in certain embodiments, the LM of a DL / UL link of a cell on a carrier may be formulated as an Average Cell Proportional Fairness Metric (ACPF) of the link or an Average Resource Pool Shared Utilization (ARPSU) of the cell. In an example embodiment, the LM of a DL cell or UL cell at a carrier may be represented as n (n=1,…,N) with m (m=1,…,M) for RAT r (r=1,…,R), respectively, as or

[0053] In the first LM option of ACPF, most packet scheduling of RATs can use proportional fair (PF) queuing when assigning users in the time and frequency domains. Further, the downlink PF metric for carrier [r, m, n] of user u for the kth time-frequency coded space resource can be given by:

[0054]

[0055] in is the achievable DL instantaneous throughput of the u-th user at the k-th resource on carrier [r,m,n], and is the average downlink throughput of the u-th user on carrier [r,m,n]. In addition, q u (q u >0) is a quality of service (QoS) weighting factor, which is based on the type of bearer service and the number of retransmissions. The instantaneous throughput may depend on the user's received signal-to-interference-and-noise ratio (SINR), multiple-input multiple-output (MIMO) rank, and data buffer size. Moreover, SINR may take into account receiver noise floor, inter-cell interference, multi-user MIMO cross-beam interference, and non-orthogonal multiple access (NOMA) interference.

[0056] Uplink PF metric Can be similarly formulated. Further, according to another example embodiment, for ACPF, the downlink LM of carrier [r, m, n] can be formulated as:

[0057]

[0058] in is the total number of resources that are part of the pool assigned for the rth RAT, is the total number of downlink users, and is the user selection indicator for the u-th to k-th user-resource pair, and E() is a sample average function, which may be across time samples or across a set of cells or across both. In an example embodiment, the sample average function may be implemented by a discrete time filter. In addition, the LB may use filters with different correlation times for E() at each stage of the LM scheme, depending on the update period of each stage. (in ) is a carrier-specific load multiplier used to optionally provide cross-carrier priorities. Priorities can be used to scale for different target users or cell throughputs across links, carriers, and bands or for other reasons.

[0059] For the single-user MIMO case with PF scheduling, the user selection indicator can be formulated as:

[0060]

[0061] Selecting the user that maximizes PF for resources is called "selection of the best". can be formulated similarly. For multi-user MIMO, for one or more users of the k-th resource

[0062] Under higher data traffic loads, data queues may start to build up at the buffers, which will result in a higher maximum PF metric.Thus, the PF metric may be used as a load measure, especially under higher loads.

[0063] As a second LM option, ARPSU may formulate downlink LM as follows:

[0064]

[0065] in is the resource selection indicator for the kth resource of the pool, and is the amount of remaining data in the buffer. The fractional term represents the predicted amount of resources required to exhaust the remaining data in the buffer, and ρ∈{0,1} is the activation / deactivation parameter of the fractional term. In the example embodiment, the uplink LM can be formulated similarly.

[0066] By using the resource pool shared utilization as a load balancing metric, the utilization of shared resources can be balanced across all RATs and all carriers. However, since the value of ARPSU is limited to between 0% and 100% of all resources for a carrier (assuming ), so load balancing using the ARPSU metric cannot handle load balancing for full-buffer or near-full-buffer traffic scenarios. On the other hand, load balancing using the ACPF metric includes QoS priorities and is able to handle full-buffer scenarios that ARPSU does not. However, load balancing using ARPSU is simpler than through ACPF and may be sufficient at lower or medium traffic loads.

[0067] In an example embodiment, a third LM option, total average resource requirement (TARR), may take into account the historical amount of resources utilized and the predicted amount of additional resources for data remaining in the buffer. The DL LM may be formulated as:

[0068]

[0069] in is the amount of remaining data in the buffer. The fractional term represents the predicted amount of resources required to exhaust the remaining data in the buffer.

[0070] According to example embodiments, an optimality formula may be provided for load balancing. For example, the optimality criterion for MC-SS LB may optimize the total log throughput for all RATs. According to example embodiments, the optimality criterion may incorporate load metrics across links, carriers, bands, and RATs. It may also contain constraints on resource pool ratios and carrier assignment thresholds, and it may be used to evaluate the reward component of the Q function.

[0071] The load metric is essentially a measure of congestion of how much buffered data there is or the number of active users competing for the available resources of the carrier. Therefore, a suitable function of load balancing performance may be the assignment of resources and users. This can maximize the per-user log throughput as a function of the load metric and the allocated resources to the carrier. In an example embodiment, the optimality formula for the load balancing metric F is detailed in equation (6) below. (6)

[0073]

[0074] Make

[0075]

[0076]

[0077]

[0078]

[0079]

[0080] In equation (6), for each carrier In addition, the RSS / PHR threshold Minimum and maximum constraints are imposed, depending on the type of multicarrier operation. Load Metrics is a function of the user's carrier assignment, which can be determined by the RSS / PHR threshold together with the offered data traffic load.

[0081] According to an example embodiment, for future convenience, the total load of the carrier, the total load of the band, the total resource of the band, the total load metric L as the sum of all load metrics may be calculated. total and the global total resource K as the sum of all resources total These metrics are shown in equations (7a) to (7e) as follows:

[0082]

[0083]

[0084]

[0085]

[0086]

[0087] The quantities from equations (7a) to (7e) can be used to express the solution to the optimality problem, where the integer constraints are removed As well as minimum and maximum constraints. For the case where LM is constant and the resource pool is about to be split, the function maximizes resource allocation It can be:

[0088]

[0089] On the other hand, in the case where carrier reassignment is to be performed and the amount of allocated resources is fixed, the target LM based on the allocated resources is:

[0090]

[0091] These solutions can maximize the sum of the log throughput of the users when the mentioned constraints are removed. However, finding the best solution can be challenging because integers, minimum constraints, and maximum constraints need to be considered in real networks, and the mapping of LM from carrier assignment thresholds depends on the SINR / RSS / PHR user distribution. To achieve this, certain example embodiments can use trained DQNs and (multiple) DNNs to find the best configuration adjustment and predict the change in LM due to the configuration adjustment.

[0092] According to an example embodiment, a multi-stage DQN can be used to find an optimized primary carrier assignment (PCA) configuration for LB. In addition, the DQN can determine an optimized new RSS / PHR threshold, while the in-band carrier assignment probability can be calculated using a closed-form equation. These carrier assignment parameters can then be passed to the RRM, which can assign or reassign users to carriers that meet these parameters.

[0093] Example embodiments may provide an RL approach in which a DQN is used to arrive at an optimized RSS / PHR threshold and in-band carrier assignment probability. The RL approach may be useful when the allowed range of update values ​​for the RSS / PHR threshold is limited (i.e., the RSS / PHR threshold is not allowed to change drastically) so that the optimized RSS / PHR threshold update may exceed the allowed range of update values. Additionally, the RL approach may have the ability to converge to an optimal load balance compared to iterative RSS / PHR threshold determination. Furthermore, since the range of update values ​​is limited, the RL approach may avoid overshooting of the optimized update value (and thus avoid overshooting of the optimal load balance), which is not addressed by supervised learning.

[0094] According to an example embodiment, the RL method may use a multi-stage model, where the vth stage uses DQN to determine the minimum threshold update. The threshold update to the previous RSS / PHR threshold may be an accumulation of micro-updates made at each stage:

[0095]

[0096] where Y is the number of RL stages of the sample, and the set of possible updates is small, e.g. Once an update is identified, you can update it from and previous RSS / PHR thresholds The sum of the RSS / PHR thresholds is obtained

[0097]

[0098] If the size of the DQN output vector is large, then a single stage may be sufficient - allowing many quantized threshold update values. Otherwise, through micro-updates in each stage, a multi-stage model can allow a larger number of possible quantized threshold update values ​​while reducing the complexity of the DQN by limiting the size of the DQN output vector.

[0099] In an example embodiment, the first frequency band corresponding to the lowest frequency may have the largest coverage among all frequency bands and may serve as a fallback when other frequency bands are out of coverage of the UE. Thus, the threshold of the first frequency band may be fixed to its minimum value to ensure maximum coverage: While allowing for a small number of possible micro-updates to the thresholds for other bands: For example, δ∈{-1, 0, 1}. According to some example embodiments, the DQN model may model the set of Q values ​​of the Bellman equation as shown in equation (12):

[0100]

[0101] In equation (12), η is the future discount factor (0<η≤1), s is the state the system is in, and a is the action taken in state s. The Q value represents the future reward observed when taking action a, and it is equal to the immediate reward r(s,a) and the potential maximum future reward of state s' The sum of , assuming that action a is taken, the system will transition to state s'. The Q value is the output of the DNN.

[0102] According to an example embodiment, a state s may be defined with respect to the input to the DQN at stage v as: (1) the previous load metric (2) Previous LB configuration: (a) RSS / PHR threshold of the previous DQN stage: m>1; (b) UL / DL split ratio (optional); and (c) resource pool split ratio (Optional) (3) Previous SINR PMF of one or more r, m, n, l (optional); (4) one or more previous RSS / PHR PMFs of r, m, n, l (optional); and (5) timestamp (optional).

[0103] In an example embodiment, inputs from measurements and configurations collected at one or more previous update intervals may be provided. Each set of DQN inputs for an update interval may have its own set of DQN input nodes, or data from several update intervals may be linearly combined to produce a single set of DQN input nodes. If sets of inputs for consecutive update intervals are to be continuously input, then a recurrent neural network architecture may be beneficial in accurately predicting the output.

[0104] According to an example embodiment, each action a corresponds to a set with a specific update to the RSS / PHR thresholds and may have a corresponding Q value. The reward value r(s,a) of the Q function is the net increase in the optimality function F of equation (5).

[0105] r(s,a)=F(s′(s,a))-F(s) (13)

[0106] The segmentation ratio can be calculated by equations (14) and (15) described below.

[0107] Assuming η = 1, the Q value of action a in stage y is expressed by Add to The remaining maximum potential increase in the optimality of Consider the micro-update of action a and the future micro-updates of subsequent Q-maximizing actions. Further, in order to obtain the optimal Q function, value iteration (i.e., exploration / exploitation) of all states and actions may be required. The RL exploration / exploitation strategy (e.g., ε-Greedy) during DQN training can be left to the implementation discretion.

[0108] Figure 6 An example of a reinforcement learning load balancer optimality DQN input and output is illustrated according to an example embodiment. Figure 6 As illustrated in , during DQN training, for a large threshold update, the update can be broken down into smaller "micro-update" steps, the sum of which is the threshold update. The DQN can then be trained for each of the micro-updates, where the Q-values ​​of these micro-updates are the same. The same Q-value can be assigned to all component micro-updates for all RATs (r), links (l), and bands (m) except the first band.

[0109] According to an example embodiment, a threshold value may determine the assignment of a band to a user. For example, if the user's RSS / PHR meets the RSS / PHR threshold of the band, then the user may be assigned to the band. The priority between bands that provide a sufficient threshold for the user may be based on the user's maximum achievable throughput on the band:

[0110]

[0111] in is the throughput per resource, which is the user's The predicted SINR function. The quantity can be a priori or a new value as calculated from (20) and (21). The expected SINR of the user can be a historical measurement or a predicted value.

[0112] In an example embodiment, a higher maximum achievable user throughput on a frequency band may be given priority over a frequency band with a lower maximum achievable user throughput. For example, a higher frequency band providing a sufficient threshold may provide a higher maximum achievable user throughput.

[0113] According to an example embodiment, once a user is assigned to a band, it is assigned to the carrier with a certain probability based on the previous LM or new LM, the total number of resources per carrier, and the SINR per carrier calculated as discussed below by equation (19). The decision on which option to use may depend on the availability of measurements and the speed of reconfiguration relative to the traffic load. If a carrier-specific SINR or broadband throughput is available for each user, then the first option (maximizing user throughput) may be more preferred, in which case the user throughput can be maximized. Otherwise, the next two options may be preferred. Specifically, when the reconfiguration period is much shorter than the rate of change of the natural fluctuations in the traffic load, the second option (balanced LM) may be preferred over the third option, while the third option (proportional resource allocation) is preferred in other cases.

[0114] In the intra-band carrier assignment of option 1 for maximizing user throughput, users may be assigned to the intra-band carrier that maximizes the predicted throughput of each user given the current load of the resources of the given carrier. According to an example embodiment, the predicted throughput of the u-th user for carrier [r, m, n] may be

[0115]

[0116] The probability of user assigning a carrier to a carrier Can be user-specific and taken as the maximum value among all in-band carriers:

[0117]

[0118] In the in-band carrier assignment of option 2, LM can be balanced. For example, the probability of carrier assignment of a user to a carrier is The LM of a carrier may be complementary to the LM of other carriers and may be formulated as:

[0119]

[0120] in and and is the previous LM or the new LM calculated from equation (19). This expression ensures that in the next update cycle, highly congested carriers are given lower priority than less congested carriers. This formula is applicable when the number of carrier resources in the same frequency band is the same.

[0121] In the intra-band carrier assignment of option 3, proportional resource allocation can be provided. In this case, the carrier assignment probability of the third option is The following equation is used, where this value is the ratio of the total number of resources of the carrier to the band:

[0122]

[0123] A third option may assume that users can be reassigned during the next update period, or that a large number of new users will be added or dropped during the next update period.

[0124] According to an example embodiment, the DNN may be used to determine an optimized TDD UL / DL split ratio and resource pool split ratio, which corresponds to the optimal carrier assignment predicted by the DQN. In an example embodiment, once the optimal RSS / PHR threshold updates and carrier assignment probabilities are determined, the corresponding updates to the load metrics for a given change in the carrier assignment thresholds may be predicted with the aid of the DNN. The predicted load metric updates may then be used to find the optimized TDD UL / DL split ratio and resource pool split ratio. In addition, the DNN model inputs and outputs may be the previous and new LB configurations and the corresponding previous and new load metrics, and training may involve backpropagation to tune the DNN weights.

[0125] In an example embodiment, the components of the RL LB may include a prediction of changes in the load metric when users are assigned or reassigned to different carriers. Further, the new LM may be used to determine the optimal split ratio via a closed-form solution. In addition, the architecture of the LM prediction DNN (such as the type of neural network (e.g., recurrent, convolutional, multi-layer perceptron), the number of layers, the number of nodes per layer, etc.) may depend on the implementation. Specifically, the set of inputs and outputs may be set to: (1) LB configuration update (change in RSS / PHR threshold) (2) Previous load measurement (3) Previous LB configuration, which includes a) RSS / PHR threshold b) UL / DL split ratio (optional) and c) resource pool split ratio (optional); (4) one or more previous SINR PMFs of r, m, n, l (optional); (5) one or more previous RSS / PHR PMFs of r, m, n, l (optional); and (6) timestamp (optional).

[0126] According to an example embodiment, optional inputs may include RSS / PHR Probability Mass Function (PMF) and timestamp. PMF may provide additional information to LB when setting RSS / PHR thresholds, while timestamp may account for periodic traffic behavior and may consist of time of day and period of week.

[0127] In an example embodiment, the inputs may come from measurements and configurations collected at one or more previous update intervals. Each DNN input set for an update interval may have its own set of DNN input nodes, or data from several update intervals may be linearly combined to produce a single set of DNN input nodes. If the input sets for consecutive update intervals are to be continuously input, then a recurrent neural network architecture may be beneficial in accurately predicting the output.

[0128] Figure 7 An example of a reinforcement learning load balancer LM updating DNN inputs and outputs according to an example embodiment is shown. In an example embodiment, the predicted changes of LM can be determined by Representation. The prior value generated by DNN can be and the predicted update value Get the predicted new LM as follows:

[0129]

[0130]

[0131] As indicated above, equation (19a) may be used during DNN prediction (including, for example, during a DNN inference step), while equation (19b) may be used during a training back-propagation step. Figure 7 Specific inputs and outputs are illustrated in .

[0132] As mentioned herein, the UL / DL split ratio and the resource pool split ratio may be calculated based on the updated load metric. According to an example embodiment, based on the predicted or measured load metric, equations (7c), (7d), (7d), (8) may be used to find an optimized split ratio. For example, the new UL / DL split ratio may be applicable to a carrier with the following conditions:

[0133]

[0134] In another example embodiment, the new resource pool split ratio may be found using the following formula:

[0135]

[0136] According to an example embodiment, for FDD carriers, the UL / DL split ratio can be constant and does not need to be calculated. The new optimized split ratio can be transmitted by the LB to the RRM, which is then applied to the network. In addition, the split ratio together with the LM can be used to calculate the load balancing metric F (Equation (6a)), which can then be used to determine the predicted reward (Equation (13)) and Q value (Equation (12)) during the DQN training phase.

[0137] In an example embodiment, the average SINR of the UE may be predicted over the update period using at least one previous SINR / RSS / PHR sample, a predicted load metric, a predicted resource slicing, and a DNN. According to an example embodiment, the predicted average SINR may be used to calculate a predicted user throughput, which may represent a criterion for carrier assignment.

[0138] According to an example embodiment, in the first carrier assignment option described herein ("Maximize User Throughput"), a carrier that potentially maximizes a user's throughput may be assigned based on the expected SINR on the carrier and the amount of resources on the carrier. The average SINR of the UE may change dynamically during the LB update period and may be different from the average SINR measured previously. Therefore, directly replacing future SINR samples with past SINR measurement samples may produce inaccurate average throughput predictions. However, by reliably predicting the average SINR of the UE via a DNN, a more optimized carrier assignment may be performed.

[0139] In an example embodiment, the DNN can be used to predict the average SINR of a user based on the input ff. According to certain example embodiments, the SINR DNN input may include: (1) the previous average SINR of the user at the same carrier: (2) The previous average RSS / PHR of the user at the same carrier: (3) Carrier Predicted Load Measurement (4) Carrier prediction resource pool division and (5) a timestamp (optional).

[0140] According to certain example embodiments, the inputs may come from measurements and configurations collected at one or more previous update intervals. Furthermore, each set of DNN inputs for an update interval may have its own set of DNN input nodes. Alternatively, data from several update intervals may be linearly combined to produce a single set of DNN input nodes. In example embodiments, if each set for an update period has its own set of DNN input nodes, then a recurrent neural network architecture may be beneficial in accurately predicting DNN outputs.

[0141] Figure 8An example of a reinforcement learning load balancer SINR prediction DNN input and output according to an example embodiment is illustrated. The predicted average SINR of a user at a carrier can be represented by: In addition, Figure 8 An input / output diagram of the SINR prediction DNN is illustrated in FIG. According to an example embodiment, during the deep learning prediction phase, the predicted LM and resource pool splits may be calculated by the LM DNN and derived from equations (18) to (20). However, during the training phase, actual measurements of the load metric may be used.

[0142] In an example embodiment, the calculation of average SINR / RSS / PHR from multiple samples during the update period can be left to the implementation. Moreover, the architecture (e.g., convolutional NN, recurrent NN, etc.), super parameterization, and the method of training DNN can be left to the implementation.

[0143] Fig. 9 An example of an online training and search process for an RL load balancer according to an example embodiment is illustrated. In an example embodiment, a method for collecting samples from simulated and real-world networks to train DQN and DNN may be provided. For example, training an RL load balancer may include collecting time-aligned training samples from simulated or real-world networks. Fig. 9 As illustrated in , this can be achieved with the help of configured time alignment, and LM measurements can be achieved with the help of delays. After online or offline training via back-propagation, the DQN and DNN weights can be applied to the RL LB.

[0144] According to example embodiments, online training may refer to continuous adaptation of DNNs and DQNs during simulation or network operation. Once the weights are updated by the latest configuration and corresponding measurements, the weights may be applied to the next configuration opportunity.

[0145] Fig.10 An example flow chart of a method according to an example embodiment is illustrated. In some example embodiments, Fig.10 The flowchart can be obtained by Fig.12a The device 10 shown in the figure is executed, and the device may include, for example, a reinforcement learning load balancer. According to an example embodiment, Fig.10 The method may include: initially, in 100, receiving a load metric related to available resources of a carrier from a radio resource manager. The method may also include: in 105, applying an optimality criterion on the load metric. According to an example embodiment, applying the optimality criterion may include: including the load metric in radio links, radio carriers, radio frequency bands, and radio access technologies. In another example embodiment, applying the optimality criterion may include: including at least one constraint on a resource pool ratio and a carrier assignment threshold.

[0146] The method may further include: in 110, calculating an update of a carrier assignment parameter for a user based on a load metric. In addition, the method may include: in 115, calculating an update of an uplink / downlink split ratio and a resource pool split ratio based on the load metric. Further, the method may include: in 120, predicting an average signal to interference plus noise ratio of the user equipment within an update period. According to an example embodiment, the prediction of the average signal to interference plus noise ratio may be performed using a previous signal to interference plus noise ratio, a previous received signal strength or a previous power headroom sample, a predicted impact on the load metric, and an updated uplink / downlink split ratio and an updated resource pool split ratio.

[0147] In addition, the method may include: predicting the impact of the update of the carrier assignment parameters, the update of the uplink / downlink split ratio, and the update of the resource pool split ratio on the load metric in 125. In addition, the method may include: calculating the Q function value in 130. Further, in 135, the method may include: based on the Q function value, passing the updated carrier assignment parameters, the updated uplink / downlink split ratio, and the updated resource pool split ratio to a subsequent stage in the multiple stages.

[0148] The method may further include: transmitting the update of the carrier assignment parameter, the update of the uplink / downlink split ratio, and the update of the resource pool split ratio to one or more radio resource managers for corresponding one or more radio access technologies applied to the communication network in 140. Further, in 145, the method may include: repeating steps 110, 115, and 120.

[0149] According to an example embodiment, the load metric may include at least one of: instantaneous throughput, average throughput, resource utilization, and amount of data waiting to be transmitted. According to another example embodiment, the updated carrier assignment parameters may include new received signal strength or new power headroom threshold. In another example embodiment, the updated carrier assignment parameters may be determined by a multi-stage deep Q network, and the updated uplink / downlink split ratio and the updated resource pool split ratio may be determined by a deep neural network based on the updated carrier assignment parameters and the predicted impact.

[0150] Fig.11 An example flow chart of another method according to an example embodiment is illustrated. In some example embodiments, Fig.11 The flowchart can be obtained by Figure 12b The method is executed by the apparatus 20 illustrated in FIG. 1 , which includes, for example, a radio resource manager.

[0151] According to an example embodiment, Fig.11The method may include: initially, in 200, sending a load metric related to available resources of a carrier to a reinforcement learning load balancer. The method may also include: receiving updated carrier assignment parameters, updated uplink / downlink split ratios, and updated resource pool split ratios in 205. In addition, the method may include: in 210, applying the updated carrier assignment parameters, updated uplink / downlink split ratios, and updated resource pool split ratios to the communication network.

[0152] According to an example embodiment, the load metric may include at least one of: an instantaneous throughput, an average throughput, a resource utilization, and an amount of data remaining in a transmit buffer. In another example embodiment, the updated carrier assignment parameters may include a new received signal strength or a new power headroom threshold. In another example embodiment, the updated carrier assignment parameters may be determined by a multi-stage deep Q network. According to another example embodiment, the updated uplink / downlink split ratio and the updated resource pool split ratio may be determined by a deep neural network based on the updated carrier assignment parameters and the predicted impact.

[0153] Fig.12a An apparatus 10 according to an example embodiment is illustrated. In an embodiment, the apparatus 10 may be a node or element in a communication network or associated with such a network, such as a reinforcement learning load balancer or other device.

[0154] In some example embodiments, the apparatus 10 may include one or more processors, one or more computer-readable storage media (e.g., memory, storage, etc.), one or more radio access components (e.g., modems, transceivers, etc.), and / or a user interface. In some embodiments, the apparatus 10 may be configured to operate using one or more radio access technologies (such as GSM, LTE, LTE-A, NR, 5G, WLAN, WiFi, NB-IoT, Bluetooth, NFC, MulteFire, and / or any other radio access technology). It should be noted that a person of ordinary skill in the art will appreciate that the apparatus 10 may include Fig.12a Components or features not shown.

[0155] like Fig.12a As shown in the example of , the device 10 may include or be coupled to a processor 12 for processing information and executing instructions or operations. The processor 12 may be any type of general or special purpose processor. In fact, as examples, the processor 12 may include one or more of the following: a general purpose computer, a special purpose computer, a microprocessor, a digital signal processor (DSP), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), and a processor based on a multi-core processor architecture. Although in Fig.12aA single processor 12 is shown in FIG. 1 , but multiple processors may be utilized according to other embodiments. For example, it should be understood that in certain example embodiments, the apparatus 10 may include two or more processors that may form a multiprocessor system that may support multiprocessing (e.g., in this case, the processor 12 may represent a multiprocessor). According to certain example embodiments, the multiprocessor system may be tightly coupled or loosely coupled (e.g., to form a computer cluster).

[0156] Processor 12 may perform functions associated with the operation of device 10, which may include, as some examples, precoding of antenna gain / phase parameters, encoding and decoding of individual bits forming communication messages, formatting of information, and overall control of device 10, including Figures 4 to 10 The process shown in the figure.

[0157] The device 10 may also include or be coupled to a memory 14 (internal or external), which may be coupled to the processor 12 for storing information and instructions that may be executed by the processor 12. The memory 14 may be one or more memories and of any type suitable for the local application environment, and may be implemented using any suitable volatile or non-volatile data storage technology (such as semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory, and / or removable memory). For example, the memory 14 can be composed of any combination of random access memory (RAM), read-only memory (ROM), static storage devices (such as magnetic disks or optical disks), hard disk drives (HDDs), or any other type of non-transitory machine or computer readable media. The instructions stored in the memory 14 may include program instructions or computer program codes that, when executed by the processor 12, enable the device 10 to perform tasks as described herein.

[0158] In an embodiment, the device 10 may also include or be coupled to a (internal or external) drive or port configured to accept and read an external computer-readable storage medium, such as an optical disk, a USB drive, a flash drive, or any other storage medium. For example, the external computer-readable storage medium may store a computer program or software for execution by the processor 12 and / or the device 10 to perform Figures 4 to 10 Any of the methods shown in the figure.

[0159] In some embodiments, the device 10 may also include or be coupled to one or more antennas 15 for receiving downlink signals from the device 10 and transmitting via an uplink. The device 10 may also include a transceiver 18 configured to transmit and receive information. The transceiver 18 may also include a radio interface (e.g., a modem) coupled to the antenna 15. The radio interface may correspond to a variety of radio access technologies, including one or more of the following: GSM, LTE, LTE-A, 5G, NR, WLAN, NB-IoT, Bluetooth, BT-LE, NFC, RFID, UWB, etc. The radio interface may include other components (such as filters, converters (e.g., digital-to-analog converters, etc.), symbol demappers, signal shaping components, inverse fast Fourier transform (IFFT) modules, etc.) to process symbols carried by a downlink or uplink, such as OFDMA symbols.

[0160] For example, the transceiver 18 may be configured to modulate information onto a carrier waveform for transmission by the antenna(s) 15 and to demodulate information received via the antenna(s) 15 for further processing by other elements of the device 10. In other embodiments, the transceiver 18 may be capable of directly transmitting and receiving signals or data. Additionally or alternatively, in some embodiments, the device 10 may include input and / or output devices (I / O devices). In some embodiments, the device 10 may also include a user interface, such as a graphical user interface or a touch screen.

[0161] In an embodiment, the memory 14 stores software modules that provide functionality when executed by the processor 12. The modules may include, for example, an operating system that provides operating system functionality for the device 10. The memory may also store one or more functional modules (such as applications or programs) to provide additional functionality for the device 10. The components of the device 10 may be implemented in hardware or as any suitable combination of hardware and software. According to an example embodiment, the device 10 may optionally be configured to communicate with the device 10 via a wireless or wired communication link 70 according to any radio access technology (such as NR).

[0162] According to some example embodiments, the processor 12 and the memory 14 may be included in or may form part of a processing circuit system or a control circuit system. In addition, in some embodiments, the transceiver 28 may be included in or may form part of a transceiver circuit system.

[0163] As discussed above, according to certain example embodiments, the device 10 may be, for example, a reinforcement learning load balancer. According to certain embodiments, the device 10 may be controlled by the memory 14 and the processor 12 to perform functions associated with the example embodiments described herein. For example, in one embodiment, the device 10 may be controlled by the memory 14 and the processor 12 to receive a load metric associated with available resources of a carrier from a radio resource manager. The device 10 may also be controlled by the memory 14 and the processor 12 to calculate an update to a carrier assignment parameter for a user based on the load metric. In addition, the device 10 may be controlled by the memory 14 and the processor 12 to calculate an update to an uplink / downlink split ratio and a resource pool split ratio based on the load metric. Further, the device 10 may be controlled by the memory 14 and the processor 12 to predict the impact of updates to carrier assignment parameters, updates to uplink / downlink split ratios, and updates to resource pool split ratios on the load metric. In addition, the device 10 can be controlled by the memory 14 and the processor 12 to transmit updates to the carrier assignment parameters, updates to the uplink / downlink split ratio, and updates to the resource pool split ratio to one or more radio resource managers of the corresponding one or more radio access technologies for application to the communication network.

[0164] Figure 12b An apparatus 20 according to an example embodiment is illustrated. In an example embodiment, the apparatus 20 may be a node, a host, or a server in a communication network or serving such a network. For example, the apparatus 20 may be a satellite, a base station, a Node B, an evolved Node B (eNB), a 5G Node B or an access point, a next generation Node B (NG-NB or gNB) and / or a WLAN access point associated with a radio access network (RAN) such as an LTE network, 5G or NR. It should be noted that a person skilled in the art will appreciate that the apparatus 20 may include Figure 12b Components or features not shown.

[0165] like Figure 12b As shown in the example of , the device 20 may include a processor 22 for processing information and executing instructions or operations. The processor 22 may be any type of general or special purpose processor. For example, as examples, the processor 22 may include one or more of the following: a general purpose computer, a special purpose computer, a microprocessor, a digital signal processor (DSP), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), and a processor based on a multi-core processor architecture. Although in Figure 12bA single processor 22 is shown in FIG. 1 , but multiple processors may be utilized according to other embodiments. For example, it should be understood that in some embodiments, the device 20 may include two or more processors that may form a multiprocessor system that may support multiprocessing (e.g., in this case, the processor 22 may represent a multiprocessor). In some embodiments, the multiple processors may be tightly coupled or loosely coupled (e.g., to form a computer cluster).

[0166] According to certain example embodiments, processor 22 may perform functions associated with the operation of apparatus 20, which may include, for example, precoding of antenna gain / phase parameters, encoding and decoding of individual bits forming communication messages, formatting of information, and overall control of apparatus 20, including Figures 4 to 9 and Fig.11 The process shown in the figure.

[0167] The device 20 may also include or be coupled to a memory 24 (internal or external), which may be coupled to the processor 22 for storing information and instructions that may be executed by the processor 22. The memory 24 may be one or more memories and of any type suitable for the local application environment, and may be implemented using any suitable volatile or non-volatile data storage technology (such as semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory, and / or removable memory). For example, the memory 24 can be composed of any combination of random access memory (RAM), read-only memory (ROM), static storage devices (such as magnetic disks or optical disks), hard disk drives (HDDs), or any other type of non-transient machine or computer readable media. The instructions stored in the memory 24 may include program instructions or computer program codes that, when executed by the processor 22, enable the device 20 to perform tasks as described herein.

[0168] In an embodiment, the device 20 may also include or be coupled to a (internal or external) drive or port configured to accept and read an external computer-readable storage medium, such as an optical disk, a USB drive, a flash drive, or any other storage medium. For example, the external computer-readable storage medium may store a computer program or software for execution by the processor 22 and / or the device 20 to perform Figures 4 to 9 and Fig.11 The method shown in the figure.

[0169] In certain example embodiments, the apparatus 20 may also include or be coupled to one or more antennas 25 for transmitting and receiving signals and / or data to and from the apparatus 20. The apparatus 20 may also include or be coupled to a transceiver 28 configured to transmit and receive information. The transceiver 28 may include, for example, a plurality of radio interfaces, which may be coupled to the antenna(s) 25. The radio interfaces may correspond to a variety of radio access technologies, including one or more of: GSM, NB-IoT, LTE, 5G, WLAN, Bluetooth, BT-LE, NFC, Radio Frequency Identifier (RFID), Ultra-Wideband (UWB), MulteFire, etc. The radio interfaces may include components (such as filters, converters (e.g., digital-to-analog converters, etc.), mappers, Fast Fourier Transform (FFT) modules, etc.) to generate symbols for transmission via one or more downlinks and receive symbols (e.g., via an uplink).

[0170] Thus, the transceiver 28 may be configured to modulate information onto a carrier waveform for transmission by the antenna(s) 25 and to demodulate information received via the antenna(s) 25 for further processing by other elements of the apparatus 20. In other embodiments, the transceiver 18 may be capable of directly transmitting and receiving signals or data. Additionally or alternatively, in some embodiments, the apparatus 20 may include input and / or output devices (I / O devices).

[0171] In an embodiment, the memory 24 may store software modules that provide functionality when executed by the processor 22. The modules may include, for example, an operating system that provides operating system functionality for the device 20. The memory may also store one or more functional modules (such as applications or programs) to provide additional functionality for the device 20. The components of the device 20 may be implemented in hardware or as any suitable combination of hardware and software.

[0172] According to some embodiments, processor 22 and memory 24 may be included in or may form part of a processing circuit system or a control circuit system. In addition, in some embodiments, transceiver 28 may be included in or may form part of a transceiver circuit system.

[0173] As used herein, the term "circuitry" may refer to a pure hardware circuitry implementation (e.g., analog and / or digital circuitry), a combination of hardware circuitry and software, a combination of analog and / or digital hardware circuitry and software / firmware, any portion of (multiple) hardware processors and software (including digital signal processors) that work together to enable a device (e.g., devices 10 and 20) to perform various functions, and / or (multiple) hardware circuitry and / or (multiple) processors or portions thereof that operate using software but that may not be present when not required for operation. As another example, as used herein, the term "circuitry" may also encompass implementations of only a hardware circuit or processor (or multiple processors), or a portion of a hardware circuit or processor and its accompanying software and / or firmware. The term "circuitry" may also encompass, for example, a baseband integrated circuit in a server, cellular network node or device, or other computing or network device.

[0174] As introduced above, in some embodiments, the device 20 may be a radio resource manager, RAT, node, host or server in a communication network or serving such a network. For example, the device 20 may be a satellite, base station, Node B, evolved Node B (eNB), 5G Node B or access point associated with a radio access network (RAN) (such as an LTE network, 5G or NR), a next generation Node B (NG-NB or gNB) and / or a WLAN access point. According to some embodiments, the device 20 may be controlled by the memory 24 and the processor 22 to perform functions associated with any of the embodiments described herein.

[0175] For example, in one embodiment, the device 20 may be controlled by the memory 24 and the processor 22 to send a load metric related to the available resources of the carrier to the reinforcement learning load balancer. The device 20 may also be controlled by the memory 24 and the processor 22 to receive updated carrier assignment parameters, updated uplink / downlink slicing ratios, and updated resource pool slicing ratios. In addition, the device 20 may be controlled by the memory 24 and the processor 22 to apply the updated carrier assignment parameters, updated uplink / downlink slicing ratios, and updated resource pool slicing ratios to the communication network.

[0176] Certain example embodiments described herein provide several technical improvements, enhancements, and / or advantages. In some example embodiments, by using an optimized joint update of LB configuration parameters of RL, it may be possible to achieve higher throughput and faster convergence than iterative load balancing. It may also be possible to provide an RL algorithm that jointly calculates and applies optimized frame configurations, resource pool configurations, and primary carrier assignments based on both historical and predicted loads. Moreover, it may also be possible to include SINR, RSS, and PHR distributions into the RL model to help in performing per-user carrier assignment decisions that may affect future loads.

[0177] According to certain example embodiments, given the ability to perform optimized, one-shot joint updates of multiple LB configuration parameters through RL, it may also be possible to achieve optimized load balancing faster and more accurately. In other example embodiments, when using a trained DQN and (multiple) DNNs, it may be possible to find optimal configuration adjustments and predict changes in LMs due to configuration adjustments. In addition, according to certain example embodiments, when using an RL approach, it may be possible to avoid overshoots of optimal update values ​​(and therefore avoid overshoots of optimal load balancing). According to other example embodiments, in a multi-stage RL approach model, it may be possible to allow a large number of possible quantization threshold update values ​​while reducing the complexity of the DQN by limiting the size of the DQN's output vector.

[0178] The computer program product may include one or more computer executable components that are configured to perform some example embodiments when the program is run. The one or more computer executable components may be at least one software code or part thereof. Modifications and configurations required to implement the functionality of the example embodiments may be performed as (multiple) routines, which may be implemented as added or updated (multiple) software routines. (Multiple) software routines may be downloaded to a device.

[0179] As an example, the software or computer program code or part thereof may be in source code form, object code form or some intermediate form, and the software or computer program code or part thereof may be stored in some carrier, distribution medium or computer readable medium, which may be any entity or device capable of carrying the program. For example, such carriers may include recording media, computer memory, read-only memory, optoelectronic and / or electrical carrier signals, telecommunication signals and software distribution packages. Depending on the processing power required, the computer program may be executed in a single electronic digital computer, or it may be distributed among several computers. The computer readable medium or computer readable storage medium may be a non-transient medium.

[0180] In other example embodiments, the functionality may be performed by hardware or circuitry included in a device (e.g., device 10 and device 20), such as by using an application specific integrated circuit (ASIC), a programmable gate array (PGA), a field programmable gate array (FPGA), or any other combination of hardware and software. In yet another example embodiment, the functionality may be implemented as a signal, a non-tangible component capable of being carried by an electromagnetic signal downloaded from the Internet or other network.

[0181] According to example embodiments, an apparatus (such as a node, a device or a corresponding component) may be configured as a circuit system, a computer or a microprocessor (such as a single-chip computer element) or a chipset including at least a memory for providing storage capacity for arithmetic operations and an operating processor for performing arithmetic operations.

[0182] A person of ordinary skill in the art will readily appreciate that the invention as discussed above may be practiced with hardware elements in a different order of steps and / or in a configuration different from that disclosed. Therefore, although the present invention has been described based on these example embodiments, it will be apparent to those skilled in the art that certain modifications, variations, and alternative constructions will be apparent while remaining within the spirit and scope of the example embodiments. Although the above embodiments refer to 5G NR and LTE technologies, the above embodiments may also be applied to any other current or future 3GPP technologies, such as LTE Advanced and / or fourth generation (4G) technologies.

[0183] Partial Glossary

[0184] ACPF Average Community Proportion Fairness

[0185] ARPSU Average Resource Pool Share Utilization

[0186] eNB Enhanced Node B

[0187] DL Downlink

[0188] DNN Deep Neural Network

[0189] DQN Deep Q Network

[0190] DSS Dynamic Spectrum Sharing

[0191] gNB5G or NR base station

[0192] HFB High Band

[0193] LB Load Balancing

[0194] LFB Low Frequency Band

[0195] LM load measurement

[0196] LTE Long Term Evolution

[0197] MC Multi-Carrier

[0198] NN Neural Network

[0199] NR New Radio

[0200] PCA Primary Carrier Assignment

[0201] PF Proportional Fairness

[0202] PHR Power Headroom

[0203] QoS Quality of Service

[0204] RAT Radio Access Technology

[0205] RL

[0206] RP Resource Pool

[0207] RRM Radio Resource Manager

[0208] RSS receiving signal strength

[0209] SS Spectrum Sharing

[0210] TC Tightly Coupled

[0211] TDD Time Division Duplex

[0212] UE User Equipment

[0213] UL Uplink

Claims

1. An apparatus included in a reinforcement learning load balancer, the apparatus comprising: at least one processor; as well as at least one memory including computer program code, The at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus to at least: receiving, from a radio resource manager, a load metric associated with available resources of a carrier; calculating an update to a carrier assignment parameter for a user based on the load metric; Calculating updates to uplink / downlink split ratios and resource pool split ratios based on the load metric; predicting the impact of the updating of the carrier assignment parameters, the updating of the uplink / downlink split ratio, and the updating of the resource pool split ratio on the load metric; as well as The updates to the carrier assignment parameters, the updates to the uplink / downlink split ratios, and the updates to the resource pool split ratios are communicated to one or more radio resource managers of corresponding one or more radio access technologies for application to the communication network.

2. The device according to claim 1, wherein the apparatus comprises a plurality of stages, and The at least one memory and the computer program code are further configured to, together with the at least one processor, cause the apparatus to at least: Calculating the Q function value; and Based on the Q function value, the updated carrier assignment parameters, the updated uplink / downlink slicing ratio, and the updated resource pool slicing ratio are passed to a subsequent stage in the plurality of stages.

3. The apparatus according to claim 1 or 2, wherein the at least one memory and the computer program code are further configured to, with the at least one processor, cause the apparatus to at least: calculating subsequent updates to carrier assignment parameters for the user; Calculating subsequent updates to the uplink / downlink split ratio and the resource pool split ratio; and A subsequent impact of the subsequent update of the carrier assignment parameters, the subsequent update of the uplink / downlink slicing ratio, and the subsequent update of the resource pool slicing ratio on the load metric is predicted. 4 . The apparatus according to claim 1 , wherein the load metric comprises at least one of the following: instantaneous throughput, average throughput, resource utilization, and amount of data waiting to be sent.

5. The apparatus according to any one of claims 1 to 4, wherein the at least one memory and the computer program code are further configured to, together with the at least one processor, cause the apparatus to at least: applying an optimality criterion to the load metric, The optimality criteria applied include: incorporating said load metrics across radio links, radio carriers, radio frequency bands and radio access technologies, and At least one constraint on resource pool ratio and carrier assignment threshold is incorporated. 6 . The apparatus according to claim 1 , wherein the updated carrier assignment parameter comprises a new received signal strength or a new power headroom threshold.

7. The device according to any one of claims 1 to 6, wherein the updated carrier assignment parameters are determined by a multi-stage deep Q network, and The updated uplink / downlink split ratio and the updated resource pool split ratio are determined by a deep neural network based on the updated carrier assignment parameters and the predicted impact.

8. The apparatus according to any one of claims 1 to 7, wherein the at least one memory and the computer program code are further configured to, together with the at least one processor, cause the apparatus to at least: The average signal to interference and noise ratio of the user equipment during the update period is predicted using the following terms: A previous signal to interference plus noise ratio, a previous received signal strength or a previous power headroom sample, the impact on the prediction of the load metric and the updated uplink / downlink split ratio and the updated resource pool split ratio.

9. An apparatus included in a radio resource manager, comprising: at least one processor; as well as at least one memory including computer program code, The at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus to at least: sending a load metric related to available resources of the carrier to a reinforcement learning load balancer; receiving updated carrier assignment parameters, updated uplink / downlink slicing ratios, and updated resource pool slicing ratios; as well as The updated carrier assignment parameters, the updated uplink / downlink slicing ratio, and the updated resource pool slicing ratio are applied to a communication network.

10. The apparatus according to claim 9, wherein the load metric comprises at least one of the following: instantaneous throughput, average throughput, resource utilization, and amount of data remaining in a sending buffer.

11. The apparatus according to claim 9 or 10, wherein the updated carrier assignment parameters include a new received signal strength or a new power headroom threshold.

12. The device according to any one of claims 9 to 11, wherein the updated carrier assignment parameters are determined by a multi-stage deep Q network, and The updated uplink / downlink split ratio and the updated resource pool split ratio are determined by a deep neural network based on the updated carrier assignment parameters and the predicted impact.

13. A method performed by a reinforcement learning load balancer, comprising: receiving, at a reinforcement learning load balancer, a load metric related to available resources of a carrier; computing, by the reinforcement learning load balancer, updates to carrier assignment parameters for users based on the load metric; The reinforcement learning load balancer calculates updates to the uplink / downlink split ratio and the resource pool split ratio based on the load metric; Predicting the impact of the update of the carrier assignment parameters, the update of the uplink / downlink split ratio, and the update of the resource pool split ratio on the load metric; as well as The updates to the carrier assignment parameters, the updates to the uplink / downlink split ratios, and the updates to the resource pool split ratios are communicated to one or more radio resource managers of corresponding one or more radio access technologies for application to the communication network.

14. The method according to claim 13, wherein the reinforcement learning load balancer comprises a plurality of stages, and The method further comprises: Calculate the Q function value, and Based on the Q function value, the updated carrier assignment parameters, the updated uplink / downlink slicing ratio, and the updated resource pool slicing ratio are delivered to a subsequent stage of the plurality of stages.

15. The method according to claim 13 or 14, wherein the method further comprises, in each subsequent stage of the plurality of stages based on a previous slicing ratio, a carrier assignment parameter and a predicted load metric: calculating subsequent updates to carrier assignment parameters for the user; Calculating subsequent updates to the uplink / downlink split ratio and the resource pool split ratio; and A subsequent impact of the subsequent update of the carrier assignment parameters, the subsequent update of the uplink / downlink slicing ratio, and the subsequent update of the resource pool slicing ratio on the load metric is predicted.

16. The method according to any one of claims 13 to 15, wherein the load metric comprises at least one of the following: instantaneous throughput, average throughput, resource utilization, and amount of data waiting to be sent.

17. The method according to any one of claims 13 to 16, wherein the method further comprises: applying an optimality criterion to the load metric, The optimality criteria applied include: incorporating said load metrics across radio links, radio carriers, radio frequency bands and radio access technologies, and At least one constraint on resource pool ratio and carrier assignment threshold is incorporated.

18. The method according to any one of claims 13 to 17, wherein the updated carrier assignment parameters include a new received signal strength or a new power headroom threshold.

19. The method according to any one of claims 13 to 18, wherein the updated carrier assignment parameters are determined by a multi-stage deep Q network, and The updated uplink / downlink split ratio and the updated resource pool split ratio are determined by a deep neural network based on the updated carrier assignment parameters and the predicted impact.

20. The method according to any one of claims 13 to 19, wherein the method further comprises: The average signal to interference and noise ratio of the user equipment during the update period is predicted using the following terms: A previous signal to interference plus noise ratio, a previous received signal strength or a previous power headroom sample, the impact on the prediction of the load metric, and the updated uplink / downlink split ratio and the updated resource pool split ratio.

21. A method performed by a radio resource manager, comprising: sending, from a radio resource manager, a load metric associated with available resources of the carrier; receiving updated carrier assignment parameters, updated uplink / downlink slicing ratios, and updated resource pool slicing ratios; as well as The updated carrier assignment parameters, the updated uplink / downlink slicing ratio, and the updated resource pool slicing ratio are applied to a communication network.

22. The method of claim 21, wherein the load metric comprises at least one of: instantaneous throughput, average throughput, resource utilization, and amount of data remaining in a sending buffer.

23. The method of claim 21 or 22, wherein the updated carrier assignment parameters include a new received signal strength or a new power headroom threshold.

24. The method according to any one of claims 21 to 23, wherein the updated carrier assignment parameters are determined by a multi-stage deep Q network, and The updated uplink / downlink split ratio and the updated resource pool split ratio are determined by a deep neural network based on the updated carrier assignment parameters and the predicted impact.

25. An apparatus for communication comprising means for performing the method according to any one of claims 13 to 24.

26. An apparatus for communication, comprising circuitry configured to cause the apparatus to perform the method according to any one of claims 13 to 24.

27. A computer program product comprising a computer program which, when executed by a processor of an apparatus, causes the apparatus to at least perform the method according to any one of claims 13 to 24.

Images