Network performance optimization

Abstract

Embodiments of the present disclosure relate to network performance optimization, and more particularly to gNB performance optimization based on reinforcement learning (RL) operations. The method includes obtaining current state information of a network environment and a reward, the reward being associated with a reinforcement learning operation and pertaining to performance of the network environment; determining a predicted future state of the network environment based on the current state information; and generating a target set of configuration parameters of the network environment based on at least the predicted future state of the network environment by using the reinforcement learning operation; and configuring the network environment using the target set of configuration parameters.

Description

Technical Field

[0001] Various example embodiments of this disclosure generally relate to the telecommunications field, and more specifically to methods, apparatuses, devices, and computer-readable storage media for network performance optimization, particularly for gNB performance optimization based on reinforcement learning (RL) operations. Background Technology

[0002] Network devices (such as gNBs) have over one hundred field-adjustable configuration parameters to address the time-varying characteristics of wireless networks. However, because different network environments can change at any time, system performance is not yet optimal. Summary of the Invention

[0003] In a first aspect of this disclosure, a first apparatus is provided. The first apparatus includes at least one processor; and at least one memory storing instructions, which, when executed by the at least one processor, cause the first apparatus to at least: acquire current state information and a reward of a network environment, the reward being associated with reinforcement learning operations and relating to the performance of the network environment; determine a predicted future state of the network environment based on the current state information; generate a target configuration parameter set for the network environment by using reinforcement learning operations, at least based on the predicted future state of the network environment; and configure the network environment using the target configuration parameter set.

[0004] In a second aspect of this disclosure, a method is provided. The method includes: acquiring current state information and a reward of a network environment, the reward being associated with a reinforcement learning operation and relating to the performance of the network environment; determining a predicted future state of the network environment based on the current state information; generating a target configuration parameter set for the network environment by using a reinforcement learning operation, at least based on the predicted future state of the network environment; and configuring the network environment using the target configuration parameter set.

[0005] In a third aspect of this disclosure, a first apparatus is provided. The first apparatus includes: units for acquiring current state information and a reward of a network environment, the reward being associated with a reinforcement learning operation and relating to the performance of the network environment; units for determining a predicted future state of the network environment based on the current state information; units for generating a target configuration parameter set for the network environment by using a reinforcement learning operation, at least based on the predicted future state of the network environment; and units for configuring the network environment using the target configuration parameter set.

[0006] In a fourth aspect of this disclosure, a computer-readable medium is provided. The computer-readable medium includes instructions stored thereon for causing a device to at least execute the method according to the second aspect.

[0007] It should be understood that the summary portion is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0008] Some example implementation schemes will now be described with reference to the accompanying drawings, in which: Figure 1 An example communication environment in which an example implementation of this disclosure may be carried out is shown; Figure 2 A schematic diagram of an RL-based performance optimization process according to some example embodiments of the present disclosure is shown; Figures 3A-3C Examples of UE distributions according to some exemplary embodiments of this disclosure are shown; Figure 4 An example of a state prediction module according to an embodiment of this disclosure is shown; Figure 5 A flowchart is shown showing a method implemented at a first device according to some example embodiments of the present disclosure; Figure 6 A simplified block diagram of an apparatus suitable for implementing an example embodiment of this disclosure is shown; and Figure 7 A block diagram of an example computer-readable medium according to some example embodiments of the present disclosure is shown.

[0009] In all the accompanying drawings, the same or similar reference numerals denote the same or similar elements. Detailed Implementation

[0010] The principles of this disclosure will now be described with reference to some example embodiments. It should be understood that these embodiments are described for illustrative purposes only and to assist those skilled in the art in understanding and implementing this disclosure, and do not imply any limitation on the scope of this disclosure. The embodiments described herein can be implemented in various ways other than those described below.

[0011] In the following description and claims, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

[0012] References to "an embodiment," "an embodiment," "an example embodiment," etc., in this disclosure indicate that the described embodiment may include a particular feature, structure, or characteristic, but not every embodiment is required to include that particular feature, structure, or characteristic. Furthermore, such phrases do not necessarily refer to the same embodiment. Additionally, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is believed that in conjunction with other embodiments, whether explicitly described or not, influencing such feature, structure, or characteristic is within the knowledge of those skilled in the art.

[0013] It should be understood that although terms such as "first," "second," etc., preceding nouns may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another, and they do not restrict the order of nouns. For example, without departing from the scope of the example embodiments, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element. As used herein, the term "and / or" includes any and all combinations of one or more of the listed terms.

[0014] As used herein, “at least one of the following: a list of two or more elements” and “at least one of the following: a list of two or more elements” and similar wording (where the list of two or more elements is connected by “and” or “or”) means at least any one of the elements, or at least any two or more of the elements, or at least all of the elements.

[0015] As used herein, unless explicitly stated otherwise, the execution step “in response to A” does not indicate that the step is performed immediately after “A” occurs, and may include one or more intermediate steps.

[0016] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments. As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that, when used herein, the terms “comprises,” “comprising,” “has,” “having,” “includes,” and / or “including” specify the presence of the stated features, elements, and / or components, but do not exclude the presence or addition of one or more other features, elements, components, and / or combinations thereof.

[0017] As used in this application, the term "circuit" may refer to one or more of the following: (a) Hardware circuit implementation only (such as implementation in analog and / or digital circuits only) and (b) A combination of hardware circuitry and software, such as (if applicable): (i) A combination of analog and / or digital hardware circuitry with software / firmware, and (ii) Any part of a hardware processor having software (including digital signal processors(s)), software, and memory, which work together to enable a device such as a mobile phone or server to perform various functions, and (c) (Multiple) hardware circuits and / or (multiple) processors, such as (multiple) microprocessors or a portion thereof, which require software (e.g., firmware) to operate, but may not exist when the software is not required to operate.

[0018] This definition of "circuit" applies to all uses of the term in this application (including in any claim). As another example, as used herein, the term "circuit" also encompasses only hardware circuitry or a processor (or multiple processors) or a portion thereof and its accompanying software and / or firmware implementation. The term "circuit" also encompasses, for example and if applicable to a particular claim element, baseband integrated circuits or processor integrated circuits for mobile devices or similar integrated circuits in servers, cellular network devices, or other computing or network devices.

[0019] As used herein, the term "communication network" refers to a network that conforms to any suitable communication standard, such as New Radio (NR), Long Term Evolution (LTE), LTE-A Advanced (LTE-A), Wideband Code Division Multiple Access (WCDMA), High-Speed ​​Packet Access (HSPA), Narrowband Internet of Things (NB-IoT), etc. Furthermore, communication between terminal devices and network devices in a communication network can be performed according to any suitable generation of communication protocol, including but not limited to first-generation (1G), second-generation (2G), 2.5G, 2.75G, third-generation (3G), fourth-generation (4G), 4.5G, fifth-generation (5G), sixth-generation (6G) communication protocols and / or any other currently known or future-developed protocols. Embodiments of this disclosure can be applied to a variety of communication systems. Given the rapid development of communications, there will naturally be future types of communication technologies and systems that embody the future types of this disclosure. The scope of this disclosure should not be construed as limited to the aforementioned systems.

[0020] As used herein, the term "network device" refers to a node in a communications network through which terminal devices access the network and receive services. Depending on the terminology and technology applied, a network device can refer to a base station (BS) or access point (AP), such as a Node B (NodeB or NB), an evolved Node B (eNodeB or eNB), an NR NB (also known as a gNB), a Remote Radio Unit (RRU), a Radio Header (RH), a Remote Radio Header End (RRH), a relay, an Integrated Access and Backhaul (IAB) node, a low-power node (such as a femto, pico, or non-terrestrial network (NTN)) or non-terrestrial network equipment (such as satellite network equipment), low Earth orbit (LEO) satellites and geostationary Earth orbit (GEO) satellites, spacecraft network equipment, etc. In some example implementations, the Radio Access Network (RAN) split architecture includes a centralized unit (CU) and a distributed unit (DU) at the IAB donor node. The IAB node includes a mobile terminal (IAB-MT) portion that behaves as a UE toward the parent node, and the DU portion of the IAB node behaves as a base station toward the next-hop IAB node.

[0021] The term "terminal device" refers to any terminal device capable of wireless communication. By way of example and not limitation, a terminal device may also be referred to as a communication device, user equipment (UE), subscriber station (SS), portable subscriber station, mobile station (MS), or access terminal (AT). Terminal devices can include, but are not limited to, mobile phones, cellular phones, smartphones, Voice over IP (VoIP) phones, wireless local loop phones, tablets, wearable terminal devices, personal digital assistants (PDAs), portable computers, desktop computers, image capture terminal devices (such as digital cameras), gaming terminal devices, music storage and playback devices, in-vehicle wireless terminal devices, wireless endpoints, mobile stations, laptop embedded devices (LEEs), laptop devices (LMEs), USB dongles, smart devices, wireless customer premises equipment (CPEs), Internet of Things (IoT) devices, watches or other wearable devices, head-mounted displays (HMDs), vehicles, drones, medical devices and applications (e.g., remote surgery), industrial devices and applications (e.g., robots and / or other wireless devices operating in industrial and / or automated processing chain environments), consumer electronics devices, devices operating on commercial and / or industrial wireless networks, etc. The terminal equipment may also correspond to the mobile termination (MT) portion of an IAB node (e.g., a relay node). In the following description, the terms "terminal equipment," "communication equipment," "terminal," "user equipment," and "UE" are used interchangeably.

[0022] As used herein, the terms “resource,” “transmission resource,” “resource block,” “physical resource block” (PRB), “uplink resource,” or “downlink resource” can refer to any resource used to perform communication, such as communication between a terminal device and a network device, including resources in the time domain, frequency domain, spatial domain, code domain, or any other combination of time, frequency, spatial, and / or code domain resources used to achieve communication. In the following, unless explicitly stated otherwise, resources in the frequency and time domains will be used as examples of transmission resources used to describe some exemplary embodiments of this disclosure. Note that the exemplary embodiments of this disclosure are equally applicable to other resources in other domains.

[0023] As mentioned above, system performance in wireless environments remains suboptimal. Challenges in system performance optimization can be considered from the following aspects: 1) The wireless environment (e.g., the number of UEs, UE distribution and radio conditions, interference) is always changing, but almost all of these parameters of the network environment are fixed, even if better settings exist in different scenarios.

[0024] 2) The wireless environment of each site and cell may also differ, and there is currently no method for specific optimization of each site or cell.

[0025] 3) Due to time and resource constraints, a large number of parameters and parameter combinations need to be tested and verified on-site, but only a small portion can be verified before delivery.

[0026] It is difficult to perform joint dynamic optimization of these parameters. Although parameter configuration optimization can be completed before the actual network deployment, these configurations are not robust enough and cannot maintain good performance under various wireless scenarios (service load, user distribution, interference level, channel quality, etc.).

[0027] Hundreds of parameter optimization-related issues arise in the field each year. Due to redundant log information and the large number of parameters to be optimized, handling these issues requires significant resources. Furthermore, it is impossible to manually adjust parameters jointly to determine the optimal parameter configuration for a specific network.

[0028] 6G wireless networks are expected to evolve towards self-sustaining networks. Machine learning plays a crucial role in maintaining high network performance in dynamically changing environments. Therefore, this disclosure proposes a gNB parameter optimization scheme based on reinforcement learning (RL). Leveraging the inherent theoretical framework of parameter closed-loop control, this reinforcement learning-based gNB parameter optimization method can effectively improve network performance.

[0029] Figure 1An example communication environment 100 in which exemplary embodiments of the present disclosure may be implemented is illustrated. Communication environment 100 includes a first device 110 that can operate as a network device (e.g., a gNB) and second devices 120-1, 120-2, and 120-3, which may be collectively referred to below as second devices 120. In some scenarios, second device 120 may be a terminal device (e.g., a UE, such as second devices 120-1 and 120-2). In some other scenarios, the second device may also be a network node (e.g., access point 120-3).

[0030] The first device 110 has the capability to perform RL operations. For example, an RL model can be deployed at the first device 110. Distributed RL-based performance optimization can be performed for each cell (e.g., cell 101) managed by the first device 110. It is also possible to perform RL-based performance optimization for all cells within a defined network coverage area.

[0031] In the following text, for illustrative purposes, the RL-based performance optimization performed by the first device 110 and / or by the RL model deployed at the first device 110 may be referred to as a distributed RL operation for optimizing the performance of the network environment within the cell.

[0032] It should be understood that Figure 1 The number of network devices and terminal devices shown is given for illustrative purposes and does not imply any limitation. The communication environment 100 may include any suitable number of network devices and terminal devices.

[0033] Communication in communication environment 100 can be implemented according to any suitable communication protocol, including but not limited to cellular communication protocols such as first-generation (1G), second-generation (2G), third-generation (3G), fourth-generation (4G), fifth-generation (5G), and sixth-generation (6G), wireless local network communication protocols such as IEEE 802.11, and / or any other currently known or future-developed protocols. Furthermore, communication can utilize any suitable wireless communication technology, including but not limited to: Code Division Multiple Access (CDMA), Frequency Division Multiple Access (FDMA), Time Division Multiple Access (TDMA), Frequency Division Duplex (FDD), Time Division Duplex (TDD), Multiple-Input Multiple-Output (MIMO), Orthogonal Frequency Division Multiple Access (OFDM), Discrete Fourier Transform Extended OFDM (DFT-s-OFDM), and / or any other currently known or future-developed technologies.

[0034] The exemplary embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0035] Now for reference Figure 2The diagram 200 illustrates a process for RL-based performance optimization according to some example implementations of this disclosure.

[0036] like Figure 2 As shown, the RL agent / model 210 can be deployed in, for example, Figure 1 The first device 110 is shown. The RL agent / model 210 can perform RL-based performance optimization on the network environment 220.

[0037] As an example, the RL agent / model 210 can be deployed at RAN devices (such as eNB or gNB). It should be understood that the RL agent / model 210 can also be deployed at network nodes / components / entities associated with network management.

[0038] Network environment 220 may refer to the cell where the first device 110 and at least one second device 120 are located. In the following, for illustrative purposes, distributed performance optimization of a cell managed by the first device 110 (e.g., a gNB) will be described. It should be understood that the solutions proposed in this disclosure can also be used for centralized performance optimization in network environments comprising more than one cell.

[0039] In the following text, RL can refer to the learning process of mapping situations (states) to actions in order to maximize numerical reward signals (rewards). RL agents can refer to software modules within a reinforcement learning (RLS) system that sense states and take actions to maximize rewards.

[0040] In this RL operation, the first device 110 or the RL agent / model 210 deployed at the first device 110 can acquire the current state information of the network environment 220. The current state information can be represented as s(t) below.

[0041] The current status information may refer to the state of the network environment 220 caused by the behavior of at least one second device 120 (e.g., UE behavior 222) and the configuration / functions provided by the first device 110 (e.g., gNB configuration / function 224).

[0042] In some scenarios, the state of network environment 220 can be obtained based on reports from UEs within network environment 220 to the gNB. In other scenarios, the state of network environment 220 can be obtained by the gNB based on corresponding measurements.

[0043] Typically, the state of network environment 220 can be associated with the state of everything outside of RL agent / model 210, such as UE behavior, gNB configuration, and interference associated with the UE and / or gNB.

[0044] For example, Table 1 shows four types of components used to describe UE behavior: capacity, radio conditions, cell load, and UE distribution.

[0045] Table 1

[0046] Regarding UE behavior, the state of the network environment 220 obtained by the RL agent / model 210 may include capacity information, such as the number of UEs included in the network environment 220, for example, the average number of active UEs connected to the first device 110. The state of the network environment 220 may also include the radio conditions of at least a portion of the at least one UE, which can be reflected by the average channel quality indication (CQI) associated with at least one UE connected to the first device 110.

[0047] In addition, the state of the network environment 220 may also include the cell load of the first device 110, which can be indicated by the average physical resource block (PRB) usage of the UE's services.

[0048] The state of network environment 220 can also involve UE distribution, such as Figure 3A As shown, it depends on angle 302 (measured by the direction of arrival). The UE distribution mentioned below may refer to the average DoA distance associated with the first device 110 and the average DoA associated with the first device 110.

[0049] The average DoA associated with the first device 110 can represent the average of the DOAs associated with the first device and at least one second device in the cell. The average DoA distance associated with the first device 110 represents the average DoA distance between at least one second device 120 and the average DOA associated with the first device 110.

[0050] There is a DoA distance between the second devices. Figure 3B The incremental DoA312 between the second devices 120-1 and 120-2 is shown. The DoA distance between the second devices can be calculated as follows: (1) The average DoA of the first device 110 (which can be associated with, for example, the DoA angle 322 of the second device 120-1 and the DoA angle 324 of the second device 120-2, such as...) Figure 3C (As shown) can be calculated as follows: (2) As described, the average DoA distance associated with the first device 110 and the average DoA associated with the first device 110 can reflect the UE distribution in the cell associated with the first device 110.

[0051] Now return to the reference Figure 2 In addition to the state of the network environment 220 (i.e., s(t)), the RL agent / model 210 can also obtain a reward (denoted as r(t)), which can represent the performance of the network environment 220 in the current state. Specifically, the reward r(t) can indicate the level of system performance measured during time t (i.e., the super critical performance indicator (KPI) score).

[0052] Typically, rewards can be used in RL operations to indicate the outcome of a combination of states and actions. The super KPI score can be considered a value that indicates the overall performance level of the system, normalizing and unifying several KPIs into a single score.

[0053] In this context, the super KPI score can be used directly as a reward. The super KPI score can be a uniform score (100 points) and can include different types of KPIs: throughput, HO SR, call drop rate, efficiency, and even customer-specific KPIs. The super KPI score can also be determined using customized weights and normalized to a score with a customized non-linear curve. An example of a super KPI score definition is shown below: Table 2

[0054] The state prediction operation can be used to predict the future state of network environment 220 by using the current state information of network environment 220 (e.g., denoted as s(t+1)), which can be considered as an auxiliary operation of the RL gNB optimization method.

[0055] For example, state prediction module 212 can perform state prediction. For example, state prediction module 212 can be based on recurrent neural network (RNN). Future states can be predicted because the state changes of network environment 220 are mainly determined by UE behavior rather than gNB, and UE behavior can be predicted to some extent.

[0056] Figure 4 An example of a state prediction module according to an embodiment of this disclosure is shown. Figure 4 The example illustrates a machine learning model 400 based on a gated recurrent unit (GRU), which can be considered a type of RNN model and can be used to predict state sequences. For example, the GRU-based machine learning model 400 can be implemented at the state prediction module 212. It should be understood that the machine learning model used for state prediction is not limited to an RNN model. Any other suitable machine learning model capable of performing state prediction may also be included within the scope of this disclosure.

[0057] The machine learning model 400 may include an input layer, one or more hidden layers, and an output layer. The current state information of the network environment 220 can be used as the input to the machine learning model 400. The output of the machine learning model 400 may be a predicted future state of the network environment 220.

[0058] The machine learning model 400 can be trained using a historical dataset associated with network environment states, such as a dataset of historical data associated with network environment states over a day, a week, or a predefined duration. Since there is always some regularity in UE behavior within each cell, the state can be predicted to some extent. The machine learning model 400 can be trained online or offline.

[0059] Based on the predicted future state of network environment 220, policy model 216 in RL agent / model 210 can select corresponding actions. The actions used below can be referred to as the set of configuration parameters of the network environment, i.e., a set of parameter settings for the network environment. First device 110 can take actions (e.g., by changing parameter settings) to affect the performance of the network environment. For example, parameters sensitive to the network environment can be selected as actions, such as MU-MIMO pairing, power control, link adaptation, power saving, load balancing, admission control, carrier aggregation, and handover.

[0060] The RL agent / model 210 may also include a query model 214, which may also be referred to below as a Q-table. Typically, a Q-table can represent the association between a set of rewards and multiple sets of configuration parameters (e.g., actions) used in the corresponding reference network environment state. The Q-table records all rewards by state and action. Values ​​in the Q-table can be represented as r(s, a). Since the state space and action space are limited to a reasonable size, Q-learning is chosen as the RL algorithm in this disclosure. However, other RL algorithms are also applicable to this disclosure.

[0061] The following is an example of a Q-table format. In a Q-table, rows represent states, columns represent actions, and rewards are recorded in cells—represented as r(s, a).

[0062] Table 3

[0063] The Q table can be updated with a new r(s, a) value at the end of each predetermined time interval (e.g., time slot t). The value r(s, a) can include the count of rewards and the average of rewards, which serves as a reference for data reliability.

[0064] For example, as mentioned above, if the current state information s(t) and the reward r(t) representing the performance of the network environment in the current state are obtained, the values ​​corresponding to the obtained data can be updated in the Q table.

[0065] The RL operations of the RL agent / model 210 can include an exploration phase and an exploitation phase. RL exploration helps gather more information in the unknown state / action space, while RL exploitation involves using the current learned policy to select the action with the best reward. RL requires a policy to balance exploration and exploitation. The role of the policy model 216 at the RL agent / model 210 determines when and how to conduct exploration in each time slot.

[0066] During the utilization phase, actions can be selected based on the predicted future state s(t+1). For example, policy model 216 can query multiple rewards corresponding to the predicted future state s(t+1) from the Q table. That is, if the predicted future state s(t+1) (such as s0, s… or sN as shown in Table 3) is determined, multiple rewards in the row corresponding to the future state s(t+1) can be read from the Q table. Then, an action corresponding to the highest / best reward among the multiple rewards can be selected. This action can include a set of target configuration parameters of the network environment under the future state s(t+1), which can be represented as a(t+1). In this case, action a(t+1) can be provided to the network environment 220 to achieve performance optimization of the network environment 220.

[0067] During the exploration phase, the exploration policy at policy model 216 can select actions based on the predicted future state s(t+1). Exploration actions can also be provided to the network environment 220. The network environment 220 can be configured based on the exploration actions. Subsequently, the actual state information at time t+1 and the corresponding actual reward can be provided to the RL agent / model 210, for example, stored and updated by a Q-table, for the RL operations of the RL agent / model 210.

[0068] It should be understood that the exploration phase can be performed at predetermined time intervals (such as certain time slots per day). Furthermore, some exploration criteria are listed. For example, the number of exploration time slots with negative rewards per day may not be allowed to exceed a predefined threshold `maxNumNegT`. Additionally, the number of exploration time slots per day may not be allowed to exceed a predefined parameter `maxNumExplorT`. The baseline for the exploration phase can be the first column of the Q table (i.e., the default parameter setting). The negative reward used in this paper can be referred to as the super KPI score that is less than the baseline (set at the client's request). The exploration criteria ensure that the performance impact meets the client's requirements before the algorithm converges.

[0069] This disclosure proposes cell-level Q-table sharing to accelerate learning. Cells can share Q-tables if the primary configuration is identical across all cells. Specifically, Each cell can periodically send Q(S, A) results to other cells with the same primary configuration. For each cell, all Q(S, A) results are collected, categorized, and summarized. The result of Q(S,A) comes from itself: If the absolute value of the difference between a previous Q value in the Q table and the reported Q value is less than a predefined threshold, it is merged into its own Q table.

[0070] If the absolute value of the difference between the previous Q value in the Q table and the reported Q value is greater than a predefined threshold, then the Q value is replaced with the latest reported value.

[0071] The Q(S, A) results are from other cells: If a Q-table with the same state and action is empty, it is merged into its own Q-table.

[0072] Furthermore, site-level Q-table sharing is also possible if the master site configuration is identical across all sites.

[0073] Based on the solution disclosed herein, performance improvements in dynamic parameter optimization can be achieved through the use of RL-based optimization operations. Compared to manual optimization, the solution proposed in this disclosure can reduce system optimization costs, time-frequency resource waste, and signaling overhead.

[0074] Figure 5 A flowchart of an example method 500 implemented at a first device according to some example embodiments of the present disclosure is shown. For discussion purposes, [the following will be discussed]. Figure 1 Method 500 is described by the angle of the first device 110 in the middle.

[0075] At box 510, the first device acquires current state information of the network environment and a reward, the reward being associated with the reinforcement learning operation and relating to the performance of the network environment; In box 520, the first device determines a predicted future state of the network environment based on current state information; and At box 530, the first device generates a set of target configuration parameters for the network environment by using reinforcement learning operations, based at least on the predicted future state of the network environment; and In box 540, the first device uses a target set of configuration parameters to configure the network environment.

[0076] In some example implementations, the current status information includes at least one of the following: capacity information associated with at least one second device connected to the first device in the network environment; radio conditions of at least a portion of the at least one second device; load information associated with the services of the first device; and / or distribution information of at least a portion of the at least one second device in the network environment.

[0077] In some example implementations, the distribution information includes at least one of the following: the average DoA associated with the first device, which represents the average of the DOAs associated with the first device and at least one second device in the cell; and / or the average DoA distance associated with the first device, which represents the average DoA distance between at least one second device and the average DOA associated with the first device.

[0078] In some example implementations, a predicted future state of the network environment is determined by using an RNN-based machine learning model, wherein current state information is used as input to the RNN-based machine learning model, and the output of the RNN-based machine learning model is the predicted future state.

[0079] In some example implementations, method 500 further includes: during reinforcement learning operations, obtaining associations between a set of reference rewards and multiple sets of reference configuration parameters used in the corresponding reference network environment state; during the exploitation phase of reinforcement learning operations, obtaining a set of candidate rewards corresponding to the predicted future state based on the predicted future state and associations; selecting the best reward from the set of candidate rewards; and determining a set of reference configuration parameters corresponding to the best reward and the predicted future state from the multiple sets of reference configuration parameters as the target configuration parameter set for the network environment.

[0080] In some example implementations, method 500 further includes: determining a set of exploration configuration parameters for a predicted future state during the exploration phase of a reinforcement learning operation; obtaining information on the corresponding reward achieved by using the set of exploration configuration parameters in another network environment state; and optimizing the reinforcement learning operation based on that information.

[0081] In some example implementations, the first device includes a network device capable of performing reinforcement learning operations, and wherein the reinforcement learning operations are performed for a network environment having cells managed by the first device.

[0082] In some example implementations, the second device includes a terminal device and / or a network device.

[0083] In some example implementations, a first device capable of performing any method 500 (e.g., Figure 1 The first device 110 may include components for performing the corresponding operation of method 500. The device may be implemented in any suitable form. For example, the device may be implemented in a circuit or software module. The first device may be implemented as or included in... Figure 1 In the first device 110.

[0084] In some example implementations, the first apparatus includes: components for acquiring current state information and a reward of a network environment, the reward being associated with a reinforcement learning operation and relating to the performance of the network environment; components for determining a predicted future state of the network environment based on the current state information; components for generating a target set of configuration parameters for the network environment by using a reinforcement learning operation, at least based on the predicted future state of the network environment; and components for configuring the network environment using the target set of configuration parameters.

[0085] In some example implementations, the current status information includes at least one of the following: a component for capacity information associated with at least one second device connected to the first device in a network environment; a component for radio conditions of at least a portion of the at least one second device; a component for load information associated with services of the first device; and / or a component for distribution information of at least a portion of the at least one second device in the network environment.

[0086] In some example implementations, the distribution information includes at least one of the following: the average DoA associated with the first device, which represents the average of the DOAs associated with the first device and at least one second device in the cell; and / or the average DoA distance associated with the first device, which represents the average DoA distance between at least one second device and the average DOA associated with the first device.

[0087] In some example implementations, a predicted future state of the network environment is determined by using an RNN-based machine learning model, wherein current state information is used as input to the RNN-based machine learning model, and the output of the RNN-based machine learning model is the predicted future state.

[0088] In some example implementations, the first apparatus further includes: components for acquiring an association between a set of reference rewards and a plurality of sets of reference configuration parameters used in a corresponding reference network environment state during reinforcement learning operations; components for acquiring a set of candidate rewards corresponding to a predicted future state based on a predicted future state and association during the exploitation phase of the reinforcement learning operations; components for selecting the best reward from the set of candidate rewards; and components for determining, from the plurality of sets of reference configuration parameters, the set of reference configuration parameters corresponding to the best reward and the predicted future state as the target configuration parameter set for the network environment.

[0089] In some example implementations, the first apparatus further includes: components for determining a set of exploration configuration parameters for a predicted future state during the exploration phase of a reinforcement learning operation; components for acquiring information about a corresponding reward achieved by using the set of exploration configuration parameters in another network environment state; and components for optimizing the reinforcement learning operation based on the information.

[0090] In some example implementations, the first device includes a network device capable of performing reinforcement learning operations, and wherein the reinforcement learning operations are performed for a network environment having cells managed by the first device.

[0091] In some example implementations, the second device includes a terminal device and / or a network device.

[0092] In some example embodiments, the first device further includes components for performing other operations in some example embodiments of method 500 or the first device 110. In some example embodiments, the components include: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the first device to perform.

[0093] Figure 6 This is a simplified block diagram of device 600 suitable for implementing example embodiments of the present disclosure. Device 600 can be provided to implement a communication device, such as... Figure 1 The first device 110 is shown. As shown, the device 600 includes one or more processors 610, one or more memories 620 coupled to the processors 610, and one or more communication modules 640 coupled to the processors 610.

[0094] Communication module 640 is used for bidirectional communication. Communication module 640 has one or more communication interfaces to facilitate communication with one or more other modules or devices. The communication interface can represent any interface required for communication with other network elements. In some example embodiments, communication module 640 may include at least one antenna.

[0095] As a non-limiting example, processor 610 can be any type suitable for a local technology network and can include one or more of the following: general-purpose computer, special-purpose computer, microprocessor, digital signal processor (DSP), and processor based on a multi-core processor architecture. Device 600 can have multiple processors, such as application-specific integrated circuit chips that are time-dependent on a clock of a synchronous main processor.

[0096] Memory 620 may include one or more non-volatile memories and one or more volatile memories. Examples of non-volatile memories include, but are not limited to, read-only memory (ROM) 624, electrically programmable read-only memory (EPROM), flash memory, hard disk, optical disc (CD), digital video disc (DVD), optical disc, laser disc, and other magnetic and / or optical storage. Examples of volatile memories include, but are not limited to, random access memory (RAM) 622 and other volatile memories that will not persist for extended periods of power-off duration.

[0097] Computer program 630 includes computer-executable instructions that are executed by an associated processor 610. The instructions of program 630 may include instructions for performing operations / actions of some example embodiments of this disclosure. Program 630 may be stored in memory (e.g., ROM 624). Processor 610 can perform any suitable actions and processes by loading program 630 into RAM 622.

[0098] The exemplary implementation of this disclosure can be achieved through program 630, enabling device 600 to perform as described in the reference. Figure 2 This applies to any process of the present disclosure as discussed in Figure DDDD 5. Example embodiments of the present disclosure may also be implemented in hardware or by a combination of software and hardware.

[0099] In some example embodiments, program 630 may be tangibly contained in a computer-readable medium, which may be included in device 600 (such as memory 620) or other storage device accessible by device 600. Device 600 may load program 630 from the computer-readable medium into RAM 622 for execution. In some example embodiments, the computer-readable medium may include any type of non-transitory storage medium, such as ROM, EPROM, flash memory, hard disk, CD, DVD, etc. As used herein, the term "non-transitory" is a limitation on the medium itself (i.e., tangible, not tactile), rather than a limitation on the persistence of data storage (e.g., RAM and ROM).

[0100] Figure 7 An example of a computer-readable medium 700 is shown, which may be in the form of a CD, DVD, or other optical storage disc. A program 630 is stored on the computer-readable medium 700.

[0101] Generally, the various embodiments of this disclosure can be implemented in hardware or special-purpose circuitry, software, logic, or any combination thereof. Some aspects can be implemented in hardware, while others can be implemented in firmware or software that can be executed by a controller, microprocessor, or other computing device. Although various aspects of the embodiments of this disclosure are illustrated and described as block diagrams, flowcharts, or using some other graphical representation, it should be understood that, as a non-limiting example, the blocks, apparatuses, systems, techniques, or methods described herein can be implemented in hardware, software, firmware, special-purpose circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0102] Some example embodiments of this disclosure also provide at least one computer program product tangibly stored on a computer-readable medium, such as a non-transitory computer-readable medium. The computer program product includes computer-executable instructions that execute in a device on a target physical or virtual processor, such as those included in a program module, to perform any of the methods described above. Typically, a program module includes routines, programs, libraries, objects, classes, components, data structures, etc., that perform a specific task or implement a specific abstract data type. In various embodiments, the functionality of a program module can be combined or split among program modules as needed. The machine-executable instructions for a program module can execute within a local or distributed device. In a distributed device, the program module can reside in both local and remote storage media.

[0103] Program code used to perform the methods of this disclosure may be written in any combination of one or more programming languages. The program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a stand-alone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0104] In the context of this disclosure, computer program code or related data may be carried by any suitable carrier to enable a device, apparatus, or processor to perform the various processes and operations described above. Examples of carriers include signals, computer-readable media, etc.

[0105] Computer-readable media can be computer-readable signal media or computer-readable storage media. Computer-readable media can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any suitable combination thereof. More specific examples of computer-readable storage media will include electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable optical disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0106] Furthermore, although the operations are described in a specific order, this should not be construed as requiring that such operations be performed in the specific order shown or sequentially, or that all the operations shown be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the discussion above, these details should not be construed as limiting the scope of this disclosure, but rather as descriptions of features that may be specific to a particular implementation. Unless explicitly stated otherwise, certain features described in the context of a single implementation may also be implemented in combination in a single implementation. Conversely, unless explicitly stated otherwise, the various features described in the context of a single implementation may also be implemented individually or in any suitable sub-combination in multiple implementations.

[0107] Although this disclosure has been described in language specific to structural features and / or methodological actions, it should be understood that the disclosure as defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are disclosed as exemplary forms for implementing the claims.

Claims

1. A first device, comprising: At least one processor; as well as At least one memory storing instructions that, when executed by the at least one processor, cause the first device to at least: Obtain the current state information and reward of the network environment, wherein the reward is associated with the reinforcement learning operation and with respect to the performance of the network environment; The predicted future state of the network environment is determined based on the current state information; as well as By using the reinforcement learning operation, a set of target configuration parameters for the network environment is generated, based at least on the predicted future state of the network environment. as well as The network environment is configured using the target set of configuration parameters.

2. The first device according to claim 1, wherein the current status information includes at least one of the following: Capacity information associated with at least one second device in the network environment that is connected to the first device; At least a portion of the radio conditions of the at least one second device Load information associated with the services of the first device, and / or Distribution information of at least a portion of the second devices in the network environment.

3. The first device according to claim 2, wherein the distribution information includes at least one of the following: The average DoA associated with the first device represents the average DOA associated with the first device and the at least one second device in the cell, and / or The average DoA distance associated with the first device represents the average DoA distance between the at least one second device and the average DoA associated with the first device.

4. The first apparatus according to any one of claims 1-3, wherein the predicted future state of the network environment is determined by using an RNN-based machine learning model, wherein the current state information is used as the input to the RNN-based machine learning model, and the output of the RNN-based machine learning model is the predicted future state.

5. The first device according to any one of claims 1-4, wherein the first device is further caused to: During the reinforcement learning operation, a set of reference rewards is obtained and associated with multiple sets of reference configuration parameters used in the corresponding reference network environment state; During the utilization phase of the reinforcement learning operation, a set of candidate rewards corresponding to the predicted future state is obtained based on the predicted future state and the association. Select the best reward from the set of candidate rewards; as well as A set of reference configuration parameters corresponding to the optimal reward and the predicted future state is determined from the plurality of reference configuration parameter sets, and is used as the target configuration parameter set of the network environment.

6. The first device according to any one of claims 1-4, wherein the first device is further caused to: In the exploration phase of the reinforcement learning operation, a set of exploration configuration parameters is determined for the predicted future state; and Obtain information about the corresponding reward achieved by using the aforementioned set of exploration configuration parameters in another network environment state; and The reinforcement learning operation is optimized based on the information provided.

7. The first apparatus according to any one of claims 1-6, wherein the first apparatus includes a network device capable of performing the reinforcement learning operation, and wherein the reinforcement learning operation is performed for a network environment having cells managed by the first apparatus.

8. The first device according to claim 2 or 3, wherein the second device includes a terminal device and / or a network device.

9. A method comprising: Obtain the current state information and reward of the network environment, wherein the reward is associated with the reinforcement learning operation and with respect to the performance of the network environment; The predicted future state of the network environment is determined based on the current state information; as well as By using the reinforcement learning operation, a set of target configuration parameters for the network environment is generated, based at least on the predicted future state of the network environment. as well as The network environment is configured using the target set of configuration parameters.

10. The method of claim 9, wherein the current status information includes at least one of the following: Capacity information associated with at least one second device connected to the first device in the network environment; At least a portion of the radio conditions of the at least one second device Load information associated with the services of the first device, and / or Distribution information of at least a portion of the second devices in the network environment.

11. The method of claim 10, wherein the distribution information comprises at least one of the following: The average DoA associated with the first device represents the average DOA associated with the first device and the at least one second device in the cell, and / or The average DoA distance associated with the first device represents the average DoA distance between the at least one second device and the average DoA associated with the first device.

12. The method according to any one of claims 9-11, wherein the predicted future state of the network environment is determined by using an RNN-based machine learning model, wherein, The current state information is used as input to the RNN-based machine learning model, and the output of the RNN-based machine learning model is the predicted future state.

13. The method according to any one of claims 9-12, further comprising: During the reinforcement learning operation, a set of reference rewards is obtained and associated with multiple sets of reference configuration parameters used in the corresponding reference network environment state; During the utilization phase of the reinforcement learning operation, a set of candidate rewards corresponding to the predicted future state is obtained based on the predicted future state and the association. Select the best reward from the set of candidate rewards; as well as A set of reference configuration parameters corresponding to the optimal reward and the predicted future state is determined from the plurality of reference configuration parameter sets, and is used as the target configuration parameter set of the network environment.

14. The method according to any one of claims 9-12, further comprising: During the exploration phase of the reinforcement learning operation, a set of exploration configuration parameters is determined for the predicted future state; as well as Obtain information about the corresponding reward achieved by using the set of exploration configuration parameters in another network environment state; as well as The reinforcement learning operation is optimized based on the information provided.

15. The method according to any one of claims 9-14, wherein the first apparatus comprises a network device capable of performing the reinforcement learning operation, and wherein, The reinforcement learning operation is performed on the network environment having cells managed by the first device.

16. The method according to claim 10 or 11, wherein the second device comprises a terminal device and / or a network device.

17. A first device, comprising: Components for acquiring current state information of the network environment and rewards, the rewards being associated with reinforcement learning operations and relating to the performance of the network environment; A component for determining the predicted future state of the network environment based on the current state information; as well as A component for generating a target configuration parameter set for the network environment by using the reinforcement learning operation, at least based on the predicted future state of the network environment; as well as A component used to configure the network environment using the target set of configuration parameters.

18. A computer-readable medium comprising instructions stored thereon for causing a device to perform at least the method of any one of claims 9-16.

Images