Method, apparatus, and computer readable medium for managing system operating in telecommunications environment

Abstract

A computer-implemented method is provided for managing a system operating in a telecommunications environment. Managing the system includes causing the system to perform an action, the action being one of a set of available actions. The method includes analyzing data related to a state of the telecommunications environment using a rule-based algorithm to determine a recommended first action from a set of available actions; and removing a second action confronting the recommended first action from the set of available actions to generate a reduced set of available actions. The method further includes analyzing data related to a state of the telecommunications environment using a reinforcement learning algorithm to select a third action from the reduced set of available actions; and causing the system to perform a third action.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to methods, apparatus, and computer-readable media for managing a system operating in a telecommunications environment, and in particular to methods for causing a system to implement an action from a set of available actions. Background Art

[0002] A radio access network (RAN) may include hundreds of thousands of cells. Each cell has many parameters that can be tuned to optimize network performance within the cell (e.g., cell coverage, cell capacity, etc.). A cell can monitor its network performance via key performance indicators (KPIs) observed within the cell (e.g., cell throughput, edge cell signal strength, etc.). However, network operators are typically interested in global KPIs (e.g., average throughput over multiple or all cells of the network, etc.) so that they can ensure that most network users experience adequate network performance. Therefore, network configuration is often intended to optimize global KPIs rather than KPIs for individual cells. This can be achieved by having the RAN act as a self-organizing network (SON).

[0003] The performance of a cell (and therefore its KPI) can be improved by adjusting cell parameters. One method of adjusting cell parameters involves cell shaping. Cell shaping uses beamforming techniques to shape the overall coverage area of ​​a cell. For example, remote electrical tilt (RET) and / or digital tilt can define the tilt of the antenna of a base station serving a cell. Changes to the antenna tilt can be performed remotely. By modifying the antenna tilt of a base station serving a particular cell, the downlink (DL) signal to interference plus noise ratio (SINR) for that cell can be changed. However, the SINR of surrounding cells may also be affected as a result.

[0004] Figure 1 106a, 106b) may affect each other due to their overlapping coverage areas. The resulting signal interference may affect the network performance experienced by the first and second user equipment (UE) 104a, 104b served by each cell. By adjusting the downtilt angle θ of one or more antennas of each RAN node 102a, 102b, the shape of each cell 106a, 106b can be tuned so that the combined network performance of the two cells is optimized. This may involve improving network performance for the first UE 104a within the first cell 106a, but reducing network performance for the second UE 104b in the second cell 106b.

[0005] Another method of adjusting cell parameters involves adjusting the power of signals sent within the cell. For example, the P0 nominal physical uplink shared channel (PUSCH) defines the target power per resource block (RB) that the cell expects in uplink (UL) communications from the UE to the base station. By increasing this target power in a particular cell, the UL SINR in the cell can be increased (due to an increase in signal strength). However, as a result, the UL SINR in surrounding cells may decrease (due to an increase in signal interference).

[0006] Both parameter adjustments described above can be implemented in various ways. One implementation method uses a rule-based algorithm that is preconfigured with logic and / or one or more functions. The current state of a cell that can determine one or more KPIs of the cell can be input into the algorithm. The algorithm outputs a recommended action to be implemented by the base station serving the cell. The rule-based algorithm can be preconfigured so that the recommended action at least maintains (but ideally improves) the state of the cell.

[0007] Machine learning techniques such as reinforcement learning (RL) are increasingly being used to replace rule-based algorithms.

[0008] In more detail, RL is a decision-making framework in which an agent interacts with an environment by exploring the state of the environment and choosing actions to perform on the environment. The goal of choosing an action is to maximize the long-term reward of the action based on a reward signal.

[0009] More formally, the RL problem is defined as follows:

[0010] ●State space S: It is the set of all possible states that the environment can exist in,

[0011] ●Action space A: It is the set of all possible actions that can be performed on the environment,

[0012] ● Transition probability distribution P: It is the probability of transitioning from one state to another based on the actions chosen by the agent,

[0013] ●Reward distribution R: It incentivizes or penalizes specific state-action pairs.

[0014] The agent’s policy π defines the control policy implemented by the agent and is a mapping from states to a policy distribution over possible actions, which indicates the probability that each possible action is the most favorable given the current state. RL interaction proceeds as follows: At each time instant t, the agent finds the environment in state s t ∈S. The agent chooses action a t ~π(·|s t )∈A, receiving a random reward r t ~R(·|st , a t ), and the environment shifts to a new state s t+1 ~P(·|s t , a t ). The goal of the agent is to find the optimal policy, that is, the policy that maximizes the expected cumulative reward over a predefined time period, also known as the policy value function

[0015] When performing the dynamic optimization process discussed above in an unknown environment (with respect to transition and reward probabilities), the RL agent needs to try or explore different state-action combinations with sufficient frequency to be able to make accurate predictions about the rewards and transition probabilities for each state-action pair. It is therefore necessary for the agent to repeatedly choose suboptimal actions that conflict with its goal of maximizing cumulative reward in order to fully explore the state-action space. At each time step, the agent must decide whether to prioritize further information collection (exploration) or to take the best action (exploitation) considering current knowledge. Exploration can create opportunities by discovering higher rewards based on previously untried actions. However, exploration also carries the risk that previously unexplored decisions will not provide increased rewards, but may have a negative impact on the environment. This negative impact may be only short-term, or it may persist, for example, if the explored action puts the environment in an undesirable state from which it cannot recover.

[0016] RL solutions have been shown to lead to reasonable performance gains in real-world deployments (e.g., AI: enhancing customer experience in a complex 5G world, Ericsson Mobility Report 2021). Summary of the invention

[0017] There are currently certain challenges. The performance improvements achieved using rule-based algorithms are limited; the logic and / or one or more functions employed by the rule-based algorithm are not iteratively trained or optimized, and therefore the performance of the rule-based algorithm is not expected to improve over time. The rule-based algorithm will simply output an action according to its pre-configured set of rules for a given state of a cell.

[0018] Reinforcement learning can be applied to the problem of optimizing cell parameters. However, although the RL agent can be continuously trained to recommend the best action, the RL agent can be regarded as a black box model in the following sense: there is no perceptible reason as to why the RL agent outputs an estimated cumulative future reward for a certain action. In addition, exploring suboptimal mechanisms may lead to unacceptable performance degradation, risk, or violation of security regulations for cellular networks. This means that the reliability and safety of the actions recommended by the RL agent cannot be guaranteed. Therefore, it may be risky to implement RL agents in telecommunications networks because any action that negatively affects network performance will cause widespread performance problems for many users. In order to utilize RL agents, the associated risks of harmful actions being implemented should be reduced and / or minimized.

[0019] By combining rule-based algorithms with RL algorithms, existing domain knowledge can be used to prevent potentially harmful actions (recommended by the RL algorithm) from being implemented. In this way, RL agents can be employed in real-world network environments while helping to ensure a high level of security for the environment.

[0020] In one aspect, a computer-implemented method for managing a system operating in a telecommunications environment is provided. Managing the system includes causing the system to implement an action, the action being one of a set of available actions. The method includes: analyzing data related to a state of the telecommunications environment using a rule-based algorithm to determine a recommended first action from the set of available actions; and removing a second action that is antagonistic to the recommended first action from the set of available actions to generate a reduced set of available actions. The method also includes: analyzing data related to the state of the telecommunications environment using a reinforcement learning algorithm to select a third action from the reduced set of available actions; and causing the system to implement the third action.

[0021] Also provided are apparatus and computer-readable media for performing the methods set forth above. For example, a management node for managing a system operating in a telecommunications environment is provided. Managing the system includes causing the system to implement an action, the action being one of a set of available actions. The management node includes a processing circuit configured to: analyze data related to a state of the telecommunications environment using a rule-based algorithm to determine a recommended first action from the set of available actions; and remove a second action that is antagonistic to the recommended first action from the set of available actions to generate a reduced set of available actions. The processing circuit is further configured to: analyze data related to a state of the telecommunications environment using a reinforcement learning algorithm to select a third action from the reduced set of available actions; and cause the system to implement the third action.

[0022] Examples of the present disclosure provide a method that facilitates the use of RL agents for controlling a telecommunications environment without putting network performance at risk. Thus, network performance can be optimized while ensuring a level of security for the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to better understand examples of the present disclosure, and to more clearly show how examples may be implemented, reference will now be made, by way of example only, to the accompanying drawings, in which:

[0024] Figure 1 is a schematic diagram showing two adjacent network cells;

[0025] Figure 2 is a schematic diagram showing a communication network according to an embodiment of the present disclosure;

[0026] Figure 3 is a schematic diagram showing a management architecture and environment according to an embodiment of the present disclosure;

[0027] Figure 4 is a signaling diagram illustrating signaling between a SON agent, a DQN agent, and a management node according to an embodiment of the present disclosure;

[0028] Figure 5 is a flowchart of a method performed by a management node according to an embodiment of the present disclosure; and

[0029] Figure 6 is a schematic diagram of a management node according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] Some embodiments contemplated herein will now be described more fully with reference to the accompanying drawings. However, other embodiments are included within the scope of the subject matter disclosed herein. The disclosed subject matter should not be construed as being limited to the embodiments set forth herein; rather, these embodiments are provided by way of example to convey the scope of the subject matter to those skilled in the art.

[0031] Examples of the present disclosure present a method for safe reinforcement learning (SRL) and an architecture on which the method can be implemented, which changes the standard RL interaction cycle in order to improve the safety of the environment in which RL techniques are employed. Conceptually, the method can be conceived as masking the set of actions from which the RL agent can select actions to be implemented by a system operating in the environment. Masking the set of available actions involves removing any actions from the set of actions that directly oppose a "safe" action. An action can be considered "safe" in the following sense: it can be known with a certain degree of certainty to improve the state of the environment or to cause no change to the state of the environment. Such safe actions can be reliably determined using a rule-based algorithm pre-configured with existing domain knowledge.

[0032] Figure 2 2 is a schematic diagram illustrating a communication network 200 according to an embodiment of the present disclosure. In some embodiments, the communication network 200 may be a self-organizing network (SON), where one or more parameters associated with the network 200 are determined autonomously by entities of the network 200.

[0033] In the illustrated embodiment, the communication network 200 includes a radio access network (RAN) and a core network 202, the RAN including one or more RAN nodes 204, and the core network 202 including one or more core network nodes. The RAN includes the RAN node 204. As used herein, the term "RAN node" refers to a device that is capable of, configured, arranged and / or operable to communicate directly or indirectly with a wireless device or user equipment (UE) and / or other network nodes or devices in a telecommunications network. Examples of network nodes include, but are not limited to, access points (APs) (e.g., radio access points), base stations (BSs) (e.g., radio base stations, node Bs, evolved node Bs (eNBs), and new radio node Bs (gNBs)). The RAN node serves one or more UEs 206 in a serving cell 208. The RAN node 204 facilitates direct or indirect connections of UEs in its serving cell 208, such as by connecting the UE 206 to the core network 202 via one or more wireless connections.

[0034] The core network 202 includes one or more core network nodes constructed using hardware and software components. Example core network nodes include functions of one or more of a mobile switching center (MSC), a mobility management entity (MME), a home subscriber server (HSS), an access and mobility management function (AMF), a session management function (SMF), an authentication server function (AUSF), a subscription identifier de-hiding function (SIDF), a unified data management (UDM), a security edge protection proxy (SEPP), a network open function (NEF) and / or a user plane function (UPF).

[0035] According to an embodiment of the present disclosure, the core network 202 includes a management architecture 216 that enables secure use of reinforcement learning (RL) to adapt, control and / or manage one or more aspects of the network 200. For example, in one embodiment, the management architecture 216 is configured to determine an action to be implemented by the RAN node 204 (e.g., a change in the downward tilt of an antenna of the RAN node 204, or a change in the P0 nominal PUSCH) so that the shape or parameters of the serving cell 208 can be adjusted. This in turn allows the performance of the network 200 to be optimized.

[0036] The management architecture 216 may be implemented using any one or more of the core network nodes discussed above, or alternatively implemented in one or more servers attached to the core network 202. In further alternative embodiments, the management architecture 216 may be implemented in the RAN, such as in the RAN node 204 itself or in another RAN node. For example, the management architecture 216 may be implemented using a service management and orchestration (SMO) system or a non-real-time radio intelligent controller (non-RT RIC). In the illustrated embodiment, the management architecture 216 includes a rule-based agent 210, an RL agent 212, and a management node 214. These aspects will be referred to in detail in the accompanying drawings. Figure 3 for a more detailed discussion.

[0037] Figure 3 310 and management architecture 312 according to an embodiment of the present disclosure. For example, management architecture 312 may correspond to the above Figure 2 Management architecture 216 is depicted. Management architecture 312 includes rule-based agent 302, RL agent 304, and management node 306 (eg, corresponding to rule-based agent 210, RL agent 212, and management node 214, respectively).

[0038] According to an embodiment of the present disclosure, the environment 310 may include a telecommunications network or a portion of a telecommunications network. For example, the environment may include or correspond to a radio access network (RAN), one or more RAN nodes (such as those described above with respect to, for example, Figure 2 The environment 310 may also include or correspond to one or more wireless devices or user equipment (UE) described above. In another example, the environment 310 may additionally or alternatively include a core network (such as the core network 202 described above) or one or more core network nodes. The system 308 operates in the environment 310 and performs actions in the environment 310 under the control of the management architecture 312. For example, the system 308 may correspond to a communication node or device within a telecommunications network, such as a RAN node, a core network node, or a UE. Alternatively, the system 308 may correspond to a relevant portion of a communication node or device within a telecommunications network, such as its processing circuit.

[0039] The management node 306 is configured with a set of available actions that can be performed by the system 308 for a particular task. For example, where the system 308 corresponds to a RAN node, one task may correspond to setting the tilt of an antenna. In this case, the set of available actions may correspond to a set of relative adjustments to the antenna tilt, such as "tilt antenna up X degrees", "tilt antenna down X degrees", and "no change" (where X is a number). The management node 306 selects an action from the set of available actions and outputs the action to the system 308 to be implemented.

[0040] According to an embodiment of the present disclosure, management architecture 312 includes at least two different mechanisms by which management node 306 selects actions to output to system 308 : a rule-based mechanism implemented by rule-based agent 302 ; and an RL mechanism implemented by RL agent 304 .

[0041] The rule-based agent 302 is configured to recommend an action (from a set of available actions configured in the management node 306) for implementation by the system 308. The action is recommended by the rule-based agent 302 based on the state of the environment 310 and using one or more pre-configured rules.

[0042] Thus, data is collected from the environment 310 and provided to the rule-based agent 302. In some embodiments, the data may include information related to the state of the environment 310.

[0043] The state of environment 310 includes the value of one or more performance parameters used to measure the performance of the environment or a portion of the environment. Thus, one or more performance parameters can measure the performance of a single portion of the environment or the performance of the environment as a whole.

[0044] As described above, the environment is a telecommunication environment and may include a telecommunication network or a portion of a telecommunication network. Therefore, in such an embodiment, the state of the environment may be related to the performance of the entire telecommunication network or the performance of a portion of the telecommunication network. The telecommunication network may include one or more physical nodes or entities (e.g., base stations, UEs, core network nodes, etc.), one or more virtual nodes or entities (e.g., network nodes or entities defined in software), and / or one or more logical entities (e.g., cells, tracking areas, public land mobile networks (PLMNs), etc.). A portion of a telecommunication network may include any one or more of these nodes or entities.

[0045] For example, the state of the environment may include one or more performance parameters measuring the performance of one or more physical or virtual nodes (such as RAN nodes). In such a case, the one or more performance parameters may include one or more of the following: Quality of Experience (QoE) associated with services provided by the RAN node; Quality of Service (QoS) associated with services provided by the RAN node; Statistics related to handovers between RAN nodes (e.g., number of handover attempts, handover success rate, etc.); number of radio measurement reports received by the RAN node from served UEs; Radio measurements reported to the RAN node (e.g., RSRP, RSRQ, SINR, etc.); and number or proportion of error reports (e.g., negative acknowledgments) received by the RAN node.

[0046] Additionally or alternatively, the state of the environment may include one or more performance parameters that measure the performance of one or more physical or virtual nodes (such as wireless devices or UEs). In such a case, the one or more performance parameters may include one or more of the following: power usage of one or more UEs; radio measurements (e.g., RSRP, RSRQ, SINR, etc.) performed by one or more UEs; the number of error reports (e.g., negative acknowledgements) sent by one or more UEs and / or received by the UEs; and the number or proportion of radio link failures or handover failures experienced by one or more UEs.

[0047] Additionally or alternatively, the state of the environment may include one or more performance parameters measuring the performance of one or more logical entities of the telecommunication environment (e.g., cells, tracking areas, PLMNs, etc.). In such a case, the performance parameters may include any one or more of the following: capacity of one or more logical entities; throughput of one or more logical entities; statistics related to handovers between cells (e.g., number of handover attempts, handover success rate, etc.); total number of users camped on the logical entity; coverage of one or more logical entities; and signal strength at the edge of one or more cells.

[0048] Any of these one or more performance parameters may be measured in real time or over a period of time. In the latter case, the value of the performance parameter may be averaged over the period of time. In addition, multiple sets of data (e.g., collected during different time periods) may be averaged to determine one or more time-averaged performance parameters for the environment.

[0049] The rules implemented by the rule-based agent 302 may be preconfigured using existing domain knowledge related to the environment 310, such that the rule-based agent 302 recommends actions that are known or expected (with a high degree of certainty) to improve or at least maintain the state of the environment 310. For example, the actions recommended by the rule-based agent 302 may maintain or improve one or more performance parameters of the state of the environment.

[0050] In this context, "improving" the state of the environment may include improving one or more performance parameters that represent the state of the environment. Note that, depending on the definition of the performance parameter, improving a parameter may include increasing or decreasing the value of the parameter. For example, performance parameters such as SINR, RSRP, handover success rate, etc. are improved by increasing the values ​​of those parameters. Conversely, performance parameters such as the number of radio link failures or handover failures, the number of received error reports, etc. are improved by decreasing the values ​​of those parameters.

[0051] When determining whether the state of the environment has improved in an overall sense, one or more performance parameters representing the state of the environment may be weighted more heavily than other performance parameters. For example, it may be considered more important to achieve improvements in one or more performance parameters representing the environment as a whole (e.g., the overall throughput of the network, etc.) rather than improvements in one or more performance parameters representing a specific portion of the environment (e.g., the throughput of any one specific network node). In one embodiment, the improvement of the environment may be determined based on the values ​​of one or more performance parameters and ignoring the values ​​of other performance parameters.

[0052] The preconfigured rules implemented by the rule-based agent 302 can operate in a logical or deterministic manner such that given the same input data, the rule-based agent recommends the same action. For example, the preconfigured rules may include one or more functions to be applied to the performance parameters, where the outputs of the functions are used to determine the recommended action (e.g., by comparing those outputs to one or more thresholds). As described above, the functions and / or thresholds may be determined based on domain knowledge of the environment 310.

[0053] Thus, the rule-based agent 302 determines a recommended action for the system 308 based on the state of the environment 310 indicated by the input data, and outputs the recommended action to the management node 306. It can be assumed (with a high degree of certainty) that the action recommended by the rule-based agent 302, once implemented by the system 308, will not negatively affect the state of the environment 310.

[0054] As described above, a traditional RL agent interacts with an environment by exploring the state of the environment and selecting actions to be performed on the environment. The purpose of selecting an action is to maximize the long-term return of the action based on a reward signal. In the context of an embodiment of the present disclosure, the RL agent 304 is configured to interact with the environment 310 by exploring the state of the environment 310 and outputting a reward value associated with each action to a management node 306. The actions to be implemented by the system 308 are selected by the management node 306.

[0055] Thus, RL agent 304 may include an RL model configured to estimate a reward value as a result of system 308 performing a given output action based on the state of environment 310. The reward value may be based on an estimated improvement in the state of environment 310 as a result of performing the action. The reward may be an estimate of at least one of: an immediate reward for performing the action and a cumulative future reward for performing the action.

[0056] In certain embodiments of the present disclosure, RL agent 304 uses the RL model and the state of environment 310 as input to calculate a plurality of reward values. Each reward value may correspond to an estimated reward for implementing a corresponding action in a set of available actions. The RL model may be a deep Q network (DQN) model, and the reward value may be a Q value. The plurality of reward values ​​are sent to management node 306.

[0057] The management node 306 thus receives the recommended action and the plurality of reward values ​​from the rule-based agent 302 and the RL agent 304, respectively. According to an embodiment of the present disclosure, in the case where an action from the set of available actions opposes the recommended action, the opposing action is removed or masked from the set of available actions. Note that if there is no action opposing the recommended action, the action may not be removed from the set of available actions.

[0058] In the context of the present disclosure, in the case where the first action indicates a change to the configuration of the system 308 or a parameter of the system 308, and the second action indicates a change to the configuration of the system 308 or a parameter of the system 308 that has an opposite effect on the configuration or parameter, the first action is opposed to the second action. For example, in the case where the first action indicates a positive change to the parameter, the second action indicating a negative change to the parameter is opposed to the first action (and vice versa). Therefore, in some embodiments, in the case where the two actions indicate changes of opposite signs to the parameters, the two actions are opposed to each other. In further embodiments, the magnitude of the change can also be related to determining whether one action is opposed to another action. In such a case, in the case where the second action indicates a change of the opposite sign to the parameter, and the absolute magnitude of the change indicated by the second action is the same as or greater than the absolute magnitude of the first action, the second action can be opposed to the first recommended action. Consider an example where the task performed by the system is to control the tilt of the antenna, and the set of available actions includes the following: -10 degree tilt; -5 degree tilt; -1 degree tilt; no change; +1 degree tilt; +5 degree tilt; and +10 degree tilt. If the rule-based agent recommends a +5 degree tilt, then the adversarial action may be considered to include a -10 degree tilt and a -5 degree tilt.

[0059] The management node 306 then selects an action to be implemented by the system 308 from the reduced set of actions (i.e., the set of available actions with any "opposing" actions removed) based on the output of the RL agent 304. For example, particularly where the management node 306 implements the exploitation phase, the management node 306 may select an objectively best action from the reduced set of actions, such as an action associated with the highest or maximum reward value. In many cases, this action will correspond to an action recommended by the rule-based agent 302. That is, both the rule-based agent 302 and the RL agent 304 will recommend the same action. Where the management node 306 implements the exploration phase, the management node 306 may select an action that is not associated with the highest or maximum reward value, but may select a different action for the purpose of exploring the available state space. In either case, an action that is opposed to an action recommended by the rule-based agent 302 is not selectable, and thus the management node 306 is prevented from selecting an action that may result in an undesirable result. The management node 306 outputs the selected action to the system 308 for implementation.

[0060] This process can be performed repeatedly (e.g., periodically). For each repetition, the RL agent 304 can be trained using training data collected during and / or after performing the action (e.g., performance data collected from the environment 310), so that the RL model is trained to output reliable reward values. In this way, the management node 306 is enabled to select an action that maximizes the reward returned for its implementation.

[0061] According to an embodiment of the present disclosure, system 308 is thus prevented from performing actions that are known to be antagonistic (or contrary) to actions that do not negatively impact the state of environment 310. As previously discussed, when environment 310 is a telecommunications environment, small changes to the state of the telecommunications environment can have a wide-ranging impact on network performance. Therefore, actions that are likely to negatively impact the state of the telecommunications environment should be avoided.

[0062] Figure 4 4 is a signaling diagram illustrating signaling between a SON agent 406, a DQN agent 408, and a management node 404 according to an embodiment of the present disclosure. The signaling diagram illustrates two time steps t-1 and t of an iterative process by which the management node 404 may determine an action a to be performed in the environment 402 at the end of a given time step. The SON agent 406 may correspond to the rule-based agent 210, 302 described above; the DQN agent 408 may correspond to the RL agent 212, 304; the management node 404 may correspond to the management node 214, 306; and the environment may correspond to the environment 310 described above.

[0063] Signaling starts at a first time step t-1. In step 410, the management node 404 forwards the action a to be performed in the environment 402. t-1 At step 412, the action is executed in environment 402 (e.g., by a system in environment 402). At step 414, the DQN agent obtains: the reward value r returned for executing the action t-1 ; and the state s of the environment after executing the action (i.e. at the end of the first time step) t .

[0064] At the second time step t, at step 416, the DQN agent stores training data including information related to the previous iteration of the method at the first time step. For example, the DQN agent may store one or more of the following: the actions performed (a t-1 ); the state s of the environment before executing the action (i.e., at the beginning of the first time step) t-1 ; The state of the environment after executing the action (i.e., at the end of the first time step) s t ; and the reward r resulting from performing the action t-1 At step 418, the DQN agent 408 trains a DQN model using the stored training data.

[0065] At step 420, the SON agent 406 uses a rule-based algorithm to recommend an action a sonThis action is one of the available actions that can be suggested by the SON agent 406. Figure 3 As discussed, the rule-based algorithm may suggest actions that are known with a high degree of certainty to maintain or improve the state of the environment 402 when implemented. In the illustrated embodiment, the output of the SON agent 406 is referred to as a "safe" action because the action is known not to degrade the performance of the environment 402. In step 424, the suggested action is forwarded to the management node 404.

[0066] In step 422, the DQN agent 408 calculates a reward value for each action in the set of available actions. In some embodiments, the reward value is a Q value that estimates the cumulative future reward for implementing the corresponding action in the set of available actions. In step 426, the reward value is sent to the management node 404.

[0067] At step 428, the management node 404 determines whether any action in the set of available actions conflicts with the action suggested by the SON. If such a conflicting action exists, the management node 404 generates a reduced set of available actions by removing the conflicting action from the set of available actions.

[0068] At step 430, the management node selects an action from the reduced set of available actions. For example, if the management node 404 determines at step 428 that no action conflicts with the action recommended by the SON agent, the management node 404 determines an action to be performed in the environment 402 from the set of available actions. t On the other hand, if the management node 404 removes an action from the set of available actions at step 428, the management node 404 determines an action a to be performed in the environment 402 from the reduced set of available actions. t In any scenario, the RL agent’s policy can be used to determine an action from the corresponding set of actions (a t ). For example, the determined action may have the highest associated reward value. Alternatively, an exploration algorithm may be used to select different actions and thus explore the state space of the environment 402. Epsilon-Greedy, Softmax, or upper confidence bound algorithms may be used for this purpose.

[0069] At step 432, the action (a t ) is performed by a system operating in environment 402, and in step 434, the action is performed in environment 402.

[0070] Figure 5 is a flow chart illustrating process steps in a computer-implemented method 500 for managing a system operating in a telecommunications environment. Figure 2 and Figure 3 The method may correspond to the management architecture described in Figure 4 Signaling and actions are described for management node 404, SON agent 406, and DQN agent 408. In some embodiments, the telecommunications environment may include at least one of a RAN and a SON. In some embodiments, the system may be a network node, such as a base station. A base station may include one or more radio antennas. Managing the system includes causing the system to implement an action, which is one of a set of available actions.

[0071] The method begins at step 502, where data related to the state of the telecommunications environment is analyzed using a rule-based algorithm to determine a recommended first action from a set of available actions. In some embodiments, the data related to the state of the telecommunications environment includes one or more parameters related to a performance characteristic of the telecommunications environment. The performance characteristic may be a KPI.

[0072] The recommended first action may be determined by a rule-based algorithm to improve the state of the telecommunications environment or to cause no change to the state of the telecommunications environment. For example, the action may improve one or more performance characteristics of the telecommunications environment or to cause no change to one or more performance characteristics of the telecommunications environment. Examples of (use case specific) rule-based algorithms are discussed further below.

[0073] Still reference Figure 5 The method further comprises, in step 504, removing from the set of available actions a second action that is antagonistic to the recommended first action to generate a reduced set of available actions. Two actions may be antagonistic (i.e., opposed) to each other in the sense that they are opposite to each other. Examples of such antagonistic actions are opposite changes to parameters, such as: increasing or decreasing the vertical tilt of an antenna, increasing or decreasing the target power per resource block received by an antenna, and increasing or decreasing the horizontal width of a cell sector.

[0074] The method further includes, in step 506, analyzing data related to the state of the telecommunications environment using a reinforcement learning algorithm to select a third action from the reduced set of available actions. The data may be the same data analyzed by the rule-based algorithm. The RL algorithm may be an algorithm separate from the rule-based algorithm. In some embodiments, the data related to the state of the telecommunications environment includes one or more parameters related to a performance characteristic of the telecommunications environment. The performance characteristic may be a KPI. The recommended first action and the third action may correspond to the same action in the set of available actions.

[0075] In some embodiments, analyzing data using a reinforcement learning algorithm may include obtaining a plurality of reward values ​​from a reinforcement learning agent. Each reward value may be associated with an estimated reward (e.g., an immediate reward and / or a cumulative future reward) for implementing a corresponding action in a set of available actions. The estimated reward may be based on the calculated impact of the corresponding action on the state of the telecommunications environment. A third action may be selected based on a plurality of reward values. For example, an action with the highest reward value among a plurality of reward values ​​may be selected as the third action, or a third action may be selected based on a reinforcement learning exploration strategy and a plurality of reward values.

[0076] In step 508, the system is caused to perform a third action.

[0077] In some embodiments, the method further includes, in response to causing the system to perform the third action, storing 510 training data including data related to the state of the telecommunication environment, the third action, a reward value corresponding to the third action, and data related to an updated state of the telecommunication environment as a result of performing the third action. The method may also include training 512 the reinforcement learning agent using the stored training data.

[0078] In some embodiments, the reinforcement learning agent is a DQN agent and the reward value is a Q-value.

[0079] In some embodiments, the method further includes, in response to determining that there is no second action that counters the recommended first action, avoiding removing any action from the set of available actions. The method may also include analyzing data related to the state of the telecommunications environment using a reinforcement learning algorithm to determine a third action from the set of available actions. Likewise, in some embodiments, the recommended first action and the third action may correspond to the same action in the set of available actions.

[0080] In some embodiments, the one or more parameters include an indication of any one or more of: coverage of at least one network cell in the telecommunications environment; capacity of at least one network cell in the telecommunications environment; quality of experience (QoE) associated with services provided by the RAN of the telecommunications environment; quality of service (QoS) associated with services provided by the RAN of the telecommunications environment; signal strength at an edge of at least one network cell in the telecommunications environment; average throughput of multiple network cells in the telecommunications environment; and average coverage of multiple network cells.

[0081] In some embodiments, using the rule-based algorithm includes applying a function to data related to the state of the telecommunications environment. The method may also include: comparing an output of the function to a threshold; and determining a recommended first action based on the comparison. The threshold may be adjusted to tune the frequency with which the rule-based algorithm determines the recommended first action as an action with an adversarial action. That is, if the recommended first action does not have an adversarial action, the second action is not removed from the set of available actions (i.e., a reduced set of available actions is not generated). By adjusting the threshold so that the rule-based algorithm has a greater likelihood of recommending an action that does not result in a change, the RL algorithm is given greater freedom in selecting a third action from the complete set of available actions, thereby allowing for increased exploration of the search space.

[0082] In some embodiments, the system includes at least one antenna, and the available action set is used to control the tilt of the at least one antenna. For example, the available action set may include: tilting the at least one antenna upward; tilting the at least one antenna downward; and not changing the tilt of the at least one antenna.

[0083] In some embodiments, the system includes at least one antenna, and the available action set is used to control the target power of each resource block received by the at least one antenna (e.g., P0 nominal PUSCH). For example, the available action set may include: increasing the target power of each resource block received by the at least one antenna; decreasing the target power of each resource block received by the at least one antenna; and not changing the target power of each resource block received by the at least one antenna.

[0084] Two example use cases are now discussed in detail to illustrate how the methods of the present disclosure may be implemented to address example technical scenarios.

[0085] Use Case 1: Remote Electrical Tilt

[0086] Modern cellular networks are increasingly faced with the need to meet consumer demands that are highly variable in both spatial and temporal domains. In order to be able to efficiently provide high levels of QoS to UEs, the network must adjust its configuration in an automatic and timely manner. Antenna vertical tilt angle, also known as downtilt angle, is one of the most important variables to control for QoS management. Downtilt angle can be modified both mechanically and electronically, but due to the costs associated with manual adjustment of downtilt angle, remote electrical tilt (RET) optimization is used in the vast majority of modern networks.

[0087] There are several KPIs that can be considered when evaluating the performance of a RET optimization strategy, and significant existing material is dedicated to coverage capacity optimization (CCO), where the coverage KPI is related to the coverage area expressed in terms of minimum received signal strength, and the capacity KPI is related to the average total throughput in a given area of ​​interest.

[0088] Control of RET involves a trade-off between coverage and capacity: an increase in antenna downtilt (θ) is associated with a stronger signal in a more concentrated area, as well as higher capacity and reduced interference radiation to other cells in the network. However, excessive downtilt may result in insufficient coverage in a given area, with some UEs unable to receive the minimum signal quality.

[0089] During the past decade, there has been a large amount of research in the field of using RL methods for RET optimization. However, most existing methods for RET optimization using RL techniques do not consider any concept of security, resulting in arbitrary performance interruptions during the learning process. Since reliability of service is one of the most important features for network providers, the possibility of performance degradation has actually hindered the deployment of RL methods for RET optimization in the real world, although these methods have been shown to outperform currently used traditional methods in simulated network environments. By addressing the problem of performance degradation via the above-mentioned SRL process, the method according to the present disclosure can achieve safe deployment of RL agents for RET optimization, as discussed below.

[0090] For this scenario, the elements of the architecture discussed above could be:

[0091] System: A physical base station with at least one radio antenna. Actions that can be implemented by the system include possible discrete changes to antenna downtilt. For example, only three actions may be available: tilt up, tilt down, and no change. In other examples, additional actions may be available, including different amounts of uptilt and downtilt.

[0092] Telecommunications environment: Consider a physical 4G or 5G mobile cellular network area for RET optimization. The state of the environment may be determined by one or more of: cell congestion, cell overlap, cell interference, cell overshooting, and low reference signal received power (RSRP) at the cell edge. Rewards returned for performing a specific action may favor actions that result in any of the following: traffic with good RSRP; traffic with good SINR; and reduced congestion.

[0093] RL Algorithm: The RL algorithm employed by the RL agent whose policy takes the state of the mobile cellular network as input. The learning agent's policy can be DQN.

[0094] Rule-based algorithm: an algorithm that uses existing network domain knowledge to recommend actions given the state of a mobile cellular network area. An example of such a rule-based algorithm is discussed in detail in the following document: "Self-tuning of remote electrical tilts based on call traces for coverage and capacity optimization in LTE", V. Buenestado et al., IEEE Transactions on Vehicular Technology, Vol. 66, No. 5, pp. 4315-4326, May 2017. The algorithm can be used to iteratively calculate recommended changes to the antenna tilt angle using cell traces. The algorithm is designed to optimize the tradeoff between cell coverage, cell quality, and cell capacity. For example, although large cell overlaps often result in better coverage, connection quality may decrease and cell overcoverage may increase. When a signal strength that is close in strength to the signal of its strongest neighbor is measured in a distant cell (for a specific cell), cell overcoverage of the specific cell occurs. Therefore, cell overcoverage may result in useless cell overlap, since cell coverage is not improved and interference levels are increased.

[0095] To monitor this trade-off, the algorithm first uses the overcoverage indicator N for the particular cell i under study. os (i) and the useless overlap indicator N ol (i) as input to the first fuzzy logic controller (FLC). The fuzzy output value UI(i) of the first FLC quantifies the unwanted interference generated by cell i and ranges from 0 (no interference) to 1 (large interference).

[0096] N os (i) is calculated as:

[0097]

[0098] Where N(i) is the set of neighbors of cell i; X rn (j, i) is a neighbor level indicator showing whether cell i is a relevant neighbor of cell j, N s (j) is the total number of RSRP samples reported by users in cell j; (uon represents the useless overlap from non-related cells) is the RSRP level difference between the serving cell and the neighbor cell i is below a predefined threshold The number of samples, and cell i is not the strongest neighbor; and R uon is a scaling factor between 0 and 1 that defines the ratio of samples required to consider source cell i as a strong interferer for neighbor cell j. The min() operator ensures that neighbor cell i contributes only to N os (i) Add 1.

[0099] Parameter X rn (j, i) is defined as:

[0100]

[0101] in (rn denotes relevant neighbors) is the number of samples whose RSRP value from neighbor cell i is higher than that of the rest of the neighbors, and R rn is the ratio of correlation samples that consider source cell i as a perfectly correlated neighbor of neighbor cell j.

[0102] N ol (i) is calculated as:

[0103]

[0104] in (uo represents useless overlap) is when the RSRP difference between serving cell j and neighbor cell i is lower than a predefined threshold The number of samples, and the signal level from serving cell j is greater than the threshold And R uo is a scaling factor between 0 and 1 that defines the ratio of low dominance samples used to consider source cell i as a strong interferer for neighboring cell j. Similarly, the min() operator ensures that neighbors with large area of ​​unwanted cell overlap with cell i are only forwarded to N ol (i) Add 1.

[0105] N os (i) and N ol (i) The first FLC as input has the following rules:

[0106] Table I

[0107] Rules in the First Fuzzy Logic Controller

[0108]

[0109] The algorithm then uses UI(i), the output of the first FLC and the poor coverage indicator R bc (i) as input to the second FLC. The proposed tilt change α(i) is output from the second FLC, which can be positive (tilting down) or negative (tilting up).

[0110] R bc (i) is calculated as:

[0111]

[0112] Where N bcce (i) is reported by the cell edge UE of cell i and is below a certain threshold The number of RSRP samples, and N ce (i) is the total number of samples reported by cell-edge UEs of cell i.

[0113] Take UI(i) and R bc (i) The second FLC as input has the following rules:

[0114] Table II

[0115] Rules in the Second Fuzzy Logic Controller

[0116]

[0117] Therefore, the proposed tilt change output from the second FLC is known to improve or maintain the cell coverage, cell quality and cell capacity of cell i.

[0118] The proposed tilt change output is provided to the management node, for example as described above with respect to Figures 2 to 4 Any one of the above, and is used to mask any action from the set of available actions that is antagonistic (ie, opposite) to the proposed tilt change output. Then, the action to be implemented can be selected from the remaining available actions.

[0119] Use case 2: P0 nominal PUSCH

[0120] As with the downward tilt described above, the cell's P0 nominal PUSCH (physical uplink shared channel) parameter is also an important network configuration variable to control for QoS management. The P0 nominal PUSCH defines the target power per resource block (RB) that the cell expects in uplink (UL) communications from the UE to the base station. By increasing this parameter, the UL SINR in a specific cell can increase (due to increased signal strength). At the same time, the UL SINR in surrounding cells may decrease (due to increased cell interference).

[0121] For this scenario, the elements of the architecture discussed above could be:

[0122] System: A physical base station with at least one radio antenna. Actions that can be implemented by the system include possible discrete changes to the target power per RB expected in UL communications. In this example, only three actions are available: increase the target power per RB, decrease the target power per RB, or make no change to the target power per RB. However, additional actions may define different amounts by which the target power per RB is increased or decreased.

[0123] Telecommunications environment: A physical mobile cellular network area (e.g., 4G or 5G). The state of the environment may be determined by at least any one of: cell congestion, cell overlap, cell interference, cell over-coverage, and low RSRP at the cell edge. Rewards returned for performing a particular action may favor actions that result in any one of: traffic with good RSRP; traffic with good SINR; and reduced congestion.

[0124] RL Algorithm: The RL algorithm employed by the RL agent whose policy takes the state of the mobile cellular network as input. The learning agent's policy can be DQN.

[0125] Rule-based algorithm: An algorithm that uses existing network domain knowledge to recommend actions given the state of a mobile cellular network area. Those skilled in the art will be able to develop many suitable rule-based algorithms to control and / or set the P0 nominal PUSCH. For example, the P0 nominal PUSCH for a particular cell may be controlled as a function of one or more of: the throughput of the particular cell; the throughput of one or more cells adjacent to the particular cell; the radio measurements performed by the UE served by the particular cell on the particular cell and one or more neighboring cells. In the event that the throughput for the particular cell drops below a threshold, and / or the throughput for the particular cell is lower than the throughput of the neighboring cells by a threshold, the P0 nominal PUSCH value for the particular cell may be increased by a certain amount. In this way, the UEs served by the particular cell increase their transmit power in order to increase the throughput of the particular cell.

[0126] Figure 6 A schematic block diagram of an apparatus 600 for managing a system operating in a telecommunications environment is shown. The apparatus 600 is operable to perform a reference Figure 5 The example methods described herein may also be performed in accordance with the present invention and any other process or method disclosed herein may be performed. Figure 5 The method is not necessarily performed only by the apparatus 600. At least some operations of the method may be performed by one or more other entities.

[0127] The device 600 includes a processing circuit 604, which may include one or more microprocessors or microcontrollers, and other digital hardware that may include a digital signal processor (DSP), dedicated digital logic, etc. The processing circuit 604 may be configured to execute program codes stored in a memory 602, which may include one or more types of memory, such as a read-only memory (ROM), a random access memory, a cache memory, a flash memory device, an optical storage device, etc. In several embodiments, the program codes stored in the memory 602 include program instructions for executing one or more telecommunication and / or data communication protocols, and instructions for executing one or more of the techniques described herein. In some implementations, the processing circuit 604 may cause the device 600 to perform corresponding functions according to one or more embodiments of the present disclosure.

[0128] According to an embodiment of the present disclosure, the processing circuit 604 is configured to enable the device 600 to: use a rule-based algorithm to analyze data related to the state of the telecommunication environment to determine a recommended first action from a set of available actions; remove a second action that is opposed to the recommended first action from the set of available actions to generate a reduced set of available actions; use a reinforcement learning algorithm to analyze data related to the state of the telecommunication environment to select a third action from the reduced set of available actions; and enable the system to implement the third action.

[0129] The device 600 may be implemented in a node of a communication network, such as a radio network, an optical network, or an electronic network. Thus, the device 600 also includes one or more interfaces 606, which are used to communicate with one or more other nodes of the communication network. Thus, the one or more interfaces 606 may include hardware and / or software for sending and / or receiving one or more of the following: radio signals; optical signals; and electronic signals.

[0130] In an alternative embodiment, the apparatus 600 may include one or more units or modules configured to perform, for example, Figure 5 In such an embodiment, the apparatus 600 may include an analyzing unit, a removing unit, and an implementing unit. The analyzing unit may be configured to analyze data related to the state of the telecommunication environment using a rule-based algorithm to determine a recommended first action from a set of available actions. The removing unit may be configured to remove a second action that is antagonistic to the recommended first action from the set of available actions to generate a reduced set of available actions. The analyzing unit may also be configured to analyze data related to the state of the telecommunication environment using a reinforcement learning algorithm to select a third action from the reduced set of available actions. The implementing unit may be configured to cause the system to implement the third action.

[0131] The term "unit" may have a conventional meaning in the field of electronics, electrical devices and / or electronic devices, and may include, for example, electrical and / or electronic circuits, devices, modules, processors, memories, logical solid-state and / or discrete devices, computer programs or instructions for performing corresponding tasks, processes, calculations, output and / or display functions, etc., such as those described in this document.

[0132] The method of the present disclosure may be implemented in hardware or as a software module running on one or more processors. The method may also be performed according to the instructions of a computer program, and the present disclosure further provides a computer-readable medium on which a program for performing any method described herein is stored. The computer program embodying the present disclosure may be stored on a computer-readable medium, or it may, for example, take the form of a signal, such as a downloadable data signal provided from an Internet website, or it may take any other form.

[0133] It should be noted that the examples mentioned above illustrate rather than limit the present disclosure, and that those skilled in the art will be able to design many alternative embodiments without departing from the scope of the attached embodiments. The word "comprising" does not exclude the presence of elements or steps other than those listed in the embodiments or claims, "a" does not exclude a plurality, and a single processor or other unit may implement the functions of several units described in the embodiments. Any reference symbols in the claims should not be interpreted as limiting their scope.

Claims

1. A computer-implemented method for managing a system (308) operating in a telecommunications environment (200, 310, 402), wherein: Managing the system includes causing the system to perform an action, the action being one of a set of available actions, the method comprising: - analyzing (502) data related to the state of the telecommunication environment (200, 310, 402) using a rule-based algorithm to determine a recommended first action from the set of available actions; - removing (504) from the set of available actions a second action that is opposed to the recommended first action to generate a reduced set of available actions; - analyzing (506) data related to the state of the telecommunication environment (200, 310, 402) using a reinforcement learning algorithm to select a third action from the reduced set of available actions; and - causing (508) said system (308) to perform said third action.

2. The method according to claim 1, wherein: Analyzing the data using the reinforcement learning algorithm includes: - obtaining a plurality of reward values ​​from a reinforcement learning agent (212, 304, 408), wherein each reward value is associated with an estimated reward for performing a corresponding action from the set of available actions, the estimated reward being based on a calculated effect of the corresponding action on the state of the telecommunication environment (200, 310, 402), and - wherein said third action is selected based on said plurality of reward values.

3. The method according to claim 2, wherein: The third action is selected as the action having the highest reward value among the plurality of reward values.

4. The method according to claim 2, wherein: The third action is selected based on a reinforcement learning exploration strategy and the plurality of reward values.

5. The method according to any one of claims 2 to 4, further comprising: In response to causing the system (308) to perform the third action: - storing (510) training data, the training data comprising data relating to the state of the telecommunication environment (200, 310, 402), the third action, a reward value corresponding to the third action, and data relating to an updated state of the telecommunication environment as a result of performing the third action; as well as - Training (512) the reinforcement learning agent (212, 304, 408) using the stored training data.

6. The method according to any one of claims 2 to 5, wherein: The reinforcement learning agent (212, 304, 408) is a deep Q network (DQN) agent, and the reward value is a Q value.

7. The method according to any one of the preceding claims, further comprising: - in response to determining that there is no second action that competes with the recommended first action, refraining from removing any action from the set of available actions; as well as - analyzing data related to the state of the telecommunication environment using the reinforcement learning algorithm to determine a third action from the set of available actions.

8. A method according to any one of the preceding claims, wherein: The data related to the status of the telecommunication environment includes: - one or more parameters related to performance characteristics of said telecommunication environment.

9. The method according to claim 8, wherein: The one or more parameters include an indication of any one or more of the following: - coverage of at least one network cell in the telecommunication environment; - the capacity of at least one network cell in the telecommunication environment; - Quality of Experience (QoE) associated with services provided by a radio access network of said telecommunication environment; - a quality of service QoS associated with a service provided by a radio access network of said telecommunication environment; - signal strength at the edge of at least one network cell in the telecommunication environment; - an average throughput of a plurality of network cells in the telecommunication environment; as well as - an average coverage of the plurality of network cells.

10. A method according to any one of the preceding claims, wherein: Use of the rule-based algorithm includes: - applying a function to data related to the state of said telecommunication environment; and - comparing an output of the function with a threshold value, determining the recommended first action based on the comparison.

11. A method according to any one of the preceding claims, wherein: The recommended first action indicates a first change in a configuration of the system, and the second action indicates a second change in the configuration of the system, wherein the first change has an opposite effect on the configuration of the system than the second change.

12. A method according to any one of the preceding claims, wherein: The system (308) includes at least one antenna, and the set of available actions is for controlling a tilt of the at least one antenna.

13. The method according to claim 12, wherein: The set of available actions includes: - tilting the at least one antenna upwards; - tilting the at least one antenna downwards; and - The tilt of the at least one antenna is not changed.

14. The method according to any one of claims 1 to 11, wherein: The system (308) includes at least one antenna, and the set of available actions is used to control a target power for each resource block received by the at least one antenna.

15. The method according to claim 14, wherein: The set of available actions includes: - increasing the target power of each resource block received by the at least one antenna; - reducing the target power of each resource block received by the at least one antenna; - The target power for each resource block received by the at least one antenna is not changed.

16. A method according to any one of the preceding claims, wherein: The recommended first action and the third action correspond to the same action in the set of available actions.

17. A method according to any one of the preceding claims, wherein: The telecommunication environment comprises a Self Organizing Network SON.

18. A management node (216, 312, 600) for managing a system (308) operating in a telecommunications environment (200, 310, 402), wherein: Managing the system includes causing the system to perform an action, the action being one of a set of available actions, the management node comprising a processing circuit (604), the processing circuit (604) being configured to: - analyzing (502) data related to the state of the telecommunication environment using a rule-based algorithm to determine a recommended first action from the set of available actions; - removing (504) from the set of available actions a second action that is opposed to the recommended first action to generate a reduced set of available actions; - analyzing (506) data related to the state of the telecommunication environment using a reinforcement learning algorithm to select a third action from the reduced set of available actions; and - causing (508) the system to perform the third action.

19. The management node according to claim 18, wherein: Being configured to analyze the data using the reinforcement learning algorithm comprises being configured to: - obtaining a plurality of reward values ​​from a reinforcement learning agent (212, 304, 408), wherein each reward value is associated with an estimated reward for performing a corresponding action from the set of available actions, the estimated reward being based on a calculated effect of the corresponding action on the state of the telecommunication environment, and - wherein said third action is selected based on said plurality of reward values.

20. The management node according to claim 19, wherein: The third action is selected as the action having the highest reward value among the plurality of reward values.

21. The management node according to claim 19, wherein: The third action is selected based on a reinforcement learning exploration strategy and the plurality of reward values.

22. The management node according to any one of claims 19 to 21, the processing circuit is further configured to, in response to causing the system to perform the third action: - storing (510) training data, the training data comprising data relating to the state of the telecommunication environment, the third action, a reward value corresponding to the third action, and data relating to an updated state of the telecommunication environment as a result of performing the third action; and - Training (512) the reinforcement learning agent using the stored training data.

23. The management node according to any one of claims 19 to 22, wherein: The reinforcement learning agent is a deep Q network (DQN) agent (408) and the reward value is a Q value.

24. The management node according to any one of claims 18 to 23, wherein the processing circuit is further configured to: - in response to determining that there is no second action that competes with the recommended first action, refraining from removing any action from the set of available actions; and - analyzing data related to the state of the telecommunication environment using the reinforcement learning algorithm to determine a third action from the set of available actions.

25. The management node according to any one of claims 18 to 24, wherein: The data related to the status of the telecommunication environment includes: - one or more parameters related to performance characteristics of said telecommunication environment.

26. The management node according to claim 25, wherein: The one or more parameters include an indication of any one or more of the following: - coverage of at least one network cell in the telecommunication environment; - the capacity of at least one network cell in the telecommunication environment; - Quality of Experience (QoE) associated with services provided by a radio access network of said telecommunication environment; - a quality of service QoS associated with a service provided by a radio access network of said telecommunication environment; - signal strength at the edge of at least one network cell in the telecommunication environment; - an average throughput of a plurality of network cells in the telecommunication environment; as well as - an average coverage of the plurality of network cells.

27. The management node according to any one of claims 18 to 26, wherein: Being configured to use the rule-based algorithm includes being configured to: - applying a function to data related to the state of said telecommunication environment; as well as - comparing an output of the function with a threshold value, determining the recommended first action based on the comparison.

28. The management node according to any one of claims 18 to 27, wherein: The recommended first action indicates a first change in a configuration of the system, and the second action indicates a second change in the configuration of the system, wherein the first change has an opposite effect on the configuration of the system than the second change.

29. The management node according to any one of claims 18 to 28, wherein: The system includes at least one antenna, and the set of available actions is for controlling a tilt of the at least one antenna.

30. The management node according to claim 29, wherein: The set of available actions includes: - tilting the at least one antenna upwards; - tilting the at least one antenna downwards; and - The tilt of the at least one antenna is not changed.

31. A management node according to any one of claims 18 to 28, wherein: The system includes at least one antenna, and the set of available actions is for controlling a target power for each resource block received by the at least one antenna.

32. The management node according to claim 31, wherein: The set of available actions includes: - increasing the target power of each resource block received by the at least one antenna; - reducing the target power of each resource block received by the at least one antenna; - The target power for each resource block received by the at least one antenna is not changed.

33. A management node according to any one of claims 18 to 32, wherein: The recommended first action and the third action correspond to the same action in the set of available actions.

34. A management node according to any one of claims 18 to 33, wherein: The telecommunication environment comprises a Self Organizing Network SON.

Images