Safe reinforcement learning for managing a system operative in a telecommunication environment

Abstract

A computer-implemented method manages a system operative in a telecommunication environment (210). The method comprises: providing (302) a first state of the environment as an input to a reinforcement learning algorithm; receiving (304), from the reinforcement learning algorithm, an output comprising a first action that is recommended for execution by the system, the first state and the first action forming a first state-action pair; and determining (306, 308) whether the first state-action pair is similar to a second state-action pair based on a measure of similarity. The second state-action pair comprises a second state of the environment (210) and a second action that was previously executed based on the second state and resulted in a violation of one or more safety constraints of the environment (210). In response to determining the first state-action pair is similar to the second state-action pair, the system refrains (318) from executing the first action.

Description

[0001]MANAGING A SYSTEM OPERATIVE IN A TELECOMMUNICATION ENVIRONMENT TECHNICAL FIELD Embodiments of the present disclosure relate to methods, apparatus and computer- readable media for managing a system operative in a telecommunications environment, and particularly to methods in which the system executes one or more actions which are recommended by a reinforcement learning (RL) algorithm based on a state of the telecommunication environment. BACKGROUND Antenna systems in radio networks are becoming increasingly complex. However, manual tuning and / or heuristic tuning strategies are costly and lack scalability, meaning they are not suited for such complex antenna systems. Instead, automated tuning is preferred. One way of automating antenna tuning is through the use of reinforcement learning (RL), which enables control strategies to be learnt through interactions of an RL agent with an environment. RL has been successfully used to tune antenna parameters such as remote electrical tilt (RET), downlink (DL) power, and uplink (UL) power. RL is a decision-making framework in which an RL agent aims to optimise its interactions with an environment by exploring its states and selecting actions to be executed on the environment. An action selected for execution and the state based on which the selection was made may be considered to form a state-action pair. Actions are selected with the aim of maximising the long-term return of the actions according to a reward signal. An RL agent is often deployed in environments, with a goal of experimenting and interacting with those environments to learn which state-action pairs lead to a high reward. During this process, actions that are suboptimal (and even sometimes unsafe) are attempted as at least a part of the state-action space may be unknown. More formally, an RL problem is defined by: - A state space ^, which is the set of all possible states that the agent can be in; - An action space ^, which is the set of all the possible actions the agent can take; - A transition probability distribution ^: ^ × ^ × ^ → [0,1], which is the result of the agent’s actions on the environment and dictates the stochastic dynamics of this space (i.e. which state the agent will end up in after its action); and - A reward distribution ^: ^ × ^ → ℝ, which defines how good the agent’s action is and can incentivise or penalise specific state-action pairs. For example, an action may be considered “good” in the sense that it results in a performance metric of the environment improving, or meeting a certain performance threshold. The agent’s behaviour in the environment (e.g., its implemented control strategy) is defined by its policy, ^. The agent’s policy determines which action it will take when it is in a given state, ^ ∈ ^. A typical agent-environment interaction can be as follows: - At timestep ^, the agent is in a state ^^∈ ^. Based on its policy, it selects an action ^^∼ ^(⋅ |^^) ∈ ^, which is accompanied by a stochastic reward ^^∼ ^(⋅ | ^^, ^^)- As a result of its action, the agent transitions to a new state, ^^^^∼ P(⋅|s^, a^). These interactions are aimed at finding an optimal policy which may maximise an expected (cumulative) reward received by the agent over a period of time. This term is often called the policy value function, defined as ^^(^)= ^ ^^^^^^^^^| ^^= ^ ^. While executing the above discussed dynamic optimisation process in an unknown environment (with respect to transition and reward probabilities), the RL agent tries out (i.e., explores) different state-action combinations such that it can learn to make accurate predictions about the rewards and transition probabilities of each state-action pair. As part of this process, the agent may therefore choose suboptimal actions, which conflicts with its goal of maximising the accumulated reward, in order to explore the state-action space. At each time step, the agent may decide whether to prioritise further gathering of information (exploration) or to make the best move given its current knowledge (exploitation). Exploration may create opportunities by discovering higher rewards on the basis of previously untried actions. However, exploration also carries the risk that previously unexplored decisions do not provide increased reward and may instead have a negative impact on the environment. For example, an action may be considered to have a “negative impact” in the sense that it results in a performance metric of the environment worsening or deviating from a certain performance threshold. This negative impact may only be short term or may persist, for example if the explored actions place the environment in an undesirable state from which it does not recover. Therefore, deploying RL agents in live telecommunications networks can be problematic given their function of exploring state-action spaces. This exploration can result in diminished performance of the telecommunications network until the RL agent learns how to properly control the network. To avoid this issue, a simulated environment (in which consequences resulting from sub- optimal actions will not be detrimental) may be used to train an RL agent. Whilst this reduces the risks associated with training the RL agent, the downside is that, when the RL agent is ultimately deployed in the real-world, there can be a mismatch between what the RL agent has been trained on and what the RL agent experiences. Indeed, most real-world environments tend to display noise in their measurements and involve more complicated dynamics than simulated environments, making deployment of RL agents in the real world challenging. To overcome this issue, “Safe Reinforcement Learning” (SRL) solutions have been developed which aim to improve RL agent performance whilst maintaining a measure of safety (see “A comprehensive survey on safe reinforcement learning” by García, J. & Fernández, F. in Journal of Machine Learning Research, vol.16, 1437-1480, 2015, ACM). In some applications of SRL, an environment is “shielded” from an RL agent in the sense that safety constraints are imposed on the RL agent which restrict the actions that may be executed in the environment, thus reducing the risks associated with deploying the RL agent in the real-world (see “Safe Reinforcement Learning via Shielding” (2017) in Proceedings of the AAAI Conference on Artificial Intelligence, vol 32, 10.1609 / aaai.v32i1.11797). Safety in the context of RL can be defined through various criteria such as: a minimum performance level of the RL agent compared to known, safe baselines; robustness of the RL agent when faced with noise and uncertainty; and / or whether the RL agent can avoid repeating unwanted and / or unsafe actions. The risks associated with RL may be present during both the training and deployment phases of an RL agent, which can make the use of RL especially risky in safety-critical situations. SUMMARY Embodiments of the present disclosure aim to reduce the risk associated with deploying RL agents in real-world environments by reducing the risk that unwanted and / or unsafe actions are executed. As discussed in detail below, an action may be considered to be “unwanted” or “unsafe” in the sense that it is known, when executed in an environment in a particular state, to violate one or more pre-determined or pre-configured safety constraints. In a first aspect of the disclosure, a computer implemented method manages a system operative in a telecommunication environment. The system executes one or more actions recommended by an RL algorithm based on a state of the telecommunication environment. The method comprises: providing a first state of a telecommunication environment as an input to the RL algorithm; and receiving, from the RL algorithm, an output comprising a first action that is recommended for execution by the system, the first state and the first action forming a first state-action pair. The method further comprises determining whether the first state-action pair is similar to a second state- action pair based on a measure of similarity. The second state-action pair comprises a second state of the telecommunication environment and a second action that was previously executed based on the second state and resulted in a violation of one or more safety constraints of the telecommunication environment. In response to determining that the first state-action pair is similar to the second state-action pair, the method further comprises refraining from executing the first action. In a second aspect of the disclosure, a management node manages a system operative in a telecommunication environment. The system executes one or more actions recommended by an RL algorithm based on a state of the telecommunication environment. The management node comprises processing circuitry configured to cause the management node to: provide a first state of a telecommunication environment as an input to the RL algorithm; and receive, from the RL algorithm, an output comprising a first action that is recommended for execution by the system, the first state and the first action forming a first state-action pair. The processing circuitry is further configured to cause the management node to determine whether the first state-action pair is similar to a second state-action pair based on a measure of similarity. The second state-action pair comprises a second state of the telecommunication environment and a second action that was previously executed based on the second state and resulted in a violation of one or more safety constraints of the telecommunication environment. The management node comprises processing circuitry configured to cause the management node to, in response to determining that the first state-action pair is similar to the second state-action pair, refrain from executing the first action. BRIEF DESCRIPTION OF THE DRAWINGS For a better understanding of examples of the present disclosure, and to show more clearly how the examples may be carried into effect, reference will now be made, by way of example only, to the following drawings in which: Figure 1 is a schematic diagram illustrating a communication network according to embodiments of the disclosure; Figure 2 is a schematic diagram illustrating an architecture according to embodiments of the disclosure; Figure 3 is a flowchart of a method according to embodiments of the disclosure; Figure 4 is schematic diagram illustrating two neighbouring cells; Figure 5 illustrates two graphs showing an average reward accumulated by RL agents in a simulation using a first seed (upper graph) and a simulation using a second seed (lower graph); and Figure 6 is a schematic diagram of an apparatus according to embodiments of the disclosure. DETAILED DESCRIPTION Some of the embodiments contemplated herein will now be described more fully with reference to the accompanying drawings. Other embodiments, however, are contained within the scope of the subject matter disclosed herein. The disclosed subject matter should not be construed as limited to only the embodiments set forth herein; rather, these embodiments are provided by way of example to convey the scope of the subject matter to those skilled in the art. When an RL agent is being trained and / or is deployed, it may recommend an action which, when executed in an environment in a particular state, violates one or more pre- determined or pre-configured safety constraints. For example, execution of the action may result in the performance of the environment being negatively impacted to an extent that it may be difficult or impossible for the RL agent and / or the environment to recover. Thus, a state-action pair may be considered to result in a “violation” if executing the action of the state-action pair in the state of the state-action pair has previously been observed to violate one or more safety constraints imposed on the system. For example, a state-action pair may be flagged as resulting in a violation automatically or by a user when the effects of the executed action on the environment are observed. The one or more safety constraints may be related to the performance of the telecommunication network and may be pre-configured or pre-defined by a user / network operator. Examples of violations of safety constraints are provided below. Whilst an RL agent which has undergone training is unlikely to recommend an action that results in a violation, these recommendations may not be avoided entirely due to the exploratory nature of RL and / or the fact that the RL agent is likely to encounter states it has not encountered before (and therefore may not know that a recommended action will result in a violation before it is executed). Of course, a system should avoid executing actions recommended by an RL agent which result in violations. Examples of the present disclosure propose methods relating to SRL (and architecture on which they may be implemented) which change the previously discussed standard RL interaction cycle so as to improve the safety of an environment in which RL techniques are being employed by enabling a system to be prevented from repeating the execution of unwanted and / or unsafe actions. That is, the system is enabled to avoid state-action pairs previously observed to result in violations of one or more safety constraints imposed on the system. Conceptually, the method may be envisaged as preventing a system from executing an action recommended by an RL agent in a given state when it is known that the execution of a substantially identical action in a substantially identical state previously violated one or more safety constraints. For example, a state-action pair comprising an action recommended by the RL agent may be compared to a stored state-action pair which resulted in a violation to see if the two are identical. However, when deployed or training in real-world environments and / or simulated environments with complicated dynamics, an RL agent is likely to encounter high- dimensional states and / or states of a continuous state-space. Therefore, it may happen that a state newly encountered by the RL agent will not be identical to any state previously encountered. This is problematic if a system is to be prevented from repeating unwanted and / or unsafe actions by comparing states and actions of the RL agent with those known to have resulted in violations. For example, a state-action pair may comprise a state of the environment newly encountered by the RL agent and an action recommended by the RL agent based on that state. If this state-action pair is similar (but not identical) to a state-action pair known to result in a violation (e.g., values representing the states and actions of the respective state-action pairs are not exactly the same), the system will not recognise that executing the recommended action is likely to result in a repeat of the violation. In order to overcome this and other issues, embodiments of the present disclosure enable a management node to prevent a system from executing actions which result in violations of safety rules and / or constraints based on a measure of similarity between a proposed state-action pair and a state-action pair that is known to result in a violation. Figure 1 is a schematic diagram illustrating a communication network 100 according to embodiments of the disclosure. In some embodiments, the communication network100 may be a self-organizing network (SON), where one or more parameters associated with the network 100 are determined by entities of the network 100 autonomously. In the illustrated embodiment, the communication network 100 comprises a radio access network (RAN), which includes one or more RAN nodes 104, and a core network 102, which includes one or more core network nodes. As used herein, the term “RAN node” refers to equipment capable, configured, arranged and / or operable to communicate directly or indirectly with a wireless device or user equipment (UE) and / or with other network nodes or equipment, in a telecommunication network. Examples of network nodes include, but are not limited to, access points (APs) (e.g., radio access points), base stations (BSs) (e.g., radio base stations, Node Bs, evolved Node Bs (eNBs) and New Radio NodeBs (gNBs)). The RAN node serves one or more UEs 106 in a serving cell 108. The network node 104 facilitates direct or indirect connection of UEs in its serving cell 108, such as by connecting the UE 106 to the core network 102 over one or more wireless connections. The core network nodes may be structured with hardware and software components. Example core network nodes include functions of one or more of a mobile switching centre (MSC), mobility management entity (MME), home subscriber server (HSS), access and mobility management function (AMF), session management function (SMF), authentication server function (AUSF), subscription identifier de-concealing function (SIDF), unified data management (UDM), security edge protection proxy (SEPP), network exposure function (NEF), and / or a user plane function (UPF). According to embodiments of the disclosure, the core network 102 comprises a management architecture 116 that enables the safe usage of RL to adapt, control and / or manage one or more aspects of the network 100. For example, in one embodiment, the management architecture 116 is configured to determine an action to be executed or initiated by the RAN node 104 (e.g., a change to a downtilt of an antenna of the RAN node 104, or a change in an UL / DL transmit power), such that the shape or parameters of the serving cell 108 may be adjusted. This in turn allows for the performance of the network 100 to be optimised. The management architecture 116 may be implemented using any one or more of the core network nodes discussed above, or alternatively in one or more servers attached to the core network 102. In a further alternative embodiment, the management architecture 116 may be implemented in the RAN, e.g., in the RAN node 104 itself, or another RAN node. For example, the management architecture 116 may be implemented using a service management and orchestration (SMO) system or a non- real time radio intelligent controller (non-RT-RIC). In the illustrated embodiment, the management architecture 116 comprises an RL agent 110, a safe baseline agent 112, and a management node 114. These aspects are discussed in more detail with respect to Figures 2 and 3. Figure 2 is a schematic diagram illustrating an architecture 200 and processing flow in accordance with embodiments of the present disclosure. Figure 3 is a flowchart of a method in accordance with embodiments of the present disclosure which may be performed, at least in part, by one or more components of the architecture 200. The architecture 200 comprises: a violation log 202, an RL agent 204, a safe baseline agent 206, a management node 208, and an environment 210. Those skilled in the art will appreciate that one or more of the entities shown in Figure 2 may be combined or implemented using the same physical hardware. For example, the management node 208 may comprise one or more other components of the architecture 200, such as the RL agent 204 and / or the violation log 202. The environment 210 is the device or collection of devices in which actions recommended by the RL agent 204 are to be executed, as well as the current situation, condition and / or circumstances of that device or collection of devices. For example, the environment 210 may comprise a telecommunications network, such as the communication network 100 described above. The environment 210 may be “live” or simulated, e.g., a live telecommunications network or a simulated telecommunications network. Reference is made herein to the “state” of the environment 210. The state of the environment refers to the current situation, condition and / or circumstances of the environment, and may include one or more of the environment’s configuration, the presence and / or position (within physical or radio space) of entities within the environment, requests currently being made of the environment, availability and / or requirements being made of resources within the environment, condition of such resources, etc. The state may thus be represented by observations, which may include values of configurable parameters for the environment and / or its contents and / or its resources, values of measurable parameters for the environment and / or its contents and / or resources, demands being made upon the environment, entities present within the environment, etc. In some embodiments, the state of the environment 210 may comprise or be represented by a multi-dimensional vector, with each dimension of the vector corresponding to a parameter of the environment 210. The parameter may be defined by a numerical value, or a category, for example. In the latter case, the category may be encoded to a numerical value to enable easier processing of the data. In some embodiments, the vector may have a large number of dimensions (e.g., more than 10). The RL agent 204 is configured to recommend, based on its policy and a current state (^^^^^^^^) of the environment 210, an action (^^^^^^) to be executed in the environment 210. The current state and recommended action form a first state-action pair: {^^^^^^, ^^^^^^^^}. In some embodiments, the RL agent 204 is further configured to determine the state of the environment 210. For example, this may involve the RL agent 204 interacting with the environment 210 in order to sample the current state of the environment 210 (e.g., by initiating a measurement process by which one or more parameters defining the current state of the environment 210 may be measured). In other embodiments, the RL agent 204 may be configured to receive information indicating a current state of the environment 210 (e.g., from a system operating in the environment 210, the management node 208, etc). Recommendation of the action may involve the use of an RL algorithm (e.g. a deep q- learning (DQN) algorithm) to learn the actions which are predicted to maximize a long- term reward signal based on exploration of the state-action space of the environment 210 and measurement of the changing reward signal. Based on this learning, the policy implemented by the RL agent 204 can be updated such that the RL agent 204 performs better and recommends actions that achieve higher rewards. One example of a DQN algorithm is presented in “Playing atari with deep reinforcement learning”, Mnih, V. et al., (2013), arXiv:1312.5602 (see algorithm 1). Those skilled in the art will appreciate that many RL algorithms are known, and the present disclosure is not limited to any particular algorithm. The management node 208 is communicatively coupled to the environment 210 and, in some embodiments, may be implemented within the environment 210 itself. For example, where the environment 210 comprises a telecommunications network, the management node 208 may be connected to or part of that network 210. In the former case, the management node may be implemented in a server which is connected to the network 210. In the latter case, the management node may be implemented within a network node or as part of a network function, such as a core network node or function (e.g., operations, administration and maintenance (OAM)). The action that is recommended by the RL agent 204 may be executed by the management node 208 directly, or it may be executed by another system operating in the environment 210 (e.g., a RAN node 104), e.g., upon instruction from the management node 208. According to embodiments of the disclosure, the violation log 202 stores a data set which comprises a plurality of state-action pairs known to have resulted in violations of one or more safety constraints that apply to the environment 210. That is, each state-action pair of the data set may comprise a state of the environment 210 and a corresponding action executed in the environment 210 based on that state which resulted in a violation of one or more safety constraints. Examples of violations of safety constraints may include any one or more of the following events: - the performance of the RL agent being negatively impacted to such an extent that it may be difficult for the RL agent and / or the environment to recover; - a negative or low reward yield being obtained when executing an action in the environment; - one or more performance metrics of the RL agent or the environment being degraded or deteriorated such that they no longer meet acceptable performance thresholds; - an executed action having to be revised; - the RL agent entering a dead-end learning trajectory; and - the RL agent needing to be reset. The violation log 202 may be created based on prior domain knowledge and / or training of an RL agent. For example, states, actions, and rewards encountered by an RL agent at each timestep during its training or deployment (which may be referred to as “interaction data” herein) may be recorded in an initial dataset. The RL agent may be the RL agent 204 of architecture 200, or it may be a different RL agent which previously interacted with the environment 210. From the initial dataset, state-action pairs that resulted in a violation of one or more safety constraints may be identified (automatically or by a user) and classified as violations. Any state-action pairs classified as violations may be added to the violation log 202 (e.g., automatically, according to a rule of the violation log 202, or by a user). In another example, a guard cell level key performance indicator (GCLK) module may operate alongside a SON optimization manager (SON OM) and may be responsible for automatically reverting a change made to a network if any accessibility, retainability, integrity and mobility (ARIMA) key performance indicators (KPIs) degrade. The one or more safety constraints may therefore relate to KPI degradation and / or the occurrence of action reversion. A violation of these safety constraints may occur when an executed action causes a GCLK module to command an action reversion. In some embodiments, the violation log 202 may be updated after its initial creation (e.g., to create an updated violation log 212). For example, if the RL agent is deployed in a live network and further violations of safety constraints occur (e.g., by a network operator), state-actions pairs which resulted in these violations may be added to the violation log 202. In Figure 2, the violation log 202 is illustrated as being stored locally in the management node 208. However, in other embodiments, the violation log 202 may be stored elsewhere in the architecture 200, such as in the RL agent 204, a local or cloud based server, and / or in a core network node. The safe baseline agent 206 is configured to provide one or more alternative actions to the management node 208, in the event that an action recommended by the RL agent 204 is found to be unsafe and / or likely to result in a violation of one or more safety constraints. Further detail regarding this component of the architecture 200 is set out below. Turning to Figure 3, in step 302 a first state of a telecommunication environment (e.g. environment 210 of Figure 2) is input into an RL algorithm (e.g., an algorithm employed by RL agent 204). For example, the first state may be obtained by the RL agent 204 itself. Alternatively, the first state may be obtained by one or more other entities (such as the management node 208) and forwarded to the RL agent 204. See the discussion above for a description of the state of an environment. In step 304, a first action is received from (i.e. recommended for execution by) the RL algorithm. The first action and the first state correspond, respectively, to ^^^^^^and ^^^^^^^^, and form a first state-action pair. See the discussion above for the implementation of an RL algorithm in the RL agent 204. The management node 208 receives the first action from the RL agent 204. As noted above, the first action forms a first state-action pair with the first state. According to embodiments of the disclosure, the management node 208 determines, in step 308, whether the first state-action pair is similar to a second state-action pair which is known to have resulted in a violation of one or more safety constraints, with this determination being based on a measure of similarity. That is, the second state-action pair comprises a second state of the environment and a second action that was previously executed based on the second state and resulted in a violation of one or more safety constraints of the environment. Step 308 may comprise determining, based on a measure of similarity, whether the first state-action pair is similar to any of the plurality of state-action pairs comprised in the violation log 202. In some embodiments, this involves the management node 208 being configured to cross-reference (i.e., compare) the first state-action pair with the state-action pairs of the violation log 202. For example, the first state-action pair may be determined to be similar to the second state-action pair based on a measure of similarity which is a distance between the two state-action pairs. In such embodiments, the first state-action pair and the second state-action pair may be represented, respectively, by a first point and a second point in a multi-dimensional geometric space. For example, the first point and the second point may both be co- ordinates of a geometric space (e.g., cartesian co-ordinates). The distance between the two state-action pairs may then be calculated any geometrical distances, including any one of the following: - the L0 norm distance between the first point and the second point; - the L1 norm distance between the first point and the second point; - the L2 norm distance (e.g., the Euclidean distance) between the first point and the second point; and - the L-infinity norm distance between the first point and the second point. Determining whether the first state-action pair is similar to the second state-action pair then comprises comparing the calculated distance between the two state-action pairs to a threshold (which may be pre-configured by a user or network operator). If the distance falls within or is equal to the threshold (e.g., is less than or equal to the threshold), the two state-action pairs may be considered similar (i.e., substantially the same). If the distance is greater than the threshold, the two state-action pairs may not be considered similar. The threshold may be defined by an ε-region located around the first state-action pair, where if the second state-action pair is located within the ε-region, it is considered to be similar (or substantially the same) to the first state-action pair. For example, the threshold may be the radius of the ε-region in a certain dimension. In some embodiments, the size of the radius of the ε-region may be the same in all dimensions. In other embodiments, the size of the radius of the ε-region in each dimension may be dimension-dependent, e.g., such that there are: - larger (i.e., more lenient) thresholds for similarity in dimensions where an executed action results in relatively large changes to the state of the environment; and - smaller (i.e., stricter) thresholds for similarity in dimensions where an executed action results in relatively small changes to the environment. The value of the threshold and / or the size of the ε-region may be determined based on: - the method of calculating the distance between the first state-action pair and the second state-action pair; and - the degree of similarity required between the first state-action pair and the second state-action pair in order for them to be considered substantially the same. A smaller ε-region or a smaller threshold results in a stricter criterion for similarity (i.e., the two state-action pairs must be relatively closer to one another to be considered substantially the same). A larger ε-region or a larger threshold results in a more lenient criterion for similarity (i.e., the two state-action pairs can be further apart from one another and still be considered similar or substantially the same). Since states of the environment may have a large number of dimensions (e.g., defined by many different parameters, such as geometric parameters and / or cell KPIs), the value of the threshold or the size (e.g., radius) of the ε-region may need to be tuned in order to find a suitable criterion for similarity. A suitable criterion for similarity is one that is strict enough that known violations are prevented from re-occurring but lenient enough that the RL agent can still perform exploration and exploitation to optimise the performance of the environment 210. In some embodiments, this tuning may be performed by treating the size of the threshold / ε-region as a hyperparameter of the RL agent 204. It may therefore be tuned via any suitable method known to the skilled person for tuning a hyperparameter of an RL agent. For example, multiple iterations of the method of Figure 3 may be performed during a training phase and / or based on a simulated environment 210, whilst adjusting the threshold / ε-region in order to determine an appropriate threshold. In an alternative embodiment, step 308 may comprise inputting the first state-action pair into a machine learning (ML) model trained to classify state-action pairs. In this embodiment, the output received from the ML model is the measure of similarity and indicates whether or not the first state-action pair is similar to the second state-action pair. For example, the ML model may be trained, based on the previously discussed violation log 202, to classify state-action pairs as, for example, “likely to result in a violation” or “not likely to result in a violation”. In one example, the ML model may be trained to learn a contrastive latent space representation of the environment 210, such that the ML model is able to separate similar samples (e.g., safe state-action pairs) from dissimilar samples (e.g., unsafe state-action pairs). Once the ML model has been trained to classify state-action pairs, the ML model is able to predict whether an action output from the RL agent 204 is likely to result in a violation. This means that the management node 208 does not need to directly cross-reference the violation log 202 in order to determine whether executing ^^^^^^in ^^^^^^^^would likely result in a violation of one or more safety constraints. Therefore, once the ML model has been sufficiently trained, the violation log 202 does not need to be accessed by the management node 208. This helps conserve network resources (e.g., for storage and transmission) when embodiments of the present disclosure are implemented. Embodiments using a ML model to determine whether a first state-action pair is similar to a second-state action pair may be particularly useful in one or more of the following scenarios: - where a trained RL agent is to be generalized by being applied to environments different to the one on which it was trained; and - where there is noise in the environment that results in partially observable scenarios / states. Determining whether the first state-action pair is similar to the second state-action pair (which is known to result in a violation) enables the management node 208 to prevent unwanted and / or unsafe actions from being repeated. The management node 208 is also enabled to initiate execution of only safe actions (^^^^^) in the environment 210. If the management node 208 determines that the first-state action pair is not similar to the second state-action pair, in step 310 the management node 208 enables or initiates the execution of ^^^^^^in the environment 210. That is, the management node 208 may determine that ^^^^^= ^^^^^^. ^^^^^^may be executed, for example, by the management node 208 itself or a system operating in the environment 210 (e.g., a RAN node 104). In step 312, if ^^^^^^is executed in the system, the management node 208 may determine whether this executed action results in a violation of one or more safety constraints. For example, the management node 208 may receive feedback information from the system which executed ^^^^^^and / or the environment 210. If ^^^^^^is determined to be a violation when executed based on the state ^^^^^^^^, in step 314 the management node 208 updates the violation log 202 such that it includes the first state- action pair {^^^^^^, ^^^^^^^^} (see updated violation log 212 in Figure 2). If executing the first action is determined not to result in a violation of one or more safety constraints, the violation log is not updated to include the first state-action pair. The method then returns to step 302, and is iterated using a new state (e.g., the state of the environment 210 in a subsequent timestep). If the first-state action pair is determined in step 308 to be similar to the second state- action pair, the management node 208 refrains from executing ^^^^^^in the environment 210 (e.g., by having a system operating in the environment 210 refraining from executing ^^^^^^). The management node 208 may also determine and execute, in step 320, a third action that is different from the first action ^^^^^^and the second action. The third action may be known, to some degree of certainty, to be safe (i.e., it can be set as ^^^^^). The safe baseline agent 206 may be configured to provide the third action to the management node 208. For example, the safe baseline agent 206 may determine, based on ^^^^^^^^and a rule-based algorithm and / or existing domain knowledge, an action (^^^^^^^^^) to be executed in the environment 210. As the underlying mechanisms of the safe baseline agent 206 used to determine ^^^^^^^^^may be preconfigured, studied, and / or tested, the safe baseline agent 206 is not a black box like the RL agent 204. Therefore, the actions recommended by the safe baseline agent 206 can be assumed to cause little or no deterioration or degradation to the state of the environment 210. As such, the actions it recommends can be considered “safe”. Thus, in some embodiments, the management node 208 may determine that ^^^^^= ^^^^^^^^^. Alternatively or additionally, in some embodiments, the third action may be a fallback (i.e., a default) action, ^^^^^^^^^, which is preconfigured in the architecture 200 or in a system operating in the environment 210. For example, ^^^^^^^^^may comprise an action which, when implemented, causes no change to a parameter being tuned (e.g., no change to an antenna tilt or UL / DL transit power). That is, the default action may be “no action”. Thus, in some embodiments, the management node 208 may determine that ^^^^^= ^^^^^^^^^. Embodiments of the disclosure are described above in the context of individual state- action pairs. This concept may be extended to sequences of state-action pairs, i.e., states and corresponding actions that were taken in a sequence of timesteps. Thus, the first state-action pair may belong to a first sequence of state-action pairs, the first sequence comprising a plurality of first states of the environment 210 and corresponding first actions previously executed based on the first states. The second state-action pair may belong to a second sequence of state-action pairs, the second sequence comprising a plurality of second states of the environment and corresponding second actions previously executed based on the second states. Executing one or more (or all) of the actions comprised in the second sequence resulted in a violation of one or more safety constraints of the environment 210. The second sequence may be stored in the violation log 202. For example, if a violation is known to have occurred at time ^, instead of storing just the state-action pair {^^, ^^} in the violation log 202, the ^ previous state-action pairs … , {^^, ^^}] which lead to {^^, ^^} may also be stored in the violation log 202. The management node 208 may then determine whether the first state-action pair or the first sequence is similar to the second sequence based on a measure of similarity (e.g., as part of step 308 of Figure 3). This may involve determining whether the first state action pair or one or more or all of first sequence are similar to one or more or all of the second sequence based on a measure of similarity. Any of the above discussed methods for determining similarity may be employed. If it is determined that the first sequence of state-action pairs is similar to the second sequence of state-action pairs, in full or in part, the management node 208 may be configured to prevent the first action from being executed. As such, in these embodiments, the management node 208 is able to determine whether the RL agent is recommending a sequence of actions which are similar to a chain of state-action pairs known to have resulted in at least one violation. The management node 208 is therefore able to pre-emptively prevent a similar chain of actions from being executed in the environment 210. Use Cases A telecommunication network (e.g., a RAN) may comprise hundreds or thousands of cells. Such networks face the challenge of satisfying QoS requirements from UEs, which may vary considerably over time in individual cells. In order for the network to efficiently provide a high QoS to UEs, the network should aim to adjust (or “tune”) its configuration in an automatic and timely manner. To this effect, each cell has a number of parameters which can be adjusted or tuned (e.g., by a base station) to optimise network performance (e.g., cell coverage, cell capacity, etc) within the cell. A base station can monitor a cell’s network performance via KPIs observed within the cell (e.g., cell throughput, edge cell signal strength, etc). However, network operators are usually interested in global KPIs (e.g., throughput averaged over multiple or all cells of a network, etc), such that they can ensure an adequate network performance is experienced by a large proportion of network users. Thus, network configuration often aims at optimising global KPIs rather than KPIs of individual cells. This may be achieved through having a RAN act as a self- organising network (SON), with the self-organizing being achieved via the use of RL agents. In addition to the use cases of tuning RET and UL / DL transmit power that are discussed below, embodiments of the present disclosure may be applied when tuning other cell parameters (e.g., cell individual offsets (CIO), cell azimuths, beam widths, and cell shape indices). Use Case 1: Remote Electronic / Digital Tilt Antenna vertical tilt angle, sometimes referred to as a downtilt angle, is one of the most important variables of an antenna system to control in terms of QoS management. This control can be either mechanical or electronic in nature, but owing to the cost associated with manually adjusting the downtilt angle, RET optimisation is used in the vast majority of modern networks. There exist several KPIs that may be taken into consideration when evaluating the performance of a RET optimization strategy, and significant existing material is dedicated to coverage capacity optimization (CCO). In CCO, a coverage KPI relates to area covered in terms of a minimum received signal strength, and a capacity KPI relates to an average total throughput in a given area of interest. Control of the RET involves a trade-off between coverage and capacity: an increase in antenna downtilt (^) correlates with a stronger signal in a more concentrated area, improving the interference plus noise ratio (SINR) for UEs in that area. It is also correlated with higher capacity and reduced interference radiation towards other cells in the network. However, excessive downtilting can result in insufficient coverage in a given area (e.g., the outer edges of a cell), with some UEs unable to receive a minimum signal quality. This is exemplified by Figure 4, which illustrates how neighbouring first and second cells 406a, 406b (served by RAN nodes 402a, 402b) may overlap one another’s coverage area. The resulting signal interference may impact the network performance experienced by first and second UEs 404a, 404b served by each cell. Through adjusting a downtilt angle, ^, of the antenna(s) of each RAN node 402a, 402b, the shape of each cell 406a, 406b may be tuned such that the combined network performance of both cells is optimised. This may involve the network performance being improved for the first UE 104a within the first cell 406a, but degraded for the second UE 404b in the second cell 406b. Methods according to embodiments of the present disclosure can enable the safe deployment of RL Agents for RET optimisation and are discussed in detail below. For this scenario, the elements of the above discussed architecture may be: System: A physical base station (e.g., a RAN node 104) with at least one radio antenna. Actions which may be implemented by the system comprise possible discrete changes to the antenna downtilt. For example, only three actions may be available: uptilt, downtilt, and no change (i.e., ^^∈ [−Δ^, Δ^, 0]). In other examples, additional actions may be available, including uptilts and downtilts of differing amounts. Telecommunication environment: The physical 4G or 5G mobile cellular network area considered for RET optimisation. The state of the environment may be determined by one or more of: cell congestion, cell overlapping, cell interference, cell overshooting, and low reference signal received power (RSRP) at a cell edge. The reward returned for executing a particular action may favour actions resulting in any one of: traffic with good RSRP; traffic with good SINR; and reduced congestion. RL algorithm: An RL algorithm employed by an RL agent whose policy takes as input the state of the mobile cellular network. For example, the RL Agent’s policy may be a DQN. Safety constraints: constraints imposed on the RL agent and / or the system to ensure the safety of the telecommunications environment. The constraints may be defined by a user and / or a network operator and may relate to the performance of the telecommunications environment (e.g., a QoS or QoE metric) and / or the performance of the RL agent (e.g., a minimum threshold reward). Violation log: A data set comprising state-action action pairs (i.e. tuples) which are known to comprise actions which have resulted in violations of the one or more safety constraints. Rule-based algorithm: A rule-based algorithm employed by a safe baseline agent which utilises existing network domain knowledge to recommend an action, given the state of the mobile cellular network area. One example of such a rule-based algorithm is discussed in detail in “Self-tuning of remote electrical tilts based on call traces for coverage and capacity optimization in LTE”, V. Buenestado et al., IEEE Transactions on Vehicular Technology, vol.66, no.5, pp.4315-4326, May 2017. This algorithm can be used to iteratively compute suggested changes to an antenna tilt angle using cell traces. The algorithm is designed to optimise the trade-off between cell coverage, cell quality, and cell capacity. For example, whilst large cell overlapping often leads to better coverage, the connection quality may be degraded and cell overshooting may be increased. Cell overshooting of a particular cell occurs when a signal strength is measured (for that cell) in distant cells which is close in strength to the signals of their strongest neighbours. Cell overshooting therefore results in useless cell overlap, since cell coverage is not improved and interference level is increased. To monitor this trade-off, the algorithm first uses an overshooting indicator, ^^^(^), and a useless overlapping indicator, ^^^(^), for a particular cell under study, ^, as inputs to a first fuzzy logic controller (FLC). A fuzzy output value of the first FLC, ^^(^), quantifies useless interference generated by cell ^, and ranges from 0 (no interference) to 1 (large interference). ^^^(^) is calculated as where ^(^) is the set of neighbors of cell ^; ^^^(^, ^) is an adjacency-level indicator showing whether cell ^ is a relevant neighbour of cell ^, ^^(^) is the total number of RSRP samples reported by users in cell ^; ^^^^^(^, ^) (^^^ for useless overlapping from a non- relevant cell) is the number of samples where the RSRP level difference between the serving cell and neighbour cell ^ is lower than a pre-defined threshold, ∆^^^^^^^^, and cell ^ is not the strongest neighbour; and ^^^^is a scaling factor between 0 and 1 that defines the ratio of samples needed to consider the source cell ^ as a strong interferer of adjacent cell ^. The min() operator ensures that a neighbour cell ^ strongly interfered by cell ^ only adds 1 to ^^^(^). The parameter ^^^(^, ^) is defined as where ^^^^(^, ^)(^^ for relevant neighbour) is the number of samples with RSRP values from neighbour cell ^ higher than the rest of neighbour, and ^^^is the ratio of relevant samples to consider the source cell ^ as a totally relevant neighbour of adjacent cell ^. ^^^(^) is calculated as where ^^^^(^, ^) (^^ for useless overlapping) is the number of samples where RSRP difference between serving cell ^ and neighbour cell ^ is lower than a pre-defined threshold, ∆^^^^^^^^, and the signal level from the serving cell ^ is larger than a threshold, ^^^^^^^^^^, and ^^^is a scaling factor between 0 and 1 defining the ratio of low dominance samples to consider source cell ^ as a strong interferer of adjacent cell ^. Again, the min() operator ensures that a neighbour with a large area of useless cell overlapping with cell ^ only adds 1 to ^^^(^). The first FLC, taking ^^^(^)and ^^^(^) as inputs, has the following rules: The algorithm then uses ^^(^), the output of the first FLC, and a bad coverage indicator, ^^^(^), as inputs to a second FLC. A suggested tilt change, ∆^(^), is output from the second FLC, which may be positive (downtilt) or negative (uptilt). ^^^(^) is calculated as where ^^^^^(^) is the number of RSRP samples below a certain threshold, ^^^^^^^^^, reported by cell-edge UEs of cell ^, and ^^^(^) is the total number of samples reported by cell edge UEs of cell ^. The second FLC, taking ^^(^) and ^^^(^) as inputs, has the following rules: The suggested tilt change output from the second FLC is therefore known to either improve or maintain the cell coverage, cell quality, and cell capacity of cell ^. This suggested tilt change output may be provided to the system and may be used as the alternative action if a state-action pair comprising an action recommended by the RL agent is determined to be similar to a state-action pair known to comprise a violation. Fallback or default action: an action which causes no change to the antenna tilt (i.e., ^^= 0). This action may be preconfigured in the system. Results of Implementation To demonstrate the effectiveness of using a management node according to embodiments of the present disclosure in the RET optimisation use case, an example is considered in which cells of a network act independently to optimise their antenna tilt based on their own measurements and those of neighbouring cells. More specifically, a simulated scenario of twenty-one cells implemented in a high-fidelity system level simulator is considered. The simulator uses ray tracing propagation models to model a Long Term Evolution (LTE) network. In this example, an RL agent (specifically, a DQN agent) is deployed in each cell in the network and iteratively recommends, after each time step, an action to be executed by a base station serving that cell. The RL agents are to explore and train in their respective cells. The action space of each RL agent is discrete and the available actions are ^^∈ [Δ^, 0, −Δ^], which correspond to “downtilt, no change, uptilt”. If the RL agent does not change the antenna tilt after multiple consecutive iterations / time steps, the simulation is terminated. In this example, the safety constraint imposed on the LTE network is that actions which have a reward yield less than 0.2 should not be executed. First, the simulated scenario is initialised using a first simulation seed. The RL agents do not use management nodes according to embodiments of the present disclosure. That is, the RET of each base station is tuned without use of a mechanism explicitly preventing the base stations from executing actions recommended by the RL agents which are known (based on previous observations) to result in a violation of the safety constraint when executed in the state of the environment. Interaction data is collected from the deployed RL agents after each timestep / iteration. The interaction data includes the states input into the RL agents, the actions recommended by the RL agents based on the states, and the returned reward for executing the RL agents’ recommended actions. After the simulated scenario ends, the interaction data is used to identify any state-action pairs which lead to a negative reward, with these state-action pairs being classified as being known to result in a violation of the safety constraint. The state-action pairs classified as being known to result in a violation of the safety constraint are stored in a violation log. After the violation log is created, the simulated scenario is re-initialized multiple times (again using the first simulation seed). During each re-initialization, the same RL agents as discussed above are deployed in the same environment, but this time with the RL agents utilising management nodes according to embodiments of the present disclosure. That is, they are capable of cross-referencing the violation log to determine whether any action recommended by the RL agent results in a violation when executed in the state of the environment. In each re-initialization, the management nodes use a similarity threshold of 0.05, 0.2, and 0.5, respectively. Therefore, in each re-initialization, the management nodes have different levels of strictness. Interaction data is also collected during the re-initializations. For each initialization, the reward accumulated by each deployed RL agent during intervals of 20 timesteps (that form 1 iteration) is calculated (using the collected interaction data). The accumulated reward of the different RL agents in a given interval of 20 steps is then averaged, such that an average reward accumulated by a deployed RL agent in a given initialization can be observed with respect to the number of iterations / timesteps that have elapsed. Indeed, Figure 5 illustrates, for each initialization, the average reward accumulated by an RL agent after intervals of 20 time steps for a total of 10 iterations (200 timesteps). The same seed was used in each initialization, as characteristics of the environment (e.g., antenna heights) are initialized based on the simulation seed. Therefore, using different seeds for a simulation can result in different network topologies, meaning the effect of the management node on the performance of the environment / RL agent would be obscured. It is also noted that using the same simulation seed in the present example mirrors use-cases in which a violation log is created using interaction data collected by an RL agent deployed in an environment, with the violation log being used to improve the safety of the RL agent when it is re-deployed in the same environment. The lower graph of Figure 5 was obtained when repeating the above discussed example using a second simulation seed in place of the first simulation seed. As with the upper graph, the lower graph illustrates, for each initialization, the average reward accumulated by an RL agent after intervals of 20 time steps As seen from the two graphs of Figure 5, the highest average reward accumulated during an interval of 20 time steps is achieved when a suitable similarity threshold for the simulation seed is used (which is 0.2 for the top graph and 0.05 for the bottom graph). Furthermore, it can be seen that the RL agents which are capable of cross referencing the violation log have smaller negative reward peaks in comparison to the negative reward peaks for the RL agent which cannot cross reference the violation log. Lastly, it can be observed in both graphs of Figure 5 that overly strict similarity thresholds (e.g., 0.5) do not allow many actions recommended by the RL agent to be executed, leading to very small, or zero, rewards and premature termination of the simulation. Use Case 2: UL / DL transmit power Another method of tuning cell parameters involves tuning the power of signals transmitted within a cell. For example, by increasing the transmit power of an UL or DL signal in a particular cell, the SINR in the cell may increase (due to the increased signal strength). However, as a result, the SINR in surrounding cells may decrease (due to increased signal interference). For this scenario, the elements of the above discussed architecture may be: System: A physical base station with at least one radio antenna. Actions which may be implemented by the system comprise possible discrete changes to the UL or DL transmit power. In the present example, only three actions may be available: increase the UL / DL transmit power, decrease the UL / DL transmit power, or cause no change to the UL / DL transmit power. Again, however, additional actions may define differing amounts by which the UL / DL transmit power may be increased or decreased. Telecommunication environment: The physical mobile cellular network area (e.g., 4G or 5G). The state of the environment may be determined by at least any one of: cell congestion, cell overlapping, cell interference, cell overshooting, and low RSRP at cell edge. The reward returned for executing a particular action may favour actions resulting in any one of: traffic with good RSRP; traffic with good SINR; and reduced congestion. RL algorithm: An RL algorithm employed by an RL agent whose policy takes as input the state of the mobile cellular network. The RL Agent’s policy may be a DQN (e.g., as provided in algorithm 1). Safety constraints: constraints imposed on the RL agent and / or the system. The constraints may be defined by a user and / or a network operator and may relate to the performance of the telecommunications environment (e.g., a QoS or QoE metric) and / or the performance of the RL agent (e.g., a minimum threshold reward). Violation log: A data set comprising state-action action pairs (i.e. tuples) which are known to comprise actions which have resulted in violations of the one or more safety constraints. Rule-based algorithm: An algorithm which utilises existing network domain knowledge to recommend an action, given the state of the mobile cellular network area. Those skilled in the art will be able to formulate many suitable rule-based algorithms for the control and / or setting of an UL / DL transmit power. For example, the UL / DL transmit power for a particular cell may be controlled as a function of one or more of: the particular cell’s throughput; the throughput of one or more cells neighbouring the particular cell; radio measurements on the particular cell and the neighbouring cell(s) performed by UEs served by the particular cell. The UL / DL transmit power value for the particular cell may be increased by an amount where the throughput for the particular cell falls below a threshold, and / or where the throughput for the particular cell is lower than the throughput of neighbouring cells by a threshold. In this way, the UL / DL transmit power may be increased so as to increase the particular cell’s throughput. Fallback or default action: an action which causes no change to the UL / DL transmit power (i.e., ^^= 0). This action may be preconfigured in the system. Figure 6 illustrates a schematic block diagram of an apparatus 600 for managing a system operative in a telecommunication environment. Apparatus 600 is operable to carry out the example method described with reference to Figure 3 and possibly any other processes or methods disclosed herein. It is also to be understood that the method of Figure 3 is not necessarily carried out solely by apparatus 600. At least some operations of the method can be performed by one or more other entities. Apparatus 600 comprises processing circuitry 604, which may include one or more microprocessor or microcontrollers, as well as other digital hardware, which may include digital signal processors (DSPs), special-purpose digital logic, and the like. The processing circuitry 604 may be configured to execute program code stored in memory 602, which may include one or several types of memory such as read-only memory (ROM), random-access memory, cache memory, flash memory devices, optical storage devices, etc. Program code stored in memory 602 includes program instructions for executing one or more telecommunications and / or data communications protocols as well as instructions for carrying out one or more of the techniques described herein, in several embodiments. In some implementations, the processing circuitry 604 may cause the apparatus 600 to perform corresponding functions according one or more embodiments of the present disclosure. According to embodiments of the disclosure, the processing circuitry 604 may be configured to cause the apparatus 600 to: provide a first state of a telecommunication environment as an input to a reinforcement learning algorithm; receive, from the reinforcement learning algorithm, an output comprising a first action that is recommended for execution by a system, the first state and the first action forming a first state-action pair; and determine whether the first state-action pair is similar to a second state-action pair based on a measure of similarity, where the second state-action pair comprises a second state of the telecommunication environment and a second action that was previously executed based on the second state and resulted in a violation of one or more safety constraints of the telecommunication environment. The processing circuitry 604 may be configured to further cause the apparatus 600 to, in response to determining that the first state-action pair is similar to the second state-action pair, refrain from executing the first action. The apparatus 600 may be implemented in a node of a communication network, such as a radio network, an optical network, or an electronic network. Thus, the apparatus 600 further comprises one or more interfaces 606 with which to communicate with one or more other nodes of the communication network. The interface(s) 606 may therefore comprise hardware and / or software for transmitting and / or receiving one or more of: radio signals; optical signals; and electronic signals. In alternative embodiments, the apparatus 600 may comprise one or more units or modules configured to perform the steps of the method, for example, as illustrated in Figure 3. In such embodiments, the apparatus 600 may comprise a providing unit, a receiving unit, and an execution unit. The providing unit may be configured to provide a first state of a telecommunication environment as an input to a reinforcement learning algorithm. The receiving unit may be configured to receive, from the reinforcement learning algorithm, an output comprising a first action that is recommended for execution by a system, the first state and the first action forming a first state-action pair. The determination unit may be further configured to determine whether the first state-action pair is similar to a second state-action pair based on a measure of similarity, where the second state-action pair comprises a second state of the telecommunication environment and a second action that was previously executed based on the second state and resulted in a violation of one or more safety constraints of the telecommunication environment. The execution unit may be configured to, in response to determining that the first state-action pair is similar to the second state-action pair, refrain from executing the first action. Embodiments of the present disclosure may be employed in both open RAN (O-RAN) and 3GPP architectures. For example, in O-RAN, the management node 208 could be implemented as an r-app in a non-real time RAN intelligent controller (RIC). It could also be implemented in the same r-app as the RL agent 204, or it could be implemented in a different application that communicates with the RL agent 204. The management node 208 may interact with base stations (e.g., according to the method of Figure 3) using the A1 interface. The term “unit” may have conventional meaning in the field of electronics, electrical devices and / or electronic devices and may include, for example, electrical and / or electronic circuitry, devices, modules, processors, memories, logic solid state and / or discrete devices, computer programs or instructions for carrying out respective tasks, procedures, computations, outputs, and / or displaying functions, and so on, as such as those that are described herein. The methods of the present disclosure may be implemented in hardware, or as software modules running on one or more processors. The methods may also be carried out according to the instructions of a computer program, and the present disclosure also provides a computer readable medium having stored thereon a program for carrying out any of the methods described herein. A computer program embodying the disclosure may be stored on a computer readable medium, or it could, for example, be in the form of a signal such as a downloadable data signal provided from an Internet website, or it could be in any other form. It should be noted that the above-mentioned examples illustrate rather than limit the disclosure, and that those skilled in the art will be able to design many alternative embodiments without departing from the scope of the appended embodiments. The word “comprising” does not exclude the presence of elements or steps other than those listed in an embodiment or claim, “a” or “an” does not exclude a plurality, and a single processor or other unit may fulfil the functions of several units recited in the embodiments. Any reference signs in the claims shall not be construed so as to limit their scope.

Claims

CLAIMS 1. A computer implemented method for managing a system operative in a telecommunication environment (210), wherein the system executes one or more actions recommended by a reinforcement learning algorithm based on a state of the telecommunication environment (210), the method comprising: - providing (302) a first state of the telecommunication environment as an input to the reinforcement learning algorithm and receiving (304), from the reinforcement learning algorithm, an output comprising a first action that is recommended for execution by the system, the first state and the first action forming a first state-action pair; - determining (306, 308) whether the first state-action pair is similar to a second state-action pair based on a measure of similarity, wherein the second state-action pair comprises a second state of the telecommunication environment (210) and a second action that was previously executed based on the second state and resulted in a violation of one or more safety constraints of the telecommunication environment (210); and - in response to determining that the first state-action pair is similar to the second state-action pair, refraining (318) from executing the first action.

2. The method according to claim 1, comprising: - in response to determining that the first state-action pair is not similar to the second state-action pair, executing (310) the first action.

3. The method according to any one of claims 1 or 2, comprising: - in response to determining that first state-action pair is similar to the second state-action pair, executing (320, 322) a third action which is different to the first action and / or the second action.

4. The method according to claim 3, wherein the third action is pre-configured in the system.

5. The method according to claim 3, wherein the third action is determined using a rule-based algorithm based on the first state.

6. The method according to any one of the preceding claims, wherein the measure of similarity comprises a distance between the first state-action pair and the second state-action pair, and wherein determining whether the first state-action pair is similar to the second state-action pair comprises: - comparing the distance to a threshold.

7. The method according to any one of claims 1 to 5, wherein determining whether the first state-action pair is similar to the second state-action pair comprises: - inputting the first state-action pair to a machine learning model trained to classify state-action pairs; and - receiving, as the measure of similarity, an output from the machine learning model indicating whether or not the first state-action pair is similar to the second state-action pair.

8. The method according to claim 7, wherein the second state-action pair belongs to a data set comprising a plurality of state-action pairs, each state-action pair comprising a respective state of the telecommunication environment (210) and a corresponding action executed based on the state which resulted in a violation of the one or more safety constraints of the telecommunication environment; and wherein the machine learning model is trained, based on the data set, to classify state-action pairs comprising actions that resulted in a violation of the one or more safety constraints.

9. The method according to any one of claims 1 to 6, wherein the second state- action pair belongs to a data set comprising a plurality of state-action pairs, each state-action pair comprising a respective state of the telecommunication environment (210) and a corresponding action executed based on the state which resulted in a violation of the one or more safety constraints of the telecommunication environment; and wherein determining whether the first state-action pair is similar to the second state-action pair comprises determining whether the first state-action pair is similar to any of the plurality of state-action pairs in the data set.

10. The method according to any one of to any one of claims 8 or 9, further comprising:- executing (310) the first action; and - in response to determining (312) that executing the first action results in a violation of the one or more safety constraints of the telecommunication environment, updating (314) the data set to include the first state-action pair.

11. The method according to any one of the preceding claims, wherein the first state-action pair belongs to a first sequence of state-action pairs comprising a plurality of first states of the telecommunication environment (210) and corresponding first actions executed based on the first states, and wherein the second state-action pair belongs to a second sequence of state-action pairs comprising a plurality of second states of the telecommunication environment (210) and corresponding second actions executed based on the states, wherein the second sequence of state-action pairs resulted in a violation of the one or more safety constraints of the telecommunication environment (210).

12. The method according to claim 11, comprising: - determining whether the first sequence of state-action pairs is similar to the second sequence of state-action pairs based on a measure of similarity.

13. The method according to claim 12, wherein determining whether the first sequence of state-action pairs is similar to the second sequence of state-action pairs comprises: - determining whether one or more of the first sequence of state-action pairs are similar to one or more of the second sequence of state-action pairs based on a measure of similarity.

14. The method according to any one of claims 12 or 13, comprising: - in response to determining that the first sequence of state-action pairs is similar to the second sequence of state-action pairs, refraining from executing the first action.

15. The method according to any one of the preceding claims, wherein the system comprises at least one antenna, and the one or more actions executed by the system are for controlling a tilt of the at least one antenna.

16. The method according to claim 15, wherein the one or more actions comprise at least one of: - an uptilt of the at least one antenna; - a down tilt of the at least one antenna; and - no change to the tilt of the at least one antenna.

17. The method according to any one of the preceding claims, wherein the system comprises at least one antenna, and the one or more actions executed by the system are for controlling an uplink transmit power or a downlink transmit power.

18. The method according to claim 17, wherein the one or more actions result in at least one of: - an increase in the uplink transmit power or the downlink transmit power; - a decrease in the uplink transmit power or the downlink transmit power; and - no change to the uplink transmit power or the downlink transmit power.

19. The method according to any one of the preceding claims, wherein the telecommunication environment (210) comprises a self-organised network, SON.

20. A management node (208, 600) for managing a system operative in a telecommunication environment (210), wherein the system executes one or more actions which are determined using a reinforcement learning algorithm based on a state of the telecommunication environment (210), the management node comprising processing circuitry (604) configured to cause the management node to: - provide (302) a first state of the telecommunication environment (210) as an input to the reinforcement learning algorithm and receive (304), from the reinforcement learning algorithm, an output comprising a first action that is recommended for execution by the system, the first state and the first action forming a first state-action pair; - determine (306, 308) whether the first state-action pair is similar to a second state-action pair based on a measure of similarity, wherein the second state-action pair comprises a second state of thetelecommunication environment (210) and a second action that was previously executed based on the second state and resulted in a violation of one or more safety constraints of the telecommunication environment; and - in response to determining that the first state-action pair is similar to the second state-action pair, refrain (318) from executing the first action.

21. The management node (208, 600) according to claim 20, wherein the processing circuitry (604) is configured to cause the management node (208, 600) to: - in response to determining that the first state-action pair is not similar to the second state-action pair, execute (310) the first action.

22. The management node (208, 600) according to any one of claims 20 or 21 wherein the processing circuitry (604) is further configured to cause the management node (208, 600) to: - in response to determining that first state-action pair is similar to the second state-action pair, execute (320, 322) a third action which is different to the first action and / or the second action.

23. The management node (208, 600) according to claim 22, wherein the third action is pre-configured in the system.

24. The management node (208, 600) according to claim 2, wherein the third action is determined using a rule-based algorithm based on the first state.

25. The management node (208, 600) according to any one of claims 20 to 24, wherein the measure of similarity comprises a distance between the first state- action pair and the second state-action pair, and wherein determining whether the first state-action pair is similar to the second state-action pair comprises: - comparing the distance to a threshold.

26. The management node (208, 600) according to any one of claims 20 to 24, wherein determining whether the first state-action pair is similar to the second state-action pair comprises: - inputting the first state-action pair to a machine learning model trained to classify state-action pairs; and- receiving, as the measure of similarity, an output from the machine learning model indicating whether or not the first state-action pair is similar to the second state-action pair.

27. The management node (208, 600) according to claim 26, wherein the second state-action pair belongs to a data set comprising a plurality of state-action pairs, each state-action pair comprising a respective state of the telecommunication environment and a corresponding action executed based on the state which resulted in a violation of the one or more safety constraints of the telecommunication environment; and wherein the machine learning model is trained, based on the data set, to classify state-action pairs comprising actions that resulted in a violation of the one or more safety constraints.

28. The management node (208, 600) according to any one of claims 20 to 26, wherein the second state-action pair belongs to a data set comprising a plurality of state-action pairs, each state-action pair comprising a respective state of the telecommunication environment and a corresponding action executed based on the state which resulted in a violation of the one or more safety constraints of the telecommunication environment; and wherein determining whether the first state-action pair is similar to the second state- action pair comprises determining whether the first state-action pair is similar to any of the plurality of state-action pairs in the data set.

29. The management node (208, 600) according to any one of to any one of claims 27 or 28, wherein the processing circuitry (604) is configured to cause the management node to: - execute (310) the first action; and - in response to determining (312) that execute the first action results in a violation of the one or more safety constraints of the telecommunication environment, update (314) the data set to include the first state-action pair.

30. The management node (208, 600) according to any one of claims 20 to 29, wherein the first state-action pair belongs to a first sequence of state-action pairs comprising a plurality of first states of the telecommunication environmentand corresponding first actions executed based on the first states, and wherein the second state-action pair belongs to a second sequence of state-action pairs comprising a plurality of second states of the telecommunication environment and corresponding second actions executed based on the states, wherein the second sequence of state-action pairs resulted in a violation of the one or more safety constraints of the telecommunication environment.

31. The management node (208, 600) according to claim 30, wherein the processing circuitry (604) is configured to cause the management node (208, 600) to: - determine whether the first sequence of state-action pairs is similar to the second sequence of state-action pairs based on a measure of similarity.

32. The management node (208, 600) according to claim 31, wherein determining whether the first sequence of state-action pairs is similar to the second sequence of state-action pairs comprises: - determining whether one or more of the first sequence of state-action pairs are similar to one or more of the second sequence of state-action pairs based on a measure of similarity.

33. The management node (208, 600) according to any one of claims 31 or 32, wherein the processing circuitry (604) is configured to cause the management node (208, 600) to: - in response to determining that the first sequence of state-action pairs is similar to the second sequence of state-action pairs, refrain from executing the first action.

34. The management node (208, 600) according to any one of claims 20 to 33, wherein the system comprises at least one antenna, and the one or more actions executed by the system are for controlling a tilt of the at least one antenna.

35. The management node (208, 600) according to claim 34, wherein the one or more actions comprise at least one of: - an uptilt of the at least one antenna; - a down tilt of the at least one antenna; and- no change to the tilt of the at least one antenna.

36. The management node (208, 600) according to any one of claims 20 to 35, wherein the system comprises at least one antenna, and the one or more actions executed by the system are for controlling an uplink transmit power or a downlink transmit power.

37. The management node (208, 600) according to claim 36, wherein the one or more actions result in at least one of: - an increase in the uplink transmit power or the downlink transmit power; - a decrease in the uplink transmit power or the downlink transmit power; and - no change to the uplink transmit power or the downlink transmit power.

38. The management node (208, 600) according to any one of claims 20 to 37, wherein the telecommunication environment (210) comprises a self-organised network, SON.

39. A non-transitory computer-readable storage medium for managing a system operative in a telecommunication environment (210), wherein the system executes one or more actions which are determined using a reinforcement learning algorithm based on a state of the telecommunication environment (210), the storage medium storing code which, when executed by processing circuitry of a management node, causes the management node to: - provide (302) a first state of the telecommunication environment (210) as an input to the reinforcement learning algorithm and receive (304), from the reinforcement learning algorithm, an output comprising a first action that is recommended for execution by the system, the first state and the first action forming a first state-action pair; - determine (306, 308) whether the first state-action pair is similar to a second state-action pair based on a measure of similarity, wherein the second state-action pair comprises a second state of the telecommunication environment and a second action that was previously executed based on the second state and resulted in a violation of one or more safety constraints of the telecommunication environment; and- in response to determining that the first state-action pair is similar to the second state-action pair, refrain (318) from executing the first action.

Images