Training a reinforcement learning agent

Abstract

There is provided a computer-implemented method (200) for training a reinforcement learning agent to adjust or maintain a parameter of a first cell of a communication network. The method (200) comprises obtaining (202) a dataset comprising a first state of the first cell at a first time instance, a second state of the first cell at a second time instance, a first action, wherein the first action, when co-occurring with the first state at any time instance, causes the first cell to transition to the second state, and a first reward value for performing the first action, determining (204), based on the dataset, a second action, wherein the second action, when co-occurring with the second state at any time instance, causes the first cell to transition to the first state, and a second reward value for performing the second action, obtaining (206) a first information comprising the first state, the second state, the first action, and the first reward value, obtaining (208) second information comprising the second state, the first state, the second action, and the second reward value, and training (210) the reinforcement learning agent based on the first information and the second information.

Description

[0001] TRAINING A REINFORCEMENT LEARNING AGENT

[0002] TECHNICAL FIELD

[0003] The present disclosure relates to a computer-implemented method for training a reinforcement learning agent to adjust or maintain a parameter of a first cell of a communication network. The present disclosure also relates to a network node, a computer program and a computer program product.

[0004] BACKGROUND

[0005] Communication networks, and particularly Radio Access Networks, involve numerous parameters which should be optimized to ensure desirable network performance. Such parameters are particularly present at layer 3 of the network and above. Several platforms have emerged that enable the effective control of these parameters. One example application for such parameter control is in controlling RAN antenna parameters such as antenna tilt, Downlink (DL) Transmission (TX) power, sector shapes, cell sleep mode, etc., to optimize energy efficiency for the RAN and user experience.

[0006] One promising method for control of RAN parameters is using machine learning (ML), in which an ML algorithm learns to change the parameter configuration to adapt to a given situation. Reinforcement learning (RL) has been used for control of parameter value changes. In RL algorithms, an RL agent receives as input the current state of the network and outputs an incremental change for values of the controlled parameters as an action, with the aim of optimizing one or more performance metrics for the network. An incremental parameter value change might be, for example, +1 or -1 degrees on an antenna tilt angle, an increase or decrease in DL TX power, etc.

[0007] The RL scheme based on the Deep Neural Network (DNN) proposed in “Multi-Agent Reinforcement Learning with Common Policy for Antenna Tilt Optimization” has been demonstrated to work very efficiently to optimize cell parameters that impact the Key Performance Indicators (KPIs) of a first cell, but also the KPIs of the neighbouring cells of the first cell. “Multi-Agent Reinforcement Learning with Common Policy for Antenna Tilt Optimization” proposes the use of actions that apply a small incremental parameter change in the cell, such that it is possible to modify all the cells in the network simultaneously. The use of multiple instances of a unique RL agent to optimize parameters (such as the Remote Electrical Tilt (RET)) per cell, using a common policy, is described in WO 2021 / 190772 A1.

[0008] The use of a static simulator as a digital twin to pre-train the model and avoid damaging the live network during an initial exploratory phase, was also considered in “Multi-Agent Reinforcement Learning with Common Policy for Antenna Tilt Optimization” and “Al: Enhancing customer experience in a complex 5G world” (retrieved 30th July 2021). In these approaches, static offline datasets are used to train the RL agent, meaning the agent does not interact with the live network during its initial training.

[0009] Figure 1 shows the expected performance evolution of an RL agent and an expert system, as described in “Multi-Agent Reinforcement Learning with Common Policy for Antenna Tilt Optimization”. As illustrated in Figure 1 , the initial performance of an optimizer based on RL is significantly worse than that of an expert system, because it needs to interact with the environment (i.e., the network) to acquire experiences and learn from them. Initially, the RL agent needs to explore the space of possible solutions as it does not have any knowledge to address the problem, and as a result, initially makes more random decisions. However, as the RL agent acquires knowledge from the network, the performance associated with its decisions improves, and eventually exceeds the performance that can be achieved with an expert system.

[0010] “Multi-Agent Reinforcement Learning with Common Policy for Antenna Tilt Optimization” and "Al: Enhancing customer experience in a complex 5G world." (retrieved 30th July 2021) propose to optimize live networks by using an RL agent previously trained with a static simulator, and then allowing the agent to continue learning from new experiences acquired as a result of interacting with the live network. The RL agent explores possible actions (i.e., possible changes of parameter values) in a real-world situation as the RL agent learns which actions will lead to improved performance. According to the results published in “Multi-Agent Reinforcement Learning with Common Policy for Antenna Tilt Optimization”, allowing the agent to continue learning from the live network can provide significant performance gain.

[0011] The same approach has also been successfully used to optimize the PZeroNominalPusch parameter for uplink Power Control (PC) in a live network, and could be applied to the optimization of many other parameters, such as maximum cell transmission power, or Cell-specific Reference Signal (CRS) gain (also known as power boosting).

[0012] As noted above, there is a potential performance gain by allowing a cell parameter optimizer based on RL to continue learning from the live network. However, in practice, the iterative process described by these articles to optimize a live network cannot be run more often than 2 or 3 times a week, as every iteration (in which the RL agent receives as input the current state of the network and outputs an incremental change for values of the controlled parameters as an action) requires collecting statistics over the course of 1 or 2 days, and then running a single step of the algorithm. This means that the number of samples that can be obtained from the live network to continue training the agent is very small, especially when compared with the number of samples available from the static simulator, where 10,000 iterations were used to pre-train the agent in these cases. In the trial carried out for RET optimization in "Al: Enhancing customer experience in a complex 5G world." (retrieved 30th July 2021), where 127 cells were considered for RET optimization, only 34 samples were obtained from the live network for online training after the first iteration. It is also noted that cells with no RET change do not produce any training samples for the relevant iteration. After 9 iterations, the number of available samples was 259.

[0013] Moreover, trials on live networks have shown that most suggested actions are downtilts (i.e., a RET change of +1 degree), which means that the agent has little opportunity to learn from uptilts (i.e., a RET change of -1 degree) applied to live networks. In the previously mentioned RET optimization trial, only 45 changes were uptilts out of the 259 changes proposed by the agent after 9 iterations.

[0014] Existing online RL training algorithms consequently suffer from a scarcity of training samples, for training the RL agent for action selection. As a result, the online training process may be slow, due to the scarcity of training samples. Furthermore, the trained RL agent may not be well adapted to the live network, due to the scarcity of training samples available from the live network.

[0015] SUMMARY One object of the present disclosure is to enable facilitated control of a cell of a communication network, when a number of training samples available from the communication network is scarce in comparison to a number of available offline training samples.

[0016] According to a first aspect of the present disclosure, there is provided a computer- implemented method for training a reinforcement learning agent to adjust or maintain a parameter of a first cell of a communication network. The method comprises obtaining a dataset, comprising a first state of the first cell at a first time instance, a second state of the first cell at a second time instance, a first action, wherein the first action, when cooccurring with the first state at any time instance, causes the first cell to transition to the second state, and a first reward value for performing the first action. The method further comprises determining, based on the dataset, a second action, wherein the second action, when co-occurring with the second state at any time instance, causes the first cell to transition to the first state, and a second reward value for performing the second action. The method further comprises obtaining a first information comprising the first state, the second state, the first action, and the first reward value. The method further comprises obtaining second information comprising the second state, the first state, the second action, and the second reward value. The method further comprises training the reinforcement learning agent based on the first information and the second information.

[0017] According to another aspect of the present disclosure, there is provided a computer program product comprising a computer readable storage medium. The computer readable storage medium comprises instructions which are executable by processing circuitry to cause a network node to perform a method according to any one or more of the aspects or examples of the present disclosure.

[0018] According to another aspect of the present disclosure, there is provided a computer program. The computer program comprises instructions which, when executed by processing circuitry, cause the processing circuitry to perform a method according to any one or more of the aspects or examples of the present disclosure.

[0019] According to another aspect of the present disclosure, there is provided a network node. The network node comprises processing circuitry configured operate in accordance with the first aspect. Aspects of the present disclosure thus provide a method and a nodes that address the scarcity of action samples in online RL for the management of communication network environments, such as for example an RAN environment. Examples of the present disclosure address the scarcity of online action samples by augmenting the available samples, such that an increased number of samples may be used for online training of the RL agent. By increasing the number of samples for use for online training of the RL agent, examples of the present disclosure are able to increase the speed of the training process, and enable an RL agent that is better adapted to the live network to be trained than in known approaches.

[0020] BRIEF DESCRIPTION OF THE FIGURES

[0021] For a better understanding of the present disclosure, and to show more clearly how it may be carried into effect, reference will now be made, by way of example, to the following drawings in which:

[0022] Figure 1 shows the expected performance evolution of an RL agent and an expert system;

[0023] Figure 2 illustrates a flow chart illustrating process steps in a computer implemented method for training a reinforcement learning agent to adjust or maintain a parameter of a first cell of a communication network;

[0024] Figures 3a to 3d illustrate another flow chart illustrating process steps in a computer implemented method for training a reinforcement learning agent to adjust or maintain a parameter of a first cell of a communication network;

[0025] Figure 4 is a block diagram illustrating an example network node;

[0026] Figure 5 illustrates a computer program product;

[0027] Figure 6 illustrates a concept to duplicate a number of training samples;

[0028] Figure 7 illustrates a concept to further augment a number of training samples; Figure 8 shows the evolution of a number of samples available for online training with respect to a selected number of top neighbours, with and without augmentation according to the methods described herein; and

[0029] Figure 9 shows the evolution of a cumulative number of training samples with respect to an iteration number, with and without augmentation according to the methods described herein.

[0030] DETAILED DESCRIPTION

[0031] Generally, all terms used herein are to be interpreted according to their ordinary meaning in the relevant technical field, unless a different meaning is clearly given and / or is implied from the context in which it is used. All references to a / an / the element, apparatus, component, means, step, etc. are to be interpreted openly as referring to at least one instance of the element, apparatus, component, means, step, etc., unless explicitly stated otherwise. The steps of any methods disclosed herein do not have to be performed in the exact order disclosed, unless a step is explicitly described as following or preceding another step and / or where it is implicit that a step must follow or precede another step. Any feature of any of the embodiments disclosed herein may be applied to any other embodiment, wherever appropriate. Likewise, any advantage of any of the embodiments may apply to any other embodiments, and vice versa. Other objectives, features and advantages of the enclosed embodiments will be apparent from the following description.

[0032] Some of the embodiments contemplated herein will now be described more fully with reference to the accompanying drawings. Other embodiments, however, are contained within the scope of the subject-matter disclosed herein, the disclosed subject-matter should not be construed as limited to only the embodiments set forth herein; rather, these embodiments are provided by way of example to convey the scope of the subject-matter to those skilled in the art.

[0033] In some instances, detailed descriptions of well-known methods, entities, interfaces, circuits, and devices are omitted so as not obscure the description with unnecessary detail. Those skilled in the art will appreciate that the functions described may be implemented in one or more entities using hardware circuitry (e.g., analogue and / or discrete logic gates interconnected to perform a specialized function, ASICs, PLAs, etc.) and / or using software programs and data in conjunction with one or more digital microprocessors or general purpose computers. Entities that communicate using the air interface also have suitable radio communications circuitry. Moreover, where appropriate the technology can additionally be considered to be embodied entirely within any form of computer-readable memory, such as solid-state memory, magnetic disk, or optical disk containing an appropriate set of computer instructions that would cause a processor to carry out the techniques described herein.

[0034] Embodiments of the present disclosure propose methods for training a reinforcement learning agent to adjust or maintain a parameter of a first cell of a communication network. The proposed methods address the challenge of online RL discussed above, that is, the scarcity of training samples for online training of the RL policy for action selection. The challenges are addressed through the obtaining of augmented samples, based on samples that represent changes that have occurred in the live network, where the augmented samples represent hypothetical reversions of these observed changes. In this manner, the reinforcement learning agent (or algorithm) is able to learn based on both changes that have occurred in the live network, and hypothetical reversions of these observed changes (where this information previously would not have been available to the RL agent).

[0035] Figure 2 is a flow chart illustrating process steps in a computer implemented method 200 for training a reinforcement learning agent to adjust or maintain a parameter (for example, a RET parameter) of a first cell of a communication network. The method 200 may be used to augment a number of samples (or an amount of information) for online training an RL-based radio network parameter optimizer. The RL-based radio network parameter optimizer may be trained to, at each iteration, receive as input a current state of the first cell, and output an incremental change of a value of the parameter of the first cell as an action. For example, in the particular case of RET optimization, the possible incremental actions may be {-1 , 0,+1} degrees. For example, in the particular case of RET optimization, the incremental change may be a change of +1 , 0, or -1 degree.

[0036] The method may be performed by a network node, which may comprise a physical or virtual node, and may be implemented in a computer system, computing device or server apparatus and / or in a virtualized environment, for example in a cloud, edge cloud, an Open Radio Access Network, O-RAN, or fog deployment. The network node may for example be instantiated in an rApp of an O-RAN architecture. Examples of a virtual node may include a piece of software or computer program, a code fragment operable to implement a computer program, a virtualised function, or any other logical entity. The network node may for example be implemented in a core network of a communication network, or in a RAN. The communication network may for example comprise an LTE network, a New Radio (NR) network or any other existing or future communication network systems, and the network node may be implemented in a core network node, or in a Radio Access node, which itself may comprise a physical node and / or a virtualized network function that is operable to exchange wireless signals. In some examples, a Radio Access node may comprise a base station node such as a NodeB, eNodeB, gNodeB, or any future implementation of this functionality. The network node may encompass multiple logical entities, as discussed in greater detail below, and may for example comprise a Virtualised Network Function (VNF). In other examples, the network node may be instantiated in a physical or virtual server in a centralised or cloud based deployment.

[0037] Referring to Figure 2, the method 200 comprises, in a first step 202, obtaining a dataset comprising a first state of the first cell at a first time instance, a second state of the first cell at a second time instance, a first action, wherein the first action, when co-occurring with the first state at any time instance, causes the first cell to transition to the second state, and a first reward value for performing the first action. In other words, the dataset relates to a pair of observed states of the first cell. The first action may have been selected for execution by the RL agent on the basis of the first state. The first reward value may indicate an observed impact of the first action on performance of the first cell and / or one or more neighbouring cells of the first cell. The dataset may be sampled by the RL agent from the communication network, or may be obtained from another entity in the communication network.

[0038] In the context of the present disclosure, the term “co-occurring” with a state defines that the action is performed while the first cell is in the particular state. The term “a time instance” defines a sample time at which the RL agent observes a state of the first cell.

[0039] A state of the first cell may represent its current situation, condition, and / or circumstances. For example, a state of the first cell may comprise intrinsic information relating to the first cell and / or extrinsic information relating to the environment of the first cell. The state of the first cell may include its configuration, the presence and position (within physical or radio space) of the first cell and / or one or more neighbouring cells of the first cell, requests currently being made of the first cell and / or one or more neighbouring cells of the first cell, availability and / or requirements being made of resources of the first cell and / or one or more neighbouring cells of the first cell, condition of such resources, etc. A state of the first cell may thus include values of configurable, operational and / or measurable parameters for the first cell and / or one or more neighbouring cells of the first cell and / or the environment of the first cell, demands being made upon the first cell and / or one or more neighbouring cells of the first cell, etc. A state of the first cell may comprise one or more Key Performance Indicators (KPIs) for the first cell and / or one or more neighbouring cells of the first cell, information about a number of wireless devices connecting to the communication network in the environment of the first cell, etc.

[0040] The first reward value may indicate an observed impact of the first action on the performance of the first cell and / or one or more neighbouring cells of the first cell. This may comprise a change in one or more KPI values following execution of the action, or any other value, combination of values, function using values as input etc. which provide an indication of how the selected action has impacted the ability of the first cell and / or one of more neighbouring cells to perform a particular task. For example, in the case of the first cell being a cell of a RAN, the reward value may comprise a function of network coverage, quality and capacity parameters.

[0041] The first time instance and the second time instance may be successive time instances, or non-successive time instances. The method 200 may be used to augment a number of samples (or an amount of information) for online training an RL agent, regardless of whether the information relates to a pair of states that occur at successive time instances, or not.

[0042] In step 204, the method 200 comprises determining, based on the dataset, a second action, wherein the second action, when co-occurring with the second state at any time instance, causes the first cell to transition to the first state, and a second reward value for performing the second action.

[0043] The second action may be determined through recognising that applying an opposite action to the first action, to the second state, will cause the first cell to transition to the first state and provide an inverse reward to the first reward. In this case, the opposite action is the second action, and the inverse reward is the second reward. In other words, the dataset provides the information required to allow a reward associated with a hypothetical change reversion (between the second state and the first state) to be determined.

[0044] In step 206, the method 200 comprises obtaining a first information comprising the first state, the second state, the first action, and the first reward value.

[0045] The first information may be an information tuple of the form (St, at, St+1 , rt), where St is the first state of the first cell, at is the first action, St+1 is the second state of the first cell, and rt is the first reward value.

[0046] In step 208, the method 200 comprises obtaining second information comprising the second state, the first state, the second action, and the second reward value.

[0047] The second information may be an information tuple of the form (St+1 , -at, St, r’t), where

[0048] St+1 is the second state of the first cell, -at is the second action, St is the first state of the first cell, and r’t is the second reward value. In this example, r’t is the inverse of rt described above.

[0049] In step 210, the method 200 comprises training the reinforcement learning agent based on the first information and the second information.

[0050] Therefore, the method 200 enables augmented second information, that represents a hypothetical reversion of the change represented by the first information, to be obtained. The obtained first information and obtained second information are then used to train the reinforcement learning agent, allowing the reinforcement learning agent to learn based on both changes that have occurred in the live network, and hypothetical reversions of these observed changes that were previously unavailable to the RL agent. Figures 3a to 3d show a flow chart illustrating another example of a computer implemented method 300 fortraining a reinforcement learning agent to adjust or maintain a parameter (for example, a RET parameter) of a first cell of a communication network. As for the method 200 discussed above, the method 300 may be performed by a network node, which may comprise a physical or virtual node, and may be instantiated substantially as discussed above with reference to the method 200. The method 300 illustrates examples of how the steps of the method 200 may be implemented and supplemented to provide the above discussed and additional functionality.

[0051] Referring initially to Figure 3a, in step 302, the method 300 comprises obtaining a dataset comprising a first state of the first cell at a first time instance, a second state of the first cell at a second time instance, a first action, wherein the first action, when co-occurring with the first state at any time instance, causes the first cell to transition to the second state, and a first reward value for performing the first action. The first action and the first reward value may be substantially as described above with reference to the method 200. As illustrated at 302a, in some examples, the first action, when co-occurring with the first state, may adjust the parameter from a first value corresponding to the first state, to a second value corresponding to the second state.

[0052] As described above, the dataset comprises pairs of observed states of the first cell. It will be appreciated the first state and the second state in this pair may be consecutive states (that is, consecutive states in a time series of observed states of the first cell). As illustrated at 302b and described above, in some examples, the first time instance and the second time instance may be successive time instances.

[0053] As illustrated at 302c and discussed above, in some examples, the parameter may be a remote electrical tilt, RET, of one or more antennas associated with the first cell. Other example parameters may include pan, azimuth, SSB beam shapes, DL TX power, P0 nominal PUSCH, etc.

[0054] As illustrated at 302d and discussed above, in some examples, a state of a cell at a time instance may comprise information relating to the cell and / or information relating to the environment of the cell. As illustrated at 302e and discussed above, in some examples, the state of the first cell at a time instance may comprise a state of one or more neighbour cells of the first cell at the time instance. As illustrated at 302f and discussed above, in some examples, the state of the first cell at a time instance may comprise one or more of: configuration parameters, operational parameters and / or key performance indicators of the first cell at the time instance, configuration parameters, operational parameters and / or key performance indicators of one or more neighbour cells of the first cell at the time instance. For example, the key performance indicators may indicate one or more of: coverage, quality, capacity, an amount of interference, a delay, etc.

[0055] Optionally, the method 300 may comprise steps 304-310. Steps 304-310 enable the method 300 to determine whether the augmented information may be obtained based on the dataset. For example, where the dataset relates to a pair of states that occur at time instances that are not successive, one or more neighbouring cells of the first cell may have also changed their values of the parameter between the pair of states. These changes may not be an incremental change, and in these cases, augmented information may not be obtained based on the dataset. However, in these cases, the augmented information may still be obtained, based on the dataset, if the one or more neighbouring cells do not have significant influence on the first cell (for example, if their respective degrees of overlap with the first cell are low).

[0056] At step 304, prior to determining the second action and the second reward value, the method 300 comprises determining a subset of one or more neighbour cells of the first cell, based on a degree of overlap between the first cell and each of the one or more neighbour cells. As illustrated at 304a, the subset of the one or more neighbour cells may comprise a predetermined number, N, of neighbour cells with the N greatest degrees of overlap with the first cell. As illustrated at 304b, in some examples, the subset of the one or more neighbour cells may comprise neighbour cells whose respective degrees of overlap with the first cell exceed a threshold. Step 304 may therefore allow a subset of one or more neighbour cells to be determined that comprises the neighbouring cells of the first cell that have the most significant influence on the first cell.

[0057] At step 306, the method 300 comprises, for each neighbour cell of subset, determining, between the first time instance and the second time instance, whether a change in a value of the parameter of the neighbour cell is an incremental change.

[0058] At step 308, the method 300 determines whether each change is an incremental change. In response to determining that each change is an incremental change, the method 300 proceeds to step 312. That is, at step 308, the method 300 determines whether the augmented information may be obtained based on the dataset. This guarantees the quality of the augmented information that is obtained.

[0059] In response to determining that at least one change is not an incremental change, the method 300 proceeds to step 310. At step 310, the method 300 comprises discarding the dataset. The method 300 then returns to step 302, and an alternative dataset is obtained.

[0060] In step 312, the method 300 comprises determining, based on the dataset, a second action, wherein the second action, when co-occurring with the second state at any time instance, causes the first cell to transition to the first state, and a second reward value for performing the second action.

[0061] As illustrated at 312a, in some examples, the first reward value may be based on the difference between a first metric associated with the first state and a second metric associated with the second state, and the second reward value may be based on the difference between the second metric and the first metric. In other words, a reward value associated with an action may depend on the state of the first cell prior to performing, and after performing, the action. In these examples, this dependency of the reward value on the pre-action and post-action states may enable an inverse reward to be determined that is associated with performing an inverse action on the first cell in the second state.

[0062] As illustrated at 312b, in some examples, the first metric may be representative of conditions in the first cell at the first time instance, and / or the second metric may be representative of conditions in the first cell at the second time instance. As illustrated at 312c, the first metric may be representative of conditions in one or more neighbour cells of the first cell at the first time instance, and / or the second metric may be representative of conditions in the one or more neighbour cells of the first cell at the second time instance. As illustrated at 312d, the conditions may comprise one or more of: downlink, DL, coverage (for example, the proportion of DL traffic that is delivered in the particular cell with good quality during a period including or immediately preceding the particular time instance); DL quality (for example, the proportion of DL traffic that is delivered in the particular cell with good quality during a period including or immediately preceding the particular time instance); and congestion (for example, the congestion rate for RRC signalling in the particular cell during a period including or immediately preceding the particular time instance). In some examples, “good coverage” means having DL reference signal received power (RSRP) greater than a predefined threshold, and / or “good quality” means having DL SINR (or reference signal received quality, RSRQ) greater than a predefined threshold.

[0063] In some examples, the first metric may indicate performance of the first cell and / or one or more neighbour cells of the first cell at the first time instance, and the second metric may indicate performance of the first cell and / or one or more neighbour cells of the first cell at the second time instance. In these examples, a reward value may then provide an indication of how the action associated with the reward value has impacted the performance of the first cell and / or one or more neighbour cells of the first cell.

[0064] As illustrated at 312e and discussed above, in some examples, the first action may adjust the parameter, and the second action may reverse the adjustment caused by the first action. In other words, as discussed above, in these examples, the second action is an opposite action to the first action. As illustrated at 312f, in some examples, performing the first action may increase the parameter and performing the second action may decrease the parameter. In some examples, performing the first action may decrease the parameter and performing the second action may increase the parameter. The first and second actions may therefore respectively control a value of the parameter, and may comprise at least one of an incremental change or no change to the parameter value.

[0065] As discussed above with reference to step 204, the dataset provides the information required to allow a reward associated with a hypothetical change reversion (between the second state and the first state) to be determined.

[0066] In step 314, the method 300 comprises obtaining a first information comprising the first state, the second state, the first action, and the first reward value.

[0067] In step 316, the method 300 comprises obtaining second information comprising the second state, the first state, the second action, and the second reward value.

[0068] In step 318, the method 300 comprises training the reinforcement learning agent based on the first information and the second information.

[0069] Optionally, at step 320, the method 300 comprises configuring the reinforcement learning agent to adjust or maintain the parameter in the first cell and / or adjust or maintain the parameter in one or more neighbour cells of the first cell. In some examples, the trained RL agent may be configured to generate an action based on an observed state of the first cell, and cause execution of the action in the environment of the first cell.

[0070] Therefore, the method 300 enables augmented second information, that represents a hypothetical change reversion of the change represented by the first information, to be obtained.

[0071] Where the first information represents a transition between non-consecutive states, the method 300 also enables augmented second information, that represents a hypothetical reversion of the transition represented by the first information, to be obtained. In examples in which the first information relates to a first state and a second state that are non-consecutive, the dataset may comprise a list of consecutive transitions from the first state to the second non-consecutive state, as described with reference to Figure 7 below.

[0072] The obtained first information and obtained second information are then used to train the reinforcement learning agent, allowing the reinforcement learning agent to learn based on both changes that have occurred in the live network, and hypothetical reversions of these observed changes that were previously unavailable to the RL agent. This enables an improved RL agent to be trained, that is better adapted to the communication network in which the online learning process is taking place.

[0073] As discussed above, the methods 200 and 300 may be performed by a network node, and the present disclosure provides a network node that is adapted to perform any or all of the steps of the above discussed methods. The network node may comprise a physical node such as a computing device, server etc., or may comprise a virtual node. A virtual node may comprise any logical entity, such as a Virtualized Network Function (VNF) which may itself be running in a cloud, edge cloud or fog deployment. The network node may be operable to be instantiated in a cloud based deployment, as an rApp, in a physical or virtual radio access node, control network node, etc.

[0074] Figure 4 is a block diagram illustrating an example network node 400 which may implement the method 200 and / or 300, as illustrated in Figures 2 and 3a to 3d, according to examples of the present disclosure, for example on receipt of suitable instructions from a computer program 450. Referring to Figure 4, the network node 400 comprises a processor or processing circuitry 402, and may comprise a memory 404 and interfaces 406. The processing circuitry 402 is operable to perform some or all of the steps of the method 200 and / or 300 as discussed above with reference to Figures 2 and 3a to 3d. The memory 404 may contain instructions executable by the processing circuitry 402 such that the network node 400 is operable to perform some or all of the steps of the method 200 and / or 300, as illustrated in Figures 2 and 3a to 3d. The instructions may also include instructions for executing one or more telecommunications and / or data communications protocols. The instructions may be stored in the form of the computer program 450. In some examples, the processor or processing circuitry 402 may include one or more microprocessors or microcontrollers, as well as other digital hardware, which may include digital signal processors (DSPs), special-purpose digital logic, etc. The processor or processing circuitry 402 may be implemented by any type of integrated circuit, such as an Application Specific Integrated Circuit (ASIC), Field Programmable Gate Array (FPGA) etc. The memory 404 may include one or several types of memory suitable for the processor, such as read-only memory (ROM), random-access memory, cache memory, flash memory devices, optical storage devices, solid state disk, hard disk drive, etc.

[0075] Figure 5 illustrates a computer program product 502 according to an embodiment. More specifically, there is provided a computer program product 502 comprising a computer readable storage medium 504. The computer readable storage medium 504 comprises instructions (e.g. computer program 506) which are executable by processing circuitry (such as the processing circuitry 402 of the network node 400 described herein) to cause the network node 400 to perform the methods described herein in respect of the network node 400. There is provided a computer program product 502 comprising a carrier 504 containing instructions (e.g. computer program 506) for causing the network node 400 (e.g. the processing circuitry 402 of the network node 400 described herein) to perform at least part of the methods described herein. In some embodiments, the carrier 504 can be any one of an electronic signal, an optical signal, an electromagnetic signal, an electrical signal, a radio signal, a microwave signal, or a computer-readable storage medium.

[0076] There is also provided a computer program 506 comprising instructions which, when executed by processing circuitry (such as the processing circuitry 402 of the network node 400 described herein), cause the processing circuitry to perform at least part of the methods described herein. Figures 2 and 3a to 3d discussed above provide an overview of methods which may be performed according to different examples of the present disclosure. These methods may be performed by a network node, as illustrated in Figure 4. These methods enable the efficient training of an improved reinforcement learning agent to adjust or maintain a parameter of a first cell of a communication network. There now follows a detailed discussion of how different process steps illustrated in Figures 2 and 3a to 3d discussed above may be implemented. The functionality and implementation detail described below is discussed with reference to the modules of Figure 4 performing examples of the methods 200 and / or 300, substantially as described above.

[0077] The examples below focus on the approach proposed in “Multi-Agent Reinforcement Learning with Common Policy for Antenna Tilt Optimization” for RET optimization, however, it will be appreciated that these examples can be applied to many other parameters and optimization approaches based on RL.

[0078] An example method of augmenting online training data for online RL training according to the methods herein is now described. The following example may augment training data for an RL agent that applies, at each of a series of iterations #t, an action atto transition a current state of an environment St to a state St+1 (that is, the state of the environment at the next iteration), and receives a reward rt associated with performing the action.

[0079] In this example, the RL agent aims to optimise the value of a RET parameter of a first cell, which is included as a part of the state S in this example. A state S may comprise information relating to the first cell and / or information relating to the environment of the first cell and / or information relating to one or more neighbouring cells of the first cell. In this example, the action applied at each iteration is an incremental value change (normally 0, +1 or -1 degrees) with respect to the previous value of the RET parameter of the first cell.

[0080] In this example, the problem (that is, the problem of Remote Electrical Tilt (RET) optimization) is particularized to a case where the reward rt is defined as the relative difference of a metric m before and after applying the action as follows:

[0081] In this case, the metric mt is inherent to state St (as is the first metric described above), and the metric mt+1 is inherent to state St+1 (as is the second metric described above).

[0082] As a result, in this example, the reward rt strictly depends on the states St+1 and St (similar to the first reward value described above).

[0083] This example also assumes that for iterations where the action is 0 degrees (and the value of the parameter does not change between the states St and St+1), the reward obtained by the RL agent is zero. This forces the RL agent (which in this example is based on a DNN) to predict no improvements nor degradations when no parameter change is applied. The network would otherwise very likely provide a non-zero reward in these situations, as neighbouring cells of the first cell may have changed their parameter values between the states St and St+1 , or simply as a result of natural network fluctuations due to external factors.

[0084] In this example, the RL agent utilises samples consisting of tuples with the following shape for training:

[0085] (St, at, St+1 , mt, mt+1).

[0086] For a number of samples, each represented as a respective tuple that comprises a pair of consecutive states, the number of samples may be duplicated as follows.

[0087] In this example, at state St, when the action at is applied, the state St+1 and the reward

[0088] (mt+1 - mt) / mt will be obtained. Due to the dependence of the reward rt on the state prior to applying the action, St, and the state following the application of the action, St+1 , it has been recognised that, at state St+1 , if the action -at (that is, the opposite action to at) is applied, the environment would transition to state St with the reward (mt-mt+1) / mt+1 (that is, an inverse reward to the reward rt). Figure 6 offers a visual explanation of this concept.

[0089] Therefore, in this example, each of sample of the form (St, at, St+1 , mt, mt+1) can be augmented to obtain a second sample of the form (St+1 , -at, St, mt+1 , mt) due to this reciprocity property. As a result, the number of samples from the real network may be duplicated for online training at RL optimization.

[0090] An example method of augmenting online training data for online RL training according to the methods herein is now described. The following example may augment training data for an RL agent that applies, at each of a series of iterations #t, an action atto transition a current state of an environment St to a state St+1 (that is, the state of the environment at the next iteration), and receives a reward rt associated with performing the action.

[0091] In this example, the RL agent aims to optimise the value of a RET parameter of a first cell, which is included as a part of the state S in this example. A state S may comprise information relating to the first cell and / or information relating to the environment of the first cell. In this example, the action applied at each iteration is an incremental value change (normally 0, +1 or -1 degrees) with respect to the previous value of the RET parameter of the first cell. It will be appreciated that, in this example, the first cell has one or more neighbouring cells.

[0092] In the aforementioned example, the samples obtained from the network comprised pairs of consecutive states. It is possible to further increase the number of samples available for online training by not restricting the samples obtained from the network to those that comprise pairs of states associated with consecutive iterations. In this example, a sample comprising any pair of states (consecutive or non-consecutive) may be obtained from the network and considered for augmentation, as long as the change in the parameter value of the first cell between the pair of states is an incremental change. A visual example of this concept is provided in Figure 7.

[0093] Figure 7 illustrates how first information relating to a pair of non-consecutive states may be obtained from a list of consecutive transitions from a first state to a second non- consecutive state. As shown in Figure 7, first information, relating to the transition between non-consecutive states Si and S3, may be obtained from the list of consecutive transitions comprising the transition from states S1 to S2, and the transition from states S2 to S3,

[0094] However, not all such samples are suitable for augmentation according to the methods described herein.

[0095] It will be appreciated that, at each iteration, each neighbouring cell of the first cell is respectively able to change the value of its RET parameter. Samples where the absolute change in the parameter value of a neighbour cell, between the pair of states, is greater than an incremental change (that is, the maximum change in the value that may occur between successive iterations), may not be suitable for augmentation according to the methods described herein. Such samples may be discarded in order to guarantee that the augmented training samples remain as valid as the original training samples obtained from the network.

[0096] However, it will be appreciated that not all the neighbouring cells of the first cell may significantly impact the first cell. For example, depending on the inter-cell distance, the relative azimuth, and the tilt, the coupling or overlapping between the first cell and one or more of the neighbouring cells may be negligible. It may be acceptable to augment training data samples where the absolute change in the parameter value of a neighbour cell, between the pair of states, is greater than an incremental change, when the overlapping between the neighbour cell and the first cell is low. Therefore, in this example, a number of samples, each represented as a respective tuple that comprises a pair of states (consecutive or non-consecutive), may be duplicated as follows.

[0097] For the first cell, a number of neighbour cells that are considered to significantly impact the first cell are determined. For example, the N top overlapping neighbours of the first cell may be determined. The number N may be predetermined and may define the number of neighbouring cells that are considered to significantly impact the first cell. For example, N may be a value between 5 and 10. Alternatively, an overlapping threshold may be defined, such that the neighbour cells that are considered to significantly impact the first cell each respectively have a degree of overlap with the first cell that exceeds the overlapping threshold. It will be appreciated that, in this example, the number of overlapping cells that are considered to significantly impact the first cell may vary, for different first cells to which the method is being applied to.

[0098] The N top overlapping neighbours may be determined by sorting the neighbouring cells from the neighbouring cell with the highest degree of overlap with the first cell, to the neighbouring cell with the lowest degree of overlap with the first cell, and selecting the first N neighbouring cells in this list. The degree of overlap for each neighbouring cell may be determined using Call Trace Recording (CTR) data, as described in “Multi-Agent Reinforcement Learning with Common Policy for Antenna Tilt Optimization”. For example, the overlapping factor between a first cell and a neighbour cell may be obtained as the periodicity in which both cells are reported simultaneously by the same user equipment (UE) in CTR, provided that the first cell is a serving cell. Additionally or alternatively, the degree of overlap between the first cell and a neighbouring cell may be calculated as the probability of having a Reference Signal Received Power (RSRP) difference between the first cell and the neighbouring cell lower than a relative RSRP threshold.

[0099] Then, for the first cell, for each iteration, samples including the current state and a state of a previous iteration, where the change in the parameter value for the first cell between these states is an incremental change, are obtained.

[0100] From the obtained samples, the samples where none of the considered neighbouring cells have an absolute change in a parameter value, between the pair of states, that is greater than an incremental change, are selected. Alternatively, in other examples, the permitted absolute change in a parameter value of a neighbour cell, between the pair of states, may depend on the degree of overlapping between the neighbour cell and the first cell. For example, if the degree of overlapping is high for the neighbour cell, only an incremental change may be permitted forthe sample to be considered valid. If the degree of overlapping is low for the neighbour cell, a greater absolute change in the parameter value may be permitted for the sample to be considered valid.

[0101] These selected samples are suitable for augmentation according to the methods described herein. In this example, for each selected sample, a tuple is formed with the shape (St, at, St+1 , mt, mt+1), and included in a bufferfor online training of the RL agent.

[0102] Each tuple may then be augmented according to the methods described herein to form a second tuple, which may then also be included in the buffer for online training.

[0103] It will be appreciated that, in this example, valid augmented samples will be obtained when the method is applied in a certain time window, in which traffic and network conditions do not vary significantly. These conditions must also be met during an RL RET optimization project (which is typically 3-5 weeks long), even when augmented samples are not being used. This guarantees that the conditions in the network will enable valid augmented training data to be obtained.

[0104] In this example, a number of samples for online training at RL optimization may be increased by greater than a factor of two (as is the case for the example above), by not restricting the samples to be augmented to those which comprise states associated with consecutive iterations.

[0105] For the purposes of evaluation of an example implementation of the methods disclosed herein, the use case of Remote Electrical Tilt (RET) optimization is considered with a real-world dataset.

[0106] A prototype has been developed in Python language as several Jupyter Notebooks to obtain all possible augmented samples for online training of an RL agent, from samples comprising observed states of a network.

[0107] The prototype has been used with the training data collected during the RET optimization trial described in "Al: Enhancing customer experience in a complex 5G world." (retrieved 30th July 2021), to calculate the number of samples for online training available with and without augmentation according to the methods described herein.

[0108] Figure 8 shows the evolution of the number of samples available for online training after the 9 iterations of the trial with respect to the selected number of top neighbours (N), with and without augmentation according to the methods described herein. With N=5, it is possible to multiply the number of training samples by 5, approximately, when augmenting the samples according to the methods described herein. With a more conservative value of N=10, the number of training samples may be augmented by a factor of 4, approximately, when augmenting the samples according to the methods described herein.

[0109] Figure 9 shows the evolution of the cumulative number of training samples with respect to the iteration number, with and without augmentation according to the methods described herein. Additionally, Figure 9 also shows the augmentation factor obtained when augmenting the samples according to the methods described herein. In Figure 9, a number of top neighbours N=5 has been considered. A high augmentation factor is appreciated from the first step, and then the augmentation factor grows with the number of iterations.

[0110] Example methods according to the present disclosure can increase the number of samples for online training during RL optimization of a live network. This increase in the number of samples accelerates the convergence of the pre-trained model (or RL agent) towards a model that is better adapted to the live network. This then allows the model to make improved decisions at earlier iterations when deployed in the live network, leading to improved network performance. Better and faster optimization of the RL agent also improves energy efficiency at the nodes in the network, both during deployment and optimization.

[0111] One particular challenge of RL training is action exploration. Exploring possible actions (i.e., possible changes of parameter values) in a real-world situation is usually accompanied by a risk of performance degradation, which may not be acceptable in a live network. However, the augmented samples formed according to the methods described herein allow an RL agent to perform action exploration in a safe fashion and learn the impact of actions that the pre-trained version of the RL agent would likely never otherwise suggest. In other words, if an action a is proposed as the best option at state Si , and then the network transitions to state S2, the RL agent would very unlikely suggest applying the opposite action -a at state S2 to go back to state Si . Example methods according to the present disclosure allow the RL agent to learn the impact of hypothetical opposite actions. Examples of these explorations include the particular case of uptilts in scenarios where downtilts are the most frequent action, or vice versa.

[0112] The methods of the present disclosure may be implemented in hardware, or as software modules running on one or more processors. The methods may also be carried out according to the instructions of a computer program, and the present disclosure also provides a computer readable medium having stored thereon a program for carrying out any of the methods described herein. A computer program embodying the disclosure may be stored on a computer readable medium, or it could, for example, be in the form of a signal such as a downloadable data signal provided from an Internet website, or it could be in any other form.

[0113] It should be noted that the above-mentioned examples illustrate rather than limit the disclosure, and that those skilled in the art will be able to design many alternative embodiments without departing from the scope of the appended claims. The word “comprising” does not exclude the presence of elements or steps other than those listed in a claim, “a” or “an” does not exclude a plurality, and a single processor or other unit may fulfil the functions of several units recited in the claims. Any reference signs in the claims shall not be construed so as to limit their scope.

Claims

CLAIMS1 . A computer-implemented method (200) for training a reinforcement learning agent to adjust or maintain a parameter of a first cell of a communication network, the method (200) comprising: obtaining (202) a dataset comprising a first state of the first cell at a first time instance, a second state of the first cell at a second time instance, a first action, wherein the first action, when co-occurring with the first state at any time instance, causes the first cell to transition to the second state, and a first reward value for performing the first action; determining (204), based on the dataset, a second action, wherein the second action, when co-occurring with the second state at any time instance, causes the first cell to transition to the first state, and a second reward value for performing the second action; obtaining (206) a first information comprising the first state, the second state, the first action, and the first reward value; obtaining (208) second information comprising the second state, the first state, the second action, and the second reward value; and training (210) the reinforcement learning agent based on the first information and the second information.

2. The method (200) of claim 1 , wherein a state of a cell at a time instance comprises one or more of: information relating to the cell, information relating to the environment of the cell.

3. The method (200) of claim 1 or 2, wherein the state of the first cell at a time instance comprises a state of one or more neighbour cells of the first cell at the time instance.

4. The method (200) of any one of the preceding claims, wherein the state of the first cell at a time instance comprises one or more of: configuration parameters, operational parameters and / or key performance indicators of the first cell at the time instance, configuration parameters, operational parameters and / or key performance indicators of one or more neighbour cells of the first cell at the time instance.

5. The method (200) of any one of the preceding claims, wherein the first reward value is based on the difference between a first metric associated with the first state and a second metric associated with the second state, and wherein the second reward value is based on the difference between the second metric and the first metric.

6. The method (200) of claim 5, wherein the first metric is representative of conditions in the first cell at the first time instance, and / or wherein the second metric is representative of conditions in the first cell at the second time instance.

7. The method (200) of claim 5 or 6, wherein the first metric is representative of conditions in one or more neighbour cells of the first cell at the first time instance, and / or wherein the second metric is representative of conditions in the one or more neighbour cells of the first cell at the second time instance.

8. The method (200) of claim 6 or 7, wherein the conditions comprise one or more of: downlink, DL, coverage; DL quality; and congestion.

9. The method (200) of any one of the preceding claims, wherein the first time instance and the second time instance are successive time instances.

10. The method (200) of claim 9, wherein the first action, when co-occurring with the first state, adjusts the parameter from a first value corresponding to the first state, to a second value corresponding to the second state.11 . The method (200) of any one of the preceding claims, wherein the parameter is a remote electrical tilt, RET, of one or more antennas associated with the first cell.

12. The method (200) of any one of the preceding claims, wherein the first action adjusts the parameter, and wherein the second action reverses the adjustment caused by the first action.

13. The method (200) of any one of the preceding claims, wherein performing the first action increases the parameter and performing the second action decreases the parameter, or wherein performing the first action decreases the parameter and performing the second action increases the parameter.

14. The method (200) of any one of the preceding claims, wherein, prior to determining (204) the second action and the second reward value, the method (200) comprises: determining (304) a subset of one or more neighbour cells of the first cell, based on a degree of overlap between the first cell and each of the one or more neighbour cells; for each neighbour cell of subset, determining (306), between the first time instance and the second time instance, whether a change in a value of the parameter of the neighbour cell is an incremental change; and in response to determining that each change is an incremental change, determining (204) the second action and the second reward value.

15. The method (200) of claim 14, wherein the method (200) further comprises: in response to determining that at least one change is not an incremental change, discarding (310) the dataset; and obtaining (202) alternative dataset.

16. The method (200) of claim 14 or 15, wherein the subset of the one or more neighbour cells comprises a predetermined number, N, of neighbour cells with the N greatest degrees of overlap with the first cell.

17. The method (200) of claim 14 or 15, wherein the subset of the one or more neighbour cells comprises neighbour cells whose respective degrees of overlap with the first cell exceed a threshold.

18. The method (200) of any one of the preceding claims, wherein the method (200) comprises: configuring (320) the reinforcement learning agent to adjust or maintain the parameter in the first cell and / or adjust or maintain the parameter in one or more neighbour cells of the first cell.

19. A network node (400) comprising: processing circuitry (402) configured to operate in accordance with a method of any one of claims 1 to 18.

20. A network node (400) as claimed in claim 19, wherein:the network node (400) comprises: at least one memory (404) for storing instructions which, when executed by the processing circuitry (402), cause the network node (400) to operate in accordance with a method of any one of claims 1 to 18.21 . A computer program (506) comprising instructions which, when executed by processing circuitry of a network node (400), cause the network node to perform the method (200) according to any one of claims 1 to 18.

22. A computer program product (502) comprising a computer readable storage medium (504), wherein the computer readable storage medium (504) comprises instructions which are executable by processing circuitry to cause: a network node to (400) perform the method (200) according to any one of claims 1 to 18.

Images