Managing an environment in a communication network

Abstract

A method (100) is disclosed for training a system to manage an environment in a communication network. The method comprises obtaining a training dataset comprising records of performance of a task of the environment during a period of management according to a reference policy (110), and using a Self-Supervised Learning process and a subset of the obtained training dataset to train an encoder ML model to generate a latent representation of the observed context. The method also comprises using a Reinforcement Learning process and the obtained training dataset to train a policy ML model to generate an action for execution in the environment according to a latent representation of an observed context of the environment, the latent representation generated by the encoder ML model (130), and outputting the trained encoder ML model and policy ML model for use in managing the communication network environment (140).

Description

[0001] Managing an environment in a communication network

[0002] Technical Field

[0003] The present disclosure relates to a method for training a system to manage an environment in a communication network, and to a method for managing an environment in a communication network. The methods may be performed by a training node and management node of the communication network respectively. The present disclosure also relates to a training node, a management node, and to a computer program product configured, when run on a computer, to carry out methods for training a system to manage an environment in a communication network, and for managing an environment in a communication network.

[0004] Background

[0005] Communication networks, and particularly Radio Access Networks, involve numerous parameters which should be optimized to ensure desirable network performance. Such parameters are particularly present at layer 3 of the network and above. Several platforms have emerged that enable the effective control of these parameters, including for example rApps in the Ericsson Intelligent Automated Platform, and Ericsson Performance Optimizers. One example application for such parameter control is in controlling RAN antenna parameters such as antenna tilt, Downlink (DL) Transmission (TX) power, sector shapes, cell sleep mode, etc., to optimize energy efficiency for the RAN and user experience. The example platforms also enable closed loop automation by providing extensive telemetry that can be used to control the above noted parameters in an intelligent manner.

[0006] One promising method for control of RAN parameters is using machine learning (ML), in which an ML agent learns to change the parameter configuration to adapt to a given situation. Reinforcement learning (RL) has been used for control of parameter value changes. In RL algorithms, an RL agent receives as input the current state of the network and outputs an incremental change for values of the controlled parameters as an action, with the aim of optimizing one or more performance metrics for the network. An incremental parameter value change might be, for example, +1 or -1 on an antenna tilt angle, an increase or decrease in DL TX power, etc.

[0007] One particular challenge of RL training is action exploration. Exploring possible actions (i.e., possible changes of parameter values) in a real-world situation is usually accompanied by a risk of performance degradation, as the RL agent learns which actions will lead to improved performance, and which will not. In order to avoid the risk of unacceptable performance degradation, alternative approaches have been proposed, including simulation-based training and offline RL, in which static offline datasets are used to train an RL agent, meaning the agent does not interact with the live network during its initial training.

[0008] While addressing the risk of performance degradation during training, offline RL comes with its own set of challenges, one of which stems from the fact that the policy used to collect the data may be suboptimal. This is problematic as the new policy will be optimized on the dataset itself and will, initially, not be able to acquire new data by interacting with the environment. An additional challenge with offline RL is that the collected data may be imbalanced in terms of the action distribution in the dataset, and may consequently lack informative training samples for training of the RL policy for action selection.

[0009] Informative samples are samples in which the action taken corresponds to a change of the parameter (e.g., increase / decrease of DL TX power, antenna tilt angle, etc.). Considering for example the case of Remote Antenna Tilt (RET), the available actions are tilting up, tilting down and not changing the angle. In a realistic scenario, tilting does not happen very often. As a consequence, most of the samples in a collected dataset for offline RL training will be generated without any change on the parameters, that is the action for the sample will be the ‘no change' action. This makes it difficult for an RL agent trained on such data to evaluate the impact of the change actions (tilt up and tilt down) on the KPIs of interest.

[0010] Offline RL training algorithms consequently suffer from the scarcity of informative samples, i.e., unbalanced distribution between ‘no change' action samples and 'change' action samples. In many cases, ‘no change' action samples, referred to as non-informative samples, represent the majority of samples in a static dataset collected in the real world. RL agents trained on such datasets tend therefore to exhibit significant bias and overfitting in their behavior. This problem gets worse as the size of the input state (the number of parameters used to represent the current state of the network, for example the number of input Key Performance Indicators (KPIs)) increases. As the complexity of communication networks, and the capabilities to observe their state and performance, increase, the available features to be used to represent the network state are expected to increase as well. Being able to process this representation in an efficient manner will consequently become even more important. Use of latent representations in RL to address this particular challenge has been proposed, and WO2022072982 presents a management method involving two-levels of policy model in an online RL setup. The policy models include a high-level controller with a task policy model and a low level-controller with primitive policy model. The primitive policy model takes as input a latent representation of state generated by the task policy model. While presenting a method for managing large state representations, WO2022072982 does not address the challenges discussed above related to offline RL, and in particular the prevalence of unbalanced datasets and scarcity of informative samples.

[0011] It is an aim of the present disclosure to provide methods, a training node, a management node, and a computer program product which at least partially address one or more of the challenges mentioned above. It is a further aim of the present disclosure to provide methods, a training node, a management node and a computer program product which cooperate to facilitate offline training of a system for management of a communication network environment in such a manner as to address the issue of imbalance in offline training datasets.

[0012] According to a first aspect of the present disclosure, there is provided a computer implemented method for training a system to manage an environment in a communication network, wherein the system comprises an encoder Machine Learning (ML) model, and a policy ML model, and wherein the communication network environment is operable to perform a task. The method, performed by a training node, comprises obtaining a training dataset comprising records of task performance of the environment during a period of management according to a reference policy. Each record of task performance comprises: an observed context for the environment, an action selected for execution in the environment by the reference policy on the basis of the observed context, and a reward value indicating an observed impact of the selected action on task performance by the environment. The method further comprises using a Self-Supervised Learning (SSL) process and a subset of the obtained training dataset to train the encoder ML model to generate a latent representation of the observed context. The method further comprises using a Reinforcement Learning (RL) process and the obtained training dataset to train the policy ML model to generate an action for execution in the environment according to a latent representation of an observed context of the environment, the latent representation generated by the encoder ML model. The method further comprises outputting the trained encoder ML model and policy ML model for use in managing the communication network environment.

[0013] According to another aspect of the present disclosure, there is provided a computer implemented method for managing an environment in a communication network, wherein the communication network environment is operable to perform a task. The method, performed by a management node, comprises obtaining a trained system from a training node, wherein the system comprises an encoder ML model, and a policy ML model, and wherein the system has been trained using a method according to any one or more of the aspects or examples of the present disclosure. The method further comprises receiving an observed environment context from a communication network node, and using the encoder ML model to generate a latent representation of the observed context. The method further comprises using the policy ML model to generate an action for execution in the environment according to the generated latent representation, and causing execution of the generated action in the environment.

[0014] According to another aspect of the present disclosure, there is provided a computer program product comprising a computer readable non-transitory medium, the computer readable medium having computer readable code embodied therein, the computer readable code being configured such that, on execution by a suitable computer or processor, the computer or processor is caused to perform a method according to any one or more of the aspects or examples of the present disclosure.

[0015] According to another aspect of the present disclosure, there is provided a training node for training a system to manage an environment in a communication network, wherein the system comprises an encoder ML model and a policy ML model, and wherein the communication network environment is operable to perform a task. The training node comprises processing circuitry configured to cause the training node to obtain a training dataset comprising records of task performance by the environment during a period of management according to a reference policy. Each record of task performance comprises an observed context for the environment, an action selected for execution in the environment by the reference policy on the basis of the observed context, and a reward value indicating an observed impact of the selected action on task performance by the environment. The processing circuitry is configured to cause the training node to use an SSL process and a subset of the obtained training dataset to train the encoder ML model to generate a latent representation of the observed context. The processing circuitry is configured to cause the training node to use an RL process and the obtained training dataset to train the policy ML model to generate an action for execution in the environment according to a latent representation of an observed context of the environment, the latent representation generated by the encoder ML model. The processing circuitry is configured to cause the training node to output the trained encoder ML model and policy ML model for use in managing the communication network environment.

[0016] According to another aspect of the present disclosure, there is provided a management node for managing an environment in a communication network, wherein the communication network environment is operable to perform a task. The management node comprises processing circuitry configured to cause the management node to obtain a trained system from a training node, wherein the system comprises an encoder ML model and a policy ML model, and wherein the system has been trained using a method according to any one or more of the aspects of examples of the present disclosure. The processing circuitry is configured to cause the management node to receive an observed environment context from a communication network node, and use the encoder ML model to generate a latent representation of the observed context. The processing circuitry is configured to cause the management node to use the policy ML model to generate an action for execution in the environment according to the generated latent representation, and cause execution of the generated action in the environment.

[0017] Aspects of the present disclosure thus provide methods and nodes that cooperate to address the sparsity of informative action samples in offline RL for the management of communication network environments, such as for example a RAN environment. Examples of the present disclosure address the sparsity of informative action samples by improving the sample efficiency of training through use of a Self-Supervised Learning (SSL) technique in the training of an encoder ML model. Examples of the present disclosure encode an input observed state of the environment into a representation, which may be a lower dimensional representation. By using an SSL process for training this encoder with offline data, examples of the present disclosure are able to make use of non-informative samples from the offline training dataset in the training of the encoder ML model. As the encoder ML model provides the input to the policy ML model, the system for management of the environment is trained in a manner that makes much more efficient use of the available dataset than in known approaches, knowing that the available dataset may include few informative samples as discussed above.

[0018] Brief of the

[0019] For a better understanding of the present disclosure, and to show more clearly how it may be carried into effect, reference will now be made, by way of example, to the following drawings in which:

[0020] Figure 1 is a flow chart illustrating process steps in a computer implemented method for training a system to manage an environment in a communication network;

[0021] Figure 2 is a flow chart illustrating process steps in a computer implemented method for managing an environment in a communication network;

[0022] Figures 3a to 3d show flow charts illustrating another example of a method for training a system to manage an environment in a communication network;

[0023] Figures 4a and 4b show flow charts illustrating another example of a method for managing an environment in a communication network;

[0024] Figure 5 is a block diagram illustrating functional modules in an example training node;

[0025] Figure 6 is a block diagram illustrating functional modules in an example management node;

[0026] Figure 7 shows an example of an online management method provided by the present disclosure;

[0027] Figure 8 the sample efficiency of training achieved by examples of the present disclosure;

[0028] Figure 9 illustrates an example centralized implementation architecture in network optimization platforms for examples of the present disclosure;

[0029] Figure 10 illustrates a Generative Predictive Architecture;

[0030] Figure 11 illustrates example KPIs that were used as target output of prediction in an experimental validation; Figure 12 illustrates Joint Embedding Predictive Architecture;

[0031] Figure 13 illustrates RL policy ML model training;

[0032] Figure 14 shows a flowchart illustrating an implementation of both SSL and RL training; and

[0033] Figure 15 illustrates results of a performance evaluation of a method according to examples of the present disclosure.

[0034] Detailed Description

[0035] Examples of the present disclosure propose methods for training and inference in respect of using a system to manage a communication network environment. The proposed methods address the two challenges of offline RL discussed above, that is the lack of informative training samples and action imbalance. The challenges are addressed through the use of an encoder ML model and a policy ML mode, and the use of SSL to train the encoder ML model. In this manner, non-informative samples from the offline training dataset may nonetheless be used to facilitate policy learning.

[0036] Figure 1 is a flow chart illustrating process steps in a computer implemented method 100 for training a system to manage an environment in a communication network, wherein the system comprises an encoder Machine Learning (ML) model and a policy ML model, and wherein the communication network environment is operable to perform a task. The task may for example comprise the provision access to the communication network, and / or the provision of communication network services, and / or additional services over the communication network.

[0037] The method is performed by a training node, which may comprise a physical or virtual node, and may be implemented in a computer system, computing device or server apparatus and / or in a virtualized environment, for example in a cloud, edge cloud, an Open Radio Access Network, O-RAN, or fog deployment. The training node may for example be instantiated in a rApp of an O-RAN architecture. Examples of a virtual node may include a piece of software or computer program, a code fragment operable to implement a computer program, a virtualised function, or any other logical entity. The training node may for example be implemented in a core network of a communication network, or in a RAN. The communication network may for example comprise an LTE network, a New Radio (NR) network or any other existing or future communication network systems, and the training node may be implemented in a core network node, or in a Radio Access node, which itself may comprise a physical node and / or a virtualized network function that is operable to exchange wireless signals. In some examples, a Radio Access node may comprise a base station node such as a NodeB, eNodeB, gNodeB, or any future implementation of this functionality. The training node may encompass multiple logical entities, as discussed in greater detail below, and may for example comprise a Virtualised Network Function (VNF). In other examples, the Training node may be instantiated in a physical or virtual server in a centralised or cloud based deployment.

[0038] Referring to Figure 1 , the method 100 comprises, in a first step 110, obtaining a training dataset comprising records of task performance of the environment during a period of management according to a reference policy. As illustrated at 1101, each record of task performance comprises an observed context for the environment, an action selected for execution in the environment by the reference policy on the basis of the observed context, and a reward value indicating an observed impact of the selected action on task performance by the environment.

[0039] For the purposes of the present disclosure, an observed context for an environment comprises any measured, recorded or otherwise observed information about the state of the environment. The state of the environment comprises its current situation, condition, and / or circumstances. The state of the environment may include its configuration, the presence and position (within physical or radio space) of entities within the environment, requests currently being made of the environment, availability and / or requirements being made of resources within the environment, condition of such resources, etc. An observed context for the environment may thus include values of configurable parameters for the environment and / or its contents, values of measurable parameters for the environment and / or its contents, demands being made upon it, entities present within it, etc. As stated above, the environment of the method 100 is an environment within a communication network, such as a cell of a cellular network, a cell sector, a group of cells, network slice, etc. An observed context for an environment may therefore comprise one or more Key Performance Indicators (KPIs) for the environment, information about a number of wireless devices connecting to the communication network in the environment, etc. KPIs that might be included in the observed context include: coverage (for example measured through Reference Signal Received Power (RSRP) of a connected device), quality (including for example some measure or impact of interference between cells), capacity (for example, throughput of a connected device), an amount of interference, delay, latency of initial access, etc. In some examples, metrics on which KPIs are based may be measured per connected device. In such examples, the KPIs may comprise aggregated metrics, such as an average value, percentile, (for example, for edge user), average of the log, etc.

[0040] The reward value indicates an observed impact of the selected action on task performance by the environment. This may comprise a change in one or more KPI values following execution of the action, or any other value, combination of values, function using values as input etc. which provide an indication of how the selected action has impacted the ability of the environment to perform its task. For example, in the case of an environment comprising a cell of a RAN, the reward value may comprise a function of network coverage, quality, capacity, and / or any of the parameters noted above with reference to observed environment context.

[0041] The reference policy may be any policy, algorithm, heuristic, or ML model used to manage the environment during the management period.

[0042] Referring still to Figure 1 , in step 120, the method 100 comprises using a Self-Supervised Learning (SSL) process and a subset of the obtained training dataset to train the encoder ML model to generate a latent representation of the observed context. Selection of the subset is discussed in more detail below with reference to Figure 3a. In step 130, the method 100 comprises using a Reinforcement Learning (RL) process and the obtained training dataset to train the policy ML model to generate an action for execution in the environment according to a latent representation of an observed context of the environment, the latent representation generated by the encoder ML model. Finally, in step 140, the method 100 comprises outputting the trained encoder ML model and policy ML model for use in managing the communication network environment. Step 140 may for example comprise sending or otherwise providing the trained encoder ML model and policy ML model to a management node for use in managing the environment. The management node may be collocated with the training node, for example instantiated as a virtual node in the same logical entity as the training node, or may be instantiated in a different logical and / or physical location. For example, in a first alternative, the training node and management node may be collocated in a RAN node serving a particular cell or group of cells. In a second alternative, the training node may be located in a centralized cloud deployment, and the management node may be located in a RAN node serving a particular cell or group of cells.

[0043] It will be appreciated that the method 100 provides an offline training pipeline for the management system comprising the encoder and policy ML models. The obtaining of a training dataset comprising records from a period of management of the environment by the reference policy places the training pipeline of the method 100 in the realm of offline training. The method 100 involves the use of an SSL process and a subset of the training data (which may for example comprise just the observed contexts and not the actions and reward values as discussed in further detail below) to train the encoder ML model, and then using the full training dataset to train the policy ML model. The method 100 allows for the production of a trained encoder ML model that is more effective at encoding essential features of the observed context, owing to the SSL training that has been able to leverage insights from what would normally be non-informative samples of the training dataset (that is samples in which the action selected did not change a managed parameter). This more effective encoder results in a more effective policy model, as well as supporting much larger and more complicated observed contexts.

[0044] According to example of the present disclosure, generating an action for execution according to a latent representation of an observed context of the environment comprises generating the action on the basis of the latent representation, that is the latent representation is input to the policy ML model, processed by the policy ML model according to current values of its trainable parameters, and a generated action for execution in the environment is output by the policy ML model.

[0045] It will be appreciated that the method 100 does not require access to a live run time system in order to train the encoder and policy ML models, and the method may consequently be run in any centralised or cloud based deployment without incurring risk of performance degradation in the live environment to be managed.

[0046] The method 100 may be complemented by a method 200 performed by a management node, in which method the trained encoder ML model and policy ML model are used for managing an environment in a communication network.

[0047] Figure 2 is a flow chart illustrating process steps in a computer implemented method 200 for managing an environment in a communication network, wherein the communication network environment is operable to perform a task. The task may for example comprise the provision access to the communication network, and / or the provision of communication network services, and / or additional services over the communication network.

[0048] The method is performed by a management node, which may comprise a physical or virtual node, and may be implemented in a computer system, computing device or server apparatus and / or in a virtualized environment, for example in a cloud, edge cloud, an Open Radio Access Network, O-RAN, or fog deployment. The management node may for example be instantiated in a rApp of an O-RAN architecture. Examples of a virtual node may include a piece of software or computer program, a code fragment operable to implement a computer program, a virtualised function, or any other logical entity. The management node may for example be implemented in a core network of a communication network, or in a RAN. The communication network may for example comprise an LTE network, a New Radio (NR) network or any other existing or future communication network systems, and the management node may be implemented in a core network node, or in a Radio Access node, which itself may comprise a physical node and / or a virtualized network function that is operable to exchange wireless signals. In some examples, a Radio Access node may comprise a base station node such as a NodeB, eNodeB, gNodeB, or any future implementation of this functionality. The management node may encompass multiple logical entities, as discussed in greater detail below, and may for example comprise a Virtualised Network Function (VNF). In other examples, the management node may be instantiated in a physical or virtual server in a centralised or cloud based deployment.

[0049] Referring to Figure 2, the method 200 comprises, in step 210, obtaining a trained system from a training node, wherein the system comprises an encoder ML model and a policy ML model, and wherein the system has been trained using a method according to any one or more of the methods disclosed herein. The system may for example have been trained using the method 100 described above, and / or examples of the method 300 described below. The method 200 further comprises, in step 220, receiving an observed environment context from a communication network node, and in step 230, using the encoder ML model to generate a latent representation of the observed context. The observed context may be as discussed above with reference to the method 100. Using the encoder ML model to generate a latent representation of the observed context in step 230 may comprise inputting the observed context to the trained encoder ML model, wherein the encoder ML model is operable to process the input observed context according to trained values of its trainable parameters, and to output the latent representation.

[0050] Referring still to Figure 2, the method 200 further comprises, at step 240, using the policy ML model to generate an action for execution in the environment according to the generated latent representation, and, at step 250, causing execution of the generated action in the environment. Using the policy ML model to generate an action for execution in the environment according to the generated latent representation in step 240 may comprise inputting the latent representation to the trained policy ML model, wherein the policy ML model is operable to process the input observed context according to trained values of its trainable parameters, and to output the generated action. According to examples of the present disclosure, causing execution of the generated action in the environment may comprise sending a message, for example instruction execution of the action, to a destination node, wherein the destination node is operable to execute the action, or to instruct a further node to execute the action. The message may for example comprise the generated action. In some examples, if the generated action is an action that does not change a managed parameter, then the message may be omitted, and the absence of a message containing a new action may be interpreted by the destination node for the message as an instruction not to change the parameter value.

[0051] The method 100 trains an encoder ML model and a policy ML model, and the method 200 uses such ML models to manage a communication network environment. For the purposes of the present disclosure, the term "ML model” encompasses within its scope the following concepts: machine Learning algorithms, comprising processes or instructions through which data may be used in a training process to generate a model artefact for performing a given task, or for representing a real- world process or system; and the model artefact that is created by such a training process, and which comprises the computational architecture that performs the task.

[0052] Generally, an ML model, or a representation of an ML model, can be transmitted or transferred between nodes using any existing model format such as Open Neural Network Exchange, ONNX (https: / / onnx.ai), or formats used in commonly used toolboxes such as Keras or PyTorch. In some examples, the ML models trained by the method 100 and used in the method 200 may comprise Artificial Neural Networks. In other examples, the ML models may comprise linear functions, polynomials, decision trees, ensemble models, etc.

[0053] Figures 3a to 3d show flow charts illustrating another example of a computer implemented method 300 for training a system to manage an environment in a communication network, wherein the system comprises an encoder ML model and a policy ML model, and wherein the communication network environment is operable to perform a task. As for the method 100 discussed above, the method 300 is performed by a Training node, which may comprise a physical or virtual node, and may be instantiated substantially as discussed above with reference to the method 100. The method 300 illustrates examples of how the steps of the method 100 may be implemented and supplemented to provide the above discussed and additional functionality.

[0054] Referring initially to Figure 3a, in step 310, the training node obtains a training dataset comprising records of task performance of the environment during a period of management according to a reference policy. As illustrated at 310a, each record of task performance comprises an observed context for the environment, an action selected for execution in the environment by the reference policy on the basis of the observed context, and a reward value indicating an observed impact of the selected action on task performance by the environment. The observed context and reward value may be substantially as described above with reference to the method 100.

[0055] As illustrated at 310b and discussed above, the communication network environment may comprise at least a part of a Radio Access Network, a cell of a Radio Access Network, a cell sector of a Radio Access Network, a group of cells of a Radio Access Network, or a slice of the communication network; and the task that the environment is operable to perform may comprise provision of connectivity to the communication network, provision of communication network services, and / or provision of other services over the communication network.

[0056] As illustrated at 310c, in some examples the environment may comprise at least one entity, a parameter of which is operable to be managed, and in the training dataset, the actions selected for execution in the environment by the reference policy on the basis of the observed context may comprise actions controlling a value of the parameter, the actions comprising at least one of an incremental change or no change to the parameter value.

[0057] For the purposes of the present disclosure, an entity comprised in a communication network environment is a physical or virtual element within the environment. The entity may for example be an entirely virtual construct, such as a virtual node, or may be a physical element such as an antenna, a basestation, etc. The entity may have a logical identity, and be associated with one or more parameters. Management of the entity may be conducted through adjustments of the values of the one or more parameters associated with the entity. Examples of entities which may be comprised within a communication network environment include a Radio Access Node, and example parameters associated with the entity may include Antenna tilt, pan, azimuth, SSB beam shapes, DL TX power, P0 nominal PUSCH, etc. It will be appreciated that the concept of an entity within a communication network environment, the entity being associated with one or more parameters whose values may be adjusted, does not encompass natural persons who may be associated with a device that is operable to connect to the communication network. As discussed above and illustrated at 31 Od, the communication network environment may comprise at least one Radio Access Node, and management of the communication network may comprise controlling a value of a parameter of the radio access node. As discussed above, a Radio Access Node of a communication network comprises a node that is operable to transmit, receive, process and / or orchestrate wireless signals. A RAN node may comprise a physical node and / or a virtualised network function. In some examples, a RAN node may comprise a NodeB, eNodeB, gNodeB, etc., or any other current or future implementation of such functionality.

[0058] Referring still to Figure 3a, in step 320, the training node uses a Self-Supervised Learning (SSL) process and a subset of the obtained training dataset to train the encoder ML model to generate a latent representation of the observed context. In some examples, the latent representation of the observed context may be of a lower dimensionality than the observed context.

[0059] As illustrated at 320a, using an SSL process and a subset of the obtained training dataset to train the encoder ML model to generate a latent representation of the observed context may comprise using a predictive architecture for the SSL process. The predictive architecture may for example comprise at least one of a Generative Predictive Architecture or a Joint Embedding Predictive Architecture.

[0060] In some examples, as shown at 320b, the subset of the obtained training dataset comprises a time series of observed contexts for the environment, each entry in the time series extracted from a record in the obtained training dataset. As illustrated at 320c, the time series of observed contexts for the environment may comprise pairs of observed contexts, each pair comprising an input observed context extracted from an input record of task performance, and a target observed context extracted from a record of task performance following execution of the selected action from the input record of task performance. It will be appreciated that the target observed context may be extracted from a record of task performance immediately following execution of the selected action from the input record of task performance, such that the input and target observed contexts may be consecutive contexts in the time series.

[0061] In some examples of the present disclosure, the time series of observed contexts for the environment may comprise pairs of observed contexts for which the selected action from the input record of task performance is an action that imposes no change on the environment, as shown in step 320d. In further examples, as shown at 320e, the time series of observed contexts for the environment may comprise pairs of observed contexts for which the selected action from the input record of task performance is an action comprising no change to the parameter value. It will be appreciated that according to such examples the encoder ML model is trained to generate a latent representation of an observed environment context using only consecutive environment contexts from the time series in which there is no imposed change on the environment through a change in the relevant managed parameter.

[0062] It will be appreciated from the above discussion that the subset may generally contain data that includes only context, not action, and contexts for which only certain actions were selected, for example contexts for which the action selected was the same (this may be the no change action as discussed above). Once the data are filtered in this manner, the subset may be obtained using a standard train / test split of the dataset (for example using 20% of the data points for validations). In other examples, the subset can be obtained after removing outliers in the dataset or other ad-hoc cleaning techniques may be used such as removing data point with missing values, and / or removing datapoints from nodes with higher privacy requirements. Steps that may be performed in order to carry out using an SSL process and a subset of the obtained training dataset to train the encoder ML model to generate a latent representation of the observed context are illustrated in Figure 3c. Referring now to Figure 3c, in order to use an SSL process and a subset of the obtained training dataset to train the encoder ML model to generate a latent representation of the observed context, the Training node may initially use the encoder ML model to generate a latent representation of an observed context from the subset at step 321. The training node may then use the generated latent representation for a prediction task relating to the observed context in step 322, before updating values of trainable parameters of the encoder ML model to minimize a loss function evaluating performance of the prediction task in step 323. In some examples, the updating may performed be via back propagation.

[0063] As discussed above, use the encoder ML model to generate a latent representation of an observed context may comprise inputting the observed context to the encoder ML mode, wherein he encoder ML model is operable to process the input context in accordance with current vales of the trainable parameters of the encoder ML model, and to output the latent representation.

[0064] As illustrated at 322a, in some examples, the prediction task may comprise predicting at least one of an observed context or a latent representation of an observed context. As illustrated at 322b, predicting at least one of an observed context or a latent representation of an observed context may comprise predicting at least one of the next observed context in the time series, or a latent representation of the next observed context in the time series. According to some examples of the present disclosure, the loss function evaluating performance of the prediction task, used in step 323, can be a comparison of the predicted next observed context with the next observed context in the time series, or can be a comparison in latent space, i.e., a comparison of the predicted latent representation with an actual latent representation of the next observed context. The latent representation can be generated using an exponential moving average (EMA) of the encoder ML model being updated.

[0065] It will be appreciated that the SSL training of the encoder ML model, for example using a prediction task relating to the next observed context in a time series, enables training of the encoder ML model regardless of the presence or absence of change actions in the training dataset. The encoder ML model may be trained to preserve in the latent representation the most important features of the context for predicting the evolution of the environment, independently of change actions that may influence the state of the environment. By encapsulating this knowledge in the encoder ML model, and using the encoder ML model to generate an input for the policy ML model, this knowledge of the evolution of the environment may be used indirectly to train the policy ML model, so making a more efficient use of the data samples in the training data set than if training were focused on maximizing the information available from samples associated with a change action. As discussed above in this disclosure, such samples tend to be scarce in real world data from communication networks.

[0066] Returning to the method 300, and referring now to Figure 3b, following training of the encoder ML model, the training node then uses a Reinforcement Learning (RL) process and the obtained training dataset to train the policy ML model to generate an action for execution in the environment according to a latent representation of an observed context of the environment in step 330, the latent representation generated by the encoder ML model. As illustrated at 330a, generating an action for execution in the environment according to a latent representation of an observed context of the environment may comprise inputting the latent representation to the policy ML model, wherein the policy ML model is operable to process the input latent representation according to current values of its trainable parameters, and to output a generated action for execution in the environment. In some examples, as discussed above and as illustrated at 330b, using an RL process and the obtained training dataset to train the policy ML model to generate an action for execution in the environment according to a latent representation of an observed context of the environment, the latent representation generated by the encoder ML model, may comprise using the trained encoder ML model to generate latent representations of the environment for training the policy ML model.

[0067] In further examples, using an RL process and the obtained training dataset to train the policy ML model to generate an action for execution in the environment according to a latent representation of an observed context of the environment, the latent representation generated by the encoder ML model, may further comprise at least one of freezing values of trainable parameters of the encoder ML model during training of the policy ML model, and / or updating values of trainable parameters of the encoder ML model during training of the policy ML model, as shown at step 330c.

[0068] Steps that may be performed in order to carry out RL training of the policy ML model at step 330 are illustrated in Figure 3d. Referring now to Figure 3d, in order to train the policy ML model, the training node may initially use the trained encoder ML model to generate a latent representation of an observed context from the training dataset at step 331. The training node may then use the policy ML model to generate an action for execution in the environment according to the generated latent representation in step 332, and updating values of trainable parameters of the policy ML model in step 333 to minimize a loss function. Parameter updating may be performed via back propagation. As illustrated at step 333, the loss function may evaluate the action generated by the policy ML model against at least one of: the action selected for execution in the environment by the reference policy on the basis of the observed context from the training dataset, and / or the reward value indicating an observed impact of the selected action from the training dataset on task performance by the environment.

[0069] It will be appreciated that using the policy ML model to generate an action for execution in the environment according to the generated latent representation may comprise inputting the generated latent representation to the policy ML model, the policy ML model processing the input latent representation using current values of its trainable parameters, and then outputting the generated action for execution in the environment.

[0070] Examples of offline RL algorithms (and appropriate example cost functions in parentheses) that may be used by the training node at step 330 include direct method (predict expected accumulated reward, use MSE loss), propensity based method (same cost function as for direct method but adds importance sampling weight), conservative Q learning (use the Bellman error with a regularization term), online off-policy algorithms (SAC, DQN) used on a dataset (different variant of the Bellman error), and imitation learning (use maximum likelihood cost to predict the probability of taking an action in a given state, possibly conditioned on the reward value).

[0071] Referring again to Figure 3b, following RL training of the policy ML model at step 330, the training node then, at step 340, outputs the trained encoder ML model and policy ML model for use in managing the communication network environment. As illustrated at 340a, the encoder ML model and policy ML model may in some cases comprise Artificial Neural Networks (ANNs), and may be output in any appropriate format. For example, the encoder ML model and policy ML model may be transferred to a management node using any existing model format such as ONNX (https: / / onnx.ai), or formats used in commonly used toolboxes such as Keras or PyTorch. As illustrated at 340b, outputting the trained encoder ML model and policy ML model for use in managing the communication network environment may comprise providing the trained encoder ML model and policy ML model to a management node, wherein the management node is operable to use the trained encoder ML model and policy ML model to manage the environment. The management node may for example use the trained models by performing he method 200 discussed above, or the method 400 presented below.

[0072] As illustrated at step 350, in some examples, the training node may update values of trainable parameters of at least one of the encoder ML model or the policy ML model during use of the system to manage the environment. This updating may be performed by the training node as part of the method 300, or by a management node using the trained encoder ML model and policy ML model as part of method 400.

[0073] Figures 4a and 4b show flow charts illustrating another example of a computer implemented method 400 for managing an environment in a communication network, wherein the communication network environment is operable to perform a task. As for the method 200 discussed above, the method 400 is performed by a management node, which may comprise a physical or virtual node, and may be instantiated substantially as discussed above with reference to the method 200. The method 400 illustrates examples of how the steps of the method 200 may be implemented and supplemented to provide the above discussed and additional functionality.

[0074] Referring initially to Figure 4a, in step 410, the management node obtains a trained system from a training node, the system comprising an encoder ML model and a policy ML model. The system has been trained using examples of the method 100 and / or 300. As illustrated at 410a, the environment to be managed using the method 400 may comprise at least one entity, a parameter of which is operable to be managed. The entity may be as discussed above with reference to the method 300.

[0075] Each of the encoder ML model and the policy ML model of the trained system may comprise an ANN, as illustrated at 410a.

[0076] In step 420, the management node receives an observed environment context from a communication network node. The observed context may be as discussed above with reference to the method 300.

[0077] As illustrated at 420a, the environment described by the observed context, and to be managed using the method 400, may comprise at least one entity and a parameter related to that entity which is operable to be managed. The entity may be as discussed above with reference to the method 300.

[0078] In some examples the entity may comprise a radio access node, and, as illustrated at 420c, the communication network environment may comprise at least one Radio Access Node, and management of the communication network may comprise controlling a value of a parameter of the radio access node. Such control may comprise incremental change (increase or decrease), or no change to the parameter value. As discussed above, a Radio Access Node of a communication network comprises a node that is operable to transmit, receive, process and / or orchestrate wireless signals. A RAN node may comprise a physical node and / or a virtualised network function. In some examples, a RAN node may comprise a NodeB, eNodeB, gNodeB, etc., or any other current or future implementation of such functionality.

[0079] As illustrated at 420b, in some examples, the communication network environment may comprise at least a part of a Radio Access Network, a cell of a Radio Access Network, a cell sector of a Radio Access Network, a group of cells of a Radio Access Network, or a slice of the communication network; and the task that the environment is operable to perform may comprise any one or more of provision of communication network access, provision of communication network services, and or provision of services via the communication network.

[0080] Referring still to Figure 4b, in step 430, the management node uses the encoder ML model to generate a latent representation of the observed context. This may comprise inputting the received observed context to the trained encoder ML model, the encoder ML model being operable to process the input observed context according to the trained values of its trainable parameters, and to output the generate latent representation, for example in the form of a vector of latent space values. In some examples, the latent representation of the observed context may be of a lower dimensionality than the observed context.

[0081] Referring now to Figure 4b, in step 440, the management node uses the policy ML model to generate an action for execution in the environment according to the generated latent representation. As illustrated at 440a, this may comprise inputting the generated latent representation to the policy ML model, wherein the policy ML model is operable to process the input latent representation according to current values of its trainable parameters, and to output a generated action for execution in the environment. As illustrated at 440b, the action generated for execution in the environment by the policy ML model may comprise an action controlling a value of the parameter referred to above, the action comprising at least one of an incremental change or no change to the parameter value.

[0082] In step 450, the management node causes execution of the generated action in the environment. As discussed above with reference to the method 200, according to examples of the present disclosure, causing execution of the generated action in the environment may comprise sending a message, for example instruction execution of the action, to a destination node, wherein the destination node is operable to execute the action, or to instruct a further node to execute the action. The message may for example comprise the generated action. In some examples, if the generated action is an action that does not change a managed parameter, then the message may be omitted, and the absence of a message containing a new action may be interpreted by the destination node for the message as an instruction not to change the parameter value.

[0083] In step 460 of the method 400, the management node may obtain a reward value indicating an observed impact of the generated action on task performance by the environment. The reward value may for example comprise a function of one or more KPIs for performance of the environment task. As discussed above with reference to the observed environment context, example KPIs that might be included within the reward function may include: coverage (for example measured through Reference Signal Received Power (RSRP) of a connected device), quality (including for example some measure or impact of interference between cells), capacity (for example, throughput of a connected device), an amount of interference, delay, latency of initial access, etc. For metrics measured per connected device, the KPIs may comprise aggregated metrics, such as an average value, percentile, (for example, for edge user), average of the log, etc. The function may for example comprise a weighted average of KPIs, or other function. In step 470, the management node may update values of trainable parameters of at least one of the encoder ML model or the policy ML model according to the obtained reward value. This step may encompass the online updating of training of the encoder ML model and / or policy ML model discussed above with reference to the method 300.

[0084] As discussed above, the methods 100 and 300 may be performed by a training node, and the present disclosure provides a training node that is adapted to perform any or all of the steps of the above discussed methods. The training node may comprise a physical node such as a computing device, server etc., or may comprise a virtual node. A virtual node may comprise any logical entity, such as a Virtualized Network Function (VNF) which may itself be running in a cloud, edge cloud or fog deployment. The training node may be operable to be instantiated in a cloud based deployment, as an rApp, in a physical or virtual radio access node, control network node, etc.

[0085] Figure 5 is a block diagram illustrating an example training node 500 which may implement the method 100 and / or 300, as illustrated in Figures 1 and 3a to 3d, according to examples of the present disclosure, for example on receipt of suitable instructions from a computer program 550. Referring to Figure 5 the training node 500 comprises a processor or processing circuitry 502, and may comprise a memory 504 and interfaces 506. The processing circuitry 502 is operable to perform some or all of the steps of the method 100 and / or 300 as discussed above with reference to Figures 1 and 3a to 3d. The memory 504 may contain instructions executable by the processing circuitry 502 such that the training node 500 is operable to perform some or all of the steps of the method 100 and / or 300, as illustrated in Figures 1 and 3a to 3d. The instructions may also include instructions for executing one or more telecommunications and / or data communications protocols. The instructions may be stored in the form of the computer program 550. In some examples, the processor or processing circuitry 502 may include one or more microprocessors or microcontrollers, as well as other digital hardware, which may include digital signal processors (DSPs), special-purpose digital logic, etc. The processor or processing circuitry 502 may be implemented by any type of integrated circuit, such as an Application Specific Integrated Circuit (ASIC), Field Programmable Gate Array (FPGA) etc. The memory 504 may include one or several types of memory suitable for the processor, such as read-only memory (ROM), random-access memory, cache memory, flash memory devices, optical storage devices, solid state disk, hard disk drive, etc.

[0086] As discussed above, the methods 200 and 400 may be performed by a management node, and the present disclosure provides a management node that is adapted to perform any or all of the steps of the above discussed methods. The management node may comprise a physical node such as a computing device, server etc., or may comprise a virtual node. A virtual node may comprise any logical entity, such as a Virtualized Network Function (VNF) which may itself be running in a cloud, edge cloud or fog deployment. The management node may be operable to be instantiated in a cloud based deployment, as an rApp, in a physical or virtual radio access node, control network node, etc.

[0087] Figure 6 is a block diagram illustrating an example management node 600 which may implement the method 200 and / or 400, as illustrated in Figures 2 and 4a to 4b, according to examples of the present disclosure, for example on receipt of suitable instructions from a computer program 650. Referring to Figure 6, the management node 600 comprises a processor or processing circuitry 602, and may comprise a memory 604 and interfaces 606. The processing circuitry 602 is operable to perform some or all of the steps of the method 200 and / or 400 as discussed above with reference to Figures 2 and 4a to 4b. The memory 604 may contain instructions executable by the processing circuitry 602 such that the management node 600 is operable to perform some or all of the steps of the method 200 and / or 400, as illustrated in Figures 2 and 4a to 4b. The instructions may also include instructions for executing one or more telecommunications and / or data communications protocols. The instructions may be stored in the form of the computer program 650. In some examples, the processor or processing circuitry 602 may include one or more microprocessors or microcontrollers, as well as other digital hardware, which may include digital signal processors (DSPs), special-purpose digital logic, etc. The processor or processing circuitry 602 may be implemented by any type of integrated circuit, such as an Application Specific Integrated Circuit (ASIC), Field Programmable Gate Array (FPGA) etc. The memory 604 may include one or several types of memory suitable for the processor, such as read-only memory (ROM), random-access memory, cache memory, flash memory devices, optical storage devices, solid state disk, hard disk drive, etc.

[0088] Figures 1 to 4b discussed above provide an overview of methods which may be performed according to different examples of the present disclosure. These methods may be performed by a training node and a management node respectively, as illustrated in Figures 5 and 6. These methods enable the efficient training of a system for managing an environment in a communication network. There now follows a detailed discussion of how different process steps illustrated in Figures 1 to 4b and discussed above may be implemented. The functionality and implementation detail described below is discussed with reference to the modules of Figures 5 and 6 performing examples of the methods 100, 200, 300 and / or 400, substantially as described above.

[0089] As discussed above, in order to address the sparsity of informative action samples in offline RL in communication network parameter optimization applications, the present disclosure proposes methods to improve sample efficiency of training by using an SSL. By encoding the input feature (observed context) into a lower dimensional representation, the proposed methods enable the policy ML model to learn more efficiently from scarce informative samples.

[0090] An example of the online management method provided by examples of the present disclosure is illustrated in Figure 7. The management makes use of a system comprising two models, referred to as encoder and policy ML models. As illustrated in Figure 7, the encoder and policy ML models are trained with SSL and RL, respectively. The encoder ML model takes an observed context comprising environment state features as input and outputs a latent vector representation which typically has lower dimension than the state features but includes useful information for decision making in RL. The RL policy ML model takes as input the latent vector from the encoder ML model and outputs an action proposal.

[0091] Examples of the present disclosure, and particularly the proposed methods 100 and 300, enable the design and training an encoder ML model by fully leveraging abundant but less informative (i.e. , ‘no change') action samples using SSL. In SSL, the encoder ML model is pre-trained for example using predictive tasks, such as predicting future states. These abundant samples usually come from the various number of network nodes being monitored and / or acted upon. Even if they do not inform about the consequence of actions, they still contain a great deal of information that can be exploited since they all have different contexts.

[0092] The example methods proposed herein enable the system to make the most of the available data by learning meaningful representations from the unlabeled samples, i.e., from state data without an action label. By using SSL and decoupling encoding of environment contexts from policy training, example methods according to the present disclosure overcome the challenges posed by the sparsity of informative action samples in offline RL.

[0093] Figure 8 illustrates the sample efficiency of training achieved by examples of the present disclosure. As illustrated in Figure 8, methods disclosed herein make use of plentiful but less informative samples (i.e., ‘no change' action samples) to design a high performing encoder based on SSL techniques. This enables the encoder to capture useful information in a lower dimensional latent vector, which in turn helps the policy RL model learn in a more sample-efficient way. It will be appreciated that as the number of observations included in the input observed context increases, so the advantages and efficiency offered by the methods proposed herein also increase. Examples methods according to the present disclosure may be particularly suited to use cases in which online or offline RL policy takes a decision on incremental changes in parameters and learns from its reward. Specifically, the methods offer particular advantages in performance when the action samples in the data are distributed in an unbalanced manner, and skewed to non- informative samples in which the action taken was no change to the managed parameter.

[0094] Figure 9 illustrates an example centralized implementation architecture in network optimization platforms for examples of the present disclosure. The training and management nodes forming the system to be trained and used may run as a centralized application or module in any network optimization platform, with the observed data collected to a central server from base stations. Based on those measurements, the management node, using the trained encoder and policy ML models, can make a decision on change of parameters at entities within the environment to be managed, which could for example comprise every single base station antenna within the environment comprising a cell, cell sector, group of cells, etc.

[0095] The training node may be formed of two logical units or trainers: an SSL-trainer and an RL-trainer, each being responsible for training the corresponding model. Both trainers use a training dataset in the form of offline static data provided by data lake at a central server.

[0096] Example implementations of different method steps of the methods 100 to 400 are presented below with reference to Figures 10 to 15.

[0097] Encoder training with SSL and target output composition - steps 120 of method 100, and 320, 320a- e, 321, 322, 322a-b and 323 of method 300.

[0098] The target output composition in SSL-training samples depends on which SSL technique is used to train the encoder ML model. As discussed above, the encoder ML model can be represented by any differentiable machine learning model such as neural networks.

[0099] The SSL training in the present example takes a predictive architecture, in which the encoder ML model constructs a latent representation based on a prediction task. The rationale behind this design is that the training enables the latent vector generated by the encoder to capture the underlying dynamics of the environment state, as represented by specific KPIs. The policy ML model is then able to consider such dynamics when it makes a decision on action for next time slots.

[0100] First example: Generative Predictive Architecture

[0101] One SSL technique that may be used for training the encoder ML model is Prediction-based architecture illustrated in Figure 10 and described, for example, in l-JEPA: The first Al model based on Yann LeCun's vision for more human-like Al

[0102] Referring to Figure 10, the observed context is represented as an input state feature, which can be any observation in Performance Monitoring (PM) counters, preprocessed KPIs in Self Organising Networks (SON), current antenna configuration in Configuration Management (CM) counters, and / or physical positions such as x, y coordinates, antenna height, etc. The target output of prediction (provided by the prediction head) is a list of KPIs that are varying over time steps. By evaluating the prediction error between predicted output and ground-truth, the back-propagation gradient update may be conducted to obtain the trained encoder ML model during SSL training. As the encoder training in this example does not need action information, one option is to use only training samples with the same action (e.g., no change) to keep consistency in the underlying pattern to predict. Given a state at a time step, different actions will have a different impact on the environment state, and so lead to different next states at following time steps. The impact of different actions can therefore act as a noise when learning a pattern of state dynamics without taking action as input. To avoid this noise, encoder ML training may use input and target output observed contexts corresponding to ‘no change' action samples for training the SSL encoder with predictive architecture (as set out for example at steps 320d and 320e of method 300).

[0103] Example KPIs that were used as target output of prediction in an experimental validation are illustrated in Figure 11.

[0104] In the present example of generative predictive architecture for encoder ML model training, the prediction error is evaluated in the state feature space, typically using Mean Squared Error (MSE), or cosine similarity.

[0105] Second example: Joint Embedding Predictive Architecture (JEPA)

[0106] Another SSL technique that may be used for training the encoder ML model is Joint Embedding Predictive architecture illustrated in Figure 12 and described, for example, the reference cited above. As illustrated in Figure 12, JEPA training applies the encoder ML model to observed contexts (state features) at both timestep t and t+1. Then, the cost function evaluates the prediction error in the latent space. The cost function can be MSE or cosine similarity.

[0107] Referring to Figure 12, the two encoder ML models of the JEPA architecture can be the same or partially shared. For stabilizing the training, techniques such as the use of target networks may be used. The target network consists of an EMA of the encoder ML model in the upper part (corresponding to state at time t) in Figure 12. The target network is used as the encoder ML model in the lower part in Figure 12 that processes the state feature at time t+1 . In other words, the parameters of the encoder ML model for encoding the future state (at time t+1 in Figure 12) is updated by EMA of the parameters of the updated encoder ML model (at time t in Figure 12).

[0108] RL policy training and inference with the trained encoder ML model - steps 130 of method 100 and 330, 330a-c, 331, 332, 333 of method 300, and method 400.

[0109] RL policy ML model training is illustrated in Figure 13.

[0110] RL policy training and inference, performed by the training node and management node respectively, may in some examples both run at RLJrainer at the centralized server. RLJrainer receives a trained encoder ML model from SSLJrainer. As in Figure 13, the encoder and policy ML models are cascaded in a chain for action inference. Then, using the reward signal and corresponding cost function, the gradient update of the RL policy ML model is conducted via back-propagation during training. The cost function and gradient are determined based on the selected offline RL training algorithm. In this update, the encoder ML model trainable parameters may in many examples be frozen, but they may also be updated through back- propagation.

[0111] A flowchart illustrating an implementation of both SSL and RL training is illustrated in Figure 14. Use Cases

[0112] As discussed above, example methods according to the present disclosure may be applied to management of a range of different communication network environments. The following example use cases are illustrative of the types of management in such environments that may be carried out using the methods disclosed herein.

[0113] Use Case 1 : Remote Electronic Tilt optimization

[0114] Antenna tilt optimization is a problem to which RL has been successfully applied. In order to be able to efficiently provide a high level of Quality of Service (QoS) to users, networks must adjust their configuration in an automatic and timely manner. Antenna vertical tilt angle, referred to as downtilt angle, is one of the most important variables to control for QoS management. The problem is to adjust the antenna tilt for each individual cell in light of a plurality of cell and user locations as well as the current antenna tilts of neighbouring cells. The objective is to maximize some trade-off between capacity (for example total cell throughput) and coverage (for example 5thpercentile user throughput).

[0115] For the purposes of this use case, a management node may be implemented in a Radio Access Node, which may for example comprise a Centralised Unit serving one or more Distributed Units, each Distributed Unit comprising one or more antennas. In another example, a management node may be implemented in a core network node. The actions generated by the policy ML model comprise, for the purposes of this use case, adjustments to the downtilt angle of individual antennas for cells in the environment. The observed context may include configuration management counters (current tilt, mechanical tilt, downlink power), performance management counters (containing coverage, quality, capacity KPIs), deployment related information (context of neighbouring cells, distance to neighbours, antenna height, etc). The performance management counters may fir example include RSRP of UEs at cell edge, average SINR and RRC congestion rate. The reward function may comprise a weighted sum of the performance indicators.

[0116] Use Case 2: PO Nominal PUSCH

[0117] PO Nominal PUSCH is another problem that has been studied in the context of RL. While increasing SINR, higher transmit power not only increases the received power for the serving cell but also creates higher interference towards users in other cells.

[0118] For the purposes of this use case, a management node may be implemented in a Radio Access Node, which may for example comprise a Centralised Unit serving one or more Distributed Units, each Distributed Unit comprising one or more antennas. In another example, a management node may be implemented in a core network node. The actions generated by the policy ML model comprise, for the purposes of this use case, adjustments to the PO Nominal PUSCH for individual cells in the environment. The observed context may include the parameters discussed above with reference to Use Case 1, and may also include configuration of the uplink power control algorithm (fractional power factor, closed loop component, max output power), as well as some interference related KPIs. The reward function may comprise a weighted sum of the performance indicators.

[0119] Use Case 3: Downlink Power Control Downlink power control setting for radio networks is another problem that has been studied in the context of reinforcement learning. Traditionally, downlink transmit power for RAN nodes is set to use the maximum available power. However, higher transmit power not only increases the received power for the UE but also creates higher interference towards users in other cells. As different UEs require different data rates, some gains may be expected if downlink transmit powers were to be adjusted based on the current situation in the network.

[0120] For the purposes of this use case, a management node may be implemented in a Radio Access Node, which may for example comprise a Centralised Unit serving one or more Distributed Units, each Distributed Unit comprising one or more antennas. In another example, a management node may be implemented in a core network node. The actions generated by the policy ML model comprise, for the purposes of this use case, adjustments to the downlink power for individual cells in the environment. The observed context may the parameters discussed above with reference to Use Case 1 , and may also include SINRs, path gains and current powers. The reward function may comprise a weighted sum of the performance indicators.

[0121] The three use cases discussed above all relate to optimization of Radio Access Network parameters. For such use cases, an rApp in O-RAN architecture other platform would be a suitable implementation option. The rApp can be implemented in the non-real time RIC and the interfaces 01 , A1, and 02 can be used to collect the data from the network nodes and to send the change action.

[0122] It is also possible to implement the SSL trainer as an independent rApp. The learned representation could be used by one or more RL trainer rApps possibly for different tasks. In this context, the R1 interface would be used along with the publish / subscribe approach in the O-RAN non real-time RIC to communicate between the SSL Trainer and the RL trainer.

[0123] The above use cases relate to the Radio Access Network of a communication network. However, it will be appreciated that other, non-RAN, use cases for the methods disclosed herein may also be envisaged. In a first example, methods according to the present disclosure could be used for admission control in network slicing. In such an example, the observed environment context could include current load levels in the existing slices, and QCI value of the user requesting access. The action would be accepting the user to a given slice, or rejecting the user. This example could be applied to slicing in the transport or core network, but could also be applied to slicing in the RAN network. In another example, methods according to the present disclosure could be used for network management automation with intent-based closed loops. This example could be related to slicing when there are multiple services that require orchestration and draw from the same pool of resources. In this example the observed environment context could include throughput, packet loss, service priority, MBR, latency, Quality of Experience, etc. The actions for this example would depend on the formulation but may include adjustment of any of the configuration parameters that affect the environment, including for example, Maximum Bit Rate (MBR), Priority, Computational resources, etc.

[0124] Performance evaluation

[0125] For the purposes of evaluation of an example implementation of the methods disclosed herein, the use case of Remote Electrical Tilt (RET) optimization is considered with a real-world dataset.

[0126] Dataset and scenario: The evaluation uses an Ericsson Centralized Self Organizing Network (ECSON) dataset collected for two months in a customer network. During the data collection period, RET of each cell is updated every weekday by a SON Fuzzy logic algorithm. More than 95% of samples in the dataset have ‘no change' action.

[0127] State and action feature

[0128] The action set of the policy is {‘no change', 'uptilt', 'downtilt'}.

[0129] The input state feature of the observed context include KPI measurement such as 'Coverage', 'Quality', 'Capacity', 'Interference', 'Congestion', etc. Input state features also include CM counters and physical configuration such as ‘carrier frequency', ‘antenna height', etc.

[0130] Training algorithm

[0131] SSL training uses a Generative Predictive architecture. By having a prediction target output as set out above, the encoder ML model is trained using ‘no change' action samples. The size of latent representation vector is 5.

[0132] RL training uses an offline RL training algorithm propensity DM technique, described in WC2022199792. During RL policy training, the encoder ML model trainable parameters values are frozen.

[0133] Result

[0134] Two performance metrics are used to evaluate the example implementation: Inverse Propensity Score (IPS) estimator, as discussed in WC2022199792, and updown-reward (average reward only for uptilt and downtilt actions). Both techniques provide methods to evaluate the performance of trained encoder and policy ML model on the offline static test dataset.

[0135] Figure 15 illustrates the estimation of IPS and updown-reward throughout the training epoch on the test dataset. 'SSL-encoder' is the curve for example implementation in accordance with the present disclosure. 'Fuzzy-feature' and ‘KPI-feature’ are policy ML models trained by offline RL algorithms. 'Fuzzy-feature' takes only two KPIs (coverage and quality DOF KPIs) as input, and both 'KPI-feature' and 'SSL-encoder' take all available KPIs and CM counters as input. It can be observed that 'SSL-encoder' exhibits better sample efficiency (i.e., faster convergence) in terms of the IPS-Reward metric and training td_error compared to other tested offline RL methods. Owing to the better sample efficiency, it is anticipated that the SSL-encoder will be particularly advantageous when dealing with larger feature sets including more KPIs

[0136] According to another aspect of the present disclosure, there is provided an arrangement comprising a training node and a management node, wherein the arrangement is configured to train and use a system for managing an environment in a communication network, wherein the environment is operable to perform a task, and wherein the system comprises an encoder ML model and a policy ML model.

[0137] The training node of the arrangement comprises processing circuitry configured to cause the training node to obtain a training dataset comprising records of task performance by the environment during a period of management according to a reference policy. Each record of task performance comprises an observed context for the environment, an action selected for execution in the environment by the reference policy on the basis of the observed context, and a reward value indicating an observed impact of the selected action on task performance by the environment. The processing circuitry is configured to cause the training node to use an SSL process and a subset of the obtained training dataset to train the encoder ML model to generate a latent representation of the observed context. The processing circuitry is configured to cause the training node to use an RL process and the obtained training dataset to train the policy ML model to generate an action for execution in the environment according to a latent representation of an observed context of the environment, the latent representation generated by the encoder ML model. The processing circuitry is configured to cause the training node to output the trained encoder ML model and policy ML model for use in managing the communication network environment.

[0138] The management node of the arrangement comprises processing circuitry configured to cause the management node to obtain a trained system from the training node. The processing circuitry is configured to cause the management node to receive an observed environment context from a communication network node, and use the encoder ML model to generate a latent representation of the observed context. The processing circuitry is further configured to cause the management node to use the policy ML model to generate an action for execution in the environment according to the generated latent representation, and cause execution of the generated action in the environment.

[0139] Examples of the present disclosure, as set out above, address two significant challenges in offline RL, the lack of informative training samples and action imbalance, by using non-informative samples to facilitate the policy learning. The present disclosure proposes a sequential training procedure, in which an encoder ML model is first trained, for example using only ‘no change' action samples, using SSL algorithms. Secondly, a policy ML model is trained using RL algorithms. Additional training of the encoder ML model using RL algorithms may also be carried out.

[0140] Contrary to known methods, the SSL training may use a particular sample composition, including time series data of evolving state representations. The state representation may be associated with ‘no change' action samples. It will be appreciated that the SSL training and RL training consequently use two different data streams, and a suitable indication may be provided enabling each training instance to identify its own data stream.

[0141] The method disclosed herein differ from the methods proposed in WO2022072982, discussed in the Background section, in that the present disclosure proposes a method to obtain latent representation encoding through SSL in an offline RL setup, and in that the combination of SSL and RL, with the possibility to learn from ‘no change' action samples, is particularly suited to unbalanced datasets skewed towards uninformative action samples, such as may be found in management situations involving incremental changes of parameter values.

[0142] Example methods according to the present disclosure can improve reward performance of the trained policy by allowing more features to be used as an input to the policy given a fixed number of training samples. Also, by avoiding the risk of overfitting to any noise in the data, the trained policy provides more reliable behavior even in cases in which an insufficient number of informative samples (e.g., increase / decrease actions) are available. Additionally, owing to the sample efficiency of the methods proposed herein, the cost for data collection (or action exploration) and the corresponding training (or data collection) period can be significantly reduced. Considering computational complexity, training can be achieved in a much more time efficient / less computationally heavy way while performance is also improved.

[0143] When applied to RAN optimization products and deployed for training in operator networks, the improved sample efficiency of methods according to the present disclosure provides a quicker convergence to a certain level of network performance, affording operators shorter time period for data collection. In case of online learning, training period during which performance degradation is imposed by data collection and exploration can be reduced. Additionally, as methods according to the present disclosure can allow a greater number of input features, including all detailed configurations and KPIs in RAN, the resulting trained policy ML model can provide higher reward performance (e.g., RAN coverage, quality, capacity, energy consumption) by taking account of a greater amount of information in generating actions. It will be appreciated that additional, non-RAN use cases may also be envisaged, and may offer similar advantages.

[0144] The methods of the present disclosure may be implemented in hardware, or as software modules running on one or more processors. The methods may also be carried out according to the instructions of a computer program, and the present disclosure also provides a computer readable medium having stored thereon a program for carrying out any of the methods described herein. A computer program embodying the disclosure may be stored on a computer readable medium, or it could, for example, be in the form of a signal such as a downloadable data signal provided from an Internet website, or it could be in any other form.

[0145] It should be noted that the above-mentioned examples illustrate rather than limit the disclosure, and that those skilled in the art will be able to design many alternative embodiments without departing from the scope of the appended claims or numbered embodiments. The word "comprising” does not exclude the presence of elements or steps other than those listed in a claim or embodiment, "a” or "an” does not exclude a plurality, and a single processor or other unit may fulfil the functions of several units recited in the claims or numbered embodiments. Any reference signs in the claims or numbered embodiments shall not be construed so as to limit their scope.

Claims

CLAIMS1. A computer implemented method (100) for training a system to manage an environment in a communication network, wherein the system comprises an encoder Machine Learning, ML, model, and a policy ML model, and wherein the communication network environment is operable to perform a task, the method, performed by a training node, comprising: obtaining a training dataset comprising records of task performance of the environment during a period of management according to a reference policy (110), wherein each record of task performance comprises (110i): an observed context for the environment, an action selected for execution in the environment by the reference policy on the basis of the observed context, and a reward value indicating an observed impact of the selected action on task performance by the environment; using a Self-Supervised Learning, SSL, process and a subset of the obtained training dataset to train the encoder ML model to generate a latent representation of the observed context (120); using a Reinforcement Learning, RL, process and the obtained training dataset to train the policy ML model to generate an action for execution in the environment according to a latent representation of an observed context of the environment, the latent representation generated by the encoder ML model (130); and outputting the trained encoder ML model and policy ML model for use in managing the communication network environment (140).

2. A method as claimed in claim 2, wherein generating an action for execution in the environment according to a latent representation of an observed context of the environment comprises: inputting the latent representation to the policy ML model, wherein the policy ML model is operable to process the input latent representation according to current values of its trainable parameters, and to output a generated action for execution in the environment (330a).

3. A method as claimed in claim 1 or 2, wherein using an SSL process and a subset of the obtained training dataset to train the encoder ML model to generate a latent representation of the observed context comprises: using a predictive architecture for the SSL process (320a).

4. A method as claimed in any one of claims 1 to 3, wherein the subset of the obtained training dataset comprises a time series of observed contexts for the environment, each entry in the time series extracted from a record in the obtained training dataset (320b).

5. A method as claimed in any one of claims 1 to 4, wherein using an SSL process and a subset of the obtained training dataset to train the encoder ML model to generate a latent representation of the observed context comprises: using the encoder ML model to generate a latent representation of an observed context from the subset (321); using the generated latent representation for a prediction task relating to the observed context (322); and updating values of trainable parameters of the encoder ML model to minimize a loss function evaluating performance of the prediction task (323).

6. A method as claimed in claim 5, wherein the prediction task comprises predicting at least one of an observed context or a latent representation of an observed context (322a).

7. A method as claimed in claim 6, when dependent on claim 4, wherein predicting at least one of an observed context or a latent representation of an observed context comprises predicting at least one of the next observed context in the time series, or a latent representation of the next observed context in the time series (322b).

8. A method as claimed in any one of the preceding claims, wherein using an SSL process and a subset of the obtained training dataset to train the encoder ML model to generate a latent representation of the observed context comprises: using at least one of a Generative Predictive Architecture or a Joint Embedding Predictive Architecture for the SSL process (320a).

9. A method as claimed in any one of the preceding claims, wherein the environment comprises at least one entity, a parameter of which is operable to be managed; and wherein in the training dataset, the actions selected for execution in the environment by the reference policy on the basis of the observed context comprise actions controlling a value of the parameter, the actions comprising at least one of an incremental change or no change to the parameter value (31 Od).

10. A method as claimed in any one of claims 4 to 9, wherein the time series of observed contexts for the environment comprises pairs of observed contexts, each pair comprising an input observed context extracted from an input record of task performance, and a target observed context extracted from a record of task performance following execution of the selected action from the input record of task performance (320c).

11. A method as claimed in claim 10, wherein the time series of observed contexts for the environment comprises pairs of observed contexts for which the selected action from the input record of task performance is an action that imposes no change on the environment (320d).

12. A method as claimed in claim 10 or 11, when dependent on claim 9, wherein the time series of observed contexts for the environment comprises pairs of observed contexts for which the selected action from the input record of task performance is an action comprising no change to the parameter value (320e).

13. A method as claimed in any one of the preceding claims, wherein using an RL process and the obtained training dataset to train the policy ML model to generate an action for execution in the environment according to a latent representation of an observed context of the environment, the latent representation generated by the encoder ML model, comprises using the trained encoder ML model to generate latent representations of the environment for training the policy ML model (330b).

14. A method as claimed in claim 13, wherein using an RL process and the obtained training dataset to train the policy ML model to generate an action for execution in the environment according to a latentrepresentation of an observed context of the environment, the latent representation generated by the encoder ML model, further comprises at least one of (330c): freezing values of trainable parameters of the encoder ML model during training of the policy ML model; updating values of trainable parameters of the encoder ML model during training of the policy ML model.

15. A method as claimed in any one of the preceding claims, wherein using an RL process and the obtained training dataset to train the policy ML model to generate an action for execution in the environment according to a latent representation of an observed context of the environment, the latent representation generated by the encoder ML model, comprises: using the trained encoder ML model to generate a latent representation of an observed context from the training dataset (331); using the policy ML model to generate an action for execution in the environment according to the generated latent representation (332); and updating values of trainable parameters of the policy ML model to minimize a loss function evaluating the action generated by the policy ML model against at least one of (333): the action selected for execution in the environment by the reference policy on the basis of the observed context from the training dataset; the reward value indicating an observed impact of the selected action from the training dataset on task performance by the environment.

16. A method as claimed in any one of the preceding claims, further comprising: updating values of trainable parameters of at least one of the encoder ML model or the policy ML model during use of the system to manage the environment (350).

17. A method as claimed in any one of the preceding claims, wherein each of the encoder ML model and the policy ML model comprise an Artificial Neural Network (340a).

18. A method as claimed in any one of the preceding claims, wherein the communication network environment comprises at least a part of a Radio Access Network, a cell of a Radio Access Network, a cell sector of a Radio Access Network, a group of cells of a Radio Access Network, or a slice of the communication network; and wherein the task that the environment is operable to perform comprises provision of communication network services (310b).

19. A method as claimed in any one of the preceding claims, wherein the communication network environment comprises at least one Radio Access Node, and wherein management of the communication network comprises controlling a value of a parameter of the radio access node (310c).

20. A method as claimed in any one of the preceding claims, wherein outputting the trained encoder ML model and policy ML model for use in managing the communication network environment comprises: providing the trained encoder ML model and policy ML model to a management node, wherein the management node is operable to use the trained encoder ML model and policy ML model to manage the environment (340b).21 . A computer implemented method (200) for managing an environment in a communication network, wherein the communication network environment is operable to perform a task, the method, performed by a management node, comprising: obtaining a trained system from a training node (210), wherein the system comprises an encoder Machine Learning, ML, model, and a policy ML model, and wherein the system has been trained using a method according to any one of claims 1 to 19 (21 Oi); receiving an observed environment context from a communication network node (220); using the encoder ML model to generate a latent representation of the observed context (230); using the policy ML model to generate an action for execution in the environment according to the generated latent representation (240); and causing execution of the generated action in the environment (250).

22. A method as claimed in claim 21, wherein using the policy ML model to generate an action for execution in the environment according to the generated latent representation comprises: inputting the generated latent representation to the policy ML model, wherein the policy ML model is operable to process the input latent representation according to current values of its trainable parameters, and to output a generated action for execution in the environment (440a).

23. A method as claimed in claim 21 or 22, further comprising: obtaining a reward value indicating an observed impact of the generated action on task performance by the environment (460); and updating values of trainable parameters of at least one of the encoder ML model or the policy ML model according to the obtained reward value (470).

24. A method as claimed in any one of claims 21 to 23, wherein the environment comprises at least one entity, a parameter of which is operable to be managed (420a); and wherein the action generated for execution in the environment by the policy ML model comprises an action controlling a value of the parameter, the action comprising at least one of an incremental change or no change to the parameter value (440b).

25. A method as claimed in any one of claims 21 to 24, wherein each of the encoder ML model and the policy ML model comprise an Artificial Neural Network (410a).

26. A method as claimed in any one of claims 21 to 25, wherein the communication network environment comprises at least a part of a Radio Access Network, a cell of a Radio Access Network, a cell sector of a Radio Access Network, a group of cells of a Radio Access Network, or a slice of the communication network; and wherein the task that the environment is operable to perform comprises provision of communication network services (420c).

27. A method as claimed in any one of claims 21 to 26, wherein the communication network environment comprises at least one Radio Access Node, and wherein management of the communication network comprises controlling a value of a parameter of the radio access node (420b).

28. A computer program product comprising a computer readable medium, the computer readable medium having computer readable code embodied therein, the computer readable code being configured such that, on execution by a suitable computer or processor, the computer or processor is caused to perform a method as claimed in any one of claims 1 to 27.

29. A training node (500) for training a system to manage an environment in a communication network, wherein the system comprises an encoder Machine Learning, ML, model, and a policy ML model, and wherein the communication network environment is operable to perform a task, the training node comprising processing circuitry (502) configured to cause the training node to: obtain a training dataset comprising records of task performance by the environment during a period of management according to a reference policy, wherein each record of task performance comprises an observed context for the environment, an action selected for execution in the environment by the reference policy on the basis of the observed context, and a reward value indicating an observed impact of the selected action on task performance by the environment; use a Self-Supervised Learning, SSL, process and a subset of the obtained training dataset to train the encoder ML model to generate a latent representation of the observed context; use a Reinforcement Learning, RL, process and the obtained training dataset to train the policy ML model to generate an action for execution in the environment according to a latent representation of an observed context of the environment, the latent representation generated by the encoder ML model; and output the trained encoder ML model and policy ML model for use in managing the communication network environment.

30. A training node as claimed in claim 29, wherein the processing circuitry is further configured to cause the training node to perform the steps of any one of claims 2 to 20.31 . A management node (600) for managing an environment in a communication network, wherein the communication network environment is operable to perform a task, the management node comprising processing circuitry (602) configured to cause the management node to: obtain a trained system from a training node, wherein the system comprises an encoder Machine Learning, ML, model, and a policy ML model, and wherein the system has been trained using a method according to any one of claims 1 to 19; receive an observed environment context from a communication network node; use the encoder ML model to generate a latent representation of the observed context; use the policy ML model to generate an action for execution in the environment according to the generated latent representation; and cause execution of the generated action in the environment.

32. A management node as claimed in claim 31 , wherein the processing circuitry is further configured to cause the management node to perform the steps of any one of claims 22 to 27.

Images