Performing a task using reinforcement learning and symbolic regression

Abstract

There is provided a computer-implemented method (300) of performing a task, including training (S310) a reinforcement learning (RL) model to perform at least part of a task, wherein the RL model includes an artificial neural network (ANN)-based function approximator. After having trained the RL model, the trained RL model is used to train (S312) a symbolic regression (SR) model to represent the trained RL model, and the task is then performed (S314) using the trained SR model. In one embodiment, the trained SR model is deployed as part of a Network Data Analytics Function (NWDAF).

Description

PERFORMING A TASK USING REINFORCEMENT LEARNING AND SYMBOLIC REGRESSION Technical field

[0001] The present disclosure relates to the field of reinforcement learning (RL). In particular, the present disclosure relates to RL models including one or more artificial neural network (ANN)-based function approximators. An example use case relates to making predictions in telecommunication networks. Background

[0002] Machine learning (ML), wherein a computer is trained and learned to perform various tasks, such as to control or make predictions about a particular system (or environment), has found more and more use within many different disciplines. Due to their generality, reinforcement learning (RL) models form a subset of ML models particularly useful when there is no or little labeled data available for training of the models.

[0003] However, if e.g. the number of possible states between which the system may transition is large, manual engineering of the state space may be difficult and traditional RL models and algorithms are often not sufficient to handle such situations. This issue may however, at least in some situations, be resolved by combining reinforcement learning with deep learning to build so-called deep reinforcement learning (deep RL) models, wherein artificial neural networks (ANNs) are used to approximate one or more functions of the traditional RL model (such as e.g. the state-value, action-value and / or policy function) which would otherwise be intractable to define due to the high-dimensionality of the state space.

[0004] In for example the field of telecommunications, an ever-growing amount of data traffic and number of connected users in combination with demands for further improvements of e.g. latency, reliability, and similar, have made e.g. the capability of a network operator and / or service provider to gather and analyze data more important than ever. For example, 3GPP TS 23.503 defines a so-called Network Data Analytics Function (NWDAF) responsible for centralized data collection and data analytics in e.g. fifth-generation (5G) telecommunications networks, and 3GPP TR 23.791 provides multiple use cases such as computations and predictions of load-level and service experience, prediction of user equipment (UE) behavior, detection of abnormal UE behavior and anomalies, prediction of UE communication patterns, prediction of congestion information, and e.g. reporting and prediction of Quality of Service (QoS) change. For at least some (if not all) of these use cases, the use of ML- and RL- / deep RL-based solutions may be considered as important or even essential.

[0005] The present disclosure seeks to develop the use of RL- / deep RL-based solutions in general, in particular such solutions within the field of telecommunications, and to mitigate one or more shortcomings thereof. Summary

[0006] A potential downside with contemporary deep RL-based solutions is that the number of hidden layers in an ANN-based function approximator required to sufficiently represent the approximated function may be large, with a resulting increase in computational cost (in terms of required processing power and / or memory) and execution time. This may make such deep RL-based solutions unsuitable in some scenarios wherein execution time is critical and wherein there are many (e.g. several hundred thousand, or even millions) of users who may concurrently request e.g. predictions from a same model. In addition, such requirements of processing power and / or memory may also make the deep RL-based solutions impossible to deploy on low-hardware devices such as various Internet-of- Things (IoT) devices and similar (wherein the available computing power and / or memory are limited due to e.g. budget and / or size constraints).

[0007] Another potential downside with contemporary deep RL-based solutions is that they are basically “black boxes”, wherein the inner workings of e.g. the ANN- based function approximators are hard if not impossible to understand for a human observer. Deep RL-based solutions are thus not easily explainable, and reasonable questions such as why a particular solution is provided by the used models may be left unanswered.

[0008] To at least partially overcome some or all of the above-identified issues, the present disclosure provides a computer-implemented method of performing a task making use of a symbolic regression (SR) model to replace the trained RL model at the deployment stage, a computer-implemented method of training an SR model,an SR training entity, an SR entity, a NWDAF entity, a system, as well as various computer programs and computer program products, and a computer-readable storage medium as defined in and by the accompanying independent claims. Various embodiments of the methods, entities, system, computer programs and computer program products, and computer-readable storage medium, are defined in and by the accompanying dependent claims.

[0009] According to a first aspect, there is provided a (computer-implemented) method of performing a task. The method includes training a reinforcement learning (RL) model to perform at least part of a task, wherein the RL model includes an artificial neural network (ANN)-based function approximator. The method further includes, after having trained the RL model, using the trained RL model to train a symbolic regression (SR) model to represent the trained RL model. The method further includes performing the task using the trained SR model. Phrased differently, during deployment, the trained SR model may be used instead of the trained RL model.

[0010] As used herein, performing a task may e.g. include controlling and / or making predictions about a particular system (or environment). The system may e.g. be a physical system, such as a telecommunications system or any other system for which control and / or predictions are needed. The term “reinforcement learning” is assigned its usual meaning, wherein an agent interacts with (i.e., take actions in) an environment, wherein a taken action may result in the environment transitioning from a current state to a next state, wherein a reward is received, and wherein the agent learns what action(s) to take for a particular state of the environment in order to maximize some notion of cumulative reward. As used herein, an ”action” may not necessarily be something which influences the environment (such as a control action), but may instead include e.g. the making of an observation / prediction about some feature of the environment (such as e.g. a next state of the environment) based on e.g. a current state of the environment, or similar. Phrased differently, the agent may control the environment, or may take a more passive role and observe the environment as controlled by some other entity in order to learn how the environment behaves, and what a likely next state of the environment is based on its current state. As used herein, an ANN-based function approximator may e.g. be an ANN trained to approximate (parametrize) a state-value function (V), state-action-value function (Q), or policy (π), or similar, of the RL model, and training the RL model may include the use of e.g. deep Q-learning, actor-critic learning, or whatever suitable approach. Using the trained RL model to train the SR model may include e.g. first deploying the RL model and observing the RL model “in action” by applying actions and gathering state information, e.g. to gather data of a current state, an action taken and a resulting next state for a plurality of time instances, and to then use such data to train the SR model in order to generate one or more equations describing e.g. what action(s) the trained RL model would recommend taking based on e.g. a current state of the system / environment. Observing the trained RL model “in action” may e.g. include deploying the trained RL model to control / observe a real environment / system, or e.g. by “simulating” such an environment by providing information about a hypothetical environment to the RL model and observe how the RL model performs.

[0011] As used herein, the RL model may have a discrete or continuous action space. Preferably, the policy of the RL model is deterministic. The term “symbolic regression model” is also assigned its normal meaning, i.e. a process of identifying (using e.g. evolutionary optimization) one or more mathematical expressions (e.g., equations) that fit some observed phenomena without direct human intervention. In the present disclosure, this phenomena may be the output from the “black box” ANN- based function approximator, such that the trained SR model may replace the ANN- based function approximator and trained RL model at deployment.

[0012] The present disclosure improves upon contemporary technology in that it, during deployment, uses a trained SR model to replace the trained RL model. Instead of having to evaluate a potentially numerically costly ANN (including e.g. multiple hidden layers) for each incoming request, the proposed solution instead relies on evaluating one or more equations generated by the trained SR model, equations which are faster to evaluate than the corresponding ANN and which sufficiently approximates the ANN. Once deployed, the proposed solution is thus faster to evaluate, requires less computational power and / or memory, and is thus suitable to handle multiple concurrent user requests and also more suitable for deployment in low-hardware devices such as IoT devices. The training of the RL model may still be done during the training phase where computational power and / or memory is often not an issue. In addition, using one or more equations to represent the ANN and RLmodel may provide a more intuitive solution which may be easier to follow for a human, and thus lead to a more explainable solution from which e.g. the reason for why a particular solution is generated is easier to deduce. The proposed solution is also generic and may be applied for RL models also in other fields than telecommunications, and for both single-agent and multi-agent models.

[0013] In one or more embodiments of the method, the SR model may be trained to generate at least a first equation representative of how to perform the at least part of the task for a current state of the environment. The SR model may also be trained to generate also at least a second equation representative of how to perform the at least part of the task for a predicted later, such as a next, state of the environment. Phrased differently, the SR model may be trained to output a sequence of actions to take based on a current state of the environment. For example, the SR model may receive a current state of the environment at time t, and return one equation for which action to take next (at time t), and another equation for which action to take next-next (e.g. at a later time t + 1). Predicting / providing not only one but a sequence of actions using the SR model may be particularly effective in terms of computational power, and also help to make the result / solution easier to explain. Exactly how many equations (i.e., the length of the sequence) may depend on the particular use case and application.

[0014] In one or more embodiments of the method, the task may include making predictions related to, and / or controlling of, a telecommunications network. As telecommunications networks may be time-critical, the use of the SR model to represent the otherwise numerically expensive RL model as a set of one or more equations (which are faster to evaluate) may render the proposed solution particularly useful in such networks, wherein there are often many concurrent requests made for e.g. predictions using a same model. Controlling and / or predicting a telecommunications network may e.g. include any of the analytics use cases described in 3GPP TR 23.791, such as load-level computation and prediction (for network slice instance), service experience computation and prediction for an application / UE group, load analytics information and prediction for a specific Network Function (NF), network load performance computation and future load prediction, UE expected behavior prediction, UE abnormal behavior / anomaly detection, UE-mobility related information and prediction, UE communicationpattern prediction, congestion information (current and predicted) for a specific location, QoS sustainability including reporting and predicting of QoS changes, and e.g. frequency monitoring and predicting frequency loss.

[0015] In one or more embodiments of the method, the at least part of the task may include predicting a data throughput of a user equipment (UE). Such a task may e.g. include to, based on a known or expected location of the UE. Consequently, in one or more embodiments of the method, the first equation may be representative of the data throughput of the UE at a current location and / or point in time, and the second equation may be representative of the data throughput of the UE at a later, such as a next, predicted location and / or later, such as a next, point in time. In one or more embodiments of the method, the task may further include controlling one or more parameters of the telecommunications network to meet the predicted throughput of the UE, as output by e.g. the first and / or second equation. The data throughput may also be predicted based on e.g. a known / predicted signal-to-noise ratio (SNR), signal strength, or similar at the known or predicted location of the user. In other embodiments, other known or predicted data regarding the UE may (instead or in addition) be used for the throughput prediction. As an example, the first equation may provide an “action” in the form of a predicted throughput at a known location of the user, and the second equation may provide an “action” in the form of a predicted throughput at a predicted next location of the user, or similar. The next location of the user may e.g. be found using a suitable model for predicting the movement of the user, based on e.g. information about a user’s current location and velocity (speed and direction), information about e.g. a road along which the user is currently travelling, or similar.

[0016] In one or more embodiments of the method, the at least part of the task may include making predictions and / or detections of at least one of churn, network conflicts and network quality in / of the telecommunications network.

[0017] In one or more embodiments of the method, the task may include controlling one or more parameters of the telecommunications network to at least partially optimize the churn, network conflicts and network quality based on the prediction and / or detections (thereof).

[0018] In one or more embodiments of the method, the trained SR model may be deployed as part of a Network Data Analytics Function (NWDAF), such as e.g.described in 3GPP TS 23.503. In a telecommunications network, various NFs may then use the NWDAF to obtain statistics and predictions about various properties of the network, and the use of the envisaged SR model instead of the RL model may help to deploy such ML-based functionality in an NWDAF possible.

[0019] According to a second aspect, there is provided a computer-implemented method of training a symbolic regression (SR). The method includes training a reinforcement learning (RL) model to perform a task, wherein the RL model includes an artificial neural network (ANN)-based function approximator. The method further includes, after training the RL model, using the RL model to train a symbolic regression (SR) model to represent the trained RL model.

[0020] In one or more embodiments of the method, the SR model may be trained to generate at least a first equation representative of how to perform the task for a current state of an environment, and at least a second equation representative of how to perform the task for a predicted later, such as a next, state of the environment. As used herein, “performing a task for a current state of an environment” may include taking a certain first action in response to the environment being in a current state. Likewise, “performing a task for a predicted later state of the environment” may include taking a certain second action after the first action, wherein the second action is also determined based on the current state of the environment. Phrased differently, in response to the environment / system being in a particular current state, the SR model may output a sequence of actions to take next, all based on the current state of the environment.

[0021] According to a third aspect, there is provided a symbolic regression (SR) training entity. The SR training entity includes processing circuitry and a memory storing instructions, wherein the instructions are such that they, when executed by the processing circuitry, cause the SR training entity to perform the method of the second aspect or any embodiment thereof as disclosed herein.

[0022] According to a fourth aspect, there is provided a symbolic regression (SR) entity. The SR entity includes processing circuitry and a memory storing instructions, wherein the instructions are such that they, when executed by the processing circuitry, cause the SR entity to implement a symbolic regression (SR) model trained in accordance with the method of the second aspect or any embodiment thereof disclosed herein.

[0023] According to a fifth aspect, there is provided a Network Data Analytics Function (NWDAF) entity. The NWDAF entity includes the SR entity of the fourth aspect.

[0024] According to a sixth aspect, there is provided a system. The system includes the SR training entity of the third aspect. The system further includes the SR entity of the fourth aspect and / or the NWDAF entity of the fifth aspect.

[0025] According to a seventh aspect, there is provided a computer program for performing a task. The computer program includes computer code which, when run on processing circuitry, causes the processing circuitry to perform the method of the first aspect or any embodiment thereof disclosed herein.

[0026] According to an eight aspect, there is provided a computer program product including the computer program of the seventh aspect, and a computer- readable storage medium on which the computer program is stored.

[0027] According to a ninth aspect, there is provided a computer program for training a symbolic regression (SR) model. The computer program includes computer code which, when run on processing circuitry of an SR training entity (such as the entity of the third aspect), causes the SR entity to perform the method of the second aspect or any embodiment thereof disclosed herein.

[0028] According to a tenth aspect, there is provided a computer program product including the computer program of the ninth aspect, and a computer-readable storage medium on which the computer program is stored.

[0029] According to an eleventh aspect, there is provided a computer program for a symbolic regression (SR) entity (such as the entity of the fourth aspect). The computer program includes computer code which, when run on processing circuitry of the SR entity, causes the SR entity to implement an SR model trained in accordance with the method of the second aspect or any embodiment thereof disclosed herein.

[0030] According to a twelfth aspect, there is provided a computer program product including the computer program of the eleventh aspect, and a computer- readable medium on which the computer program is stored.

[0031] According to a thirteenth aspect, there is provided a computer-readable storage medium, on which data representative of an SR model trained in accordancewith the method of the second aspect or any embodiment thereof disclosed herein is stored.

[0032] As used herein, a computer-readable storage medium (such as that of e.g. the eight, tenth, twelfth or thirteenth aspect) may e.g. be non-transitory, and be provided as e.g. a hard disk drive (HDD), solid state drive (SDD), USB flash drive, SD card, CD / DVD, and / or as any other storage medium capable of non-transitory storage of data. In other embodiments, the computer-readable storage medium may be transitory and e.g. correspond to a signal (electrical, optical, mechanical, or similar) present on e.g. a communication link, wire, or similar means of signal transferring.

[0033] Other objects and advantages of the present disclosure will be apparent from the following detailed description, the drawings and the claims. Within the scope of the present disclosure, it is envisaged that all features and advantages described with reference to e.g. the method of the first aspect are relevant for, apply to, and may be used in combination with also the method of the second aspect, the entities of the third, fourth and fifth aspects, the system of the sixth aspect, the computer programs of the seventh, ninth and eleventh aspects, the computer program products of the eight, tenth and twelfth aspects, and the computer-readable storage medium of the thirteenth aspect, and vice versa. Brief description of the drawings

[0034] Exemplifying embodiments will be described below with reference to the accompanying drawings, in which: Figures 1A and 1B illustrate conventional systems for performing a task using reinforcement learning (RL); Figures 2A-2D illustrate various examples of using RL in combination with symbolic regression (SR) to perform a task according to embodiments of the present disclosure; Figure 3 illustrates a flow of a method of performing a task according to embodiments of the present disclosure;Figure 4 illustrates a flow of a method of training an SR model according to embodiments of the present disclosure; Figures 5A and 5B illustrate an SR training entity according to embodiments of the present disclosure; Figures 6A and 6B illustrate an SR entity according to embodiments of the present disclosure; Figure 7 illustrates a Network Data Analytics Function (NWDAF) entity according to embodiments of the present disclosure; Figure 8 illustrates a system according to embodiments of the present disclosure; Figure 9 illustrates computer programs, computer program products and computer readable means / media according to embodiments of the present disclosure; Figures 10 and 11 illustrate an NWDAF in a telecommunications network according to embodiments of the present disclosure, and Figures 12 and 13 illustrate sequence diagrams according to embodiments of the present disclosure.

[0035] In the drawings, like reference numerals will be used for like elements unless stated otherwise. Unless explicitly stated to the contrary, the drawings show only such elements that are necessary to illustrate the example embodiments, while other elements, in the interest of clarity, may be omitted or merely suggested. Detailed description

[0036] Figure 1A schematically illustrates an example of a conventional (computer) system 100 configured to perform a task using reinforcement learning (RL). In the system 100, an RL model 110 makes observations about a state of an environment 120 and decides, based on the observed state of its / the environment 120, which action to take as part of performing the task. The RL model 110 observes how the state of the environment 120 changes due to the taken action, and learns (usually by receiving a reward) to optimize which action to take for each observed state of the environment 120. The RL model 110 is also referred to as an agent.

[0037] More specifically, the RL model 110 usually interacts with its / the environment 120 in discrete time steps. Consequently, at each time (step) , the RL model 110 observes a current state of the environment 120, and receives a reward that is given in response to whatever previous action (at time1) that caused the environment 120 to move into the state . Based on the state and thereward , the RL model 110 decides upon an action (from a set of available actions). As a result of the RL model 110 taking the action ^[^], the environment 120 transitions from state to a next stateand the process continued by the RL model 110 observing the state and receiving a reward and so on.

[0038] The environment 120 and the interaction therewith is often modelled mathematically as a Markov decision process (MDP). To become better at performing the task, a goal of the RL model 110 may be to learn a policywhich maximizes an expected cumulative reward. Here, is a set of possible states in which the environment 120 may be in, and the policy thus corresponds to the probability of the next action being , given that the current state of the environment 120 is the state . The RL model 110 may then decide which action to take next using the learned policy , i.e. by sampling based on the probability distribution provided by the policy. In this case, the policy is said to be stochastic, as there is no one-to-one mapping between a particular state and a particular action. In other examples of how to implement conventional reinforcement, the policy may instead be deterministic, such that it directly maps a particular state to a particular actionPhrased differently, the policy is a (perhaps probabilistic) mapping from the state space to the action space .

[0039] Common methods used to find optimal policies include various forms of value function estimations and direct policy searching. Examples of value function estimation may e.g. include Monte Carlo methods, Temporal difference (TD) methods, function approximation methods, or similar. Examples of direct policy searching may various methods related to stochastic optimization, including both gradient-based (e.g. REINFORCE / likelihood ratio methods) and gradient-free methods (e.g. simulated annealing, cross-entropy search, evolutionary computation, and similar).

[0040] Figure 1B schematically illustrates another conventional example of a (computer) system 101, in which an RL model 111 is implemented using deep learning based on one or more artificial neural networks (ANNs) 112 trained to approximate one or more functions (such as the policy or some value function) of the RL model 111. Using the ANN 112 to learn and represent / parametrize e.g. the policy may be beneficial if e.g. the state space are high-dimensional and cannot be solved using any of the algorithms mentioned with reference to Figure 1A. Deep RL strategies / algorithms are often classified as either model-based or model-free. Model-based algorithms may e.g. be estimated forward models of the dynamics of the environment 120 (by use of e.g. supervised learning and model predictive control, MPC), while model-free algorithms allows to learn the policy without explicitly modelling the forward dynamics of the environment 120. For example, it is known to use e.g. dynamic programming based on temporal difference learning and / or Q-learning, wherein e.g. the ANN 112 is trained to provide a function estimating the future return(s) of taking action in state , or similar, also referred to as deep Q-learning.

[0041] As an example, the goal of an RL model may be to find a policy ^ such that it optimizes an expected return defined as (a V-value function)where is the expectation value operator and a discount rate factor (usually defined to be smaller than one). An optimal expected return (i.e. optimal V-value function) may be defined asThe optimal V-value function is thus the expected discounted reward when, in a given state ^[^] at time step , the RL model follows the policy when deciding what to do for all subsequent time steps

[0042] One may also define a quality-value functionand an optimal Q-value function may be defined asThe optimal Q-value function is thus the expected discounted return when, in a given state at time step and for a given action ^[^], the RL model follows the policy when deciding what to do for all subsequent time steps . In particular,the optimal policy may be obtained directly from the optimal Q-value function as

[0043] In value-based methods such as Q-learning, the optimal Q-value function is iteratively approximated as the state-action pair is updated, e.g.where 4 is a learning-rate factor and indicates the approximation of at time step . The Q-value function may be represented as a table, where, for each possible state- action pairthere is provided a reward value which is updated during training such that, once in steady-state, it matches the reward with discount (i.e. the value of However, as soon as thestate-space grows sufficiently large, the number of elements in such a table becomes too large to handle. For example, if assuming that the environment 120 is e.g. the pixels in a small, 128 by 128 pixel grayscale image, the number of possible states in which the image may be in is (if assuming 256 possible grayscale values for each pixel), i.e. the number of elements in the matrix / table representing the Q-value function would exceed the estimated number of atoms in the universe and consequently also exceed the storage / processing capability of any modern computer.

[0044] Using deep RL and the ANN 112, the Q-value function may insteadbe approximated in / by the RL model 111 aswhere is a parametrization of the Q-value function provided by the ANN 112 and defined by the various weights of the network. A loss function may e.g. be defined asand the parameters C may beupdated / learned by minimizing this loss using e.g. a (stochastic) gradient descentmethod. Various tricks, such as only updating the parameters of the target Q-network every N:th iteration and / or the use of experience replay may be used to avoid instabilities and slow / no convergence. In addition, it is known to e.g. use multiple networks to e.g. both select actions and evaluate actions, and similar.

[0045] In addition to value-based deep RL (e.g. deep Q-learning), one may also use the ANN 112 to parametrize the policy, e.g. by approximating , and train the network by adjusting the weights providing C in order to achieve more reward. It is also known to combine both value- and policy-based learning, e.g. as used in actor-critic models and similar including e.g. a value-based actor and policy-based critic.

[0046] Independent of whether the ANN 112 is used for value- and / or policy- based deep RL, the number of hidden layers (and nodes in general) of the ANN 112 may become large, and the execution time required to evaluate the ANN 112 for each time step may become too long to e.g. meet multiple incoming requests. As described earlier herein, this may be particularly relevant in telecommunications network, wherein several hundred thousand or even millions of requests has to be processed by the RL model 111 in order to e.g. provide analytics as part of an NWDAF or similar. As a result, and as the inventors have realized, the use of conventional deep RL methods in e.g. telecommunications networks may thus be limited, and there is a need to provide other methods which may handle e.g. such amounts of requests in real-time. It should of course be noted that even though telecommunications networks are used herein as a particular example of where conventional deep RL methods may suffer, the improvements that will be presented herein of course apply to any other field in which there is a need to improve upon conventional deep RL methods for similar reasons.

[0047] Various examples of improved systems and methods for RL as envisaged herein will now be described in more detail with reference to Figures 2A-2D, 3, 4, 5A, 5B, 6A, 6B, 7, 8, 9, 10, 11, 12 and 13.

[0048] In general, to improve upon conventional systems and methods using RL combined with ANN function approximators (i.e., deep RL) to perform a task, the present disclosure proposes to combine deep RL with symbolic regression (SR), and in particular to replace the deep RL model with a trained SR model at the deploymentstage. The general principle of the present disclosure is schematically illustrated in Figures 2A, 2B and 2C.

[0049] Figure 2A depicts a system 200 as envisaged herein in a training stage. As already described with reference to Figure 1A, an RL model 111 is first trained and learned how to interact with the environment 120 to perform a particular task. The task may e.g. be to control the environment 120, but may also (or instead) include to make predictions about a future behavior of the environment 120 (which may or may not include directly interacting with the environment 120). However, in contrast to conventional such solutions, the present disclosure envisages to provide a symbolic regression (SR) model 210. The SR model 210 may e.g. observe the states of the environment 120 and the actions taken by the (trained) RL model 111, and based thereon e.g. learn how to represent the trained RL model 111. This may include e.g. learning a mapping between state and action (e.g., the policy ) by observing the trained RL model 111, and to represent such a mapping in terms of one or more explicit equations / formulas, i.e. equations which may more easily be evaluated then the ANN 112 and which may thus reduce the execution time needed to handle an incoming request (e.g., a request to provide for example an action to take based on a current state of the environment 120). Phrased differently, Figure 2A depicts a training phase of the present disclosure, wherein the SR model 210 of the system 200 is trained to represent the trained RL model 111. Although not shown in Figure 2A, the training of the SR model 210 is not necessarily performed by observing the trained RL model 111 interacting with an environment. As an alternative, the trained RL model 111 may be used to generate the data for training the SR model 210 by simulating an environment (i.e., by providing what appears to be states of an environment to the trained RL model 111), and by observing what action(s) the trained RL model 111 outputs in response. Phrased differently, training of the SR model 210 may be performed both with the trained RL model 111 deployed in a “live situation”, or with the trained RL model 111 used in an environment emulating such a “live situation”.

[0050] Figure 2B schematically illustrates a system 201 as envisaged herein in a deployment stage. Here, the trained RL model 111 is not required, and instead replaced with the trained SR model 210. Thus, the trained SR model 210 observes a state (e.g. ^[^], if assuming to proceed in discrete time steps) of the environment120, and outputs (using e.g. one or more equations directly mapping a particular state to a particular action ) an action , i.e. the trained SR model 210 is used instead of the trained RL model 111 to perform the task. If the task includes controlling the environment 120, the action may of course be taken in the environment 120, and the SR model 210 may observe a next state (e.g., ) of the environment 120 and take a next action, etc., all according to e.g. the one or more equations generated by the trained SR model 210 in order to represent the trained RL model 111. Phrased differently, the SR model 210 has been trained to represent the trained RL model 111 and in particular the ANN 112 thereof, such that the trained RL model 111 and the ANN 112 does not need to be evaluated to control / make predictions about the environment 120.

[0051] As used and envisaged herein, the symbolic regression is a type of (regression) analysis in which one or more mathematical expressions which best fit a given dataset are found by searching the space of possible such mathematical expressions. For example, the space of possible mathematical expressions may include any suitable combination of mathematical operators, analytic functions, constants, state variables, and similar, and finding the most suitable such mathematical expression(s) may e.g. be performed using genetic programming, Bayesian methods, neural networks, or other methods therefor known in the art. There may of course be restrictions added in order to limit the space of possible mathematical expressions, i.e. by specifying more in detail which mathematical operators, analytic functions, constants, state variables, etc., that are considered “possible”. Once such mathematical expressions have been found, the expressions may be used to represent the trained RL model 111.

[0052] As envisaged herein, the training of the SR model 210 may use e.g. experience replay, as the trained RL model 111 may be used to generate as much data for training of the SR model 210 as needed (e.g., if using the trained RL model 111 in an “emulated” environment as described earlier herein). In particular, even if the SR model 210 is deployed in a scenario wherein the computational resources are limited, the training of the SR model 210 is performed before the deployment and both the training of the RL model 111 and the usage of the trained RL model 111 is thus not necessarily bound by such limited computational resources. To the contrary, it is envisaged that during the training and using of the RL model 111, computationalresources may well be abundant and the complexity of the RL model 111 and ANN 112 in particular is therefore not necessarily an issue. If the RL model 111 is policy-based, the SR model 210 may learn to represent the ANN 112. If the RL model 111 is value- based, the SR model 210 may (instead) be trained on the data output by the trained RL model 111. In the case of a value-based RL model, the network may e.g. output a Q-value of the state for each possible action and the RL model 111 may chose the action corresponding to the highest Q-value. The SR model 210 may be trained on the chosen best action and states.

[0053] With reference also to Figures 2C and 2D, in particular embodiments, an SR model 211 as envisaged herein may be trained to output multiple equations which provide not only what action to take next (if using the trained RL model 111), but also one or more additional, more future actions which the trained RL model 111 would likely suggest if the trained RL model 111 and ANN 112 were still used to control / make predictions of the environment 120. During deployment, as shown in Figure 2C, the trained SR module 211 may then be used to output a sequence of equations describing a sequence of (current and / or future) actions.

[0054] For example, the SR model 211 may be trained to, as a function of a current state (i.e., at time step ), generate a first function I3describing what action ^[^] to take now, i.e.

[0055] The SR model 211 may further be trained to, also as a function of the current state ^[^], generate also a second function I;describing what actionto take after the action i.e.

[0056] In some embodiments, the one or more equations describing an action or later (such as I;) may be obtained by predicting what the next state (e.g.or later) of the environment 120 will be, and to use the data generated by the trained RL model 111 in order to find the mathematical expression I;which describes what the trained RL model 111 would have done (e.g. which action it would have output / recommended) for such a next state of the environment 120. The prediction about the next state may e.g. be based on the current state, and similar. For example, it is envisaged to provide a separate model for performing such prediction, e.g. asimplemented by a state predictor 220 which takes as input a current state ^[^] and output a predicted next state or similar, e.g.where K3is some function used to represent the prediction (i.e. a trained neural network or other prediction method suitable for the particular task that is to be solved) of the next state based on the current state. The state predictor 220 may, in some embodiments, rely on replay experience data to make the prediction of the next (or future) state.

[0057] As an example of the above, and as illustrated in Figure 2D, in a system 202 as envisaged herein in a training stage, the trained RL model 111 (and in particular the one or more ANN-based function approximator 112 thereof) may take as input a current state and output an actionwhere is a function representing the trained RL model 111, and where C is some parametrization learned by the trained RL model / ANN(s) 112. What the function is may be at least partially deduced by observing how the trained RL model 111 behaves (e.g. by gathering data by feeding the trained RL model 111 different states and observing what the proposed actions are in response thereto), and the SR model 211 (e.g. a first SR (sub)module 211a of the SR model 211) may be trained to generate an equation representing ^[^] as a function of Thecurrent state may also be provided to the state predictor 220, and the predicted next state may be provided to the trained RL module 111 in order to obtainits proposed actioni.e. information about. Such information (i.e. at least and may then be provided to train (a second SR (sub)module 211b) of the SR model 210 to generate an equation for as a function of , i.e. . Similarly, thesystem 202 may be further expanded to provide equations for even more future actions may also be generated by the trained SR model 210,where K;is some functionrepresenting a prediction of the next-next state based on the current state (provided by a therefor suitable state predictor), and so on. As used herein, when saying that the trained SR model 211 may provide multiple equations, it may of course (as illustrated in Figure 2D) also be assumed that the trained SR model 211 is a collection of multiple trained SR models (i.e. one for each equation, such as SR module 211a for expression , SR module 211b for expression , etc.).

[0058] Having access to such a sequence of actions may be particularly useful in terms of low computational demands, and may also help to e.g. explain why the actions are to be taken in terms of the current state of the environment 120. Thus, instead of relying on a complex “black box” ANN 112 in order to suggest what action to take next, the solution of the present disclosure may illustrate what action(s) to take next in a more intuitive way, in the form of one or more mathematical expressions.

[0059] As used herein, and as will be exemplified in more detail later herein, the term “action” is not necessarily a true action (i.e., an action taken in an environment) but may instead be a prediction about one or more future value(s) of one or more properties of the environment 120. The output from the trained SR model 210 (or 211) may also be used to indirectly control the environment 120, i.e. by providing the output from the trained SR model 210 (or 211) to some other entity responsible for controlling the environment 120, as will also be further exemplified in more detail later herein.

[0060] Flowcharts of various methods as envisaged herein will now be described with reference also to Figures 3 and 4.

[0061] Figure 3 schematically illustrates a flowchart of an exemplary computer- implemented method 300 of performing a task as envisaged herein. In an operation S310, the method 300 includes training the RL model 111 to perform at least part of task, wherein the RL model includes an ANN-based function approximator 112. In an operation S312, the method 300 includes, after having trained the RL model 111, using the trained RL model 111 to train the SR model 210 (or 211) to represent the trained RL model 111. In an operation S314, the method 300 includes performing the task using the trained SR model 210 (or 211), e.g. instead of using the trained RL model 111.

[0062] Figure 4 schematically illustrates a flowchart of an exemplary computer- implemented method 400 of training an SR model as envisaged herein. In an operation S410, the method 400 includes training the RL model 111 to perform a task, wherein the RL model 111 includes the ANN-based function approximator 112. The method 400 further includes, in an operation S412, using the trained RL model 111 to train the SR model 210 (or 211) to represent the trained RL model 111.

[0063] Various entities and systems as envisaged herein will now be described in more detail with reference also to Figures 5A, 5B, 6A, 6C, 7 and 8.

[0064] Figure 5A schematically illustrates, in terms of a number of functional units, the components of an embodiment of an SR training entity 500 according to the present disclosure. The SR training entity 500 is configured for training an SR model (such as the SR model 210 or 211) to perform a task in accordance with e.g. the method 400 described herein with reference to Figure 4, and includes processing circuitry 510. The processing circuitry 510 is provided using any combination of one or more of a suitable central processing unit (CPU), multiprocessor, microcontroller, digital signal processor (DSP), etc., capable of executing software instructions stored in a computer program product 910a (see Figure 9 and the description thereof), e.g. in form of a storage medium 530 that may also form part of the SR training entity 500. The processing circuit 510 may further be provided as at least one application specific integrated circuit (ASIC), or field-programmable gate array (FPGA).

[0065] Particularly, the processing circuitry 510 is configured to cause the SR training entity 500 to perform a set of operations, or steps, as disclosed above e.g. when describing the method 400 illustrated in Figure 4. For example, the storage medium 530 may store a set of operations, and the processing circuitry 510 may be configured to retrieve the set of operations from the storage medium 530 to cause the SR training entity 500 to perform the set of operations. The set of operations may be provided as a set of executable instructions. Thus, the processing circuitry 510 is thereby arranged to execute methods associated with training of an SR model as disclosed herein e.g. with reference to Figure 4.

[0066] The storage medium 530 may also include persistent storage, which, for example, can be any single or combination of magnetic memory, optical memory, solid state memory or even remotely mounted memory.

[0067] The SR training entity 500 may further include a communications interface 520 for communications with other entities, functions, nodes, and devices, such as e.g. those of a telecommunications network or any other network associated with the task to be solved. For example, the communications interface 520 may allow the SR training entity 600 to communicate with e.g. an SR entity, with one or more Network Functions (NFs) of a telecommunications network, etc. As such, the communication interface 520 may include one or more transmitters and receivers,including analogue and / or digital components. For example, the communications interface 520 may be used to obtain data about the state of an environment in which the task is to be solved, or similar. The communications interface 520 may for example receive data from one or more sensors configured to detect / measure the state of the environment. The communications interface 520 may also be capable of communicating one or more actions that are to be taken in the environment, e.g. to various actuators or controllers which may influence the environment.

[0068] The processing circuitry 510 controls the general operation of the SR training entity 500 e.g. by sending data and control signals to the communications interface 520 and the storage medium 530, by receiving data and reports from the communications interface 520, and by retrieving data and instructions from the storage medium 530. Other components, as well as their related functionality, of the SR training entity 500 are omitted in order not to obscure the concepts presented herein.

[0069] Figure 5B schematically illustrates, in terms of a number of functional modules 510a, 510b and 510c, the components of an SR training entity 500 according to one embodiment of the present disclosure. The SR training entity 500 includes at least an RL training module 510a configured to perform operation S410 of the method 400 described with reference to Figure 4. The SR training entity 500 also includes an SR training module 510b configured to perform operation S412. The SR training entity 500 may also include one or more optional functional modules (illustrated by the dashed box 510c), such as for example an environment state module configured to obtain various parameters indicative of the state of the environment, an / or e.g. a task solving module configured to solve the task using the trained SR model generated by the SR training module 510b (in which case the SR training entity is also capable to perform the operations S310, S312 and S314 of the method 300 described with reference to Figure 3). In some embodiments, the SR training entity 500 may be operable between a training stage and a deployment stage. In the training stage, the RL training module 510a and SR training module 510b may work together to generate a trained SR model configured to solve the task. In the deployment stage, the SR training entity 500 may be configured to switch to using the trained SR model to solve the task, including e.g. deactivating the RL training module 510a and the SR training module 510b. The SR training entity 500 may also beoperable to a retraining stage, in which one or both of the RL training module 510a and SR training module 510b are once again activated in order to e.g. update the trained RL model and / or the trained SR model, e.g. in response to the environment changing in a way which makes the trained SR model less successful in solving the task than before, and similar.

[0070] In general terms, each functional module 510a-c may be implemented in hardware or in software. Preferably, one or more or all functional modules 510a-c may be implemented by the processing circuitry 510, possibly in cooperation with the communications interface 520 and / or the storage medium 530. The processing circuitry 510 may thus be arranged to from the storage medium 530 fetch instructions as provided by a functional module 510a-c, and to execute these instructions and thereby perform any operations of the method 400 performed by / in the SR training entity 500 as disclosed herein. The SR training entity 500 may also be referred to as an SR model training entity, or similar.

[0071] Figure 6A schematically illustrates, in terms of a number of functional units, the components of an embodiment of an SR entity 600 according to the present disclosure. The SR entity 600 is configured to implement an SR model (such as the SR model 210) to perform a task, and includes processing circuitry 610. The processing circuitry 610 is provided using any combination of one or more of a suitable central processing unit (CPU), multiprocessor, microcontroller, digital signal processor (DSP), etc., capable of executing software instructions stored in a computer program product 910b (see Figure 9 and the description thereof), e.g. in form of a storage medium 630 that may also form part of the SR entity 600. The processing circuit 610 may further be provided as at least one application specific integrated circuit (ASIC), or field-programmable gate array (FPGA).

[0072] Particularly, the processing circuitry 610 is configured to cause the SR entity 600 to perform a set of operations, or steps, needed to implement a trained SR model as envisaged herein, e.g. as resulting from (or trained according to) the method 300 and / or the method 400 described with reference to Figure 3 and Figure 4, respectively. For example, the storage medium 630 may store a set of operations, and the processing circuitry 610 may be configured to retrieve the set of operations from the storage medium 630 to cause the SR entity 600 to perform the set of operations. The set of operations may be provided as a set of executable instructions. Thus, theprocessing circuitry 610 is thereby arranged to execute methods associated with implementing and using a trained SR model as disclosed herein, e.g. with reference to any one of Figures 2A-2D.

[0073] The storage medium 630 may also include persistent storage, which, for example, can be any single or combination of magnetic memory, optical memory, solid state memory or even remotely mounted memory.

[0074] The SR entity 600 may further include a communications interface 620 for communications with other entities, functions, nodes, and devices, such as e.g. those of a telecommunications network or any other network associated with the task to be solved. For example, the communications interface 620 may allow the SR entity 600 to communicate with e.g. an SR training entity (such as the SR training entity 500), with one or more Network Functions (NFs) of a telecommunications network, etc. As such, the communication interface 620 may include one or more transmitters and receivers, including analogue and / or digital components. For example, the communications interface 620 may be used to obtain data including details about the trained SR model which is to be implemented, data about the state of an environment in which the task is to be solved and from which e.g. the state of the environment may be estimated / obtained and provided to the trained SR model, or similar. The communications interface 620 may for example receive data from one or more sensors configured to detect / measure the state of the environment. The communications interface 620 may also be capable of communicating one or more actions that are to be taken in the environment, e.g. to various actuators or controllers which may influence the environment. The communications interface 620 may for example receive an initial trained SR model, and may receive one or more subsequent updates of the trained SR model from e.g. the SR training entity 500.

[0075] The processing circuitry 610 controls the general operation of the SR entity 600 e.g. by sending data and control signals to the communications interface 620 and the storage medium 630, by receiving data and reports from the communications interface 620, and by retrieving data and instructions from the storage medium 630. Other components, as well as their related functionality, of the SR entity 600 are omitted in order not to obscure the concepts presented herein.

[0076] Figure 6B schematically illustrates, in terms of a functional module 610a, the components of an SR entity 600 according to one embodiment of the presentdisclosure. The SR training entity 600 includes at least an SR (model) implementation module 610a configured to implement an SR model trained in accordance with e.g. method 400. The SR entity 600 may also include one or more optional functional modules (illustrated by the dashed box 610b), such as for example an environment state module configured to obtain various parameters indicative of the state of the environment, and / or e.g. an action module configured to communicate one or more actions which are to be taken in the environment based on the output from the implemented trained SR model.

[0077] In general terms, each functional module 610a and 610b may be implemented in hardware or in software. Preferably, one or more of the functional modules 610a and 610b may be implemented by the processing circuitry 610, possibly in cooperation with the communications interface 620 and / or the storage medium 630. The processing circuitry 610 may thus be arranged to from the storage medium 630 fetch instructions as provided by a functional module 610a and / or 610b, and to execute these instructions and thereby implement the SR model trained in accordance with e.g. method 400 as disclosed herein.

[0078] Figure 7 schematically illustrates, in terms of a number of functional units, the components of an embodiment of an NWDAF entity 700 according to the present disclosure. The NWDAF entity 700 includes one or both of the SR training entity 500 described with reference to Figures 5A and 5B and the SR entity 600 described with reference to Figures 6A and 6B. The NWDAF entity 700 may e.g. include only the SR entity 600 and thus be configured for deployment stage only. The NWDAF entity 700 may e.g. include both the SR training entity 500 and the SR entity 600, and thus be operable between a training stage and a deployment stage. The NWDAF entity 700 may of course also include various other functional units required to implement the functionality of an NWDAF as defined in e.g.3GPP TS 23.503. Such optional other functional units are however not illustrated in Figure 7. Although also not shown, the NWDAF entity 700 may of course also include additional processing circuitry and / or memory / storage as required to implement any required functionality not provided by the SR training entity 500 and / or SR entity 600.

[0079] The NWDAF entity 700 may include a communications interface 710 with which the NWDAF entity 700 may communicate with one or more other nodes of the telecommunications network, such as e.g. one or more other NF (entities) or similar.The communications interface 710 may for example be configured to receive information about an environment (such as its state) which is to be controlled and / or predicted (as part of the task to be solved), and may communicate such information 712 to one or both of the SR training entity 500 and SR entity 600. In addition, the communications interface 710 may be configured to distribute data / information 502 from the SR training entity 500 (such as information about a trained SR model and / or one or more actions proposed, and / or one or more predictions made, by such a trained SR model), and / or data / information 602 from the SR entity 600 (such as one or more actions proposed, and / or one or more predictions made, by the trained SR model of the SR entity 600), to e.g. one or more other NF (entities) of the network, and / or to e.g. one or more controllers or similar with which the environment (network) may be controlled based on actions and / or predictions from the trained SR model.

[0080] The communications interface 710 may thus be in communication with the communication interfaces 520 and / or 620 of the SR training entity 500 and SR entity, respectively. In other embodiments of the NWDAF 700, the communications interface 710 may not be included and the NWDAF 700 may instead be configured such that the communication interfaces 520 and / or 620 are allowed to communicate directly with other nodes / NFs of the network, or similar. Various embodiments of the NWDAF entity 700 will also be described later herein with reference to Figures 10, 11, 12 and 13.

[0081] Figure 8 schematically illustrates an embodiment of a system 800 of the present disclosure. The system 800 includes the SR training entity 500, and at least one of the SR entity 600 and NWDAF entity 700. The SR training entity 500 is configured to train the SR model and provide the trained model to the SR entity 600 and / or NWDAF entity 700.

[0082] In all of the entities and systems described herein, the RL model may e.g. be the RL model 111, the trained SR model may e.g. be the trained SR model 210, and similar.

[0083] Figure 9 schematically illustrates a computer program product 910a, 910b, 910c including computer readable means 930. On the computer readable means 930, a computer program 920a can be stored, which computer program 920a can cause the processing circuitry 510 and thereto operatively coupled entities and devices, suchas the communication interface 520 and the storage medium 530, of the SR training entity 500 to execute method 400 according to embodiments described herein with reference to Figure 4. The computer program 920a and / or computer program product 910a may thus provide means for performing any operations of the method 400 performed by the SR training entity 500 as disclosed herein.

[0084] On the computer readable means 930, a computer program 920b can also be stored, either in addition to or instead of the computer program 920a, which computer program 920b can cause the processing circuitry 610 and thereto operatively coupled entities and devices, such as the communication interface 620 and the storage medium 630, of the SR entity 600 to implement the SR model trained in accordance with e.g. method 400 according to embodiments described herein with reference to Figure 4. The computer program 920b and / or computer program product 910b may thus provide means for performing any operations required to implement the trained SR model.

[0085] On the computer readable means 930, a computer program 920c can also be stored, either in addition to or instead of one or both of the computer programs 920a and 920b, which computer program 920c can cause e.g. (processing circuitry and thereto operatively coupled entities and devices, such as the communication interface and the storage medium), of the SR training entity 500 and / or SR entity 600 and / or NWDAF 700 and / or system 800 to perform the method 300 of solving the task as described with reference to e.g. Figure 3. The computer program 920c and / or computer program product 910c may thus provide means for performing any operations required to perform the method 300. Of course, the computer program 920c may also be suitable to run on any processing circuitry suitable to perform method 300 in order to solve the task.

[0086] Figure 9 also serves to illustrate an embodiment of the computer readable means 930 in which it stores data representative of the trained SR model (i.e. an SR model trained in accordance with method 400). It is envisaged that the computer readable means 930 may include only such data, or such data in combination with one or more of the computer programs 920a, 920b and 920c described above.

[0087] In the example of Figure 9, the computer program product 910a, 910b, 910c and computer readable means 930 are illustrated as an optical disc, such as a CD (compact disc) or a DVD (digital versatile disc) or a Blu-Ray disc. The computerprogram product 910a, 910b, 910c and computer readable means 930 could also be embodied as a memory, such as a random-access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), or an electrically erasable programmable read-only memory (EEPROM) and more particularly as a non-volatile storage medium of a device in an external memory such as a USB (Universal Serial Bus) memory or a Flash memory, such as a compact Flash memory. Thus, while the computer program 920a, 920b, 920c and data 920d are here schematically shown as a track on the depicted optical disk, the computer program 920a, 920b, 920c and data 920d may be stored in any way which is suitable for the computer program product 910a, 910b, 910c and computer readable means 930.

[0088] With reference to Figures 10, 11, 12 and 13, various envisaged implementations and embodiments of the above concepts in a telecommunications network environment will now be described in more detail.

[0089] Figure 10 schematically illustrates the NWDAF (or NWDAF entity) 700 wherein the envisaged (trained) SR model 210 is interfaced with the NWDAF 700 in order to provide predictions about one or more properties / parameters of a telecommunications network 1000. Data 1012 is collected from a (5G) core 1010 of the network 1000 using an NF 1020. The data 1012 is pre-processed (as part of e.g. the NF 1020 or some other, non-shown entity) and provided as pre-processed data 1022 to the NWDAF 700. In the NWDAF 700, during the training stage, there is provided the SR training entity 500 which includes the RL model 111 (under training) as well as the SR model 210, such that the RL model 111 is first trained and then used to trained the SR model 210 as described earlier herein with reference to e.g. Figures 2A-2D. The RL model 111 is a generalized model that may be trained on any applications relevant for the telecommunications network 1000 and the NWDAF 700. The RL model 111 may for example include one or more of a churn prediction module 111a, a conflict detection module 111b, a network quality prediction module 111c, and a throughput prediction module 111d. Once the RL model 111 is trained for such prediction tasks, the SR model 210 is trained to represent the trained RL model 111, using data provided by running the trained RL model 111 either live or in an emulated environment. Once the SR model 210 is trained, the NWDAF 700 is configured to switch from the training stage to the deployment stage, wherein(preferably only) the trained SR model 210 is used to make the predictions. The predictions 702 are then output from the NWDAF 700 and sent as data 1032 to e.g. an end-user, using e.g. a same / other NF 1020 / 1030. As illustrated in Figure 10, the envisaged trained SR model 210 may thus be integrated in the NWDAF for e.g. various business support systems (BSSs) or similar. In addition, it is also envisaged that in some embodiments, the predictions 702 from the NWDAF 700 about the environment (telecommunications network 1000) may instead, or in addition, be used to control the network 1000 to optimize e.g. churn, conflict, network quality, throughput, and similar. Such optimization may e.g. be performed by the NF 1020 / 1030. Thus, as discussed earlier herein, although the output from the NWDAF 700 is one or more predictions, such predictions may be used to generate one or more actions which influences the environment (network 1000), and the NWDAF 700 plus the NF responsible for actually taking actions in the environment may thus be considered as an agent in the sense described with reference to e.g. Figures 2A-2D.

[0090] Figure 11 schematically illustrates a particular envisaged use-case in which the RL and SR models are used to predict data throughput for a particular user equipment (UE) 1140. In this particular example of a telecommunications network 1110, data 1112 indicative of various properties of the radio network 1110 is received by an Access and Mobility management Function (AMF) 1120 and communicated (e.g. as part of a signal 1122) to a pre-processing module 1124 in which e.g. outliers are removed, missing values are replaced / handled, and similar, such that the data is more suitable for use in the training and / or deployment stage. The pre-processed data 1126 is provided to the NWDAF (or NWDAF entity) 700, in which the RL model 111d and resulting SR model 210 (or 211) is for making data throughput predictions based on the data 1126. As already described, the NWDAF 700 is operable between a training stage and a deployment stage. In e.g. the deployment stage, the data 1126 is used to represent the state of the environment (network 1100) and used as input to the trained SR model 210, such that the trained SR model 210 may provide predictions / estimates of the data throughput of the UE 1140. For example, the NWDAF 700 and the trained SR model 210 may predict a next data throughput (requirement) for the UE 1140, depending on e.g. a location, signal-to-noise ratio and / or signal strength of the UE 1140 as estimated by the radio network 1110 and / or AMF 1120 based on communication 1142 between the UE 1140 and radio network 1110. Such predictions 702 are provided to a Session Management Function 1130which is configured to modify one or more parameters of the telecommunications network 1110 (such as e.g. one or more parameters of the UE 1140, as communicated from the SMF 1130 to the UE 1140 via a signal / message 1132), in order to maintain / meet the predicted data throughput (requirement) of the UE 1140 obtained from the NWDAF 700 and trained SR model 210. By using predictions from the NWDAF 700, the SMF 1130 is thus able to optimize the behavior of the network 1100, which in turn increases e.g. Quality-of-experience (QoE) for the owner / user of the UE 1140, as the UE 1140 will experience that the network 1100 is capable to provide the required data throughput / bandwidth. The envisaged solution thus enables e.g. network operators and / or service providers to improve their network operations and e.g. prevent customer abandonment from services by increasing customer satisfaction.

[0091] Figure 12 schematically illustrates a communication flow / sequence diagram 1200 between the various entities of the telecommunications network 1100 shown in Figure 11.

[0092] In general, the AMF 1120 receives a current Quality-of-Service (QoS) value and / or other features 1112 from the UE 1140 (via the radio network 1110). The AMF 1120 provides the relevant data 1122 to the pre-processing module 1124, which performs a corresponding pre-processing operation 1125. The pre-processed data 1126 is then sent to the NWDAF (or NWDAF entity) 700, wherein (during the deployment stage) the trained SR model 210 is deployed. The trained SR model 210 consumes the data 1126 and performs a throughput prediction operation, and the output 602 is provided to the NWDAF 700 which sends the predicted throughput 702 to the SMF 1130. The SMF 1130 then updates / changes, in an operation 1134, the relevant parameters (like e.g. priority and / or maximum bit rate, MBR) of the UE 1140 and / or radio network 1110 in order to achieve the predicted throughput, thus enhancing the experience of the owner / user of the UE 1140.

[0093] Another envisaged use-case includes prediction / detection / explaining of product churn. Product churn (or just “churn”) may be defined as a metric that measures how many customers that decides to leave a certain product / service over a certain time period. Such a metric may also be referred to as e.g. customer churn, (customer) attrition, and similar. Churn thus provides a measure of how many users that stopped using e.g. a specific product over a specific time period, and may includee.g. users who have either downgraded or cancelled a subscription plan, or similar. Other reasons may include product-market fit failure, high prices, poor user experience, and / or bad customer support. Churn is considered to occur as a result of users considering themselves having stopped receiving value-for-money for the product, or e.g. because they do not consider the product to be necessary any longer. One way of detecting churn is to divide a number of lost customers during the specific time period by a number of customers present at the start of the specific time period, and then e.g. multiply the result by 100.

[0094] As envisaged herein, using the trained SR model may help to understand the reasons underlying churn, and in particular as the one or more equations generated by the trained SR model may help explaining the underlying reasons easier than if e.g. having access only to a neural network used to approximate one or more functions of an RL model. As an example, various features could be used to analyze churn, such as characteristic, specification category, service specification, resource specification, etc. For exemplary purpose only, it may be assumed that there three features F1, F2 and F3 attributed to product churn. Given a list of population size (x) and population (R) churned out of a product, an action output by the SR model may be selected from e.g. a set of three possible actions wherein an actionin this example is a reason for churn of the population (R). Each such action / reason may thus be provided as a function of the features and . Also, each output is represented with sub-script t which represents the action taken at time instant ‘t’. For example, it may be assumed that action / reason at time instant ‘t’ can be written aswhere the coefficients U33, U;3and U>3are weighting coefficients for the respective feature-terms. Expressions / equations for the actions / reasons at future time steps ‘t+1’ and ‘t+2’ are represented as andIf of interest, actions may thus be predicted for two future time steps. Based on such equations output from the trained SR model, the reasons for churn may be made more explainable than if using only a trained RL / ANN-based model.

[0095] Figure 13 schematically illustrates a communication flow / sequence diagram 1300 of an exemplary churn prediction use-case as described above. In a product data collection operation 1312, a data collector 1310 collects product data and sends the data 1134 to an RL trainer 1320. The RL trainer 1320 may e.g. be the part of the SR training entity 500 responsible for training the RL model 111. In an RL training operation 1322, the RL trainer 1320 trains the RL model and sends the trained RL model 1324 to an SR model module 1330. The SR model module 1330 may e.g. be the part of the SR training entity 500 responsible for training the SR model 210 using the trained RL model 111 and / or the SR entity 600. In an SR training operation 1332, the SR model module 1330 trains the SR model and sends the trained / updated SR model 1334 to an NWDAF 1340. The NWDAF 1340 may e.g. be the NWDAF (entity) 700. The NWDAF 1340 receives a query 1352 from a user 1350 for a product and sends data 1352 associated therewith to the NWDAF 1340. The NWDAF 1340 sends data 1342 associated therewith to the SR model module 1330, which sends the predictions and SR model 1336 to the NWDAF 1340. The NWDAF 1340 then sends the predictions and model parameters 1336 to the user 1350. In an operation 1354, the user 1350 may visualize the predictions along with affecting features and their weightage, to better understand the underlying reasons for product churn.

[0096] As mentioned already herein, a technical advantage of the proposed solution is that using a trained SR model to represent a trained RL model allows to process more incoming requests (for e.g. one or more predictions) using the same processing power. Phrased differently, the proposed solution enables to make the same (or similar) predictions in e.g. low-level hardware such as various IoT devices, which would otherwise not be possible in some scenarios (i.e. with hundreds of thousands or millions of requests per time frame) if having to evaluate the full RL model and ANN-based function approximator for each incoming requests.

[0097] For example, predictive QoE may be important for many (5G) use cases, including e.g. for autonomous vehicles, delivery robots, drones, gaming, and other scenarios in which e.g. low and consistent latency is required to provide a satisfyingexperience. Network performance and reliability may also be critical for e.g. autonomous vehicles and other remotely operated devices.

[0098] To verify the validity of the proposed solution, and to verify that the claimed technical advantages are obtainable, two validation tests related to data uplink throughput prediction were performed.

[0099] In a first validation test, a dataset from HUAWEI downloadable from https: / / vehicle2x.net / v2x-measurements / was used. This dataset provides the throughput value of a UE / device for different locations of the UE / device and signal strength. The data was split such that 60% of the data was used to train the RL model, and the remaining 40% was used to test the trained RL and SR models. The location and signal strength was used as input (i.e. state information), and the corresponding output or “action” (or rather, prediction) was the throughput value. For the sake of training the SR model, a sequence of two actions / predictions were generated (as described with reference to Figures 2C and 2D), i.e. one actionindicating the current throughput and another action indicating the nextthroughput. To compare the efficacy of the envisaged approach, comparisons were made to a conventional deep neural network (DNN) model as well as an RL model using an ANN-based function approximator in form of a deep Q network (DQN). The results of the first validation test are shown in Table 1.Table 1. Outcome of first validation test.

[0100] In Table 1, “MAE” is the mean absolute error of the predicted throughput vis-à-vis the real throughput (ground truth). From the MAE values, it may be concluded the envisaged SR-based approach results in a lower MAE compared to the DNN-based model. Also, when compared to the RL (DQN)-based model, there is only a slight degradation in MAE (0.01) but a substantially reduced execution time (the execution time of the envisaged SR-based approach is approximately only 1 / 20thofthat of the RL (DQN)-based model). The equations generated by the trained SR model are summarized as andwhere Z^^ and Z[\ is the latitude and longitude of the UE, respectively, and ^]^_ is the signal-to-interference-plus-noise ratio used as an indication of signal quality. Consequently, in this validation test, the proposed SR-based solution is concluded to provide both a sufficient accuracy, a substantially faster execution time, as well as a sequence of actions helping to more easily explain the reasons for the proposed actions / predictions (in contrast to an ANN-based function approximator).

[0101] In a second validation test, a dataset downloadable from https: / / ieee- dataport.org / open-access / berlin-v2x was used. This dataset provides GPS-located wireless measurements across diverse urban environments in the city of Berlin, for both cellular and sidelink radio access technologies, acquired using multiple vehicles over multiple days. The features from the dataset used to define the environment state in the validation test was Latitude, Longitude, jitter, PCell_Downlink_Num_RBs, PCell_Downlink_TB_Size, SCell_Downlink_Num_RBs, SCell_Downlink_TB_Size and SCell_Downlink_RBs_MCS_31. The feature datarate was used to define the throughput to be predicted. As before, the goal was to trained the RL model to predict the throughput based on the state of the environment, and to then train the SR model to represent the trained RL model.

[0102] As in the first validation test, the data was split such that 60% was used for training, and the remaining 40% was used to test the trained models. For the sake of training the SR model, equations for three actions / predictions (as described with reference to Figures 2C and 2D) were generated, and the results of the envisaged SR- based approach was once again compared with those of a DNN model and RL model using DQN. The results of the second validation test are shown in Table 2.Table 2. Outcome of second validation test.

[0103] From the MAE values, it can be deduced that the envisaged SR-based approach resulted in a lower MAE compared to the DNN-based model. Also, when compared to the RL (DQN)-based model, there is only a slight increase in MAE but a substantial reduction of execution time (the execution time of the envisaged SR- based approach / model is once again only approximately 1 / 20thof that of the RL (DQN)-based model). Once again, it is thus confirmed that the envisaged SR-based approach may have similar accuracy as conventional methods, while providing a substantial reduction of execution time.

[0104] In summary of all of the above, the present disclosure provides an improved way of controlling and / or making predictions about an environment, wherein instead of using a trained RL model to perform such controlling and / or prediction making, the trained RL model is instead used to train an SR model which then replaces the RL model during the deployment stage. As a consequence, the SR model is substantially faster to evaluate and results in a substantially lower execution time more suitable for lower-end devices such as IoT devices and similar. The equations generated by the trained SR model are also easier to understand by a human compared to black-box ANN-based solution, and thus improves the explainability of why a certain action is proposed in response to a particular environmental state. The present disclosure may be used in e.g. a telecommunications network scenario, e.g. as part of an NWDAF in order to meet the predicted demand in prediction capability as the number of connected users continue to grow, without exceeding the expected capability of the involved devices in terms of e.g. computing power and memory / storage capability. Although features and elements may be described above in particular combinations, each feature or element may be used alone without the other features and elements or in various combinations with or without other features and elements. Additionally, variations to the disclosed embodiments may be understood and effected by the skilled person in practicing the claimed invention as defined by the appended patent claims, from a study of the drawings, the disclosure, and the appended claimsthemselves. In the claims, the words “comprising” and “including” does not exclude other elements, and the indefinite article “a” or “an” does not exclude a plurality. The mere fact that certain features are recited in mutually different dependent claims does not indicate that a combination of these features cannot be used to advantage.

Claims

CLAIMS 1. A computer-implemented method (300) of performing a task, comprising: - training (S310) a reinforcement learning, RL, model to perform at least part of a task, wherein the RL model comprises an artificial neural network, ANN,-based function approximator; - after having trained the RL model, using (S312) the trained RL model to train a symbolic regression, SR, model to represent the trained RL model, and - performing the task (s314) using the trained SR model.

2. The method according to claim 1, wherein the SR model is trained to generate at least a first equation representative of how to perform said at least part of the task for a current state of an environment, and at least a second equation representative of how to perform said at least part of the task for a predicted later, such as a next, state of the environment.

3. The method according to claim 1 or 2, wherein the task comprises making predictions related to, and / or controlling of, a telecommunications network.

4. The method according to claim 3, wherein said at least part of the task comprises predicting a data throughput of a user equipment, UE.

5. The method according to claim 5 depending on claim 2, wherein the first equation is representative of the data throughput of the UE at a current location and / or point in time, and wherein the second equation is representative of the data throughput of the UE at a later, such as a next, predicted location and / or later, such as a next, point in time.

6. The method according to claim 4 or 5, wherein the task comprises controlling one or more parameters of the telecommunications network to meet the predicted data throughput of the UE.

7. The method according to any one of the preceding claims, wherein said at least part of the task comprises making predictions and / or detections of at least one of churn, network conflicts and network quality in / of the telecommunications network.

8. The method according to claim 7, wherein the task comprises controlling one or more parameters of the telecommunications network to at least partially optimize said at least one of churn, network conflicts and network quality based on said predictions and / or detections.

9. The method according to any one of claims 2 to 8, wherein the trained SR model is deployed as part of a Network Data Analytics Function, NWDAF (700).

10. A computer-implemented method (400) of training a symbolic regression, SR, model, comprising: - training (S410) a reinforcement learning, RL, model to perform a task, wherein the RL model comprises an artificial neural network, ANN, -based function approximator; - after training the RL model, using (S412) the RL model to train a symbolic regression, SR, model to represent the trained RL model.

11. The method according to claim 10, wherein the SR model is trained to generate at least a first equation representative of how to perform the task for a current state of an environment, and at least a second equation representative of how to perform the task for a predicted later, such as a next, state of the environment.

12. A symbolic regression, SR, training entity (500), comprising processing circuitry (510) and a memory (530) storing instructions, wherein the instructions are such that they, when executed by the processing circuitry, cause the SR training entity to: - train a reinforcement learning, RL, model to perform a task, wherein the RL model comprises an artificial neural network (ANN)-based function approximator, and - after training the RL model, use the RL model to train a symbolic regression, SR, model of an SR entity (600) to represent the trained RL model.

13. The SR training entity according to claim 12, wherein the instructions are further such that they, when executed by the processing circuitry, cause the SR training entity to perform the method according to claim 11.

14. A symbolic regression, SR, entity (600), comprising processing circuitry (610) and a memory (630) storing instructions, wherein the instructions are such that they, when executed by the processing circuitry, cause the SR entity to implement a symbolic regression, SR, model trained in accordance with the method according to claim 10 or 11.

15. A Network Data Analytics Function, NWDAF, entity (700), comprising the SR training entity (500) according to claim 112 and / or the SR entity (600) according to claim 14.

16. A system (800), comprising: - the symbolic regression, SR, training entity (500) according to claim 12 or 13, and - the SR entity (600) according to claim 14 or the NWDAF entity (700) according to claim 15.

17. A computer program (920c) for performing a task, the computer program comprising computer code which, when run on processing circuitry, causes the circuitry to: - train a reinforcement learning, RL, model to perform at least part of a task, wherein the RL model comprises an artificial neural network, ANN,-based function approximator; - after having trained the RL model, use the trained RL model to train a symbolic regression, SR, model to represent the trained RL model, and - perform the task using the trained SR model.

18. A computer program product (910c) comprising a computer program (920c) according to claim 17, and a computer-readable storage medium (930) on which the computer program is stored.

19. A computer program (920a) for training a symbolic regression, SR, model, the computer program comprising computer code which, when run on processing circuitry of an SR training entity (500), causes the SR training entity to: - train a reinforcement learning, RL, model to perform a task, wherein the RL model comprises an artificial neural network, ANN, -based function approximator; - after training the RL model, use the RL model to train a symbolic regression, SR, model to represent the trained RL model.

20. A computer program product (910a) comprising a computer program (920a) according to claim 19, and a computer-readable storage medium (930) on which the computer program is stored.

21. A computer program (920b) for a symbolic regression, SR, entity (600), the computer program comprising computer code which, when run on processing circuitry of the SR entity, causes the SR entity to implement a symbolic regression, SR, model trained in accordance with the method according to claim 10 or 11.

22. A computer program product (910b) comprising a computer program (920b) according to claim 21, and a computer-readable storage medium (930) on which the computer program is stored.

23. A computer-readable storage medium (930) storing data (920d) representative of a symbolic regression, SR, model trained in accordance with the method according to claim 1o or 11.

Images