First node, second node and methods performed thereby for handling analytics reports

Abstract

A computer-implemented method performed by a first node (111) operating in a communications system (100). The first node (111) determines (308), using a first ML model, a respective prediction of one or more indicators of performance of the communications system (100) at a first time period based on: i) an action taken at a second time period preceding the first time period, ii) a respective first observed value of the respective one or more indicators at the second time period, and iii) an analytic report obtained at the second time period from a second node (112). The first node (111) determines (309) a respective second observed value of the respective one or more indicators of performance at the first time period. The first node (111) then provides (311) a first indication to the second node (112) indicating a difference between the respective prediction and respective second observed value as feedback.

Description

[0001] FIRST NODE, SECOND NODE AND METHODS PERFORMED THEREBY FOR HANDLING

[0002] ANALYTICS REPORTS

[0003] TECHNICAL FIELD

[0004] The present disclosure relates generally to a first node and methods performed thereby for handling analytic reports. The present disclosure further relates generally to a second node and methods performed thereby, for handling the analytic reports. The present disclosure also relates generally to computer programs and computer-readable storage mediums, having stored thereon the computer programs to carry out these methods.

[0005] BACKGROUND

[0006] Computer systems in a communications network or communications system may comprise one or more nodes. A node may comprise one or more processors which, together with computer program code may perform different functions and actions, a memory, a receiving port, and a sending port. A node may be, for example, a server. Nodes may perform their functions entirely on the cloud.

[0007] Computer systems may be comprised in a telecommunications network. The telecommunications network may cover a geographical area which may be divided into cell areas, each cell area being served by a type of node, a network node in the Radio Access Network (RAN), radio network node or Transmission Point (TP), for example, an access node such as a Base Station (BS), e.g., a Radio Base Station (RBS), which sometimes may be referred to as e.g., gNB, evolved Node B (“eNB”), “eNodeB”, “NodeB”, “B node”, or Base Transceiver Station (BTS), depending on the technology and terminology used. The base stations may be of different classes such as e.g., Wide Area Base Stations, Medium Range Base Stations, Local Area Base Stations and Home Base Stations, based on transmission power and thereby also cell size. A cell may be understood to be the geographical area where radio coverage may be provided by the base station at a base station site. One base station, situated on the base station site, may serve one or several cells. Further, each base station may support one or several communication technologies. The telecommunications network may also comprise network nodes which may serve receiving nodes, such as user equipments, with serving beams.

[0008] The standardization organization Third Generation Partnership Project (3GPP) is currently in the process of specifying a New Radio Interface called Next Generation Radio or New Radio (NR), as well as a Fifth Generation (5G) Packet Core Network, which may be referred to as 5G Core Network (5GC). The advantages of 5G NR may include higher bandwidth, more resources, low latency and network slicing. 5G may provide services to various applications, such as enhanced Mobile Broad Band (eMBB), machine to Machine type communication (mMTC), Ultra Reliable Low Latency Communication (URLLC), etc.

[0009] 5G may be understood to bring in sizeable flexibility with technological advancements along with innovations of cloud and Artificial Intelligence (Al). This may be understood to bring a whole new set of opportunities in the enterprise segment.

[0010] For many enterprises, mobile cellular technology has already proven to bring great value to their digitalization process, which may include numerous use cases, such as autonomous robotics, enhanced video services, connected vehicles, remote operations, hazard, and maintenance sensors etc. This may be understood to not only enhance productivity in connected factories, but also make workplaces safer.

[0011] In the course of operations of the telecommunications network, data may be collected via the telecommunications network, which may enable to monitor and manage different functions.

[0012] The advent of for example, the Internet of Things (loT) has exponentially increased the amount of data to be monitored. The availability of large amounts of data, such as those collected for example, from loT devices, may be understood to enable the possibility of analysing such data to make predictions on events, with a high predictive power. To make predictions on events may be understood to refer to building mathematical models that may fit those data, which mathematical models may then be used to make predictions for such events. Within this context, machine learning models may be used to analyze the data collected, and enable an improved management of different types of operations via the telecommunications network.

[0013] Machine Learning

[0014] Machine learning (ML) may be understood as the study of computer algorithms that may improve automatically through experience. It is seen as a part of AL ML algorithms may build a model based on sample data, known as "training data", in order to make predictions or decisions without being explicitly programmed to do so. ML algorithms may be used in a wide variety of applications, such as email filtering and computer vision, where it may be difficult or unfeasible to develop conventional algorithms to perform the needed tasks.

[0015] There may be basically three types of ML Algorithms: Supervised Learning, Unsupervised Learning, and Reinforcement Learning (RL).

[0016] Supervised Learning algorithms may comprise a target / outcome variable, or dependent variable, which may have to be predicted from a given set of predictors, that is, independent variables. Using this set of variables, a function may be generated that may map inputs to desired outputs. The training process may continue until the model may achieve a desired level of accuracy on the training data. Once an ML model may have been trained, an inference process may begin, whereby new data may be run through the ML model to calculate an output. Examples of Supervised Learning may be Regression, Decision Tree, Random Forest, KNN, Logistic Regression etc.

[0017] In Unsupervised Learning algorithms, there may be no target or outcome variable to predict / estimate. It may be used for clustering a population into different groups, which may be widely used for segmenting customers in different groups for specific intervention. Examples of Unsupervised Learning may be K-means, mean-shift clustering, Density-Based Spatial Clustering of Applications with Noise (DBSCAN), Expectation-Maximization (EM) Clustering using Gaussian Mixture Models (GMM), Agglomerative Hierarchical Clustering, etc....

[0018] Cluster analysis or clustering may be understood as an ML technique which may comprise grouping a set of objects in such a way that objects in the same group, which may be called a cluster, may be understood to be more similar, in some sense, to each other than to those in other groups, that is, other clusters. It may be understood as a main task of exploratory data mining, and a common technique for statistical data analysis, used in many fields, including pattern recognition, image analysis, information retrieval, bioinformatics, data compression, computer graphics and ML.

[0019] Using an RL algorithm, a machine may be trained to make specific decisions. It may be understood to work as follows: the machine may be exposed to an environment where it may train itself continually using trial and error. This machine may learn from past experience and may try to capture the best possible knowledge to make accurate decisions. An example of RL may be a Markov Decision Process (MDP). The training using RL may comprise generating an ML model. To train such an ML model, an agent, given a state of the environment, may take an action in this environment and receive a reward. The action may result in a new state of the environment. This process may be repeated in a loop. Over time, the agent may learn to take actions that may result in larger immediate and future rewards, meaning that it may be understood to be in the best interest of the agent not to take the action that may only lead to the highest reward in the next state, but the action that may cumulatively lead to the highest reward in the next state and in a future number of states.

[0020] The agent may comprise a neural network which may input the state and may produce an action. There may be several ML algorithms that may be used for training the network of the agent, e.g., policy-learning based, such as actor-critic approaches, or value-based learning, such as deep-q networks.

[0021] A Network Data Analytics Function (NWDAF) may be understood to be a Network Function (NF) in the 5G Core Network (5GC) [1]. The NWDAF may be understood to be designed to collect data from diverse data sources such as User Equipments (UEs), other NFs in 5GC, and Operation, Administration, and Maintenance systems (OAM), Cloud, and Edge networks. The NWDAF may exploit and process the collected data to train ML models which may be used to provide predictions and information from the past to generate analytic reports and deliver them to an NF. Accordingly, different NFs, called service consumers, may be subscribed to the NWDAF. The different NFs may benefit from the capability of the NWDAF and request information about the network state, such as slice load level-related network data, User equipment (UE)-related, and user data congestion analytics.

[0022] One NF may subscribe to the NWDAF to receive: an event-based, periodic, threshold, or one-time notification, which may be on-demand. In the one-time subscription, the NF may be automatically unsubscribed once it may get the requested analytics report from the NWDAF. However, the users may receive multiple analytic reports in an event-based, periodic, and threshold-based subscription. The NWDAF may be equipped with Machine Learning (ML) techniques to generate the required analytic reports about the previous and future system states. The NWDAF may then provide these reports to the subscribed NFs to know about the system evolution and eventual events that may happen.

[0023] Figure 1 is a schematic diagram depicting the different steps between the NWDAF 11 and one NF 12 to produce the feedback on the received analytic report. As depicted in Figure 1 , the NF 12 may request an analytic report from the NWDAF 11 at (1). Once an NF 12 may receive an analytic report at (2), the NF 12 may interact with its environment through specific actions to achieve a goal. Based on the received analytic report, the NF 12 may interact with the surrounding system, the network 13, using a set of selected actions at (3), to improve key performance information, e.g., Key Performance Indicators (KPIs), such as delay and traffic throughput. The NF 12 may observe the KPIs evolution at (4).

[0024] As a result, the accuracy of the ML agent that the NWDAF 11 may use to output analytic reports may be understood to play a critical role in impacting network performance. For this reason, after receiving an analytic report, the NF 12 may need to send feedback on the received report to the NWDAF 11 , as schematically represented in Figure 1 at (5). The NWDAF 11 may have produced the analytic report based on an ML model 14 it may have trained on data obtained from a data source 15. Once the model 14 may have been trained and deployed, it may be used during a model inference phase 16. Model performance feedback may be provided back for model training to improve its performance.

[0025] Providing feedback by an NF service consumer to an NWDAF has widely been discussed, and several approaches have been proposed. In one of the approaches, approach #28 in [2], feedback may be sent by an NF service consumer to the NWDAF Analytical Logical Function (AnLF) about the KPIs that have been affected after receiving an analytics report. In this approach, the NF service consumer may provide information which may contain the result of the triggered actions related to the received analytics report. The result of the triggered actions may be defined as an effect on the network state when actions are applied. Such feedback may be used by NWDAF to evaluate the correctness of an ML model for an analytics Identifier (ID) and perform proper action e.g., re-training if needed. Another example of existing approaches so that NWDAF(AnLF) may provide feedback about ML model performance to an NWDAF, Model Training Logical Function (MTLF) may be Solution #32 in [2], The feedback may contain ML model performance data, such as analytics deviation, comparing predictions to ground truth, data set for the detected deviation, etc. Based on the received ML model performance feedback, the NWDAF(MTLF) may decide to take an action to improve correctness of the ML model.

[0026] Currently, the NFs may share with the NWDAF a set of KPIs that their actions may impact.

[0027] In a first example, Example 1 , it may be assumed that one NF, which may be referred to as NF1 , is responsible for physical resource blocks (PRB) allocation. NF1 may ask the NWDAF for predictions about arrival of users traffic in 30 seconds. Based on the received analytic report and the policy of the NF1 , the NF1 may decide to increase the number of PRBs. In this scenario, the feedback may be the newly observed delay, the data rate of the users, and the packet loss rate.

[0028] In a second example, Example 2, another NF, which may be referred to as NF2 may be responsible for the minimization of the power transmission of the base stations (BSs). NF2 may ask the NWDAF for the traffic arrival of the users in 30s, in the same manner as NF1 may have done. Thus, NF2 may change the power of transmission of the Base Station (BS) while keeping the data rate, the delay, and the packet loss rate of the users under or above a specific threshold. In this example, the same as in Example 1 , the feedback may be the observed delay, the data rate of the users, and the packet loss rate.

[0029] Existing methods to provide feedback to an NWDAF which may provide analytic reports to NFs, may fail to improve the accuracy of the models run by the NWDAF and may result in a poor performance of a communications network.

[0030] SUMMARY

[0031] As part of the development of embodiments herein, one or more problems with the existing technology will first be identified and discussed.

[0032] The ML solutions that the NWDAF may deploy to provide requested analytic reports to an NF may strongly impact the influence of the NF in their surrounding environment. Accordingly, the different NFs may need to provide feedback on the quality of the received analytic reports to enable the NWDAF to improve its ML model.

[0033] Designing the feedback as a set of observed KPIs may be inefficient and has many limitations.

[0034] One limitation may be understood to be that this form of feedback does not provide clear information about the satisfaction of the NF with the received analytic report. For example, in Examples 1 and 2, NF1 and NF2 have different objectives. Accordingly, sharing feedback in the form of observed delay, the data rate of the user, and the packet loss rate may not robustly help the NWDAF to improve its internal model since this does not provide information on whether the received analytic reports helped the corresponding NF achieve its objective.

[0035] Another limitation of existing methods may be understood to be that the feedback provided by one NF may be understood to be dependent on its policy and its objective. For example, NF1 may focus on the PRB allocation for Ultra-Reliable Low Latency Communications (URLLC) traffic, so it may aim to minimize the URLLC traffic delay and increase the associated data rate. NF2, on the other hand, may be deployed to minimize the power of transmission of the base stations, which may improve or reduce the power of transmission while keeping the traffic delay and the data rate under or above a given threshold. Thus, for the same KPIs, i.e., delay and data rate, NF1 and NF2 have opposite objectives. The NWDAF may be understood to generally provide open-loop predictions, that is, independent of the NF actions. Thus, to improve the accuracy of the model of the NWDAF, the provided feedback may need to indicate the quality of the analytic report, while considering the action and / or policy of the NF, since every NF may have a different evaluation.

[0036] A further limitation of existing methods may be understood to be that the NFs may have privacy issues. However, this form of feedback may quickly reveal the privacy of the NF. When one NF sends to the NWDAF the set of KPIs impacted by its actions, this may indicate the state that the NF may be interested in and, as a result, the policy and action space of the NFs.

[0037] Thus, a novel policy-dependent feedback of the NF may be required. The feedback that expresses the satisfaction of the NFs about the provided analytic report may have to be designed. At the same time, this feedback may need to not reveal the policy, objective, and KPIs one NF may be interested in. A closed-loop privacy-preserving feedback that protects the privacy of the NFs may have to be involved to enable the NWDAF to correct its model and then improve the quality of its prediction.

[0038] According to the foregoing, it is an object of embodiments herein to improve the handling of analytic reports in a communications system.

[0039] According to a first aspect of embodiments herein, the object is achieved by a computer- implemented method, performed by a first node. The method is for handling analytic reports. The first node operates in a communications system. The first node determines, using a first machine-learning model, a respective prediction of one or more indicators of performance of the communications system at a first time period. The determining is based on: i) an action taken at a second time period preceding the first time period on an environment of the communications system for which the one or more indicators of performance are predicted, ii) a respective first observed value of the respective one or more indicators of performance in the environment at a second time period preceding the first time period, and iii) an analytic report obtained at the second time period from a second node operating in the communications system on a past or future state of the communications system. The first node determines a respective second observed value of the respective one or more indicators of performance in the environment at the first time period. The first node then provides a first indication to the second node indicating a difference between the respective prediction and respective second observed value of the one or more indicators of performance at the first time period as feedback on the analytic report

[0040] According to a second aspect of embodiments herein, the object is achieved by a computer-implemented method, performed by the second node. The method is for handling the analytic reports. The second node operates in the communications system. The second node provides the analytic report on the past or future state of the communications system to the first node operating in the communications system. The analytic report has been obtained at the second time period. The second node then receives the first indication, from the first node, indicating the difference between the respective prediction and the respective second observed value of the one or more indicators of performance of the communications system at the first time period as feedback on the analytic report. The respective prediction is based on the provided analytic report.

[0041] According to a third aspect of embodiments herein, the object is achieved by the first node. The first node may be understood to be for handling the analytic reports. The first node is configured to operate in the communications system. The first node is configured to determine, using the first machine-learning model, the respective prediction of the one or more indicators of performance of the communications system at the first time period. The determining is configured to be based on: i) the action taken at the second time period preceding the first time period on the environment of the communications system for which the one or more indicators of performance are configured to be predicted, ii) the respective first observed value of the respective one or more indicators of performance in the environment at the second time period preceding the first time period, and iii) the analytic report configured to be obtained at the second time period from the second node configured to be operating in the communications system on the past or future state of the communications system. The first node is also configured to determine the respective second observed value of the respective one or more indicators of performance in the environment at the first time period. The first node is further configured to provide the first indication to the second node configured to indicate the difference between the respective prediction and the respective second observed value of the one or more indicators of performance at the first time period as feedback on the analytic report.

[0042] According to a fourth aspect of embodiments herein, the object is achieved by the second node. The second node may be understood to be for handling the analytic reports. The second node is configured to operate in the communications system. The second node is configured to provide the analytic report on the past or future state of the communications system to the first node configured to operate in the communications system. The analytic report is configured to have been obtained at the second time period. The second node is further configured to receive the first indication, from the first node, configured to indicate the difference between the respective prediction and the respective second observed value of one or more indicators of performance of the communications system at the first time period as feedback on the analytic report. The respective prediction is configured to be based on the analytic report configured to be provided.

[0043] According to a fifth aspect of embodiments herein, the object is achieved by a computer program, comprising instructions which, when executed on at least one processing circuitry, cause the at least one processing circuitry to carry out the method performed by the first node.

[0044] According to a sixth aspect of embodiments herein, the object is achieved by a computer-readable storage medium, having stored thereon the computer program, comprising instructions which, when executed on at least one processing circuitry, cause the at least one processing circuitry to carry out the method performed by the first node.

[0045] According to a seventh aspect of embodiments herein, the object is achieved by a computer program, comprising instructions which, when executed on at least one processing circuitry, cause the at least one processing circuitry to carry out the method performed by the second node.

[0046] According to an eighth aspect of embodiments herein, the object is achieved by a computer-readable storage medium, having stored thereon the computer program, comprising instructions which, when executed on at least one processing circuitry, cause the at least one processing circuitry to carry out the method performed by the second node.

[0047] By determining the respective prediction of the one or more indicators of performance of the communications system at the first time period, the first node may be enabled to assess the correctness of the analytic report obtained at the second time period from the second node using the internally trained first machine learning model that may be understood to predict values of the relevant one or more indicators of performance, e.g., KPIs, by including triggered actions. In particular examples wherein the first node may be a service consumer of the second node, and the second node may be an NWDAF, this functionality may be understood to be an enhancement of for example, service consumer NFs of an NWDAF.

[0048] By determining the respective second observed value of the respective one or more indicators of performance, the first node may then be enabled to use the current and predicted values of the one or more indicators of performance for evaluating the accuracy of the analytics report received by considering the impact of the triggered actions.

[0049] By providing the first indication to the second node, the first node may be able to produce a closed-loop feedback. The feedback may be understood to be closed-loop feedback because it may be understood to quantify and measure the satisfaction of the first node from the accuracy and guidance of the received analytic report received from the second node, in order to take suitable actions.

[0050] BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Examples of embodiments herein are described in more detail with reference to the accompanying drawings, according to the following description.

[0052] Figure 1 is a schematic diagram illustrating an example of different steps between NWDAF and one NF to produce feedback on a received analytic report, according to existing methods.

[0053] Figure 2 is a schematic diagram illustrating two non-limiting examples, in panels a) and b), of a communications system, according to embodiments herein.

[0054] Figure 3 is a flowchart depicting a method in a first node, according to embodiments herein.

[0055] Figure 4 is a flowchart depicting a method in a second node, according to embodiments herein.

[0056] Figure 5 is a schematic diagram depicting particular aspects of a non-limiting example of the method performed by the first node and the second node, according to embodiments herein.

[0057] Figure 6 is a schematic diagram depicting particular aspects of another non-limiting example of the method performed by the first node and the second node, according to embodiments herein.

[0058] Figure 7 is a schematic diagram depicting a non-limiting example of the method performed by the first node and the second node, according to embodiments herein.

[0059] Figure 8 is a schematic diagram depicting aspects of non-limiting example of the method performed by the first node and the second node, according to embodiments herein.

[0060] Figure 9 is a schematic diagram depicting another non-limiting example of the method performed by the first node and the second node, according to embodiments herein.

[0061] Figure 10 is a schematic block diagram illustrating an embodiment of a first node, according to embodiments herein.

[0062] Figure 11 is a schematic block diagram illustrating an embodiment of a second node, according to embodiments herein.

[0063] DETAILED DESCRIPTION

[0064] Certain aspects of the present disclosure and their embodiments address the challenges identified in the Background and Summary sections with the existing methods and provide solutions to the challenges discussed. Embodiments herein may be understood to relate to a privacy-preserving feedback learning for 5G core networks.

[0065] Particular embodiments herein may relate to a novel ML approach to design the feedback that the NFs may send to NWDAF. This designed feedback may depend on the policy of the NFs and their target reward. At the same time, embodiments herein may respect the privacy constraints of the NFs. The feedback of embodiments herein may be understood to not reveal the action space of the NF, the KPIs it may be interested in, and its objective. This feedback may enable the NWDAF to evaluate its deployed ML model and improve its performance.

[0066] Embodiments herein may comprise two innovative steps or components to produce feedback that may respect the privacy of an NF. First, a novel ML model may be deployed in the NF. This model may predict the future system state, e.g., the set of future KPIs. This prediction will be mainly based on the considered NF action made, the current observed KPIs, and the received analytic report. Once the model may be well-trained, every NF may calculate the feedback on the received analytic report, which may represent the accuracy of the state prediction, and send it to the NWDAF.

[0067] Some of the embodiments contemplated will now be described more fully hereinafter with reference to the accompanying drawings, in which examples are shown. In this section, the embodiments herein will be illustrated in more detail by a number of exemplary embodiments. Other embodiments, however, are contained within the scope of the subject matter disclosed herein. The disclosed subject matter should not be construed as limited to only the embodiments set forth herein; rather, these embodiments are provided by way of example to convey the scope of the subject matter to those skilled in the art. It should be noted that the exemplary embodiments herein are not mutually exclusive. Components from one embodiment may be tacitly assumed to be present in another embodiment and it will be obvious to a person skilled in the art how those components may be used in the other exemplary embodiments.

[0068] Several embodiments and examples are comprised herein. It should be noted that the embodiments and / or examples herein are not mutually exclusive. Components from one embodiment or example may be tacitly assumed to be present in another embodiment or example and it will be obvious to a person skilled in the art how those components may be used in the other exemplary embodiments and / or examples.

[0069] Figure 2 depicts two non-limiting examples, in panels “a” and “b”, respectively, of a communications system 100, in which embodiments herein may be implemented. In some example implementations, such as that depicted in the non-limiting example of Figure 2a, the communications system 100 may be a computer network. In other example implementations, such as that depicted in the non-limiting example of Figure 2b, the communications system 100 may be implemented in a telecommunications system, sometimes also referred to as a telecommunications network, cellular radio system, cellular network, or wireless communications system. In some examples, the telecommunications system may comprise network nodes which may serve receiving nodes, such as wireless devices. The communications system 100 may for example be a network such as a 5G system, or a newer system supporting similar functionality. The telecommunications system may additionally support other technologies such as, for example, Long-Term Evolution (LTE), e.g., LTE Frequency Division Duplex (FDD), LTE Time Division Duplex (TDD), LTE Half-Duplex Frequency Division Duplex (HD-FDD), or LTE operating in an unlicensed band. The telecommunications system may also support yet other technologies, such as Wideband Code Division Multiple Access (WCDMA), Universal Mobile Telecommunications System Terrestrial Radio Access (UTRA) TDD, Global System for Mobile communications (GSM) network, GSM / Enhanced Data Rate for GSM Evolution (EDGE) Radio Access Network (GERAN) network, Ultra-Mobile Broadband (UMB), EDGE, any combination of Radio Access Technologies (RATs) such as e.g. Multi-Standard Radio (MSR) base stations, multi-RAT base stations etc., any 2rd Generation Partnership Project (3GPP) cellular network, Wireless Local Area Network / s (WLAN) or WiFi network / s, Worldwide Interoperability for Microwave Access (WiMax), IEEE 802.15.4-based low-power short-range networks such as IPv6 over Low-Power Wireless Personal Area Networks (6LowPAN), Zigbee, Z-Wave, Bluetooth Low Energy (BLE), or any cellular network or system. The telecommunications system may for example support a Low Power Wide Area Network (LPWAN). LPWAN technologies may comprise Long Range physical layer protocol (LoRa), Haystack, SigFox, LTE-M, and Narrow-Band loT (NB-loT).

[0070] The communications system 100 comprises nodes, whereof a first node 111 and a second node 112 are depicted in Figure 2. In some examples, such as that depicted in Figure 2 a), the first node 111 and the second node 112 may be co-located or be the same node. The communications system 100 may comprise additional nodes. In particular examples, the communications system 100 may comprise a plurality of first nodes 111. Any of the first nodes 111. In some of such examples, the communications system 100 may comprise a third node.

[0071] Any of the first node 111 and the second node 112 may be understood, respectively, as a first computer system or server and a second computer system or server. Any of the first node 111 and the second node 112 may be implemented as a standalone server in e.g., a host computer in the cloud 115, as depicted in the non-limiting example of Figure 2b) for the first node 111 and the second node 112. In other examples, any of the first node 111 and the second node 112 may be a distributed node or distributed server, such as a virtual node in the cloud 115, and may perform some of its respective functions locally, e.g., by a client manager, and some of its functions in the cloud 115, by e.g., a server manager. In other examples, any of the first node 111 and the second node 112 may perform its functions entirely on the cloud 115, or partially, in collaboration or collocated with a radio network node. Yet in other examples, any of the first node 111 and the second node 112 may also be implemented as processing resources in a server farm.

[0072] Yet in other examples, any of the first node 111 and the second node 112 may also be implemented as virtual network functions, e.g., according to a Network Functions Virtualization (NFV) Architecture.

[0073] Any of the first node 111 and the second node 112 may be under the ownership or control of a service provider or may be operated by the service provider, or on behalf of the service provider.

[0074] Any of the first node 111 and the second node 112, may be co-localized. However, in typical embodiments, the first node 111 and the second node 112 may be different nodes. first node 111 may be understood as a node having a capability to consume a service offered by the second node 112. In a particular non-limiting example, wherein the communications system 100 may be a 5G network, the first node 111 may, in such examples be a Consumer NF.

[0075] The second node 112 may have a capability to perform machine-implemented learning procedures, which may be also referred to as “machine learning” (ML).

[0076] The second node 112 may have a capability to manage an artificial neural network. The artificial neural network may be understood as a machine learning framework, which may comprise a collection of connected nodes, where in each node or perceptron, there may be an elementary decision unit. Each such node may have one or more inputs and an output. The input to a node may be from the output of another node or from a data source. Each of the nodes and connections may have certain weights or parameters associated with it. In order to solve a decision task, the weights may be learnt or optimized over a data set which may be representative of the decision task. The most commonly used node may have each input separately weighted, and the sum may be passed through a non-linear function which may be known as an activation function. The nature of the connections and the node may determine the type of the neural network, for example a feedforward network, recurrent neural network etc. That the second node 112 may have the capability to manage the artificial neural network may be understood herein as having the capability to store the training data set and the models that may result from the machine learning, to train a new model, and once the model may have been trained, to use this model for prediction. In some embodiments, such as those depicted in Figure 2 b), the system that may be used for training the model and the one used for prediction may be different.

[0077] The second node 112 used for training the artificial neural network 121 may require more computational resources than the first node 111. The second node 112 may, for example, support running python / Java with Tensorflow or Pytorch, Theano etc... The second node 112 may also have GPU capabilities.

[0078] The second node 112 be understood as an operator managed network analytics logical function. That is, as a node that may have a capability to handle data collection and analysis from different sources in the communications system 100. The second node 112 may interact with different entities for different purposes, such as to data collection provided by, e.g., Access and Mobility Function (AMF), Session Management Function (SMF), Policy Control Function (PCF), Unified Data Management Function (UDM), Application Function (AF), based on event subscription, directly or via a Network Exposure Function (NEF), and Operations And Management (OAM), retrieval of information from data repositories, e.g., Unified Data Repository (UDR) via UDM for subscriber-related information, retrieval of information about NFs, e.g., NRF for NF-related information, and Network Slice Selection Function (NSSF) for slice-related information, on demand provision of analytics to consumers, and storage in an Analytics Data Repository Function, e.g., Analytics Data Repository Function (ADRF), for two types of data: collected Data, e.g., Event Exposure data, and Analytics reports. As depicted in Figure 2, a non-limiting example of the second node 112, wherein the communications system 100 may be a 5G network, may be an NWDAF.

[0079] The third node may be an orchestrator in a FL scenario.

[0080] The communications system 100 may in some examples, comprise one or more radio network nodes, such as radio network node 130, depicted in Figure 2 b). The radio network node 130 may be, e.g., comprised in a Radio Access Network of the telecommunications system. That is, the radio network node 130 may be a transmission point such as a radio base station, for example a gNB, an eNB, or any other network node with similar features capable of serving a wireless device, such as a user equipment or a machine type communication device, in the communications system 100. In typical examples, the radio network node 130 may be a base station, such as a gNB or an eNB. In other examples, the radio network node 130 may be a distributed node, such as a virtual node in the cloud 115, and may perform its functions entirely on the cloud 115, or partially, in collaboration with a radio network node.

[0081] The telecommunications system may cover a geographical area, which in some embodiments may be divided into cell areas, wherein each cell area may be served by a radio network node 130, although, one radio network node 130 may serve one or several cells. In the example of Figure 2, the cells are not depicted to simplify the figure. The radio network node 130 may be of different classes, such as, e.g., macro eNodeB, home eNodeB or pico base station, based on transmission power and thereby also cell size. In some examples, the radio network node 130 may serve receiving nodes with serving beams. The radio network node 130 may be directly connected to one or more core networks. Any of the first node 111 and the second node 112, and / or any of the nodes comprised in the communications system 100 may support one or several communication technologies, and its name may depend on the technology and terminology used.

[0082] A device 140 may be comprised in the telecommunication network. The device 140 comprised in the communications system 100 may be a wireless communication device such as a 5G UE, or a UE, which may also be known as e.g., mobile terminal, wireless terminal and / or mobile station, a Customer Premises Equipment (CPE) a mobile telephone, cellular telephone, or laptop with wireless capability, just to mention some further examples. The device 140 comprised in the telecommunications system may be, for example, portable, pocket-storable, hand-held, computer-comprised, or a vehicle-mounted mobile device, enabled to communicate voice and / or data, via the RAN, with another entity, such as a server, a laptop, a Personal Digital Assistant (PDA), or a tablet, Machine-to-Machine (M2M) device, device equipped with a wireless interface, such as a printer or a file storage device, modem, sensor, loT device, or any other radio network unit capable of communicating over a radio link in a communications system. In typical examples, the device 140 may be, or comprise, a microphone. The device 140 comprised in the telecommunications system may be enabled to communicate wirelessly in the telecommunications system. The communication may be performed e.g., via a RAN, and possibly the one or more core networks, which may be comprised within the telecommunications system.

[0083] It may be understood that the telecommunications network may comprise additional radio network nodes 130 and / or additional devices 140.

[0084] The first node 111 may be configured to communicate within the communications system 100 with the second node 112 over a first link 141 , e.g., a radio link, or a wired link. The first node 111 may be configured to communicate within the communications system 100 with the radio network node 130 over a second link 142, e.g., a radio link, or a wired link. The second node 112 may be configured to communicate within the communications system 100 with the radio network node 130 over a third link 143, e.g., a radio link, or a wired link. The radio network node 130 may be configured to communicate within the communications system 100 with the device 140 over a fourth link 144, e.g., a radio link.

[0085] Any of the first link 141 , the second link 142, the third link 143 and the fourth link 144 may be a direct link or may be comprised of a plurality of individual links, wherein it may go via one or more computer systems or one or more core networks in the communications system 100, which are not depicted in Figure 2, or it may go via an optional intermediate network. The intermediate network may be one of, or a combination of more than one of, a public, private or hosted network; the intermediate network, if any, may be a backbone network or the Internet; in particular, the intermediate network may comprise two or more sub-networks, which is not shown in Figure 2. In general, the usage of “first”, “second”, “third” and / or “fourth” herein may be understood to be an arbitrary way to denote different elements or entities, and may be understood to not confer a cumulative or chronological character to the nouns they modify.

[0086] Some of the embodiments contemplated herein will now be described more fully with reference to the accompanying drawings. Other embodiments, however, are contained within the scope of the subject matter disclosed herein, the disclosed subject matter should not be construed as limited to only the embodiments set forth herein; rather, these embodiments are provided by way of example to convey the scope of the subject matter to those skilled in the art.

[0087] Embodiments of a computer-implemented method, performed by the first node 1 11 , will now be described with reference to the flowchart depicted in Figure 3. The method is for handling analytic reports. The first node 1 1 1 operates in the computer system 100.

[0088] Several embodiments are comprised herein. In some embodiments all the actions may be performed. In some embodiments, some actions may be optional. In Figure 3, optional actions are indicated with dashed lines. It should be noted that the examples herein are not mutually exclusive. Components from one embodiment may be tacitly assumed to be present in another embodiment and it will be obvious to a person skilled in the art how those components may be used in the other exemplary embodiments. One or more embodiments may be combined, where applicable. All possible combinations are not described to simplify the description.

[0089] In some embodiments, at least one of the following may apply: a) the communications system 100 may be 5G system, b) the first node 11 1 may be an NF and c) the second node 1 12 may be an NWDAF.

[0090] Action 301

[0091] In this Action 301 , the first node 1 11 may send, to the second node 1 12, a respective request for a respective analytic report on a respective past or future state of the communications system 100, for a respective time period (t) of a plurality of time periods.

[0092] Sending in this Action 301 may be performed, e.g., via the first link 141 .

[0093] Action 302

[0094] In this Action 302, the first node 1 11 may obtain the respective analytic report (p ) from the second node 1 12, for the respective time period (t) of the plurality of time periods, responsive to the sent respective request.

[0095] Obtaining in this Action 301 may comprise receiving from the second node 1 12, e.g.,, via the first link 141 . Action 303

[0096] In this Action 303, the first node 11 1 may determine, for each respective time period, a respective action (at) to be taken on the environment. The respective action may be based on the respective analytic report (p ) and a policy (IT) of the first node 1 11.

[0097] Determining may be understood as calculating, estimating, deriving, or similar, or obtaining or receiving from another node.

[0098] The policy may be understood to refer to an algorithm used by the first node 1 11 to determine the action it may take based on a set of inputs, including the respective analytic report

[0099] Initially, the first node 1 11 , as in some examples, every NF, may have its strategy to act on the surrounding environment and impact a set of one or more indicators of performance, e.g., KPIs, it may be interested in. This strategy may be, for example, a set of rules described by an expert. In some examples, the determining in this Action 303 may be performed by an ML, which may be referred to herein as a second machine learning model. The second machine learning model may be an ML, for example, RL, that may output the action (a). For example, the first node 1 11 , e.g., the NF i (NR), may have an RL agent that may decide an action (a), following a policy TT, every time step t. This agent may observe the received analytic report (p ) requested from the second node 112, and may determine the action (at), as is represented schematically in Figure 6, which will be described later. Generally, this action (at) may impact a set of one or more indicators of performance, e.g., KPIs, at time t such as traffic delay, data rate, and packet loss rate. Thus, a new state of these one or more indicators of performance, e.g., KPIs, may appear at t+1 , which may be denoted herein as

[0100] Action 304

[0101] In this Action 304, the first node 11 1 may train a first machine learning model (Mi) to predict a respective first prediction of the one or more indicators of performance of the communications system 100 at the respective first time period (t‘+1). The training may be based on: i) the determined respective action, as determined in Action 303, taken for the respective time period (at) on the environment, ii) a respective observed value of the respective one or more indicators of performance in the environment at a respective second time period preceding the respective time period (7f ), and iii) the respective analytic report obtained from the second node 1 12 in Action 302 (pl). In other words, as will be depicted in Figure 6, the first machine learning model (Mi) may predict the evolution over time of the one or more indicators of performance, e.g., the set of KPIs Ki+1, indicating the state of the communications system 100, based on the following: i) the action at, ii) the set of KPIs Kt' observed at t, and iii) the received analytic report pltat time t.

[0102] An observed value may be understood as a real value that may be measured or detected in the environment. That is, a value that may not be the result of a prediction by a model, e.g., the first machine learning model.

[0103] That the training may be based on i), ii) and iii) may be understood to mean that the training takes i), ii) and iii) as input.

[0104] The first machine learning model (Mi) may be understood as a new ML model, which may comprise a supervised ML algorithm. Examples of such a supervised ML algorithm may be Deep Neural Networks which may comprise multiple layers of so-called neurons with trainable weights, auto-regressive models such as Autoregressive Moving-Average Model (ARMA), or Random Forests.

[0105] The training in this Action 304 may be performed until an accuracy level may be reached. It may be understood that Actions 301 -304 may be performed iteratively until the desired accuracy level may be reached.

[0106] Local training

[0107] In a first group of examples of embodiments herein, the training in this Action 304 may be a local training. That is, the training of the first machine learning model (Mi) may be performed entirely by the first node 11 1. The local training of the first machine learning model (Mi) may be realized in a way to get an accurate prediction of the one or more indicators of performance, e.g., the KPIs, thereby, minimizing the following Loss function:

[0108] The received analytic report (ptl) may be understood to mainly impact a quality of the prediction (^t‘+1). Thus, by training the first machine learning model, the first node 1 11 may then be enabled to use the first machine learning model (Mi) to evaluate the quality and / or accuracy of the provided analytic report pi.

[0109] FL training

[0110] In a second group of examples of embodiments herein, another approach may be used to train the first machine learning model (Mi) in this Action 304, which may be different from the local training. The approach in the second group of examples may be based on federated learning (FL). In FL, the model, e.g., (Mi), may be trained across multiple plurality of first nodes 1 11 , e.g., decentralized edge devices, or servers, holding local data samples, without exchanging them. In such examples, the horizontal and vertical federated learning [3] may be distinguished. Horizontal FL may be understood to be a specific type of FL where the participating first nodes 11 1 , e.g., devices or nodes, may have similar features but different samples of data. Horizontal FL may be used when data collected by the plurality of first nodes 1 11 , e.g., NFs, such as the one or more indicators of performance, e.g., KPIs, may overlap a lot. That is, the locally collected data instance at the different training first nodes 1 11 , e.g., devices, may have the same set of features. Vertical FL may be used when the KPIs of the two datasets may overlap little, but the users may overlap a lot. Vertical FL may be understood to be the second type of FL where the participating first nodes 1 11 , e.g., devices or nodes, may have different sets of features, and the collaboration may aim to jointly analyze datasets with complementary features. When the different one or more indicators of performance, e.g., KPIs of the different first nodes 11 1 may be the same, the first machine learning model (Mi) may be trained using the horizontal FL approach.

[0111] Horizontal FL

[0112] In some examples, a set of n first nodes 11 1 , e.g., NFs, F (F = {NFi, NFn}) may be interested in the same one or more indicators of performance, e.g., in the same set of KPIs, and the set of n first nodes 1 11 may collaborate and train their respective first machine learning models using FL. In such examples, there may be a third node, which may be understood to be a trustworthy orchestrator, that may periodically, e.g., every T period of times, select a set G of m first nodes 11 1 , e.g., NFs, (m <= n, G c F) to send the parameters wt' of their respective local first machine learning model Mi. This orchestrator may then determine a new common model, M*={w*}: herein Xi may be understood to be the size of data points a particular first node 1 11 , e.g., NFi , may use to train Mi locally, and X may be: X = ^NF^G}xi-

[0113] The orchestrator may iteratively achieve the averaging of the respective local first machine learning models received from the different first nodes 1 11 , e.g., NF, until convergence. Thus, every T periods, any of the first nodes 1 11 , e.g., NFi, may exchange with the orchestrator their respective local first machine learning model Mi and then receive the new common model M* from the orchestrator.

[0114] Vertical FL

[0115] When the first nodes 1 11 , e.g., NFs, may have data with different one or more indicators of performance, e.g., KPIs, the training in this Action 304 may be achieved using vertical federated learning. In such examples, before starting the training, the first nodes 1 1 1 , e.g., NFs, may need to align the data. In vertical FL, each sample which may comprise features, may be split among workers. When the locally trained ML models may be concatenated it may be necessary to know which feature may belong to which sample, in order to make the data aligned. This may be performed in a privacy-preserving entity alignment step, wherein private set intersection methods may be used to identify shared sample identifiers (IDs) while keeping the unaligned dataset concealed. Following the alignment, the collaborating parties may commence the training of the vertical federated learning first machine learning model by utilizing the aligned samples in a privacy-preserving training by exchanging intermediate results. Each worker may send the locally trained ML model, even if the training is not complete, therefore it may be called intermediate, to a server, which may coordinate the process to measure the accuracy for instance. The predominant training protocol may often involve the application of gradient descent. As in the horizontal FL, an orchestrator may be used to achieve the training.

[0116] Action 305

[0117] Once the first machine learning model may have been trained in Action 304 to reach a desired level of accuracy, the first node 111 may use it to make predictions on newly obtained data, that is, the first node 111 may start an inferencing phase once the training phase may have concluded.

[0118] Accordingly, in this Action 305, the first node 111 may send a request for an analytic report to the second node 112. This Action 305 may be understood to be performed in a similar manner as Action 301 , e.g., via the first link 141 .

[0119] Action 306

[0120] In this Action 306, the first node 111 may obtain the analytic report (plt) from the second node 112.

[0121] This Action 306 may be understood to be performed in a similar manner as Action 302, e.g., via the first link 141 .

[0122] Action 307

[0123] In this Action 307, the first node 111 may determine an action (at) to be taken at a second time period (t) based on the obtained analytic report (p ) and the policy (IT) of the first node 111 and initiate performing the action (at). The second time period may precede the first time period. The action may be to be taken on an environment of the communications system 100. This Action 307 may be understood to be performed in a similar manner as Action 303, e.g., via the first link 141 .

[0124] In some embodiments, the determining in this Action 307 of the action may be performed using one of: a set of rules and the second machine-learning model described earlier. Determining may be understood as calculating, estimating, deriving, or similar, or obtaining or receiving from another node.

[0125] Action 308

[0126] Once the first machine learning model (Mi) may be efficiently trained, it may be used to predict the evolution of the one or more indicators of performance, e.g., KPIs, over time.

[0127] In this Action 308, the first node 111 determines, using the first machine-learning model, a respective prediction of the one or more indicators of performance of the communications system 100 at the first time period (t‘+1). The determining in this Action 308 is based on: i) the action taken at the second time period (at) preceding the first time period on an environment of the communications system 100 for which the one or more indicators of performance are predicted, ii) a respective first observed value of the respective one or more indicators of performance in the environment at the second time period preceding the first time period and iii) the analytic report obtained at the second time period (p‘) from the second node 112 operating in the communications system 100 on a past or future state of the communications system 100. The action may have been performed in Action 307. The analytic report may have been obtained in Action 306.

[0128] The determining in this Action 308 of the respective prediction may be performed using the trained first machine learning model (Mi) in Action 304. That is, the first machine learning model may be understood to be used in an inference phase.

[0129] By determining the respective prediction of the one or more indicators of performance of the communications system 100 at the first time period (t‘+1) in this Action 308, the first node

[0130] 111 may be enabled to assess the correctness of the analytic report obtained at the second time period (p‘) from the second node 112 using the internally trained first machine learning model that may be understood to predict values of the relevant one or more indicators of performance, e.g., KPIs, by including triggered actions (at). In particular examples wherein the first node 111 may be a service consumer of the second node 112, and the second node

[0131] 112 may be an NWDAF, this functionality may be understood to be an enhancement of for example, service consumer NFs of an NWDAF.

[0132] Action 309

[0133] In this Action 309, the first node 111 determines a respective second observed value of the respective one or more indicators of performance in the environment at the first time period (Kt+1').

[0134] By determining the respective second observed value of the respective one or more indicators of performance (Kt+i') in this Action 309, the first node 111 may then be enabled to use the current and predicted values of the one or more indicators of performance for evaluating the accuracy of the analytics report received in Action 306 by considering the impact of the triggered actions in Action 307.

[0135] Action 310

[0136] Once the first machine learning model (Mi) may be well trained, it may also be deployed to enable to determine and ultimately provide feedback ( )) to the second node 112 on its provided analytic reports (ptl). After the first node 111 may have predicted the values of the one or more indicators of performance, e.g., KPIs, in Action 308, the first node 111 may then be enabled to compare them with the actual values obtained in Action 309 to assess the correctness of the analytic reports from the second node 112. In this Action 310, the first node 111 determines a first indication. The first indication may be understood to be a result of such a comparison. The first indication may indicate a difference between the respective prediction (^+1) and the respective second observed value of the one or more indicators of performance at the first time period (7 ‘+1) as feedback on the analytic report (p ).

[0137] In contrast with existing methods, the impact of the triggered actions by the first node 111 , e.g., a service consumer of the second node 112, may be understood to be considered by the first node 111 in the assessment of the accuracy of the analytics report.

[0138] The first indication may be understood to be an indication of the quality of the prediction of the first machine learning model based on the analytic report received from the second node 112. Thus, if the first node 111 determines that the quality of the prediction from the second node 112, p is degraded, then the first node 111 may conclude that the received analytic report, p , from the second node 112 may not be efficient and robust. Thus, the second node 112 may then need to be notified to retrain, or to improve, the model the second node 112 may use that may have been responsible for the analytic report design.

[0139] The first node 111 may determine the first indication in this Action 310 as the error in the prediction as feedback in the first indication that is, the loss, according to the following equation: ft = |^+i- ^+i|

[0140] Ki+1may be understood to be the prediction of the one or more indicators of performance, e.g., KPIs, using the first machine learning model Mtand

[0141] Ki+1may be understood to be observation the of the one or more indicators of performance, e.g., KPIs, at time t+1 , that is, the first time period.

[0142] By determining the first indication in this Action 310, the first node 111 may then be enabled to share the first indication with the second node 112. Action 311

[0143] In this Action 311 , the first node 111 provides the first indication to the second node 112 indicating the difference between the respective prediction and the respective second observed value of the one or more indicators of performance at the first time period as feedback on the analytic report, that is, the analytic report received in Action 306.

[0144] In some embodiments, when providing in this Action 311 the first indication to the second node 112, the first node 111 may refrain from indicating the action taken on the environment for the first time period. That is, the action determined and performed in Action 307.

[0145] In some embodiments, when providing in this Action 311 the first indication to the second node 112, the first node 111 may refrain from indicating the one or more indicators of performance.

[0146] By providing, e.g., sending, the first indication to the second node 112 refraining from indicating the action taken on the environment and / or the one or more indicators of performance, the first node 111 may be able to produce a closed-loop privacy-preserving feedback. The feedback may be understood to be closed-loop feedback because it may be understood to quantify and measure the satisfaction of the first node 111 from the accuracy and guidance of the received analytic report received from the second node 112, in order to take suitable actions. The feedback may be also understood to a privacy-preserving feedback transmission, since the first node 111 may be able to send feedback about the correctness of the analytics report to the second node 112, which feedback may be understood to consider the triggered actions, while no information about the actions may be revealed. This may be understood to be since, using the first machine learning model, the first node 111 may be able to measure the accuracy of the provided analytic report, as the first node 111 may have requested, and its impact on a particular action. Thus, the first node 111 may send this measurement as feedback, (fi) to the second node 112. This feedback may be understood to not reveal the privacy of the first node 111.

[0147] Embodiments of a computer-implemented method, performed by the second node 112, will now be described with reference to the flowchart depicted in Figure 4. The method is for handling the analytic reports. The second node 112 operates in the communications system 100.

[0148] Several embodiments are comprised herein. In some embodiments all the actions may be performed. In some embodiments, some embodiments of the actions may be optional. It should be noted that the examples herein are not mutually exclusive. Components from one embodiment may be tacitly assumed to be present in another embodiment and it will be obvious to a person skilled in the art how those components may be used in the other exemplary embodiments. One or more embodiments may be combined, where applicable. All possible combinations are not described to simplify the description. The detailed description of some of the following corresponds to the same references provided above, in relation to the actions described for the first node 111 , and will thus not be repeated here. For example, the one or more indicators of performance may be one or more KPIs.

[0149] In some embodiments, at least one of the following may apply: a) the communications system 100 may be 5G system, b) the first node 111 may be an NF and c) the second node 112 may be an NWDAF.

[0150] Action 401

[0151] In this Action 401 , the second node 112 may receive the respective request for the respective analytic report from the first node 111 , for the respective time period of a plurality of time periods.

[0152] Action 402

[0153] In this Action 402, the second node 112 may provide the respective analytic report to the first node 111 , for the respective time period of the plurality of time periods.

[0154] Action 403

[0155] In this Action 403, the second node 112 may receive the request for the analytic report from the first node 111.

[0156] Action 404

[0157] In in this Action 404, the second node 112 provides the analytic report on the past or future state of the communications system 100 to the first node 111 operating in the communications system 100. The analytic report has been obtained at the second time period.

[0158] The providing in this Action 404 of the analytic report may be responsive to the received request in Action 403.

[0159] Action 405

[0160] In this Action 405, the second node 112 receives the first indication, from the first node 111 , indicating the difference between the respective prediction and the respective second observed value of the one or more indicators of performance of the communications system 100 at the first time period as feedback on the analytic report, The respective prediction is based on the provided analytic report in Action 404. The received first indication may be based on the output of the first machine learning model to predict the respective first prediction of the one or more indicators of performance of the communications system 100 at the respective first time period. The first machine learning model may have been trained based on the provided respective analytic reports.

[0161] The respective prediction may be further based on the action taken at the second time period preceding the first time period on the environment of the communications system 100 for which the one or more indicators of performance may be predicted. The respective prediction may be further based on the respective first observed value of the respective one or more indicators of performance in the environment at the second time period preceding the first time period.

[0162] The first indication from the first node 1 11 may omit indicating the action taken on the environment for the first time period.

[0163] The first indication from the first node 11 1 may omit indicating the one or more indicators of performance.

[0164] Figure 5 is a schematic diagram depicting a non-limiting example of a method performed by the first node 1 11 and the second node 1 12 during the inference phase of the first machine learning model, according to embodiments herein. Also depicted in Figures 5 is an architecture to utilize the local, first machine learning model inside the first node 1 11. In this non-limiting example, the second node 1 12 is an NWDAF, and the first node 1 11 is a consumer NF of the NWDAF. The first node 1 11 , according to Action 306, may obtain the analytic report pi that the second node 112 may provide according to Action 404. The first node 11 1 may then, according to Action 307, determine the action to be taken at the first time period atbased on the obtained analytic report ptland the policy (IT) of the first node 1 11 and initiate performing the action at. The first node 11 1 may then, with the first machine learning model (Mi) 501 , predict, in accordance to Action 308, the respective prediction of the one or more indicators of performance of the communications system 100, e.g., the set of KPIs, at the first time period based on the following: i) the action 502 at, ii) the set of KPIs 503 Kt' observed at t, and iii) the received analytic report 504 pi at time t. The first node 11 1 may then, in accordance with Action 309, determine the respective second observed value of the respective one or more indicators of performance in the environment at the first time period Kt+i' . The first node 11 1 may also, according to Action 310, determine the first indication ftl, and in accordance with Action 311 , provide the first indication to the second node 1 12, which may receive it according to Action 405.

[0165] Figure 6 is a schematic diagram depicting a non-limiting example of a method performed by the first node 11 1 and the second node 1 12 during the training phase of the first machine learning model, according to embodiments herein. Also depicted in Figures 6 is an architecture to utilize the local, first machine learning model inside the first node 1 11. In this non-limiting example, the second node 112 is an NWDAF, and the first node 11 1 is a service consumer NF of the NWDAF. As depicted in Figure 6, the first node 11 1 may, in accordance with Action 302 and Action 402, obtain the respective analytic report ptlfrom the second node 1 12, for the respective time period of the plurality of time periods. The first node 1 11 may then, according to Action 303, determine, for each respective time period, the respective action atto be taken on the environment, based on the policy (IT) of the first node 1 11. The first node 1 11 may then, with the first machine learning model 601 (Mi), predict, in accordance to Action 304, the evolution of the one or more indicators of performance, e.g., the set of KPIs Ki+1, based on the following: i) the action 602 at, ii) the set of KPIs 603 Kt' observed at t, and iii) the received analytic report 604 p at time t. The first machine learning model 601 (Mi) may be understood as a new ML model, which may comprise a supervised ML algorithm.

[0166] Figure 7 is a signalling diagram depicting a non-limiting example of a method of training and inference of the local first machine learning model inside the first node 11 1 to produce the closed-loop privacy-preserving feedback, according to embodiments herein. In this non-limiting example, the second node 112 is an NWDAF, and the first node 11 1 is a service consumer NF of the NWDAF. During the training phase, in accordance with Action 301 and Action 401 , the first node 11 1 may request the analytic report from the second node 1 12. The second node 1 12 may provide the requested analytic report p to the first node 1 11 in accordance with Action 402 and Action 302. In accordance with Action 304, the first node 11 1 may train the first machine learning model Mi and observe the evolution of the one or more indicators of performance, e.g., the set of KPIs Kt+i'. These actions may be performed in a loop during the training phase. Once the first machine learning model may reach the desired level of accuracy, the first machine learning model Mi may be deployed. During the inference phase, the first node 11 1 may then, according to Action 305 and Action 403, request a new analytic report, which it may receive according to Action 306 and Action 404. The first node 11 1 may then calculate the feedback using the trained first machine learning model Mi, in accordance with Action 308 and 309. The first node 1 11 mat then send the first indication ftlto the second node 1 12 in accordance with Action 31 1 and Action 405.

[0167] Figure 8 is a signalling diagram depicting a non-limiting example of the a method of Federated Learning for training the first machine learning model inside the first node 11 1 , according to embodiments herein. In this non-limiting example, the second node 112 is an NWDAF, and the first node 11 1 is a service consumer NF of the NWDAF. During the training phase of the local first machine learning model at the first node 1 11 , as described in Figure 7, in accordance with Action 301 and Action 401 , the first node 1 11 may request the analytic report from the second node 1 12. The second node 1 12 may provide the requested analytic report pi to the first node 1 11 in accordance with Action 402 and Action 302. In accordance with Action 304, the first node 1 11 may train the first machine learning model Mi and observe the evolution of the one or more indicators of performance, e.g., the set of KPIs Kt+i' using local data. These actions may be performed in a loop during the training phase. Once the local first machine learning model may reach the desired level of accuracy, the first machine learning model Mi may send the new respective first local machine learning model Mi to the orchestrator 801 with its respective parameters wt'. As seen in Figure 8, the orchestrator 801 may iteratively achieve the averaging of the models received from the different first nodes 11 1 , e.g., NFs, until convergence. Thus, every T periods, any of the first nodes 1 11 , e.g., NFi, may exchange with the orchestrator 801 its local model Mi and then receive the new common model M*.

[0168] Figure 9 is a signalling diagram depicting a non-limiting example of a method of inference of the local first machine learning model inside the first node 1 11 to produce the closed-loop privacy-preserving feedback, according to embodiments herein. In this non-limiting example, the second node 1 12 is an NWDAF, and the first node 11 1 is a service consumer NF of the NWDAF. The first node 1 11 , according to Action 305 and Action 403, may request the analytic report ptlfrom the second node 1 12. The first node 11 1 , according to Action 306 and Action 404, may obtain the analytic report ptlthat the second node 1 12 may provide according to Action 404. The first node 11 1 may then, according to Action 307, determine the action to be taken at the first time period atbased on the obtained analytic report pi and the policy (IT) of the first node 1 11 and initiate performing the action at. The first node 11 1 may then, use the first machine learning model (Mi) to predict, in accordance to Action 308, the respective prediction of the one or more indicators of performance of the communications system 100, e.g., the set of KPIs, at the first time period Ki+1, that is, the system state evolution. The first node 1 11 may then, in accordance with Action 309, determine the respective second observed value of the respective one or more indicators of performance in the environment, that is, the network, at the first time period Kt+i' . The first node 1 11 may also, according to Action 310, determine the first indication ftl, and in accordance with Action 311 , may provide the first indication to the second node 112, which may receive it according to Action 405.

[0169] As a summarized overview of the foregoing, embodiments herein may be understood to provide service consumer NFs of NWDAF, as an enhancement, with a functionality to be able to assess the correctness of an analytics report using an internally trained ML model, the first machine learning mode, that may predict values of the relevant indicators of performance, e.g., KPIs, by including triggered actions.

[0170] Certain embodiments herein may provide one or more of the following technical advantage(s). Embodiments herein may be understood to enable that the first node 111 , an NF, may measure and / or predict, thanks to the new first machine learning model of embodiments herein, the impact of the triggered actions and the received analytic reports on the indicators of performance, e.g., the KPIs, of interest. Any first node 111 , e.g., NFs, may provide feedback to the second node 112, e.g., an NWDAF, that may express the satisfaction of the respective first node 111 with the received respective analytic reports, thereby the quality of the ML model of the second node 112 to produce accurate analytic reports.

[0171] Figure 10 depicts an example of the arrangement that the first node 111 may comprise to perform the method described in Figure 3 and / or Figures 6-11 . The first node 111 may be understood to be for handling the analytic reports. The first node 111 is configured to operate in the communications system 100.

[0172] Several embodiments are comprised herein. It should be noted that the examples herein are not mutually exclusive. One or more embodiments may be combined, where applicable. All possible combinations are not described to simplify the description. Components from one embodiment may be tacitly assumed to be present in another embodiment and it will be obvious to a person skilled in the art how those components may be used in the other exemplary embodiments. The detailed description of some of the following corresponds to the same references provided above, in relation to the actions described for the first node 111 , and will thus not be repeated here. For example, the one or more indicators of performance may be configured to be one or more KPIs.

[0173] The first node 111 is configured to determine, using the first machine-learning model, the respective prediction of the one or more indicators of performance of the communications system 100 at the first time period. The determining is configured to be based on: i) the action taken at the second time period preceding the first time period on the environment of the communications system 100 for which the one or more indicators of performance are configured to be predicted, ii) the respective first observed value of the respective one or more indicators of performance in the environment at the second time period preceding the first time period, and iii) the analytic report configured to be obtained at the second time period from the second node 112 configured to be operating in the communications system 100 on the past or future state of the communications system 100. The first node 111 is also configured to determine the respective second observed value of the respective one or more indicators of performance in the environment at the first time period.

[0174] The first node 111 is further configured to provide the first indication to the second node 112 configured to indicate the difference between the respective prediction and the respective second observed value of the one or more indicators of performance at the first time period as feedback on the analytic report.

[0175] In some embodiments, when providing the first indication to the second node 112, the first node 111 may be configured to refrain from indicating the action configured to be taken on the environment for the first time period.

[0176] In some embodiments, when providing the first indication to the second node 112, the first node 111 may be configured to refrain from indicating the one or more indicators of performance.

[0177] In some embodiments, the first node 111 may be further configured with at least one of the following four configurations.

[0178] In some embodiments, the first node 111 may be configured to send the request for the analytic report to the second node 112.

[0179] In some embodiments, the first node 111 may be configured to obtain the analytic report from the second node 112.

[0180] In some embodiments, the first node 111 may be further configured to determine the action configured to be taken at the second time period based on the analytic report configured to be obtained and the policy of the first node 111 and initiate performing the action.

[0181] In some embodiments, the first node 111 may be further configured to determine the first indication.

[0182] In some embodiments, the determining of the action may be configured to be performed using one of: the set of rules and the second machine-learning model.

[0183] In some embodiments, the first node 111 may be further configured with at least one of the following four configurations.

[0184] In some embodiments, the first node 111 may be further configured to send, to the second node 112, the respective request for the respective analytic report on the respective past or future state of the communications system 100, for the respective time period of the plurality of time periods.

[0185] In some embodiments, the first node 111 may be further configured to obtain the respective analytic report from the second node 112, for the respective time period of the plurality of time periods, responsive to the respective request configured to be sent. In some embodiments, the first node 111 may be further configured to determine, for each respective time period, the respective action to be taken on the environment. The respective action may be configured to be based on the respective analytic report and the policy of the first node 111.

[0186] In some embodiments, the first node 111 may be further configured to train the first machine learning model to predict the respective first prediction of the one or more indicators of performance of the communications system 100 at the respective first time period. The training may be configured to be based on: i) the determined respective action configured to be taken for the respective time period on the environment, ii) the respective observed value of the respective one or more indicators of performance in the environment at the respective second time period preceding the respective time period, and iii) the respective analytic report configured to be obtained from the second node 112.

[0187] In some embodiments, the training may be configured to be performed until the accuracy level may be reached. The determining of the respective prediction may be configured to be performed using the trained first machine learning model.

[0188] In some embodiments, at least one of the following may apply: a) the communications system 100 may be configured to be a 5G system, b) the first node 111 may be configured to be an NF, and c) the second node 112 may be configured to be a an NWDAF.

[0189] The embodiments herein in the first node 111 may be implemented through one or more processors, such as a processing circuitry 1001 in the first node 111 depicted in Figure 10, together with computer program code for performing the functions and actions of the embodiments herein. A processor, as used herein, may be understood to be a hardware component. The program code mentioned above may also be provided as a computer program product, for instance in the form of a data carrier carrying computer program code for performing the embodiments herein when being loaded into the first node 111. One such carrier may be in the form of a CD ROM disc. It is however feasible with other data carriers such as a memory stick. The computer program code may furthermore be provided as pure program code on a server and downloaded to the first node 111.

[0190] The first node 111 may further comprise a memory 1002 comprising one or more memory units. The memory 1002 is arranged to be used to store obtained information, store data, configurations, schedulings, and applications etc. to perform the methods herein when being executed in the first node 111.

[0191] In some embodiments, the first node 111 may receive information from, e.g., the second node 112, the third node, the radio network node 130, the device 140, and / or another structure in the computer system 100, through a receiving port 1003. In some embodiments, the receiving port 1003 may be, for example, connected to one or more antennas in first node 111. In other embodiments, the first node 111 may receive information from another structure in the computer system 100 through the receiving port 1003. Since the receiving port 1003 may be in communication with the processing circuitry 1001 , the receiving port 1003 may then send the received information to the processing circuitry 1001. The receiving port 1003 may also be configured to receive other information.

[0192] The processing circuitry 1001 in the first node 111 may be further configured to transmit or send information to e.g., the second node 112, the third node, the radio network node 130, the device 140, and / or another structure in the computer system 100, through a sending port 1004, which may be in communication with the processing circuitry 1001 , and the memory 1002.

[0193] Those skilled in the art will also appreciate that the units comprised within the first node 111 described above as being configured to perform different actions, may refer to a combination of analog and digital circuits, and / or one or more processors configured with software and / or firmware, e.g., stored in memory, that, when executed by the one or more processors such as the processing circuitry 1001 , perform as described above. One or more of these processors, as well as the other digital hardware, may be included in a single Application-Specific Integrated Circuit (ASIC), or several processors and various digital hardware may be distributed among several separate components, whether individually packaged or assembled into a System-on-a-Chip (SoC).

[0194] The first node 111 may be configured to perform any of the Actions described in relation to Figure 3 and / or Figures 5-9, e.g., by means of the processing circuitry 1001 within the first node 111 , configured to perform any of such actions.

[0195] Also, in some embodiments, different units comprised within the first node 111 may be configured to perform the different actions described above, implemented as one or more applications running on one or more processors such as the processing circuitry 1001.

[0196] Thus, the methods according to the embodiments described herein for the first node 111 may be respectively implemented by means of a computer program 1005 product, comprising instructions, i.e. , software code portions, which, when executed on at least one processing circuitry 1001 , cause the at least one processing circuitry 1001 to carry out the actions described herein, as performed by the first node 111. The computer program 1005 product may be stored on a computer-readable storage medium 1006. The computer- readable storage medium 1006, having stored thereon the computer program 1005, may comprise instructions which, when executed on at least one processing circuitry 1001 , cause the at least one processing circuitry 1001 to carry out the actions described herein, as performed by the first node 111. In some embodiments, the computer-readable storage medium 1006 may be a non-transitory computer-readable storage medium, such as a CD ROM disc, or a memory stick. In other embodiments, the computer program 1005 product may be stored on a carrier containing the computer program 1005 just described, wherein the carrier is one of an electronic signal, optical signal, radio signal, or the computer-readable storage medium 1006, as described above.

[0197] The first node 111 may comprise a communication interface configured to facilitate, or an interface unit to facilitate, communications between the first node 111 and other nodes or devices, e.g., the second node 112, the third node, the radio network node 130, the device 140, and / or another structure in the computer system 100. The interface may, for example, include a transceiver configured to transmit and receive radio signals over an air interface in accordance with a suitable standard.

[0198] In other embodiments, the first node 111 may comprise a radio circuitry 1007, which may comprise e.g., the receiving port 1003 and the sending port 1004.

[0199] The radio circuitry 1007 may be configured to set up and maintain at least a wireless connection with the second node 112, the third node, the radio network node 130, the device 140, and / or another structure in the computer system 100. Circuitry may be understood herein as a hardware component.

[0200] Hence, embodiments herein also relate to the first node 111 operative to operate in the computer system 100. The first node 111 may comprise the processing circuitry 1001 and the memory 1002, said memory 1002 containing instructions executable by said processing circuitry 1001 , whereby the first node 111 is further operative to perform the actions described herein in relation to the first node 111 , e.g., in Figure 3 and / or Figures 5-9.

[0201] Figure 11 depicts an example of the arrangement that the second node 112 may comprise to perform the method described in Figure 4 and / or Figures 5-9. The second node 112 may be understood to be for handling the analytic reports. The second node 112 is configured to operate in the communications system 100.

[0202] Several embodiments are comprised herein. It should be noted that the examples herein are not mutually exclusive. One or more embodiments may be combined, where applicable. All possible combinations are not described to simplify the description. Components from one embodiment may be tacitly assumed to be present in another embodiment and it will be obvious to a person skilled in the art how those components may be used in the other exemplary embodiments. The detailed description of some of the following corresponds to the same references provided above, in relation to the actions described for the second node 112, and will thus not be repeated here. For example, the one or more indicators of performance may be configured to be one or more KPIs.

[0203] The second node 112 is configured to provide the analytic report on the past or future state of the communications system 100 to the first node 111 configured to operate in the communications system 100. The analytic report is configured to have been obtained at the second time period. The second node 112 is further configured to receive the first indication, from the first node 111 , configured to indicate the difference between the respective prediction and the respective second observed value of one or more indicators of performance of the communications system 100 at the first time period as feedback on the analytic report. The respective prediction is configured to be based on the analytic report configured to be provided.

[0204] In some embodiments, the respective prediction may be further configured to be based on: a) the action configured to be taken at the second time period preceding the first time period on the environment of the communications system 100 for which the one or more indicators of performance may be configured to be predicted, and b) the respective first observed value of the respective one or more indicators of performance in the environment at the second time period preceding the first time period.

[0205] In some embodiments, the first indication form the first node 111 may be configured to omit indicating the action configured to be taken on the environment for the first time period.

[0206] In some embodiments, the first indication from the first node 111 may be configured to omit indicating the one or more indicators of performance.

[0207] In some embodiments, the second node 112 may further configured to receive the request for the analytic report from the first node 111 , and the providing of the analytic report may be configured to be responsive to the request configured to be received.

[0208] In some embodiments, the second node 112 may be further configured with at least one of the following two configurations.

[0209] In some embodiments, the second node 112 may further configured to receive the respective request for the respective analytic report from the first node 111 , for the respective time period of the plurality of time periods.

[0210] In some embodiments, the second node 112 may further configured to, provide the respective analytic report to the first node 111 , for the respective time period of the plurality of time periods. The first indication configured to be received may be configured to be based on the output of the first machine learning model to predict the respective first prediction of the one or more indicators of performance of the communications system 100 at the respective first time period. The first machine learning model may be configured to have been trained based on the respective analytic reports configured to be provided.

[0211] In some embodiments, at least one of the following may apply: a) the communications system 100 may be configured to be a 5G system, b) the first node 111 may be configured to be an NF, and c) the second node 112 may be configured to be a an NWDAF.

[0212] The embodiments herein in the second node 112 may be implemented through one or more processors, such as a processing circuitry 1101 in the second node 112 depicted in Figure 11 , together with computer program code for performing the functions and actions of the embodiments herein. A processor, as used herein, may be understood to be a hardware component. The program code mentioned above may also be provided as a computer program product, for instance in the form of a data carrier carrying computer program code for performing the embodiments herein when being loaded into the second node 112. One such carrier may be in the form of a CD ROM disc. It is however feasible with other data carriers such as a memory stick. The computer program code may furthermore be provided as pure program code on a server and downloaded to the second node 112.

[0213] The second node 112 may further comprise a memory 1102 comprising one or more memory units. The memory 1102 is arranged to be used to store obtained information, store data, configurations, schedulings, and applications etc. to perform the methods herein when being executed in the second node 112.

[0214] In some embodiments, the second node 112 may receive information from, e.g., the first node 111 , the third node, the radio network node 110, the device 140, and / or another structure in the computer system 100, through a receiving port 1103. In some embodiments, the receiving port 1103 may be, for example, connected to one or more antennas in second node 112. In other embodiments, the second node 112 may receive information from another structure in the wireless communications network 110 through the receiving port 1103. Since the receiving port 1103 may be in communication with the processing circuitry 1101 , the receiving port 1103 may then send the received information to the processing circuitry 1101. The receiving port 1103 may also be configured to receive other information.

[0215] The processing circuitry 1101 in the second node 112 may be further configured to transmit or send information to e.g., the first node 111 , the third node, the radio network node 110, the device 140, and / or another structure in the computer system 100, through a sending port 1104, which may be in communication with the processing circuitry 1101 , and the memory 1102.

[0216] Those skilled in the art will also appreciate that the units comprised within the second node 112 described above as being configured to perform different actions, may refer to a combination of analog and digital circuits, and / or one or more processors configured with software and / or firmware, e.g., stored in memory, that, when executed by the one or more processors such as the processing circuitry 1101 , perform as described above. One or more of these processors, as well as the other digital hardware, may be included in a single Application-Specific Integrated Circuit (ASIC), or several processors and various digital hardware may be distributed among several separate components, whether individually packaged or assembled into a System-on-a-Chip (SoC).

[0217] The second node 112 may be configured to perform any of the Actions described in relation to Figure 4 and / or Figures 5-9, e.g., by means of the processing circuitry 1101 within the second node 112, configured to perform any of such actions. Also, in some embodiments, different units comprised within the second node 112 may be configured to perform the different actions described above, implemented as one or more applications running on one or more processors such as the processing circuitry 1101.

[0218] Thus, the methods according to the embodiments described herein for the second node 112 may be respectively implemented by means of a computer program 1105 product, comprising instructions, i.e. , software code portions, which, when executed on at least one processing circuitry 1101 , cause the at least one processing circuitry 1101 to carry out the actions described herein, as performed by the second node 112. The computer program 1105 product may be stored on a computer-readable storage medium 1106. The computer- readable storage medium 1106, having stored thereon the computer program 1105, may comprise instructions which, when executed on at least one processing circuitry 1101 , cause the at least one processing circuitry 1101 to carry out the actions described herein, as performed by the second node 112. In some embodiments, the computer-readable storage medium 1106 may be a non-transitory computer-readable storage medium, such as a CD ROM disc, or a memory stick. In other embodiments, the computer program 1105 product may be stored on a carrier containing the computer program 1105 just described, wherein the carrier is one of an electronic signal, optical signal, radio signal, or the computer-readable storage medium 1106, as described above.

[0219] The second node 112 may comprise a communication interface configured to facilitate, or an interface unit to facilitate, communications between the second node 112 and other nodes or devices, e.g., the first node 111 , the third node, the radio network node 110, the device 140, and / or another structure in the computer system 100. The interface may, for example, include a transceiver configured to transmit and receive radio signals over an air interface in accordance with a suitable standard.

[0220] In other embodiments, the second node 112 may comprise a radio circuitry 1107, which may comprise e.g., the receiving port 1103 and the sending port 1104.

[0221] The radio circuitry 1107 may be configured to set up and maintain at least a wireless connection with the first node 111 , the third node, the radio network node 110, the device 140, and / or another structure in the computer system 100. Circuitry may be understood herein as a hardware component.

[0222] Hence, embodiments herein also relate to the second node 112 operative to operate in the wireless communications network 110. The second node 112 may comprise the processing circuitry 1101 and the memory 1102, said memory 1102 containing instructions executable by said processing circuitry 1101 , whereby the second node 112 is further operative to perform the actions described herein in relation to the second node 112, e.g., in Figure 4 and / or Figures 5-9. When using the word "comprise" or “comprising”, it shall be interpreted as non- limiting, i.e., meaning "consist at least of".

[0223] The embodiments herein are not limited to the above-described preferred embodiments. Various alternatives, modifications and equivalents may be used. Therefore, the above embodiments should not be taken as limiting the scope of the invention.

[0224] Generally, all terms used herein are to be interpreted according to their ordinary meaning in the relevant technical field, unless a different meaning is clearly given and / or is implied from the context in which it is used. All references to a / an / the element, apparatus, component, means, step, etc. are to be interpreted openly as referring to at least one instance of the element, apparatus, component, means, step, etc., unless explicitly stated otherwise. The steps of any methods disclosed herein do not have to be performed in the exact order disclosed, unless a step is explicitly described as following or preceding another step and / or where it is implicit that a step must follow or precede another step. Any feature of any of the embodiments disclosed herein may be applied to any other embodiment, wherever appropriate. Likewise, any advantage of any of the embodiments may apply to any other embodiments, and vice versa. Other objectives, features and advantages of the enclosed embodiments will be apparent from the following description.

[0225] As used herein, the expression “at least one of:” followed by a list of alternatives separated by commas, and wherein the last alternative is preceded by the “and” term, may be understood to mean that only one of the list of alternatives may apply, more than one of the list of alternatives may apply or all of the list of alternatives may apply. This expression may be understood to be equivalent to the expression “at least one of:” followed by a list of alternatives separated by commas, and wherein the last alternative is preceded by the “or” term.

[0226] Any of the terms processor and circuitry may be understood herein as a hardware component.

[0227] As used herein, the expression “in some embodiments” has been used to indicate that the features of the embodiment described may be combined with any other embodiment or example disclosed herein.

[0228] As used herein, the expression “in some examples” has been used to indicate that the features of the example described may be combined with any other embodiment or example disclosed herein.

[0229] REFERENCES

[0230] 1 . 3GPP TS 23.288, v18.2.0, 3rd Generation Partnership Project; Technical Specification Group Services and System Aspects; Architecture enhancements for 5G System (5GS) to support network data analytics services. 3GPP TR 23.700-81 v18.0.0, 3rd Generation Partnership Project; Technical Specification Group Services and System Aspects; Study of Enablers for Network Automation for the 5G System (5GS); Phase 3. Yang, Qiang, et al. "Federated machine learning: Concept and applications." ACM Transactions on Intelligent Systems and Technology (TIST) 10.2 (2019): 1 -19.

Claims

CLAIMS:1 . A computer-implemented method, performed by a first node (1 11 ), for handling analytic reports, the first node (11 1 ) operating in a communications system (100), the method comprising:- determining (308), using a first machine-learning model, a respective prediction of one or more indicators of performance of the communications system (100) at a first time period, the determining (308) being based on: i. an action taken at a second time period preceding the first time period on an environment of the communications system (100) for which the one or more indicators of performance are predicted, ii. a respective first observed value of the respective one or more indicators of performance in the environment at the second time period preceding the first time period, and iii. an analytic report obtained at the second time period from a second node (1 12) operating in the communications system (100) on a past or future state of the communications system (100),- determining (309) a respective second observed value of the respective one or more indicators of performance in the environment at the first time period, and- providing (311 ) a first indication to the second node (112) indicating a difference between the respective prediction and respective second observed value of the one or more indicators of performance at the first time period as feedback on the analytic report.

2. The method according to claim 1 , wherein when providing (311 ) the first indication to the second node (1 12), the first node (111 ) refrains from indicating the action taken on the environment for the first time period.

3. The method according to any of claims 1 -2, wherein when providing (31 1 ) the first indication to the second node (1 12), the first node (11 1 ) refrains from indicating the one or more indicators of performance.

4. The method according to any of claims 1 -3, further comprising at least one of:- sending (305) a request for the analytic report to the second node (112),- obtaining (306) the analytic report from the second node (1 12),- determining (307) the action to be taken at the second time period based on the obtained analytic report and a policy of the first node (111) and initiating performing the action, and- determining (310) the first indication.

5. The method according to claim 4, wherein the determining (307) of the action is performed using one of: a set of rules and a second machine-learning model.

6. The method according to claim any of claims 1 -5, further comprising at least one of:- sending (301 ), to the second node (112), a respective request for a respective analytic report on a respective past or future state of the communications system (100), for a respective time period of a plurality of time periods,- obtaining (302) the respective analytic report from the second node (112), for the respective time period of the plurality of time periods, responsive to the sent respective request,- determining (303), for each respective time period, a respective action to be taken on the environment, the respective action being based on the respective analytic report and a policy of the first node (111 ),- training (304) the first machine learning model to predict a respective first prediction of the one or more indicators of performance of the communications system (100) at the respective first time period, the training (304) being based on: i. the determined respective action taken for the respective time period on the environment, ii. a respective observed value of the respective one or more indicators of performance in the environment at a respective second time period preceding the respective time period, and iii. the respective analytic report obtained from the second node (112), wherein the training (304) is performed until an accuracy level is reached, and wherein the determining (308) of the respective prediction is performed using the trained first machine learning model.

7. The method according to any of claims 1 -6, wherein at least one of: a. the communications system (100) is a Fifth Generation, 5G, system, b. the first node (111) is a Network Function, NF, and c. the second node (112) is a Network Data Analytics Function, NWDAF.

8. A computer-implemented method, performed by a second node (112), for handling analytic reports, the second node (112) operating in a communications system (100), the method comprising:- providing (404) an analytic report on a past or future state of the communications system (100) to a first node (111) operating in the communications system (100), wherein the analytic report has been obtained at a second time period, and- receiving (405) a first indication, from the first node (111), indicating a difference between a respective prediction and a respective second observed value of one or more indicators of performance of the communications system (100) at a first time period as feedback on the analytic report, wherein the respective prediction is based on the provided analytic report.

9. The method according to claim 8, wherein the respective prediction is further based on: a. an action taken at the second time period preceding the first time period on an environment of the communications system (100) for which the one or more indicators of performance are predicted, and b. a respective first observed value of the respective one or more indicators of performance in the environment at the second time period preceding the first time period.

10. The method according to any of claims 8-9, further comprising:- receiving (403) a request for the analytic report from the first node (111), and wherein the providing (404) of the analytic report is responsive to the received request.11 . The method according to claim any of claims 8-10, further comprising at least one of:- receiving (401 ) a respective request for a respective analytic report from the first node (111 ), for a respective time period of a plurality of time periods,- providing (402) a respective analytic report to the first node (111 ), for a respective time period of a plurality of time periods, and wherein the received first indication is based on an output of a first machine learning model to predict a respective first prediction of the one or more indicators of performance of the communications system (100) at a respective first time period, the first machine learning model having been trained based on the provided respective analytic reports.

12. A first node (1 11 ), for handling analytic reports, the first node (1 11 ) being configured to operate in a communications system (100), the first node (1 11 ) being further configured to:- determine, using a first machine-learning model, a respective prediction of one or more indicators of performance of the communications system (100) at a first time period, the determining being configured to be based on: i. an action taken at a second time period preceding the first time period on an environment of the communications system (100) for which the one or more indicators of performance are configured to be predicted, ii. a respective first observed value of the respective one or more indicators of performance in the environment at the second time period preceding the first time period, and iii. an analytic report configured to be obtained at the second time period from a second node (112) configured to be operating in the communications system (100) on a past or future state of the communications system (100),- determine a respective second observed value of the respective one or more indicators of performance in the environment at the first time period, and- provide a first indication to the second node (112) configured to indicate a difference between the respective prediction and respective second observed value of the one or more indicators of performance at the first time period as feedback on the analytic report.

13. The first node (111 ) according to claim 12, being further configured to at least one of:- send a request for the analytic report to the second node (1 12),- obtain the analytic report from the second node (112),- determine the action configured to be taken at the second time period based on the analytic report configured to be obtained and a policy of the first node (1 11 ) and initiate performing the action, and- determine the first indication.

14. The first node (111 ) according to claim any of claims 12-13, being further configured to at least one of: send, to the second node (1 12), a respective request for a respective analytic report on a respective past or future state of the communications system (100), for a respective time period of a plurality of time periods,- obtain the respective analytic report from the second node (1 12), for the respective time period of the plurality of time periods, responsive to the respective request configured to be sent,- determine, for each respective time period, a respective action to be taken on the environment, the respective action being configured to be based on the respective analytic report and a policy of the of the first node (111 ),- train the first machine learning model to predict a respective first prediction of the one or more indicators of performance of the communications system (100) at the respective first time period, the training being configured to be based on: a. the determined respective action configured to be taken for the respective time period on the environment, b. a respective observed value of the respective one or more indicators of performance in the environment at a respective second time period preceding the respective time period, and c. the respective analytic report configured to be obtained from the second node (112), wherein the training is configured to be performed until an accuracy level is reached, and wherein the determining of the respective prediction is configured to be performed using the trained first machine learning model.

15. A second node (1 12), for handling analytic reports, the second node (1 12) being configured to operate in a communications system (100), the second node (1 12) being further configured to:- provide an analytic report on a past or future state of the communications system (100) to a first node (1 11 ) configured to operate in the communications system (100), wherein the analytic report is configured to have been obtained at a second time period, and- receive a first indication, from the first node (11 1 ), configured to indicate a difference between a respective prediction and a respective second observed value of one or more indicators of performance of the communications system (100) at a first time period as feedback on the analytic report, wherein the respective prediction is configured to be based on the analytic report configured to be provided.

16. The second node (1 12) according to claim 15, being further configured to:- receive a request for the analytic report from the first node (111 ), and wherein the providing of the analytic report is configured to be responsive to the request configured to be received.

17. The second node (112) according to claim any of claims 15-16, being further configured to at least one of:- receive a respective request for a respective analytic report from the first node (111), for a respective time period of a plurality of time periods,- provide a respective analytic report to the first node (111 ), for a respective time period of a plurality of time periods, and wherein the first indication configured to be received is configured to be based on an output of a first machine learning model to predict a respective first prediction of the one or more indicators of performance of the communications system (100) at a respective first time period, the first machine learning model being configured to have been trained based on the respective analytic reports configured to be provided.

Images