Generating network analytics using cluster-associated generic models

Abstract

Embodiments include computer-implemented methods for generating analytics related to performance of network entities of a communication network. Such methods include receiving, from a consumer network function (NF) of the communication network, a request for an analytic related to performance of one or more network entities of the communication network. Such methods include, for each of the one or more network entities, assigning the network entity to one of a plurality of clusters and, using a generic model associated with the assigned cluster, determining a coarse version of the analytic for the network entity. Such methods include sending the coarse versions of the analytic, for the respective network entities, to the consumer NF in response to the request. In some embodiments, such methods also include initiating data collection for the respective network entities, such that the coarse versions of the analytic are sent while the data collection is ongoing.

Description

[0001] GENERATING NETWORK ANALYTICS USING CLUSTER-ASSOCIATED GENERIC MODELS

[0002] TECHNICAL FIELD

[0003] The present disclosure relates generally to communication networks and more specifically to techniques for expediting a response to a request for a performance-related analytic that may otherwise be delayed while collecting data necessary to train a model used to generate the analytic, specifically by generating prompt responses using previously trained generic models.

[0004] BACKGROUND

[0005] The fifth generation (5G) of cellular systems, also referred to as New Radio (NR), was initially standardized 3GPP Rel-15 and continues to evolve in subsequent releases. NR is developed for maximum flexibility to support a variety of different use cases including enhanced mobile broadband (eMBB), machine type communications (MTC), ultra-reliable low latency communications (URLLC), side-link device-to-device (D2D), and several other use cases. 5G / NR technology shares many similarities with fourth-generation LTE.

[0006] At a high level, the 5G System (5GS) consists of a radio access network (RAN) and a Core Network (CN). The RAN provides UEs connectivity to the CN, e.g., via base stations such as gNBs or ng-eNBs. As described in more detail below, the CN includes a variety of Network Functions (NF) that provide a range of different functionalities such as session management, connection management, charging, authentication, etc.

[0007] In 5G networks, traditional peer-to-peer interfaces and protocols found in earlier- generation networks are modified and / or replaced by a Service Based Architecture (SBA) in which Network Functions (NFs) provide one or more services to one or more service consumers. This can be done, for example, by Hyper Text Transfer Protocol / Representational State Transfer (HTTP / REST) application programming interfaces (APIs). In general, the various services are self-contained functionalities that can be changed and modified in an isolated manner without affecting other services. Furthermore, the services are composed of various “service operations,” which are more granular divisions of the overall service functionality.

[0008] The ever increasing complexity of communication networks, including 5G networks, drives the evolution of analytics systems that support operation, optimization, and planning of these networks. This includes detecting and addressing sudden, undesired changes in network operation and / or performance (e.g., failures). These analytics systems, in turn, require collecting and processing enormous amounts of data.

[0009] Advanced analytics systems, such as Ericsson Expert Analytics (EEA), are based on collecting and correlating elementary network events from different network domains, such as core, radio, and transport networks. Such analytics systems calculate user- and session-level E2E service quality metrics (S-KPIs) as well as radio and network resource metrics (R-KPIs) that characterize the radio environment or network operation at user and session level. These types of solutions are suitable for session-based troubleshooting and analysis of network issues.

[0010] Event-based analytics require real-time collection and correlation of node and protocol events from different RAN and CN nodes, probing signaling interfaces, and sampling of userplane traffic. Additionally, event-based analytics require an advanced database, a rule engine, and a “big data” analytics platform. It is expected that 5G networks will serve a higher number of UEs and support a wider variety of service types than 4G and previous-generation networks. This will significantly increase the incoming event rate and type to be processed by analytics systems to support network QoS and end-user quality-of-experience (QoE)

[0011] A 5G NF of particular interest to the present disclosure is the Network Data Analytics Function (NWDAF). NWDAF can provide network analytics information (e.g., statistical information of past events and / or predictive information) to other NFs, and can collect network- related data from various sources such as RAN nodes CN NFs, and UEs. The NWDAF can collect this data continuously or on-demand. NF consumers can request analytics information from NWDAF via two generic interfaces. The Analytics Info interface provides on-demand information using request-response signaling. In contrast, the Event Subscription interface enables NF consumers to subscribe to analytics data, based on which they receive notifications when available.. 3GPP TS 29.520 (vl8.5.0) specifies NWDAF functionality in greater detail, including a list of supported analytics identifiers (IDs) that can be requested or subscribed to.

[0012] NWDAF analytics may be computed using Machine learning (ME), which is a type of artificial intelligence (Al) that imitates the way that humans learn, gradually improving accuracy as more data becomes available. ML algorithms build models based on sample (or “training”) data, with the models being used subsequently to make predictions, estimates, or decisions. ML algorithms can be used in a wide variety of use cases or applications (e.g., medicine, email filtering, speech recognition, etc.) in which it is difficult or unfeasible to develop conventional algorithms to perform the needed tasks. 3GPP TR 23.791 (vl6.2.0) describes example use cases for ML in 5G networks, but other network-related use cases may be defined and supported by NWDAF.

[0013] SUMMARY

[0014] NWDAF can provide prompt responses to analytics requests based on historical data collected. However, this capability requires continuous monitoring of all network data sources since it is not known in advance which data and / or analytics will be requested by any consumer at any time. The hardware processing requirements of this continuous monitoring is very high, particularly for user plane (UP) monitoring and collection of RAN measurement reports.

[0015] Alternately, analytics measurements and data collection may be initiated or triggered by a received analytics request. Depending on the requested data and / or analytics, it may take a significant amount of time to collect the required data, such as seconds, hours, or even days. This approach is time consuming and introduces response delays that are very undesirable for requesting consumers.

[0016] Embodiments of the present disclosure address these and other problems, issues, and / or difficulties, thereby facilitating prompt and / or timely responses by NWDAF (or other analytics system) to analytics requests by consumers.

[0017] Some embodiments include methods (e.g., procedures) for generating analytics related to performance of network entities of a communication network in a communication network (e.g., 5G network).

[0018] These exemplary methods include receiving, from a consumer network function (NF) of the communication network, a request for an analytic related to performance of one or more network entities of the communication network. These exemplary methods also include the following operations for each of the one or more network entities:

[0019] • assigning the network entity to one of a plurality of clusters, and

[0020] • using a generic model associated with the assigned cluster, determining a coarse version of the analytic for the network entity; and

[0021] These exemplary methods also include sending the coarse versions of the analytic, for the respective network entities, to the consumer NF in response to the request.

[0022] In some embodiments, these exemplary methods also include, for each of the one or more network entities, determining whether an entity-specific model usable to determine a refined version of the requested analytic exists. Assigning the network entity to one of the plurality of clusters is based on determining that the entity-specific model does not exist.

[0023] In some embodiments, these exemplary methods also include, in response to assigning the network entity to one of the clusters, initiating data collection for the network entity. In some of these embodiments, the coarse versions of the analytic are sent while the data collection is ongoing.

[0024] In some of these embodiments, initiating data collection for the network entity is performed selectively based on a data collection resource limit. In some variants of these embodiments, the data collection resource limit is based on a maximum number of network entities for which data collection can be performed concurrently. In some variants of these embodiments, initiating data collection for the network entity includes the following operations: • determining whether data collection for the network entity will cause data collection resources to exceed the data collection resource limit;

[0025] • when it is determined that data collection for the network entity will cause data collection resources to exceed the data collection resource limit, determining whether the network entity should be prioritized over at least one other network entity for which data collection is ongoing; and

[0026] • initiating data collection for the network entity based on determining that data collection for the network entity will not cause data collection resources to exceed the data collection resource limit or that the network entity should be prioritized.

[0027] In some further variants, initiating data collection for the network entity also includes, when it is determined that the network entity should be prioritized, terminating data collection for a lowest-priority one of the at least one other network entity. For example, the lowest-priority other network entity is associated with one or more of the following: a least recent analytic request, a least complete data collection, a least complete training of an entity- specific model, and an associated interest metric that is lowest or less than a first threshold.

[0028] In some of these embodiments, these exemplary methods also include the following operations:

[0029] • collecting training data from a plurality of network entities of a second communication network;

[0030] • determining the plurality of clusters based on the collected training data and characteristics of the network entities of the second communication network; and

[0031] • for each of the plurality of clusters, training the associated generic model based on the training data collected from the network entities of the second communication network.

[0032] In some variants of these embodiments, determining the plurality of clusters includes the following operations:

[0033] • selecting one more characteristics of the network entities of the second communication network that are most relevant for analytics of interest to consumer NFs of the communication network; and

[0034] • assigning the network entities of the second communication network to the plurality of clusters based on the selected characteristics,

[0035] In some variants of these embodiments, the data collected from the communication network and the training data collected from the second communication network differ according to one or more of the following characteristics:

[0036] • resolution or frequency, with the data collected from the communication network having a higher resolution or frequency; and • numbers and / or types of parameters collected, with the data collected from the communication network having a greater number and / or more types of parameters collected.

[0037] Other embodiments include network analytics systems (e.g., NWDAF) configured to perform operations corresponding to any of the exemplary methods described herein. Other embodiments include non-transitory, computer-readable media storing program instructions that, when executed by processing circuitry, configure such network analytics systems to perform operations corresponding to any of the exemplary methods described herein.

[0038] These and other embodiments described herein may provide various benefits and / or advantages. For example, embodiments may be applied to a wide variety of NWDAF analytics that require estimation, prediction, or a combination thereof. By using generic models trained on easily observable (e.g., network- wide) data at for groups (or clusters) of network entities, embodiments may reduce or eliminate the need to collect entity-level data and train entity-level models in advance. Since entity-level data is only collected as needed, NWDAF resources needed for data collection and model training may be significantly reduced. Moreover, the described “spotlight” data collection includes a fully representative selection of network entities according to resource limitations. In addition, embodiments may facilitate prompt NWDAF responses to analytics requests based on the generic models, with more accurate responses based on entity-specific models provided later as needed

[0039] These and other objects, features, and advantages of embodiments of the present disclosure will become apparent upon reading the following Detailed Description in view of the Drawings briefly described below.

[0040] BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is a high-level block diagram of an exemplary 5G / NR network architecture.

[0042] Figure 2 shows an exemplary non-roaming architecture of a 5G network with servicebased interfaces and various 3GPP-defined NFs.

[0043] Figure 3 shows an exemplary multi-domain network comprising a RAN, a packet-based core network (CN), and an IP Multimedia Subsystem (IMS).

[0044] Figure 4 shows a high-level view of an exemplary system according to some embodiments of the present disclosure.

[0045] Figure 5 illustrates the creation and use of generic ML models, according to some embodiments of the present disclosure.

[0046] Figure 6 shows a procedure for generic ML model training, according to some embodiments of the present disclosure. Figure 7 shows a procedure for determining a requested analytic during operation in an operational (e.g., live) network, according to some embodiments of the present disclosure.

[0047] Figure 8 illustrates end-user quality of experience (QoE) estimation based on a generic model and a service specific model, according to some embodiments of the present disclosure.

[0048] Figure 9 is a cumulative distribution function (CDF) plot that compares QoE estimation accuracy using the generic and service specific models shown in Figure 8.

[0049] Figure 10 illustrates estimation of cell resource utilization based on a generic model and a service specific model, according to some embodiments of the present disclosure.

[0050] Figures 11A-B show an exemplary method (e.g., procedure) for generating analytics related to performance of network entities of a communication network, according to various embodiments of the present disclosure.

[0051] Figure 12 shows a communication system according to various embodiments of the present disclosure.

[0052] Figure 13 shows a network node according to various embodiments of the present disclosure.

[0053] Figure 14 is a block diagram of a virtualization environment in which functions implemented by some embodiments of the present disclosure may be virtualized.

[0054] DETAILED DESCRIPTION

[0055] Some of the embodiments contemplated herein will now be described more fully with reference to the accompanying drawings. Other embodiments, however, are contained within the scope of the subject matter disclosed herein, the disclosed subject matter should not be construed as limited to only the embodiments set forth herein; rather, these embodiments are provided by way of example to convey the scope of the subject matter to those skilled in the art.

[0056] In general, all terms used herein are to be interpreted according to their ordinary meaning to a person of ordinary skill in the relevant technical field, unless a different meaning is expressly defined and / or implied from the context of use. All references to a / an / the element, apparatus, component, means, step, etc. are to be interpreted openly as referring to at least one instance of the element, apparatus, component, means, step, etc. , unless explicitly stated otherwise or clearly implied from the context of use. The operations of any methods and / or procedures disclosed herein do not have to be performed in the exact order disclosed, unless an operation is explicitly described as following or preceding another operation and / or where it is implicit that an operation must follow or precede another operation. Any feature of any embodiment disclosed herein can apply to any other disclosed embodiment, as appropriate. Likewise, any advantage of any embodiment described herein can apply to any other disclosed embodiment, as appropriate. Note that the description given herein focuses on a 3GPP cellular communications system and, as such, 3GPP terminology or terminology similar to 3GPP terminology is oftentimes used. However, the concepts disclosed herein are not limited to a 3GPP system and can be applied to any communication system that may benefit from them.

[0057] Figure 1 illustrates a high-level view of an exemplary 5G network architecture, including a Next Generation Radio Access Network (NG-RAN, 199) and a 5G Core (5GC, 198). The NG- RAN can include one or more gNodeB’s (gNBs) connected to the 5GC via one or more NG interfaces, such as gNBs (100, 150) connected via respective interfaces (102, 152). More specifically, the gNBs can be connected to one or more Access and Mobility Management Functions (AMFs) in the 5GC via respective NG-C interfaces and to one or more User Plane Functions (UPFs) in 5GC via respective NG-U interfaces. The 5GC can include various other network functions (NFs), such as Session Management Function(s) (SMF).

[0058] In addition, the gNBs can be connected to each other via one or more Xn interfaces, such as Xn interface (140) between gNBs (100, 150). The radio technology for the NG-RAN is often referred to as “New Radio” (NR). With respect to the NR interface to UEs, each of the gNBs can support frequency division duplexing (FDD), time division duplexing (TDD), or a combination thereof. Each of the gNBs can serve a geographic coverage area including one or more cells and, in some cases, can also use various directional beams to provide coverage in the respective cells.

[0059] NG RAN logical nodes (e.g., gNB 100) include a Central Unit (CU or gNB-CU, e.g., 110) and one or more Distributed Units (DU or gNB-DU, e.g., 120, 130). CUs are logical nodes that host higher-layer protocols and perform various gNB functions such controlling the operation of DUs. DUs are decentralized logical nodes that host lower layer protocols and can include, depending on the functional split option, various subsets of the gNB functions. Each CU and DU can include various circuitry needed to perform their respective functions, including processing circuitry, communication interface circuitry e.g., transceivers), and power supply circuitry.

[0060] A gNB-CU connects to one or more gNB-DUs over respective Fl logical interfaces (e.g., 122 and 132 shown in Figure 1). However, each gNB-DU can be connected to only one gNB-CU. The gNB-CU and its connected gNB-DU(s) are only visible to other gNBs and the 5GC as a gNB. In other words, the Fl interface is not visible beyond gNB-CU.

[0061] As briefly mentioned above, in 5G networks, traditional peer-to-peer interfaces and protocols found in earlier- generation networks are modified and / or replaced by a Service Based Architecture (SB A) in which Network Functions (NFs) provide one or more services to one or more service consumers. Furthermore, the services are composed of various “service operations,” which are more granular divisions of the overall service functionality. Figure 2 shows an exemplary non-roaming architecture of a 5G network (200) with service-based interfaces and various 3GPP-defined NFs. These include the following NFs, with additional details provided for those most relevant to the present disclosure:

[0062] • Application Function (AF, with Naf interface) - interacts with 5GC to provision information to the network operator and to subscribe to certain events happening in operator's network. An AF offers applications for which service is delivered in a different layer (i.e., transport layer) than the one in which the service has been requested (i.e., signaling layer), the control of flow resources according to what has been negotiated with the network. An AF communicates dynamic session information to PCF (via N5 interface), including description of media to be delivered by transport layer.

[0063] • Policy Control Function (PCF, with Npcf interface) - supports unified policy framework to govern the network behavior, via providing PCC rules (e.g., on the treatment of each service data flow that is under PCC control) to the SMF via the N7 reference point. PCF provides policy control decisions and flow based charging control, including service data flow detection, gating, QoS, and flow-based charging (except credit management) towards the SMF. The PCF receives session and media related information from the AF and informs the AF of traffic (or user) plane events.

[0064] • User Plane Function (UPF) supports handling of user plane traffic based on the rules received from SMF, including packet inspection and different enforcement actions (e.g., event detection and reporting). UPFs communicate with the RAN (e.g., NG-RNA) via the N3 reference point, with SMFs (discussed below) via the N4 reference point, and with an external packet data network (PDN) via the N6 reference point. The N9 reference point is for communication between two UPFs.

[0065] • Session Management Function (SMF, with Nsmf interface) interacts with the decoupled traffic (or user) plane, including creating, updating, and removing Protocol Data Unit (PDU) sessions and managing session context with the User Plane Function (UPF), e.g., for event reporting. For example, SMF performs data flow detection (based on filter definitions included in PCC rules), online and offline charging interactions, and policy enforcement.

[0066] • Charging Function (CHF, with Nchf interface) is responsible for converged online charging and offline charging functionalities. It provides quota management (for online charging), re-authorization triggers, rating conditions, etc. and is notified about usage reports from the SMF. Quota management involves granting a specific number of units (e.g., bytes, seconds) for a service. CHF also interacts with billing systems. Access and Mobility Management Function (AMF, with Namf interface) - terminates the RAN CP interface and handles all mobility and connection management of UEs (similar to MME in EPC). AMFs communicate with UEs via the N1 reference point, with SMFs via the Ni l reference point, and with RAN (e.g., NG-RAN) via the N2 reference point.

[0067] • Network Exposure Function (NEF, with Nnef interface) - acts as the entry point into operator's network, by securely exposing to AFs the network capabilities and events provided by 3GPP NFs and by providing ways for the AF to securely provide information to 3GPP network. For example, NEF provides a service that allows an AF to provision specific subscription data (e.g., expected UE behavior) for various UEs. In general, NEF provides services similar to services provided by SCEF in EPC.

[0068] • Network Repository Function (NRF, with Nnrf interface) - provides service registration and discovery, enabling NFs to identify appropriate services available from other NFs.

[0069] • Network Slice Selection Function (NSSF, with Nnssf interface) - enables other NFs to identify a network slice instance that is appropriate for a UE’s desired service. A “network slice” is a logical partition of a 5G network that provides specific network capabilities and characteristics, e.g., in support of a particular service. A “network slice instance” is a set of NF instances and the required network resources (e.g., compute, storage, communication) that provide a network slice’s capabilities and characteristics.

[0070] • Authentication Server Function (AUSF, with Nausf interface) - based in a user’s home network (HPLMN), it performs user authentication and computes security key materials for various purposes.

[0071] • Network Data Analytics Function (NWDAF, 210, with Nnwdaf interface) - interacts with other NFs to collect relevant data and provides network analytics information (e.g., statistical information of past events and / or predictive information) to other NFs.

[0072] • Location Management Function (LMF) with Nlmf interface - supports various functions related to determination of UE locations, including location determination for a UE and obtaining any of the following: DL location measurements or a location estimate from the UE; UL location measurements from the NG RAN; and non-UE associated assistance data from the NG RAN.

[0073] • Unified Data Management function (UDM, with Nudm interface) - supports generation of 3GPP authentication credentials, user identification handling, access authorization based on subscription data, and other subscriber-related functions. To provide this functionality, the UDM uses subscription data (including authentication data) stored in the 5GC unified data repository (UDR). UDR also supports storage and retrieval of policy data by the PCF, as well as storage and retrieval of application data by NEF. The terms “UDM” and “UDM function” are used interchangeably herein.

[0074] IP Multimedia Subsystem (IMS) is an architectural framework for delivering multimedia services to wireless devices based on these Internet-centric protocols. IMS was originally specified by 3rd Generation Partnership Project (3GPP) in Release 5 (Rel-5) as a technology for evolving mobile networks beyond GSM, e.g., for delivering Internet services over GPRS. IMS has evolved in subsequent releases to support other access networks and a wide range of services and applications.

[0075] At a high-level, the functionality of the IMS network can be sub-divided into two types: control and media, and application enablers. The control functionality comprises Call Session Control Function (CSCF) and Home Subscriber Server (HSS). The CSCF is used for session control for devices and applications that are using the IMS network. Session control includes the secure routing of the session initiation protocol (SIP) messages, subsequent monitoring of SIP sessions, and communicating with a policy architecture to support media authorization. CSCF functionality can also be divided into Proxy CSCF (P-CSCF), Serving CSCF (S-CSCF), and Interrogating CSCF (I-CSCF).

[0076] CSCF also interacts with the HSS, which is the master database containing user and subscriber information to support the network entities handling calls and sessions. For example, HSS provides functions such as identification handling, access authorization, authentication, mobility management (e.g., which session control entity is serving the user), session establishment support, service provisioning support, and service authorization support.

[0077] A Media Resource Function (MRF) can provide media services in a user’ s home network and can manage and process media streams such as voice, video, speech-to-text, and real-time transcoding of multimedia data. In general, a WebRTC Gateway allows native- and browser-based devices to access services in the network securely.

[0078] Event-based analytics require real-time collection and correlation of node and protocol events from different RAN and CN nodes, probing signaling interfaces, and sampling of userplane traffic. Additionally, event-based analytics require an advanced database, a rule engine, and a “big data” analytics platform. It is expected that 5G networks will serve a higher number of UEs and support a wider variety of service types than 4G and previous-generation networks. This will significantly increase the incoming event rate and type to be processed by analytics systems to support network QoS and end-user quality-of-experience (QoE).

[0079] As briefly mentioned above, NWDAF can provide network analytics information (e.g., statistical information of past events and / or predictive information) to other NFs, and can collect network-related data from various sources such as RAN nodes CN NFs, and UEs. NWDAF can collect this data continuously or on-demand.

[0080] Figure 3 shows an exemplary multi-domain network comprising a RAN, a packet-based CN, and an IMS. As shown in Figure 3, the RAN includes eNBs that provide the FTE-Uu radio interface and gNBs that provide the NR-Uu interface to UEs. The CN includes SMF, AMF, and UPF in 5GC discussed above, as well as mobility management entity (MME), serving gateway (SGW), and packet gateway (PGW) that are part of the Evolved Packet Core (EPC) associated with LTE networks. The UPF connects to the IMS via the N6 interface, such that IMS in Figure 3 is an instance of the PDN shown in Figure 2.

[0081] Figure 3 also shows various “tapping points” where data can be collected from the three domains of the network, e.g., by NWDAF. For example, node events (e.g., PM counters) can be collected from eNBs, gNBs, AMF, SMF, UPF, MME, and PGW. Eikewise, interface events can be collected from S5-U (user), S5-C (control), Sl-U, and S5-U interfaces in CN as well as from Mw interface between P-CSCF and IS-CSCSF in IMS. In addition to detecting events and / or conditions at the individual nodes and / or interfaces, NWDAF may also combine information collected from the multiple domains to determine “user experience” analytics that represent performance experienced by an end user for a specific service.

[0082] NF consumers can request analytics information from NWDAF via two generic interfaces. The Analytics Info interface provides on-demand information using request-response signaling. In contrast, the Event Subscription interface enables NF consumers to subscribe to analytics data, based on which they receive notifications when available.. 3GPP TS 29.520 (vl8.5.0) specifies NWDAF functionality in greater detail, including a list of supported analytics identifiers (IDs) that can be requested or subscribed to.

[0083] NWDAF analytics may be derived using Machine learning (ML), which is a type of artificial intelligence (Al) that focuses on the use of data and algorithms to imitate the way that humans learn, gradually improving accuracy as more data becomes available. ML algorithms build models based on sample (or “training”) data, with the models being used subsequently to make predictions or decisions. ML algorithms can be used in a wide variety of use cases or applications (e.g., medicine, email filtering, speech recognition, etc.) in which it is difficult or unfeasible to develop conventional algorithms to perform the needed tasks.

[0084] 3GPP TR 23.791 (vl 6.2.0) describes example use cases for ML in 5G networks, but other network-related use cases may be defined and supported by NWDAF. According to 3GPP TS 29.520 (vl8.5.0), NWDAF may implement ML models for at least the following types of analytics events:

[0085] • network slice load level information; • network slice instance load level information;

[0086] • service experience;

[0087] • NF load;

[0088] • network performance;

[0089] • abnormal UE behavior;

[0090] • UE mobility;

[0091] • UE communication;

[0092] • abnormal UE behavior;

[0093] • user data congestion;

[0094] • quality-of-service (QoS) sustainability;

[0095] • dispersion;

[0096] • SMF congestion control experience;

[0097] • redundant transmission experience;

[0098] • WLAN performance;

[0099] • DN performance; and

[0100] • E2E data volume transfer time.

[0101] Based on these ML models, the NWDAF may determine actual current states, estimates of current states (e.g., that are not directly measurable), predictions about future state, and actual or predicted abnormal behavior. Based on these outputs, a consumer NF (e.g., PCF) can make intelligent decisions about network operation, mitigate network performance degradation, improve user quality of experience (QoE), and / or reduce operational costs.

[0102] With sufficient resources, NWDAF can provide prompt responses to analytics requests based on historical data collected. However, this capability requires continuous monitoring of all network data sources since it is not known in advance which data and / or analytics will be requested by any given consumer. The hardware processing requirements of this continuous monitoring is very high, particularly for UP monitoring and collection of RAN measurement reports.

[0103] Alternately, analytics measurements and data collection may be initiated or triggered by a received analytics request. Depending on the requested data and / or analytics, it may take a significant amount of time for NWDAF to collect the required data, such as seconds, hours, or even days. This approach is time consuming and introduces response delays that are very undesirable for requesting consumers.

[0104] Embodiments of the present disclosure address these and other problems, issues, and / or difficulties by novel, flexible, and efficient techniques for answering requests for NWDAF analytics IDs that require estimation and / or prediction, but data needed for the estimates and / or predictions is not available when a request is received. Even if data collection requires some time to answer the requests completely, embodiments facilitate prompt and / or timely response based on more generic estimated or predicted values associated with a “cluster” that includes the specific network entities (e.g., cells, base stations, NFs) that are subjects of the request.

[0105] Embodiments include procedures that involve first operations performed based on a test network and second operations performed based on an operational network (e.g., commercial network). The first operations include classifying network entities (e.g., cells, base stations, NFs) in clusters based on parameters relevant to the analytics use case, such as cell type, size, frequency, radio access technology (RAT), location, traffic mixture, traffic load, etc. Subsequently, different ML models are trained for each cluster (for estimation, prediction, or both) using labelled training data is available from a test network. These ML models are specific to each cluster but are “generic” in the sense that the apply to all network entities in a cluster. At this stage, there is no need to train ML models specific to each network entity.

[0106] In the second operations, when the NWDAF receives an analytics request for a network entity or for a group of network entities, which requires estimation and / or prediction to provide a response, NWDAF determines to which cluster the network entities (or group) belong. Based on this determination, the NWDAF uses the cluster-specific ML model for the required estimation and / or prediction, which the NWDAF is able to promptly provide in a response to the request.

[0107] Concurrently, the NWDAF starts collecting data required for more precise and / or specific estimations and / predictions in accordance with the request. The NWDAF identifies a “spotlight” that includes a set of network entities for which data collection is enabled. The spotlight has a predefined capacity, such that detailed data can be collected only for a limited number of network entities according to a collection resource limit. Network entities can be added to and removed from the spotlight using different strategies.

[0108] After the necessary data has been collected from the spotlight, the NWDAF applies entityspecific ML model(s) for more accurate estimations and / or predictions in accordance with the request. When outputs of the entity-specific ML models are available, they are used to check the accuracy of the corresponding outputs of the cluster-specific ML models, previously provided to the requestor. If needed, a second response including the more accurate estimations and / or predictions is sent to the consumer. Upon receiving the more accurate estimations and / or predictions, the consumer may determine to update, modify, cancel, and / or retrieve previous actions or decisions based on the less accurate estimations and / or predictions that were initially received. In some embodiments, the more accurate estimations and / or predictions output by the entity-specific ML models are used as labels for training and / or updating the corresponding cluster-specific ML models (i.e., for clusters that includes the entities). In this manner, the more generic ML models are continuously updated and adapted to changes in actual network conditions.

[0109] In some embodiments, upon receiving an analytics request for a network entity, the NWDAF may determine whether an appropriate entity-specific ML model is available to fulfill the request. If so, the NWDAF use the entity-specific model such that the response is both prompt and accurate.

[0110] Embodiments are described below in the context of three use cases: service quality estimation, QoS provisioning, and cell congestion. Even so, these use cases are only examples and skilled persons will recognize that the described principles may be applied to other use cases that may benefit from them.

[0111] Embodiments may provide various benefits and / or advantages. For example, embodiments may be applied to a wide variety of NWDAF analytics that require estimation, prediction, or a combination thereof, including any NWDAF analytic that may benefit from these techniques. By using generic models trained on easily observable (e.g., network- wide) data at for groups (or clusters) of network entities, embodiments may reduce or eliminate the need to collect entity-level data and train entity-level models in advance. Since entity-level data is only collected as needed, the NWDAF resources needed for data collection and model training may be significantly reduced. Moreover, the described “spotlight” data collection includes a fully representative selection of network entities according to resource limitations. In addition, embodiments may facilitate prompt NWDAF responses to analytics requests based on the generic models, with more accurate responses based on entity- specific models provided later as needed.

[0112] Figure 4 shows a high-level view of an exemplary system according to some embodiments of the present disclosure. The system includes a live network (400) and a test network (450). The live network includes a core domain (410) and a RAN domain (420), an example of which is a 5G network including a 5GC and an NG-RAN with gNBs. The core domain of the live network includes an NWDAF (411) and various other NFs, including an analytics consumer NF (412).

[0113] The NWDAF may collect network-related data from various sources in the RAN and CN domains of the live network, including NFs, gNBs, UEs, etc. The NWDAF may collect this data directly or via an operations / administration / maintenance (0AM) system (not shown). The collected data may vary by source, such as performance management (PM) counters from the core domain and EBM or CTR from the RAN domain. The goal of collecting the data is to be able to produce various analytics supported by the NWDAF, such as the examples discussed above. Each of these analytics may be associated with an analytics ID, which a consumer may use to request a specific analytic. The consumer may be another NF (e.g., 412), an AF, or possibly an external consumer via NEF. Different analytics IDs may be associated with different types of network entities, such as network slices (e.g., network slice instance load), radio cells (e.g., user data congestion), and NFs (e.g., NF load). The 3GPP NWDAF specification allows the data collection to be dynamically controlled, such that data is collected about a network entity only if / when a consumer NF is interested in an analytic ID related to the network entity.

[0114] According to embodiments of the present disclosure, when NWDAF is deployed in the live network, it performs data collection from a subset of all network entities in the live network, such as a subset of all cells of the live network. This subset is referred to as a “spotlight”. In contrast, NWDAF does not perform data collection for network entities outside of the spotlight. Accordingly, when an analytics request identifies a particular network entity, NWDAF may not have the necessary data to immediately compute and respond with the requested analytic.

[0115] The test network is used for data collection and training of generic ME models, prior to the NWDAF deployment in the live network. The sources of the data collected from the test network may be similar to, or representative of, the sources from which data will be collected in the live network after NWDAF deployment. Moreover, the data collected from the test network is used to create and / or train generic ML models that can produce predictions and / or estimates associated with analytics IDs that the NWDAF will support once deployed in the live network. These generic ML models are instantly usable by the NWDAF upon deployment in the live network.

[0116] Figure 5 illustrates the creation and use of generic ML models, according to some embodiments of the present disclosure. In this example, three generic ML models are generated based on data collected in the test network. The first generic ML model (Gl) is associated with a first cluster of network entities 1-2 (e.g., network slices, NFs, cells, etc.), the second generic ML model (G2) is associated with a second cluster of network entities 7 and 9-10, while the third generic ML model (G3) is associated with a third cluster of network entities 15-16. Each of GIGS is usable to generate one or more analytics related to any of their associated network entities.

[0117] The clustering of network entities may be based on various parameters and metrics. For example, cells can be clustered based on one or more of cell size, cell location (e.g., urban or rural), cell RAT (e.g., 5G or 4G), typical cell measurements (e.g., level of RSRP and / or RSRQ), etc. Once clustered, the data collected for the network entities within each cluster can be used to train one or more cluster- specific generic ML models. Moreover, compared to typical ML models, less data is needed for training the generic ML models since they are not required to be highly accurate for any specific network entities but merely representative of the cluster.

[0118] Once NWDAF is deployed in the live network, it uses the generic ML models and rules about the clusters, such as boundaries of the different dimensions for each cluster. When the NWDAF is deployed, certain network entities of the live network can be assigned to the spotlight, for which data are collected continuously by NWDAF. For example, these network entities in the spotlight may be deemed “important” by some criteria. After some period of data collection, there will be sufficient data to train entity-specific ML models for the important network entities. For example, in Figure 5, there is a first specific ML model (SI) associated with network entity 1, even though network entity 1 also is associated with generic ML model G1 trained from the test network.

[0119] If an analytic is requested for a network entity without a specific ML model, the NWDAF identifies a cluster that best represents the network entity and uses the generic ML model associated with that cluster to compute the requested analytic (e.g., by prediction and / or estimation). In the example shown in Figure 5, an analytic related to network entity 18 is requested by a consumer, and NWDAF lacks a specific ML model for network entity 18. The NWDAF identifies that network entity 18 is best represented by (or belongs to) the third cluster, which is associated with generic ML model G3. Thus, NWDAF uses G3 the requested analytic for network element 18.

[0120] In some embodiments, if NWDAF determines that the network entity subject to the request is important (e.g., requested repeatedly), NWDAF can initiate continuous data collection for that network entity and subsequently train an entity-specific ML model. For example, NWDAF may add the network entity to the spotlight and, if needed, remove another network entity from the spotlight to comply with resource limitations. In the example shown in Figure 5, NWDAF collects data and trains a specific ML model S18 associated with network entity 18.

[0121] If an analytics request is for a future time interval, NWDAF needs to provide a prediction. An example is a request for expected network slice load in a particular cell during the next hour. If the analytics request is for a metric whose precise value is not easily determined from data collected by NWDAF, then the NWDAF may apply an ML model to estimate the metric. An example is user data congestion, which may be estimated from a selection of RAN / CN counters and event data. The estimate may be for a past, current, or future time interval.

[0122] A combination of prediction and estimation may be needed for asome analytics requests. For example, a consumer requests expected congestion levels for a future time interval. NWDAF uses a model to predict the RAN / CN counter values and events for the future time interval and uses the same or different model to estimate a congestion level based on the predicted values and events. As an alternative, NWDAF may use a model to estimate congestion levels for a past time interval and use the same or a different model to predict the congestion level for the future time interval based on the estimated past congestion levels.

[0123] Figure 6 shows a procedure for generic ML model training, according to some embodiments of the present disclosure. The procedure shown in Figure 6 may be performed by an NWDAF prior to deployment and / or activation in a live network, or by another NF or system (e.g., OAM) that provides generic ML models to NWDAF prior to deployment and / or activation in the live network. As another alternative, the procedure shown in Figure 6 may be performed in a laboratory environment, such as during design of NWDAF algorithms. Various combinations of these alternatives are also possible.

[0124] In operation 610, training data is collected from various network entities in a test network. As an alternative, data may be collected from one or more live networks, to the extent this data collection is not burdensome to the live networks. For example, historical data representative of conditions in live networks can be collected. The network entities for which data is collected depend on the analytics (or analytic IDs) to be supported by the generic ML models. Even so, to support subsequent clustering, it is desirable that data is collected from network entities (e.g., cells, RAN nodes, NFs, etc.) having a range of configurations or types, such as sizes, RATs, locations, traffic level, traffic mixture, etc.

[0125] In some embodiments, the data collected for generic model training can be a subset of data collected in a live network for training entity-specific models. For example, the data collected for generic model training can be the same or similar parameters but at a lower resolution or frequency as data collected in a live network for training entity- specific models. Alternately or in addition, the data collected for generic model training can include fewer numbers and / or types of parameters as data collected in a live network for training entity-specific models. As a more specific example, the data collected for generic model training can include node counters, which are generated network wide continuously, or event data but only for a few parameters with lower time resolution. Other more specific examples are discussed below in relation to specific use cases.

[0126] In any case, labelled data is needed for generic model training. For example, in a test network, a UE or application server can measure end-user QoE for network conditions under which the training data is collected. The UE or application server can provide these end-user QoE measurements (e.g., to NWDAF), which are then appended to the collected network data as labels. For example, an end-user QoE measurement at time t is used to label the network data collected at time t (and possibly before time t), indicating actual end-user QoE for the measured network conditions at time t. In the case of collection from a live network, service agreements with end users or application providers may be needed to collect the end-user QoE needed for labelling. In general, these labels may be generated in various ways, including one or more of the following:

[0127] • Subjective evaluation of service quality, e.g., mean opinion score (MOS), QoE score, etc.

[0128] • Objective comparison of corresponding media input to and output from the communication network to detect distortion, degradation, etc. (e.g., SNR); and

[0129] • Application service quality logs, including statistics such as WebRTC parameters specified by the W3 organization in “Identifiers for WebRTC's Statistics API” (https : / / www.w3.org / TR / webrtc-stats / ) .

[0130] Alternately, some combination of the above may be used to calculate a more complex QoE score used for labeling. When a generic model is used to generate a prediction of a future value based on historical data, the actual future value may be measured and used as a label.

[0131] In operation 620, the network entities for which data was collected are classified in clusters according to parameters relevant to the analytics (or analytic IDs) to be supported by the generic ML models. For example, as discussed above, cells can be clustered based on any one or more of cell size, cell location (e.g., urban or rural), cell RAT (e.g., 5G or 4G), typical cell measurements (e.g., level of RSRP and / or RSRQ), etc.

[0132] In operation 630, input parameters for the generic ML models can be identified from the collected data, either automatically or manually (e.g., by data scientists). These input parameters used to generate the ML model outputs may be different from the parameters used for clustering.

[0133] In operation 640, using the labelled training data, different ML models are trained for each cluster (for estimation, prediction or both for estimation and prediction). These are the generic per-cluster models. At this stage, there is no need to train entity-specific models.

[0134] Figure 7 shows a procedure for determining a requested analytic during operation in a live network, according to some embodiments of the present disclosure. The procedure shown in Figure 7 may be performed by an NWDAF after deployment in a live network.

[0135] In operation 710, a request for a prediction-based analytic associated with a network element is received. In this instance, there is no entity-specific ML model for the requested analytic. In block 720, the NWDAF determines whether data should be collected for the network entity associated with the request. If it is determined that data should be collected (YES), in operation 730 the NWDAF initiates the data collection. When there is enough collected data to train an entity-specific model, in operation 740 the NWDAF uses the collected data to perform this training. In some embodiments (e.g., depending on availability), the training data may be labelled in a similar manner as discussed above in relation to training generic, cluster-associated models. In operation 750, the NWDAF uses the trained entity-specific model to perform the prediction needed for the requested analytic. If it is determined in block 720 that data should not be collected (NO), in operation 760 the NWDAF identifies a cluster that best represents the network entity. This may be based on matching configuration parameters or characteristics (e.g., cell size, location, etc.) of the network entity to corresponding configuration parameters or characteristics associated with the various clusters. The NWDAF selects the generic ML model associated with that cluster and uses it in operation 770 to perform the prediction needed for the requested analytic. In addition, upon starting the data collection in operation 730 but before obtaining enough data to train the entityspecific model, the NWDAF can perform the operations of blocks 760-770. In either case, the NWDAF can promptly provide the requested analytic, albeit in a possibly less accurate form.

[0136] Note that the consumer may base a decision initially on the less accurate (or coarse) version of the analytic provided promptly using a generic model. Upon later receiving the more accurate (or refined) version of the analytic produced by the entity-specific model, the consumer may decide to update, modify, cancel, and / or retrieve the previous decision based on the less accurate version. For example, the consumer may base this decision on the relative difference between the two versions, e.g., in comparison to some threshold.

[0137] When the requested analytic is a prediction of some quantity at some future time, the actual quantity may be measured or obtained later at the future time. For example, a QoS in a particular cell may be predicted for some future time t and then later measured at time t. The QoS prediction by the entity-specific model for the cell can be used together with the measured QoS can be used as part of a labelled dataset for training the generic and / or entity-specific models for the cell. In this manner, the models may be continuously updated and adapted to network or traffic changes.

[0138] The data collection is needed for training of entity-specific, high-performance ML models. On the other hand, the collection, processing and storage of this data may require significant resources. As mentioned above, the term “spotlight” refers to the set of network entities for which data collection is enabled at any given time. The spotlight is based on a pre-defined data collection capacity, according to limited resources available for data collection. In various embodiments, entities may be added to and removed from the spotlight according to different strategies, some examples of which are discussed below.

[0139] In some embodiments, an initial spotlight at system deployment may be defined by manual configuration, based on some geographic area of interest or some non-geographic dimension (e.g., network slice).

[0140] In some embodiments, a hard limit on the number of the number of entities in the spotlight may be configured, due to resource limitations. Thus, when a new entity needs to be added to a spotlight that has reached the limit, another entity currently in the spotlight needs to be removed. Alternately, the hard limit may be based on some resource metric, with each entity associated with some known data collection resource requirement.

[0141] The NWDAF may select which entity to remove from the spotlight to make room for a new entity in various ways. For example, the entity associated with the oldest (i.e., least recent) analytic request may be removed. In addition to removal for replacement, an entity may be removed from the spotlight for other reasons. For example, an entity may be removed when enough data has been collected to train the entity-specific model. In this case, the entity is marked ready and the ML model gets deployed. The ML models is continuously inspected for eventual model drift. If its performance is below a defined threshold, the “ready” mark is removed from the entity and it becomes a candidate for addition to the spotlight again.

[0142] As another example, an entity may be removed when its “interest metric” goes below a defined threshold. This “interest metric” may be calculated based on the analytic requests arriving to the NWDAF. Each analytic request may relate to one or more network entities. For each entity, the interest metric may be a ratio between the queries for that entity and all queries may be calculated. Subscriptions may be calculated with higher weight than one-time queries.

[0143] In addition, the time needed to collect enough data for entity-specific model training may be considered in entity removal selection. For example, likelihood of removal may decrease as the data collected for training an entity-specific model approaches the needed amount, so that “almost- trained” entities will remain in the spotlight until ready to be removed when fully trained. On the other hand, likelihood of removal may increase when the data collected for training an entityspecific model is low relative to the needed amount. One way to do this is by artificially increasing or decreasing the entity’s interest ratio, which is a tradeoff between actual interest level from consumers and production of fully trained entity-specific models.

[0144] In addition, an entity may be added to the spotlight in response to a related analytic request from a consumer or if the entity’s interest ratio exceeds a threshold. If adding the entity would cause the spotlight to exceed resource limits, another entity would need to be removed according to techniques discussed above.

[0145] Embodiments described above may be based on advanced filtering capabilities of data collection entities, either an NWDAF or the entities (e.g., NFs) that produce the data. A specific filtering capability is per-entity filtering, such that data is collected and processed only from specific entities in the spotlight. When entities are service types, filtering may be implemented directly on these entities. An alternative is to create dynamic lists of end users of a certain service type and filter based on this user list.

[0146] In general, the entities within a mobile network may change dynamically: network slices may be created or removed, cells may be activated or deactivated, new UEs or UE types may be introduced, etc. When NWDAF detects a new network entity (e.g., based on an analytic request), it assigns it to one of the existing clusters based on the clustering techniques discussed above. The existing cluster-specific model will initially be used for analytic requests related to this entity. In addition, the new entity is subject to the strategies that control the spotlight. For example, if the entity’s interest ratio reaches the defined threshold, it may be added to the spotlight causing data collection for the entity to be initiated for training of an entity-specific model.

[0147] While new entities may be detected based on the incoming analytic requests, detection of removed entities may require NWDAF to actively monitor for removal events (e.g., by OAM). Whenever a removal event is detected for an entity, the NWDAF also removes the entity from the spotlight (if necessary) and mark the entity-specific model (if existing) as “obsolete”. Optionally, NWDAF may delete the obsolete entity-specific model after a configured retention period, if needed to save storage space.

[0148] Network operators do not have direct access to UEs and thus cannot measure end user service quality. Moreover, over-the-top (OTT) applications (e.g., Netflix) do not share their own end user service quality information with network operators, at least not in real-time. As such, NWDAF implements service quality ML models that estimates end user service quality (also referred to as quality of experience, QoE) based on measured transport parameters such as packet loss, delay, jitter, burst parameters, etc. These high-resolution measurements are resourceintensive and thus are enabled only upon request, which causes delay in the availability of predicted values while data is being collected.

[0149] Even so, it is possible to detect service quality degradation by a simplified or generic ML model, based on lower resolution and a simplified set of transport parameters and radio-related measurements (e.g. bitrate, RSRP, RSRQ) that are continuously monitored for all flows. Accordingly, when a service quality analytic is requested by a consumer, the NWDAF uses the generic ML model to estimate or predict QoE based on the simplified set of parameters. The result is sent to the consumer. The requested entities may be added to the spotlight, if needed. When enough data has been collected to estimate or predict QoE using the service specific model (including training if needed), the updated result is compared with the original result produced by the general model. If the difference is greater than some threshold, the updated result is sent to the consumer.

[0150] Figure 8 shows an example that illustrates QoE estimation based on a generic model and a service specific model, according to some embodiments of the present disclosure. The generic model use throughput measurements with 30-second resolution and radio measurements with 10- second resolution, both of which are obtained from readily available network counters. The resulting QoE estimate less accurate but sufficient to detect existence of a service quality issue to reported to a consumer (e.g., that subscribed to the corresponding analytic).

[0151] When a network entity is added to the spotlight for the QoE estimation, collection is initiated for detailed per-flow transport data from the UPF and higher-resolution radio meaurements from the RAN. As a more specific example, one or more of the following transport key preformance indicators (KPIs) may be collected with 1 -second resolution:

[0152] • Interarrival time (avg / min / max / std);

[0153] • Bitrate (Kbps) ;

[0154] • Packet loss percent;

[0155] • RTT Round trip time in (ms);

[0156] • Jitter (ms);

[0157] • Packet rate (packets / sec);

[0158] • Burst rate (bursts / sec);

[0159] • Burst size in packets (avg / min / max / std over last second);

[0160] • Burst size in bytes (avg / min / max / std over last second);

[0161] • Burst length (in ms, avg / min / max / std over last second), where length of one burst is the difference of the timestamps of first and last packets in that burst; and

[0162] • Burst separation length (in ms, avg / min / max / std over last second), where separation length of two consecutive bursts is the difference between the timestamps of the last packet of the initial burst and the first packet of the subsequent burst.

[0163] Similarly, one or more of the following radio measurements may be collected with 1 -second resolution:

[0164] • Reference Signal Received Power (RSRP, in dBm)

[0165] • Reference Signal Received Quality (RSRQ, in dB or other metric);

[0166] • Signal to Interference & Noise Ratio (SINR, in dB);

[0167] • Physical Cell ID (PCI), used to detect handover;

[0168] • DL Transmission Mode (e.g., 7=Beamforming);

[0169] • UL frequency;

[0170] • DL frequency;

[0171] • UL modulation and coding scheme (MCS);

[0172] • DL MCS;

[0173] • Uplink power on the Physical Uplink Control Channel (PPUCCH), e.g., for transmitting Uplink Control Information (UCI); • Uplink power on the Physical Uplink Shared Channel (PPUSCH), e.g., for transmitting signaling and user / application data;

[0174] • Uplink power for Sounding Reference Signal (PSRS), which are used to estimate the UL channel;

[0175] • Uplink power on the Physical Random Access Channel (PPRACH), which is used by UEs to access the network.

[0176] Figure 9 is a cumulative distribution function (CDF) plot that compares QoE estimation accuracy using the generic and service specific models shown in Figure 8. The horizontal axis is QoE estimation error (for QoEs on scale of 1-5) and the vertical axis is the proportion of test samples for which QoE estimation was less than the value on the horizontal axis. The curve for the service specific model is significantly above the curve for the generic model, indicating reduction in QoE estimation error.

[0177] As another example, embodiments of the present disclosure may be applied to the use case of QoS provisioning and sustainability for a new service type. 5G networks provide a QoS framework to support different QoS requirements of various traffic types. User application traffic is carried in PDU sessions, with session-specific QoS parameters maintained by the network during the lifetime of each PDU session.

[0178] According to 3GPP TS 23.501 (vl8.5.0), the 5G QoS model is based on QoS Flows, and supports QoS Flows that require guaranteed flow bit rate (GBR QoS Flows) and QoS Flows that do not require guaranteed flow bit rate (Non-GBR QoS Flows). A QoS Flow is the finest granularity of QoS differentiation in a PDU Session, and is identified by a QoS Flow ID (QFT). UP traffic with the same QFT within a PDU Session receives the same traffic forwarding treatment (e.g. scheduling, admission threshold).

[0179] Within the 5GS, a QoS Flow is controlled by the SMF and may be preconfigured, or established via the PDU Session Establishment procedure or the PDU Session Modification procedure. Furthermore, each QoS Flow is characterized by a QoS profile provided by the SMF to the AN via the AMF over the N2 reference point or preconfigured in the AN, as well as by one or more QoS rules and (optionally) associated QoS Flow level QoS parameters. These QoS rules and parameters may be provided by the SMF to the UE via the AMF and / or derived by the UE by applying Reflective QoS control.

[0180] It can be complex for the CP NFs (e.g., SMF, PCF) to derive QoS profiles, especially for non-standardized values. Thus, it may be beneficial for the CP NFs (e.g., SMF, PCF) to utilize NWDAF to perform various analytics that facilitate SMF derivation of suitable QoS profiles. In this manner, the NWDAF provides analytics used for improving the network resource utilization and end user QoS experience. For example, PCF requests NWDAF to provide analytics for QoS profile settings of a new service, to fulfill network-level QoS targets specified in a service level agreement (SLA) under different network (e.g., radio-related) conditions. When this request arrives, NWDAF assigns or classifies the service to one of the existing service clusters and estimates QoS profile settings based on the generic model associated with the assigned cluster. The PCF executes a policy configuration for the new service by assigning the QoS profile settings determined by NWDAF based on the cluster-generic model.

[0181] At the same time, users of the new service are added to the spotlight and detailed transport and radio data are collected for the new service and in order to train a new service-specific model. When the service-specific model is trained, NWDAF estimates QoS profile settings for the new service network-wide, and provides these to the PCF. The PCF checks these QoS profile settings against the SLA and, if SLA violation is detected, re-assigns the service to a different QoS class. In some variants, the NWDAF may perform the SLA checking and re-assignment.

[0182] As another example, embodiments of the present disclosure may be applied to the use case of cell congestion prediction. The consumer requests an analytic related to cell load, including a prediction of possible congestion. NWDAF starts measuring used radio resources and compares it to the available resources to obtain a resource utilization ratio. Over some time, NWDAF determines a trend of resource utilization ratio and extrapolates that trend to future times to determine a prediction of resource utilization. If this prediction is at or above some critical value, the analytic consumers is notified and may take corrective action before a cell gets congested as predicted.

[0183] However, measuring resource utilization trends takes time. According to embodiments of the present disclosure, a generic model for a representative cell cluster is trained for estimating the radio resource usage based on actual traffic load and historical daily profile of traffic load. In this context, “traffic load” includes the different traffic types present and the amount (or fraction) of each traffic type.

[0184] When an analytic request for a cell arrives, NWDAF measures the actual traffic load in the cell and assigns the cell to one of the representative clusters associated with generic models (e.g., based on cell size, frequency, actual traffic load, etc.). Using the generic model associated with the assigned representative cell cluster, NWDAF predicts future resource utilization for the requested cell and promptly provides this prediction to the consumer (or an indication when the predicted future resource utilization exceeds a threshold). The NWDAF may also trigger data collection for the cell and subsequent training of a cell-specific model.

[0185] Figure 10 shows an example that illustrates estimation of cell resource utilization based on a generic model and a service specific model, according to some embodiments of the present disclosure. The top graph shows historical cell resource utilization (or load) measured over the period of one day, with peaks during morning and afternoon. These measurements may be for all traffic, for all network slices, for a single network slice, etc. The values shown in the top graph may be for an average or typical day.

[0186] Cells can be clustered based on having similar characteristics, such as size, frequency, location (urban / rural), etc. For each cluster, lower-granularity aggregated data can be collected and used to train a generic model that can provide a resource utilization prediction for the next day based on actual measured resource utilization (e.g., for most recent 24-hour period) and the historical resource utilization shown in the top graph. The middle graph shows resource utilization over a 24-hour period as predicted by the generic model. This prediction follows the general shape of the historical resource utilization but does not capture the minimum between the two peaks.

[0187] Once a sufficient amount of data (e.g., days) is collected to train a specific model for the cell (or slice within the cell), the specific model can be used to predict resource utilization for existing or subsequent requests for analytics related to the cell. This prediction may be based on current measurements and / or historical resource utilization (top graph). As the bottom graph shows, the prediction based on the specific model more accurately tracks the shape of the historical daily resource utilization in the top graph.

[0188] Various features of the embodiments described above correspond to various operations illustrated in Figure 11 (including parts A and B), which shows an exemplary method e.g., procedure) for generating analytics related to performance of network entities of a communication network, according to various embodiments of the present disclosure. In other words, various features of the operations described below correspond to various embodiments described above. Although Figure 11 shows specific blocks in a particular order, the operations of the exemplary method can be performed in a different order than shown and can be combined and / or divided into blocks having different functionality than shown. Optional blocks or operations are indicated by dashed lines.

[0189] The following description is based on the exemplary method being performed by a network analytics system associated the communication network. For example, the network analytics system can be implemented in (or as) a service management and orchestration (SMO) system for a RAN, an analytics-related CN node such as NWDAF, a network management node in an OAM system, or an application running in a host computing system external to the network (e.g., public or private cloud environment).

[0190] The exemplary method includes the operations of block 1120, where the network analytics system receives, from a consumer network function (NF) of the communication network, a request for an analytic related to performance of one or more network entities of the communication network. The exemplary method also includes the following operations for each of the one or more network entities, labelled with corresponding block numbers:

[0191] • (1130) assigning the network entity to one of a plurality of clusters, and

[0192] • (1135) using a generic model associated with the assigned cluster, determining a coarse version of the analytic for the network entity; and

[0193] The exemplary method also includes the operations of block 1150, where the network analytics system sends the coarse versions of the analytic, for the respective network entities, to the consumer NF in response to the request.

[0194] In some embodiments, the exemplary method also includes the operations of block 1125, where for each of the one or more network entities, the network analytics system determines whether an entity-specific model usable to determine a refined version of the requested analytic exists. Assigning the network entity to one of the plurality of clusters in block 1130 is based on determining that the entity- specific model does not exist.

[0195] In some embodiments, the exemplary method also includes the operations of block 1140, where in response to assigning the network entity to one of the clusters, the network analytics system initiates data collection for the network entity. In some of these embodiments, the coarse versions of the analytic are sent while the data collection is ongoing.

[0196] In some of these embodiments, initiating data collection for the network entity in block 1140 is performed selectively based on a data collection resource limit. In some variants of these embodiments, the data collection resource limit is based on a maximum number of network entities for which data collection can be performed concurrently. In some variants of these embodiments, initiating data collection for the network entity in block 1140 includes the following operations, labelled with corresponding sub-block numbers:

[0197] • (1141) determining whether data collection for the network entity will cause data collection resources to exceed the data collection resource limit;

[0198] • (1142) when it is determined that data collection for the network entity will cause data collection resources to exceed the data collection resource limit, determining whether the network entity should be prioritized over at least one other network entity for which data collection is ongoing; and

[0199] • (1144) initiating data collection for the network entity based on determining that data collection for the network entity will not cause data collection resources to exceed the data collection resource limit or that the network entity should be prioritized.

[0200] In some further variants, the respective data collection priorities of the network entity and the at least one other network entity are based on one or more of the following:

[0201] • timing of analytic requests associated with the at least one other network entity; • degree of completion of data collection and / or training of entity-specific models for the at least one other network entity; and

[0202] • respective interest metrics associated with the network entity and the at least one other network entity.

[0203] In some further variants, initiating data collection for the network entity in block 1140 also includes the operations of sub-block 1143, where when it is determined that the network entity should be prioritized, the network analytics system terminates data collection for a lowest-priority one of the at least one other network entity. For example, the lowest-priority other network entity is associated with one or more of the following: a least recent analytic request, a least complete data collection, a least complete training of an entity-specific model, and an associated interest metric that is lowest or less than a first threshold. This termination of data collection of the other network entity in favor of the data collection for the (new) network entity, under a data collection resource limit, is an example of the “spotlight” technique discussed above.

[0204] In some further variants, the network entity is prioritized based on one or more of the following: the request from the consumer NF, and an associated interest metric that is greater than a second threshold or greater than respective interest metrics associated with the at least one other network entity.

[0205] In some of these embodiments, the exemplary method also includes the following operations for each of the one or more network entities, labelled with corresponding block numbers:

[0206] • (1155) using the collected data, training an entity- specific model for the network entity; and

[0207] • (1160) using the trained entity-specific model and a portion of the collected data, determining a refined version of the requested analytic for the network entity.

[0208] Also, in block 1170, the network analytics system sends the refined versions of the analytic, for the respective network entities, to the consumer NF in response to the request.

[0209] In some variants of these embodiments, the exemplary method also includes the operations 1165, where for each of the one or more network entities, the network analytics system terminates data collection for the network entity in response to one of the following: the entity- specific model has been trained, or the network entity is lowest priority among network entities for which data collection is ongoing. In other words, the network analytics system terminates data collection for any of the network entities that meet one of these conditions.

[0210] In some variants of these embodiments, the exemplary method also includes the following operations, labelled with corresponding block numbers: • (1185) subsequently receiving from the consumer NF a second request for the analytic for the one or more network entities; and

[0211] • (1190) for each of the one or more network entities, using the trained entity- specific model, instead of the generic model, and a further portion of the collected data to determine a second refined version of the analytic for the network entity; and

[0212] • (1195) sending the second refined versions of the analytic to the consumer NF in response to the second request.

[0213] In some variants of these embodiments, the analytic is a prediction of a performance metric at a future time and the exemplary method also includes the following operations, labelled with corresponding block numbers:

[0214] • (1175) measuring the performance metric at the future time; and

[0215] • (1180) training the generic model associated with the assigned cluster using the portion of the collected data, the measured performance metric, and the refined prediction of the performance metric.

[0216] In some of these embodiments, the exemplary method also includes the following operations, labelled with corresponding block numbers:

[0217] • (1105) collecting training data from a plurality of network entities of a second communication network;

[0218] • (1110) determining the plurality of clusters based on the collected training data and characteristics of the network entities of the second communication network; and

[0219] • (1115) for each of the plurality of clusters, training the associated generic model based on the training data collected from the network entities of the second communication network. In some variants of these embodiments, determining the plurality of clusters in block 1110 includes the following operations, labelled with corresponding sub-block numbers:

[0220] • (1111) selecting one more characteristics of the network entities of the second communication network that are most relevant for analytics of interest to consumer NFs of the communication network; and

[0221] • (1112) assigning the network entities of the second communication network to the plurality of clusters based on the selected characteristics,

[0222] Training the generic model associated with each cluster in block 1115 is based on training data collected from the network entities assigned to the cluster in sub-block 11121.

[0223] In some variants of these embodiments, the communication network is a commercially operational network and the second communication network is one of the following: a test network, a laboratory network, the communication network, or a second commercially operational network. In some variants of these embodiments, the data collected from the communication network and the training data collected from the second communication network differ according to one or more of the following characteristics:

[0224] • resolution or frequency, with the data collected from the communication network having a higher resolution or frequency; and

[0225] • numbers and / or types of parameters collected, with the data collected from the communication network having a greater number and / or more types of parameters collected.

[0226] In some variants of these embodiments, the exemplary method also includes the operations of block 1108, where the network analytics system labels the collected training data based on observed values of performance metrics that the respective generic models will be used to estimate or predict. In such case, training the generic model associated with each cluster in block 1115 is based on the labelled training data.

[0227] In some embodiments, assigning the network entity to one of a plurality of clusters in block 1130 is based on one or more of the following characteristics associated with the network entity and the respective clusters: size, location, coverage area, bandwidth, frequency, capacity, radio access technology (RAT), typical signal strength and / or quality experienced by user equipment (UEs), traffic level, traffic mixture, number of UEs served, and types of UEs served.

[0228] In some embodiments, the generic model for each cluster is usable to compute an analytic based on one or more of the following: a prediction of a future value of a performance metric, an estimate of a past value of a performance metric, and an estimate of a current value of a performance metric.

[0229] In some embodiments, the requested analytic is related to one of the following: end user quality of experience (QoE), quality of service (QoS) provisioning for a new service type, and network entity resource utilization.

[0230] Although various embodiments are described herein above in terms of methods, apparatus, devices, computer-readable medium and receivers, the person of ordinary skill will readily comprehend that such methods can be embodied by various combinations of hardware and software in various systems, communication devices, computing devices, control devices, apparatuses, non-transitory computer-readable media, etc.

[0231] Figure 12 shows an example of a communication system 1200 in accordance with some embodiments. In this example, communication system 1200 includes a telecommunication network 1202 that includes an access network 1204 (e.g., RAN) and a core network 1206, which includes one or more core network nodes 1208. Access network 1204 includes one or more access network nodes, such as network nodes 1210a-b (one or more of which may be generally referred to as network nodes 1210), or any other similar 3GPP access nodes or non-3GPP access points. Moreover, as will be appreciated by those of skill in the art, a network node is not necessarily limited to an implementation in which a radio portion and a baseband portion are supplied and integrated by a single vendor. Thus, it will be understood that network nodes include disaggregated implementations or portions thereof. For example, in some embodiments, telecommunication network 1202 includes one or more Open-RAN (ORAN) network nodes. An ORAN network node is a node in telecommunication network 1202 that supports an ORAN specification (e.g., a specification published by the O-RAN Alliance, or any similar organization) and may operate alone or together with other nodes to implement one or more functionalities of any node in telecommunication network 1202, including one or more network nodes 1210 and / or core network nodes 1208.

[0232] Examples of an ORAN network node include an open radio unit (O-RU), an open distributed unit (O-DU), an open central unit (O-CU), including an O-CU control plane (O-CU- CP) or an O-CU user plane (O-CU-UP), a RAN intelligent controller (near-real time or non-real time) hosting software or software plug-ins, such as a near-real time control application (e.g., xApp) or a non-real time control application (e.g., rApp), or any combination thereof (the adjective “open” designating support of an ORAN specification). The network node may support a specification by, for example, supporting an interface defined by the ORAN specification, such as an Al, Fl, Wl, El, E2, X2, Xn interface, an open fronthaul user plane interface, or an open fronthaul management plane interface. Moreover, an ORAN access node may be a logical node in a physical node. Furthermore, an ORAN network node may be implemented in a virtualization environment (described further below) in which one or more network functions are virtualized. For example, the virtualization environment may include an O-Cloud computing platform orchestrated by a Service Management and Orchestration Framework via an O-2 interface defined by the O-RAN Alliance or comparable technologies. Network nodes 1210 facilitate direct or indirect connection of UEs, such as by connecting UEs 1212a-d (one or more of which may be generally referred to as UEs 1212) to core network 1206 over one or more wireless connections.

[0233] In some embodiments, telecommunication network 1202 can also include one or more Network Management (NM) nodes 1220, which can be part of an operation support system (OSS), a business support system (BSS), and / or an operation / administration / maintenance (0AM) system. NM nodes 1220 may monitor and / or control operations of other nodes in access network 1204 and core network 1206. Although not shown in Figure 12, NM node 1220 may be configured to communicate with other nodes in access network 1204 and core network 1206 for these purposes. Example wireless communications over a wireless connection include transmitting and / or receiving wireless signals using electromagnetic waves, radio waves, infrared waves, and / or other types of signals suitable for conveying information without the use of wires, cables, or other material conductors. Moreover, in different embodiments, communication system 1200 may include any number of wired or wireless networks, network nodes, UEs, and / or any other components or systems that may facilitate or participate in the communication of data and / or signals whether via wired or wireless connections. Communication system 1200 may include and / or interface with any type of communication, telecommunication, data, cellular, radio network, and / or other similar type of system.

[0234] UEs 1212 may be any of a wide variety of communication devices, including wireless devices arranged, configured, and / or operable to communicate wirelessly with network nodes 1210 and other communication devices. Similarly, network nodes 1210 are arranged, capable, configured, and / or operable to communicate directly or indirectly with UEs 1212 and / or with other network nodes or equipment in telecommunication network 1202 to enable and / or provide network access, such as wireless network access, and / or to perform other functions, such as administration in telecommunication network 1202.

[0235] In the depicted example, core network 1206 connects network nodes 1210 to one or more hosts, such as host 1216. These connections may be direct or indirect via one or more intermediary networks or devices. In other examples, network nodes may be directly coupled to hosts. Core network 1206 includes one or more core network nodes (e.g., 1208) that are structured with hardware and software components. Features of these components may be substantially similar to those described with respect to the UEs, network nodes, and / or hosts, such that the descriptions thereof are generally applicable to the corresponding components of core network node 1208. Example core network nodes include functions of one or more of a Mobile Switching Center (MSC), Mobility Management Entity (MME), Home Subscriber Server (HSS), Access and Mobility Management Function (AMF), Session Management Function (SMF), Authentication Server Function (AUSF), Subscription Identifier De-concealing function (SIDE), Unified Data Management (UDM), Security Edge Protection Proxy (SEPP), Network Exposure Function (NEF), and / or a User Plane Function (UPF).

[0236] Host 1216 may be under the ownership or control of a service provider other than an operator or provider of access network 1204 and / or telecommunication network 1202, and may be operated by the service provider or on behalf of the service provider. Host 1216 may host a variety of applications to provide one or more service. Examples of such applications include live and pre-recorded audio / video content, data collection services such as retrieving and compiling data on various ambient conditions detected by a plurality of UEs, analytics functionality, social media, functions for controlling or otherwise interacting with remote devices, functions for an alarm and surveillance center, or any other such function performed by a server.

[0237] In some embodiments, access network 1204 can include a service management and orchestration (SMO) system or node 1220, which can monitor and / or control operations of the access network nodes 1210. This arrangement can be used, for example, when access network 1204 utilizes an O-RAN architecture. SMO system 1220 can be configured to communicate with core network 1206 and / or host 1216, as shown in Figure 12.

[0238] In some embodiments, one or more of core network node 1208, NM node 1220, and SMO system 1218 can be configured to perform various operations of exemplary methods (e.g., procedures) for generating analytics related to performance of network entities of a communication network, such as described above in relation to other figures.

[0239] As a whole, communication system 1200 of Figure 12 enables connectivity between the UEs, network nodes, and hosts. In that sense, the communication system may be configured to operate according to predefined rules or procedures, such as specific standards that include, but are not limited to: Global System for Mobile Communications (GSM); Universal Mobile Telecommunications System (UMTS); Long Term Evolution (LTE), and / or other suitable 2G, 3G, 4G, 5G standards, or any applicable future generation standard (e.g., 6G); wireless local area network (WLAN) standards, such as the Institute of Electrical and Electronics Engineers (IEEE) 802.11 standards (WiFi); and / or any other appropriate wireless communication standard, such as the Worldwide Interoperability for Microwave Access (WiMax), Bluetooth, Z-Wave, Near Field Communication (NFC) ZigBee, LiFi, and / or any low-power wide-area network (LPWAN) standards such as LoRa and Sigfox.

[0240] In some examples, telecommunication network 1202 is a cellular network that implements 3GPP standardized features. Accordingly, telecommunication network 1202 may support network slicing to provide different logical networks to different devices that are connected to telecommunication network 1202. For example, telecommunication network 1202 may provide Ultra Reliable Low Latency Communication (URLLC) services to some UEs, while providing Enhanced Mobile Broadband (eMBB) services to other UEs, and / or Massive Machine Type Communication (mMTC) / Massive loT services to yet further UEs.

[0241] In some examples, UEs 1212 are configured to transmit and / or receive information without direct human interaction. For instance, a UE may be designed to transmit information to access network 1204 on a predetermined schedule, when triggered by an internal or external event, or in response to requests from access network 1204. Additionally, a UE may be configured for operating in single- or multi-RAT or multi-standard mode. For example, a UE may operate with any one or combination of Wi-Fi, NR (New Radio) and LTE, i.e. being configured for multi-radio dual connectivity (MR-DC), such as E-UTRAN (Evolved-UMTS Terrestrial Radio Access Network) New Radio - Dual Connectivity (EN-DC).

[0242] In the example, hub 1214 communicates with access network 1204 to facilitate indirect communication between one or more UEs (e.g., 1212c and / or 1212d) and network nodes (e.g., network node 1210b). In some examples, hub 1214 may be a controller, router, content source and analytics, or any of the other communication devices described herein regarding UEs. For example, hub 1214 may be a broadband router enabling access to core network 1206 for the UEs. As another example, hub 1214 may be a controller that sends commands or instructions to one or more actuators in the UEs. Commands or instructions may be received from the UEs, network nodes 1210, or by executable code, script, process, or other instructions in hub 1214. As another example, hub 1214 may be a data collector that acts as temporary storage for UE data and, in some embodiments, may perform analysis or other processing of the data. As another example, hub 1214 may be a content source. For example, for a UE that is a VR headset, display, loudspeaker or other media delivery device, hub 1214 may retrieve VR assets, video, audio, or other media or data related to sensory information via a network node, which hub 1214 then provides to the UE either directly, after performing local processing, and / or after adding additional local content. In still another example, hub 1214 acts as a proxy server or orchestrator for the UEs, in particular if one or more of the UEs are low energy loT devices.

[0243] Figure 13 shows a network node 1300 in accordance with some embodiments. Examples of network nodes include, but are not limited to, access points (APs) (e.g., radio access points), base stations (e.g., radio base stations, Node Bs, eNBs, gNBs), and O-RAN nodes or components of an O-RAN node (e.g., O-RU, O-DU, O-CU).

[0244] Base stations may be categorized based on the amount of coverage they provide (or, stated differently, their transmit power level) and so, depending on the provided amount of coverage, may be referred to as femto base stations, pico base stations, micro base stations, or macro base stations. A base station may be a relay node or a relay donor node controlling a relay. A network node may also include one or more (or all) parts of a distributed radio base station such as centralized digital units, distributed units (e.g., in an O-RAN access node) and / or remote radio units (RRUs), sometimes referred to as Remote Radio Heads (RRHs). Such remote radio units may or may not be integrated with an antenna as an antenna integrated radio. Parts of a distributed radio base station may also be referred to as nodes in a distributed antenna system (DAS).

[0245] Other examples of network nodes include multiple transmission point (multi-TRP) 5G access nodes, multi-standard radio (MSR) equipment such as MSR BSs, network controllers such as radio network controllers (RNCs) or base station controllers (BSCs), base transceiver stations (BTSs), transmission points, transmission nodes, multi-cell / multicast coordination entities (MCEs), Operation and Maintenance (O&M) nodes, Operations Support System (OSS) nodes, Self-Organizing Network (SON) nodes, positioning nodes (e.g., Evolved Serving Mobile Location Centers (E-SMLCs)), and / or Minimization of Drive Tests (MDTs).

[0246] In some embodiments, network node 1300 can be configured to perform various operations of exemplary methods e.g., procedures) for generating analytics related to performance of network entities of a communication network, such as described above in relation to other figures.

[0247] Network node 1300 includes processing circuitry 1302, memory 1304, communication interface 1306, and power source 1308. Network node 1300 may be composed of multiple physically separate components (e.g., a NodeB component and a RNC component, or a BTS component and a BSC component, etc.), which may each have their own respective components. In certain scenarios in which network node 1300 comprises multiple separate components (e.g., BTS and BSC components), one or more of the separate components may be shared among several network nodes. For example, a single RNC may control multiple NodeBs. In such a scenario, each unique NodeB and RNC pair, may in some instances be considered a single separate network node. In some embodiments, network node 1300 may be configured to support multiple radio access technologies (RATs). In such embodiments, some components may be duplicated (e.g., separate memory 1304 for different RATs) and some components may be reused (e.g., a same antenna 1310 may be shared by different RATs). Network node 1300 may also include multiple sets of the various illustrated components for different wireless technologies integrated into network node 1300, for example GSM, WCDMA, LTE, NR, WiFi, Zigbee, Z-wave, LoRaWAN, Radio Frequency Identification (RFID) or Bluetooth wireless technologies. These wireless technologies may be integrated into the same or different chip or set of chips and other components within network node 1300.

[0248] Processing circuitry 1302 may comprise a combination of one or more of a microprocessor, controller, microcontroller, central processing unit, digital signal processor, application-specific integrated circuit, field programmable gate array, or any other suitable computing device, resource, or combination of hardware, software and / or encoded logic operable to provide, either alone or in conjunction with other network node 1300 components, such as memory 1304, to provide network node 1300 functionality.

[0249] In some embodiments, processing circuitry 1302 includes a system on a chip (SOC). In some embodiments, processing circuitry 1302 includes radio frequency (RF) transceiver circuitry 1312 and / or baseband processing circuitry 1314. In some embodiments, RF transceiver circuitry 1312 and baseband processing circuitry 1314 may be on separate chips (or sets of chips), boards, or units, such as radio units and digital units. In alternative embodiments, part or all of RF transceiver circuitry 1312 and / or baseband processing circuitry 1314 may be on the same chip or set of chips, boards, or units.

[0250] Memory 1304 may comprise any form of volatile or non-volatile computer-readable memory including, without limitation, persistent storage, solid-state memory, remotely mounted memory, magnetic media, optical media, random access memory (RAM), read-only memory (ROM), mass storage media (for example, a hard disk), removable storage media (for example, a flash drive, a Compact Disk (CD) or a Digital Video Disk (DVD)), and / or any other volatile or non-volatile, non-transitory device-readable and / or computer-executable memory devices that store information, data, and / or instructions that may be used by processing circuitry 1302. Memory 1304 may store any suitable instructions, data, or information, including a computer program, software, an application including one or more of logic, rules, code, tables, and / or other instructions (collected denoted computer program 1304a, which may be in the form of a computer program product) capable of being executed by processing circuitry 1302 and utilized by network node 1300. Memory 1304 may be used to store any calculations made by processing circuitry 1302 and / or any data received via communication interface 1306. In some embodiments, processing circuitry 1302 and memory 1304 is integrated.

[0251] Communication interface 1306 is used in wired or wireless communication of signaling and / or data between a network node, access network, and / or UE. As illustrated, communication interface 1306 comprises port(s) / terminal(s) 1316 to send and receive data, for example to and from a network over a wired connection. Communication interface 1306 also includes radio frontend circuitry 1318 that may be coupled to, or in certain embodiments a part of, antenna 1310. Radio front-end circuitry 1318 comprises filters 1320 and amplifiers 1322. Radio front-end circuitry 1318 may be connected to an antenna 1310 and processing circuitry 1302. The radio front-end circuitry may be configured to condition signals communicated between antenna 1310 and processing circuitry 1302. Radio front-end circuitry 1318 may receive digital data that is to be sent out to other network nodes or UEs via a wireless connection. Radio front-end circuitry 1318 may convert the digital data into a radio signal having the appropriate channel and bandwidth parameters using a combination of filters 1320 and / or amplifiers 1322. The radio signal may then be transmitted via antenna 1310. Similarly, when receiving data, antenna 1310 may collect radio signals which are then converted into digital data by radio front-end circuitry 1318. The digital data may be passed to processing circuitry 1302. In other embodiments, the communication interface may comprise different components and / or different combinations of components.

[0252] In certain alternative embodiments, network node 1300 does not include separate radio front-end circuitry 1318, instead, processing circuitry 1302 includes radio front-end circuitry and is connected to antenna 1310. Similarly, in some embodiments, all or some of RF transceiver circuitry 1312 is part of communication interface 1306. In still other embodiments, communication interface 1306 includes one or more ports or terminals 1316, radio front-end circuitry 1318, and RF transceiver circuitry 1312, as part of a radio unit (not shown), and communication interface 1306 communicates with baseband processing circuitry 1314, which is part of a digital unit (not shown).

[0253] Antenna 1310 may include one or more antennas, or antenna arrays, configured to send and / or receive wireless signals. Antenna 1310 may be coupled to radio front-end circuitry 1318 and may be any type of antenna capable of transmitting and receiving data and / or signals wirelessly. In certain embodiments, antenna 1310 is separate from network node 1300 and connectable to network node 1300 through an interface or port.

[0254] Antenna 1310, communication interface 1306, and / or processing circuitry 1302 may be configured to perform any receiving operations and / or certain obtaining operations described herein as being performed by the network node. Any information, data and / or signals may be received from a UE, another network node and / or any other network equipment. Similarly, antenna 1310, communication interface 1306, and / or processing circuitry 1302 may be configured to perform any transmitting operations described herein as being performed by the network node. Any information, data and / or signals may be transmitted to a UE, another network node and / or any other network equipment.

[0255] Power source 1308 provides power to the various components of network node 1300 in a form suitable for the respective components (e.g., at a voltage and current level needed for each respective component). Power source 1308 may further comprise, or be coupled to, power management circuitry to supply the components of network node 1300 with power for performing the functionality described herein. For example, network node 1300 may be connectable to an external power source (e.g., the power grid, an electricity outlet) via an input circuitry or interface such as an electrical cable, whereby the external power source supplies power to power circuitry of power source 1308. As a further example, power source 1308 may comprise a source of power in the form of a battery or battery pack which is connected to, or integrated in, power circuitry. The battery may provide backup power should the external power source fail.

[0256] Embodiments of network node 1300 may include additional components beyond those shown in Figure 13 for providing certain aspects of the network node’s functionality, including any of the functionality described herein and / or any functionality necessary to support the subject matter described herein. For example, network node 1300 may include user interface equipment to allow input of information into network node 1300 and to allow output of information from network node 1300. This may allow a user to perform diagnostic, maintenance, repair, and other administrative functions for network node 1300. Figure 14 is a block diagram illustrating a virtualization environment 1400 in which functions implemented by some embodiments may be virtualized. In the present context, virtualizing means creating virtual versions of apparatuses or devices which may include virtualizing hardware platforms, storage devices and networking resources. As used herein, virtualization can be applied to any device described herein, or components thereof, and relates to an implementation in which at least a portion of the functionality is implemented as one or more virtual components. Some or all of the functions described herein may be implemented as virtual components executed by one or more virtual machines (VMs) implemented in one or more virtual environments 1400 hosted by one or more of hardware nodes, such as a hardware computing device that operates as a network node, UE, core network node, or host. Further, in embodiments in which the virtual node does not require radio connectivity (e.g., a core network node or host), then the node may be entirely virtualized. In some embodiments, the virtualization environment 1400 includes components defined by the O-RAN Alliance, such as an O-Cloud environment orchestrated by a Service Management and Orchestration Framework via an O-2 interface.

[0257] Applications 1402 (which may alternatively be called software instances, virtual appliances, network functions, virtual nodes, virtual network functions, etc.) are run in the virtualization environment 1400 to implement some of the features, functions, and / or benefits of some of the embodiments disclosed herein. In some embodiments, one or more applications 1402 can be configured to perform various operations of exemplary methods (e.g., procedures) for generating analytics related to performance of network entities of a communication network, such as described above in relation to other figures.

[0258] Hardware 1404 includes processing circuitry, memory that stores software and / or instructions (collected denoted computer program 1404a, which may be in the form of a computer program product) executable by hardware processing circuitry, and / or other hardware devices as described herein, such as a network interface, input / output interface, and so forth. Software may be executed by the processing circuitry to instantiate one or more virtualization layers 1406 (also referred to as hypervisors or virtual machine monitors (VMMs)), provide VMs 1408a-b (one or more of which may be generally referred to as VMs 1408), and / or perform any of the functions, features and / or benefits described in relation with some embodiments described herein. Virtualization layer 1406 may present a virtual operating platform that appears like networking hardware to VMs 1408.

[0259] VMs 1408 comprise virtual processing, virtual memory, virtual networking or interface and virtual storage, and may be run by a corresponding virtualization layer 1406. Different embodiments of the instance of a virtual appliance 1402 may be implemented on one or more of VMs 1408, and the implementations may be made in different ways. Virtualization of the hardware is in some contexts referred to as network function virtualization (NFV). NFV may be used to consolidate many network equipment types onto industry standard high volume server hardware, physical switches, and physical storage, which can be located in data centers, and customer premise equipment.

[0260] In the context of NFV, each VM 1408 may be a software implementation of a physical machine that runs programs as if they were executing on a physical, non-virtualized machine. Each VM 1408, and that part of hardware 1404 that executes that VM, be it hardware dedicated to that VM and / or hardware shared by that VM with others of the VMs, forms separate virtual network elements. Still in the context of NFV, a virtual network function is responsible for handling specific network functions that run in one or more VMs 1408 on top of the hardware 1404 and corresponds to application 1402.

[0261] Hardware 1404 may be implemented in a standalone network node with generic or specific components. Hardware 1404 may implement some functions via virtualization. Alternatively, hardware 1404 may be part of a larger cluster of hardware (e.g. such as in a data center or CPE) where many hardware nodes work together and are managed via management and orchestration function 1410, which, among others, oversees lifecycle management of applications 1402. In some embodiments, hardware 1404 is coupled to one or more radio units that each include one or more transmitters and one or more receivers that may be coupled to one or more antennas. Radio units may communicate directly with other hardware nodes via one or more appropriate network interfaces and may be used in combination with the virtual components to provide a virtual node with radio capabilities, such as a radio access node or a base station. In some embodiments, some signaling can be provided with the use of a control system 1412 which may alternatively be used for communication between hardware nodes and radio units.

[0262] The foregoing merely illustrates the principles of the disclosure. Various modifications and alterations to the described embodiments will be apparent to those skilled in the art in view of the teachings herein. It will thus be appreciated that those skilled in the art will be able to devise numerous systems, arrangements, and procedures that, although not explicitly shown or described herein, embody the principles of the disclosure and can be thus within the spirit and scope of the disclosure. Various embodiments can be used together with one another, as well as interchangeably therewith, as should be understood by those having ordinary skill in the art.

[0263] The term unit, as used herein, can have conventional meaning in the field of electronics, electrical devices and / or electronic devices and can include, for example, electrical and / or electronic circuitry, devices, modules, processors, memories, logic solid state and / or discrete devices, computer programs or instructions for carrying out respective tasks, procedures, computations, outputs, and / or displaying functions, and so on, as such as those that are described herein.

[0264] Any appropriate steps, methods, features, functions, or benefits disclosed herein may be performed through one or more functional units or modules of one or more virtual apparatuses. Each virtual apparatus may comprise a number of these functional units. These functional units may be implemented via processing circuitry, which may include one or more microprocessor or microcontrollers, as well as other digital hardware, which may include Digital Signal Processor (DSPs), special-purpose digital logic, and the like. The processing circuitry may be configured to execute program code stored in memory, which may include one or several types of memory such as Read Only Memory (ROM), Random Access Memory (RAM), cache memory, flash memory devices, optical storage devices, etc. Program code stored in memory includes program instructions for executing one or more telecommunications and / or data communications protocols as well as instructions for carrying out one or more of the techniques described herein. In some implementations, the processing circuitry may be used to cause the respective functional unit to perform corresponding functions according one or more embodiments of the present disclosure.

[0265] As described herein, device and / or apparatus can be represented by a semiconductor chip, a chipset, or a (hardware) module comprising such chip or chipset; this, however, does not exclude the possibility that a functionality of a device or apparatus, instead of being hardware implemented, be implemented as a software module such as a computer program or a computer program product comprising executable software code portions for execution or being run on a processor. Furthermore, functionality of a device or apparatus can be implemented by any combination of hardware and software. A device or apparatus can also be regarded as an assembly of multiple devices and / or apparatuses, whether functionally in cooperation with or independently of each other. Moreover, devices and apparatuses can be implemented in a distributed fashion throughout a system, so long as the functionality of the device or apparatus is preserved. Such and similar principles are considered as known to a skilled person.

[0266] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. It will be further understood that terms used herein should be interpreted as having a meaning that is consistent with their meaning in the context of this specification and the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0267] In addition, certain terms used in the present disclosure, including the specification and drawings, can be used synonymously in certain instances (e.g., “data” and “information”). It should be understood, that although these terms (and / or other terms that can be synonymous to one another) can be used synonymously herein, there can be instances when such words can be intended to not be used synonymously.

Claims

CLAIMS1. A computer-implemented method for generating analytics related to performance of network entities of a communication network, the method comprising: receiving (1120), from a consumer network function, NF, of the communication network, a request for an analytic related to performance of one or more network entities of the communication network; for each of the one or more network entities: assigning (1130) the network entity to one of a plurality of clusters, and using a generic model associated with the assigned cluster, determining (11135) a coarse version of the analytic for the network entity; and sending (1150) the coarse versions of the analytic, for the respective network entities, to the consumer NF in response to the request.

2. The method of claim 1, further comprising, for each of the one or more network entities, determining (1125) whether an entity-specific model usable to determine a refined version of the requested analytic exists, wherein assigning (1130) the network entity to one of the plurality of clusters is based on determining that the entity-specific model does not exist.

3. The method of any of claims 1-2, further comprising, in response to assigning (1130) the network entity to one of the clusters, initiating (1140) data collection for the network entity.

4. The method of claim 3, wherein the coarse versions of the analytic are sent while the data collection is ongoing.

5. The method of any of claims 3-4, wherein initiating (1140) data collection for the network entity is performed selectively based on a data collection resource limit.

6. The method of claim 5, wherein the data collection resource limit is based on a maximum number of network entities for which data collection can be performed concurrently.

7. The method of any of claims 5-6, wherein initiating (1140) data collection for the network entity comprises:determining (1141) whether data collection for the network entity will cause data collection resources to exceed the data collection resource limit; when it is determined that data collection for the network entity will cause data collection resources to exceed the data collection resource limit, determining (1142) whether the network entity should be prioritized over at least one other network entity for which data collection is ongoing; and initiating (1144) data collection for the network entity based on determining that data collection for the network entity will not cause data collection resources to exceed the data collection resource limit or that the network entity should be prioritized.

8. The method of claim 7, wherein respective data collection priorities of the network entity and the at least one other network entity are based on one or more of the following: timing of analytic requests associated with the at least one other network entity; degree of completion of data collection and / or training of entity-specific models for the at least one other network entity; and respective interest metrics associated with the network entity and the at least one other network entity.

9. The method of any of claims 7-8, initiating (1140) data collection for the network entity further comprises, when it is determined that the network entity should be prioritized, terminating (1143) data collection for a lo west-priority one of the at least one other network entity.

10. The method of claim 9, wherein the lowest-priority other network entity is associated with one or more of the following: a least recent analytic request, a least complete data collection, a least complete training of an entity-specific model, and an associated interest metric that is lowest or less than a first threshold.

11. The method of any of claims 8-10, wherein the network entity is prioritized based on one or more of the following: the request from the consumer NF, and an associated interest metric that is greater than a second threshold or greater than respective interest metrics associated with the at least one other network entity.

12. The method of any of claims 3-11, further comprising:for each of the one or more network entities: using the collected data, training (1155) an entity- specific model for the network entity; and using the trained entity- specific model and a portion of the collected data, determining (1160) a refined version of the requested analytic for the network entity; and sending (1170) the refined versions of the analytic, for the respective network entities, to the consumer NF in response to the request.

13. The method of claim 12, further comprising terminating (1165) data collection for the network entity in response to one of the following: the entity-specific model has been trained, or the network entity is lowest priority among network entities for which data collection is ongoing.

14. The method of any of claims 12-13, further comprising: subsequently receiving (1185) from the consumer NF a second request for the analytic for the one or more network entities; and for each of the one or more network entities, using (1190) the trained entity- specific model, instead of the generic model, and a further portion of the collected data to determine a second refined version of the analytic for the network entity; and sending (1195) the second refined versions of the analytic to the consumer NF in response to the second request.

15. The method of any of claims 12-14, wherein the analytic is a prediction of a performance metric at a future time and the method further comprises: measuring (1175) the performance metric at the future time; and training (1180) the generic model associated with the assigned cluster using the portion of the collected data, the measured performance metric, and the refined prediction of the performance metric.

16. The method of any of claims 3-15, further comprising: collecting (1105) training data from a plurality of network entities of a second communication network; determining (1110) the plurality of clusters based on the collected training data and characteristics of the network entities of the second communication network; andfor each of the plurality of clusters, training (1115) the associated generic model based on the training data collected from the network entities of the second communication network.

17. The method of claim 16, wherein determining (1110) the plurality of clusters comprises: selecting (1111) one more characteristics of the network entities of the second communication network that are most relevant for analytics of interest to consumer NFs of the communication network; and assigning (1112) the network entities of the second communication network to the plurality of clusters based on the selected characteristics, wherein training (1115) the generic model associated with each cluster is based on training data collected from the network entities assigned to the cluster.

18. The method of any of claims 16-17, wherein the communication network is a commercially operational network and the second communication network is one of the following: a test network, a laboratory network, the communication network, or a second commercially operational network.

19. The method of any of claims 16-18, wherein the data collected from the communication network and the training data collected from the second communication network differ according to one or more of the following characteristics: resolution or frequency, with the data collected from the communication network having a higher resolution or frequency; and numbers and / or types of parameters collected, with the data collected from the communication network having a greater number and / or more types of parameters collected.

20. The method of any of claims 16-19, further comprising labelling (1108) the collected training data based on observed values of performance metrics that the respective generic models will be used to estimate or predict, wherein training (1115) the generic model associated with each cluster is based on the labelled training data.

21. The method of any of claims 1-20, wherein assigning (1130) the network entity to one of a plurality of clusters is based on one or more of the following characteristics associated with the network entity and the respective clusters: size; location; coverage area; bandwidth;frequency; capacity; radio access technology, RAT; typical signal strength and / or quality experienced by user equipment, UEs; traffic level; traffic mixture; number of UEs served; and types of UEs served.

22. The method of any of claims 1-21, wherein the generic model for each cluster is usable to compute an analytic based on one or more of the following: a prediction of a future value of a performance metric, an estimate of a past value of a performance metric, and an estimate of a current value of a performance metric.

23. The method of any of claims 1-22, wherein the requested analytic is related to one of the following: end user quality of experience, QoE; quality of service, QoS, provisioning for a new service type; and network entity resource utilization.

24. A network analytics system (210, 411, 1208, 1218, 1220, 1300, 1402) configured to generate analytics related to performance of network entities of a communication network (200, 400, 1202), the network analytics system comprising: communication interface circuitry (1306, 1404) configured to communicate with nodes or functions in the communication network; and processing circuitry (1302, 1404) operatively coupled to the communication interface circuitry, whereby the processing circuitry and the communication interface circuitry are configured to: receive, from a consumer network function, NF (412, 1208, 1300, 1402) of the communication network, a request for an analytic related to performance of one or more network entities of the communication network; for each of the one or more network entities: assign the network entity to one of a plurality of clusters, and using a generic model associated with the assigned cluster, determine a coarse version of the analytic for the network entity; and send the coarse versions of the analytic, for the respective network entities, to the consumer NF in response to the request.

25. The network analytics system of claim 24, wherein the processing circuitry and the communication interface circuitry are further configured to perform operations corresponding to any of claims 2-23.

26. A network analytics system (210, 411, 1208, 1218, 1220, 1300, 1402) configured to generate analytics related to performance of network entities of a communication network (200, 400, 1202), the network analytics system being further configured to: receive, from a consumer network function, NF (412, 1208, 1300, 1402) of the communication network, a request for an analytic related to performance of one or more network entities of the communication network; for each of the one or more network entities: assign the network entity to one of a plurality of clusters, and using a generic model associated with the assigned cluster, determine a coarse version of the analytic for the network entity; and send the coarse versions of the analytic, for the respective network entities, to the consumer NF in response to the request.

27. The network analytics system of claim 26, being further configured to perform operations corresponding to any of the methods of claims 2-23.

28. A non-transitory, computer-readable medium (1304, 1404) storing computer-executable instructions that, when executed by processing circuitry (1302, 1404) associated with a network analytics system (210, 411, 1208, 1218, 1220, 1300, 1402) configured to generate analytics related to performance of network entities of a communication network (200, 400, 1202), configure the network analytics system to perform operations corresponding to any of the methods of claims 1-23.

29. A computer program product (1304a, 1404a) storing computer-executable instructions that, when executed by processing circuitry (1302, 1404) associated with a network analytics system (210, 411, 1208, 1218, 1220, 1300, 1402) configured to generate analytics related to performance of network entities of a communication network (200, 400, 1202), configure the network analytics system to perform operations corresponding to any of the methods of claims 1-23.

Images