Training data augmentation

Abstract

An analytics system (160) operating in a mobile communications network for training a machine learning, ML, model (164) configured to predict service quality (200) is provided. The analytics system obtains (212) a first set of training data for the ML model. The first set of training data is generated by a first communications network (100) and comprises network-level metrics and application-level metrics. The analytics system then receives (214) service quality metrics associated with one or more User Equipments, UEs (170), operating in a second mobile communications network (150), which is a live production mobile communications network. The analytics system then generates (216) an augmented set of training data (180) to train the ML model by augmenting the first set of training data with the service quality metrics associated with the one or more UEs in accordance with one or more transport patterns (130) in the network-level metrics and based on a set of predetermined rules.

Description

[0001] TRAINING DATA AUGMENTATION

[0002] TECHNICAL FIELD

[0003] This application relates generally to network management, and more particularly to a system and method for training a machine learning, ML, model configured to predict service quality.

[0004] BACKGROUND

[0005] Per session, event-based analytics systems are part of the network management domain. For example, such systems can be implemented at a Management Data Analytics Function (MDAF) in an Operations, Administration, and Management (OA&M) domain, or at a Network Data Analytics Function (NWDAF) in the core network. Regardless of where they are located, however, analytics systems such as these are based on collecting and correlating elementary network events received or obtained from different network domains, such as the core network, the Radio Access Network (RAN), and the transport networks. Conventionally, these systems calculate radio and network Key Performance Indicators (KPIs) that characterize the radio / environment or network operation at one or both of a user level and a session level. Conventional analytics systems also implement service quality models to estimate end user Quality of Service (QoS) and Quality of Experience (QoE) characteristics. These types of solutions are suitable for performing session-based troubleshooting and / or analyzing network issues.

[0006] SUMMARY

[0007] The present disclosure provides, inter alia, an analytics system configured to augment a set of training data obtained from a first communications network with a large amount of data obtained from a live production network. To accomplish this, the present embodiments use session segments that have similar transport parameter patterns as those in the first communications network.

[0008] Accordingly, in a first aspect, the present disclosure provides a method, for an analytics system operating in a mobile communications network, for training a machine learning, ML, model configured to predict service quality. In this aspect, the method comprises obtaining a first set of training data for a ML model configured to predict service quality, wherein the first set of training data is generated by a first communications network and comprises network-level metrics and application-level metrics, receiving service quality metrics associated with one or more User Equipments, UEs, operating in a second mobile communications network, wherein the second mobile communications network is a live production mobile communications network, and generating an augmented set of training data to train the ML model configured to predict service quality by augmenting the first set of training data with the service quality metrics associated with the one or more UEs according to one or more transport patterns in the network-level metrics and based on a set of predetermined rules. A second aspect of the present disclosure provides a network node, for an analytics system operating in a mobile communications network, for training a machine learning, ML, model configured to predict service quality. In this aspect, the network node comprises communications circuitry configured to communicatively connect the network node to a communications network and processing circuitry operatively connected to the communications circuitry. In this embodiment, the processing circuitry is configured to obtain a first set of training data for a ML model configured to predict service quality, wherein the first set of training data is generated by a first communications network and comprises network-level metrics and application-level metrics, receive service quality metrics associated with one or more User Equipments, UEs, operating in a second mobile communications network, wherein the second mobile communications network is a live production mobile communications network, and generate an augmented set of training data to train the ML model configured to predict service quality by augmenting the first set of training data with the service quality metrics associated with the one or more UEs according to one or more transport patterns in the network-level metrics and based on a set of predetermined rules.

[0009] A third aspect of the present disclosure provides a network node, for an analytics system operating in a mobile communications network, for training a machine learning, ML, model configured to predict service quality. In this aspect, the network node is configured to obtain a first set of training data for a ML model configured to predict service quality, wherein the first set of training data is generated by a first communications network and comprises network-level metrics and application-level metrics, receive service quality metrics associated with one or more User Equipments, UEs, operating in a second mobile communications network, wherein the second mobile communications network is a live production mobile communications network, and generate an augmented set of training data to train the ML model configured to predict service quality by augmenting the first set of training data with the service quality metrics associated with the one or more UEs according to one or more transport patterns in the network-level metrics and based on a set of predetermined rules.

[0010] A fourth aspect of the present disclosure provides a non-transitory computer-readable storage medium comprising a computer program stored thereon. In this aspect, the computer program comprises executable instructions that, when executed by processing circuitry in a network node, causes the network node to obtain a first set of training data for a ML model configured to predict service quality, wherein the first set of training data is generated by a first communications network and comprises network-level metrics and application-level metrics, receive service quality metrics associated with one or more User Equipments, UEs, operating in a second mobile communications network, wherein the second mobile communications network is a live production mobile communications network, and generate an augmented set of training data to train the ML model configured to predict service quality by augmenting the first set of training data with the service quality metrics associated with the one or more UEs according to one or more transport patterns in the network-level metrics and based on a set of predetermined rules.

[0011] A fifth aspect of the present disclosure provides a computer program comprising instructions that, when executed by processing circuitry of a network node, cause the network node to perform the first aspect.

[0012] In a sixth aspect, the present disclosure provides a carrier containing the computer program described the fifth aspect. In this regard, the carrier is one of an electronic signal, optical signal, radio signal, or computer readable storage medium.

[0013] BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a functional block diagram illustrating some of the network domains and network elements involved in implementing a Machine Learning (ML) model, in accordance with embodiments of the present disclosure.

[0015] Figure 2 is a functional block diagram illustrating a process for training a ML model configured to predict service quality, in accordance with embodiments of the present disclosure.

[0016] Figure 3 is a graph illustrating the performance of a model relative to dataset size and a noise level applied to a target variable, in accordance with embodiments of the present disclosure.

[0017] Figure 4 is a flow diagram illustrating a method, implemented by an analytics system, for training a ML model configured to predict service quality, in accordance with embodiments of the present disclosure.

[0018] Figure 5 is a flow diagram illustrating a method for determining one or more patterns of degradation in a network, in accordance with embodiments of the present disclosure.

[0019] Figure 6 is a flow diagram illustrating a method for labeling service quality metrics associated with User Equipment (UEs) operating in a live, production network, in accordance with embodiments of the present disclosure.

[0020] Figure 7 is a block diagram illustrating components of a network node configured to train a ML model configured to predict service quality, in accordance with embodiments of the present disclosure.

[0021] Figure 8 is a functional block diagram illustrating an example of a communication system in accordance with embodiments of the present disclosure.

[0022] Figure 9 is a functional block diagram illustrating an example User Equipment (UE) in accordance with embodiments of the present disclosure.

[0023] Figure 10 is a functional block diagram illustrating an example network node in accordance with embodiments of the present disclosure.

[0024] Figure 11 is a functional block diagram of a host, which may be an embodiment of the host illustrated in Figure 8, in accordance with embodiments of the present disclosure.

[0025] Figure 12 is a functional block diagram illustrating a virtualization environment in which functions implemented by some embodiments of the present disclosure may be virtualized. Figure 13 is a functional block diagram illustrating a communication diagram of a host communicating via a network node with a UE over a partially wireless connection in accordance with embodiments of the present disclosure.

[0026] DETAILED DESCRIPTION

[0027] Embodiments of the present disclosure configure an analytics system to train machine learning (ML) models used for predicting the service quality of a communication network. As described in more detail below, the present embodiments configure the analytics system to augment a basic set of training data obtained from a first communications network with actual service quality data associated with one or more user devices (e.g., UEs) operating in a live, production network. As defined herein, a live, production network is an operational network actively engaged in transmitting and receiving live traffic (e.g., user data and / or control data / signals). Such live, production networks are distinguished from test networks, which are networks that execute in a controlled environment and are used to develop, deploy, and test the functions and services executing in a live, production network.

[0028] As stated above, per session, event-based analytics systems are part of the network management domain and are suitable for performing session-based troubleshooting and analysis of various network issues. However, event-based analytics systems are also used in Service Operation Centers (SOC). In such centers, event-based analytics systems monitor the quality of a wide variety of services used at a network level, as well as the quality of the customer experience at an individual per-subscriber level. These tools, and the information they provide, are widely used in customer care and other business scenarios.

[0029] Event-based analytics typically requires the real-time collection and correlation of characteristic node and protocol events from different radio and core nodes. Additionally, eventbased analytics can require the probing of signaling interfaces (IFs) as well as the sampling of user-plane traffic. To accomplish its functions, conventional analytics systems require, in addition to its data collection and correlation functions, an advanced database, a rule engine, and a “big data” analytics platform.

[0030] Generally, analytics systems obtain Key Performance Indicators (KPIs) related to the transport of the data through the network and the Quality of Service (QoS) from probe systems and / or user plane network function measurements. The Quality of Experience (QoE) and end- to-end (e2e) QoS metrics are estimated by service quality models, provided that such metrics are not directly available. Service quality models are service and application specific, and require appropriate training and training data. The published document WO 2024 / 023554, which was filed July 28, 2022, and is incorporated herein by reference in its entirety, describes such a system and method for collecting appropriate transport and radio data together with end client WebRTC statistics parameters and media recording. The measurement system is suitable for introducing different transport and radio degradations into a network to emulate actual network degradations. However, obtaining the needed data can be problematic. For example, the amount of data that is needed may not be available. Additionally, or alternatively, there may be privacy issues associated with the user’s data that must be considered. Therefore, synthetic data can be generated and used instead of real network data. Synthetic data generation is a technique used to generate new data points with a learning model that was previously trained using an original set of data. Synthetic data generation can be used to overcome the privacy issues that operators and vendors often face in live networks. Additionally, synthetic data generation models can be configured to identify the key statistical features of the original, possibly sensitive dataset, and generate any amount of new data that follows those key statistical features when the exact original data points cannot be retrieved or obtained. Alternatively, it is possible to combine such newly-generated data (i.e., the synthetic data) with the original data if, for example, the original data has small sample size. To accomplish such tasks, there are various synthetic data generation tools currently available to the public. Such tools include, but are not limited to, YData, Mostly. Al, and synthpop.

[0031] Generally, training datasets should be augmented in situations where there is not enough row data to train an ML model. One option for augmenting the training datasets is to use synthetic data generation - i.e., train a generative model using available data and then use the trained generative model to generate one or more similar models. Typically, however, this usually results only in generalizing the dataset or adding noise to the dataset. The best results are expected in cases when new information, such as the labelling or filtering of the generated synthetic data by Subject Matter Experts (SMEs), can be added to the synthetic data in the training dataset.

[0032] Network data coming from a network rarely has an assigned label. This is because operators rarely have access to application level QoE / QoS data. Further, these are limited to test phones, drive tests, and / or specific applications deployed to UEs, which require user consent.

[0033] For all their usefulness, however, conventional solutions can be problematic. For example, service quality models should be developed or trained for each service and application type. However, such training often requires a large number of training sessions for a wide variation of network conditions. Further, the training data also requires labels.

[0034] In a test system, it is difficult to realize real radio degradation scenarios and obtain enough training data for such degradation scenarios. Additionally, in actual commercial or live, production networks, access to the user terminals and end user / data needed for labeling are either limited or not available.

[0035] Live operating networks (i.e., live production networks), on the other hand, are often well-optimized. Typically, this means that most of the communication sessions experience good network conditions. However, service quality models should be trained for different degradation scenarios as well. Training service quality models also requires obtaining as much training data from suboptimal network conditions as possible. However, the test equipment that obtains such data does not always provide enough data. Further, the user data of real users should also be used to train the models as well. This last aspect, however, can be especially problematic due to the privacy concerns that surround the data of real users.

[0036] Labeling data, therefore, typically requires manual work and human test users. Additionally, conventional labeling methods are expensive and time consuming when performed over a large number of sessions. Further, conventional test systems are typically only able to access a limited number of sessions in a test network, and thus, are only able to collect a limited number of labels. Accordingly, collecting an appropriate amount of labels for the data while also covering a wide range of network degradation scenarios using conventional techniques and systems is very time consuming.

[0037] The present embodiments address these and other issues related to conventional techniques and systems by providing a multi-stage data augmentation method for a mobile network analytics system. In a first stage, one or more transport and / or radio degradations are artificially introduced into a first communications network to obtain a set of basic training data. The normal values of the network-wide measurable radio and transport data are also determined using the set of basic training data. Additionally, end user service quality metrics for the one or more transport and / or radio degradations are determined and used to label a set of primary data obtained in a second stage of the data augmentation method.

[0038] More particularly, the set of primary data is obtained from a second communications network in the second stage. According to the present embodiments, the set of primary data is produced by the second communications network in response to one or more “naturally occurring” primary transport and / or radio degradations in the second communications network. That is, the one or more primary transport and / or radio degradations are not artificially introduced into the second communications network as they were in the first communications network. Rather, they occur “naturally” as part of normal network operation. Then, using multiple different, “live” or “real” communication sessions (i.e., not test sessions) occurring in the second communications network, the set of primary data is analyzed and searched to determine whether there are any patterns that match those of the primary transport and / or radio degradations. If a pattern is found, session segments corresponding to the determined pattern are saved and used to augment the set of basic training data.

[0039] The present embodiments augment the set of basic training data under the following conditions using the following options.

[0040] • If there is a sufficient number of similar communication sessions (e.g., drive tests or test users) for which end user service quality is available, then the observed service quality is used for labeling the set of basic training data. • If the end user service quality is not available, however, a measurable parameter that correlates well with the end user service quality is used for labeling.

[0041] • If there is no such measurable parameter, the same labels used to label the set of primary data are used to label the set of basic training data.

[0042] The set of primary data and the set of basic training data are then used as training data for training service quality models.

[0043] It should be noted here that in at least one embodiment, the first communications network is a test network or test system into which a network operator can artificially introduce the one or more transport and / or radio degradations. The second communications network, however, is a live, production mobile communications network. However, those of ordinary skill in the art should understand that the present embodiments are not so limited. In another embodiment, for example, the first communications network is also a live, production network. Further, the first communications network, regardless of whether it is a test network or a live, production network, can be a mobile communications network or a wired communication network.

[0044] Embodiments of the present disclosure provide advantages and benefits that conventional methods and systems do not provide. For example, conventional methods and systems require a human to identify patterns and label the data accordingly. Such manual processes, however, are labor intensive, time consuming, and often lead to inaccurate labeling. Contrast this with the present embodiments, which also allow for the automated labelling of data. In doing so, the present embodiment negate the labor and time constraints associated with conventional human-based techniques and avoid or greatly reduce the possibility of inaccurate labeling. Moreover, as will be described in more detail below, embodiments of the present disclosure ensure the data privacy of the mobile network subscribers. This, as those of ordinary skill in the art will readily appreciate, is an important basic requirement for analytics data collection.

[0045] Additionally, using actual network data produced by a live, production network increases the amount of training data for training the ML models. This significantly improves the accuracy of the ML models in situations where the amount of training data available for training the ML models (e.g., test data obtained from a test network) is insufficient.

[0046] Using the actual network data to augment the other training data, such as the test data, is advantageous for other reasons as well. For example, test data produced by a test (or other) network is based on a limited number of input parameters and degradation types. In comparison, the actual network data produced by a live production network provides a larger variation of input parameters and represents a greater number of actual network degradations across multiple radio and transport parameters. Thus, using actual network data produced under real, actual network conditions to augment the test data trains ML models to detect service quality issues more accurately than do ML models trained using conventional methods. This, in turn, enables the more accurate estimation of service quality degradations in live production networks.

[0047] Embodiments of the present disclosure also define an advantage over conventional systems in that the present embodiments are able to directly account for network degradations when labels (i.e., direct labels or intermediate labels) used for labeling the data associated with the network degradations are available in the live production networks. Such labels, while they may exist in a live production system, are not always available in a test network. Nor is it possible to emulate and / or generate these network degradations in test networks. Therefore, augmenting the test data used to train the ML models greatly increases, inter alia, the types of network degradations that can be detected by the ML models.

[0048] Similarly, the present embodiments are configured to classify new network degradation scenarios for which the test network may not have any direct labels. In conventional systems, test networks are not configured to perform this function. Therefore, to accomplish this function, embodiments of the present disclosure classify such degradations based on measurable transport parameters and labeling associated with test scenarios run on / in the test network. By way of example only, consider a degradation such as interference. Interference can cause certain packet loss and jitter issues. These issues can be measured test scenarios with labels, and further, be applied to interference test cases with certain inaccuracy.

[0049] Additionally, with conventional systems, it is not possible to emulate a plurality of transport degradations (e.g., jitter, loss, delay, etc.) simultaneously. Such emulation, however, is possible with the embodiments of the present disclosure. Further, labeling in live production networks is historically inaccurate (e.g., in cases where intermediate parameters are present and / or where the same labels are used). The present embodiments, however, compensate for this by using a larger volume of training data (i.e., the training data received from a first network, such as a test network, augmented with the training data obtained from the live production network. Not only does this improve the accuracy of the service quality model, but it also negates the need for access to the UEs and the QoE feedback from the UEs.

[0050] Moreover, as stated previously, the present embodiments are specially configured to handle a wide variety of privacy-related issues. For example:

[0051] • Data that is measured and obtained in a test network does not have privacy issues, because it is owned by the network operator.

[0052] • Labeled data obtained in a live, production network that is generated by test phones owned by the operator have no privacy issues. This is because the test phones are owned by the network operators. Even when generated by actual users in a test network, however, those users have generally consented to share the information with the network operator. Therefore, there are no privacy issues here either.

[0053] • In cases where consent from actual users is not available, labels are not collected from the UE. However, the network measurements on encrypted data streams are still measured. Use of these measurements does not require user consent because it does not reveal the content of the encrypted data. Further, these measurements are required for proper operation of the network.

[0054] Turning now to the drawings, Figure 1 is a functional block diagram illustrating a system architecture 10 configured to support the present embodiments as closed loop actions based on decisions made by an analytics function. As seen in Figure 1 , system architecture 10 comprises a Network Management (NM) domain 20 comprising a Management Data Analytics Function (MDAF) 22, a Radio Access Network (RAN) domain 30 comprising one or more gNodeBs 32, and a Core Network (CN) domain 40 comprising a Network Data Analytics Function (NWDAF) 42.

[0055] The MDAF 22, which comprises or has access to one or more ML models 24, typically collects and correlates data from various entities in other network domains. These include, but are not limited to, the NWDAF 42 in CN domain 40. The correlated data is rich, and as such, allows for (i.e., supports) a large number of analytics use cases. The ML model(s) 24 in MDAF 22 are configured to either provide predictions about future network conditions or propose reconfiguration actions to be implemented, for example, in the radio network.

[0056] The NWDAF 42 in CN domain 40 can be tightly coupled to individual CN functions (e.g., an Access and Mobility Management Function (AMF) 46, a Session Management Function (SMF) 48, and a User Plane Function (UPF) 50 configured to communicate with one or more server devices in an external cloud network 70), or it can be configured to provide central analytics functions. As seen in Figure 1 , NWDAF 42 collects or obtains data (e.g., node events and / or probe reports) mainly from the entities in CN domain 40 (i.e., AMF 46, SMF 48, UPF 50), but RAN data from RAN domain 30 is also accessible to NWDAF 42 via the CAM domain (i.e., NM domain 20). In operation, NWDAF 42 leverages one or more ML models 44 primarily for predicting future values of analytics data regardless of whether that data is or is not directly measurable or available to the analytics system of the present embodiments. Based on these predictions, or estimated service quality values, one or more controlling nodes, such as a Policy Control Function (PCF) 52, can make intelligent decisions, mitigate degradations, improve QoE, and / or decrease costs.

[0057] The RAN domain 30, as stated above, includes one or more gNodeBs 32. As is known in the art, gNodeBs are responsible for effecting radio communications with one or more UEs, and more particularly, with one or more applications 60 that are executing on, and / or are accessible to, those UEs. In operation, the gNodeBs 32 provide radio-centric measurements and other information to the MDAF 22 in NM domain 20. The MDAF 22, in some embodiments, will provide the information received from the gNodeBs 32 to the NWDAF 42.

[0058] Regardless of whether the present embodiments are implanted at the MDAF 22 or at the NWDAF 42, the ML models 24, 44 that calculate service quality are key enablers of closed loops. The goal of these closed loops is to improve service quality, while the goal of other closed loops is to decrease costs without unacceptable service quality degradations. However, service quality labels are usually not available at the application level in these analytics systems. Therefore, there is a lack of training data for these ML models 24, 44. The present embodiments, however, address these issues.

[0059] Figure 2 is a functional block diagram illustrating a process 80 for training a ML model (e.g., ML model 24 and / or ML model 44), and / or for predicting service quality, according to embodiments of the present disclosure. More specifically, Figure 2 illustrates a Testing and Measurement Collection system 90 having a first network 100, and a second network 150 having an analytics system 160. The first network 100 in this embodiment is a test network. However, those of ordinary skill in the art should appreciate that the present disclosure is not so limited. In other embodiments, for example, first network 100 is a live production network. Additionally, in some embodiments, first network 100 may be a wired network or a mobile communications network. Regardless, according to the present embodiments, Testing and Measurement Collection system 90 is deployed proximate the first network 100 and collects test data. In this embodiment, collecting the test data may be realized by drive tests or one or more UEs 102a, 102b, 102c (collectively, UEs 102) equipped with special data collection software. The first network 100 is equipped with data collection facilities for the collection of counter and / or event data produced by the network functions. The goal is to correlate network-side and application-side data to the same set of sessions. In one embodiment, for example, such data may be Time Series Data of Network and Application level Quality of Service (QoS)ZQuality of Experience (QoE) data 110.

[0060] The data 110 collected by the Testing and Measurement Collection system 90 is valuable because the data is “labelled” and can therefore be utilized to train ML models. Particularly, network-level data can be tagged with corresponding application-level metrics. However, this type of data collection is time consuming and costly. Due to these constraints, conventional systems do not collect the amount of labelled data that is needed to train ML models.

[0061] The data that is collected from the Testing and Measurement Collection system 90 is, as seen in Figure 2, collected in a controlled environment, such as a “lab” environment, and can include, but is not limited to, data collected in accordance with the application of one or more of the following degradation scenarios.

[0062] • Radio shielding, low coverage, varying shielding levels;

[0063] • Handover (soft handover in same frequency, hard handover: inter-frequency Handover, I RAT handover, softer Handover, X2 handover, S1 handover, network, UE controlled handover;

[0064] • Packet loss (e.g., 5-40%);

[0065] • Delay, jitter (e.g., 50-200 ms);

[0066] • Bandwidth limitation (e.g., 200-2000 kbps); • Background traffic (e.g., 1-2 Mbps);

[0067] • Bit corruption (e.g., 0.1-1%); and

[0068] • A combination two or more of the above degradation scenarios.

[0069] Once collected, ML classification methods are used to identify patterns in the networklevel data. By way of example, such patterns may indicate a sudden increase / decrease / peak of certain KPIs for a given time period and are indicative of changes in the network environment that influence application-level metrics. In some embodiments, a Subject Matter Expert (SME) 120 may analyze the patterns, select the patterns 130 that are considered important, and annotate the selected patterns 130 with meaningful, descriptive explanations.

[0070] It should be noted here that the present embodiments do not require the creation of a full-fledged ML model for predicting the application QoS / QoE. Rather, according to the present disclosure, it is enough to identify only the relevant patterns.

[0071] The selected patterns 130 identified by the SME 120 are the same patterns that will be searched for by the analytics system 130 associated with the second network 150. Examples of such patterns 130 include, but are not limited to, the following example patterns indicating that:

[0072] • an average packet loss ratio in a given time period is between 0-5%, 5-10%, etc.;

[0073] • an average jitter of 0-20 ms, 20-40 ms, etc.;

[0074] • an RSRP is smaller than -120 dBm;

[0075] • an RSRP is between -120 dBm and -110 dBm, etc.

[0076] Additionally, the present disclosure allows for the identification of patterns associated with any number of combined scenarios for which test data is available. For example, one such “combined” pattern may indicate that an RSRP is -120 dBm to -110 dBm with packet loss at 5% - 10%. Further, in at least one embodiment, the average values of one or more test scenarios can be defined for detecting changes in KPI values. For example, in one embodiment, the present disclosure may define one pattern as an average packet loss that increases from 5% to 10%, and another pattern as an RSRP that drops from -90 dBm to -110 dBM. Of course, other patterns tied to other data threshold values and / or ranges are also possible.

[0077] The first and second networks 100, 150 may or may not be the same network. Regardless, the second network 150 is a live, production network that handles actual users and actual traffic in an actual environment (e.g., UEs 170a, 170b, and 170c, hereinafter, UEs 170). As stated above, the second network 150 is equipped with the analytics system 160, which according to the present disclosure, is configured to collect, filter, and analyze network level data, as well as host one or more ML models 164. Based on the patterns 130 defined by the SME 120, filters 162, which are deployed in the analytics system 160, identify sessions and time intervals where the data of those sessions matches the patterns 130 defined by the SME 120. According to the present embodiments, all relevant session-level, cell-level, NF-level, and network data associated with the identified sessions are saved for the identified time intervals. In accordance with the present disclosure, the amount of saved data is only a fraction of the total amount of data produced in the second network 150. Additionally, after removing personally identifiable information (e.g., the IMSI, SUPI, etc.), these small fractions of data can be considered safe from a data privacy perspective. For example, absent such information, it is not possible to track a single user for a long period of time and identify the person based on a daily geographical track.

[0078] Next, embodiments of the present disclosure label the data 140 that is collected from UEs 170 and processed by various network entities in the CN domain 40, for example, with assumed application-level QoS / QoE labels. This labeling is based, at least in part, on rules created by the SME 120 that defined the patterns 130. The result is labelled data 180, where the amount of labeled data 180 is much greater than the amount of time-series data 110 obtained from Testing and Measurement Collection system 90. As previously described, this provides an important advantage over conventional analytics systems. Specifically, the increased amount of labeled data 180 means that the ML models 190 trained with this data will be able to detect a wider variety of network degradations more accurately in the second network 150. On the other hand, it is possible that labelling the data at this stage may be less accurate than the labeled data obtained from Testing and Measurement Collection system 90. However, as described in more detail below, the increased amount of training data achieved by labeling the collected data 140 outweighs the effects of any possible labeling inaccuracies.

[0079] Then, as previously described, the increased amount of labelled data 180 is used to train an ML model 190. Once trained, the ML model 190 is deployed in the analytics system 160 of first network 150 (i.e., as ML model 164). The analytics system 160 then uses the trained ML model 164 to predict application-level QoS / QoE metrics 200 from data that is provided by UEs 170 and collected and processed by the first network 150.

[0080] The goal of the data augmentation, therefore, is to train an ML model 190 to more accurately estimate a QoS / QoE or any other service quality parameter. According to the present disclosure, this is accomplished by augmenting the set of training data generated by Testing and Measurement Collection system 90. However, there are various techniques that can be used to implement this function. For example, one embodiment of the present disclosure uses the following method and strategy to select which sessions to use to augment the set of “basic test data” generated by Testing and Measurement Collection system 90.

[0081] Particularly, the present embodiments first determine whether any drive test data for which the targeted service quality parameter is measured exists. If so, the sessions associated with that drive test data are selected because they provide accurate labels for labeling the collected data 140. However, if no drive tests were performed for situations where the targeted service quality parameter was measured, or if there is an insufficient amount of drive test data for such situations, the present embodiments will select normal user sessions occurring in second network 150 for labeling the collected data 140 based on one or more KPI patterns identified by SME 120 in the pattern recognition stage.

[0082] In at least one embodiment, the selection of a session for labeling the collected data 140 is based on the patterns of one or more test degradation scenarios performed in the Testing and Measurement Collection system 90. By way of example only, consider a test degradation scenario where, for a service of service type A, the pattern indicates a 10% packet loss with a 50 ms jitter. According to the present embodiments, only sessions in the second network 150 that use a service of service type A and that experience the same degradations as those in the test scenario (i.e., 10% packet loss and a 50 ms jitter) are searched. This allows the analytics system 160 to identify labels from sessions that experienced the same conditions as those experienced in the Testing and Measurement Collection system 90.

[0083] In another example, consider a test degradation scenario in the Testing and Measurement Collection system 90 where a service indicates a radio signal strength of -110 dBm (e.g., RSRP = -110 dBm). In such scenarios, only those sessions in the second network 150 that experienced the same degradation scenario applied in the Testing and Measurement Collection system 90 will be searched for use in labeling the collected data 140.

[0084] Typically, there are many different sessions that occur in the second network 150 that experience this same degradation scenario. Therefore, in at least one embodiment, the present disclosure can be configured to filter the sessions further based on some additional aspect experienced by both the Testing and Measurement Collection system 90 and the second network 150. Such aspects include, but are not limited to, one or more of the same packet loss, the same delay, and the same jitter. Regardless of the particular aspect, though, the present embodiments will, for the same scenarios, label the data collected in the second network (i.e., collected data 140) with the same QoE as in the Testing and Measurement Collection system 90.

[0085] In practice, not all degradation scenarios can be executed in a lab (e.g., the Testing and Measurement Collection system 90). However, according to the present embodiments, the data obtained from such systems can be augmented with data obtained from new test scenarios.

[0086] For instance, assume that interference (e.g., Reference Signals Received Quality (RSRQ)) degradation scenarios cannot be executed in a test lab, such as Testing and Measurement Collection system 90. In such cases, the present embodiments are configured to select sessions for the required RSRQ values from the live production network (e.g., second network 150), and then obtain the transport and radio data for these sessions. The present embodiments then label the data obtained from the live production network with QoE value(s) measured in the Testing and Measurement Collection system 90 for other degradation scenarios having the same transport and radio degradation values.

[0087] Additionally, the present disclosure also configures the analytics system 160 to refine the labeling. Particularly, if there are measurable service quality parameters, such as WebRTC parameters, in the live production network that are closely related to the QoE parameters, then the present embodiments can use these measurable service quality parameters to label the collected data 140 instead of the labels obtained from the Testing and Measurement Collection system 90. For example, in some situations, it may not be possible to measure QoE directly in the network. However, the actual video resolution may be available from signaling information, for example. Such signaling information is closely related to end user experienced service quality, and thus, the actual video resolution can be used to label the collected data 140.

[0088] In a more detailed example, consider a situation where video resolution is measured in the Testing and Measurement Collection system 90. At a video resolution of 820p, the measured QoE = 3.5, and at a video resolution of 420p, the measured QoE average resolution is 3.1. Given this information, and at a 10% packet loss degradation scenario in the live production network (e.g., second network 150), the video in the live production network will be labeled with a QoE = 3.5 label when the video resolution is 820p, and with a QoE = 3.1 label when the video resolution is 420p.

[0089] Alternatively, consider a situation where the QoE label associated with an average 10% packet loss is 3.3. In such cases, the QoE label of 3.3 for average packet loss can be used to label the video in the live production network. This is because the video resolution parameter is closer to the QoE than is the average packet loss.

[0090] As stated above, the ML models (e.g., ML model 164) take various radio and core transport parameters as input. Such parameters include, but are not limited to: Packet Parameters

[0091] • Packet loss ratio

[0092] • Delay

[0093] • Round-Trip-Time (RTT);

[0094] • Jitter; and

[0095] • Packet interarrival time.

[0096] Burst parameters:

[0097] • Burst length;

[0098] • Burst separation length;

[0099] • Burst size; and

[0100] • Burst rate.

[0101] Flow parameters:

[0102] • Bitrate; and

[0103] • Throughput.

[0104] Radio parameters:

[0105] RSRP;

[0106] RSRQ; • Signal to Interference & Noise Ratio (SINR);

[0107] • PPUCCH;

[0108] • PPUSCH;

[0109] • Frequency; and

[0110] • Radio Access Technology (RAT) type.

[0111] Additionally, the ML model 164, trained according to the present embodiments, will output various, service-specific KPIs. Such KPIs include, for example:

[0112] WebRTC services:

[0113] • trackidentifier;

[0114] • framesDecoded;

[0115] • keyFramesDecoded;

[0116] • frameWidth;

[0117] • frameHeight;

[0118] • framesPerSecond;

[0119] • qpSum;

[0120] • totalDecodeTime;

[0121] • totallnterFrameDelay;

[0122] • totalSquaredlnterFrameDelay;

[0123] • lastPacketReceivedTimestamp;

[0124] • headerBytesReceived;

[0125] • packetsDiscarded;

[0126] • fecPacketsReceived;

[0127] • fecPacketsDiscarded;

[0128] • bytes Received;

[0129] • nackCount;

[0130] • firCount;

[0131] • pliCount;

[0132] • totalProcessingDelay;

[0133] • estimatedPlayoutTimestamp;

[0134] • jitterBufferDelay;

[0135] • jitterBufferTargetDelay;

[0136] • jitterBufferEmittedCount;

[0137] • jitterBufferMinimumDelay;

[0138] • totalSamplesReceived;

[0139] • concealedSamples;

[0140] • silentConcealedSamples;

[0141] • concealmentEvents;

[0142] • insertedSamplesForDeceleration; • removedSamplesForAcceleration;

[0143] • audioLevel;

[0144] • totalAudioEnergy;

[0145] • totalSamplesDuration;

[0146] • framesReceived; and

[0147] • audio or video QoE or MOS.

[0148] Streaming video:

[0149] • video resolution;

[0150] • frame loss ratio;

[0151] • buffer health;

[0152] • connection speed;

[0153] • stall time ratio; and

[0154] • QoE or MOS.

[0155] As stated above, increasing the amount of data with which an ML model (e.g., ML model 164) is trained, such as is done in the present embodiments, outweighs the negative effects of any possible labeling inaccuracies. This aspect is shown in the example of Figure 3, which is a graph illustrating the performance of a ML model 164 relative to dataset size and a noise level applied to a target variable, in accordance with embodiments of the present disclosure.

[0156] In more detail, a simulation environment was designed to validate assumptions regarding how different levels of labeling accuracy may affect the machine learning quality. In this example, a time-series dataset containing measurements of radio and core parameters under different conditions is used. The selected target parameter in this example is jitter, which was predicted from other parameters. The goal of the simulation is not to find an optimal solution for jitter prediction. Rather, the goal is to obtain a ML model that is sufficiently trained to validate the assumptions.

[0157] As seen in Figure 3, the results show that the increased size of the training dataset can yield better ML training despite the noisy labeling. Particularly, the uncertainty associated with the different labeling techniques was simulated by introducing different random noise levels to the target while repeating the training. Further, the amount of data the ML model can train on was controlled by starting from only 20% of the available data and increasing it gradually. The result of these simulations is shown in Figure 3.

[0158] In this embodiment, the evaluation of the models occurs using a loss function, which is an error-like metric. Figure 3 illustrates the training results along different noise levels and with different training dataset sizes. As seen in this figure, the increased amount of training data will benefit the ML model as the loss function decreases by using larger datasets to train the ML model. This remains true even in cases of high noise levels.

[0159] Connecting the simulation results to the augmentation cases, the average loss (i.e., the y-axis) refers to a QoE parameter. For example, in one embodiment: • 0% noise corresponds to cases where the basic test dataset (i.e. , the data generated by first network 100) is augmented with drive test data measured in the live production network (i.e., the second network 150). In the drive tests, the QoE is directly measured.

[0160] • 20% noise corresponds to cases where the QoE parameter cannot be measured directly in the live production network, but other service quality parameters closely related to the QoE (e.g., WebRTC parameters) are used to augment and label the collected data 140.

[0161] • 40% noise corresponds to cases where WebRTC parameter measurements are not available. Therefore, the present embodiments label the augmented data using the QoE label of the test dataset that was used for selecting the data to be augmented (e.g. the same transport degradations).

[0162] • 60% noise corresponds to cases where there is no basic test data available for the primary degradation (e.g., interference). For example, in the embodiment of Figure 3, neither the QoE nor the WebRTC parameters are available for labeling the data to be augmented. Additionally, the selection of augmenting sessions was performed on equivalent transport patterns.

[0163] Thus, the augmentation method of the present disclosure can be applied to other use cases that lack a sufficient amount of training data. As seen in Figure 3, the accuracy of the ML model can be significantly improved simply by adding more data with which to train the ML model. Particularly, the improvement should be better than the uncertainty caused by the inaccurate labeling of the data to be augmented in the first network 150.

[0164] Accordingly, the present disclosure provides an analytics system 160 that is configured to augment the test data obtained from the first network 100 (e.g., in the Testing and Measurement Collection system 90) with a large amount of data obtained from a live production network, such as second network 150. The analytics system 160 augments the data using session segments having transport patterns that are the same as, or similar to, the transport patterns identified in the test data obtained from the first network 100.

[0165] To perform the data augmentation, the present embodiments select sessions from the first network 100 that also appear in the live production network (e.g., second network 150) and for which end user service quality information is available. When such information is not available, the present embodiments select sessions using one or more other parameters that are measurable in the live production network and that correlate well with service quality. Additionally or alternatively, the present embodiments may select sessions based on a measurable service quality parameter for labeling. In situations where previously used service quality parameters are not available, the present embodiments apply the same labels used when labeling the test data obtained from the first network 100 to the collected data 140 in the second network 150.

[0166] The techniques described in the present disclosure are particularly beneficial when less accurate labeling is sufficient but there are many variations in the input parameters. In such situations, adequately training an ML model requires a greater amount, and a wider range, of training data. However, in at least one embodiment of the present disclosure, it is determined whether the accuracy of the ML model increases when one or more labels associated with the service quality metrics are inaccurate (i.e., less than 100% accurate).

[0167] Figure 4 is a flow diagram illustrating a method 210, implemented by an analytics system 160, for training a ML model 164 configured to predict service quality, in accordance with embodiments of the present disclosure. As seen in Figure 4, the analytics system, which may be, for example, a network node implementing the functions of MDAF 22 or NWDAF 42, obtains a first set of training data for a ML model configured to predict service quality (box 212). In this embodiment, the first set of training data is generated by a first communications network 100 and comprises network-level metrics and application-level metrics. The analytics system 160 then receives service quality metrics associated with one or more UEs 170 operating in a second mobile communications network 150 (box 214). In this embodiment, the second mobile communications network 150 is a live production mobile communications network. The analytics system 160 then generates an augmented set of training data to train the ML model 164 configured to predict service quality (box 216). To generate the augmented set of training data, analytics system 160 augments the first set of training data with the service quality metrics associated with the one or more UEs 170 according to one or more transport patterns in the network-level metrics and based on a set of predetermined rules. The analytics system 160 can then train the ML model using the augmented set of training data (box 218), and thereafter, determine a QoS / QoE for a UE in the second mobile communications network 150 based on the ML model 164 trained using the augmented set of training data.

[0168] Figure 5 is a flow diagram illustrating a method 230 for determining one or more patterns of degradation in a network, in accordance with embodiments of the present disclosure. In more detail, the analytics system 160 first determines that the service quality metrics associated with the one or more UEs comprise a transport pattern that matches at least one transport pattern in the network-level metrics (box 232). In one embodiment, the transport pattern in the service quality metrics associated with the one or more UEs matches the transport pattern in the network-level metrics when, for a given session and time interval, data in the service quality metrics associated with the one or more UEs matches data in the transport pattern in the network-level metrics. To accomplish this function, one embodiment of the analytics system 160 filters the labeled network-level metrics in the first set of training data to determine one or more selected transport patterns (box 234). The analytics system 160 then compares the service quality metrics associated with the one or more UEs 170 to the labeled network-level metrics associated with the one or more selected transport patterns 130 (box 236). Then, based on a result of the comparison, the analytics system 160 determines that the service quality metrics associated with the one or more UEs 170 comprises at least one of the selected transport patterns 130 (box 238). The analytics system 160 then stores one or more session segments of a communication session in the second mobile communications network 150 responsive to determining that the service quality metrics associated with the one or more UEs 170 comprises the at least one of the selected transport patterns (box 240).

[0169] Figure 6 is a flow diagram illustrating a method 250 for labeling service quality metrics associated with one or more UEs 170 operating in a live, production network (e.g., second mobile communications network 150), in accordance with embodiments of the present disclosure. As seen in the embodiment of Figure 6, analytics system 160 generates an augmented set of training data to train an ML model 164 by labeling the service quality metrics associated with the one or more UEs 170 (box 252). For example, in one embodiment, analytics system 160 first determines whether the service quality metrics associated with the one or more UEs 170 comprises a service quality label (box 254). If so, the analytics system 160 labels the service quality metrics associated with the one or more UEs 170 using the service quality label (box 256). If not, analytics system 160 determines whether the service quality metrics associated with the one or more UEs 170 comprise a different service quality label that is related to a first service quality measure (box 258). If so, analytics system 160 labels the service quality metrics associated with the one or more UEs 170 using the related service quality label (box 260). If analytics system 160 determines that a related service quality label is not available, however (box 258), analytics system 160 labels the service quality metrics associated with the one or more UEs 170 using a service quality label that was previously used to label the networklevel metrics and / or the application-level metrics in the first set of training data when the network-level metrics and / or the application-level metrics were measured in the first communications network 100 (box 262).

[0170] In one embodiment, the first communications network is a live production communications network.

[0171] In another embodiment, the first communications network is a test communications network.

[0172] In at least one embodiment, the first communications network is a mobile communications network, while in other embodiments, the first communications network is a wired network.

[0173] In one embodiment, the first set of training data is generated by the first communications network based on network-level degradations present in the first communications network.

[0174] In one embodiment, the network-level degradations are introduced into the first communications network.

[0175] In one embodiment, the network-level metrics in the first set of training data is labeled according to one or both of the application-level metrics corresponding to the network-level metrics and the network-level degradations. In one embodiment, the one or more transport patterns in the network-level metrics are associated with the network-level degradations introduced into the first communications network.

[0176] In one embodiment, the one or more selected transport patterns against which the service quality metrics associated with the one or more UEs are compared represent a pattern of one or more of a packet loss ratio, jitter, a change in one or more Key Performance Indicators (KPI), interference, bit corruption, and Reference Signal Received Power (RSRP).

[0177] In one embodiment, the one or more selected transport patterns are time-qualified.

[0178] In one embodiment, the one or more selected transport patterns are relative to one or more of a predefined threshold value and a predefined range of values.

[0179] In one embodiment, it is determined whether an accuracy of the ML model increases when one or more labels associated with the service quality metrics are inaccurate.

[0180] In one embodiment, the network-level metrics and the application-level metrics in the first set of training data are service quality metrics.

[0181] In one embodiment, the first communications network and the second mobile communications network are the same networks.

[0182] In another embodiment, however, the first communications network and the second mobile communications network are different networks.

[0183] In one embodiment, the analytics system 160 of the present disclosure is implemented in a Network Data Analytics Function (NWDAF) in a core network (CN).

[0184] In one embodiment, the analytics system 160 of the present disclosure is implemented in a Management Data Analytics Function (MDAF) in a management network.

[0185] In one embodiment, at least one of the one or more transport patterns in the networklevel metrics represents a pattern of degradation in a radio network.

[0186] According to the present disclosure, an apparatus can perform any of the methods herein described by implementing any functional means, modules, units, or circuitry. In one embodiment, for example, the apparatuses comprise respective circuits or circuitry configured to perform the steps shown in the method figures. The circuits or circuitry in this regard may comprise circuits dedicated to performing certain functional processing and / or one or more microprocessors in conjunction with memory. For instance, the circuitry may include one or more microprocessors or microcontrollers, as well as other digital hardware, which may include Digital Signal Processors (DSPs), special-purpose digital logic, and the like. The processing circuitry may be configured to execute program code stored in memory, which may include one or several types of memory such as read-only memory (ROM), random-access memory, cache memory, flash memory devices, optical storage devices, etc. Program code stored in memory may include program instructions for executing one or more telecommunications and / or data communications protocols as well as instructions for carrying out one or more of the techniques described herein, in several embodiments. In embodiments that employ memory, the memory stores program code that, when executed by the one or more processors, carries out the techniques described herein.

[0187] Figure 7 is a block diagram illustrating some of the main functional components of a network node 300 configured to train a ML model 164 configured to predict service quality 200, in accordance with embodiments of the present disclosure. The network node 300 in this embodiment is configured to execute the analytics system 160 and comprises, inter alia, communication circuitry 302, processing circuitry 304, and memory 306.

[0188] In some embodiments, the communication circuitry 302 comprises both radio frequency (RF) circuitry 308 and network interface circuitry (NIC) 310. In other embodiments, however, the network node may comprise only NIC 310. More particularly, the RF circuitry 308 can be located at one or more TRPs and comprises the RF components necessary for communicating with UEs over a wireless communication link. According to the present embodiments, the RF circuitry 308 may comprise, for example, a transmitter and receiver configured to operate according to the 5G standards or other wireless communication standard.

[0189] The communication circuitry 302 also comprises network interface circuitry (e.g., NIC 310) for communication with other RAN nodes, OA&M nodes, core network nodes, and / or other nodes in external systems. The network interface circuitry 310 may, for example, comprise an Ethernet interface, optical network interface, or a wireless interface.

[0190] The processing circuitry 304 comprises one or more microprocessors, hardware, firmware, or a combination thereof that controls the overall operation of the network node 300. The processing circuitry 304 in this regard can be configured by software to perform one or more of the methods herein described, including the methods 210, 230, and 250 as shown in Figures 4-6, respectively.

[0191] Memory 306 comprises both volatile and non-volatile memory for storing computer program code and data needed by the processing circuitry 304 for operation. Memory 306 may comprise any tangible, non-transitory computer-readable storage medium for storing data including electronic, magnetic, optical, electromagnetic, or semiconductor data storage. Memory 306 stores a computer program 312 comprising executable instructions that configure the processing circuit 304 in the network node 300 to perform one or more of the methods herein described, including the methods 210, 230, and 250 as shown in Figures 4-6, respectively. A computer program 312 in this regard may comprise one or more code modules corresponding to the means or units described above.

[0192] In general, computer program instructions and configuration information are stored in a non-volatile memory, such as a ROM, erasable programmable read only memory (EPROM) or flash memory. Temporary data generated during operation may be stored in a volatile memory, such as a random access memory (RAM). In some embodiments, computer program 312 for configuring the processing circuitry 304 as herein described may be stored in a removable memory, such as a portable compact disc, portable digital video disc, or other removable media. The computer program 312 may also be embodied in a carrier such as an electronic signal, optical signal, radio signal, or computer readable storage medium.

[0193] Those skilled in the art will also appreciate that embodiments herein further include corresponding computer programs. A computer program comprises instructions which, when executed on at least one processor of an apparatus, cause the apparatus to carry out any of the respective processing described above. A computer program in this regard may comprise one or more code modules corresponding to the means or units described above.

[0194] Embodiments of the present disclosure further include a carrier containing such a computer program 312. This carrier may comprise one of an electronic signal, optical signal, radio signal, or computer readable storage medium.

[0195] In this regard, the embodiments described herein also include a computer program product stored on a non-transitory computer readable (storage or recording) medium and comprising instructions that, when executed by a processor of an apparatus, cause the apparatus to perform as described above.

[0196] Embodiments further include a computer program product comprising program code portions for performing the method of any of the embodiments herein when the computer program product is executed by a computing device. This computer program product may be stored on a computer readable recording medium.

[0197] Additional embodiments will now be described. At least some of these embodiments may be described as applicable in certain contexts and / or wireless network types for illustrative purposes, but the embodiments are similarly applicable in other contexts and / or wireless network types not explicitly described.

[0198] Figure 8 shows an example of a communication system 1100 in accordance with some embodiments.

[0199] In the example, the communication system 1100 includes a telecommunication network 1102 that includes an access network 1104, such as a radio access network (RAN), and a core network 1106, which includes one or more core network nodes 1108. The access network 1104 includes one or more access network nodes, such as network nodes 1110a and 1110b (one or more of which may be generally referred to as network nodes 1110), or any other similar 3rd Generation Partnership Project (3GPP) access nodes or non-3GPP access points. Moreover, as will be appreciated by those of skill in the art, a network node is not necessarily limited to an implementation in which a radio portion and a baseband portion are supplied and integrated by a single vendor. Thus, it will be understood that network nodes include disaggregated implementations or portions thereof. For example, in some embodiments, the telecommunication network 1102 includes one or more Open-RAN (ORAN) network nodes. An ORAN network node is a node in the telecommunication network 1102 that supports an ORAN specification (e.g., a specification published by the O-RAN Alliance, or any similar organization) and may operate alone or together with other nodes to implement one or more functionalities of any node in the telecommunication network 1102, including one or more network nodes 1110 and / or core network nodes 1108.

[0200] Examples of an ORAN network node include an open radio unit (O-RU), an open distributed unit (O-DU), an open central unit (O-CU), including an O-CU control plane (O-CU- CP) or an O-CU user plane (O-CU-UP), a RAN intelligent controller (near-real time or non-real time) hosting software or software plug-ins, such as a near-real time control application (e.g., xApp) or a non-real time control application (e.g., rApp), or any combination thereof (the adjective “open” designating support of an ORAN specification). The network node may support a specification by, for example, supporting an interface defined by the ORAN specification, such as an A1 , F1 , W1 , E1 , E2, X2, Xn interface, an open fronthaul user plane interface, or an open fronthaul management plane interface. Moreover, an ORAN access node may be a logical node in a physical node. Furthermore, an ORAN network node may be implemented in a virtualization environment (described further below) in which one or more network functions are virtualized. For example, the virtualization environment may include an O-Cloud computing platform orchestrated by a Service Management and Orchestration Framework via an 0-2 interface defined by the O-RAN Alliance or comparable technologies. The network nodes 1110 facilitate direct or indirect connection of user equipment (UE), such as by connecting UEs 1112A, 1112B, 1112C, and 1112D (one or more of which may be generally referred to as UEs 1112) to the core network 1106 over one or more wireless connections.

[0201] Example wireless communications over a wireless connection include transmitting and / or receiving wireless signals using electromagnetic waves, radio waves, infrared waves, and / or other types of signals suitable for conveying information without the use of wires, cables, or other material conductors. Moreover, in different embodiments, the communication system 1100 may include any number of wired or wireless networks, network nodes, UEs, and / or any other components or systems that may facilitate or participate in the communication of data and / or signals whether via wired or wireless connections. The communication system 1100 may include and / or interface with any type of communication, telecommunication, data, cellular, radio network, and / or other similar type of system.

[0202] The UEs 1112 may be any of a wide variety of communication devices, including wireless devices arranged, configured, and / or operable to communicate wirelessly with the network nodes 1110 and other communication devices. Similarly, the network nodes 1110 are arranged, capable, configured, and / or operable to communicate directly or indirectly with the UEs 1112 and / or with other network nodes or equipment in the telecommunication network 1102 to enable and / or provide network access, such as wireless network access, and / or to perform other functions, such as administration in the telecommunication network 1102.

[0203] In the depicted example, the core network 1106 connects the network nodes 1110 to one or more hosts, such as host 1116. These connections may be direct or indirect via one or more intermediary networks or devices. In other examples, network nodes may be directly coupled to hosts. The core network 1106 includes one more core network nodes (e.g., core network node 1108) that are structured with hardware and software components. Features of these components may be substantially similar to those described with respect to the UEs, network nodes, and / or hosts, such that the descriptions thereof are generally applicable to the corresponding components of the core network node 1108. Example core network nodes include functions of one or more of a Mobile Switching Center (MSC), Mobility Management Entity (MME), Home Subscriber Server (HSS), Access and Mobility Management Function (AMF), Session Management Function (SMF), Authentication Server Function (AUSF), Subscription Identifier De-concealing function (SIDF), Unified Data Management (UDM), Security Edge Protection Proxy (SEPP), Network Exposure Function (NEF), and / or a User Plane Function (UPF).

[0204] The host 1116 may be under the ownership or control of a service provider other than an operator or provider of the access network 1104 and / or the telecommunication network 1102, and may be operated by the service provider or on behalf of the service provider. The host 1116 may host a variety of applications to provide one or more service. Examples of such applications include live and pre-recorded audio / video content, data collection services such as retrieving and compiling data on various ambient conditions detected by a plurality of UEs, analytics functionality, social media, functions for controlling or otherwise interacting with remote devices, functions for an alarm and surveillance center, or any other such function performed by a server.

[0205] As a whole, the communication system 1100 of Figure 8 enables connectivity between the UEs, network nodes, and hosts. In that sense, the communication system may be configured to operate according to predefined rules or procedures, such as specific standards that include, but are not limited to: Global System for Mobile Communications (GSM); Universal Mobile Telecommunications System (UMTS); Long Term Evolution (LTE), and / or other suitable 2G, 3G, 4G, 5G standards, or any applicable future generation standard (e.g., 6G); wireless local area network (WLAN) standards, such as the Institute of Electrical and Electronics Engineers (IEEE) 802.11 standards (WiFi); and / or any other appropriate wireless communication standard, such as the Worldwide Interoperability for Microwave Access (WiMax), Bluetooth, Z-Wave, Near Field Communication (NFC) ZigBee, LiFi, and / or any low- power wide-area network (LPWAN) standards such as LoRa and Sigfox.

[0206] In some examples, the telecommunication network 1102 is a cellular network that implements 3GPP standardized features. Accordingly, the telecommunications network 1102 may support network slicing to provide different logical networks to different devices that are connected to the telecommunication network 1102. For example, the telecommunications network 1102 may provide Ultra Reliable Low Latency Communication (URLLC) services to some UEs, while providing Enhanced Mobile Broadband (eMBB) services to other UEs, and / or Massive Machine Type Communication (mMTC)ZMassive loT services to yet further UEs. In some examples, the UEs 1112 are configured to transmit and / or receive information without direct human interaction. For instance, a UE may be designed to transmit information to the access network 1104 on a predetermined schedule, when triggered by an internal or external event, or in response to requests from the access network 1104. Additionally, a UE may be configured for operating in single- or multi-RAT or multi-standard mode. For example, a UE may operate with any one or combination of Wi-Fi, NR (New Radio) and LTE, i.e. being configured for multi-radio dual connectivity (MR-DC), such as E-UTRAN (Evolved-UMTS Terrestrial Radio Access Network) New Radio - Dual Connectivity (EN-DC).

[0207] In the example, the hub 1114 communicates with the access network 1104 to facilitate indirect communication between one or more UEs (e.g., UE 1112c and / or 1112D) and network nodes (e.g., network node 1110B). In some examples, the hub 1114 may be a controller, router, content source and analytics, or any of the other communication devices described herein regarding UEs. For example, the hub 1114 may be a broadband router enabling access to the core network 1106 for the UEs. As another example, the hub 1114 may be a controller that sends commands or instructions to one or more actuators in the UEs. Commands or instructions may be received from the UEs, network nodes 1110, or by executable code, script, process, or other instructions in the hub 1114. As another example, the hub 1114 may be a data collector that acts as temporary storage for UE data and, in some embodiments, may perform analysis or other processing of the data. As another example, the hub 1114 may be a content source. For example, for a UE that is a VR headset, display, loudspeaker or other media delivery device, the hub 1114 may retrieve VR assets, video, audio, or other media or data related to sensory information via a network node, which the hub 1114 then provides to the UE either directly, after performing local processing, and / or after adding additional local content. In still another example, the hub 1114 acts as a proxy server or orchestrator for the UEs, in particular if one or more of the UEs are low energy loT devices.

[0208] The hub 1114 may have a constant / persistent or intermittent connection to the network node 1110B. The hub 1114 may also allow for a different communication scheme and / or schedule between the hub 1114 and UEs (e.g., UE 1112C and / or 1112D), and between the hub 1114 and the core network 1106. In other examples, the hub 1114 is connected to the core network 1106 and / or one or more UEs via a wired connection. Moreover, the hub 1114 may be configured to connect to an M2M service provider over the access network 1104 and / or to another UE over a direct connection. In some scenarios, UEs may establish a wireless connection with the network nodes 1110 while still connected via the hub 1114 via a wired or wireless connection. In some embodiments, the hub 1114 may be a dedicated hub - that is, a hub whose primary function is to route communications to / from the UEs from / to the network node 1110B. In other embodiments, the hub 1114 may be a non-dedicated hub - that is, a device which is capable of operating to route communications between the UEs and network node 1110B, but which is additionally capable of operating as a communication start and / or end point for certain data channels.

[0209] Figure 9 shows a UE 1200 in accordance with some embodiments. As used herein, a UE refers to a device capable, configured, arranged and / or operable to communicate wirelessly with network nodes and / or other UEs. Examples of a UE include, but are not limited to, a smart phone, mobile phone, cell phone, voice over IP (VoIP) phone, wireless local loop phone, desktop computer, personal digital assistant (PDA), wireless cameras, gaming console or device, music storage device, playback appliance, wearable terminal device, wireless endpoint, mobile station, tablet, laptop, laptop-embedded equipment (LEE), laptop-mounted equipment (LME), smart device, wireless customer-premise equipment (CPE), vehicle, vehicle-mounted or vehicle embedded / integrated wireless device, etc. Other examples include any UE identified by the 3rd Generation Partnership Project (3GPP), including a narrow band internet of things (NB- loT) UE, a machine type communication (MTC) UE, and / or an enhanced MTC (eMTC) UE.

[0210] A UE may support device-to-device (D2D) communication, for example by implementing a 3GPP standard for sidelink communication, Dedicated Short-Range Communication (DSRC), vehicle-to-vehicle (V2V), vehicle-to-infrastructure (V2I) , or vehicle-to-everything (V2X). In other examples, a UE may not necessarily have a user in the sense of a human user who owns and / or operates the relevant device. Instead, a UE may represent a device that is intended for sale to, or operation by, a human user but which may not, or which may not initially, be associated with a specific human user (e.g., a smart sprinkler controller). Alternatively, a UE may represent a device that is not intended for sale to, or operation by, an end user but which may be associated with or operated for the benefit of a user (e.g., a smart power meter).

[0211] The UE 1200 includes processing circuitry 1202 that is operatively coupled via a bus 1204 to an input / output interface 1206, a power source 1208, a memory 1210, a communication interface 1212, and / or any other component, or any combination thereof. Certain UEs may utilize all or a subset of the components shown in Figure 9. The level of integration between the components may vary from one UE to another UE. Further, certain UEs may contain multiple instances of a component, such as multiple processors, memories, transceivers, transmitters, receivers, etc.

[0212] The processing circuitry 1202 is configured to process instructions and data and may be configured to implement any sequential state machine operative to execute instructions stored as machine-readable computer programs in the memory 1210. The processing circuitry 1202 may be implemented as one or more hardware-implemented state machines (e.g., in discrete logic, field-programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), etc.); programmable logic together with appropriate firmware; one or more stored computer programs, general-purpose processors, such as a microprocessor or digital signal processor (DSP), together with appropriate software; or any combination of the above. For example, the processing circuitry 1202 may include multiple central processing units (CPUs). In the example, the input / output interface 1206 may be configured to provide an interface or interfaces to an input device, output device, or one or more input and / or output devices. Examples of an output device include a speaker, a sound card, a video card, a display, a monitor, a printer, an actuator, an emitter, a smartcard, another output device, or any combination thereof. An input device may allow a user to capture information into the UE 1200. Examples of an input device include a touch-sensitive or presence-sensitive display, a camera (e.g., a digital camera, a digital video camera, a web camera, etc.), a microphone, a sensor, a mouse, a trackball, a directional pad, a trackpad, a scroll wheel, a smartcard, and the like. The presence-sensitive display may include a capacitive or resistive touch sensor to sense input from a user. A sensor may be, for instance, an accelerometer, a gyroscope, a tilt sensor, a force sensor, a magnetometer, an optical sensor, a proximity sensor, a biometric sensor, etc., or any combination thereof. An output device may use the same type of interface port as an input device. For example, a Universal Serial Bus (USB) port may be used to provide an input device and an output device.

[0213] In some embodiments, the power source 1208 is structured as a battery or battery pack. Other types of power sources, such as an external power source (e.g., an electricity outlet), photovoltaic device, or power cell, may be used. The power source 1208 may further include power circuitry for delivering power from the power source 1208 itself, and / or an external power source, to the various parts of the UE 1200 via input circuitry or an interface such as an electrical power cable. Delivering power may be, for example, for charging of the power source 1208. Power circuitry may perform any formatting, converting, or other modification to the power from the power source 1208 to make the power suitable for the respective components of the UE 1200 to which power is supplied.

[0214] The memory 1210 may be or be configured to include memory such as random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic disks, optical disks, hard disks, removable cartridges, flash drives, and so forth. In one example, the memory 1210 includes one or more application programs 1214, such as an operating system, web browser application, a widget, gadget engine, or other application, and corresponding data 1216. The memory 1210 may store, for use by the UE 1200, any of a variety of various operating systems or combinations of operating systems.

[0215] The memory 1210 may be configured to include a number of physical drive units, such as redundant array of independent disks (RAID), flash memory, USB flash drive, external hard disk drive, thumb drive, pen drive, key drive, high-density digital versatile disc (HD-DVD) optical disc drive, internal hard disk drive, Blu-Ray optical disc drive, holographic digital data storage (HDDS) optical disc drive, external mini-dual in-line memory module (DIMM), synchronous dynamic random access memory (SDRAM), external micro-DIMM SDRAM, smartcard memory such as tamper resistant module in the form of a universal integrated circuit card (UICC) including one or more subscriber identity modules (SIMs), such as a USIM and / or ISIM, other memory, or any combination thereof. The UICC may for example be an embedded UICC (eUlCC), integrated UICC (iUICC) or a removable UICC commonly known as ‘SIM card.’ The memory 1210 may allow the UE 1200 to access instructions, application programs and the like, stored on transitory or non-transitory memory media, to off-load data, or to upload data. An article of manufacture, such as one utilizing a communication system may be tangibly embodied as or in the memory 1210, which may be or comprise a device-readable storage medium.

[0216] The processing circuitry 1202 may be configured to communicate with an access network or other network using the communication interface 1212. The communication interface 1212 may comprise one or more communication subsystems and may include or be communicatively coupled to an antenna 1222. The communication interface 1212 may include one or more transceivers used to communicate, such as by communicating with one or more remote transceivers of another device capable of wireless communication (e.g., another UE or a network node in an access network). Each transceiver may include a transmitter 1218 and / or a receiver 1220 appropriate to provide network communications (e.g., optical, electrical, frequency allocations, and so forth). Moreover, the transmitter 1218 and receiver 1220 may be coupled to one or more antennas (e.g., antenna 1222) and may share circuit components, software or firmware, or alternatively be implemented separately.

[0217] In the illustrated embodiment, communication functions of the communication interface 1212 may include cellular communication, Wi-Fi communication, LPWAN communication, data communication, voice communication, multimedia communication, short-range communications such as Bluetooth, near-field communication, location-based communication such as the use of the global positioning system (GPS) to determine a location, another like communication function, or any combination thereof. Communications may be implemented in according to one or more communication protocols and / or standards, such as IEEE 802.11 , Code Division Multiplexing Access (CDMA), Wideband Code Division Multiple Access (WCDMA), GSM, LTE, New Radio (NR), UMTS, WiMax, Ethernet, transmission control protocol / internet protocol (TCP / IP), synchronous optical networking (SONET), Asynchronous Transfer Mode (ATM), QUIC, Hypertext Transfer Protocol (HTTP), and so forth.

[0218] Regardless of the type of sensor, a UE may provide an output of data captured by its sensors, through its communication interface 1212, via a wireless connection to a network node. Data captured by sensors of a UE can be communicated through a wireless connection to a network node via another UE. The output may be periodic (e.g., once every 15 minutes if it reports the sensed temperature), random (e.g., to even out the load from reporting from several sensors), in response to a triggering event (e.g., when moisture is detected an alert is sent), in response to a request (e.g., a user initiated request), or a continuous stream (e.g., a live video feed of a patient). As another example, a UE comprises a mobile device, such as phone or tablet having a depth sensor, Augmented Reality (AR) glasses, a Mixed Reality (MR) device, a Virtual Reality (VR) device, an actuator, a motor, or a switch, related to a communication interface configured to receive wireless input from a network node via a wireless connection. In response to the received wireless input the states of the actuator, the motor, or the switch may change. For example, the UE may comprise a motor that adjusts the control surfaces or rotors of a drone in flight according to the received input or to a robotic arm performing a medical procedure according to the received input.

[0219] A UE, when in the form of an Internet of Things (loT) device, may be a device for use in one or more application domains, these domains comprising, but not limited to, city wearable technology, extended industrial application and healthcare. Non-limiting examples of such an loT device are a device which is or which is embedded in: a connected refrigerator or freezer, a TV, a connected lighting device, an electricity meter, a robot vacuum cleaner, a voice controlled smart speaker, a home security camera, a motion detector, a thermostat, a smoke detector, a door / window sensor, a flood / moisture sensor, an electrical door lock, a connected doorbell, an air conditioning system like a heat pump, an autonomous vehicle, a surveillance system, a weather monitoring device, a vehicle parking monitoring device, an electric vehicle charging station, a smart watch, a fitness tracker, a head-mounted display for Augmented Reality (AR) or Virtual Reality (VR), a wearable for tactile augmentation or sensory enhancement, a water sprinkler, an animal- or item-tracking device, a sensor for monitoring a plant or animal, an industrial robot, an Unmanned Aerial Vehicle (UAV), and any kind of medical device, like a heart rate monitor or a remote controlled surgical robot. A UE in the form of an loT device comprises circuitry and / or software in dependence of the intended application of the loT device in addition to other components as described in relation to the UE 1200 shown in Figure 9.

[0220] As yet another specific example, in an loT scenario, a UE may represent a machine or other device that performs monitoring and / or measurements, and transmits the results of such monitoring and / or measurements to another UE and / or a network node. The UE may in this case be an M2M device, which may in a 3GPP context be referred to as an MTC device. As one particular example, the UE may implement the 3GPP NB-loT standard. In other scenarios, a UE may represent a vehicle, such as a car, a bus, a truck, a ship and an airplane, or other equipment that is capable of monitoring and / or reporting on its operational status or other functions associated with its operation.

[0221] In practice, any number of UEs may be used together with respect to a single use case. For example, a first UE might be or be integrated in a drone and provide the drone’s speed information (obtained through a speed sensor) to a second UE that is a remote controller operating the drone. When the user makes changes from the remote controller, the first UE may adjust the throttle on the drone (e.g. by controlling an actuator) to increase or decrease the drone’s speed. The first and / or the second UE can also include more than one of the functionalities described above. For example, a UE might comprise the sensor and the actuator, and handle communication of data for both the speed sensor and the actuators.

[0222] Figure 10 shows a network node 1300 in accordance with some embodiments. As used herein, network node refers to equipment capable, configured, arranged and / or operable to communicate directly or indirectly with a UE and / or with other network nodes or equipment, in a telecommunication network. Examples of network nodes include, but are not limited to, access points (APs) (e.g., radio access points), base stations (BSs) (e.g., radio base stations, Node Bs, evolved Node Bs (eNBs) and NR NodeBs (gNBs)), O-RAN nodes or components of an O-RAN node (e.g., O-RU, O-DU, O-CU).

[0223] Base stations may be categorized based on the amount of coverage they provide (or, stated differently, their transmit power level) and so, depending on the provided amount of coverage, may be referred to as femto base stations, pico base stations, micro base stations, or macro base stations. A base station may be a relay node or a relay donor node controlling a relay. A network node may also include one or more (or all) parts of a distributed radio base station such as centralized digital units, distributed units (e.g., in an O-RAN access node) and / or remote radio units (RRUs), sometimes referred to as Remote Radio Heads (RRHs). Such remote radio units may or may not be integrated with an antenna as an antenna integrated radio. Parts of a distributed radio base station may also be referred to as nodes in a distributed antenna system (DAS).

[0224] Other examples of network nodes include multiple transmission point (multi-TRP) 5G access nodes, multi-standard radio (MSR) equipment such as MSR BSs, network controllers such as radio network controllers (RNCs) or base station controllers (BSCs), base transceiver stations (BTSs), transmission points, transmission nodes, multi-cell / multicast coordination entities (MCEs), Operation and Maintenance (O&M) nodes, Operations Support System (OSS) nodes, Self-Organizing Network (SON) nodes, positioning nodes (e.g., Evolved Serving Mobile Location Centers (E-SMLCs)), and / or Minimization of Drive Tests (MDTs).

[0225] The network node 1300 includes a processing circuitry 1302, a memory 1304, a communication interface 1306, and a power source 1308. The network node 1300 may be composed of multiple physically separate components (e.g., a NodeB component and a RNC component, or a BTS component and a BSC component, etc.), which may each have their own respective components. In certain scenarios in which the network node 1300 comprises multiple separate components (e.g., BTS and BSC components), one or more of the separate components may be shared among several network nodes. For example, a single RNC may control multiple NodeBs. In such a scenario, each unique NodeB and RNC pair, may in some instances be considered a single separate network node. In some embodiments, the network node 1300 may be configured to support multiple radio access technologies (RATs). In such embodiments, some components may be duplicated (e.g., separate memory 1304 for different RATs) and some components may be reused (e.g., a same antenna 1310 may be shared by different RATs). The network node 1300 may also include multiple sets of the various illustrated components for different wireless technologies integrated into network node 1300, for example GSM, WCDMA, LTE, NR, WiFi, Zigbee, Z-wave, LoRaWAN, Radio Frequency Identification (RFID) or Bluetooth wireless technologies. These wireless technologies may be integrated into the same or different chip or set of chips and other components within network node 1300.

[0226] The processing circuitry 1302 may comprise a combination of one or more of a microprocessor, controller, microcontroller, central processing unit, digital signal processor, application-specific integrated circuit, field programmable gate array, or any other suitable computing device, resource, or combination of hardware, software and / or encoded logic operable to provide, either alone or in conjunction with other network node 1300 components, such as the memory 1304, to provide network node 1300 functionality.

[0227] In some embodiments, the processing circuitry 1302 includes a system on a chip (SOC). In some embodiments, the processing circuitry 1302 includes one or more of radio frequency (RF) transceiver circuitry 1312 and baseband processing circuitry 1314. In some embodiments, the radio frequency (RF) transceiver circuitry 1312 and the baseband processing circuitry 1314 may be on separate chips (or sets of chips), boards, or units, such as radio units and digital units. In alternative embodiments, part or all of RF transceiver circuitry 1312 and baseband processing circuitry 1314 may be on the same chip or set of chips, boards, or units.

[0228] The memory 1304 may comprise any form of volatile or non-volatile computer-readable memory including, without limitation, persistent storage, solid-state memory, remotely mounted memory, magnetic media, optical media, random access memory (RAM), read-only memory (ROM), mass storage media (for example, a hard disk), removable storage media (for example, a flash drive, a Compact Disk (CD) or a Digital Video Disk (DVD)), and / or any other volatile or non-volatile, non-transitory device-readable and / or computer-executable memory devices that store information, data, and / or instructions that may be used by the processing circuitry 1302. The memory 1304 may store any suitable instructions, data, or information, including a computer program, software, an application including one or more of logic, rules, code, tables, and / or other instructions capable of being executed by the processing circuitry 1302 and utilized by the network node 1300. The memory 1304 may be used to store any calculations made by the processing circuitry 1302 and / or any data received via the communication interface 1306. In some embodiments, the processing circuitry 1302 and memory 1304 is integrated.

[0229] The communication interface 1306 is used in wired or wireless communication of signaling and / or data between a network node, access network, and / or UE. As illustrated, the communication interface 1306 comprises port(s) / terminal(s) 1316 to send and receive data, for example to and from a network over a wired connection. The communication interface 1306 also includes radio front-end circuitry 1318 that may be coupled to, or in certain embodiments a part of, the antenna 1310. Radio front-end circuitry 1318 comprises filters 1320 and amplifiers 1322. The radio front-end circuitry 1318 may be connected to an antenna 1310 and processing circuitry 1302. The radio front-end circuitry may be configured to condition signals communicated between antenna 1310 and processing circuitry 1302. The radio front-end circuitry 1318 may receive digital data that is to be sent out to other network nodes or UEs via a wireless connection. The radio front-end circuitry 1318 may convert the digital data into a radio signal having the appropriate channel and bandwidth parameters using a combination of filters 1320 and / or amplifiers 1322. The radio signal may then be transmitted via the antenna 1310. Similarly, when receiving data, the antenna 1310 may collect radio signals which are then converted into digital data by the radio front-end circuitry 1318. The digital data may be passed to the processing circuitry 1302. In other embodiments, the communication interface may comprise different components and / or different combinations of components.

[0230] In certain alternative embodiments, the network node 1300 does not include separate radio front-end circuitry 1318, instead, the processing circuitry 1302 includes radio front-end circuitry and is connected to the antenna 1310. Similarly, in some embodiments, all or some of the RF transceiver circuitry 1312 is part of the communication interface 1306. In still other embodiments, the communication interface 1306 includes one or more ports or terminals 1316, the radio front-end circuitry 1318, and the RF transceiver circuitry 1312, as part of a radio unit (not shown), and the communication interface 1306 communicates with the baseband processing circuitry 1314, which is part of a digital unit (not shown).

[0231] The antenna 1310 may include one or more antennas, or antenna arrays, configured to send and / or receive wireless signals. The antenna 1310 may be coupled to the radio front-end circuitry 1318 and may be any type of antenna capable of transmitting and receiving data and / or signals wirelessly. In certain embodiments, the antenna 1310 is separate from the network node 1300 and connectable to the network node 1300 through an interface or port.

[0232] The antenna 1310, communication interface 1306, and / or the processing circuitry 1302 may be configured to perform any receiving operations and / or certain obtaining operations described herein as being performed by the network node. Any information, data and / or signals may be received from a UE, another network node and / or any other network equipment. Similarly, the antenna 1310, the communication interface 1306, and / or the processing circuitry 1302 may be configured to perform any transmitting operations described herein as being performed by the network node. Any information, data and / or signals may be transmitted to a UE, another network node and / or any other network equipment.

[0233] The power source 1308 provides power to the various components of network node 1300 in a form suitable for the respective components (e.g., at a voltage and current level needed for each respective component). The power source 1308 may further comprise, or be coupled to, power management circuitry to supply the components of the network node 1300 with power for performing the functionality described herein. For example, the network node 1300 may be connectable to an external power source (e.g., the power grid, an electricity outlet) via an input circuitry or interface such as an electrical cable, whereby the external power source supplies power to power circuitry of the power source 1308. As a further example, the power source 1308 may comprise a source of power in the form of a battery or battery pack which is connected to, or integrated in, power circuitry. The battery may provide backup power should the external power source fail.

[0234] Embodiments of the network node 1300 may include additional components beyond those shown in Figure 10 for providing certain aspects of the network node’s functionality, including any of the functionality described herein and / or any functionality necessary to support the subject matter described herein. For example, the network node 1300 may include user interface equipment to allow input of information into the network node 1300 and to allow output of information from the network node 1300. This may allow a user to perform diagnostic, maintenance, repair, and other administrative functions for the network node 1300.

[0235] Figure 11 is a block diagram of a host 1400, which may be an embodiment of the host 1116 of Figure 8, in accordance with various aspects described herein. As used herein, the host 1400 may be or comprise various combinations hardware and / or software, including a standalone server, a blade server, a cloud-implemented server, a distributed server, a virtual machine, container, or processing resources in a server farm. The host 1400 may provide one or more services to one or more UEs.

[0236] The host 1400 includes processing circuitry 1402 that is operatively coupled via a bus 1404 to an input / output interface 1406, a network interface 1408, a power source 1410, and a memory 1412. Other components may be included in other embodiments. Features of these components may be substantially similar to those described with respect to the devices of previous figures, such as Figures 18 and 19, such that the descriptions thereof are generally applicable to the corresponding components of host 1400.

[0237] The memory 1412 may include one or more computer programs including one or more host application programs 1414 and data 1416, which may include user data, e.g., data generated by a UE for the host 1400 or data generated by the host 1400 for a UE. Embodiments of the host 1400 may utilize only a subset or all of the components shown. The host application programs 1414 may be implemented in a container-based architecture and may provide support for video codecs (e.g., Versatile Video Coding (VVC), High Efficiency Video Coding (HEVC), Advanced Video Coding (AVC), MPEG, VP9) and audio codecs (e.g., FLAC, Advanced Audio Coding (AAC), MPEG, G.711 ), including transcoding for multiple different classes, types, or implementations of UEs (e.g., handsets, desktop computers, wearable display systems, heads-up display systems). The host application programs 1414 may also provide for user authentication and licensing checks and may periodically report health, routes, and content availability to a central node, such as a device in or on the edge of a core network. Accordingly, the host 1400 may select and / or indicate a different host for over-the-top services for a UE. The host application programs 1414 may support various protocols, such as the HTTP Live Streaming (HLS) protocol, Real-Time Messaging Protocol (RTMP), Real-Time Streaming Protocol (RTSP), Dynamic Adaptive Streaming over HTTP (MPEG-DASH), etc.

[0238] Figure 12 is a block diagram illustrating a virtualization environment 1500 in which functions implemented by some embodiments may be virtualized. In the present context, virtualizing means creating virtual versions of apparatuses or devices which may include virtualizing hardware platforms, storage devices and networking resources. As used herein, virtualization can be applied to any device described herein, or components thereof, and relates to an implementation in which at least a portion of the functionality is implemented as one or more virtual components. Some or all of the functions described herein may be implemented as virtual components executed by one or more virtual machines (VMs) implemented in one or more virtual environments 1500 hosted by one or more of hardware nodes, such as a hardware computing device that operates as a network node, UE, core network node, or host. Further, in embodiments in which the virtual node does not require radio connectivity (e.g., a core network node or host), then the node may be entirely virtualized. In some embodiments, the virtualization environment 1500 includes components defined by the O-RAN Alliance, such as an O-Cloud environment orchestrated by a Service Management and Orchestration Framework via an 0-2 interface.

[0239] Applications 1502 (which may alternatively be called software instances, virtual appliances, network functions, virtual nodes, virtual network functions, etc.) are run in the virtualization environment Q400 to implement some of the features, functions, and / or benefits of some of the embodiments disclosed herein.

[0240] Hardware 1504 includes processing circuitry, memory that stores software and / or instructions executable by hardware processing circuitry, and / or other hardware devices as described herein, such as a network interface, input / output interface, and so forth. Software may be executed by the processing circuitry to instantiate one or more virtualization layers 1506 (also referred to as hypervisors or virtual machine monitors (VMMs)), provide VMs 1508A and 1508B (one or more of which may be generally referred to as VMs 1508), and / or perform any of the functions, features and / or benefits described in relation with some embodiments described herein. The virtualization layer 1506 may present a virtual operating platform that appears like networking hardware to the VMs 1508.

[0241] The VMs 1508 comprise virtual processing, virtual memory, virtual networking or interface and virtual storage, and may be run by a corresponding virtualization layer 1506. Different embodiments of the instance of a virtual appliance 1502 may be implemented on one or more of VMs 1508, and the implementations may be made in different ways. Virtualization of the hardware is in some contexts referred to as network function virtualization (NFV). NFV may be used to consolidate many network equipment types onto industry standard high volume server hardware, physical switches, and physical storage, which can be located in data centers, and customer premise equipment. In the context of NFV, a VM 1508 may be a software implementation of a physical machine that runs programs as if they were executing on a physical, non-virtualized machine. Each of the VMs 1508, and that part of hardware 1504 that executes that VM, be it hardware dedicated to that VM and / or hardware shared by that VM with others of the VMs, forms separate virtual network elements. Still in the context of NFV, a virtual network function is responsible for handling specific network functions that run in one or more VMs 1508 on top of the hardware 1504 and corresponds to the application 1502.

[0242] Hardware 1504 may be implemented in a standalone network node with generic or specific components. Hardware 1504 may implement some functions via virtualization. Alternatively, hardware 1504 may be part of a larger cluster of hardware (e.g. such as in a data center or CPE) where many hardware nodes work together and are managed via management and orchestration 1510, which, among others, oversees lifecycle management of applications 1502. In some embodiments, hardware 1504 is coupled to one or more radio units that each include one or more transmitters and one or more receivers that may be coupled to one or more antennas. Radio units may communicate directly with other hardware nodes via one or more appropriate network interfaces and may be used in combination with the virtual components to provide a virtual node with radio capabilities, such as a radio access node or a base station. In some embodiments, some signaling can be provided with the use of a control system 1512 which may alternatively be used for communication between hardware nodes and radio units.

[0243] Figure 13 shows a communication diagram of a host 1602 communicating via a network node 1604 with a UE 1606 over a partially wireless connection in accordance with some embodiments. Example implementations, in accordance with various embodiments, of the UE (such as a UE 1112 A of Figure 8 and / or UE 1200 of Figure 9), network node (such as network node 1110A of Figure 8 and / or network node 1300 of Figure 10), and host (such as host 1116 of Figure 8 and / or host 1400 of Figure 11) discussed in the preceding paragraphs will now be described with reference to Figure 13.

[0244] Like host 1400, embodiments of host 1602 include hardware, such as a communication interface, processing circuitry, and memory. The host 1602 also includes software, which is stored in or accessible by the host 1602 and executable by the processing circuitry. The software includes a host application that may be operable to provide a service to a remote user, such as the UE 1606 connecting via an over-the-top (OTT) connection 1650 extending between the UE 1606 and host 1602. In providing the service to the remote user, a host application may provide user data which is transmitted using the OTT connection 1650.

[0245] The network node 1604 includes hardware enabling it to communicate with the host 1602 and UE 1606. The connection 1660 may be direct or pass through a core network (like core network 1106 of Figure 8) and / or one or more other intermediate networks, such as one or more public, private, or hosted networks. For example, an intermediate network may be a backbone network or the Internet. The UE 1606 includes hardware and software, which is stored in or accessible by UE 1606 and executable by the UE’s processing circuitry. The software includes a client application, such as a web browser or operator-specific “app” that may be operable to provide a service to a human or non-human user via UE 1606 with the support of the host 1602. In the host 1602, an executing host application may communicate with the executing client application via the OTT connection 1650 terminating at the UE 1606 and host 1602. In providing the service to the user, the UE's client application may receive request data from the host's host application and provide user data in response to the request data. The OTT connection 1650 may transfer both the request data and the user data. The UE's client application may interact with the user to generate the user data that it provides to the host application through the OTT connection 1650.

[0246] The OTT connection 1650 may extend via a connection 1660 between the host 1602 and the network node 1604 and via a wireless connection 1670 between the network node 1604 and the UE 1606 to provide the connection between the host 1602 and the UE 1606. The connection 1660 and wireless connection 1670, over which the OTT connection 1650 may be provided, have been drawn abstractly to illustrate the communication between the host 1602 and the UE 1606 via the network node 1604, without explicit reference to any intermediary devices and the precise routing of messages via these devices.

[0247] As an example of transmitting data via the OTT connection 1650, in step 1608, the host 1602 provides user data, which may be performed by executing a host application. In some embodiments, the user data is associated with a particular human user interacting with the UE 1606. In other embodiments, the user data is associated with a UE 1606 that shares data with the host 1602 without explicit human interaction. In step 1610, the host 1602 initiates a transmission carrying the user data towards the UE 1606. The host 1602 may initiate the transmission responsive to a request transmitted by the UE 1606. The request may be caused by human interaction with the UE 1606 or by operation of the client application executing on the UE 1606. The transmission may pass via the network node 1604, in accordance with the teachings of the embodiments described throughout this disclosure. Accordingly, in step 1612, the network node 1604 transmits to the UE 1606 the user data that was carried in the transmission that the host 1602 initiated, in accordance with the teachings of the embodiments described throughout this disclosure. In step 1614, the UE 1606 receives the user data carried in the transmission, which may be performed by a client application executed on the UE 1606 associated with the host application executed by the host 1602.

[0248] In some examples, the UE 1606 executes a client application which provides user data to the host 1602. The user data may be provided in reaction or response to the data received from the host 1602. Accordingly, in step 1616, the UE 1606 may provide user data, which may be performed by executing the client application. In providing the user data, the client application may further consider user input received from the user via an input / output interface of the UE 1606. Regardless of the specific manner in which the user data was provided, the UE 1606 initiates, in step 1618, transmission of the user data towards the host 1602 via the network node 1604. In step 1620, in accordance with the teachings of the embodiments described throughout this disclosure, the network node 1604 receives user data from the UE 1606 and initiates transmission of the received user data towards the host 1602. In step 1622, the host 1602 receives the user data carried in the transmission initiated by the UE 1606.

[0249] One or more of the various embodiments improve the performance of OTT services provided to the UE 1606 using the OTT connection 1650, in which the wireless connection 1670 forms the last segment. More precisely, the teachings of these embodiments provide the capability to build and train service quality models that can be used for monitoring service quality of OTT services. More particularly, the method of the present disclosure can be used for detecting and identifying service quality degradations by radio issues, and further, can be used for improving and optimizing the radio network. By way of example, with some localization method(s), service quality issues due to the radio environment can be identified. So identified, problematic areas of the radio network can be tuned to provide better coverage, capacity, etc.

[0250] In an example scenario, factory status information may be collected and analyzed by the host 1602. As another example, the host 1602 may process audio and video data which may have been retrieved from a UE for use in creating maps. As another example, the host 1602 may collect and analyze real-time data to assist in controlling vehicle congestion (e.g., controlling traffic lights). As another example, the host 1602 may store surveillance video uploaded by a UE. As another example, the host 1602 may store or control access to media content such as video, audio, VR or AR which it can broadcast, multicast or unicast to UEs. As other examples, the host 1602 may be used for energy pricing, remote control of non-time critical electrical load to balance power generation needs, location services, presentation services (such as compiling diagrams etc. from data collected from remote devices), or any other function of collecting, retrieving, storing, analyzing and / or transmitting data.

[0251] In some examples, a measurement procedure may be provided for the purpose of monitoring data rate, latency and other factors on which the one or more embodiments improve. There may further be an optional network functionality for reconfiguring the OTT connection 1650 between the host 1602 and UE 1606, in response to variations in the measurement results. The measurement procedure and / or the network functionality for reconfiguring the OTT connection may be implemented in software and hardware of the host 1602 and / or UE 1606. In some embodiments, sensors (not shown) may be deployed in or in association with other devices through which the OTT connection 1650 passes; the sensors may participate in the measurement procedure by supplying values of the monitored quantities exemplified above, or supplying values of other physical quantities from which software may compute or estimate the monitored quantities. The reconfiguring of the OTT connection 1650 may include message format, retransmission settings, preferred routing etc.; the reconfiguring need not directly alter the operation of the network node 1604. Such procedures and functionalities may be known and practiced in the art. In certain embodiments, measurements may involve proprietary UE signaling that facilitates measurements of throughput, propagation times, latency and the like, by the host 1602. The measurements may be implemented in that software causes messages to be transmitted, in particular empty or ‘dummy’ messages, using the OTT connection 1650 while monitoring propagation times, errors, etc.

[0252] Although the computing devices described herein (e.g., UEs, network nodes, hosts) may include the illustrated combination of hardware components, other embodiments may comprise computing devices with different combinations of components. It is to be understood that these computing devices may comprise any suitable combination of hardware and / or software needed to perform the tasks, features, functions and methods disclosed herein. Determining, calculating, obtaining or similar operations described herein may be performed by processing circuitry, which may process information by, for example, converting the obtained information into other information, comparing the obtained information or converted information to information stored in the network node, and / or performing one or more operations based on the obtained information or converted information, and as a result of said processing making a determination. Moreover, while components are depicted as single boxes located within a larger box, or nested within multiple boxes, in practice, computing devices may comprise multiple different physical components that make up a single illustrated component, and functionality may be partitioned between separate components. For example, a communication interface may be configured to include any of the components described herein, and / or the functionality of the components may be partitioned between the processing circuitry and the communication interface. In another example, non-computationally intensive functions of any of such components may be implemented in software or firmware and computationally intensive functions may be implemented in hardware.

[0253] In certain embodiments, some or all of the functionality described herein may be provided by processing circuitry executing instructions stored on in memory, which in certain embodiments may be a computer program product in the form of a non-transitory computer- readable storage medium. In alternative embodiments, some or all of the functionality may be provided by the processing circuitry without executing instructions stored on a separate or discrete device-readable storage medium, such as in a hard-wired manner. In any of those particular embodiments, whether executing instructions stored on a non-transitory computer- readable storage medium or not, the processing circuitry can be configured to perform the described functionality. The benefits provided by such functionality are not limited to the processing circuitry alone or to other components of the computing device, but are enjoyed by the computing device as a whole, and / or by end users and a wireless network generally.

[0254] The present embodiments may, of course, be carried out in other ways than those specifically set forth herein without departing from characteristics described herein. The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, and all changes coming within the meaning and equivalency range of the appended claims are intended to be embraced therein.

Claims

CLAIMSWhat is claimed is:

1. A method (210), for an analytics system (160) operating in a mobile communications network, for training a machine learning, ML, model (164) configured to predict service quality (200), the method comprising: obtaining (212) a first set of training data for a ML model configured to predict service quality, wherein the first set of training data is generated by a first communications network (100) and comprises network-level metrics and application-level metrics; receiving (214) service quality metrics associated with one or more User Equipments, UEs (170), operating in a second mobile communications network (150), wherein the second mobile communications network is a live production mobile communications network; and generating (216) an augmented set of training data (180) to train the ML model configured to predict service quality by augmenting the first set of training data with the service quality metrics associated with the one or more UEs according to one or more transport patterns (130) in the network-level metrics and based on a set of predetermined rules.

2. The method of claim 1 , wherein the first communications network is a live production communications network.

3. The method of claim 1 , wherein the first communications network is a test communications network.

4. The method of claim 1 , wherein the first communications network is a mobile communications network.

5. The method of claim 1 , wherein the first communications network is a wired network.

6. The method of any of claims 1-5, wherein the first set of training data is generated by the first communications network based on network-level degradations present in the first communications network.

7. The method of claims 5-6, wherein the network-level degradations are introduced into the first communications network.

8. The method of claim 6, wherein the network-level metrics in the first set of training data is labeled according to one or both of: the application-level metrics corresponding to the network-level metrics; andthe network-level degradations.

9. The method of any of claims 6-8, wherein the one or more transport patterns in the networklevel metrics are associated with the network-level degradations introduced into the first communications network.

10. The method of any of claims 6-9, further comprising determining (232) that the service quality metrics associated with the one or more UEs comprise a transport pattern that matches at least one transport pattern in the network-level metrics.11 . The method of claim 10, wherein the transport pattern in the service quality metrics associated with the one or more UEs matches the transport pattern in the network-level metrics when, for a given session and time interval, data in the service quality metrics associated with the one or more UEs matches data in the transport pattern in the network-level metrics.

12. The method of any of claims 10-11 , wherein determining that the service quality metrics associated with the one or more UEs comprise a transport pattern that matches at least one transport pattern in the network-level metrics comprises: filtering (234) the labeled network-level metrics in the first set of training data to determine one or more selected transport patterns; comparing (236) the service quality metrics associated with the one or more UEs to the labeled network-level metrics associated with the one or more selected transport patterns; and determining (238) that the service quality metrics associated with the one or more UEs comprises at least one of the selected transport patterns based on a result of the comparison.

13. The method of claim 12, wherein the one or more selected transport patterns against which the service quality metrics associated with the one or more UEs are compared represent a pattern of one or more of: a packet loss ratio; jitter; a change in one or more Key Performance Indicators (KPI); interference; bit corruption; andReference Signal Received Power (RSRP).

14. The method of claims 12-13, wherein the one or more selected transport patterns are time- qualified.

15. The method of claims 12-14, wherein the one or more selected transport patterns are relative to one or more of: a predefined threshold value; and a predefined range of values.

16. The method of claims 11-15, further comprising storing (240) one or more session segments of a communication session in the second mobile communications network responsive to determining that the service quality metrics associated with the one or more UEs comprises the at least one of the selected transport patterns.

17. The method of any of claims 1-16, wherein generating an augmented set of training data to train an ML model comprises labeling (252) the service quality metrics associated with the one or more UEs.

18. The method of claim 17, wherein labeling the service quality metrics associated with the one or more UEs comprises: determining (254) whether the service quality metrics associated with the one or more UEs comprises a service quality label; and responsive to the determining, labeling (256) the service quality metrics associated with the one or more UEs using the service quality label.

19. The method of claims 17-18, wherein labeling the service quality metrics associated with the one or more UEs comprises:Determining (258) whether the service quality metrics associated with the one or more UEs comprise a related service quality label related to a first service quality measure; and responsive to the determining, labeling (260) the service quality metrics associated with the one or more UEs using the related service quality label.

20. The method of claims 17-19, wherein labeling the service quality metrics associated with the one or more UEs comprises: determining (254, 258) that the service quality metrics associated with the one or more UEs do not comprise the related service quality label related to a first service quality measure and that there are no other service quality labels that are different from, but related to, the service quality label; andresponsive to the determining, labeling (262) the service quality metrics associated with the one or more UEs using a service quality label that was previously used to label the network-level metrics and / or the application-level metrics in the first set of training data when the network-level metrics and / or the application-level metrics were measured in the first communications network.21 . The method of any of claims 17-20, further comprising determining whether an accuracy of the ML model increases when one or more labels associated with the service quality metrics are inaccurate.

22. The method of any of the preceding claims, wherein the network-level metrics and the application-level metrics in the first set of training data are service quality metrics.

23. The method of any of the preceding claims, further comprising training (218) the ML model using the augmented set of training data.

24. The method of any of the preceding claims further comprising determining (220) a Quality of Service (QoS) / Quality of Experience (QoE) for a UE in the second mobile communications network based on the ML model trained using the augmented set of training data.

25. The method of any of the preceding claims, wherein the first communications network and the second mobile communications network are the same networks.

26. The method of any of the preceding claims, wherein the first communications network and the second mobile communications network are different networks.

27. The method of any of the preceding claims, implemented in a Network Data Analytics Function (NWDAF) (42) in a core network (CN) domain (40).

28. The method of any of the preceding claims, implemented in a Management Data Analytics Function (MDAF) (22) in a management network domain (20).

29. The method of any of the preceding claims, wherein at least one of the one or more transport patterns in the network-level metrics represents a pattern of degradation in a radio network.

30. A network node (300), for an analytics system (160) operating in a mobile communications network, for training a machine learning, ML, model (164) configured to predict service quality(200), the network node comprising: communications circuitry (302) configured to communicatively connect the network node to a communications network; and processing circuitry (304) operatively connected to the communications circuitry and configured to: obtain (212) a first set of training data for a ML model (164) configured to predict service quality (200), wherein the first set of training data is generated by a first communications network (100) and comprises network-level metrics and applicationlevel metrics; receive (214) service quality metrics associated with one or more User Equipments, UEs (170), operating in a second mobile communications network (150), wherein the second mobile communications network is a live production mobile communications network; and generate (216) an augmented set of training data (180) to train the ML model configured to predict service quality by augmenting the first set of training data with the service quality metrics associated with the one or more UEs according to one or more transport patterns (130) in the network-level metrics and based on a set of predetermined rules.31 . The network node of claim 23, wherein the processing circuitry is further configured to perform the method according to any one of claims 2-29.

32. A network node (300), for an analytics system (160) operating in a mobile communications network, for training a machine learning, ML, model (164) configured to predict service quality (200), the network node being configured to: obtain (212) a first set of training data for a ML model configured to predict service quality, wherein the first set of training data is generated by a first communications network (100) and comprises network-level metrics and application-level metrics; receive (214) service quality metrics associated with one or more User Equipments, UEs (170), operating in a second mobile communications network (150), wherein the second mobile communications network is a live production mobile communications network; and generate (216) an augmented set of training data (180) to train the ML model configured to predict service quality by augmenting the first set of training data with the service quality metrics associated with the one or more UEs according to one or more transport patterns (130) in the network-level metrics and based on a set of predetermined rules.

33. The network node of claim 25, wherein the network node is further configured to perform themethod according to any one of claims 2-29.

34. A computer program (312) comprising instructions that, when executed by processing circuitry (314) of a network node (300), cause the network node to perform the method according to any of claims 1 -29.

35. A carrier containing the computer program of claim 27, wherein the carrier is one of an electronic signal, optical signal, radio signal, or computer readable storage medium.

36. A non-transitory computer-readable storage medium (306) comprising a computer program(312) stored thereon, the computer program comprising executable instructions that, when executed by processing circuitry (304) in a network node (300), causes the network node to perform the method of any one of claims 1 -29.

Images