Data collection and test system for delayed critical services

Abstract

A mobile system (120) trains a Quality of Experience (QoE) model. To this end, a mobile device captures packets of an ultra-reliable low-latency communication URLLC service transmitted over a wireless connection. A mobile device introduces interference that affects the URLLC service. The mobile system (120) monitors the quality of service of the URLLC service when the URLLC service is affected by the interference and when the URLLC service is not affected by the interference. The mobile system (120) updates the QoE model (118) based on the monitored quality of service.

Description

Technical Field

[0001] The present disclosure relates generally to the field of wireless communication networks, and more particularly to collecting and analyzing data collected within a wireless communication network to evaluate network performance. Background Art

[0002] It has been predicted that by the end of 2027, the number of 5G users worldwide will reach 4.4 billion, accounting for almost half of all mobile subscriptions and contributing more than 60% of the total mobile data traffic worldwide. Over the past decade, the research community and industry players have invested a lot of effort in investigating the traffic characteristics and user experience of traditional mobile broadband (MBB) services, such as web browsing and video streaming, for which Long Term Evolution (LTE) networks are optimized. Current trends indicate that 5G will become the dominant radio access technology in the coming years and shape the future of mobile communications by enabling a wide range of new services and use cases.

[0003] 5G New Radio (NR) is a new air interface developed by the 3rd Generation Partnership Project (3GPP) and is designed to meet the needs of emerging ultra-reliable low-latency communication (URLLC) services. Such needs include 360-degree video streaming, cloud gaming, immersive AR / VR applications, remote control, and industrial automation. Unlike MBB services, which typically require high-bandwidth connections with only best-effort latency, URLLC services have strict requirements on latency and reliability due to their real-time nature. Adaptive video streaming, which is driving the growth of mobile services in terms of data volume, is able to tolerate variations in network quality of service (QoS) because client buffers are able to mask latency fluctuations from the underlying transport mechanism or the application itself. In contrast, for low-latency communications, data delivery must typically be guaranteed to be within a specific latency range with at least 99.999% probability, depending on the given use case. For example, while cloud gaming platforms can provide a latency-free gaming experience with round-trip times of up to 100 milliseconds in some cases, industrial robots cannot operate reliably unless latency is kept below 5 milliseconds.

[0004] To measure service quality, MNOs have traditionally deployed passive probes in their networks that are capable of collecting the packet-level information needed to compute meaningful metrics for various types of network traffic. Earlier approaches for predicting Quality of Experience (QoE) have primarily consisted of simplified analytical models built on human expert knowledge and observation.

[0005] Per-session advanced analytics systems, such as Ericsson Expert Analytics (EEA), are based on collecting and correlating basic network events from different network domains, such as core network, radio network, and transport network. They compute user- and session-level end-to-end (e2e) service quality metrics, such as service key performance indicators (S-KPIs) or QoE metrics, as well as radio and network resource KPIs (R-KPIs), characterizing the radio environment or network operation at user and session levels. These types of solutions are typically applied to session-based troubleshooting and analysis of network issues.

[0006] Event-based analytics systems are also used in service operations centers (SOCs) to monitor the quality of a wide range of services used at the network level, as well as to monitor the customer experience on an individual basis at each user level. These tools are widely used in customer care and other business scenarios.

[0007] Event-based analysis usually requires real-time collection and correlation of characteristic node and protocol events from different radio and core nodes, as well as detection of signaling interfaces (IFs) and sampling of user plane services. In addition to data collection and correlation capabilities, the system usually requires advanced databases, rule engines, and big data analysis platforms.

[0008] Due to the introduction of 5G mobile networks, it is expected that the mobile networks will serve a large number of various new service types (such as delay-critical URLLC service types). In addition, it is expected that these networks will serve a much higher number of devices or user equipment (UE) than previous network technologies. Summary of the invention

[0009] Embodiments of the present disclosure are generally directed to the collection of training data for machine learning models to predict the quality of service for new delay-critical service types, such as those transported over, for example, UDP / RTP, UDP / SRTP, or UDP / QUIC protocols. Specific embodiments are applicable to testing new radio bearers and settings for these new service types in 5G or post-5G mobile networks, and for deriving the relationship between different radio issues, transmission characteristics, and end-user quality of service.

[0010] A specific embodiment includes a method for training a QoE model implemented by a mobile system. The method includes: capturing packets of a URLLC service transmitted over a wireless connection, and introducing interference that affects the URLLC service. The method further includes: monitoring the quality of service of the URLLC service when the URLLC service is affected by interference and when the URLLC service is not affected by interference. The method further includes: updating the QoE model based on the monitored quality of service.

[0011] In some embodiments, the method further comprises: associating each of the captured packets with when the URLLC service is affected by interference or when the URLLC service is not affected by interference. Updating the QoE model is further based on the associated packets.

[0012] In some embodiments, monitoring the quality of service of the URLLC service includes: collecting radio environment measurements of the uplink and / or downlink supporting the URLLC service. In addition, based on the quality of service, updating the QoE model includes: using the collected radio environment measurements as training data to train the QoE model. In some such embodiments, the radio environment measurements include signal quality measurements of the downlink and transmit power of the uplink.

[0013] In some embodiments, monitoring the quality of service of the URLLC service includes: collecting radio configuration parameters of one or more radio devices supporting the URLLC service. In addition, based on the quality of service, updating the QoE model includes: using the collected radio configuration parameters as training data to train the QoE model. In some such embodiments, the radio configuration parameters include timing advance and / or the frequency used.

[0014] In some embodiments, monitoring the quality of service of the URLLC service includes collecting QoE configuration parameters associated with the URLLC service. In addition, based on the quality of service, updating the QoE model includes using the collected QoE configuration parameters as training data to train the QoE model. In some such embodiments, the QoE configuration parameters include packet loss, jitter, and / or round-trip time requirements for providing QoE for the URLLC service.

[0015] In some embodiments, monitoring the quality of service of the URLLC service includes collecting transmission measurements of the URLLC service. In addition, based on the quality of service, updating the QoE model includes using the collected transmission measurements as training data to train the QoE model. In some such embodiments, the transmission measurements include bit rate and / or throughput measurements of the URLLC service.

[0016] In some embodiments, monitoring the quality of service of the URLLC service includes: generating periodic sample data of the quality of service, the periodic sample data having a resolution of no more than one second. In addition, based on the quality of service, updating the QoE model includes: using the periodic sample data as training data to train the QoE model.

[0017] In some embodiments, capturing packets of the URLLC service includes capturing Real-time Transport Protocol (RTP), Secure RTP and / or Fast UDP Internet Connection packets.

[0018] In some embodiments, the grouping of the URLLC service includes: an encrypted portion and an unencrypted portion. In addition, monitoring the quality of service includes: deriving performance data of the URLLC service based on the unencrypted portion and without decrypting the encrypted portion. In addition, based on the quality of service, updating the QoE model includes: using the performance data as training data to train the QoE model.

[0019] In some embodiments, monitoring the quality of service of the URLLC service includes: generating a round-trip data sample for each of a plurality of spin bit inversions detected within the captured packet. In addition, based on the quality of service, updating the QoE model includes: using the round-trip data samples as training data to train the QoS model.

[0020] In some embodiments, the method further comprises: performing a handover procedure in response to the interference, wherein monitoring the quality of service comprises: monitoring the quality of service before the handover procedure and after the handover procedure.

[0021] In some embodiments, introducing interference includes introducing a transmission data disturbance. In some such embodiments, introducing a transmission data disturbance includes introducing packet loss, jitter, delay, packet reordering, congestion, and / or packet fragmentation.

[0022] In some embodiments, introducing interference includes introducing a disturbance to a radio condition of the wireless connection.

[0023] Other embodiments include a mobile system. The mobile system includes a processing circuit and a memory circuit. The memory circuit stores instructions executable by the processing circuit, whereby the mobile system is configured to capture packets of a URLLC service transmitted over a wireless connection. The mobile system is further configured to introduce interference that affects the URLLC service, and to monitor the quality of service of the URLLC service when the URLLC service is affected by the interference and when the URLLC service is not affected by the interference. The mobile system is further configured to update the QoE model based on the monitored quality of service.

[0024] In some embodiments, the mobile system is further configured to perform any embodiment of the above method.

[0025] Still other embodiments comprise a computer program comprising instructions which, when executed on a processing circuit of a mobile system, cause the mobile system to perform any embodiment of the above-described method.

[0026] Still other embodiments include a carrier embodying the computer program. The carrier is one of an electronic signal, an optical signal, a radio signal or a computer-readable storage medium. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Aspects of the present disclosure are illustrated by way of example and are not limited to the accompanying drawings, which have like reference numerals indicating like elements. In general, the use of reference numerals should be considered to refer to the depicted subject matter according to one or more embodiments, while discussion of specific instances of illustrated elements will be followed by an alphabetical designation (e.g., discussing computing device 110 in general, rather than discussing specific instances of computing devices 110a, 110b).

[0028] Figure 1 is a block diagram schematically illustrating the architecture of a wireless communication network according to one or more embodiments of the present disclosure.

[0029] Figure 2 is a flow chart schematically illustrating an example method performed by mobile system 120 according to one or more embodiments of the present disclosure.

[0030] Figure 3 is a timeline schematically illustrating an example of a measurement descriptor according to one or more embodiments of the present disclosure.

[0031] Figure 4 A time-stamped and parameterized data structure used by various monitors according to one or more embodiments of the present disclosure is shown.

[0032] Figure 5 An example combined output data structure is shown in accordance with one or more embodiments of the present disclosure.

[0033] Figure 6 is a block diagram schematically illustrating example uplink and downlink processing according to one or more embodiments of the present disclosure.

[0034] Figure 7-13 is a graph illustrating example measurements according to one or more embodiments of the present disclosure.

[0035] Fig.14 is a flow chart illustrating an example method implemented by a mobile system according to one or more embodiments of the present disclosure.

[0036] Fig.15 is a schematic block diagram illustrating an example mobile system according to one or more embodiments of the present disclosure. DETAILED DESCRIPTION

[0037] Due to the tight constraints on latency-critical services, it is increasingly important for mobile network operators (MNOs) to detect quality impairments and react quickly to performance degradations (e.g., through automated closed-loop actions). However, various factors currently hinder such efforts. For example, the growing adoption of end-to-end encryption makes the development of reliable traffic monitoring solutions increasingly difficult. As a result, customers may receive an increasingly suboptimal user experience and become increasingly dissatisfied.

[0038] An important technology driver behind encryption is the introduction of new protocols that are designed with more attention to security and privacy aspects than ever before. Specifically, the latest versions of Transport Layer Security (TLS) and its variant tailored for data transmission based on User Datagram Protocol (UDP), namely Datagram Transport Layer Security (DTLS), provide enhanced security over their predecessors through forward secrecy and encrypted handshakes. Large-scale deployment of the recently standardized Quick UDP Internet Connections (QUIC) protocol has begun, which also represents a big step towards Internet-wide encryption, as QUIC is designed to encrypt all data at the transport layer. QUIC ensures reliable transmission similar to the Transmission Control Protocol (TCP) and incorporates some additional features and improvements, but greatly limits the observability of protocol messages, leaving few options for performance monitoring and troubleshooting.

[0039] The trend of prioritizing security and privacy has led to a significant decline in data visibility and accessibility. Most network traffic is encrypted via TLS and QUIC protocols, making it challenging to predict the quality of service. In addition to the packet payload, some parts of the protocol handshake messages and header fields are also hidden from passive monitoring devices such as network probes. Existing QoE evaluation methods are tailored for traditional MBB applications, while the quality provided by latency-critical services is poorly understood. Therefore, new frameworks are needed to identify and extract relevant information from encrypted data streams that helps characterize the QoE of these emerging 5G services.

[0040] The typical approach to building QoE models is to collect and utilize transport-level data obtained from monitoring nodes. While end-to-end network QoS parameters (latency, packet loss, etc.) are indeed important indicators of possible performance degradation, signal quality and mobility events also highly impact the user experience in mobile networks. For example, inter-frequency handovers or transitions between different radio access technologies may interrupt ongoing data transmission, which can lead to suboptimal QoE, especially in the case of delay-sensitive applications. In order to improve prediction accuracy, in addition to analyzing core network data, it is also crucial to explore the impact of radio conditions on transmission characteristics and service quality in several real-world scenarios.

[0041] Obtaining e2e transmission metrics (latency, packet loss, jitter, etc.) through network probing alone is difficult or impossible to accomplish because in many cases, e2e transmission reports are missing (e.g., in the case of UDP or Real-time TCP (RTCP)). Alternatively, e2e metrics should probably be derived from multiple path segments, and the resolution of the data is different or data for some segments is not available.

[0042] While some methods for evaluating the QoE of traditional MBB applications are already available and can be sufficient for specific needs, the business characteristics of URLLC services and their complex relationship with user-perceived quality are poorly understood. In order to develop accurate performance metrics and models to address these scenarios, extensive research will likely be required, which requires large and diverse labeled datasets under various network and radio conditions. Despite having massive data as a prerequisite for properly training QoE models, current solutions do not support massive data collection without human intervention. In addition, diversity is essential for building models that generalize well across different services or under various network conditions. These requirements cannot be met by running manual tests, which are time-consuming and expensive and are only applicable to covering a limited number of scenarios and parameter values ​​that occur in actual network environments. Therefore, earlier methods for model development based on deep expert knowledge are no longer feasible.

[0043] Compared to these traditional methods of measuring service quality, leveraging the power of machine learning (ML) seems to be the most viable solution in the era of encryption, which can extract relevant patterns from raw traffic features and publicly accessible packet headers. ML algorithms can reveal hidden relationships and interdependencies between various network features.

[0044] To cope with the high processing overhead and storage footprint, state-of-the-art analytics systems typically apply advanced filtering algorithms and aggregate data at different levels. While quality of service models can estimate metrics for each session, detailed datasets are required during the training phase. To this end, it may be necessary or advantageous to capture experimental data with up to one-second resolution in both the radio network and the core network to deeply understand traffic patterns in the case of performance issues.

[0045] In LTE networks, QoS differentiation enables mobile operators to prioritize network services for users based on service-specific requirements. For example, conversational voice and video streaming services will be treated differently due to differences in packet delay budgets and error rates. 5G brings enhancements to QoS processing, where the Evolved Packet Switched System (EPS) bearer is replaced by the so-called "QoS flow" concept, which allows policy implementation at a more granular level. In addition, 3GPP has defined QoS profiles (e.g., 5G QoS Identifier (5QI) classes) to support advanced URLLC services. Despite this, in practice, there is a lack of available solutions to test and investigate these new capabilities.

[0046] In view of the above, an integrated data collection system and method for mobile networks is described herein. Specific embodiments of the system are primarily intended to collect training data for ML models to predict the quality of service of new delay-critical service types transported over UDP / Real-time Transport Protocol (RTP), UDP / Secure RTP (SRTP), or UDP / QUIC protocols.

[0047] Embodiments of the system may be applicable to testing new radio bearers and settings used for these new service types in 5G mobile networks. Embodiments may additionally or alternatively be applicable to deriving relationships between different radio issues and transmission characteristics and end-user quality of service.

[0048] The system can be integrated into various computing platforms (e.g., laptops or other mobile devices). Specifically, the system can be integrated and mobile so that the system can be used to test real radio conditions and mobile conditions. For example, the system can be suitable for conducting drive tests. The system can additionally or alternatively include different transmission monitors (e.g., spindump, tcpdump) based on the latest Unix utilities and / or monitors for recording high-resolution downlink and uplink radio environment measurements.

[0049] The output data of the radio and transmission monitors may be converted to a given resolution and format defined for the QoE ML model.

[0050] The end-user quality of service can also be monitored and logged.The system includes software and physical infrastructure based manipulators that can be used to introduce transmission (packet loss, delay, etc.) and radio environment (coverage, interference) degradations in a planned manner.

[0051] In some embodiments, the user designs and configures measurement flows and test periods within measurement flows, where different transmission and radio degradations can be applied, even combined.

[0052] End-user quality of service can be labeled online or offline. Labeled data can be used to train QoE ML models. More detailed output of transmission and radio data can be used to derive the relationship between radio metrics, transmission metrics and end-user quality of service. Detailed packet dumps provided by the system can enable investigation of individual flows at the protocol level, for example using Wireshark.

[0053] The integrated data collection system and method captures high-resolution transmission traces together with radio environment measurement data and clear end-user service quality observations while potentially introducing various radio and transmission interferences.

[0054] Although the system is suitable for monitoring voice and traditional MBB traffic types, the system specifically enables investigation of the quality of service of URLLC traffic types in mobile networks, as well as collection of training data for quality of service models for URLLC traffic types.

[0055] The data collection system can be used to collect tagged data for unbuffered or short-buffered delay-critical service types transported over UDP / RTP, UDP / SRTP, or UDP / QUIC. Uplink and downlink radio and transport data for two-way sessions are captured simultaneously. The collected packets include: downlink radio environment measurements, reference signal received power (RSRP), reference signal received quality (RSRQ), frequencies used, timing advance, handovers, uplink radio measurements (e.g., transmit power), and transport-level QoS parameters such as packet loss, jitter, round-trip time (RTT), and high-resolution transport measurements (e.g., bit rate, throughput). Transport and radio data can be collected with high temporal resolution (e.g., 1 second or less).

[0056] E2E service quality information can be derived for the above service types. The corresponding data may be equivalent to cross-domain data and node events obtained by network probes for all sessions in the mobile network analysis system.

[0057] It should be noted that this data equivalence is not trivial. For example, packet loss is obtained from transmission traces in a data collection system, but this is not available in packet probing due to encryption. Therefore, embodiments replace traditional packet loss obtained through such measurements with radio protocol layer packet loss, which can be used by network analysis systems in radio events. The resolution of the available data can also be different.

[0058] The labeled data collected by this system can be suitable for training quality of service models running in large-scale, network-wide analysis systems.

[0059] The data collection framework allows for the execution of planned data collection and test scenarios focusing on potential transport and radio issues occurring in mobile networks. Therefore, they may be particularly suitable for training quality of service models to detect typical network issues.

[0060] Particular embodiments may be flexible in multiple aspects. For example, embodiments may support testing and data collection of virtual bearers, real bearers, 5QI classes, QoS profiles, etc. (particularly those dedicated to URLLC service types).

[0061] Certain embodiments may additionally or alternatively be comprehensive in one or more aspects. For example, by artificially introducing typical radio and transmission problems, characterization training data may be obtained for training a comprehensive QoE ML model. High-resolution transmission data may be obtained, which allows for in-depth investigation of transmission patterns associated with different radio and transmission degradations. e2e transmission metrics for test sessions may be derived. High-resolution transmission data may be obtained together with high-resolution radio environment and mobility data. Thus, the relationship between radio interference and transmission errors may be explored in a wide variety of ways.

[0062] Although transport packet characteristics are important indicators of QoE, signal quality and mobility events can also highly influence the user experience in mobile networks. Therefore, it is expected that more accurate URLLC service quality models can be developed by using transport and radio packet level characteristics and uplink and downlink radio environment measurements as inputs to these models, and then using only transport network characteristics. It is worth noting that the encryption protocol field can also be used for investigation by capturing the encryption key at the end user.

[0063] Figure 1 1 is a block diagram schematically illustrating the architecture of a wireless communication network 100 according to one or more embodiments of the present disclosure. Specific embodiments of the network 100 comply with one or more 3GPP standards. The network 100 includes a computing system 160, a core network 130, and one or more radio access networks (RANs) 140. The computing system 160 includes an analysis system 110 and a mobile system 120, each of which includes a corresponding collection of one or more physical computing devices. For example, the analysis system 110 may include one or more server computers, and the mobile system 120 may include a tablet computer, a smartphone, and / or a laptop computer. The one or more RANs 140 may include a 4G RAN 142 and / or a 5G RAN 144. As will be discussed in further detail below, the analysis system 110 and / or the mobile system 120 may also include one or more logical components (e.g., subsystems, records, models, functions, monitors, services, controllers, etc.).

[0064] In some embodiments, the core network 130 uses a service-based architecture, in which architectural elements are defined in terms of network functions rather than network entities (e.g., as in the 5G Core (5GC)). Therefore, network functions can be implemented by any number of discrete computing devices (e.g., controlling network nodes). Network functions can be interconnected via interfaces of a common framework to provide services to users. For example, Figure 1As shown in , core network 130 may include access and mobility management function (AMF) 134, session management function (SMF) 136, and / or user plane function (UPF) 132. In general, UPF 132 may be responsible for processing user plane data exchanged with data networks external to core network 130 (e.g., the Internet, not shown). UPF 132 may also support processing user plane traffic based on rules received from SMF 136. AMF 134 may be responsible for handling, for example, connection and mobility management tasks for one or more mobile components of computing system 160 (discussed further below).

[0065] According to one or more specific embodiments, the analysis system 110 includes an event correlation subsystem 112, a per-session correlation record 114, one or more QoE models 118, and / or one or more analysis functions 116. During operation, the analysis system 110 may receive signaling events from one or more control functions in the core network (e.g., AMF 134 and / or SMF 136) and / or UPF 132. The analysis system 110 may additionally or alternatively receive cell tracking events from one or more radio base stations included in one or more RAN 140. The event correlation subsystem 112 may be configured to generate a per-session correlation record 114 based on these received events. The per-session correlation record 114 may be used to generate and / or improve one or more QoE models 118. One or more analysis functions 116 may use one or more QoE models 118 and / or records 114, for example, to predict and / or detect one or more network conditions.

[0066] The analysis system 110 further receives training data and / or labels for one or more QoE models 118 from the mobile system 120. Since existing (and possibly future) URLLC services are typically very sensitive to interference, the mobile system 120 can be configured to collect a wide variety of data that potentially affects QoE. To the extent possible or practical, it is advantageous that the data provided by the mobile system 120 to the analysis system 110 is comprehensive and high-resolution, for example, for use by one or more QoE models 118. In this regard, in some embodiments, the mobile system 120 can be configured with test scenarios and / or applications for one or more tests by human users 150. The user 150 can additionally or alternatively label the data based on observations of the quality of service experienced by the end user for use by one or more QoE models 118.

[0067] In this regard, in some embodiments, the mobile system 120 may include a controller 122, one or more services 124, one or more monitors 126, and / or one or more manipulators 128, each of which will be discussed in more detail below. In general, the controller 122 is configured to control the activation and deployment of one or more services 124 in the infrastructure of the network, and to control the activation and deployment of one or more monitors 126. The monitor 126 is configured to monitor the network and radio environment, and the one or more manipulators 128 are configured to manipulate characteristics of the network infrastructure.

[0068] Figure 2 2 is a flow chart illustrating an example method 200 performed by the mobile system 120 according to one or more embodiments of the present disclosure. According to the method 200, data collection is performed to collect labeled data for delay-critical service types with no buffering or short buffering transmitted through the network and radio environment. The goal of the method 200 is to obtain radio and transmission metrics (e.g., S-KPI and R-KPI) to obtain training data for a quality of service machine learning model and predict QoE.

[0069] The method 200 first begins by obtaining one or more measurement definitions (block 210). For example, to obtain metrics for a service under examination, a user 150 may create a "measurement descriptor" for a given service test. The measurement descriptor may be, for example, a yaml file that includes a single or multiple cycles during which infrastructure characteristics may change (e.g., different packet transmit and receive delay configurations in each cycle). In order to have a baseline to compare the results to, the first cycle may be configured to use an original or default state. That is, the measurements to be performed may run one or more services and any corresponding one or more monitors without any modification to existing infrastructure characteristics.

[0070] Figure 3 Schematically illustrates an example of a measurement descriptor according to one or more embodiments of the present disclosure. Figure 3 In the example of , the measurement descriptors describe two services (i.e., 360-degree video streaming and cloud gaming) occurring in each of five cycles (i.e., cycles 0-4) monitored by the packet and radio capture tool. Cycle 0 represents the default state, where the infrastructure is not manipulated. The following four cycles are described as having continuously increasing packet loss and background traffic.

[0071] Each of the number and type of services, monitoring tools, and infrastructure handlers may be described by a measurement descriptor, respectively. In some embodiments, the measurement descriptors may be user defined. In other embodiments, the measurement descriptors may be pre-configured.

[0072] Services and monitors are instructed to run in each of the cycles (e.g., continuously). In contrast, infrastructure manipulators are defined only for simple cycles, the length of which is configurable in at least some embodiments. Output data generated by monitors can be saved in output packages, for example, in a given destination directory at the end of a measurement.

[0073] The controller 122 of the mobile system 120 may be configured to control data collection and testing performed by the mobile system 120. In some embodiments, the controller 122 acts as an interface between the measurements performed by the mobile system 120 and a user 150 who wants to obtain the collected data. The user 150 defines the measurement descriptor discussed above, which specifies which monitor(s) 126 to use and which service(s) 124 to check with what infrastructure configuration(s) to use.

[0074] Back to Figure 2 When the controller 122 processes the descriptor file, the controller 122 may be configured to start and / or deploy one or more services into the network infrastructure (blocks 225, 235), start and / or deploy one or more monitors (blocks 227, 237), and create one or more manipulators that affect characteristics of the infrastructure (block 223). The controller 122 may also configure the infrastructure according to the measurement descriptors. The mobile system 120 monitors (block 240) and manipulates the infrastructure as specified by the measurement descriptors (block 250), saving measurement-related data collected along the way for later processing, e.g., after all periods indicated in the measurement descriptors have concluded (block 260).

[0075] One or more of the services 124 monitored by the mobile system 120 may be URLLC applications, and the mobile system 120 may be configured to predict the QoE for the URLLC application based on network and radio characteristics. Examples of such one or more services 124 include 360-degree video streaming, cloud gaming, immersive augmented reality (AR) / virtual reality (VR) applications, remote control, and industrial automation. Each service 124 may include start and stop methods by which the service 124 may be automatically controlled, such as by the controller 122.

[0076] As described above, one or more monitors 126 may be configured to monitor the underlying infrastructure and its radio environment, for example by evaluating data generated by one or more deployed services 122. Each monitor 126 may be a tool for collecting data about the monitored environment. Such tools may include, for example, tcpdump or spindump.

[0077] Collecting radio data may, for example, include using a customized application, since different devices may have different ways of extracting radio data. The collection of radio quality data allows the system to specifically measure low-latency services, since the quality of these services can be strongly affected by radio quality and mobility events, unlike non-URLLC services, which can be protected from such errors at the application level.

[0078] Similar to services 124, each monitor may include start and stop methods by which they may be controlled during measurement (eg, by controller 122). The data collected by each monitor 126 may be time-stamped and saved in separate folders for later use in producing a single structured data set. Figure 4 An example of the data output format for each infrastructure monitor is shown in the table below. Figure 4 As shown in , each monitor can generate a series of time-stamped entries, each entry being associated with one or more parameters obtained by the monitor.

[0079] Figure 5 Examples of combined outputs of one or more selected parameters are shown in the table. Figure 5 In the example of , a combined output structure of selected parameters with a common timestamp having a predefined time resolution is shown. Entries in the combined output may be tagged with end-user quality of service information, such as a QoE tag.

[0080] The infrastructure monitor provides at least the same or equivalent data and metrics as those typically used as input to the QoE model 118. That is, the infrastructure monitor can provide more detailed output for detailed investigations (e.g., deriving relationships between radio and transport metrics) and for different transport and radio degradations. These relationships can be used to improve the accuracy of the QoE model 118. Detailed packet capture (e.g., pcap) records can also be used for packet-level analysis, such as using Wireshark.

[0081] The manipulator 128 is responsible for configuring the infrastructure characteristics in each cycle of the measurement. For example, by using the linux command "tc", (among other things) packet loss, packet transmission delay and / or packet reception delay can be set. The start method of the manipulator 128 is called at the beginning of each cycle to configure the infrastructure, and the end method resets the infrastructure settings at the end of each cycle.

[0082] Figure 6An example of uplink and downlink data flows through example monitors 126 and manipulators 128 within components of a mobile system 120 is shown in accordance with one or more embodiments of the present disclosure. Radio environment manipulation 320 (e.g., shielding, interference) is performed at the radio interface 300 external to the data collection system 310. Uplink and downlink radio monitoring 330 is performed at the radio interface within the data collection system 310. Transmission manipulation 340 is performed between the radio interface 310 and packet capture 350 (e.g., using the "tc" tool). Transmission monitor 360 (e.g., spindump) is performed between packet capture 350 and QoE observation 370. QoE observation 370 (i.e., where the end-user quality of service is observed) is located at the end of the data flow.

[0083] Monitor 126 and manipulator 128 are adapted to monitor and manipulate both downlink and uplink data flows. Figure 6 In this example, the introduced degradation is directly monitored only in the downlink direction, since the degradation is introduced before the downlink data flow monitoring and after the uplink data flow monitoring.

[0084] Figure 6 The arrangement shown in enables simulation (e.g. using the "tc" tool) of packet loss, jitter, etc. in both the transport and radio interfaces. These can have different characteristics (e.g. statistics, burstiness), which can be set in the "tc" tool. It is also possible to add background traffic and / or bandwidth limitations (e.g. modeling congested cells and / or traffic shaping).

[0085] Although the uplink radio environment and transmission performance degradation cannot be directly observed with this arrangement, the impact of uplink performance degradation on the downlink data flow (eg, if an acknowledgement (ACK) or other packet is lost) can still be monitored and observed indirectly.

[0086] The monitored infrastructure is responsible for running services 124 started and / or deployed by the mobile system 120. The system is designed so that any browser-based service can be automatically started for measurement. Since the system is modular in design, testing a new service 124 only requires writing the code snippets needed to start and stop the service and adding that code to the code base.

[0087] Simulations of cloud gaming, video conferencing, and / or live video services can be implemented and controlled using the Selenium framework. Selenium can be used to simulate real-time users, for example, by simulating mouse movements and clicks in a predefined order and timing. Specific applications that have been experimentally simulated include NVIDIA GeForce Now, Jitsi Meet, and YouTube Live.

[0088] The data generated while using the service is saved, and the quality of service experienced by the end user is marked offline (manually or automatically). This can include, for example, saving all downloaded audio-visual data. Thus, according to one or more embodiments, the quality of service can be determined manually by watching the video manually or automatically using different algorithms.

[0089] High-resolution packet capture for URLLC example service types can be performed using RTP and QUIC protocols. Currently, most applications and services that require real-time communication (e.g., video conferencing applications, cloud gaming) use the WebRTC framework. The essence of the WebRTC framework is that the communicating parties use the RTP protocol to communicate directly point-to-point. In fact, DTLS-SRTP is used to encrypt the RTP payload, but not the RTP header. The synchronization source (SSRC) can be read from the common header and used to identify different audio and video streams. Timestamps and sequence numbers can be used to calculate packet loss, jitter, and other QoS KPIs.

[0090] A protocol analyzer such as Wireshark (or its terminal version tshark) can be used to capture and collect packets that may be associated with WebRTC communications. In addition to the information that can be extracted from the packets, the system can also be prepared to retrieve and use WebRTC's internal logs and statistics. This information is typically used for debugging and development, but for embodiments of the present disclosure, contains valuable data such as the decoding speed of audio and video packets, packet retransmission requests, and other data that is not normally visible due to encryption. For example, for a WebRTC session initiated from a Google Chrome browser, this information can be accessed from the "chrome: / / webrtc-internals / " page.

[0091] To analyze the measurements, the collected data is visualized. WebRTC data can be analyzed using an open source real-time analyzer tool called "Retina". Figure 7 An example measurement of a Jitsi Meet video conference is shown, where the packets per second of two video streams and the corresponding signal power are shown. Figure 7 As shown, radio degradation (masking) was applied around 14:07:15, which resulted in a sudden drop in signal power followed by a drop in the frequency of incoming video packet transmissions.

[0092] QUIC is a relatively new transport protocol, and there are an increasing number of packet capture tools available for capturing QUIC streams. Although only a few applications and services currently use QUIC, an increasing number of publications propose real-time communications over QUIC. Certain embodiments of the present disclosure collect and analyze QUIC packets.

[0093] The system includes an infrastructure monitor that captures pcap files that can be analyzed by a protocol analyzer (e.g., Wireshark) capable of capturing QUIC packets transmitted over UDP. Much of the data in QUIC packets is encrypted, and fields that are typically used as a convenient basis for analysis in TCP-based measurements are not available unless the encryption key is known. Thus, embodiments of the present disclosure evaluate the performance of a transport network without significant reliance on encrypted portions of the underlying protocol implementation. In fact, QUIC leaves little room for network operators to observe data flows, and developers have traditionally had to rely on a few unencrypted fields to measure round-trip time, throughput, bandwidth, and to compute metrics that can be input into machine learning modules to predict user experience or QoS.

[0094] Such measurements require a deep understanding of how QUIC works. There are applications tailor-made to measure the above parameters. To do this, the system can use the Spindump tool, which is a very convenient tool for measuring the characteristics of the transport network. A QUIC stream usually consists of several individual connections, and these connections are identified by a parameter called connectionID. Spindump is able to perform measurements for each connection, detect rotation transitions and report the RTT value.

[0095] The testbed implemented on the mobile system 120 can be built on the Spindump code base and can utilize this tool to measure the round-trip time for each QUIC connection. Since Spindump in debug mode typically generates very long output files, the data can be filtered to obtain specific relevant data. In particular, RTT values ​​for each connection can be obtained. Spindump outputs these values ​​when it receives a previously detected packet from the sender with a reversed rotation bit value. In QUIC, the rotation bit is reversed once per end-to-end RTT. The system can collect these rotation bit transition events and log them to an output folder so that these values ​​can be used later to correlate with the rest of the collected data.

[0096] The mobile system 120 uses a 4G / LTE connection. In some embodiments, the 4G / LTE connection is provided by an external radio device connected via a universal serial bus (USB) to support radio communications. The device may have an accessible application programming interface that can be called to retrieve different radio parameters. These API calls may be performed periodically and automatically, for example, using a Python script. The script may extract the desired values, such as RSRP, RSRQ, SINR, global cell identifier, eNodeB identifier, mode, uplink / downlink carrier frequency, modulation and coding, and / or uplink and downlink transmit power, from a hypertext transfer protocol (HTTP) response.

[0097] Each record is time stamped so that the data can be associated with the captured packets which are also time stamped. This data collection mechanism can be customized according to the measurement length and data polling frequency (ie, how often the LTE modem is polled for data).

[0098] The radio measurements are performed under different disturbance scenarios (ie during a given disturbance). Different active and passive methods may be used, such as partially and / or completely blocking the radio signals, thereby generating scenarios that can be repeated as needed.

[0099] For example, if we completely cover the LTE stick with a Faraday cage (if we can generate 100% coverage), we can block incoming radio signals and make it impossible to access not only 4G / LTE, but even GSM. Partially opening the Faraday cage or covering the stick with different foils is used to apply different levels of reproducible shielding.

[0100] In order to trigger a handover to another serving cell, the mobile system 120 may be physically moved. To this end, a vehicle may be used to move through a particular place and experience a handover. There may be many areas where a 4G / LTE network is not accessible or has a poor quality due to insufficient coverage. Measurements may be recorded under different conditions, such as in crowded urban areas and in areas with poor coverage (e.g. valleys, uninhabited areas). In this way, the way the signal strength or transmit power changes before / during a handover may be observed and the corresponding data may be associated with the captured packets.

[0101] Some test measurements were created experimentally to test the output format and the effects of Faraday cage-based radio disturbances. Figure 8 In the figure, you can see the RSRP values ​​for normal and disturbance measurements within 60 seconds. Fig. 9In , the power of the physical uplink shared channel (PPUSCH) is associated with the RSRP value during the 60-second measurement period. The Faraday cage is removed at the 30-second mark. Fig.10 In Figure 2, the RSRQ / RSRP correlation is visualized.

[0102] In another example test scenario, the radio receiver was blocked for about 20 seconds. As a result, inter-frequency switching occurred and the signal strength dropped. Figure 11-13 The observed transmission degradation is shown.

[0103] The mobile system 120 can be used, for example, to test new radio bearers and QoS profiles dedicated to new URLLC service types. As previously described, the mobile network operator can configure new 5QI classes and dedicated QoS profiles for new service types. Testing and investigating the operation and performance of such new profiles will be an important task in the service introduction phase. In this scenario, the mobile system 120 is used to run the new service and observe the end-user service quality under different radio and transmission conditions, while the radio and core network with the new profile are considered as the system under test.

[0104] The mobile system 120 may be adapted to determine boundary transmission characteristics where QoS and QoE degradation may have been experienced in the examined service. The framework implemented by the mobile system 120 may enable standardized and reproducible measurements of various services and generate training data for the QoE ML model. Due to the reproducibility of the measurements, boundary points in each characteristic (e.g., latency, packet loss, available bandwidth, etc.) may be determined after which QoS and QoE degradation may be experienced. In addition, the amount of training data for each service may be generated separately, enabling the sensitivity of the service to network transmission degradation to be examined separately. In some embodiments, multiple service-level QoE models may also be generated.

[0105] Different radio conditions (e.g., low coverage or high interference) may result in significant degradation in the quality of service at the end user. Mobile device 120 may associate weak radio parameters with poor user experience (e.g., jitter, delay, or completely unavailable service). Crowded urban areas are expected to result in high interference and poor QoE for users.

[0106] According to one or more embodiments, another example where a disturbed radio environment is used is to trigger a handover, which will eventually cause problems in QoE. Driving through several cells may enable the cell handover mechanism to be experienced. It is expected that the handover will severely affect the radio signal and service quality.

[0107] The data collected by the mobile system 120 can be used to train machine learning models that identify the relationship between network and radio conditions and QoE. These models can be used to predict the QoE experienced by the end user based on the encrypted packets collected by the network operator in the middle of the network. The predicted QoE can then be used in various network management operations to more effectively achieve the promised QoE.

[0108] In view of all of the above, embodiments of the present disclosure include, for example, a method 400 for training a QoE model 118, such as Fig.14 . The method 400 is implemented by the mobile system 120 and includes capturing packets of a URLLC service transmitted over a wireless connection (block 410). The method 400 further includes: introducing interference that affects the URLLC service (block 420). The method 400 further includes: monitoring the quality of service of the URLLC service when the URLLC service is affected by the interference and when the URLLC service is not affected by the interference (block 430). The method 400 further includes: updating the QoE model based on the monitored quality of service (block 440).

[0109] Other embodiments of the present disclosure include Fig.15 The example of FIG. 1 schematically shows the implemented mobile system 120 . Fig.15 The example mobile system 120 includes a processing circuit 710, a memory circuit 720, and an interface circuit 730. The processing circuit 710 is communicatively coupled to the memory circuit 720 and the interface circuit 730, for example, via a bus 705. The processing circuit 710 may include one or more microprocessors, microcontrollers, hardware circuits, discrete logic circuits, hardware registers, digital signal processors (DSPs), field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), or combinations thereof. For example, the processing circuit 710 may be programmable hardware that is capable of executing software instructions, such as stored as a machine-readable computer program 740 in the memory circuit 720. The memory circuit 720 of various embodiments may include any non-transitory machine-readable medium known or that can be developed in the art, whether volatile or non-volatile, including but not limited to solid-state media (e.g., SRAM, DRAM, DDRAM, ROM, PROM, EPROM, flash memory, solid-state drives, etc.), removable storage devices (e.g., Secure Digital (SD) cards, miniSD cards, microSD cards, memory sticks, thumb drives, USB flash drives, ROM cartridges, universal media optical disks), fixed drives (e.g., magnetic hard drives), etc., all of the above or any combination thereof.

[0110] The interface circuit 730 may be a controller hub configured to control input and output (I / O) data paths of the mobile system 120. Such I / O data paths may include data paths for exchanging signals over a network. The interface circuit 730 may be implemented as a single physical component, or as a plurality of physical components arranged in series or separately, any one of which may be communicatively coupled to any other physical component, or may communicate with any other physical component via the processing circuit 710. For example, the interface circuit 730 may include a transmitter 732 configured to send wireless communication signals and a receiver 734 configured to receive wireless communication signals.

[0111] According to a specific embodiment, the interface circuit 730 is configured to exchange signals wirelessly with the network. The processing circuit 710 is configured to capture packets of URLLC services transmitted via a wireless connection and interfere with radio conditions affecting the URLLC services. The processing circuit 710 is further configured to introduce interference affecting the URLLC services. The processing circuit 710 is further configured to monitor the quality of service of the URLLC services when the URLLC services are affected by interference and when the URLLC services are not affected by interference. The processing circuit 710 is further configured to update the QoE model based on the monitored quality of service.

[0112] Other embodiments include a computer program 740 comprising instructions that, when executed on the processing circuit 710 of the mobile system 120 , cause the mobile system 120 to perform the method 400 .

[0113] Other embodiments include a carrier embodying the computer program 740. The carrier is one of an electronic signal, an optical signal, a radio signal or a computer-readable storage medium.

[0114] Of course, without departing from the essential features of the present invention, the present invention may be implemented in other ways than those specifically described herein. The present embodiment is considered in all aspects to be illustrative and not restrictive, and all changes from the meaning and equivalent scope of the appended claims are intended to be included therein.

Claims

1. A method (400) implemented by a mobile system (120) for training a quality of experience (QoE) model (118), the method comprising: capturing (410) packets of an ultra-reliable low-latency communication (URLLC) service transmitted over a wireless connection; Introducing (420) interference affecting the URLLC service; monitoring (430) a quality of service of the URLLC service when the URLLC service is affected by the interference and when the URLLC service is not affected by the interference; Based on the monitored quality of service, the QoE model (118) is updated (440).

2. The method according to claim 1, further comprising: Each of the captured packets is associated with a time when the URLLC service is affected by the interference or a time when the URLLC service is not affected by the interference, wherein updating the QoE model (118) is further based on the associated packets.

3. The method according to any one of claims 1 to 2, wherein: Monitoring the quality of service of the URLLC service comprises: collecting uplink and / or downlink radio environment measurements supporting the URLLC service; and Based on the quality of service, updating the QoE model (118) includes using the collected radio environment measurements as training data to train the QoE model (118).

4. The method according to claim 3, wherein: The radio environment measurements include signal quality measurements of the downlink and transmit power of the uplink.

5. The method according to any one of claims 1 to 4, wherein: Monitoring the service quality of the URLLC service includes: collecting radio configuration parameters of one or more radio devices supporting the URLLC service; and Based on the quality of service, updating the QoE model (118) includes using the collected radio configuration parameters as training data to train the QoE model (118).

6. The method according to claim 5, wherein: The radio configuration parameters include timing advance and / or used frequency.

7. The method according to any one of claims 1 to 6, wherein: Monitoring the quality of service of the URLLC service comprises: collecting QoE configuration parameters associated with the URLLC service; and Based on the service quality, updating the QoE model (118) includes: using the collected QoE configuration parameters as training data to train the QoE model (118).

8. The method according to claim 7, wherein: The QoE configuration parameters include packet loss, jitter, and / or round-trip time requirements for providing QoE with respect to the URLLC service.

9. The method according to any one of claims 1 to 8, wherein: Monitoring the quality of service of the URLLC service comprises: collecting transmission measurements of the URLLC service; and Based on the quality of service, updating the QoE model (118) includes using the collected transmission measurements as training data to train the QoE model (118).

10. The method according to claim 9, wherein: The transmission measurements include bit rate and / or throughput measurements of the URLLC service.

11. The method according to any one of claims 1 to 10, wherein: Monitoring the quality of service of the URLLC service includes: generating periodic sample data of the quality of service, the periodic sample data having a resolution of no more than one second; and Based on the service quality, updating the QoE model (118) includes: using the periodic sample data as training data to train the QoE model (118).

12. The method according to any one of claims 1 to 11, wherein: Capturing packets of the URLLC service includes capturing real-time transport protocol RTP, secure RTP, and / or fast UDP Internet connection packets.

13. The method according to any one of claims 1 to 12, wherein: The packet of the URLLC service includes: an encrypted portion and an unencrypted portion; Monitoring the quality of service includes: deriving performance data of the URLLC service based on the unencrypted portion and without decrypting the encrypted portion; Based on the quality of service, updating the QoE model (118) includes using the performance data as training data to train the QoE model (118).

14. The method according to any one of claims 1 to 13, wherein: Monitoring the quality of service of the URLLC service includes: generating a round trip data sample for each of a plurality of rotated bit inversions detected within the captured packets; and Based on the quality of service, updating the QoE model (118) includes using the round-trip data samples as training data to train the QoS model.

15. The method according to any one of claims 1 to 14, further comprising: In response to the interference, a handover procedure is performed, wherein monitoring the quality of service includes monitoring the quality of service before the handover procedure and after the handover procedure.

16. The method according to any one of claims 1 to 15, wherein: Introducing the interference includes: introducing transmission data disturbance.

17. The method according to claim 16, wherein: Introducing the transmission data disturbance includes: introducing packet loss, jitter, delay, packet reordering, congestion, and / or packet fragmentation.

18. The method according to any one of claims 1 to 17, wherein: Introducing the interference includes introducing a disturbance to a radio condition of the wireless connection.

19. A mobile system (120), comprising: processing circuitry and memory circuitry, the memory circuitry storing instructions executable by the processing circuitry, whereby the mobile system (120) is configured to: Capture packets of an Ultra-Reliable Low Latency Communication (URLLC) service transmitted over a wireless connection; Introducing interference affecting the URLLC service; Monitoring the quality of service of the URLLC service when the URLLC service is affected by the interference and when the URLLC service is not affected by the interference; Based on the monitored quality of service, the QoE model (118) is updated.

20. The mobile system according to the preceding claim, further configured to perform the method of any one of claims 2-18.

21. A computer program comprising instructions which, when executed on a processing circuit of a mobile system (120), cause the mobile system (120) to perform the method according to any one of claims 1-18.

22. A carrier comprising a computer program as claimed in any preceding claim, wherein: The carrier is one of an electronic signal, an optical signal, a radio signal or a computer-readable storage medium.

Images