Automated management of network resources based on customer quality of experience

Abstract

Described herein are systems and methods that automate management of network resources based on an objective function that determines compliance with service level objectives (SLOs). The disclosed technologies use measurements with fine granularity to estimate customer quality of experience (QoE) and to establish SLOs. The disclosed technologies score SLO performance using these fine-grained measurements, such as passive speed measurements. The disclosed technologies use these measurements to improve or optimize SLO performance across a network in an automated fashion to thereby improve customer QoE.

Description

AUTOMATED MANAGEMENT OF NETWORK RESOURCES BASED ON CUSTOMER QUALITY OF EXPERIENCECROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 620,695 filed January 12, 2024, which is expressly incorporated by reference herein in its entirety for all purposes.BACKGROUNDField

[0002] The present disclosure relates to automatically managing network resources in a communications network based on customer quality of experience.Description of Related Art

[0003] Internet service providers (ISPs) offer network access to customers and may have different tiers of service with different advertised network speeds. Customers may decide on which ISP to use and to which tier to subscribe based at least in part on the offered network speeds. Network speed or throughput is typically measured using active measurement techniques. Active speed measurements involve introducing traffic into the network, also referred to as probe traffic, and tracking the probe traffic to measure network speed based at least in part on the time it takes the probe traffic to traverse one or more network links. In some instances, the ISP manages network resources based on quality-of-service objectives which may be related to these speed measurements.SUMMARY

[0004] According to a number of implementations, the present disclosure relates to a method of assessing customer quality of experience (QoE) in a communications network. The method includes acquiring fine-grained passive measurements of network resources of the communications network. The method includes acquiring application-layer measurements of network applications used on the communications network. The method includes applying one or more QoE models to generate customer QoE objectives based on a correlation between the acquired fine-grained passive measurements and the acquired application-layer measurements.

[0005] In some embodiments, the fine-grained passive measurements include passive speed measurements. In some embodiments, the fine-grained passive measurements include passive latency measurements. In some embodiments, the fine-grained passive measurements include passive usage measurements. In some embodiments, the fine-grained passive measurements are acquired with a rate that is at least 10 times the rate of network parameter adjustment.

[0006] In some embodiments, the application-layer measurements include a resolution of media content streamed to a customer. In some embodiments, the application-layer measurements include a bitrate of a network application provided to a customer.

[0007] In some embodiments, the fine-grained passive measurements are determined for service flow classes. In some embodiments, the customer QoE objectives are determined using machine learning algorithms that relate the finegrained passive measurements and the application-layer measurements to customer QoE. In some embodiments, the customer QoE objectives are expressed in terms of network metrics for individual customers.

[0008] According to a number of implementations, the present disclosure relates to a method of service planning in a communications network based on customer quality of experience (QoE). The method includes receiving one or more service level objectives (SLOs). The method includes determining an SLO score for individual SLOs based at least in part on compliance with the SLO as determined using an objective function. The method includes allocating network capacity based on the determined SLO score, the network capacity allocated to improve SLO compliance.

[0009] In some embodiments, the SLO score is determined for individual service flows. In some embodiments, the SLO score is determined for individual service flow classes. In some embodiments, the objective function uses finegrained passive measurements to determine the SLO score. In some embodiments, the fine-grained passive measurements include passive speed measurements.

[0010] According to a number of implementations, the present disclosure relates to a method of controlling a communications network based on customer quality of experience (QoE). The method includes adjusting scheduling weights ofindividual user terminals within a carrier based at least in part on compliance with an SLO, compliance with the SLO determined based at least in part on an objective function. The method includes balancing network utilization of user terminals for individual carriers of the communications network. The method includes allocating network resources to improve compliance with the SLO thereby improving customer QoE due at least in part to the relationship between the SLO and customer QoE.

[0011] In some embodiments, the method further includes determining a carrier SLO score using the objective function. In some embodiments, the method further includes determining a beam SLO score using the objective function.

[0012] In some embodiments, the communications network is a satellite communications network. In some embodiments, the objective function uses passive speed measurements to determine SLO compliance.

[0013] According to a number of implementations, the present disclosure relates to a method of monitoring service in a communications network based on customer quality of experience (QoE). The method includes detecting one or more QoE anomalies, the one or more QoE anomalies determined based at least in part on an SLO score calculated using an objective function. The method includes correlating one or more network events with the one or more QoE anomalies. The method includes generating an alert to indicate the one or more network events that caused the one or more QoE anomalies.

[0014] In some embodiments, the SLO score is determined using finegrained passive measurements as input to the objective function. In some embodiments, the fine-grained passive measurements include passive speed measurements.

[0015] According to a number of implementations, the present disclosure relates to a communications network that includes a gateway; a scheduler communicatively coupled to the gateway; and a network manager communicatively coupled to the scheduler. The network manager is configured to: receive a plurality of service level objectives (SLOs); determine an SLO score for each SLO using an objective function that uses fine-grained measurements and the corresponding SLO to determine the SLO score; and automatically adjust network resource allocation to improve an aggregate SLO score, the aggregate SLO score based on an aggregation of individual SLO scores.

[0016] In some embodiments, improving the aggregate SLO score automatically improves average customer QoE across the communications network.

[0017] According to a number of implementations, the present disclosure relates to a communications network that includes a gateway; a scheduler communicatively coupled to the gateway; and a network manager communicatively coupled to the scheduler. The network manager is configured to: acquire finegrained passive measurements of network resources of the communications network; acquire application-layer measurements of network applications used on the communications network; and apply one or more QoE models to generate customer QoE objectives based on a correlation between the acquired fine-grained passive measurements and the acquired application-layer measurements.

[0018] According to a number of implementations, the present disclosure relates to a communications network that includes a gateway; a scheduler communicatively coupled to the gateway; and a network manager communicatively coupled to the scheduler. The network manager is configured to: receive one or more service level objectives (SLOs); determine an SLO score for individual SLOs based at least in part on compliance with the SLO as determined using an objective function; and allocate network capacity based on the determined SLO score, the network capacity allocated to improve SLO compliance.

[0019] According to a number of implementations, the present disclosure relates to a communications network that includes a gateway; a scheduler communicatively coupled to the gateway; and a network manager communicatively coupled to the scheduler. The network manager is configured to: adjust scheduling weights of individual user terminals within a carrier based at least in part on compliance with an SLO, compliance with the SLO determined based at least in part on an objective function; balance network utilization of user terminals for individual carriers of the communications network; and allocate network resources to improve compliance with the SLO thereby improving customer QoE due at least in part to the relationship between the SLO and customer QoE.

[0020] According to a number of implementations, the present disclosure relates to a communications network that includes a gateway; a scheduler communicatively coupled to the gateway; and a network manager communicatively coupled to the scheduler. The network manager is configured to: detect one ormore QoE anomalies, the one or more QoE anomalies determined based at least in part on an SLO score calculated using an objective function; correlate one or more network events with the one or more QoE anomalies; and generate an alert to indicate the one or more network events that caused the one or more QoE anomalies.

[0021] For purposes of summarizing the disclosure, certain aspects, advantages and novel features have been described herein. It is to be understood that not necessarily all such advantages may be achieved in accordance with any particular embodiment. Thus, the disclosed embodiments may be carried out in a manner that achieves or optimizes one advantage or group of advantages as taught herein without necessarily achieving other advantages as may be taught or suggested herein.BRIEF DESCRIPTION OF THE DRAWINGS

[0022] FIG. 1A illustrates an example communications network that enables communication between a plurality of clients and an external network.

[0023] FIG. 1 B illustrates an example satellite communications network.

[0024] FIG. 2A illustrates a block diagram of an example network manager configured to automate network management driven by quality-of- experience (QoE) metrics.

[0025] FIG. 2B illustrates a block diagram of an example system for network optimization based on QoE metrics.

[0026] FIG. 2C illustrates that the system of FIG. 2B can be also used in a network that includes a plurality of satellites.

[0027] FIG. 3 illustrates a flow chart of an example method for assessing customer QoE in a communications network.

[0028] FIG. 4 illustrates a flow chart of an example method for service planning in a communications network based on customer QoE.

[0029] FIG. 5 illustrates a flow chart of an example method for controlling a communications network based on customer QoE.

[0030] FIG. 6 illustrates a flow chart of an example method for monitoring service in a communications network based on customer QoE.

[0031] FIG. 7 illustrates a block diagram of a subsystem of a communications network, such as a scheduler or gateway.DETAILED DESCRIPTION OF SOME EMBODIMENTS

[0032] The headings provided herein, if any, are for convenience only and do not necessarily affect the scope or meaning of the claimed invention.Overview

[0033] Network service providers (e.g., Internet service providers (ISPs) and other such entities) provide, manage, and control network resources for their customers. Typically, network service providers focus on quality of service (QoS) when managing their network resources. However, it may be more advantageous to focus on ensuring that customers have a good experience using their network. This can be done by focusing on quality of experience (QoE) in addition to or as opposed to focusing on QoS. As used herein, the quality of experience, or QoE, can refer to the customer’s perceptions of a service provider’s network based on the customer’s experience using that network and / or metrics associated therewith.

[0034] Accordingly, the technologies disclosed herein enable a network service provider to manage network resources in an automated fashion to improve or optimize around QoE. This is distinct from managing network resources based solely on QoS where a network service provider manages network resources based on service level objectives (SLOs) that are directly related to observable network characteristics such as speed, latency, jitter, etc. When a network service provider manages resources to QoE rather than to QoS, the QoE SLOs are related to achieving targeted levels of customer experience for various end-user applications as opposed to the QoS SLOs, which may be related to ensuring that one or more of the particular observable network characteristics is achieving respective target levels for that QoS. That is, managing to QoS and to QoE both involve managing network resources based on measurable network characteristics; however, managing to QoS does not have a direct connection to customer QoE, which is why managing to customer QoE, as described herein, is configured to provide a superior experience for the customer.

[0035] The end result of managing resources to QoE rather than to QoS is that customers perceive that the network is providing superior performance. This may even be the case where one or more network characteristics for the customer in the QoE-managed network are inferior to those in a network managed around optimizing QoS. For example, QoS may be strongly affected by the allocation ofbandwidth and other such resources to customers, even where there is not a commensurate demand. Because relatively high bandwidth may be allocated to customers, a network service provider may claim a high QoS. However, the average or aggregate customer experience in this situation is not as high as it could be because there are unused resources that could be allocated to other customers which may improve the overall or average customer experience. That is, with a finite amount of bandwidth (or any other network resource), managing based on bandwidth allocation alone may result in an over-allocation of bandwidth to some customers and an under-allocation of bandwidth to other customers. Thus, the disclosed systems that optimize around customer QoE may allocate fewer network resources for certain network applications (e.g., web browser applications) and more network resources for other network applications (e.g., video applications) with the result of an overall improvement in customer QoE. In these situations, customer QoE may improve where typical QoS metrics would otherwise indicate a degradation in the quality of service.

[0036] As introduced above, the technologies disclosed herein enable automatic management of a communications network around customer QoE. The disclosed technologies for automated network management are configured to globally improve or optimize network resource utilization using a single objective function. To achieve this functionality, disclosed herein are technologies that link customer QoE to network metrics. The objective function serves to express an SLO in terms of network performance that can be measured, and the SLO in turn is reflective of customer QoE. In some instances, economic considerations can be added to the single objective function. For example, the objective function may determine customer QoE weighted by economic value to reflect business objectives in conjunction with QoE objectives. The objective function can express economic value as a yield that considers the value obtained from utilization of the available bandwidth and the cost for not achieving a bandwidth target. That is, the objective function can account for the value obtained by delivering a bit and / or the economic loss associated with not being able to deliver the bit.

[0037] A challenge in this approach is how to capture customer QoE in terms of measurable parameters (e.g., network metrics, network measurements, observable network characteristics, etc.). This can be accomplished using SLOs that express performance objectives in terms of network metrics, where theperformance objectives are constructed to achieve targeted customer QoE. Then, the challenge becomes how to manage network resources to change the network metrics to improve customer QoE. This can be accomplished using the objective function (as introduced above) that uses the network metrics to determine SLO compliance (e.g., using an SLO score), thereby connecting measurements and resources to customer QoE. Thus, the disclosed automated network management technologies improve or optimize utilization of network resources using the objective function to increase or maximize SLO compliance, the logical consequence of which is to improve or optimize customer QoE.

[0038] By way of example, a customer QoE objective can be to provide customers with a high-definition video streaming experience. The customer QoE objective can be restated in terms of an end-user application objective. For example, the end-user application objective can be to support a live video stream with a median of 720p resolution and 30 frames per second during peak busy hours, with the live video stream dropping down to 480p resolution no more than 5% of the time. In some embodiments, the player video resolution can be first translated to encoded video bitrate and then the networking conditions required to deliver that bitrate can be determined. It should be noted that the bitrate required for a given resolution can vary widely as it is a function of, for example, codec, content, and / or encoding techniques. However, by measuring the cumulative density functions of bitrate versus encoding for a wide variety of content, a statistical assessment can be made of the likelihood that a given bitrate supports a given resolution. The bitrate objectives can then be expressed as SLOs and the ability to achieve these SLOs can be determined in multiple ways. In some embodiments, machine learning models can be used that employ a large set of features based on network metrics (e.g., various statistics derived from passive speed measurements over a range of time windows) to predict bitrate performance and, hence, SLO compliance. In this example, the objective function can use an SLO score that is based on the likelihood of achieving the target bitrates indicated in the corresponding SLO. In certain embodiments, the bitrate objectives can be restated in terms of the required link bandwidth for the video player to request to achieve various bitrates at specified percentiles. In certain implementations, the SLO can be defined to apply to each user sharing the network link, in which case the number of concurrent video streams sharing the network link can be measuredand the supportable bitrate can be estimated based on a combination of network speed measurements and traffic flow analysis. For example, an SLO that is configured to provide HD video and only drop to SD video a small percentage of the time to each concurrent video stream can be expressed in terms of achieving a certain link bandwidth 95% of the time and another link bandwidth 5% of the time. In this case, the objective function can be configured to generate an SLO score based on the ratio of the delivered bandwidth to the target bandwidth for each video quality level and percentile indicated in the SLO. That is, because the network can be configured to measure link bandwidth, the statement of link bandwidth can become the SLO for the high-definition video experience, which forms at least a portion of the customer QoE objective. As described herein, the objective function can be configured to generate an SLO score which reflects compliance with the corresponding SLO.

[0039] By way of overview, it is useful to define a product (e.g., a product offering by a network service provider) as a collection or bundle of primitives. In the context of a communications network, each primitive is an SLO for a group of similar network applications (e.g., video streaming, latency-sensitive traffic such as web browsing and gaming, and bulk downloads such as game and software downloads). A product collects these primitives and bundles them together to define key attributes of the target customer QoE. Thus, it is useful to determine a target customer QoE for individual primitives (e.g., collections of similar network applications) and then to express that target customer QoE as an SLO. In some embodiments, a traffic service primitive can be defined as a combination of a set of traffic associated with related applications and an associated SLO.

[0040] To achieve automated network management, these primitives can be entered into the disclosed automated network management systems, the primitives together forming a customer product. The disclosed communications networks are self-optimizing to achieve the target outcomes (e.g., as close as possible to achieving all SLOs that are set). This self-optimization is achieved based at least in part on the use of objective functions that are configured to quantify compliance with the primitives and the product. In short, the disclosed systems accept as input the SLOs (or primitives) that together form a product and the systems use one or more objective functions to assess compliance with the SLOs (e.g., via an SLO score). The systems then manage network resources toimprove or optimize the objective function(s) (e.g., globally improving or optimizing SLO scores). The output of this process is an improved or optimized communications network. In some implementations, this eliminates the need to independently optimize portions of the network and / or to independently plan capacity (e.g., to manually adjust scheduling weights, balance carrier utilization, schedule carriers in the system, etc.). In other words, customers select a product and generate network traffic. SLOs are constructed that represent performance objectives that are to be achieved based on the selected product, where the SLOs correlate to customer QoE. The SLOs drive automation of the network and capacity planning so that the communications network delivers on the promises of the selected products.

[0041] Accordingly, the disclosed technologies map network resources to customers on a network based on target customer QoE objectives. The disclosed automated network management systems and methods are configured to take available network resources and map them to customers to improve or maximize utility for the network applications the customers are using. This can be done by putting primitives into separate flows or virtual pipes, where one or more network applications are mapped to an individual virtual pipe. The disclosed systems and methods automatically improve or optimize network utility based on products, where a product is a collection of primitives (e.g., which makes the virtual pipe size of the product a sum of the virtual pipes for the primitives). Individual virtual pipes are assigned an SLO such that a product can also be considered an aggregation of multiple SLOs. The disclosed systems and methods use the SLOs and the mapping of network applications to SLOs as inputs to adjust and optimize the communications network to achieve those SLOs. As described herein, this logically results in improving or optimizing customer QoE. Thus, the disclosed technologies use SLOs as input and provide globally improved or optimized network resource allocation as output, where the connection between the input and the output is provided by objective functions that determine SLO scores based on network metrics (e.g., measured network parameters), with the ultimate outcome being improved customer QoE.

[0042] The disclosed technologies are advantageous in a constrained communications network (e.g., where one or more network resources are insufficient to meet every demand put on the network (e.g., demands from all usersof the network)). Advantages arise from the disclosed technologies due at least in part to automated network management determining how to allocate constrained network resources to improve or maximize customer satisfaction. By focusing on customer experience and creating SLOs that reflect this focus, these priorities are propagated throughout the network via capacity planning and network control, with the outcome of improved customer QoE. In other words, the QoE-driven network management systems and methods use SLOs as input, perform measurements of network parameters, and produce SLO scoring via the objective function where SLO scoring is then used to optimize the communications network by automatically adjusting network control parameters (e.g., weighting, load balancing, etc.) to improve the SLO scores. In some implementations, SLOs can be entered into the system (e.g., via APIs) and the system then self-optimizes with the simple objective to optimize customer QoE which is accomplished by optimizing around SLO scores (the output of the objective function).

[0043] Objective functions may account for economic value as well. For example, the objective function can express economic value as a yield that considers the value and cost associated with delivery and non-delivery of bits. For example, the SLO may establish a target rate of 5 Mbps for a particular service flow but the objective function, weighted by economic value, may dictate providing 120% of that speed because the overall economic value of providing that speed is increased or optimized as compared to the target rate.

[0044] Objective functions, such as QoE assessments and SLO scoring, can be used to manage network resources. The objective function that is implemented to automate the management of network resources can be configured to improve or optimize customer QoE. In other words, SLOs can be established that directly correlate with customer QoE objectives.

[0045] Accordingly, network resources can be managed to improve or optimize around customer QoE via an objective function, such as the SLO score. The objective function can be used as an input to a service planning process, which involves generating service level objectives (SLOs), scoring compliance with those SLOs, and managing capacity based on those scores (e.g., capacity planning). Capacity planning may involve balancing network resources to achieve targeted customer QoE via SLO compliance. Objective functions, such as SLO scoring, can also be used to control network resources by using the SLO scores to manageshared network traffic, manage network segment traffic, and to manage global network resource traffic. Objective functions can also be used to monitor QoE and to generate alerts around QoE anomalies, as described in greater detail herein. Thus, objective functions and SLOs are valuable for managing network resources to achieve a desired or targeted customer QoE. The objective functions described herein can be used as feedback that can be used to automate network control and management. In some embodiments, a single objective function can be used within the network to automate capacity planning and the allocation of network resources. In such embodiments, the automated network control system is configured to automatically adjust network parameters to improve or optimize the objective function, where improvement or optimization of the objective function consequently improves or optimizes customer QoE.

[0046] In some embodiments, different parts of a network can be optimized or configured at different time scales using the objective function. At each time scale, a control loop or other control algorithm or mechanism (e.g., machine learning) can be used that improves or optimizes that part of the network based on the objective function. By keeping the time scales of individual control loops separated by a sufficient amount, the control loops can run independently of one another, improving, or optimizing the same objective function. In this way, the disclosed technologies optimize network resource allocation around network utility and / or value. The disclosed network automation technologies are particularly applicable in networks that have capacity constraints where an objective function can be used to allocate capacity to increase network utilization and value and, relatedly, to increase customer QoE. It should be noted that although control loops are discussed herein, control loops represent an example of a way to implement automated network control. As the number of variables or dimensions around which network control is automated increases, different control techniques can be implemented such as machine learning techniques for network optimization. Moreover, in addition to control loops or as an alternative to control loops, other mathematical functions can be used for multidimensional improvement or optimization.

[0047] In some embodiments, different parts of the network are optimized or configured at the same time scale using the objective function. In such embodiments, the various dimensions of the network can be improved or optimizedconcurrently. This can be achieved using artificial intelligence techniques, such as a machine learning (ML) based optimizer. In various implementations, it may be difficult or impossible to update all network configurations concurrently. In such implementations, the sets of parameters that are improvable or optimizable at the same time scale are improved or optimized together. Control loops or ML-based optimizers may be used in such implementations, depending at least in part on the number of variables or dimensions being concurrently improved or optimized.

[0048] The disclosed QoE-driven network automation technologies establish a target QoE in the network metrics. Compliance with the target QoE can be measured using an objective function. In some embodiments, the objective function is called an SLO score. By way of example, the SLO score can be a ratio of the speed experienced by a customer to the speed in the SLO. In this example, an SLO score of less than 1 indicates that the SLO is not being met. Relatedly, a network can be considered full if there is at least one SLO that is not being met. That is, the network resource can be considered to be fully loaded where there is an SLO score that is less than 1 . Thus, in this example, network resources can be allocated in an automated fashion so that the SLO scores across the network are greater than or equal to 1 . Relatedly, customer QoE and utilization of network capacity can be maximized by achieving an SLO score of 1 for customers across the network.

[0049] In some implementations, the disclosed technologies use passive speed measurements to estimate customer quality of experience and to establish SLOs. In some implementations, the disclosed technologies use fine-grained measurements, such as passive speed measurements, to assess SLO performance. In some implementations, the disclosed technologies use these finegrained measurements to improve or optimize SLO performance across a network in an automated fashion.

[0050] Thus, the disclosed technologies can manage network resource utilization to improve or optimize customer QoE based on the output of one or more objective functions. A result of improving or optimizing network resource utilization around QoE is that a network service provider can improve customer perception of the network while reducing actual utilization of the network for that customer. In other words, the customer may perceive the network as being faster even though actual network speeds (e.g., bandwidth) are slower or unchanged for thatcustomer. This results in an improved or optimal distribution of network resources that allows the network service provider to achieve targeted customer QoE for a larger number of customers.

[0051] Customer QoE is typically related to the customer’s expectations and experience. To illustrate this relationship, the following example is provided. A customer that is not using throughput-intensive applications will not experience a difference between the speed offered as part of an SLA and the speed actually experienced on the network. In other words, the network applications the customer uses will transfer data as expected without experiencing unexpected delays or buffering because the bandwidth the customer is using is smaller than the bandwidth available to the customer. This translates to a high quality of experience for the customer because the customer cannot tell any difference between the experienced speed and the offered speed. This is true even though the customer may not be actually using (or provided) the offered speed. Thus, any bandwidth the customer is not using can be redistributed to other customers without affecting the customer QoE.

[0052] Service level agreements (SLAs) are agreements between the network service provider and a customer that dictate promised service levels around one or more measurable metrics (e.g., uptime, responsiveness, speed, bandwidth, throughput, latency, etc.). An SLO serves as a benchmark for indicators, parameters, or metrics defined with specific service level targets. Individual objectives may be an optimal range or a specific value for a particular service function or process that is a part of the network service promised to the customer. Thus, SLAs represent a contractual agreement with a customer whereas the SLOs represent the experience that the network service provider desires to provide to the customer. In some situations, SLOs align with SLAs. However, in some cases, the SLAs may represent achievable network performance standards but meeting those standards may not lead to high customer satisfaction. In those situations, SLOs can be set that exceed the SLA so that the customer QoE is such that the customer desires to stay with the network service provider rather than finding another provider. Thus, SLOs can be considered to be performance benchmarks that the network service provider targets to achieve a desired customer QoE, which typically results in compliance with the SLA. This means thatSLOs can be used to set targets for customer QoE and to manage network resources to achieve targeted customer QoE.

[0053] In addition, SLOs can vary between network applications. For example, SLOs can be configured to trade off performance between background downloads and video streaming. The SLOs can be configured to give video streaming more capacity than background downloads when there is congestion, thereby increasing network utility to the customers utilizing video streaming applications over those customers utilizing background download applications. In this way, SLOs can be configured to allocate network resources more efficiently so that available network capacity is used in a way that increases or maximizes utility of all network applications.

[0054] Functions that assess or score SLO compliance thus represent objective functions that can be used in capacity planning and management of network resources. Compliance with SLOs can be determined based at least in part on real time network measurements (e.g., passive speed measurements or other fine-grained measurements). In a QoE assessment phase, network measurements can be correlated with customer QoE based on QoE measurements and QoE modeling, as described herein. The resulting QoE assessments can be used in the generation of SLOs. Compliance with the SLOs, as determined by the objective function, can be used as feedback for managing network resources.

[0055] SLO scores can be used for QoE-driven network control wherein the disclosed technologies adjust network resource allocation based on SLO scores. Network resource allocation can happen at the level of an individual customer. For example, network resource allocation for individual customers’ service flows can be based at least in part on packet scheduling that uses scheduling weights for service flows and customers. These scheduling weights influence the resulting SLO scores and can thus be adjusted to improve SLO scores. Network resource allocation can happen at the level of groups of customers. For groups of customers, network resources can be adjusted to have SLO scores of the groups meet the targets set in the SLOs in a fair manner. Network resource allocation can happen at the level of carriers or frequency bands of modulated signals (e.g., customers can be assigned to different carriers in a network such as different carrier frequencies in a satellite network). For carriers,network resources can be adjusted to have the average SLO scores of the carriers meet the targets set in the SLOs in a fair manner. In that way, SLO scores can be used to allocate network resources to customers and to allocate customers to network resources. In some embodiments, network resource allocation can happen at the level of network instantiations. For example, in the case of a satellite network, one satellite could be treated as one network instantiation.

[0056] A problem arising in network resource management for a network service provider is how to manage resources on a congested network. The network service provider must determine how many resources to allocate to different services (e.g., background downloads, media content streaming, etc.). As described herein, to improve or optimize customer QoE in cases where the network is fully congested, the network service provider can decide to allocate resources to service flows that bring more value to a larger number of customers, attempting to maximize utility of the network from the customers’ point of view based on QoE. In certain instances, this may result in making tradeoffs globally across the network to improve customer QoE rather than attempting to optimize bandwidth allocation in accordance with QoS metrics. Network measurements, such as passive speed measurements, can be a measure or indication of congestion on the network. Because network congestion affects customer QoE, the technologies disclosed herein can utilize network measurements acquired with high frequency and / or fine granularity (e.g., passive speed measurements) to improve or optimize network utilization based on QoE.

[0057] Advantageously, the disclosed technologies enable network service providers to focus on selling network access rather than focusing on offering network speeds while still satisfying customer demands. This is done by managing network resources in an automated fashion around customer QoE. Managing network resources around customer QoE allows for trade-offs in network capacity. That is, a network service provider can allocate more network resources where the additional resources are likely to result in increased customer QoE and decrease network resource allocation where it is unlikely to decrease customer QoE. For example, optimizing customer utility rather than network speeds allows a network to prioritize video streaming while de-prioritizing bulk downloads when the network is congested. On average, this increases customer QoE. In contrast,focusing on QoS may result in reserving bandwidth for network applications such as speed tests, which may provide less value to customers.

[0058] Although unexpected, the disclosed approach to QoE-driven network management (rather than QoS-driven network management) has been tested and has been shown to improve customer QoE. For various customers, network resource allocations for speed tests were reduced while network resource allocations were increased for video applications. In this situation, customers reported a superior experience, reporting faster perceived speeds even though speed tests showed slower network speeds.

[0059] More particularly, various network parameters were adjusted for various customers and then these customers were polled to receive feedback on their network experience. Overall, customers reported perceived higher network speeds even when the network was not optimized to provide the highest speeds possible to customers. For example, at the beginning of the testing period, speed tests were weighted highest in the scheduler so that customers would see higher speeds. After, applications that provide a superior customer experience (e.g., video) were weighted higher in the scheduler with the result being that customers perceived higher speeds and reported being more satisfied with the network. During this period, at first there was a focus on optimizing video quality. In response, customers commented on the quality of the video experience but reported unsatisfactory lags elsewhere and stated a desire for higher network speeds. Then, SLO targets were adjusted which resulted in reducing the ratio of speed test-to-video weights by a factor of 2, to further bolster the video experience at the expense of speed tests. In response, customers reported that they noticed that the network seemed faster, e.g., faster loading and opening of applications, better handling of multiple devices, etc.Example Network Measurements for QoE-Driven Network Automation

[0060] Customer QoE is influenced by the customer’s expectations and experiences which, in some situations, relate to the expected network speed (e.g., the bandwidth offered, advertised, or promised by the service provider) and the experienced network speed (e.g., the bandwidth actually experienced by the customer) as well as other factors (e.g., latency, usage, jitter, etc.). To measure and understand the customer experience, then, the network service provider canuse network speed metrics to assess the QoE of the customer. Network speed is a system metric often used to measure or characterize a network carrier, where the network speed may correspond to a speed at which the network carrier can transfer data packets. Techniques exist that attempt to measure network speeds. For example, network speed measurements such as active speed measurements (ASMs) or embedded service platform (ESP) speed tests can be used to provide information on certain aspects of the customer’s network speed on the service provider’s network. However, such measurements may not fully represent the network speed experienced by the customer. In addition, service providers may offer network speeds as part of service level agreements with customers and customers may expect to experience the offered network speeds. However, the offered network speed may not represent the network speed experienced by the customer when the customer does not have enough data demand to utilize the allocatable capacity, when the customer is experiencing high latency, and so forth.

[0061] Furthermore, although active speed measurements may accurately measure network speeds, such measurements introduce additional traffic into the network, which may be undesirable, especially in congested communication networks. Furthermore, because active speed measurements add load to the network, they cannot be run with sufficiently high frequency to provide measurements at the timescale at which network applications make their decisions. Consequently, active speed measurements do not support modeling the application-layer QoE, in contrast with the case with passive speed measurements or other fine-granularity measurements described herein. In addition, active speed measurements may not accurately measure or reflect the network speed experienced by a customer due at least in part to the active speed measurements measuring only certain aspects of network speed over a communications network or over portions of the communications network.

[0062] In contrast, passive speed measurements do not introduce additional traffic into the network and can be used to more accurately reflect the speed experienced by the customer. Passive speed measurements can be characterized as measurements of network speeds that do not inject traffic into the network to determine network speeds. Rather, passive speed measurements use data arising from actual usage of the network by customers to determine network speed metrics. Passive speed measurement techniques can be used to determinea QoE speed metric that accurately characterizes the customer experience. In general, passive speed measurements can be used to derive QoE metrics that can be used to better manage the customer’s experience on a communications network. That is, passive speed measurements can be used as a metric to assess, measure, or determine customer quality of experience without adversely affecting network resources as may be done using active speed measurements. Examples of passive speed measurements are further provided in Int’l Pat. App. No. PCT / US2024 / 045713 filed September 6, 2024, which is incorporated in its entirety herein for all purposes.

[0063] Although the disclosure herein focuses on using passive speed measurements, it is to be understood that other passive measurements can be used in place of passive speed measurements, such as passive latency measurements, passive jitter measurements, passive usage measurements, and the like. These passive measurement techniques can be differentiated from other passive techniques that attempt to predict customer QoE, such as determining a signal strength at a user device (e.g., a cellular phone) to assess customer QoE. A differentiating factor is that the passive measurement techniques referred to herein include measurement techniques that utilize information related to actual network usage as opposed to measurements techniques that utilize characteristics or performance of a customer’s device (e.g., signal strength). Additional metrics may be used in the described technologies as well, such as delay and / or packet loss. In some embodiments, delay can be measured in a number of different ways including, for example, by using information in IP headers to assess the time deltas. In addition, although the disclosure herein discusses passive measurements, other measurements can be used as input to objective functions that are used to manage network resources. In certain implementations, active speed measurements can be used in determining network metrics. These can be used in addition to or as an alternative to passive measurements.

[0064] The disclosed technologies leverage the use of fine-grained measurements of network resources. To achieve these fine-grained measurements (e.g., measurements made with high frequency), passive measurement techniques are used so as to not adversely affect network performance as may be the case with active measurement techniques, as described herein. As used herein, high frequency measurements or fine-grainedmeasurements refer to measurements that are made at a rate faster than an application is making decisions. For example, if network resources are adjusted every 10 seconds, then measurements can be acquired every second or faster. It should be noted, however, that the rate at which network applications make decisions depends on the application and other related factors. The more real-time the application is, the faster the application should respond. For example, for video applications, the rate at which the video player application makes decisions is variable-where there is less data buffered the application responds more quickly to changes in the network whereas where there is more data buffered the application responds more slowly. In general, the measurement rate can be between 10 to 20 times the decision rate (e.g., at least 10 measurements for every decision) to achieve responsive QoE-driven network control.

[0065] The high rate of measurements allows for automated network systems to model customer experience (e.g., using machine learning) and to set SLOs to achieve targeted performance. For example, targeted performance may be providing HD video at least 90% of the time for a customer. High rates of measurements allow faster adjustment of network parameters to set and achieve SLOs. Thus, because high rates of measurements are used in the disclosed technologies, active measurement techniques are undesirable because they would introduce too much load into the system, thereby making passive measurement techniques advantageous. Passive speed measurements, for example, enable high measurement rates resulting in fine granularity in information which are advantageous for the disclosed technologies around QoE assessment, service planning, network control, and service monitoring. It is to be understood, however, that any measurement technique that can provide high rates of measurement or fine granularity in information similar to passive speed measurements can be used with the disclosed technologies for QoE-driven network automation.

[0066] The disclosed technologies use high frequency measurements in association with objective functions to control network resources in an automated fashion. For example, QoE measurements and network measurements (e.g., passive speed measurements) are acquired over a period of time (e.g., minutes, hours, days, weeks, months, etc.) to assess customer QoE on the network. The assessed customer QoE, based on measurements acquired over the period of time, is mapped to an objective, such as an SLO.

[0067] Once SLOs are established, high-frequency network measurements (e.g., passive speed measurements) are acquired in real time (or near real time) and compliance with the SLO is assessed or scored (e.g., via an objective function) to produce an objective score or metric (e.g., an SLO score). Network resources are then managed to improve or optimize the objective score. Thus, the technologies disclosed herein manage network resources using objective functions.Example Communications Networks

[0068] FIG. 1 illustrates an example communications network 100a that enables communication between a plurality of clients 110 and an external network 160 (e.g., the Internet). The communications network 100a includes a communications system 140a having a scheduler 130a, a gateway 150a, and a QoE-driven network manager 145a. The communications system 140a is configured to manage network traffic between the plurality of clients 1 10 and the external network 160 using the QoE-driven network manager 145a. The QoE- driven network manager 145a manages network resources in an automated fashion based on high-frequency network measurements, one or more SLOs, and an objective function, as described herein. In some embodiments, the communications system 140a can be a wireless communications system configured to wirelessly communicate at least a portion of the network traffic. In certain embodiments, the communications system 140a can be a terrestrial communications system. In various embodiments, the communications system 140a can be a satellite communications system. In some embodiments, the communications system 140a is a combination of terrestrial, wireless, and / or satellite communications systems.

[0069] The scheduler 130a is configured to manage network resources. To send data to the external network 160, the scheduler 130a allocates network resources to the plurality of clients 1 10, creating a schedule of transmission for devices. Then, based on the schedule, the individual clients of the plurality of clients 1 10 transmit data using the allocated resources. Similarly, to send data from the external network 160 to the plurality of clients 1 10, the scheduler 130a allocates network resources to network devices and components configured to transmit data over the communications system 140a to the plurality of clients 1 10, creating aschedule of transmission for those devices. The QoE-driven network manager 145a is configured to provide input to the scheduler 130a to manage network resources in a way that improves customer QoE, as described herein.

[0070] The gateway 150a is configured to direct network traffic between the plurality of clients 1 10 and the external network 160. The gateway 150a can receive network traffic from network applications and direct the received network traffic to a targeted destination in the external network 160. Similarly, the gateway 150a can receive network traffic from the external network 160 and direct the network traffic to the intended recipient client of the plurality of clients 110. The gateway 150a can manage network traffic divided into service flows, as described herein.

[0071] The scheduler 130a can be configured to allocate network resources to service flows from the plurality of clients 1 10 and / or to allocate network resources to service flows destined for plurality of clients 1 10. In some implementations, the scheduler 130a can allocate resources using scheduling frames that divide time into discrete chunks referred to as scheduling epochs. Each scheduling frame can be divided into a plurality of scheduling epochs with each scheduling epoch having a duration (e.g., 1 ms, 5 ms, 20 ms, etc.). Within a scheduling epoch, resource blocks can be allocated to individual client devices. Each resource block can represent an allocation of network resources such as frequency and time. Each scheduling epoch is capable of transmitting a certain amount of network data within the scheduling epoch. The scheduler 130a can be configured to allocate network resources to multiple clients of the plurality of clients 1 10 within a single scheduling epoch.

[0072] The QoE-driven network manager 145a is configured to acquire fine-grained measurements related to network performance (e.g., passive speed measurements, bytes, etc.). The QoE-driven network manager 145a is further configured to determine QoE assessments based on the fine-grained measurements. In some embodiments, the QoE assessments are determined using machine learning. In some embodiments, the QoE assessments are determined using other algorithms that are trained on statistical models. The statistical models can be trained on data acquired during operation of the network. The QoE-driven network manager 145a can be configured to improve or optimize network resource allocation given a set of traffic flow primitives and associatedSLOs. The QoE-driven network manager 145a can be further configured to determine SLO scores based on network measurements and the generated SLOs. This feedback can be used for capacity planning, service monitoring, and automated network control on the communications system 140a.

[0073] The SLO scores can be generated using an objective function and can be used for QoE-driven network control. For example, the output of the objective function can be used in an automated fashion to adjust scheduling weights used by the scheduler 130a. This can be used to adjust network resource allocation for individual service flows and / or customers, for groups of customers, and / or for carriers in a network. In other words, the results of the objective function can be used to manage shared network traffic, network segment traffic, and global resource traffic, as described in greater detail herein. In certain embodiments, QoE-driven control loops informed by passive metrics can be used to adjust weights used by a layer 2 media access controller (MAC) scheduler (e.g., the scheduler 130a), to load-balance terminals across carriers in a beam (e.g., a beam of a satellite network), to modify power allocated to a beam, to modify beam footprints, to change satellite TDMA (time-division multiple access) slot allocations, to manage beam frequency allocations, to configured group bandwidth control, and the like.

[0074] In some implementations, SLO scores for customers can be used to determine a carrier SLO score that can be used to balance resources in a beam (e.g., in a satellite network). This can then be used to determine a beam SLO score that can be used as input to the scheduler 130a to balance and manage network resources. The SLO scores can also be used by the QoE-driven network manager 145a to detect anomalies in SLO scores to determine correlated events and to generate alerts regarding degradations in network performance, as described in greater detail herein.

[0075] In some implementations, the QoE-driven network manager 145a is configured to receive network performance parameters and statistics from the scheduler 130a and the gateway 150a and to derive speed metrics to monitor a quality of experience of the plurality of clients 1 10. The speed metrics and / or the quality of experience can be determined using passive measurement techniques. In some implementations, the scheduler 130a and / or the gateway 150a determine the speed metrics, which can then be shared with the QoE-driven network manager145a. In some implementations, the QoE-driven network manager 145a is configured to use speed metrics and / or other performance metrics to manage network performance in the communications network 100a. This can be done to ensure a satisfactory quality of experience for the plurality of clients 1 10.

[0076] FIG. 1 B illustrates an example satellite communications network 100b. The satellite communications network 100b includes a satellite network 140b that communicatively couples client devices 1 10a, 1 10b and an access network 150b to one another and to the external network 160 (such as the Internet). The satellite communications network 100b includes a scheduler 130b configured to allocate network resources to the client devices 1 10a, 110b. The access network 150b is similar to the gateway 150a described herein with reference to FIG. 1 A and the scheduler 130b is similar to the scheduler 130a described herein with reference to FIG. 1 A. The satellite communications network 100b also includes a QoE-driven network manager 145b configured to provide QoE-driven network automation, similar to the QoE-driven network manager 145a described herein with reference to FIG. 1 A.

[0077] The satellite communications network 100b may utilize various network architectures that include space and ground segments. For example, the space segment may include one or more satellites, while the ground segment may include one or more satellite user terminals, gateway terminals, network operations centers (NOCs), satellite and gateway terminal command centers, and / or the like. Some of these elements are not shown in the figure for clarity. The satellite network 140b can include a geosynchronous earth orbit (GEO) satellite or satellites, a medium earth orbit (MEO) satellite or satellites, and / or a low earth orbit (LEO) satellite or satellites.

[0078] The client devices 1 10a, 1 10b can include a router and can be configured to receive data to be routed over the satellite communications network 100b. The client devices 1 10a, 110b can include any type of consumer premises or mobile equipment (e.g., a telephone, modem, router, computer, set-top box, and the like).

[0079] The client devices 110a, 110b are configured to route data to the satellite network 140b (via respective customer satellite transceivers 120a, 120b). The satellite network 140b includes a forward link for sending information from the access network 150b to the client devices 1 10a, 1 10b, and a return link for sendinginformation from the client devices 1 10a, 110b to the access network 150b. The forward link includes a transmission path from the access network 150b through a gateway satellite transceiver 131 , through a satellite 105 via a satellite uplink channel, to the customer satellite transceivers 120a, 120b via a satellite downlink channel, and to the client devices 110a, 1 10b. The return link includes a transmission path from the customer satellite transceivers 120a, 120b, to the satellite 105 via the satellite uplink channel, to the gateway satellite transceiver 131 via the satellite downlink channel, and to the access network 150b. Each transmission channel may utilize multiple satellites and transceivers.

[0080] Each of the client devices 1 10a, 110b is configured to request return-link grants on the satellite network 140b from the scheduler 130b via the access network 150b. The scheduler 130b determines a return-link allocation schedule and transmits it to each client device 1 10a, 110b via the access network 150b. The scheduler 130b can utilize any suitable scheduling technique such as a demand assigned multiple access (DAMA) scheduling model, an enhanced mobile satellite services (EMSS) scheduling model, and the like. Responsive to receiving a request for bandwidth allocation from the client devices 1 10a, 1 10b, the scheduler 130b analyzes the request, network status, network congestion, prior requests, similar requests, and the like to determine a schedule for return-link bandwidth. Data may be transmitted from a particular client device 1 10a, 1 10b through the satellite 105 to the access network 150b using bandwidth requested by the client devices 1 10a, 1 10b and allocated by the scheduler 130b.

[0081] Based on the allocated resource grants from the scheduler 130b, the client devices 1 10a, 110b transmit data to the access network 150b through the satellite network 140b via the return link. After reaching the access network 150b, the data can then be directed to the external network 160. Data from the external network 160 can be sent to the client devices 1 10a, 110b by the access network 150b via the forward link of the satellite network 140b. The scheduler 130b can allocate network resources on the forward link similar to the return link, as described herein. In some implementations, part or all of the access network 150b and / or the scheduler 130b can be located in a virtual device residing in a public or private computing cloud.

[0082] As discussed herein, the communications system 140a and / or the satellite network 140b (or any component thereof such as the scheduler 130a,130b, the gateway 150a, the access network 150b, or the QoE-driven network manager 145a, 145b) can be configured to passively determine speed measurements for network applications of a service flow. Any of the components of the satellite network 140b can be configured to determine speed measurements or speed metrics. The QoE-driven network manager 145b can then use the determined measurements to manage network resources on the satellite network 140b. For example, the QoE-driven network manager 145b can use determined speed metrics as an input to an objective function, the output of which is used by a network control system to improve or optimize network resource allocation.

[0083] The disclosed systems and methods function in any suitable communications network. For example, the communications network can be provided by satellites, by terrestrial-based equipment, or a combination of satellites and terrestrial networks. Thus, the concepts disclosed herein regarding managing network resources using an objective function with passive measurements as input to the objective function can be applied to service flows or similar logical network data structures provided by any variety of communications networks.

[0084] Similarly, the disclosed techniques can be used to understand the virtual pipe provided by a network service provider to the customer. For example, passive speed measurements and resulting SLO scores can be used by the network service provider to determine whether the virtual pipe being provided achieves the targeted experience for the customer. Passive speed measurements can be understood to be associated with or to represent a virtual pipe size (e.g., network capacity or bandwidth) through a communications network allocated to service flows in the service flow class. These passive measurement techniques do not require any additional probe traffic to be introduced into a communications network. Rather, service flow statistics are extracted from communications network components and / or subsystems and these service flow statistics are used to determine speeds experienced by customers to better understand the customer QoE on the communications network and to identify congestion.

[0085] As used herein, service flows can refer to a layer 2 connection associated with a network application. A service flow class extends the concept of a speed experienced by a customer for a network application by grouping similar service flows together into a service flow class. Although it is possible to measure the speed experience of individual network applications, this may introduceundesirable overhead for communications networks, or components in a communications network such as a scheduler, gateway, or QoE-driven network manager. Thus, the passive measurement technologies used herein can define a service flow class that groups a plurality of similar service flows together. To define a service flow class, service flows can be grouped together based on shared characteristics such as, for example and without limitation, priority, minimum reserved transmission rate (MRTR), maximum sustained transmission rate (MSTR), scheduling weight, etc. QoE-Driven Network Automation

[0086] The systems and methods disclosed herein provide QoE-driven network automation. The disclosed technologies leverage passive measurements of speed, latency, and / or usage (or other fine-grained or high frequency measurements) and QoE measurements to derive QoE-related metrics. These QoE-related metrics are then used to generate SLOs. Fine-grained, high- frequency, and / or passive measurements are then acquired in real time or near real time and those measurements are used to determine compliance with the SLOs (e.g., SLO scores) using an objective function. Based on the output of the objective function, network resource allocations can be adjusted to improve or optimize customer QoE by improving or optimizing the output of the objective function. For example, the disclosed technologies can implement control loops around SLO compliance scoring that are configured to adjust network parameters to improve or optimize SLO scores on the network. This, in turn, advantageously improves or optimizes SLO outcomes which improves customer QoE.

[0087] In some implementations, high frequency network statistics (e.g., QoE-related metrics) are used as input to algorithms that determine QoE speeds for service flows or service flow classes. The algorithms correlate the network statistics to customer QoE, referred to as a QoE assessment. The algorithms can correlate the input network statistics to customer QoE. Based on the targeted customer experience the network service provider is aiming to achieve, network parameters can be adjusted to maintain targeted network statistics, due at least in part to the correlation between the network statistics and the customer QoE. In some embodiments, the algorithms can be implemented using machine learning.

[0088] In some implementations, establishing SLOs that result in an improved or optimal customer experience combines both objective and subjective assessments. During the service planning phase, QoE assessments can be usedto inform how to set SLOs to deliver the desired customer experience. Service planning can use network measurements (e.g., fine-granularity measurements, high-frequency measurements, passive measurements) and QoE modelling that correlates these network measurements with customer QoE to estimate the network resources required to deliver new services in the future that achieve the QoE-based SLOs.

[0089] In some implementations, a control loop can be implemented that is configured to ensure that each subservice sharing the same network resource is meeting a targeted SLO. Based on the SLO and passive speed measurements, the control loop can be configured to ensure that the SLOs are being met by adjusting parameters (e.g., scheduling weights) in a QoE-driven network controller. An input to this control loop can be the output of the objective function used in the service planning.

[0090] In some implementations, the QoE assessment systems can be combined with the service planning systems so that the SLO score determined as part of the QoE assessment procedures can be used as input to the network controller. In this way, the network controller can be configured to adjust network parameters to meet a targeted customer QoE for specific network applications. In some embodiments, the disclosed systems are configured to be fully automated to achieve QoE-driven network automation.

[0091] FIG. 2A illustrates a block diagram of an example network manager 200 configured to automate network management driven by QoE metrics. The network manager 200 is configured to implement the systems and methods disclosed herein related to QoE assessment using a modeling system 210, QoE- based service planning using a service planning system 220, QoE-driven network control using a network optimization system 230, and QoE-driven service monitoring using a service monitoring system 240.

[0092] The network manager 200 includes the modeling system 210 that has a network measurement subsystem 212, a QoE measurement subsystem 214, and a QoE modeling subsystem 216. The modeling system 210 is configured to determine or estimate a QoE assessment based on fine-grained measurements acquired or received by the network measurement subsystem 212. The finegrained measurements can include, for example, passive speed measurements,passive latency measurements, passive usage measurements, measurements of throughput or bytes, and the like.

[0093] The QoE measurement subsystem 214 is configured to receive application layer statistics and to determine or estimate QoE metrics that are related to the quality of experience of the customer. The QoE measurement subsystem 214 can be configured to receive statistics, data, information, and the like from end-user applications or network applications. This information from the applications can be used to understand or estimate customer experience. This is advantageous because application layer statistics (e.g., frame rate, resolution, etc.) are not measurable at the network layer by the network measurement subsystem 212. The QoE modeling subsystem 216 implements one or more algorithms that correlate measurements of network statistics (e.g., passive speed measurements) from the network measurement subsystem 212 and the measurements of end-user applications from the QoE measurement subsystem 214 with customer QoE. By way of example, the QoE modeling subsystem 216 can use the fine-grained measurements from the network measurement subsystem 212 and the resolution provided by a video application as determined by the QoE measurement subsystem 214 to correlate determined network speed with the resolution a customer is actually seeing on their device while watching streaming media content. The resolution can be correlated to customer QoE because a higher resolution while watching media content typically correlates with a better customer experience.

[0094] The QoE modeling subsystem 216 then uses one or more algorithms (e.g., machine learning algorithms) to generate QoE assessments based on the fine-grained measurements. For example, the QoE modeling subsystem 216 can use the viewing resolution to generate an estimated customer QoE. For example, machine learning can be implemented in the QoE modeling subsystem 216 that correlates passive speed measurements acquired by the network measurement subsystem 212 to video bitrates based on the information from the QoE measurement subsystem 214. Knowing the video bitrate, the QoE modeling subsystem 216 can infer or otherwise determine an estimate of the video resolution. The QoE modeling subsystem 216 can use a probabilistic method to assess the likelihood of achieving a given resolution based on the video bitrate. Although there are many factors that go into determining customer QoE, themodeling system 210 is particularly effective in measuring customer QoE during times where the network is the limiting factor in customer QoE.

[0095] In some embodiments, the modeling system 210 operates over a period of time, such as a number of days, weeks, or months, to determine the output QoE assessments. Because the modeling system 210 has access to both network layer statistics (via the network measurement subsystem 212) and to application layer statistics (via the QoE measurement subsystem 214), the QoE modeling subsystem 216 can generate QoE assessments that correlate measurable network metrics with customer QoE. Thus, the modeling system 210 provides a correlating process to help the network manager 200 ensure that targets from application layer measurements are being met. In other words, the modeling system 210 is configured to correlate application layer measurements to network layer measurements to generate target performance metrics that increase the probability that customer QoE targets are being met. For example, the QoE assessment provided by the modeling system 210 can indicate the network speed that should be provided to achieve a target application QoE (e.g., a target network speed to support a target video resolution and frame rate in a video application). This QoE assessment can thus be used to generate SLOs (e.g., in the service planning system 220).

[0096] By way of example, a machine learning model can be built and implemented in the QoE modeling subsystem 216 that relates network measurements to video quality based on measurements taken over a month. This relationship is not expected to change significantly over time, unless there is a rapid adoption of a new video codec, for example. However, if a new codec came out and was rapidly adopted, the modeling system 210 can be configured to retrain the models in the QoE modeling subsystem 216. Thus, triggers can initiate a process of retraining models in the QoE modeling subsystem 216. Examples of triggers include, without limitation, new technologies, new usage profiles, new users that use the network differently, etc. The models (e.g., machine learning algorithms) in the QoE modeling subsystem 216 are configured to translate measurements (network and application measurements) into customer QoE to enable the setting of measurable objectives or targets. In some implementations, this can be done for individual service flows and / or for service flow classes.

[0097] The network manager 200 includes the service planning system 220 that has an SLO generation subsystem 222, a network measurement subsystem 223, an SLO scoring subsystem 224, and a capacity planning subsystem 226. The service planning system 220 is configured to generate SLOs with the SLO generation subsystem 222 based on the QoE assessment from the modeling system 210. The SLO generation subsystem 222 can use the QoE assessments to generate one or more SLOs that focus on improving or increasing customer QoE because the modeling system 210 provides performance objectives that are correlated to customer QoE. By way of example, it may be desirable to provide a customer a video experience with a resolution of at least 480p almost all the time during a peak busy period. This is an application-layer objective provided by the modeling system 210. The SLO generation subsystem 222 can be configured to turn this objective into a measurement that can be acquired (e.g., based on the output of the QoE modeling subsystem 216). For example, the SLO generation subsystem 222 can determine that to achieve the application-layer objective, the network should deliver a bit rate of about 2 Mbps to the video player 90% of the time. From that, the SLO generation subsystem 222 can determine that the network needs a channel capacity of about 3 Mbps 90% of the time, as measured with passive speeds when the network latency is on the order of 600 ms. The SLO generation subsystem 222 then recasts the application-layer objective in terms of the measurable quantity (e.g., passive speed measurements).

[0098] The service planning system 220 is configured to score SLO compliance with the SLO scoring subsystem 224 based on network measurements acquired by the network measurement subsystem 223. The network measurement subsystem 223 can be configured to acquire high-frequency, fine-granularity, and / or passive measurements of network layer metrics. These measurements can be configured to be acquired in real time or near real time to provide real time feedback for the SLO scoring subsystem 224. The SLO scoring subsystem 224 implements an objective function to determine the SLO score using the network measurements as input. In some embodiments, the time frame in which the SLO scoring subsystem 224 operates can be on the order of milliseconds, seconds, minutes, or hours.

[0099] The SLO scoring subsystem 224 can output an SLO score for individual service flows and / or service flow classes. The SLO score is used in thenetwork optimization system 230 and / or the service monitoring system 240 as input to these respective systems. The SLO score is configured to reflect compliance with one or more SLOs generated by the SLO generation subsystem 222. In some embodiments, the SLO score is the ratio of the measured speed divided by the SLO target speed. Hence, a score of at least 1 means the SLO was achieved and a score of less than 1 means that the SLO was not achieved. Many SLO scores can be determined at each point in time. Because of this, the SLO scoring subsystem 224 can be configured to aggregate these scores. Aggregating the scores can be done in any number of ways and multiple ways may be used concurrently. For example, aggregating the scores can use a weighted average where the importance of each score is weighted by the number of customers or economic value, or some combination of both. Aggregating the scores can use a simple average. Aggregating the scores can use a simple or weighted averages across a particular span of time (e.g., one hour). Aggregating the scores can be done by aggregating scores across the same period day over day. The resulting score can be used in network control automation. For example, the service planning system 220 can take network resources from an entity with a score of 1 .1 and provide them to an entity with a score of 0.9 to attempt to achieve a score of 1 for both entities (e.g., customers, groups of customers, carriers, beams, etc.). In such an example, the service planning system 220 or the network optimization system 230 can attempt to balance a large number of scores (e.g., 30 or more scores) to try to get the scores to be exactly 1 . This can be done by adjusting weights among the entities to get each as close to 1 as possible. As an example, this can be done every 5 s to 10 s. This reflects the overarching principle that superior performance and satisfaction can be achieved when the network allocates resources so that each entity’s score is as close to 1 as possible and as even among the entities as possible.

[0100] The service planning system 220 is configured to plan or allocate capacity with the capacity planning subsystem 226. The capacity planning subsystem 226 can be configured to allocate capacity to improve SLO scores. This can be done by improving QoE assessments which in turn involves adjusting network parameters to improve the network statistics derived from the fine-grained measurements. For example, the capacity planning subsystem 226 can adjust network parameters so that different customers have similar experiences. Oncenetwork parameters are adjusted to equalize customer experience, any additional capacity can be allocated based on deviation from quality targets indicated by the SLO scores determined by the SLO scoring subsystem 224. In other words, the capacity planning subsystem 226 is configured to manage network capacity based on customer QoE, which is correlated with the SLO scores determined by the SLO scoring subsystem 224 which in turn is dependent on the SLOs generated by the SLO generation subsystem 222, each of which depends on the QoE assessments generated by the modeling system 210. Thus, the service planning system 220 is configured to provide QoE-based service planning. In some embodiments, static resource allocation management that cannot be performed by the global resource traffic manager 236 is performed by capacity planning subsystem 226. For example, a certain satellite system cannot change the satellite schedule in real or near-real time. In such cases, capacity planning to allocate capacity to each beam / carrier makes sense.

[0101] In some embodiments, capacity planning involves determining the amount and location of the allocation of network resources. Typical networks allocate resources based on demand models, where the amount of usage is used to assess the resources that are needed. However, that is a flawed approach. For example, in cases where the network is highly congested and QoE is not meeting customer expectations, the usage is underestimating demand. Conversely, when the network is uncongested, the system may be delivering more bandwidth than is needed to meet customer expectations (e.g., where customer expectations are captured in the SLOs). This may be particularly true for video delivery. For example, for a given service offering the system may seek to deliver a HD video experience (e.g., 720p) as opposed to a UHD video experience (e.g., 4k). If the demand model is based on usage, then more resources will be allocated than are needed to achieve the SLOs. Accordingly, the capacity planning subsystem 226 is configured to attempt to deliver the capacity needed to achieve each SLO in the network at all times. Using machine learning and other models, the capacity planning subsystem 226 can be configured to estimate the QoE (e.g., based on input passive speed targets) that results from a given carrier loading and carrier size, where carrier loading is a function of the combined flow weight of all the traffic flows and where flow weight is the sum of each customer’s flow usage times the scheduler weight normalized by total usage. Thus, the capacity planningsubsystem 226 is configured to model how much capacity is needed for a given customer cohort mix to achieve the associated customer QoE targets.

[0102] The network manager 200 includes the network optimization system 230 that has a shared network traffic manager 232, a network segment traffic manager 234, and a global resource traffic manager 236. Each of the shared network traffic manager 232, the network segment traffic manager 234, and the global resource traffic manager 236 can receive the output of the objective function, such as the SLO score, from the SLO scoring subsystem 224. In addition, the shared network traffic manager 232 can be configured to aggregate SLO scores for shared network portions (e.g., a carrier SLO score for a carrier in a beam) and the network segment traffic manager 234 can be configured to aggregate SLO scores for network segments (e.g., a beam SLO score for a beam in a satellite network). The shared network traffic manager 232, the network segment traffic manager 234, and the global resource traffic manager 236 can operate on different time scales, on similar time scales, or on the same time scale. In some embodiments, the network optimization system 230 is configured to improve or optimize the shared portions of the network, the network segments, and the global network resources independently or at different times. In some embodiments, the network optimization system 230 is configured to improve or optimize the shared portions of the network, the network segments, and the global network resources simultaneously.

[0103] In some embodiments, the shared network traffic manager 232 manages scheduling weights for the scheduler. In some embodiments, the shared network traffic manager 232 operates as a layer 2 scheduler, e.g., a media access controller (MAC) scheduler. The shared network traffic manager 232 can be configured to allocate weights to achieve a weighted fair share among customers, groups of customers, or carriers. In some embodiments, the shared network traffic manager 232 is configured to generate SLO scores by aggregating customer SLO scores for each carrier in the system, examples of which are provided herein.

[0104] In some embodiments, the network segment traffic manager 234 is configured to distribute customer terminals across different carriers in a beam, where individual beams include a plurality of carriers. The network segment traffic manager 234 is also configured to generate a beam or access network SLO scoreby aggregating carrier SLO scores provided by the shared network traffic manager 232.

[0105] In some embodiments, the global resource traffic manager 236 is configured to adjust network parameters and to schedule network resources to improve or optimize SLO scores. In a satellite system, the global resource traffic manager 236 can be configured to adjust the beam pattern projected onto the ground based at least in part on the beam SLO scores determined by the network segment traffic manager 234.

[0106] For each traffic service primitive, where a traffic service primitive is a combination of a set of applications and an SLO, a score for an individual carrier can be determined using the aggregating techniques described herein. The carrier score determined by the shared network traffic manager 232, for example, can be a weighted average based on each customer associated with the traffic service primitive. The network segment traffic manager 234 can then be configured to move a customer from one carrier to another carrier so that carrier scores are balanced. Then, the global resource traffic manager 236 can be configured to balance beam scores. In this way, the network can be managed in a distributed way. The network is not managed solely based on SLOs, rather there is an objective function that operates globally across the network that allows the network optimization system 230 to approximately achieve global optimization across the network. This can be achieved due at least in part to using the same objective function to adjust network resources.

[0107] Because the SLOs are determined based on customer QoE in the service planning system 220, the network optimization system 230 is configured to schedule network resources to ultimately improve customer QoE. This may result in different network resource allocations than would be the case if the network manager were configured to schedule network resources based on optimal utilization of network resources, as described in greater detail herein. The network manager 200 uses high-frequency measurements to improve or optimize SLO performance, and thereby customer QoE, across a communications network in an automated fashion. In this way, the network manager 200 is configured to control the communications network around customer QoE. Without estimating customer QoE, the network manager 200 cannot accomplish QoE-driven network management. Typical network managers perform SLO-driven networkmanagement. However, because the SLOs described herein are generated based at least in part on customer QoE, the network manager 200 is configured to perform QoE-driven network management. The network manager 200 is configured to reallocate resources based on customer QoE because network capacity is constrained in the communications network. The network manager 200 is configured to move capacity around to improve or optimize customer QoE.

[0108] In some embodiments, the network optimization system 230 can be implemented in a terrestrial network and / or a cellular network. In such embodiments, different cell towers can provide a plurality of frequency bands (e.g., 4 or 5 bands) for communicating with mobile terminals (e.g., phones). The cell tower can use the shared network traffic manager 232 to manage the share of resources to allocate to mobile terminals for individual carriers. In addition, individual cell towers can perform hard hand offs to move a mobile terminal from one carrier (or frequency band) to another carrier, thereby acting as the network segment traffic manager 234. Furthermore, cell towers in a communications network can be positioned to cover adjacent areas. Based on usage distribution, the communications network can increase or decrease power used by a particular cell tower which results in decreasing or increasing power in an adjacent cell tower to avoid interference, performing a similar function to the global resource traffic manager 236. Thus, in such embodiments, the network optimization system 230 can be used to globally improve or optimize service in a network of cell towers and / or ground stations.

[0109] The network manager 200 includes a service monitoring system 240 that has an anomaly detector 242, an event correlator 244, and an alerts subsystem 246. The anomaly detector 242 is configured to receive SLO scores from the service planning system 220 and to determine one or more QoE events that represent anomalies in network performance. A QoE event is a drop in the QoE score relative to what is normal for that area. The anomaly detector 242 is different from typical network monitoring and alerting systems that are based on detecting outages. Instead, the QoE events from the anomaly detector 242 indicate when the customer experiences degradations in performances, not just outages.

[0110] The event correlator 244 is configured to identify correlations between the QoE events determined by the anomaly detector 242. For example, when there is an impairment in the network, the anomaly detector 242 will generatenumerous QoE events. The event correlator 244 is configured to use this set of QoE events to localize the root cause of the impairment. In response to determined correlated events, the alerts subsystem 246 is configured to generate alerts to identify adverse events affecting customer QoE. Thus, the service monitoring system 240 is configured to respond to situations in which there is degraded performance, not just outages. The service monitoring system 240 identifies when network quality falls outside of an acceptable range, creating an alert via a correlation engine, to identify a root of the problem causing the reduction in network quality.

[0111] In some embodiments, the network manager 200 or portions of the network manager 200 (e.g., the modeling system 210, the service planning system 220, the network optimization system 230, and / or the service monitoring system 240) can be implemented on a router, a base station, a gateway terminal, a scheduler, or other such component or combination of components in a communications network.

[0112] FIG. 2B illustrates a block diagram of an example system 250 for network optimization based on QoE metrics. In some implementations, the system 250 can be similar to the network optimization system 230 described herein with reference to FIG. 2A. For example, the system 250 can implement control loops to accomplish network improvement or optimization similar to the network optimization system 230. Control loops implemented by the system 250 can be similar, for example, to the control provided by the shared network traffic manager 232, the network segment traffic manager 234, and / or the global resource traffic manager 236, as described herein. The system 250 can be configured to improve or optimize network resource allocation based on SLOs or based on SLOs combined with metrics associated with business value for the network service provider, expressed or represented using an objective function as described herein. The system 250 can achieve these goals using nested loops of controllers, multi-variable feedback controllers, reinforcement learning mechanisms, and the like. The system 250 is configured to allocate network resources based on an objective function, examples of which have been provided herein.

[0113] The system 250 can be configured to improve or optimize network resource allocation using a multi-dimensional approach or a parallel or simultaneous optimization approach. For example, although the networkoptimization system 230 of FIG. 2A is illustrated as a sequential process, it is to be understood that two or more of the shared network traffic manager 232, the network segment traffic manager 234, and the global resource traffic manager 236 can be performed in parallel. Thus, the system 250 can control network resource allocation using different optimization approaches that can be performed in parallel. In some implementations, the system 250 is configured to attempt to globally optimize network resource allocation and usage using a multi-dimensional approach and / or using control loops that may or may not include nested control loops. Where nested control loops are implemented, individual control loops can have similar or different time granularity or time scales. In some embodiments, scheduling can include local optimization techniques within a global optimization scheme. That is, the local and global optimization techniques can use the same utility function due at least in part to the lack of a conflict of interest in terms of optimization goals at the local and global levels.

[0114] The system 250 includes a plurality of service areas (or beams) 260. Within individual service areas 260 there is a plurality of carriers 270. Individual carriers 270 can provide network services to a plurality of user terminals (UTs) and / or mobile terminals (MTs). By way of example, network services provided to the UTs and MTs can be grouped into service flows (SFs) and service flows can be grouped into service flow classes (SFCs). A service flow (SF) can be a layer 2 connection that provides transport of packets, such as on an uplink or on a downlink. A service flow can be characterized by a set of quality-of-service (QoS) parameters including, for example and without limitation, priority, minimum reserved transmission rate (MRTR), maximum sustained transmission rate (MSTR), parameters specifying requested resources (e.g., bandwidth, latency, etc.), parameters specifying a level of service to be provided, and the like. A service flow can be identified by a description of the service flow type (e.g., web service, video, etc.) and other identifiers. Service flows allow network service providers to offer different services and to segregate traffic flows having different QoS requirements. A service flow class (SFC) includes a plurality of service flows that are logically grouped together because of certain shared characteristics. For example, service flows with the same QoS configuration can be logically grouped together into a service flow class. A QoS configuration can include, for example and without limitation, a scheduling priority, a minimum reserved transmission rate(MRTR), a maximum sustained transmission rate (MSTR), and / or the like. As another example, service flows with the same QoS configuration and scheduling weight can be logically grouped together into a service flow class. Scheduling weight can be used to weight the allocation of network resources for service flows in different service flow classes. For example, a service flow with a higher scheduling weight will typically be assigned more network resources, if required, than a service flow with a lower scheduling weight. Scheduling priority can be used to allocate network resources to a higher priority service flow before allocating network resources to a lower priority service flow. In some implementations, a network application (e.g., a layer 3 or 4 application) is associated with a service flow (a layer 2 connection).

[0115] The system 250 includes a capacity allocation engine 252 that is configured to accept inputs such as usage per service area and SLO score per service area and to output allocated RF resources to the plurality of carriers 270. In some implementations, the capacity allocation engine 252 can be referred to as a satellite beam carrier capacity allocation engine due at least in part to the capacity allocation engine 252 allocating capacity per service area 260 (or per satellite beam) and per carrier 270 within individual service areas (or satellite beams). The capacity allocation engine 252 is configured to globally manage network resource allocation based on usage metrics and SLO score metrics acquired and aggregated by the plurality of carriers 270 and the plurality of service areas 260. These usage metrics and SLO score metrics are measured and aggregated at the carrier level in the plurality of carriers 270 and at the level of the service areas in the plurality of service areas 260, and then parameters are adjusted to improve or optimize network resource allocation based on these metrics, as described herein. That is, the capacity allocation engine 252 allocates RF resources for the plurality of service areas 260 and the plurality of carriers 270, which is input to a carrier capacity module 271 associated with each carrier 270. Systems within each carrier 270 improve or optimize network resource allocation within the bounds dictated by the carrier capacity module 271 . Further improvement or optimization is performed within each service area of the plurality of service areas 260. This serves to globally improve or optimize network resource allocation based on QoE metrics by performing these optimizations at the level of the carriers 270 and the service areas260 within the bounds of the carrier capacity dictated by the capacity allocation engine 252 and using SLO-based objective functions.

[0116] Within individual service areas 260 (or satellite beams), the system 250 includes a usage aggregator 262, a service area load balancer 264, and an SLO score aggregator 266. The usage aggregator 262 is configured to receive usage statistics per terminal from each carrier 270 and to aggregate these usage statistics to send to the capacity allocation engine 252 and to the service area load balancer 264. Similarly, the SLO score aggregator 266 is configured to receive SLO scores from each carrier 270 and to aggregate these SLO scores to send to the capacity allocation engine 252 and to the service area load balancer 264. The service area load balancer 264 is configured to adjust the mapping of user terminals to carriers within a service area based at least in part on the aggregated usage metrics from the usage aggregator 262 and the SLO score metrics from the SLO score aggregator 266.

[0117] Within individual carriers, the system 250 includes a scheduler 272, an SLO score aggregator 273, a usage controller 276, and an SFC SLO score balancer 277. The scheduler 272 can be similar to the scheduler 130a and / or the scheduler 130b, described herein with reference to FIGS. 1 A and 1 B. For example, the scheduler 272 can be configured to allocate network resources to service flows to and from a plurality of user terminals and / or mobile terminals. The scheduler 272 is configured to interface with a carrier capacity module 271 that determines the carrier capacity within the associated carrier based on the network resource allocation from the capacity allocation engine 252. The scheduler 272 receives from the service area load balancer 264 a demand per service flow for user terminals and / or mobile terminals served by the carrier 270. In addition, as described herein, the scheduler 272 receives an adjusted weight per service flow using a control loop that includes the usage controller 276 and the SFC SLO score balancer 277. Using these inputs, the scheduler 272 allocates network resources to improve or optimize network resource allocation within the carrier 270 based on SLOs, as represented by the objective function.

[0118] The objective function serves to express an SLO in terms of network performance that can be measured, and the SLO in turn is reflective of customer QoE. The system 250 is configured to utilize the SLOs (or primitives) that together form a product and the system 250 uses one or more objective functionsto assess compliance with the SLOs (e.g., via an SLO score). The system 250 then manages network resources to improve or optimize the objective function(s) (e.g., globally improving or optimizing SLO scores).

[0119] The scheduler 272 is configured to send usage statistics based on user terminal and mobile terminal network usage to the usage aggregator 262 of the associated service area 260. The SLO score aggregator 273 is configured to receive SLO score metrics that are based on SLO measurements for service flows or service flow classes provided by the scheduler 272. The SLO score aggregator 273 aggregates these SLO scores and sends the SLO score metrics to the SLO score aggregator 266 of the associated service area 260. The SFC SLO score balancer 277 also receives the SLO score metrics per service flow class, which the SFC SLO score balancer 277 uses to determine base weights per service flow class to help to balance network resource allocation based on SLO scores. The base weights provided by the SFC SLO score balancer 277 are adjusted by a function 275 (e.g., a multiplier or other function or mapping) that is determined by the usage controller 276. The usage controller 276 receives from the scheduler 272 an input corresponding to a low pass filter (LPF) of peak busy hour (PBH) usage per terminal in the carrier 270. Based on the provisioning rate per plan, the usage controller 276 determines a multiplier per user terminal and / or mobile terminal to generate the adjusted weight per service flow that is used as input at the scheduler 272.

[0120] To determine the SLO score per service flow class, the scheduler 272 provides SLO measurements per service flow class to a function 274 (e.g., a division or other function or mapping) that compares the SLO measurements to an SLO target to determine an SLO score per service flow class. The SLO score per service flow class is provided to the SLO score aggregator 273 and to the SFC SLO score balancer 277. The SLO target can be a plurality of target SLO measurements, with an SLO target for each service flow class. Thus, the function 274 compares the SLO measurement per SFC provided by the scheduler 272 with an SLO target for the corresponding SFC.

[0121] In some implementations, control loops of the system 250 can be performed on a variety of time scales to improve or optimize network resource utilization. In certain implementations, different control loops can be performed on similar time scales. In such implementations, the system 250 is configured so thatthe different control loops do not collide with one another. In various implementations, different control loops are performed on different time scales. Similarly, in such implementations, the system 250 is configured so that the different control loops do not collide with one another. In some embodiments, the system 250 operates on a time scale that does not allow the use of nested control loops or control loops in general. In such embodiments, the system 250 uses local and / or global optimization techniques to allocate network resources. Examples of such optimization techniques include, for example and without limitation, feedback control using artificial intelligence systems (e.g., machine learning, neural networks, etc.).

[0122] The system 250 can be configured to perform load balancing as part of its allocation of network resources. In some implementations, network resource optimization can be done using feedback control over different time scales. In various implementations, the system 250 is configured to analyze the state of the network, analyzing a plurality of variables, and to determine adjustments to the network to improve or optimize network resource allocation. This can be done locally and globally nearly simultaneously. In such implementations, for example, control loops can be replaced or augmented with reinforcement learning techniques that utilize multiple variables. Such implementations may be particularly useful where it is possible to switch between carriers relatively rapidly. In networks where rapid carrier switching is implemented, the time scales for control loops shrink as well. As this time scale shrinks, it becomes advantageous to perform local and global optimization without the use of nested control loops but to rather use multi-variable optimization techniques. Similarly, such implementations may be particularly useful where handover times between beams (e.g., between service areas) shrink. This may be the case in networks that use phased arrays for satellite beams, for example. Again, where control loop time scales shrink, it becomes advantageous to perform local and global optimization without the use of nested control loops but to rather use multivariable optimization techniques.

[0123] Even in such multi-variable optimization techniques, the system 250 utilizes SLO scores and associated objective functions to determine improved or optimal network resource allocations. The SLO scores provide target performance metrics for service flows and service flow classes for user terminalsand mobile terminals. In some implementations, the SLO scores incorporate economic value to the network service provider in addition to the QoE of the customer. In some implementations, the objective function uses the SLO score as a primary objective function and economic value to the network service provider as a secondary objective function. In such implementations, the system 250 can optimize to the primary objective function and then to the secondary objective function. In various implementations, the system 250 uses multi-objective optimization techniques to achieve a plurality of objectives that may include SLO score, economic value to the network service provider, and other such objectives. Thus, the system 250 can be configured to achieve the SLOs where possible or feasible and then to consider economic value to the network service provider in instances where it is not possible or feasible to achieve one or more SLOs. This allows the system 250 to drive the customers’ network experience based on QoE metrics while also still considering economic value to the network service provider.

[0124] Where control loops are implemented in the system 250, which may include nested control loops, a variety of control loops can be used. For example, the system 250 can be configured to perform a usage control loop. The usage control loop may be particularly beneficial for fixed customers, or customers whose location remains relatively fixed (e.g., as opposed to mobile terminals whose location changes more frequently). The usage control loop uses the usage controller 276 to determine a UT multiplier for individual user terminals. The UT multiplier can be a value greater than 0 and less than or equal to 1 , for example. The UT multiplier is applied to the base weight of the service flow (e.g., as provided by the SFC SLO score balancer 277). The base weight is an output of the SFC SLO score balancer 277 that is configured to balance SLO scores for service flow classes. The UT multiplier can be configured to bring peak busy hour usage of each user terminal to a targeted provisioning value as the carrier gets full. Thus, the scheduler 272 provides to the usage controller 276 data that is a low pass filter of peak busy hour usage for each user terminal. In some implementations, the usage control loop interval is on the order of a day or multiple days. In various implementations, the usage control loop has targeted convergence within about a month. The usage control loop may be particularly advantageous in managing heavy data users. For such users, it is desirable to reduce the scheduling weight in times of congestion so that such users get lower speeds and their usage islimited during congested times. This has the beneficial effect of filtering out conformant users to protect them from non-conformant users (e.g., heavy data users).

[0125] As another example control loop, the system 250 can be configured to perform a service flow SLO balancing control loop. The service flow SLO balancing control loop can be performed in conjunction with other control loops described herein to optimize the application of network capacity. The service flow SLO balancing control loop is performed within a carrier 270 using scheduling weight control. The SFC SLO score balancer 277 determines a base service flow weight to balance SLO scores of all service flows that belong to customers, assuming a UT multiple of 1. In some implementations, the time frame of the service flow SLO balancing control loop is on the order of one or more days with targeted convergence on the order of one to two weeks. Thus, the service flow SLO balancing control loop includes the SFC SLO score balancer 277 receiving the SLO score per service flow class from the function 274 (which divides the SLO measurement per SFC from the scheduler 272 with a SLO target) and then determines the base service flow weight to balance SLO scores.

[0126] As another example control loop, the system 250 can be configured to perform a capacity allocation control loop. The system 250 dynamically allocates capacity within individual service areas or beams 260 to match demand with the SLO scores of each carrier 270. In some implementations, the system 250 allocates capacity so that the aggregated SLO score per carrier is greater than or equal to 1 in instances where an SLO score of 1 indicates that the SLO targets are being met. The capacity allocation control loop can aggregate SLO scores in the SLO score aggregator 266 and send the aggregated scores to the capacity allocation engine 252. Similarly, the usage aggregator 262 can send aggregated usage metrics to the capacity allocation engine 252. The capacity allocation engine 252 can adjust the carrier capacity in the carrier capacity module 271 to balance SLO scores of service areas. In some implementations, the time frame of the capacity allocation control loop is on the order of 30 minutes to an hour.

[0127] As another example control loop, the system 250 can be configured to perform a service area load balancing control loop. The system 250 is configured to balance user terminal SLO scores across carriers 270 of eachservice area 260. This may be particularly advantageous for mobile terminal handover events. As a mobile terminal enters a new service area, the system 250 can find the best carrier for the mobile terminal in its new service area. This may also be advantageous to perform for every control epoch where the system 250 can adjust the distribution of user terminals and / or mobile terminals in each carrier 270 of individual service areas 260 to balance the load in the corresponding service area 260. The system 250 can use the service area load balancer 264 to balance SLO scores across carriers. In this way, the service area load balancer 264 is configured to update mapping of terminals to carriers using the service area load balancing control loop. That is, the usage of each carrier and the SLO scores of each carrier can be used to move terminals from one carrier to another within a service area to balance loads across the carriers. In some implementations, the time frame of the service area load balancing control loop is on the order of 1 to 3 minutes. In some implementations, the service area load balancing control loop is triggered by mobile terminal handover events.

[0128] As another example control loop, the system 250 can be configured to perform local SLO control loop for certain mobile terminals, such as mobile terminals with premium service contracts. Within a service area 260, the local SLO control loop can be configured to ensure that the SLO of the associated mobile terminal is met for each carrier 270. To ensure the SLO is met, the system 250 is configured to adjust the scheduling weight of the associated mobile terminal at a relatively rapid time scale. In some implementations, the time frame of the local SLO control loop is on the order of 10 seconds.

[0129] As described herein, the control loops can be performed on similar or different time scales. The functions of the described control loops can be performed outside of a control loop structure, that is, the functionality of the disclosed control loops can be incorporated into local and global optimization techniques, such as those described in greater detail herein. Thus, one or more of the usage control loop, the service flow SLO balancing control loop, the capacity allocation control loop, the service area load balancing control loop, and the local SLO control loop can be incorporated into a local and / or global optimization technique rather than being operated as a traditional control loop.

[0130] FIG. 2C illustrates that the system 250 can be also used in a network that includes a plurality of satellites 280. Individual satellites of the pluralityof satellites 280 can be similar to the satellite 105 described herein with reference to FIG. 1 B. That is, a communications system serviced by the system 250 can include one or more satellites and / or other communications devices or systems, similar to the communications system 140a of FIG. 1 A or the satellite network 140b of FIG. 1 B.

[0131] The system 250 can perform the disclosed control loops and / or local and global optimization techniques in communications systems that include a plurality of satellites 280 that each include a plurality of service areas 260 or beams that in turn each include a plurality of carriers 270. In such implementations, the capacity allocation engine 252 is configured to allocate RF resources for satellites of the plurality of satellites 280, service areas (or beams) of the plurality of service areas 260, and carriers of the plurality of carriers 270. The resulting carrier capacity for each carrier can be used to improve or optimize network resource allocation across the plurality of carriers 270, the plurality of service areas 260, and the plurality of satellites 280. The capacity allocation engine 252 is configured to receive usage metrics across the plurality of satellites 280 and SLO scores across the plurality of satellites 280 to dynamically allocate resources to the plurality of satellites 280, the plurality of service areas 260, and the plurality of carriers 270.

[0132] By way of example, mobile terminals (e.g., terminals on an airplane) in one satellite can frequently be alternatively service via an overlapping satellite. In such instances, load balancing decisions can be made by the capacity allocation engine 252 to balance loads across satellites. As another example, the capacity allocation engine 252 can manage network resource allocation among the plurality of satellites 280 where individual satellites may move their beams and / or change beam characteristics to dynamically place supply where customer demand is located.Example Methods for Managing Network Resources Using QoE

[0133] FIG. 3 illustrates a flow chart of an example method 300 for modeling customer QoE in a communications network. The method 300 can be performed in any of the schedulers or other components of a communications network described herein with reference to FIGS. 1 A, 1 B, and 7. For ease of description, the method 300 will be described as being performed by a QoEmodeling system, such as the modeling system 210 described herein with reference to FIG. 2A. This is not to be understood to limit the scope of the disclosure. Rather, any step or portion of the method 300 can be performed by any component or combination of components of the communications networks described herein.

[0134] In block 305, the QoE modeling system receives or determines fine-grained measurements of network statistics. The fine-grained measurements can include passive measurements such as passive speed measurements, passive latency measurements, passive usage measurements, and the like. The fine-grained measurements can be acquired at a rate that is faster than network applications in the communications network make decisions. For example, the rate of the fine-grained measurements can be at least 5 times the decision rate of the network application, at least 10 times the decision rate of the network application, or at least 20 times the decision rate of the network application, and so forth.

[0135] In block 310, the QoE modeling system receives or determines application-layer measurements (e.g., QoE measurements). The application-layer measurements comprise measurements, statistics, metrics, and the like from network applications. The application-layer measurements can include, for example and without limitation, bitrates, download speeds, video resolution, etc.

[0136] In block 315, the QoE modeling system uses QoE models to generate customer QoE objectives. For example, the QoE modeling system is configured to analyze the fine-grained measurements to correlate those measurements to customer QoE based on the application-layer measurements. For example, the QoE modeling system can implement one or more algorithms that use the fine-grained measurements as input and generates a likely customer QoE based on the correlation of the fine-grained measurements with the application-layer measurements. One or more algorithms can be implemented that use machine learning or other similar methods. The output of the one or more algorithms can represent an estimate of customer QoE in the communications network based on the received or determined fine-grained measurements and the received or determined application-layer measurements.

[0137] The resulting QoE assessment can represent an estimate of the customer QoE based on models that correlate customer QoE to network measurements (e.g., passive speed measurements), examples of which are- M -described herein. Based on the estimated customer QoE, the QoE assessment can be used to inform other systems and / or to help in generating SLOs.

[0138] FIG. 4 illustrates a flow chart of an example method 400 for service planning in a communications network based on customer QoE. The method 400 can be performed in any of the schedulers or other components of a communications network described herein with reference to FIGS. 1A, 1 B, and 7. For ease of description, the method 400 will be described as being performed by a network manager, such as the QoE-driven network manager 145a, 145b described herein with reference to FIGS. 1A, 1 B, respectively. This is not to be understood to limit the scope of the disclosure. Rather, any step or portion of the method 400 can be performed by any component or combination of components of the communications networks described herein.

[0139] In block 405, the network manager sets one or more SLOs. The one or more SLOs are set based at least in part on QoE assessments determined by the communications network (e.g., using the method 300 described herein with reference to FIG. 3). The network manager can also be configured to analyze and adjust existing SLOs based on customer QoE. This can be done relatively frequently due at least in part to the high-frequency or fine-grained measurements acquired by the communications network. In some embodiments, the SLOs can be adjusted to capture the desired customer experience more accurately or to accommodate evolution of customer expectations to provide a different user experience.

[0140] In some embodiments, high-frequency measurements allow establishment of statistically sound SLOs for small SLO compliance assessment intervals, since there are many samples for a short time duration. For example, the SLO for a service flow class could be to achieve a 90th-percentile speed of 3 Mbps for each 10-minute period, where passive speed measurement generates a speed experience sample every 5 seconds. With 120 passive speed measurement samples generated every 10 minutes, it is possible to evaluate whether the system met the 90thpercentile SLO. Short SLO compliance assessment intervals may allow QoE-driven network managers (e.g., QoE-driven network managers 145a, 145b of FIGS. 1 A and 1 B) to implement near-real-time feedback control loops.

[0141] In block 410, the network manager determines SLO scores based on compliance with the SLOs set in block 405. The SLO scores can also be basedon the QoE assessments determined by the communications network (e.g., using the method 300 described herein with reference to FIG. 3). In some embodiments, the SLO scores are determined for individual service flows or for service flow classes. In some embodiments, the SLO scores are determined for different customers. The determined SLO scores are configured to assess performance of the communications network in meeting the SLOs. The rate at which SLO compliance can be determined is based on the high-frequency measurements acquired by the network manager (e.g., as described in the method 300). Thus, the rate at which SLO scores are generated is correlated with the rate of measurements acquired by the network manager, which is higher than typical communications networks as described herein.

[0142] In block 415, the network manager allocates network capacity in the communications network based at least in part on the SLO scores determined in block 410. By identifying deviation from targeted performance, based on the SLO scores, the network manager is able to adjust network parameters to achieve targeted performance with respect to SLO scores, which results in improvements in customer QoE. In some embodiments, the network manager is configured to adjust network parameters so different customers have similar experiences. After equalization, the network manager is configured to determine how far away from quality targets the network is and to allocate capacity to account for that difference. Thus, the network manager is configured to manage network capacity based at least in part on customer QoE.

[0143] FIG. 5 illustrates a flow chart of an example method 500 for controlling a communications network based on customer QoE. The method 500 can be performed in any of the schedulers or other components of a communications network described herein with reference to FIGS. 1A, 1 B, and 7. For ease of description, the method 500 will be described as being performed by a network manager, such as the QoE-driven network manager 145a, 145b described herein with reference to FIGS. 1A, 1 B, respectively. This is not to be understood to limit the scope of the disclosure. Rather, any step or portion of the method 500 can be performed by any component or combination of components of the communications networks described herein.

[0144] In block 505, the network manager manages shared network traffic based on the output of an objective function. The network manager isconfigured to receive SLO scores (e.g., using the method 400 described herein with reference to FIG. 4) and to use those scores to manage the allocation of shared network resources within individual network segments in the communications network. For example, the network manager receives SLO scores and adjusts scheduling weights for individual user terminals based at least in part on SLO compliance. Network resources (e.g., bandwidth) in a shared portion of a network can be allocated based at least in part on these scheduling weights.

[0145] In some embodiments, the network manager can be configured to determine a shared SLO score for the shared network based on the received SLO scores. The shared SLO score can represent an aggregation of SLO scores for user terminals assigned to the same shared network resource, such as a carrier.

[0146] As described herein, the SLO scores are based on fine-grained or high-frequency measurements, such as passive speed measurements, which are correlated to customer QoE. The network manager can adjust scheduling weights to achieve more balanced QoE scores and / or to achieve more QoE scores that achieve targeted performance, examples of which are disclosed herein.

[0147] In block 510, the network manager manages network segment traffic based on the output of the objective function. The network manager is configured to receive SLO scores and to use those scores to manage the allocation of network resources on individual segments of the network in the communications network. In some embodiments, the network segment traffic is an aggregation of the shared network traffic across a number of carriers, frequency bands, or the like. In such embodiments, the network segment comprises a plurality of shared network resources that connect from one node or location in the communications network to another node or location. The network manager is configured to balance capacity and usage on individual network segments. Examples of network segments include access networks and / or beams, such as where the communications network includes a satellite network.

[0148] The network manager can be configured to determine a segment SLO score based on the received SLO scores and / or the shared SLO scores determined in block 505. In some embodiments, the segment SLO score represents an aggregation of shared SLO scores assigned to the same network segment. The network manager can be configured to adjust the user terminalsassigned to particular shared network resources (e.g., carriers) to improve segment SLO scores, as described in greater detail herein.

[0149] In block 515, the network manager manages global resource traffic based on the output of the objective function. The network manager is configured to receive SLO scores and to use those scores to globally manage the allocation of network resources. The network manager is configured to allocate global network resources to improve SLO compliance. The network manager is configured to reallocate resources based on customer QoE, moving capacity around the communications network globally to improve or optimize customer QoE.

[0150] Because the SLOs are determined based on customer QoE, the network manager is thus configured to allocate network resources to improve customer QoE. In this way, the network manager is configured to use high- frequency measurements (e.g., passive speed measurements) to improve or optimize SLO performance across a communications network in an automated fashion. Similarly, the network manager is thus configured to control the communications network around customer QoE. In some embodiments, the steps of the method 500 can be performed in sequence or one or more of the steps can be performed simultaneously.

[0151] By way of example, each plan is made up of service flow classes. In a communications network where customers or users share cell towers in a network with multiple cell towers, the network manager is configured to adjust capacity of individual towers in real time to improve customer QoE. The network manager is configured to determine how much capacity to put where to increase or maximize utility. The network manager is configured to attempt to maximize customer QoE (e.g., via QoE assessments) rather than met customer demand. That is, the network manager is configured to attempt to maximize network utility based on QoE scores.

[0152] FIG. 6 illustrates a flow chart of an example method 600 for monitoring service in a communications network based on customer QoE. The method 600 can be performed in any of the schedulers or other components of a communications network described herein with reference to FIGS. 1A, 1 B, and 7. For ease of description, the method 600 will be described as being performed by a network manager, such as the QoE-driven network manager 145a, 145b described herein with reference to FIGS. 1A, 1 B, respectively. This is not to beunderstood to limit the scope of the disclosure. Rather, any step or portion of the method 600 can be performed by any component or combination of components of the communications networks described herein.

[0153] In block 605, the network manager is configured to detect one or more QoE or SLO anomalies using SLO scores as input (e.g., as determined in the method 400 described herein with reference to FIG. 4). The network manager is configured to determine one or more QoE events based on the QoE or SLO anomalies. As described in greater detail herein, a QoE event is an event where measurements indicate a significant deviation from expected performance within a particular region (either topological region or geographical region). A significant deviation can be characterized as a deviation of more than one standard deviation from expected behavior or a deviation of more than two standard deviations from expected behavior.

[0154] In block 610, the network manager is configured to correlate the determined QoE events with each other to identify any correlated anomalies. As described in greater detail herein, correlated anomalies may typically indicate an issue in the network that should be addressed or fixed. These issues may result in network impairments in addition to network outages which are detected by the network manager. In block 615, the network manager is configured to generate one or more alerts based on the correlated events determined in block 610. The alerts can be used by other systems or by personnel to monitor the performance of the communications network. Thus, the network manager is configured to provide QoE-driven service monitoring because the network manager receives SLO scores (where the SLOs and SLO scores are based on QoE assessments) as input to generate alerts and to monitor performance of the communications network.Example Component of a Communications network

[0155] FIG. 7 illustrates a block diagram of a subsystem 770 of a communications system, such as a network manager, scheduler, or gateway, examples of which are described herein with reference to FIGS. 1 A and 1 B. The subsystem 770 is configured to manage network capacity and network resource allocation based at least in part on speed metrics and customer QoE determined as described herein. The subsystem 770 can employ any method described hereinfor QoE-driven network automation, such as the example methods 300, 400, 500, and 600 described herein with reference to FIGS. 3, 4, 5, and 6, respectively.

[0156] The subsystem 770 can include hardware, software, and / or firmware components for managing resource grant allocations. The subsystem 770 includes a data store 771 , one or more processors 773, one or more network interfaces 775, a QoE modeling module 772, a service planning module 774, a network optimization module 776, and a service monitoring module 778. Components of the subsystem 770 can communicate with one another, with external systems, and with other components of a network using communication bus 779. The subsystem 770 can be implemented using one or more computing devices. For example, the subsystem 770 can be implemented using a single computing device, multiple computing devices, a distributed computing environment, or it can be located in a virtual device residing in a public or private computing cloud. In a distributed computing environment, one or more computing devices can be configured to provide the modules 772, 774, 776, 778 to provide the described functionality. In some implementations, the QoE assessments and SLO scores disclosed herein can be calculated by a first component of a communications network, such as a network manager, and exported to other components of the communications network, such as a gateway or a scheduler.

[0157] The subsystem 770 includes the QoE modeling module 772 to assess customer QoE based on fine-grained or high frequency measurements, such as passive speed measurements, as described herein. In some embodiments, the QoE modeling module 772 is configured to implement the method 300, described herein with reference to FIG. 3. In some embodiments, the QoE assessment module is configured to implement the functionality of the modeling system 210 described herein with reference to FIG. 2A.

[0158] The subsystem 770 includes the service planning module 774 to manage and allocate network capacity based on compliance with SLOs to improve customer QoE, as described herein. In some embodiments, the service planning module 774 is configured to implement the method 400, described herein with reference to FIG. 4. In some embodiments, the service planning module 774 is configured to implement the functionality of the service planning system 220 described herein with reference to FIG. 2A.

[0159] The subsystem 770 includes the network optimization module 776 to adjust network parameters in a communications network based on compliance with SLOs to provide QoE-driven network control, as described herein. In some embodiments, the network control module is configured to implement the method 500, described herein with reference to FIG. 5. In some embodiments, the network optimization module 776 is configured to implement the functionality of the network optimization system 230 or the system 250 described herein with reference to FIGS. 2A and 2B, respectively.

[0160] The subsystem 770 includes the service monitoring module 778 to analyze SLO scores to identify events or conditions affecting customer QoE to generate related alerts, thereby providing QoE-driven service monitoring, as described herein. In some embodiments, the service monitoring module is configured to implement the method 600, described herein with reference to FIG. 6. In some embodiments, the service monitoring module 778 is configured to implement the functionality of the service monitoring system 240 described herein with reference to FIG. 2A.

[0161] The subsystem 770 includes one or more processors 773 that are configured to control operation of the modules 772, 774, 776, 778 and the data store 771. The one or more processors 773 implement and utilize the software modules, hardware components, and / or firmware elements configured to provide QoE-driven network automation. The one or more processors 773 can include any suitable computer processors, application-specific integrated circuits (ASICs), field programmable gate array (FPGAs), or other suitable microprocessors. The one or more processors 773 can include other computing components configured to interface with the various modules and data stores of the subsystem 770.

[0162] The subsystem 770 includes the data store 771 configured to store configuration data, user requirements, network statuses, network characteristics and capabilities, control commands, databases, algorithms, executable instructions (e.g., instructions for the one or more processors 773), and the like. The data store 771 can be any suitable data storage device or combination of devices that include, for example and without limitation, random access memory, read-only memory, solid-state disks, hard drives, flash drives, and the like. The data store 771 can be a non-transitory computer-readable medium. The data store771 can store processor-executable instructions to implement one or more of the modules 772, 774, 776, 778 and / or the methods 300, 400, 500, and 600.Additional Embodiments and Terminology

[0163] The present disclosure describes various features, no single one of which is solely responsible for the benefits described herein. It will be understood that various features described herein may be combined, modified, or omitted, as would be apparent to one of ordinary skill. Other combinations and subcombinations than those specifically described herein will be apparent to one of ordinary skill, and are intended to form a part of this disclosure. Various methods are described herein in connection with various flowchart steps and / or phases. It will be understood that in many cases, certain steps and / or phases may be combined together such that multiple steps and / or phases shown in the flowcharts can be performed as a single step and / or phase. Also, certain steps and / or phases can be broken into additional sub-components to be performed separately. In some instances, the order of the steps and / or phases can be rearranged and certain steps and / or phases may be omitted entirely. Also, the methods described herein are to be understood to be open-ended, such that additional steps and / or phases to those shown and described herein can also be performed.

[0164] Some aspects of the systems and methods described herein can advantageously be implemented using, for example, computer software, hardware, firmware, or any combination of computer software, hardware, and firmware. Computer software can comprise computer executable code stored in a computer readable medium (e.g., non-transitory computer readable medium) that, when executed, performs the functions described herein. In some embodiments, computer-executable code is executed by one or more general purpose computer processors. A skilled artisan will appreciate, in light of this disclosure, that any feature or function that can be implemented using software to be executed on a general purpose computer can also be implemented using a different combination of hardware, software, or firmware. For example, such a module can be implemented completely in hardware using a combination of integrated circuits. Alternatively or additionally, such a feature or function can be implemented completely or partially using specialized computers designed to perform the particular functions described herein rather than by general purpose computers.

[0165] Multiple distributed computing devices can be substituted for any one computing device described herein. In such distributed embodiments, the functions of the one computing device are distributed (e.g., over a network) such that some functions are performed on each of the distributed computing devices.

[0166] Some embodiments may be described with reference to equations, algorithms, and / or flowchart illustrations. These methods may be implemented using computer program instructions executable on one or more computers. These methods may also be implemented as computer program products either separately, or as a component of an apparatus or system. In this regard, each equation, algorithm, block, or step of a flowchart, and combinations thereof, may be implemented by hardware, firmware, and / or software including one or more computer program instructions embodied in computer-readable program code logic. As will be appreciated, any such computer program instructions may be loaded onto one or more computers, including without limitation a general purpose computer or special purpose computer, or other programmable processing apparatus to produce a machine, such that the computer program instructions which execute on the computer(s) or other programmable processing device(s) implement the functions specified in the equations, algorithms, and / or flowcharts. It will also be understood that each equation, algorithm, and / or block in flowchart illustrations, and combinations thereof, may be implemented by special purpose hardware-based computer systems which perform the specified functions or steps, or combinations of special purpose hardware and computer-readable program code logic means.

[0167] Furthermore, computer program instructions, such as embodied in computer-readable program code logic, may also be stored in a computer readable memory (e.g., a non-transitory computer readable medium) that can direct one or more computers or other programmable processing devices to function in a particular manner, such that the instructions stored in the computer- readable memory implement the function(s) specified in the block(s) of the flowchart(s). The computer program instructions may also be loaded onto one or more computers or other programmable computing devices to cause a series of operational steps to be performed on the one or more computers or other programmable computing devices to produce a computer-implemented process such that the instructions which execute on the computer or other programmableprocessing apparatus provide steps for implementing the functions specified in the equation(s), algorithm(s), and / or block(s) of the flowchart(s).

[0168] Some or all of the methods and tasks described herein may be performed and fully automated by a computer system. The computer system may, in some cases, include multiple distinct computers or computing devices (e.g., physical servers, workstations, storage arrays, etc.) that communicate and interoperate over a network to perform the described functions. Each such computing device typically includes a processor (or multiple processors) that executes program instructions or modules stored in a memory or other non- transitory computer-readable storage medium or device. The various functions disclosed herein may be embodied in such program instructions, although some or all of the disclosed functions may alternatively be implemented in applicationspecific circuitry (e.g., ASICs or FPGAs) of the computer system. Where the computer system includes multiple computing devices, these devices may, but need not, be co-located. The results of the disclosed methods and tasks may be persistently stored by transforming physical storage devices, such as solid state memory chips and / or magnetic disks, into a different state.

[0169] Unless the context clearly requires otherwise, throughout the description and the claims, the words “comprise,” “comprising,” and the like are to be construed in an inclusive sense, as opposed to an exclusive or exhaustive sense; that is to say, in the sense of “including, but not limited to.” The word “coupled”, as generally used herein, refers to two or more elements that may be either directly connected, or connected by way of one or more intermediate elements. Additionally, the words “herein,” “above,” “below,” and words of similar import, when used in this application, shall refer to this application as a whole and not to any particular portions of this application. Where the context permits, words in the above Detailed Description using the singular or plural number may also include the plural or singular number respectively. The word “or” in reference to a list of two or more items, that word covers all of the following interpretations of the word: any of the items in the list, all of the items in the list, and any combination of the items in the list. The word “exemplary” is used exclusively herein to mean “serving as an example, instance, or illustration.” Any implementation described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other implementations.

[0170] The disclosure is not intended to be limited to the implementations shown herein. Various modifications to the implementations described in this disclosure may be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other implementations without departing from the scope of this disclosure. The teachings of the invention provided herein can be applied to other methods and systems, and are not limited to the methods and systems described above, and elements and acts of the various embodiments described above can be combined to provide further embodiments. Accordingly, the novel methods and systems described herein may be embodied in a variety of other forms; furthermore, various omissions, substitutions and changes in the form of the methods and systems described herein may be made without departing from the scope of the disclosure. The accompanying claims and their equivalents are intended to cover such forms or modifications as would fall within the scope of the disclosure.

Claims

WHAT IS CLAIMED IS:1 . A method of assessing customer quality of experience (QoE) in a communications network, the method comprising: acquiring fine-grained passive measurements of network resources of the communications network; acquiring application-layer measurements of network applications used on the communications network; and applying one or more QoE models to generate customer QoE objectives based on a correlation between the acquired fine-grained passive measurements and the acquired application-layer measurements.

2. The method of claim 1 , wherein the fine-grained passive measurements include passive speed measurements.

3. The method of claim 1 , wherein the fine-grained passive measurements include passive latency measurements.

4. The method of claim 1 , wherein the fine-grained passive measurements include passive usage measurements.

5. The method of any one of claims 1 to 4, wherein the fine-grained passive measurements are acquired with a rate that is at least 10 times the rate of network parameter adjustment.

6. The method of any one of claims 1 to 5, wherein the application-layer measurements include a resolution of media content streamed to a customer.

7. The method of any one of claims 1 to 6, wherein the application-layer measurements include a bitrate of a network application provided to a customer.

8. The method of any one of claims 1 to 7, wherein the fine-grained passive measurements are determined for service flow classes.

9. The method of any one of claims 1 to 8, wherein the customer QoE objectives are determined using machine learning algorithms that relate the fine-grained passive measurements and the application-layer measurements to customer QoE.

10. The method of any one of claims 1 to 9, wherein the customer QoE objectives are expressed in terms of network metrics for individual customers.1 1. A method of service planning in a communications network based on customer quality of experience (QoE), the method comprising: receiving one or more service level objectives (SLOs); determining an SLO score for individual SLOs based at least in part on compliance with the SLO as determined using an objective function; and allocating network capacity based on the determined SLO score, the network capacity allocated to improve SLO compliance.

12. The method of claim 11 , wherein the SLO score is determined for individual service flows.

13. The method of claim 11 , wherein the SLO score is determined for individual service flow classes.

14. The method of any one of claims 11 to 13, wherein the objective function uses fine-grained passive measurements to determine the SLO score.

15. The method of claim 14, wherein the fine-grained passive measurements include passive speed measurements.

16. A method of controlling a communications network based on customer quality of experience (QoE), the method comprising: adjusting scheduling weights of individual user terminals within a carrier based at least in part on compliance with an SLO, compliance with the SLO determined based at least in part on an objective function; balancing network utilization of user terminals for individual carriers of the communications network; and allocating network resources to improve compliance with the SLO thereby improving customer QoE due at least in part to a relationship between the SLO and customer QoE.

17. The method of claim 16 further comprising determining a carrier SLO score using the objective function.

18. The method of claim 17 further comprising determining a beam SLO score using the objective function.

19. The method of any one of claims 16 to 18, wherein the communications network is a satellite communications network.

20. The method of any one of claims 16 to 19, wherein the objective function uses passive speed measurements to determine SLO compliance.21 . A method of monitoring service in a communications network based on customer quality of experience (QoE), the method comprising: detecting one or more QoE anomalies, the one or more QoE anomalies determined based at least in part on an SLO score calculated using an objective function; correlating one or more network events with the one or more QoE anomalies; and generating an alert to indicate the one or more network events that caused the one or more QoE anomalies.

22. The method of claim 21 , wherein the SLO score is determined using fine-grained passive measurements as input to the objective function.

23. The method of claim 22, wherein the fine-grained passive measurements include passive speed measurements.

24. A communications network comprising: a gateway; a scheduler communicatively coupled to the gateway; and a network manager communicatively coupled to the scheduler, the network manager configured to: receive a plurality of service level objectives (SLOs);determine an SLO score for each SLO using an objective function that uses fine-grained measurements and a corresponding SLO to determine the SLO score; and automatically adjust network resource allocation to improve an aggregate SLO score, the aggregate SLO score based on an aggregation of individual SLO scores.

25. The communications network of claim 24, wherein improving the aggregate SLO score automatically improves average customer QoE across the communications network.

26. A communications network comprising: a gateway; a scheduler communicatively coupled to the gateway; and a network manager communicatively coupled to the scheduler, the network manager configured to: acquire fine-grained passive measurements of network resources of the communications network; acquire application-layer measurements of network applications used on the communications network; and apply one or more QoE models to generate customer QoE objectives based on a correlation between the acquired fine-grained passive measurements and the acquired application-layer measurements.

27. A communications network comprising: a gateway; a scheduler communicatively coupled to the gateway; and a network manager communicatively coupled to the scheduler, the network manager configured to: receive one or more service level objectives (SLOs); determine an SLO score for individual SLOs based at least in part on compliance with the SLO as determined using an objective function; andallocate network capacity based on the determined SLO score, the network capacity allocated to improve SLO compliance.

28. A communications network comprising: a gateway; a scheduler communicatively coupled to the gateway; and a network manager communicatively coupled to the scheduler, the network manager configured to: adjust scheduling weights of individual user terminals within a carrier based at least in part on compliance with an SLO, compliance with the SLO determined based at least in part on an objective function; balance network utilization of user terminals for individual carriers of the communications network; and allocate network resources to improve compliance with the SLO thereby improving customer QoE due at least in part to a relationship between the SLO and customer QoE.

29. A communications network comprising: a gateway; a scheduler communicatively coupled to the gateway; and a network manager communicatively coupled to the scheduler, the network manager configured to: detect one or more QoE anomalies, the one or more QoE anomalies determined based at least in part on an SLO score calculated using an objective function; correlate one or more network events with the one or more QoE anomalies; and generate an alert to indicate the one or more network events that caused the one or more QoE anomalies.

30. A network optimization system comprising: a capacity allocation engine configured to receive usage metrics for a plurality of service areas, each service area including a plurality of carriers, and to receive service level objective (SLO) score metrics for the plurality ofservice areas and to allocate network resources to individual service areas of the plurality of service areas and to individual carriers within individual service areas of the plurality of service areas; a usage aggregator for each service area of the plurality of service areas, each usage aggregator configured to aggregate usage metrics from the plurality of carriers associated with the service area; a SLO score aggregator for each service area of the plurality of service areas, each SLO score aggregator configured to aggregate SLO score metrics from the plurality of carriers associated with the service area; and a service area load balancer for each service area of the plurality of service areas, the service area load balancer configured to receive the aggregated usage metrics from the usage aggregator and to receive the aggregated SLO score metrics from the SLO score aggregator and to update mapping of user terminals to the plurality of carriers in the associated service area to load balance the associated service area across the plurality of carriers.31 . The system of claim 30 further comprising a scheduler for each carrier of a service area of the plurality of service areas, the scheduler configured to receive a carrier capacity determination from the capacity allocation engine and a service flow demand from the service area load balancer to schedule network resources for user terminals within the carrier of the service area.

32. The system of claim 31 further comprising a usage controller configured to receive usage metrics for individual user terminals from the scheduler to determine a weight multiplier that adjusts a base weight for individual service flow classes to determine and adjusted scheduling weight per service flow, the scheduler configured to use the adjusted scheduling weight to adjust network resource allocation among user terminals within the carrier.

33. The system of claim 32 further comprising a service flow class (SFC) SLO score balancer configured to provide the base weight for individual service flow classes based at least in part on SLO measurements per SFC provided by the scheduler and a target SLO.

34. The system of claim 33 further comprising a carrier SLO score aggregator configured to receive SLO scores per SFC that are results of comparing the SLO measurements per SFC to the target SLO, the carrier SLO score aggregator configured to aggregate the received SLO scores and to provide the aggregated SLO scores to the SLO score aggregator.

35. The system of claim 34, wherein the scheduler is further configured to provide usage metrics per user terminal to the usage aggregator.

Images