System and method for energy-efficient resource allocation for microservice-based application in the cloud

Abstract

Methods and systems are described that optimize the performance and energy consumption of microservice-based applications. Some embodiments can comprise methods that combine a fine-grained core allocation, core mapping, and CPU frequency scaling considering the impact on other concurrently running microservices. Some embodiments can comprise methods that consider the characteristics of the underlying environment (e.g., capability of the hardware platform in support of DVFS, service characteristics, requirements, etc.). Some embodiments can comprise methods that manage different application types, e.g., real-time and best-effort services, and handle prioritization. Some embodiments comprise methods where the hardware power management is applicable to services running on top of a higher layer cloud-based management systems.

Description

SYSTEM AND METHOD FOR ENERGY-EFFICIENT RESOURCE ALLOCATION FOR MICROSERVICE-BASED APPLICATION IN THE CLOUDTECHNICAL FIELD

[0001] The present disclosure generally relates to the technical field of cloud computing and more particularly to application service chains.BACKGROUND

[0002] Applications are becoming complex and are moving from a single unified unit, also called monolithic application, to services comprising smaller units, also called microservices, with complex dependencies. The service may consist of multiple execution paths in which traffic or requests are directed. The requests paths (herein onwards are referred to as service chain) imply the connection of an ordered list of microservices forming a complete end- to-end service. Some microservices can be part of multiple service chains simultaneously which can complicate the interaction pattern among microservices. These microservice instances belonging to a service chain may be deployed in a distributed manner, e.g., in different nodes based on the requirements associated to these microservices, resource limitation in each node, etc. The nodes can be heterogenous in terms of available capacity, capability to handle service load, performance, energy usage, etc.

[0003] Though microservice-based architecture brings many benefits such as high agility, flexibility, and scalability, it introduces additional complexity and challenge in the management of services, e.g., assurance of service performance. Performance can be impacted by the distance between the source and destination node if microservices are distributed across nodes, as well as the nodes’ capability and capacity. The microservices typically run concurrently in a shared underlying environment that might adversely affect performance if not carefully managed. In addition, a performance bottleneck in a microservice might have a cascading effect on other dependent microservices (i.e., intra-service chain dependencies and inter-service chain dependencies) and the whole service chain affecting the end-to-end performance requirement. Furthermore, the impact of change in the performance of individual microservice to the overallchain performance might be different. Depending on the microservice type, the chain performance could be sensitive or tolerant to performance changes of a given microservice.

[0004] Moreover, some service chains might have strict performance requirements while some have relaxed requirements. For example, in a media service, video streaming service chain might have a real-time requirement (referred to as real-time service chains) compared to a service chain that provides movie recommendation or backend services performing caching and backups (referred to as best effort service chains).

[0005] At the node level, several resource allocation techniques have been proposed to manage the performance of services. One such mechanism has been to dynamically change the core allocation to tasks / applications. In the cloud context, this is called vertical scaling where, for instance, the CPU (central processing unit) resources available to virtual instances are changed to handle dynamic loads during the service operation. In systems that use Linux™ control groups (CGroups) for resource management CPUs can be assigned at fraction levels for more finegrained CPU allocation. In addition, virtual CPUs could be mapped / pinned to physical CPUs for exclusive access, which can facilitate CPU resource isolation and better guaranteed performance, mainly at the cost of efficient resource usage (which is achieved through CPU sharing).

[0006] One factor that affects the performance of cores is the frequencies they are running as it affects the number of cycles executed in a second. CPU power consumption is also impacted by the frequency it is running and the associated voltage. Processors have a range of frequencies and corresponding voltages over which they may operate. Techniques such as voltage / frequency scaling, i.e., Dynamic Voltage and Frequency Scaling (DVFS) have been used to adjust the power consumption of cores based on the workload. DVFS have been shown to save power at times of lower utilization but its effectiveness in general depends on how much performance could be sacrificed. That requires detailed understanding and accurate estimation of the performance and power usage of the system. Some processor architectures (e.g., Intel Sky Lake™, Intel Sandy Bridge™) support a per-chip / per-cluster DVFS, where a uniform frequency and voltage are assigned to all the cores of a chip together sacrificing fine-grained energy management. Other processors (e.g. Intel Haswell™, AMD Ryzen™) can support per-core DVFS, allowing to adjust individual cores independently with the support of core level voltage regulator but may entail extra overhead. Software such as Linux cpufreq use power governors to adjust the frequency.

[0007] In a cloud environment, by default cores are shared by multiple instances (e.g. pods) and DVFS is controlled by the power management system that is not in the controller of the cloud orchestration framework (e.g., K8s). Recent advancement has enabled the exposure as well as controller of hardware level information in the cloud management system, using for example CRI-RM (Container Runtime Interface - Resource Manager). CRI-RM is an opensource framework from Intel™ that extends the functionality of Kubernetes™ to support OS and hardware level optimizations. It is an add-on plugin between the Kubernetes kubelet and the container runtime used for controlling the resource allocation of containers in a Kubernetes cluster.

[0008] There currently exist certain challenges. Existing solutions all have drawbacks, as set forth below.

[0009] One existing solution is a mechanism where DVFS decision making is specific to the hardware platform considered. Solutions that apply DVFS focus on adjusting frequency either at a core-level or at a chip-level based on what is supported in the system they are implemented. As described in the previous section a per-core DVFS solution (see e.g., Acun, Bilge, Kavitha Chandrasekar, and Laxmikant V. Kale. "Fine-grained energy efficiency using per- core dvfs with an adaptive runtime system." 2019 Tenth International Green and Sustainable Computing Conference (IGSC). IEEE, 2019), if available, provides fine-grained power management and more power savings. On the other hand, for a per-chip DVFS solutions (e.g., Sarood, Osman, et al. "“Cool” load balancing for high performance computing data centers." IEEE Transactions on Computers 61.12 (2012): 1752-1764), if the workloads / tasks running have different characteristics (as in the case in many cloud application), they cannot run at their optimal frequency and would have a lower gain in energy-savings if not applied appropriately, for instance by combining with other fine-grained resource management technique. This requires a mechanism that considers the capability of the underlying power management system and adapts accordingly.

[0010] Other solutions may comprise DFVS-aware mechanisms, but these are not necessarily DVFS-optimized mechanisms. Some solutions (e.g., Chen, Jing, et al. "ERASE: Energy efficient task mapping and resource management for work stealing runtimes." A CM Transactions on Architecture and Code Optimization (TACO) 19.2 (2022): 1-29) take the DVFS that is controlled externally into account when making decisions such as scheduling decision. This allows solutions to be energy-aware, avoids unnecessary or conflicting decisions. It however provides limited opportunity to save more power due to lack of close coordination and if the DFVSis controlled for a system wide optimization, it has limited impact on application-aware power optimization, particularly in virtualized environments.

[0011] Solutions that coordinate and control DFVS with other resource allocation mechanisms (e.g., resource scaling, scheduling) could maybe result in better energy performance for applications running in virtualized environments.

[0012] Other solutions use resource allocation approaches focusing on one of these mechanisms: DVFS or core scaling. Some solutions (e.g., Chen, Shuang, et al. "Retail: Opting for learning simplicity to enable qos-aware power management in the cloud." 2022 IEEE International Symposium on High-Performance Computer Architecture (HPCA). IEEE, 2022; Sarood; Acun) use DVFS mechanisms that adjust the frequency of cores for specific tasks. In a shared environment this may impact the performance / energy usage of other collocated tasks. This is mainly handled by core assignment that map a given task to a specific core to provide better isolation. This on the other hand can lead to wastage of resources and energy, for example if the task is small and does not utilize a core well. In addition, in the case of per-chip DVFS, the uniform core frequency assignment may result in the under or over-provisioning of processing power for tasks running on the cores (the DVFS is even more coarse grained here). Other mechanisms (e.g. CPU scaling) must be in place to compensate for the under provisioning (in case of a lower frequency setting than needed) and the take off the over provisioning (in case of a higher frequency setting than needed).

[0013] CPU scaling for services (e.g., Vertical Pod Autoscaling, available at btpsj^cjoud. google.has been used to dynamically adapt the number of cores due to changes in in CPU load. Firstly, the CPU scaling alone may not lead to low power usage if not supported by other power management techniques. Secondly, in the presence of DVFS mechanism, current CPU scaling solutions are not DVFS aware: they are mainly triggered by raw CPU utilization. However, CPU utilization is not a good indicator of the performance of cores. A core highly utilized but running at low CPU frequency might be performing equally well with a core that is utilized low but running at a high frequency. Thirdly, CPU scaling solutions are not performed in close coordination with DVFS control. These decisions are mainly performed separately, the DFVS mainly triggered by the overall load of the system while CPU scaling is triggered by the performance of individual application instances. In a virtualized environment the challenges will be bigger in applying the hardware powermanagement system in coordination with CPU scaling for containerized applications (running on a shared environment) with minimum impact on other collected instances.

[0014] To handle these issues, a more accurate and coordinated resource allocation that combines CPU and frequency scaling is needed.

[0015] In existing solutions, the impact of dependent tasks is not considered for energy performance optimization. Many solutions (e.g. Chen, Shuang et al.; Acun; Chen, Jing, et al.; and Sarood) address the problem of resource allocation, i.e., frequency scaling and CPU scaling, for tasks or services with no dependency. However, this might be poorly applicable for services that have dependency among its components (e.g., microservice-based services) as the resource management becomes complex for these types of services. For example, a poorly chosen resource for a given component / microservice has a cascading effect on performance and energy usage of other dependent microservices of a chain. This could also impact the performance and energy usage of the whole chain. In addition, the efficient assignment / mapping of resources (e.g., cores) to microservices might be impacted by dependencies between the microservices; execution might be sequential if two dependent microservices are allocated on single core, or parallel if there is no dependency among them.

[0016] Understanding and modeling the performance and energy usage of microservices and service chain is required to make informed decisions.

[0017] Some existing solutions only consider a single type of application. Many solutions (e.g. Chen, Shuang et.al.) address the problem of resource allocation for a particular type of application. For example, several solutions, e.g., frequency scaling, or CPU scaling mechanisms, have been designed for short-lived, best-effort tasks that tolerate performance penalty. On the other hand, there are solutions that target online, long running and performance critical applications. A mechanism that works well for one type of application may not work well for another. For example, batching of tasks to create effective periods to apply DVFS for bestefforts tasks may not work well for time-critical applications that have stricter performance requirement. On the other hand, resource allocations that over-provision resources for time-critical applications to provide better isolation and improved performance (mainly with power management disabled) might lead to inefficient resource utilization and less energy performance.SUMMARY

[0018] One embodiment under the present disclosure comprises a method for optimizing performance of one or more microservices being run in one or more nodes comprising one or more hardware architectures. The method comprises generating a representation model of one or more microservices based at least in part on one or features of one or more service chains related to the one or more microservices; extracting one or more statuses of the one or more service chains, the one or more statuses identifying one or more nodes hosting the one or more microservices; collecting one or more metrics related to the one or more nodes and the one or more microservices. It further includes extracting one or more capabilities of the one or more hardware architectures; collecting one or more resource usage metrics related to the one or more nodes or one or more microservices; estimating service chain performance based at least in part on the one or more performance metrics. It further comprises estimating energy consumption of the one or more microservices based at least in part on the one or more resource usage metrics; allocating, based at least in part on the one or more capabilities, one or more cores comprising the one or more nodes for performing the one or more microservices; mapping the one or more microservices to the one or more cores, the mapping comprising a configuration; and applying the configuration to the one or more nodes and the one or more microservices.

[0019] Another embodiment under the present disclosure is a system for modeling and monitoring one or more nodes running one or more microservices, the one or more microservices associated with one or more service chains and the one or more nodes comprising one or more hardware architectures. The system comprises an application modeler configured to generate a representation model of the one or more microservices based at least in part on the one or more service chains. The system also comprises a monitor configured to monitor the one or more nodes and to perform one or more of, detect a status of the one or more service chains, collect one or more metrics related to the one or more nodes or one or more microservices, collect one or more capabilities of the one or more hardware architectures. The system also comprises a chain performance estimator coupled to the monitor and the application modeler and configured to estimate a performance of the one or more service chains based at least in part on the one or more metrics. The system further comprises a microservice energy estimator coupled to the monitor and configured to estimate energy consumption of the one or more microservices based at least in part on the one or more metrics. It further comprises a CPU resource allocator coupled to theapplication modeler, the chain performance estimator, and the microservice energy estimator and configured to allocate, based at least in part on the one or more capabilities, one or more cores comprising the one or more nodes for performing the one or more microservices, and further configured to map the one or more microservices to the one or more cores, the mapping comprising a configuration, the resource allocator further configured to provide the configuration to an actuator configured to apply the configuration to the one or more nodes. The CPU resource allocator is further configured to set CPU frequency for the one or more cores.

[0020] Another embodiment under the present disclosure is a method for modeling and monitoring one or more nodes running one or more microservices, the one or more microservices associated with one or more service chains and the one or more nodes comprising one or more hardware architectures. The method comprises receiving a service description describing one or more microservices to be performed in one or more nodes; generating a representation model based at least in part on the service description, the representation model describing one or more service chains for performing the one or more microservices in the one or more nodes; collecting one or more energy usage metrics, one or more CPU metrics, and one or more performance metrics related to the one or more service chains or one or more nodes; detecting one or more performance violations related to at least one of the one or more energy usage metrics, the one or more CPU metrics, or one or more performance metrics; requesting one or more additional performance metrics and one or more additional energy usage metrics; detecting one or more statuses related to the one or more nodes; extracting one or more microservice statuses related to the one or more microservices; estimating a performance level of the one or more service chains based at least in part on the one or more microservice statuses; estimating an energy usage level of the one or more microservices; detecting one or more capabilities of the one or more nodes or one or more hardware architectures; creating a new resource configuration for the one or more service chains, one or more microservices, one or more nodes, and / or one or more hardware architectures; translating the new resource configuration to one or more policies, the one or more policies formatted to be used by a cloud orchestrator to apply the new resource configuration to the one or more nodes; and transmitting the one or more policies to the cloud orchestrator.

[0021] Another embodiment is a system for modeling and monitoring one or more nodes running one or more microservices, the one or more microservices associated with one or more service chains and the one or more nodes comprising one or more hardware architectures.The system comprises processing circuitry and a memory. The memory contains instructions executable by the processing circuitry whereby the system is operative to perform the steps of: receiving a service description describing one or more microservices to be performed in one or more nodes; generating a representation model based at least in part on the service description, the representation model describing one or more service chains for performing the one or more microservices in the one or more nodes; collecting one or more energy usage metrics, one or more CPU metrics, and one or more performance metrics related to the one or more service chains or one or more nodes; detecting one or more performance violations related to at least one of the one or more energy usage metrics, the one or more CPU metrics, or one or more performance metrics; requesting one or more additional performance metrics and one or more additional energy usage metrics; detecting one or more statuses related to the one or more nodes; extracting one or more microservice statuses related to the one or more microservices; estimating a performance level of the one or more service chains based at least in part on the one or more microservice statuses; estimating an energy usage level of the one or more microservices; detecting one or more capabilities of the one or more nodes or one or more hardware architectures; creating a new resource configuration for the one or more service chains, one or more microservices, one or more nodes, and / or one or more hardware architectures; translating the new resource configuration to one or more policies, the one or more policies formatted to be used by a cloud orchestrator to apply the new resource configuration to the one or more nodes; and transmitting the one or more policies to the cloud orchestrator.

[0022] This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an indication of the scope of the claimed subject matter.BRIEF DESCRIPTION OF THE DRAWINGS

[0023] For a more complete understanding of the present disclosure, reference is now made to the following descriptions taken in conjunction with the accompanying drawings, in which:

[0024] Fig. 1 illustrates an example of a system embodiment under the present disclosure;

[0025] Fig. 2 illustrates an example of a system embodiment under the present disclosure;

[0026] Fig. 3 illustrates a flow-chart of a method embodiment under the present disclosure;

[0027] Fig. 4 illustrates a flow-chart of a method embodiment under the present disclosure;

[0028] Fig. 5 shows a schematic of a communication system embodiment under the present disclosure;

[0029] Fig. 6 shows a schematic of a user equipment embodiment under the present disclosure;

[0030] Fig. 7 shows a schematic of a network node embodiment under the present disclosure;

[0031] Fig. 8 shows a schematic of a host embodiment under the present disclosure;

[0032] Fig. 9 shows a schematic of a virtualization environment embodiment under the present disclosure; and

[0033] Fig. 10 shows a schematic representation of an embodiment of communication amongst nodes, hosts, and user equipment under the present disclosure.DETAILED DESCRIPTION

[0034] Before describing various embodiments of the present disclosure in detail, it is to be understood that this disclosure is not limited to the parameters of the particularly exemplified systems, methods, apparatus, products, processes, and / or kits, which may, of course, vary. Thus, while certain embodiments of the present disclosure will be described in detail, with reference to specific configurations, parameters, components, elements, etc., the descriptions are illustrative and are not to be construed as limiting the scope of the claimed embodiments. In addition, the terminology used herein is for the purpose of describing the embodiments and is not necessarily intended to limit the scope of the claimed embodiments. Some of the embodiments contemplated herein will now be described more fully with reference to the accompanying drawings.

[0035] As discussed above, the current solutions fail to address all the challenges identified. Systems and methods with careful resource allocation that collocates best-effort with time critical application could provide better opportunity to improve resource utilization (and energy reduction) while meeting performance requirement of time-critical application, for example by throttling best-efforts tasks at times of resource congestion.

[0036] Certain aspects of the disclosure and their embodiments may provide solutions to the problems identified above or other challenges. Certain embodiments can comprise systems and / or methods that dynamically scale the number of CPUs as well as CPU frequency for microservices considering the impact of other concurrently running microservices and by considering the capability of the hardware in scaling the CPU frequency. To achieve the optimization of energy usage and performance, embodiments can also perform a mapping of cores to time-critical and best efforts microservices by examining the impact of collocation on interference and efficient usage of resources. Embodiments may achieve these aspects or other benefits through continuous monitoring of the internal dynamics of the system, knowledge extraction and control mechanisms. For example, a closed-loop control mechanism can provide a continuous scaling decision, capable of detecting deviations from the expected performance requirements (e.g., latency) and detecting energy savings opportunities. It monitors the performance of individual microservices and the service chain, as well as other metrics including resource usage, the supported CPU frequency scaling, and / or the CPU frequency settings of cores. It can analyze the extracted information and model the performance end energy usages to determine e.g., the right number of cores, the frequency to be used, the mapping of virtual CPUs to physical cores for microservices to reduce energy consumption while meeting performance requirement. The necessary configuration changes can be applied by communicating with the cloud orchestration framework (e.g., CRI-RM in Kubernetes).

[0037] Certain embodiments include methods that optimize the performance and energy consumption of microservice-based applications. Some embodiments can comprise methods that combine a fine-grained core allocation, core mapping, and CPU frequency scaling considering the impact on other concurrently running microservices. Some embodiments can comprise methods that consider the characteristics of the underlying environment (e.g., capability of the hardware platform in support of DVFS, service characteristics, requirements, etc.). Some embodiments can comprise methods that manage different application types, e.g., real-time andbest-effort services, and handle prioritization. Some embodiments comprise methods where the hardware power management is applicable to services running on top of a higher layer cloud-based management systems.

[0038] Certain embodiments may provide one or more of the following technical advantages. Certain embodiments can allocate the right resources through the combined frequency and fractional core scaling, which is important for optimized energy performance. Certain embodiments can also manage dependency effectively in a shared environment, reducing energy hotspots, and SLO (service level objective) violation by using microservice and service chain knowledge, knowledge about the underlying platform and making informed decisions. Certain embodiments can also tackle resource overprovisioning and resource usage inefficiency through sharing of resources between microservices of time-critical and best-effort service chains.

[0039] As described previously, embodiments can provide a fine-grained resource allocation that controls CPU allocation including fractional CPU assignment and a mapping of cores to specific physical CPUs as well as different CPU frequency scaling options based on the capabilities supported by the hardware platforms for microservices deployed in the cloud. Objectives include fulfilling performance requirements while minimizing energy usage.

[0040] Figure 1 displays a system 10 embodiment under the present disclosure for modeling and monitoring microservices and service chains. Application modeler 15 receives a service description 5 and models the microservice chain for the service description as a workflow. It takes into consideration information provided by e.g., the application developer / operator regarding the topology of a microservice-based application, the associated performance requirements, and service type in order to keep a representational model of the application. Monitor 30 is responsible to monitor the state of the microservices and underlying system including resource usage of microservices, the capability of the hardware architecture in DFVS management, the frequency of the individual cores if applicable, and the performance of individual microservices. Monitor 30 can be distributed across nodes 80, 90 to monitor the different parts of the nodes 80, 90 the microservice chain is running. It can also be part of the application platform / orchestrator to retrieve service-related metrics (e.g., microservice CPU usage and performance). Monitor 30 can interface with chain performance estimator 20 and microservice energy estimator 50.

[0041] Chain performance estimator 20 is responsible for estimating the service chain performance from the microservice performance data received from monitor 30. This can be done by analyzing, for example relevant trace data (if tracing technique is used for performance metrics extraction) that reflects the flow of requests through the microservices and providing aggregated statistical performance data from it. Chain performance estimator 20 also profiles the characteristics of the microservice in terms of sensitivity to collocation with other microservice running on same cores, and variabilities of performance of microservice overtime.

[0042] Microservice energy estimator 50 on the other hand estimates the energy consumption (EC) of microservices based on the metrics extracted from monitor 30. The estimation is mainly done based on software-based methods as microservices run in a shared environment. Based on the service resource usage, microservice energy estimator 50 can consider the CPU (and CPU frequency), memory, disk and / or network in the EC calculation. For computeintensive services, for example, a model can be used that considers the CPU usages, and CPU frequency in the estimation. The extracted information from the chain performance estimator 20 and microservice energy estimator 50 is sent to CPU resource allocator 40 when requested.

[0043] CPU resource allocator 40 considers the information retrieved from application modeler 15, chain performance estimator 20, and microservice energy estimator 50, and in case of detecting performance deviations from the expected performance (e.g., latency) requirements, it will decide on new allocation. It can also trigger resource allocation in case of identifying configuration that yields to lower energy compared to the current state while meeting performance requirements. It extracts the DVFS capabilities of nodes 80, 90. Nodel 80 to nodeN 90 can each comprise one or more chips 88, 98, which can each comprise one or more cores 89, 99. For, example if a per-core DFVS is supported, it can decide the frequency of the cores of individual microservices, by determining the minimal frequency and number of (fractional) cores that could fulfil the chain performance requirement and reduce energy usage. For a per-core DVFS, the cores are preferably pinned to the microservices as well. In case of fractional core assignment for a time-critical microservice, the remaining fraction of a core can be assigned to best-effort microservice type to facilitate higher utilization. In case of per-chip DVFS, CPU resource allocator 40 would select the lowest frequency for all the cores of a chip and scale the number of cores for individual microservices that run on the chip. This (i.e., by also considering the support of CPU scaling) will avoid the lower or higher processing power that can result due to applying uniformCPU frequency only. Microservice- to-core mapping can also be considered here to avoid two time- critical microservices from competing for the same resources. CPU resource allocator 40 can be seen as consisting of two logical components, a core and frequency allocator 42 and a microservice-to-core mapper 44. Core and frequency allocator 42 is for the combined setting / scaling of cores and frequency. Ms-to-core mapper 44 determines the mapping of microservices to physical cores based on the topology of the architecture (e.g., using NUMA (non- uniform memory access) topology). If there is change in resource allocation, CPU resource allocator 40 communicates with actuator 60 for the application of new resource configuration.

[0044] Actuator 60 can apply new configurations from CPU resource allocator 40 to managed system 70 or nodes 80,90 by instantiating e.g., 1) the core scaler 62 for the scaling of CPU for a given microservice, 2) the frequency scaler 64 for the scaling of CPU frequency of identified cores 3) Ms-to-Core mapper 66 for the mapping of microservices to physical cores.

[0045] Monitor 30 can continuously collect related energy, performance, resource usage, and hardware parameters. Chain performance estimator 20 and microservice energy estimator 50 can perform further analysis. Resource allocator 40 can access these metrics and analyses from monitor 30, chain performance estimator 20, and microservice energy estimator 50, and based on the result, CPU resource allocator 40 can continuously update the resource allocation decision in order to improve energy performance of services.

[0046] Managed system 70 can comprise one or more nodes, such as nodel 80 up to nodeN 90. Each node can comprise / run one or more microservices, such as e.g., Msl 81, Ms2, 83, Ms5 82 of nodel 80, and Ms3 91, Ms4 93, and Ms6 92 of nodeN 90.

[0047] A potential implementation of a proposed system 200 in a cloud platform is shown in Figure 2, showing a potential mapping to management functions. Application modeler 215, the CPU resource allocator 240, the chain performance estimator 220, the microservice energy estimator 250, and the monitor 230 can be part of a cloud platform manager 210 or cloud management platform that monitors and controls resources, and services in a cloud environment (e.g., Google™ Cloud Platform that provides infrastructure as a service, and platform as a service). Monitor 230 can be based on a metric server or Prometheus™ resource usage monitoring, open telemetry for extracting performance of microservices (tracing-based), while low level hardware and software components of containerized application can be monitored using Intel™ Performance Monitoring Unit (PMU) plugin. Chain performance estimator 220 and microservice energyestimator 250 can be implemented using analytics-based techniques such as statistical regression methods to predict performance and energy usage of microservices / chain. CPU resource allocator 240 can be implemented using machine learning (ML) techniques such as remote learning (RL)- based methods that incrementally learn from their environment while taking actions. For actuation, the CRI-RM running in the Kubernetes nodes can be used as a proxy to control the resource allocation of the containers. The CPU allocation (by controlling container CPU shares and allocations), the CPU frequency setting, and (NUMA-aware) CPU mapping can be applied by extending the CRI-RM with new policies. Actuator 265 (with core scaler 262, frequency scaler 264, and Ms-to-core mapper 266) can be stored in a Kubernetes instance / container 260. A managed system 270 including e.g. nodes (e.g., nodel 280... nodeN 290) can comprise a portion of Kubernetes container 260. Each node 280, 290 can comprise e.g., microservices, e.g., Msl 281, Ms2283, Ms5 282, Ms3 284, Ms4286, Ms6285.

[0048] Further description is given below of the components of system 200. The following descriptions can be applicable to a variety of embodiments, such as system 10 of Figure 1.

[0049] Application modeler 215 parses a received / accessed service description 205 provided by the application developer / service provider and generates a representation model that illustrates, e.g., the relationship, dependencies of microservices, service type (e.g., time-critical, best-effort) and performance requirements (e.g. latency) of service chains. Regarding dependencies between microservices, in some embodiments the relationship can be extracted from the service description provided by the application developer / service provider, which might be described for example using TOSCA modeling language (Topology and Orchestration Specification for Cloud Applications). The TOSCA may describe the microservices, their type, requests with associated the performance requirements and type, the sequence of microservices (i.e., dependent microservices). In some embodiments the relationship between microservices can be extracted during runtime from traffic performance metrics (e.g., using service mesh Istio Observability) or from distributed tracing data (e.g. using the Query Service of a distributed tracing tool, Jaeger). Traffic performance metrics can show the per service interaction and dependencies among microservices. Tracing data can also show the actual interaction between the source and target microservices in the handling of requests. Application modeler 215 can generate a representation model of the service by extracting dependencies from service description / trafficperformance metrics / distributed tracing data. It also maps other service properties (e.g., service type, performance requirement). It may use storage systems such as a graph database to store relationships between the microservices. This way it is possible to easily get the dependency graph / service chain-level information. For example, by specifying the service name, it would be possible to retrieve a dependency graph with associated vertices and edges.

[0050] Monitor 230 is responsible to monitor the state of the microservices and underlying system including resource usage of microservices, the capability of the hardware architecture in DFVS management, the CPU frequency at a chip level or of individual cores if applicable, and the performance of individual microservices. Monitor 230, or components thereof, can be distributed across nodes to monitor the different parts of the nodes running a service. It can also be part of the application platform / orchestrator to retrieve service-related metrics (e.g, microservice CPU usage and performance). In the example of microservice-based applications deployed in Kubernetes clusters, the performance of microservices (e.g., response latency) can be extracted from the traffic performance metric (e.g. by using Prometheus or service mesh Istio sidecar) or from the trace data (e.g., by using OpenTelemetry). Specific resource usages (e.g. CPU, Memory, Storage, Networking) of microservices can be extracted using observability tools (e.g. Metric-server, Prometheus, cAdvisor). Hardware level information (e.g., CPU frequency, NUMA topology, component-level power measurement) can be monitored using for example hardware performance counters such as Intel Performance Monitoring Unit (PMU), Running Average Power Limit (RAPL), etc. The capability of the processor in frequency setting can be inferred from the processor generation of the node where the service is hosted. Monitor 230 and / or its distributed components / sensors can be configured with the necessary granularity, for example, the frequency and duration at which the monitoring data will be collected based on the subscription by the client (i.e., the Performance Estimator, the Energy Estimator, the Resource Allocator).

[0051] Monitor 230 is interfaced with the chain performance estimator 220 and microservice energy estimator 250 for further analysis of the collected data. Monitor 230 collects the raw performance data (e.g., response latency of individual microservices and / or latency between two microservices). To arrive at a chain level performance, chain performance estimator 220 pre-processes the received data, for example, the trace data reported by microservice sidecar proxy or collected by, for example distributed tracing solutions, and constructs an end-to-end performance for the request path. For any request path that has relevant trace data, chainperformance estimator 220 could analyze the duration of the trace and provide statistical performance data (e.g., average and percentile latency) by performing in-memory calculation or directly querying the graph database. It exposes the information for the CPU resource allocator 240 to consume. Since the trace data reflects the flow of requests through the microservices, it can dynamically pinpoint performance characteristics of the microservices based on workload changes. Chain performance estimator 220 also communicates this information to the CPU resource allocator 240 that makes decision by assessing the impact of performance of individual microservices on the performance of service chain.

[0052] Microservice energy estimator 250 handles the estimation of the energy consumption of microservices. The estimation can be done based on software-based methods as microservices run in a shared environment. In some embodiments the estimation can be provided by external components (e.g., using tools such as Scaphandre, Smartwatt, Kepler). In some embodiments, microservice energy estimator 250 can do the estimation based on the metrics extracted from monitor 230. For example, microservice energy estimator 250 can consider the CPU (and CPU frequency), memory, disk and / or network in the EC calculation. It may use different metrics or weights for metrics to estimate the energy consumption of different type of microservices (i.e., compute- / memory- / disk- / network-intensive). For example, for computeintensive services, it can use a model that considers the CPU usages, and CPU frequency in the power estimation. The estimated power usage is sent to the CPU resource allocator 240 when requested.

[0053] CPU resource allocator 240 considers the information retrieved from application modeler 215 (e.g. service chain performance requirement, type, and priority), monitor 230 (e.g., DVFS capability of the nodes’ processor, CPU usage, CPU frequency of cores), chain performance estimator 220 (e.g., performance of microservices and service chain), and microservice energy estimators 250 (e.g., energy consumption of microservices). In case of detecting performance deviations from the expected performance (e.g., latency) requirements, it will decide on new allocation. It can also trigger resource allocation in case of identifying configuration that yields to lower energy compared to the current state while meeting performance requirements. Based on the DVFS capability of the nodes, it makes the CPU frequency and core allocation. For, example if a per-core DFVS is supported, it can decide the frequency of the cores of individual microservices, by determining the minimal frequency and number of (fractional)cores that could fulfil the chain performance requirement and at the same time reduce energy usage. For a per-core DVFS, the cores need to be pinned to the microservices as well. In case of fractional core assignment for a time-critical microservice, the remaining fraction of a core can be assigned to best-effort microservice type by analyzing the impact of collocation in order to facilitate higher utilization. In case of per-chip DVFS, for example, CPU resource allocator 240 could select the lowest frequency for all the cores of a chip and scale the number of cores for individual microservices that run on the chip. This scaling, e.g., the scaling of CPU, will avoid the lower or higher processing power that can result due to applying uniform CPU frequency. Microservice-to-core mapping can also be considered here to avoid two time-critical microservices from competing for the same resources. CPU resource allocator 240 can be seen in some embodiments as comprising two logical components, one is for the combined setting / scaling of cores and frequency (using core and frequency allocator 242) and the second one determines the mapping of microservices to physical cores, which can be based on the topology of the architecture (e.g., using NUMA topology) (using Ms-to-core mapper 244) and sensitivity of microservice to collocation.

[0054] In some embodiments, CPU resource allocator 240 solves an optimization problem that minimizes the energy consumption while meeting the performance (e.g., latency) requirement of a service chain. The optimization problem can in some embodiments be given as:Minimize ECservicechain frequency, core) Subject to.< LotertcyTor iet_servicec}iainEk Xs corejnicroservices < corenode

[0055] To solve this optimization problem, the CPU resource allocator 240 can be implemented using ML techniques such as RL-based methods that incrementally learn from its environment while taking actions. For example, the RL agent can consider metrics such as service chain performance requirement, type, and priority, performance of microservices and service chain and the energy consumption of microservices, the DVFS capability of the nodes’ processors, CPU usage, and CPU frequency of cores to make an action. The agent can learn the impact of the performance of a microservice on the end-to-end performance of the chain and its sensitivity to collocation. It then makes an action that scales the CPU cores and frequency, and the mapping of microservices to physical cores. An example output can be:msID: < CPUX, CPUfrequencyY, physicalCoreZ> where msID, CPUX, CPUfrequencyY, and physicalCoreZ denote the microservice id, the number of CPU cores, the CPU frequency the cores should be running and the (list) of physical cores it should be mapped to respectively. If there is a change in resource allocation, CPU resource allocator 240 communicates with actuator 265 for the application of new resource configurations.

[0056] Actuator 265 can apply the new configuration by instantiating the 1) core scaler 262 for the scaling of CPU for a given microservice, 2) frequency scaler 264 for the scaling of CPU frequency of identified cores, and 3) Ms-to-core mapper 266 for the mapping of microservices to physical cores. Actuator 265 can be located in or comprise a Kubernetes cluster 260.

[0057] In Kubernetes cluster 260, the CRI-RM running in the nodes 280, 290 can be used as a proxy for controlling the resource allocation of the containers. Nodel 280 to nodeN 290 can each comprise one or more chips 288, 298, which can each comprise one or more cores 289, 299. The CPU allocation (by controlling container CPU shares and quota allocations in Uinux Cgroup), the CPU frequency setting (by setting the right CPU power governor and user-controlled CPU frequencies in Uinux power / Frequency control), and (NUMA-aware) CPU mapping (by using Cgroup / cpuset) can be applied by extending the CRI-RM with new policies.

[0058] Figure 3 displays a possible method embodiment under the present disclosure. Component references are made with respect to Figure 2, by way of example only. At step 410, application modeler 215 generates a representation model of the service by considering the information provided by e.g., the application developer / operator regarding the topology of the service (i.e., dependencies of microservices), service type (e.g., time-critical, best-effort) and performance requirements (e.g. latency) of service chains. Step 410 comprises what can generally be referred to as the initial data steps 470.

[0059] At 415, monitor 230 extracts deployment status of the chains’ microservices(e.g., Msl 281, Ms2283, Ms5282, Ms3 284, Ms4286, Ms6285) to identify the nodes (e.g., nodel 280... nodeN 290) hosting the microservices and monitor metrics accordingly. At 420, monitor 230 collects metrics related to the microservices and underlying system including resource usage (e.g., CPU, memory, disk, memory) of microservices, node resource usage (e.g., CPU, memory, disk, memory), NUMA topology of the node, etc. At step 425, monitor 230 extracts the capability of the hardware architecture in DFVS management (e.g., per-core DVFS, per-chip DVFS), thefrequency of the cores microservices are assigned if applicable, power usage of components (e.g., CPU, RAM) from RAPL if available, etc. At 430, monitor 230 scrapes, monitors, or tracks performance metric data (e.g., latency) of individual microservices. This can be done through trace analysis or from the traffic performance of the microservices extracted from traffic management tools such as Istio™. Steps 415 to 430 can generally be referred to as monitoring steps 475.

[0060] At step 435, performance estimator 220 estimates the service chain performance from the monitored microservice performance data received from monitor 230. It provides aggregated statistical performance data at the chain level. The performance estimation logic also profiles the characteristics of the microservice in terms of sensitivity to collocation with other microservice running on the same cores and / or variabilities of performance of microservice overtime. At 440, energy estimator 250 estimates the energy consumption of microservices based on the metrics extracted from monitor 230. Using the metrics extracted about the microservice regarding their resource usage, energy estimator 250 can consider part or all of CPU (and CPU frequency), memory, disk and network resources in the EC calculation. Steps 435-440 can generally be referred to as the analysis steps 495.

[0061] At step 445, based on the monitored and analyzed information, resource allocator 240 decides the number of cores, which can be fractional cores, to be allocated for each microservice. This is done in coordination with CPU frequency setting. At 450, the CPU frequency scaling is performed (e.g., by resource allocator 240 through actuator 265) based on the DVFS capability of the hardware. For, example if a per-core DFVS is supported, it can decide the frequency of the cores of individual microservices, by determining the minimal frequency and number of (fractional) cores that could fulfil the chain performance requirement and reduce energy usage. In case of per-chip DVFS, resource allocator 240 would select the lowest frequency for all the cores of a chip and scale the number of cores for individual microservices that run on the chip to fulfil requirements. In case of fractional core assignment for microservice of a time-critical service chain, the remaining fraction of a core can be assigned to best-effort microservice type to facilitate higher utilization, where the later can be throttled down easily at times of resource shortage. To facilitate isolation, resource allocator 240 at step 455, also performs mapping of microservices to cores based on for example NUMA topology of the node to allow the assignment of CPU cores for the microservice (and / or dependent microservices) from the same NUMA node. Steps 445 to 455 can generally be referred to as the decision-making steps 490.

[0062] At step 460, actuator 265 (e.g., CRI-RM) applies the new configuration by instantiating the core scaler 262, the CPU frequency scaler 264, and the Ms-to-core mapper 266. Step 460 can generally be referred to as the actuation steps 485.

[0063] At 465, the monitoring 475, analysis 495, and decision making 490 is performed continuously in a closed-loop manner to ensure improved energy performance. Step 465 can generally be referred to as the closed-loop steps 480.

[0064] Figure 4 displays another possible method embodiment 600 under the present disclosure. Method 600 comprises a method for monitoring or optimizing microservice performance. At step 601, a service description is created by a service provider 620 and received by application modeler 625. The service provider 620 of the service preferably creates the service description, that can be described using modeling language such as TOSCA. The description can illustrate the dependencies between microservices, type (e.g., time-critical / best-effort, CPU- / memory- / disk / network-bound) and performance requirements (e.g. latency) of service chains. The service description is sent to application modeler 625, that then creates representation model of the service in a usable form.

[0065] At 602, application modeler 625 generates an application or representation model for the service. Application modeler 625 can generate a representation model of the service by extracting the information stored in the service description. For easy manipulation of dependencies among microservices, storage systems such as graph database can be used to store relationships between the microservices and other properties such as service type and performance requirements.

[0066] At 603, microservice energy estimator 630, CPU resource allocator 635, and chain performance estimator 640 subscribe to monitor data for monitor 645. These components preferably subscribe to monitor 645 with proper frequency and duration at which the monitoring data should be collected.

[0067] At 604, as part of continuous performance monitoring, e.g., as begun in step 603, a component, e.g., CPU resource allocator 635, detects a performance violation (e.g., latency) of a service chain.

[0068] At 605a / b, CPU resource allocator 635 sends requests to the chain performance estimator 640 and microservice energy estimator 630 to receive performance and energy data (e.g. current performance and energy usage of microservices / service chain).

[0069] At 606, CPU resource allocator 635, chain performance estimator 640, and microservice energy estimator 630 communicate with the monitor 645 to query data such as deployment status (to identify the nodes the microservices are running), NUMA topology of the nodes, resource (e.g., CPU, memory, disk, network) usage, CPU frequency of the cores used by microservices of chains, etc.

[0070] At 607, to estimate performance (e.g., latency) of the chain, the chain performance estimator 640 also extracts performance of individual microservices from monitor 645.

[0071] At 608, chain performance estimator then pre-processes the received performance data from monitor 645, by performing aggregation (e.g., through in-memory calculation or directly querying a graph database if used as storage system) to arrive at a chain level performance data. It then sends the chain and microservice performance data to the CPU resource allocator. The CPU frequency scaling is performed based on the DVFS capability of the hardware. For example, if a per-core DFVS is supported, it can decide the frequency of the cores of individual microservices. In case of per-chip DVFS, CPU resource allocator 635 would select the lowest frequency for all the cores of a chip and scale the number of cores for individual microservices that run on the chip to fulfil requirements.

[0072] At 609, microservice energy estimator 630 handles the estimation of the energy consumption of microservices. The estimation can be provided by external components (e.g., using tools such as Scaphandre, Smartwatt, Kepler) or microservice energy estimator 630 can do the actual estimation based on the metrics (e.g., CPU / CPU frequency, memory, disk, network usage, power consumption of components such as CPU, RAM ) extracted from monitor 645, using for example regression methods.

[0073] At 610, CPU resource allocator 635 queries the DVFS capabilities of nodes (e.g., per-core DVFS, per-chip DVFS) hosting microservices to identify which optimization option to use.

[0074] At 611a / b / c, based on e.g., the monitored and analyzed information from application modeler 625 (e.g. service chain performance requirement, type, and priority), monitor 645 (e.g., DVFS capability of the nodes’ processor, NUMA topology, resource usage, CPU frequency of cores), chain performance estimator 640 (e.g., performance of microservices and service chain) and microservice energy estimator 630 (e.g., energy consumption of microservices),CPU resource allocator 635 decides to scale the number of cores, CPU frequency and the mapping of ms to physical cores so as to meet the performance requirement with minimal energy usage. For example, at 61 la, CPU resource allocator 635 decides the number of cores, which can be fractional cores, to be allocated for each microservices. This can be done in coordination with CPU frequency setting. At 611b, in case a per-core DFVS is supported, CPU resource allocator 635 can decide the frequency of the cores of individual microservices, by determining the minimal frequency and number of (fractional) cores that could fulfil the chain performance requirement and reduce energy usage. In case of per-chip DVFS, CPU resource allocator 635 would select the lowest frequency for all the cores of a chip and scale the number of cores for individual microservices that run on the chip to fulfil requirements. At 611c, to facilitate isolation, CPU resource allocator 635 can also perform mapping of microservices to cores based on for example NUMA topology of the node to allow the assignment of CPU cores for the microservice (and / or dependent microservices) from the same NUMA node. In case of fractional core assignment, it makes sure that no two time-critical microservices are assigned the same core. However, a best-effort microservice can be collocated with the time-critical microservice to facilitate higher utilization, where the former can be throttled down easily at times of resource shortage.

[0075] At 612, CPU resource allocator sends the new resource configurations to actuator 650 for the application of the configurations.

[0076] At 613, actuator 650 generates templates in a format used by the cloud orchestrator (e.g., K8s CRI RM policies) for the new resource configurations. Step 613 could comprise e.g., steps 613a / b / c. For example, at 613a, actuator 650 generates a CPU scaling policy for the scaling of cores of microservices. At 613b, actuator 650 generates a CPU frequency scaling policy for the scaling of CPU frequency of cores used by microservices. At 613c, actuator 650 generates a CPU pinning policy for the mapping of microservices to specific physical cores.

[0077] At 614, actuator 650 sends the policies to cloud orchestrator 655 for their execution.

[0078] At 615, cloud orchestrator 655 executes e.g., CPU scaling, CPU frequency scaling, and CPU pinning policies. For example in CRI-RM enabled K8s cluster, the CRI-RM can be extended with new policies for CPU allocation (by controlling container CPU shares and quota allocations in Linux Cgroup), the CPU frequency setting (by setting the right CPU power governor 1and user-controlled CPU frequencies in Linux power / Frequency control), and (NUMA-aware) CPU mapping (by using Cgroup / cpuset).Additional Embodiments

[0079] Figure 5 shows an example of a communication system 2100 in accordance with some embodiments. In the example, the communication system 2100 includes a telecommunication network 2102 that includes an access network 2104, such as a RAN, and a core network 2106, which includes one or more core network nodes 2108. The access network 2104 includes one or more access network nodes, such as network nodes 2110a and 2110b (one or more of which may be generally referred to as network nodes 2110), or any other similar 3rd Generation Partnership Project (3GPP) access node or non-3GPP access point. The network nodes 2110 facilitate direct or indirect connection of UE, such as by connecting UEs 2112a, 2112b, 2112c, and 2112d (one or more of which may be generally referred to as UEs 2112) to the core network 2106 over one or more wireless connections.

[0080] Example wireless communications over a wireless connection include transmitting and / or receiving wireless signals using electromagnetic waves, radio waves, infrared waves, and / or other types of signals suitable for conveying information without the use of wires, cables, or other material conductors. Moreover, in different embodiments, the communication system 1100 may include any number of wired or wireless networks, network nodes, UEs, and / or any other components or systems that may facilitate or participate in the communication of data and / or signals whether via wired or wireless connections. The communication system 2100 may include and / or interface with any type of communication, telecommunication, data, cellular, radio network, and / or other similar type of system.

[0081] The UEs 2112 may be any of a wide variety of communication devices, including wireless devices arranged, configured, and / or operable to communicate wirelessly with the network nodes 2110 and other communication devices. Similarly, the network nodes 2110 are arranged, capable, configured, and / or operable to communicate directly or indirectly with the UEs 2112 and / or with other network nodes or equipment in the telecommunication network 2102 to enable and / or provide network access, such as wireless network access, and / or to perform other functions, such as administration in the telecommunication network 2102.

[0082] In the depicted example, the core network 2106 connects the network nodes 2110 to one or more hosts, such as host 2116. These connections may be direct or indirect via one or more intermediary networks or devices. In other examples, network nodes may be directly coupled to hosts. The core network 2106 includes one more core network nodes (e.g., core network node 2108) that are structured with hardware and software components. Features of these components may be substantially similar to those described with respect to the UEs, network nodes, and / or hosts, such that the descriptions thereof are generally applicable to the corresponding components of the core network node 2108. Example core network nodes include functions of one or more of a Mobile Switching Center (MSC), Mobility Management Entity (MME), Home Subscriber Server (HSS), Access and Mobility Management Function (AMF), Session Management Function (SMF), Authentication Server Function (AUSF), Subscription Identifier De-concealing function (SIDF), Unified Data Management (UDM), Security Edge Protection Proxy (SEPP), Network Exposure Function (NEF), and / or a User Plane Function (UPF).

[0083] The host 2116 may be under the ownership or control of a service provider other than an operator or provider of the access network 2104 and / or the telecommunication network 2102, and may be operated by the service provider or on behalf of the service provider. The host 2116 may host a variety of applications to provide one or more service. Examples of such applications include live and pre-recorded audio / video content, data collection services such as retrieving and compiling data on various ambient conditions detected by a plurality of UEs, analytics functionality, social media, functions for controlling or otherwise interacting with remote devices, functions for an alarm and surveillance center, or any other such function performed by a server.

[0084] As a whole, the communication system 2100 of Figure 5 enables connectivity between the UEs, network nodes, and hosts. In that sense, the communication system may be configured to operate according to predefined rules or procedures, such as specific standards that include, but are not limited to: Global System for Mobile Communications (GSM); Universal Mobile Telecommunications System (UMTS); Long Term Evolution (LTE), and / or other suitable 2G, 3G, 4G, 5G standards, or any applicable future generation standard (e.g., 6G); wireless local area network (WLAN) standards, such as the Institute of Electrical and Electronics Engineers (IEEE) 802.11 standards (WiFi); and / or any other appropriate wireless communication standard, such as the Worldwide Interoperability for Microwave Access (WiMax), Bluetooth, Z-Wave, Near Field Communication (NFC) ZigBee, LiFi, and / or any low-power wide-area network (LPWAN) standards such as LoRa and Sigfox.

[0085] In some examples, the telecommunication network 2102 is a cellular network that implements 3GPP standardized features. Accordingly, the telecommunications network 2102 may support network slicing to provide different logical networks to different devices that are connected to the telecommunication network 2102. For example, the telecommunications network 2102 may provide Ultra Reliable Low Latency Communication (URLLC) services to some UEs, while providing Enhanced Mobile Broadband (eMBB) services to other UEs, and / or Massive Machine Type Communication (mMTC)ZMassive loT services to yet further UEs.

[0086] In some examples, the UEs 2112 are configured to transmit and / or receive information without direct human interaction. For instance, a UE may be designed to transmit information to the access network 2104 on a predetermined schedule, when triggered by an internal or external event, or in response to requests from the access network 2104. Additionally, a UE may be configured for operating in single- or multi-RAT or multi-standard mode. For example, a UE may operate with any one or combination of Wi-Fi, NR (New Radio) and LTE, i.e. being configured for multi-radio dual connectivity (MR-DC), such as E-UTRAN (Evolved-UMTS Terrestrial Radio Access Network) New Radio - Dual Connectivity (EN-DC).

[0087] In the example, the hub 2114 communicates with the access network 2104 to facilitate indirect communication between one or more UEs (e.g., UE 2112c and / or 2112d) and network nodes (e.g., network node 2110b). In some examples, the hub 2114 may be a controller, router, content source and analytics, or any of the other communication devices described herein regarding UEs. For example, the hub 2114 may be a broadband router enabling access to the core network 2106 for the UEs. As another example, the hub 2114 may be a controller that sends commands or instructions to one or more actuators in the UEs. Commands or instructions may be received from the UEs, network nodes 2110, or by executable code, script, process, or other instructions in the hub 2114. As another example, the hub 2114 may be a data collector that acts as temporary storage for UE data and, in some embodiments, may perform analysis or other processing of the data. As another example, the hub 2114 may be a content source. For example, for a UE that is a VR headset, display, loudspeaker or other media delivery device, the hub 2114 may retrieve VR assets, video, audio, or other media or data related to sensory information via anetwork node, which the hub 2114 then provides to the UE either directly, after performing local processing, and / or after adding additional local content. In still another example, the hub 2114 acts as a proxy server or orchestrator for the UEs, in particular in if one or more of the UEs are low energy loT devices.

[0088] The hub 2114 may have a constant / persistent or intermittent connection to the network node 2110b. The hub 2114 may also allow for a different communication scheme and / or schedule between the hub 2114 and UEs (e.g., UE 2112c and / or 2112d), and between the hub 2114 and the core network 2106. In other examples, the hub 2114 is connected to the core network 2106 and / or one or more UEs via a wired connection. Moreover, the hub 2114 may be configured to connect to an M2M service provider over the access network 1104 and / or to another UE over a direct connection. In some scenarios, UEs may establish a wireless connection with the network nodes 2110 while still connected via the hub 2114 via a wired or wireless connection. In some embodiments, the hub 2114 may be a dedicated hub - that is, a hub whose primary function is to route communications to / from the UEs from / to the network node 2110b. In other embodiments, the hub 2114 may be a non-dedicated hub - that is, a device which is capable of operating to route communications between the UEs and network node 2110b, but which is additionally capable of operating as a communication start and / or end point for certain data channels.

[0089] Figures 6-9 show possible UE 2200, network node 2300, host 3300, and virtualization embodiments 4400 under the present disclosure. In various embodiments, any of these components, or portions thereof, and / or combinations thereof, can comprise e.g., system 10 or 200 or portions thereof. In addition, UE 2200, network node 2300, host 3300, and virtualization embodiments 4400 or portions thereof, and / or combinations thereof, can perform together or separately, the steps of possible method embodiments described herein, such as methods 400 or 600.

[0090] Figure 6 shows a UE 2200 in accordance with some embodiments. As used herein, a UE refers to a device capable, configured, arranged and / or operable to communicate wirelessly with network nodes and / or other UEs. Examples of a UE include, but are not limited to, a smart phone, mobile phone, cell phone, voice over IP (VoIP) phone, wireless local loop phone, desktop computer, personal digital assistant (PDA), wireless cameras, gaming console or device, music storage device, playback appliance, wearable terminal device, wireless endpoint, mobilestation, tablet, laptop, laptop-embedded equipment (LEE), laptop-mounted equipment (LME), smart device, wireless customer-premise equipment (CPE), vehicle-mounted or vehicle embedded / integrated wireless device, etc. Other examples include any UE identified by the 3rd Generation Partnership Project (3GPP), including a narrow band internet of things (NB-IoT) UE, a machine type communication (MTC) UE, and / or an enhanced MTC (eMTC) UE.

[0091] A UE may support device-to-device (D2D) communication, for example by implementing a 3 GPP standard for sidelink communication, Dedicated Short-Range Communication (DSRC), vehicle-to-vehicle (V2V), vehicle-to-infrastructure (V2I), or vehicle-to- everything (V2X). In other examples, a UE may not necessarily have a user in the sense of a human user who owns and / or operates the relevant device. Instead, a UE may represent a device that is intended for sale to, or operation by, a human user but which may not, or which may not initially, be associated with a specific human user (e.g., a smart sprinkler controller). Alternatively, a UE may represent a device that is not intended for sale to, or operation by, an end user but which may be associated with or operated for the benefit of a user (e.g., a smart power meter).

[0092] The UE 2200 includes processing circuitry 2202 that is operatively coupled via a bus 2204 to an input / output interface 2206, a power source 2208, a memory 2210, a communication interface 2212, and / or any other component, or any combination thereof. Certain UEs may utilize all or a subset of the components shown in Figure 10. The level of integration between the components may vary from one UE to another UE. Further, certain UEs may contain multiple instances of a component, such as multiple processors, memories, transceivers, transmitters, receivers, etc.

[0093] The processing circuitry 2202 is configured to process instructions and data and may be configured to implement any sequential state machine operative to execute instructions stored as machine- readable computer programs in the memory 2210. The processing circuitry 2202 may be implemented as one or more hardware-implemented state machines (e.g., in discrete logic, field-programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), etc.); programmable logic together with appropriate firmware; one or more stored computer programs, general-purpose processors, such as a microprocessor or digital signal processor (DSP), together with appropriate software; or any combination of the above. For example, the processing circuitry 2202 may include multiple central processing units (CPUs).

[0094] In the example, the input / output interface 2206 may be configured to provide an interface or interfaces to an input device, output device, or one or more input and / or output devices. Examples of an output device include a speaker, a sound card, a video card, a display, a monitor, a printer, an actuator, an emitter, a smartcard, another output device, or any combination thereof. An input device may allow a user to capture information into the UE 2200. Examples of an input device include a touch-sensitive or presence-sensitive display, a camera (e.g., a digital camera, a digital video camera, a web camera, etc.), a microphone, a sensor, a mouse, a trackball, a directional pad, a trackpad, a scroll wheel, a smartcard, and the like. The presencesensitive display may include a capacitive or resistive touch sensor to sense input from a user. A sensor may be, for instance, an accelerometer, a gyroscope, a tilt sensor, a force sensor, a magnetometer, an optical sensor, a proximity sensor, a biometric sensor, etc., or any combination thereof. An output device may use the same type of interface port as an input device. For example, a Universal Serial Bus (USB) port may be used to provide an input device and an output device.

[0095] In some embodiments, the power source 2208 is structured as a battery or battery pack. Other types of power sources, such as an external power source (e.g., an electricity outlet), photovoltaic device, or power cell, may be used. The power source 2208 may further include power circuitry for delivering power from the power source 2208 itself, and / or an external power source, to the various parts of the UE 2200 via input circuitry or an interface such as an electrical power cable. Delivering power may be, for example, for charging of the power source 2208. Power circuitry may perform any formatting, converting, or other modification to the power from the power source 2208 to make the power suitable for the respective components of the UE 2200 to which power is supplied.

[0096] The memory 2210 may be or be configured to include memory such as random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic disks, optical disks, hard disks, removable cartridges, flash drives, and so forth. In one example, the memory 2210 includes one or more application programs 2214, such as an operating system, web browser application, a widget, gadget engine, or other application, and corresponding data 2216. The memory 2210 may store, for use by the UE 2200, any of a variety of various operating systems or combinations of operating systems.

[0097] The memory 2210 may be configured to include a number of physical drive units, such as redundant array of independent disks (RAID), flash memory, USB flash drive, external hard disk drive, thumb drive, pen drive, key drive, high-density digital versatile disc (HD- DVD) optical disc drive, internal hard disk drive, Blu-Ray optical disc drive, holographic digital data storage (HDDS) optical disc drive, external mini-dual in-line memory module (DIMM), synchronous dynamic random access memory (SDRAM), external micro-DIMM SDRAM, smartcard memory such as tamper resistant module in the form of a universal integrated circuit card (UICC) including one or more subscriber identity modules (SIMs), such as a USIM and / or ISIM, other memory, or any combination thereof. The UICC may for example be an embedded UICC (eUICC), integrated UICC (iUICC) or a removable UICC commonly known as ‘SIM card.’ The memory 2210 may allow the UE 2200 to access instructions, application programs and the like, stored on transitory or non-transitory memory media, to off-load data, or to upload data. An article of manufacture, such as one utilizing a communication system may be tangibly embodied as or in the memory 2210, which may be or comprise a device-readable storage medium.

[0098] The processing circuitry 2202 may be configured to communicate with an access network or other network using the communication interface 2212. The communication interface 2212 may comprise one or more communication subsystems and may include or be communicatively coupled to an antenna 2222. The communication interface 2212 may include one or more transceivers used to communicate, such as by communicating with one or more remote transceivers of another device capable of wireless communication (e.g., another UE or a network node in an access network). Each transceiver may include a transmitter 2218 and / or a receiver 2220 appropriate to provide network communications (e.g., optical, electrical, frequency allocations, and so forth). Moreover, the transmitter 2218 and receiver 2220 may be coupled to one or more antennas (e.g., antenna 2222) and may share circuit components, software or firmware, or alternatively be implemented separately.

[0099] In the illustrated embodiment, communication functions of the communication interface 2212 may include cellular communication, Wi-Fi communication, LPWAN communication, data communication, voice communication, multimedia communication, short-range communications such as Bluetooth, near-field communication, location-based communication such as the use of the global positioning system (GPS) to determine a location, another like communication function, or any combination thereof. Communicationsmay be implemented in according to one or more communication protocols and / or standards, such as IEEE 802.11, Code Division Multiplexing Access (CDMA), Wideband Code Division Multiple Access (WCDMA), GSM, LIE, New Radio (NR), UMTS, WiMax, Ethernet, transmission control protocol / internet protocol (TCP / IP), synchronous optical networking (SONET), Asynchronous Transfer Mode (ATM), QUIC, Hypertext Transfer Protocol (HTTP), and so forth.[000100] Regardless of the type of sensor, a UE may provide an output of data captured by its sensors, through its communication interface 2212, via a wireless connection to a network node. Data captured by sensors of a UE can be communicated through a wireless connection to a network node via another UE. The output may be periodic (e.g., once every 15 minutes if it reports the sensed temperature), random (e.g., to even out the load from reporting from several sensors), in response to a triggering event (e.g., when moisture is detected an alert is sent), in response to a request (e.g., a user initiated request), or a continuous stream (e.g., a live video feed of a patient).[000101] As another example, a UE comprises an actuator, a motor, or a switch, related to a communication interface configured to receive wireless input from a network node via a wireless connection. In response to the received wireless input the states of the actuator, the motor, or the switch may change. For example, the UE may comprise a motor that adjusts the control surfaces or rotors of a drone in flight according to the received input or to a robotic arm performing a medical procedure according to the received input.[000102] A UE, when in the form of an Internet of Things (loT) device, may be a device for use in one or more application domains, these domains comprising, but not limited to, city wearable technology, extended industrial application and healthcare. Non-limiting examples of such an loT device are a device which is or which is embedded in: a connected refrigerator or freezer, a TV, a connected lighting device, an electricity meter, a robot vacuum cleaner, a voice controlled smart speaker, a home security camera, a motion detector, a thermostat, a smoke detector, a door / window sensor, a flood / moisture sensor, an electrical door lock, a connected doorbell, an air conditioning system like a heat pump, an autonomous vehicle, a surveillance system, a weather monitoring device, a vehicle parking monitoring device, an electric vehicle charging station, a smart watch, a fitness tracker, a head-mounted display for Augmented Reality (AR) or Virtual Reality (VR), a wearable for tactile augmentation or sensory enhancement, a water sprinkler, an animal- or item-tracking device, a sensor for monitoring a plant or animal, anindustrial robot, an Unmanned Aerial Vehicle (UAV), and any kind of medical device, like a heart rate monitor or a remote controlled surgical robot. A UE in the form of an loT device comprises circuitry and / or software in dependence of the intended application of the loT device in addition to other components as described in relation to the UE 2200 shown in Figure 10.[000103] As yet another specific example, in an loT scenario, a UE may represent a machine or other device that performs monitoring and / or measurements, and transmits the results of such monitoring and / or measurements to another UE and / or a network node. The UE may in this case be an M2M device, which may in a 3 GPP context be referred to as an MTC device. As one particular example, the UE may implement the 3 GPP NB-IoT standard. In other scenarios, a UE may represent a vehicle, such as a car, a bus, a truck, a ship and an airplane, or other equipment that is capable of monitoring and / or reporting on its operational status or other functions associated with its operation.[000104] In practice, any number of UEs may be used together with respect to a single use case. For example, a first UE might be or be integrated in a drone and provide the drone’s speed information (obtained through a speed sensor) to a second UE that is a remote controller operating the drone. When the user makes changes from the remote controller, the first UE may adjust the throttle on the drone (e.g. by controlling an actuator) to increase or decrease the drone’s speed. The first and / or the second UE can also include more than one of the functionalities described above. For example, a UE might comprise the sensor and the actuator, and handle communication of data for both the speed sensor and the actuators.[000105] Figure 7 shows a network node 3300 in accordance with some embodiments. As used herein, network node refers to equipment capable, configured, arranged and / or operable to communicate directly or indirectly with a UE and / or with other network nodes or equipment, in a telecommunication network. Examples of network nodes include, but are not limited to, access points (APs) (e.g., radio access points), base stations (BSs) (e.g., radio base stations, Node Bs, evolved Node Bs (eNBs) and NR NodeBs (gNBs)).[000106] Base stations may be categorized based on the amount of coverage they provide (or, stated differently, their transmit power level) and so, depending on the provided amount of coverage, may be referred to as femto base stations, pico base stations, micro base stations, or macro base stations. A base station may be a relay node or a relay donor node controlling a relay. A network node may also include one or more (or all) parts of a distributedradio base station such as centralized digital units and / or remote radio units (RRUs), sometimes referred to as Remote Radio Heads (RRHs). Such remote radio units may or may not be integrated with an antenna as an antenna integrated radio. Parts of a distributed radio base station may also be referred to as nodes in a distributed antenna system (DAS).[000107] Other examples of network nodes include multiple transmission point (multi-TRP) 5G access nodes, multi-standard radio (MSR) equipment such as MSRBSs, network controllers such as radio network controllers (RNCs) or base station controllers (BSCs), base transceiver stations (BTSs), transmission points, transmission nodes, multi-cell / multicast coordination entities (MCEs), Operation and Maintenance (O&M) nodes, Operations Support System (OSS) nodes, Self-Organizing Network (SON) nodes, positioning nodes (e.g., Evolved Serving Mobile Location Centers (E-SMLCs)), and / or Minimization of Drive Tests (MDTs).[000108] The network node 3300 includes a processing circuitry 3302, a memory 3304, a communication interface 3306, and a power source 3308. The network node 3300 may be composed of multiple physically separate components (e.g., a NodeB component and a RNC component, or a BTS component and a BSC component, etc.), which may each have their own respective components. In certain scenarios in which the network node 3300 comprises multiple separate components (e.g., BTS and BSC components), one or more of the separate components may be shared among several network nodes. For example, a single RNC may control multiple NodeBs. In such a scenario, each unique NodeB and RNC pair, may in some instances be considered a single separate network node. In some embodiments, the network node 1300 may be configured to support multiple radio access technologies (RATs). In such embodiments, some components may be duplicated (e.g., separate memory 3304 for different RATs) and some components may be reused (e.g., a same antenna 3310 may be shared by different RATs). The network node 3300 may also include multiple sets of the various illustrated components for different wireless technologies integrated into network node 1300, for example GSM, WCDMA, LTE, NR, WiFi, Zigbee, Z-wave, LoRaWAN, Radio Frequency Identification (RFID) or Bluetooth wireless technologies. These wireless technologies may be integrated into the same or different chip or set of chips and other components within network node 1300.[000109] The processing circuitry 3302 may comprise a combination of one or more of a microprocessor, controller, microcontroller, central processing unit, digital signal processor, application-specific integrated circuit, field programmable gate array, or any other suitablecomputing device, resource, or combination of hardware, software and / or encoded logic operable to provide, either alone or in conjunction with other network node 3300 components, such as the memory 3304, to provide network node 3300 functionality.[000110] In some embodiments, the processing circuitry 3302 includes a system on a chip (SOC). In some embodiments, the processing circuitry 3302 includes one or more of radio frequency (RF) transceiver circuitry 3312 and baseband processing circuitry 3314. In some embodiments, the radio frequency (RF) transceiver circuitry 3312 and the baseband processing circuitry 3314 may be on separate chips (or sets of chips), boards, or units, such as radio units and digital units. In alternative embodiments, part or all of RF transceiver circuitry 3312 and baseband processing circuitry 3314 may be on the same chip or set of chips, boards, or units.[000111] The memory 3304 may comprise any form of volatile or non-volatile computer-readable memory including, without limitation, persistent storage, solid-state memory, remotely mounted memory, magnetic media, optical media, random access memory (RAM), readonly memory (ROM), mass storage media (for example, a hard disk), removable storage media (for example, a flash drive, a Compact Disk (CD) or a Digital Video Disk (DVD)), and / or any other volatile or non-volatile, non-transitory device-readable and / or computer-executable memory devices that store information, data, and / or instructions that may be used by the processing circuitry 3302. The memory 3304 may store any suitable instructions, data, or information, including a computer program, software, an application including one or more of logic, rules, code, tables, and / or other instructions capable of being executed by the processing circuitry 3302 and utilized by the network node 3300. The memory 3304 may be used to store any calculations made by the processing circuitry 3302 and / or any data received via the communication interface 3306. In some embodiments, the processing circuitry 3302 and memory 3304 is integrated.[000112] The communication interface 3306 is used in wired or wireless communication of signaling and / or data between a network node, access network, and / or UE. As illustrated, the communication interface 3306 comprises port(s) / terminal(s) 3316 to send and receive data, for example to and from a network over a wired connection. The communication interface 3306 also includes radio front-end circuitry 3318 that may be coupled to, or in certain embodiments a part of, the antenna 3310. Radio front-end circuitry 3318 comprises filters 3320 and amplifiers 3322. The radio front-end circuitry 3318 may be connected to an antenna 3310 and processing circuitry 3302. The radio front-end circuitry may be configured to condition signalscommunicated between antenna 3310 and processing circuitry 3302. The radio front-end circuitry 3318 may receive digital data that is to be sent out to other network nodes or UEs via a wireless connection. The radio front-end circuitry 3318 may convert the digital data into a radio signal having the appropriate channel and bandwidth parameters using a combination of filters 3320 and / or amplifiers 3322. The radio signal may then be transmitted via the antenna 3310. Similarly, when receiving data, the antenna 3310 may collect radio signals which are then converted into digital data by the radio front-end circuitry 3318. The digital data may be passed to the processing circuitry 3302. In other embodiments, the communication interface may comprise different components and / or different combinations of components.[000113] In certain alternative embodiments, the network node 3300 does not include separate radio front-end circuitry 3318, instead, the processing circuitry 3302 includes radio frontend circuitry and is connected to the antenna 3310. Similarly, in some embodiments, all or some of the RF transceiver circuitry 3312 is part of the communication interface 3306. In still other embodiments, the communication interface 3306 includes one or more ports or terminals 3316, the radio front-end circuitry 3318, and the RF transceiver circuitry 3312, as part of a radio unit (not shown), and the communication interface 3306 communicates with the baseband processing circuitry 3314, which is part of a digital unit (not shown).[000114] The antenna 3310 may include one or more antennas, or antenna arrays, configured to send and / or receive wireless signals. The antenna 3310 may be coupled to the radio front-end circuitry 3318 and may be any type of antenna capable of transmitting and receiving data and / or signals wirelessly. In certain embodiments, the antenna 3310 is separate from the network node 3300 and connectable to the network node 3300 through an interface or port.[000115] The antenna 3310, communication interface 3306, and / or the processing circuitry 3302 may be configured to perform any receiving operations and / or certain obtaining operations described herein as being performed by the network node. Any information, data and / or signals may be received from a UE, another network node and / or any other network equipment. Similarly, the antenna 3310, the communication interface 3306, and / or the processing circuitry 3302 may be configured to perform any transmitting operations described herein as being performed by the network node. Any information, data and / or signals may be transmitted to a UE, another network node and / or any other network equipment.[000116] The power source 3308 provides power to the various components of network node 3300 in a form suitable for the respective components (e.g., at a voltage and current level needed for each respective component). The power source 3308 may further comprise, or be coupled to, power management circuitry to supply the components of the network node 3300 with power for performing the functionality described herein. For example, the network node 3300 may be connectable to an external power source (e.g., the power grid, an electricity outlet) via an input circuitry or interface such as an electrical cable, whereby the external power source supplies power to power circuitry of the power source 3308. As a further example, the power source 3308 may comprise a source of power in the form of a battery or battery pack which is connected to, or integrated in, power circuitry. The battery may provide backup power should the external power source fail.[000117] Embodiments of the network node 3300 may include additional components beyond those shown in Figure 7 for providing certain aspects of the network node’s functionality, including any of the functionality described herein and / or any functionality necessary to support the subject matter described herein. For example, the network node 3300 may include user interface equipment to allow input of information into the network node 3300 and to allow output of information from the network node 3300. This may allow a user to perform diagnostic, maintenance, repair, and other administrative functions for the network node 3300.[000118] Figure 8 is a block diagram of a host 4400, which may be an embodiment of the host 2116 of Figure 5, in accordance with various aspects described herein. As used herein, the host 4400 may be or comprise various combinations hardware and / or software, including a standalone server, a blade server, a cloud-implemented server, a distributed server, a virtual machine, container, or processing resources in a server farm. The host 4400 may provide one or more services to one or more UEs.[000119] The host 4400 includes processing circuitry 4402 that is operatively coupled via a bus 4404 to an input / output interface 4406, a network interface 4408, a power source 4410, and a memory 4412. Other components may be included in other embodiments. Features of these components may be substantially similar to those described with respect to the devices of previous figures, such as Figures 6 and 7, such that the descriptions thereof are generally applicable to the corresponding components of host 4400.[000120] The memory 4412 may include one or more computer programs including one or more host application programs 4414 and data 4416, which may include user data, e.g., data generated by a UE for the host 4400 or data generated by the host 4400 for a UE. Embodiments of the host 4400 may utilize only a subset or all of the components shown. The host application programs 4414 may be implemented in a container-based architecture and may provide support for video codecs (e.g., Versatile Video Coding (WC), High Efficiency Video Coding (HEVC), Advanced Video Coding (AVC), MPEG, VP9) and audio codecs (e.g., FLAC, Advanced Audio Coding (AAC), MPEG, G.711), including transcoding for multiple different classes, types, or implementations of UEs (e.g., handsets, desktop computers, wearable display systems, heads-up display systems). The host application programs 4414 may also provide for user authentication and licensing checks and may periodically report health, routes, and content availability to a central node, such as a device in or on the edge of a core network. Accordingly, the host 4400 may select and / or indicate a different host for over-the-top services for a UE. The host application programs 4414 may support various protocols, such as the HTTP Live Streaming (HLS) protocol, Real-Time Messaging Protocol (RTMP), Real-Time Streaming Protocol (RTSP), Dynamic Adaptive Streaming over HTTP (MPEG-DASH), etc.[000121] Figure 9 is a block diagram illustrating a virtualization environment 5500 in which functions implemented by some embodiments may be virtualized. In the present context, virtualizing means creating virtual versions of apparatuses or devices which may include virtualizing hardware platforms, storage devices and networking resources. As used herein, virtualization can be applied to any device described herein, or components thereof, and relates to an implementation in which at least a portion of the functionality is implemented as one or more virtual components. Some or all of the functions described herein may be implemented as virtual components executed by one or more virtual machines (VMs) implemented in one or more virtual environments 5500 hosted by one or more of hardware nodes, such as a hardware computing device that operates as a network node, UE, core network node, or host. Further, in embodiments in which the virtual node does not require radio connectivity (e.g., a core network node or host), then the node may be entirely virtualized.[000122] Applications 5502 (which may alternatively be called software instances, virtual appliances, network functions, virtual nodes, virtual network functions, etc.) are run in thevirtualization environment 5500 to implement some of the features, functions, and / or benefits of some of the embodiments disclosed herein.[000123] Hardware 5504 includes processing circuitry, memory that stores software and / or instructions executable by hardware processing circuitry, and / or other hardware devices as described herein, such as a network interface, input / output interface, and so forth. Software may be executed by the processing circuitry to instantiate one or more virtualization layers 5506 (also referred to as hypervisors or virtual machine monitors (VMMs)), provide VMs 5508a and 5508b (one or more of which may be generally referred to as VMs 5508), and / or perform any of the functions, features and / or benefits described in relation with some embodiments described herein. The virtualization layer 5506 may present a virtual operating platform that appears like networking hardware to the VMs 5508.[000124] The VMs 5508 comprise virtual processing, virtual memory, virtual networking or interface and virtual storage, and may be run by a corresponding virtualization layer 5506. Different embodiments of the instance of a virtual appliance 5502 may be implemented on one or more of VMs 5508, and the implementations may be made in different ways. Virtualization of the hardware is in some contexts referred to as network function virtualization (NFV). NFV may be used to consolidate many network equipment types onto industry standard high volume server hardware, physical switches, and physical storage, which can be located in data centers, and customer premise equipment.[000125] In the context of NFV, a VM 5508 may be a software implementation of a physical machine that runs programs as if they were executing on a physical, non-virtualized machine. Each of the VMs 5508, and that part of hardware 5504 that executes that VM, be it hardware dedicated to that VM and / or hardware shared by that VM with others of the VMs, forms separate virtual network elements. Still in the context of NFV, a virtual network function is responsible for handling specific network functions that run in one or more VMs 5508 on top of the hardware 5504 and corresponds to the application 5502.[000126] Hardware 5504 may be implemented in a standalone network node with generic or specific components. Hardware 5504 may implement some functions via virtualization. Alternatively, hardware 5504 may be part of a larger cluster of hardware (e.g., such as in a data center or CPE) where many hardware nodes work together and are managed via management and orchestration 5510, which, among others, oversees lifecycle management of applications 5502. Insome embodiments, hardware 5504 is coupled to one or more radio units that each include one or more transmitters and one or more receivers that may be coupled to one or more antennas. Radio units may communicate directly with other hardware nodes via one or more appropriate network interfaces and may be used in combination with the virtual components to provide a virtual node with radio capabilities, such as a radio access node or a base station. In some embodiments, some signaling can be provided with the use of a control system 5512 which may alternatively be used for communication between hardware nodes and radio units.[000127] Figure 10 shows a communication diagram of a host 6602 communicating via a network node 6604 with a UE 6606 over a partially wireless connection in accordance with some embodiments. Example implementations, in accordance with various embodiments, of the UE (such as a UE 2112a of Figure 5 and / or UE 2200 of Figure 6), network node (such as network node 2110a of Figure 5 and / or network node 3300 of Figure 7), and host (such as host 2116 of Figure 5 and / or host 4400 of Figure 8) discussed in the preceding paragraphs will now be described with reference to Figure 10.[000128] Like host 4400, embodiments of host 6602 include hardware, such as a communication interface, processing circuitry, and memory. The host 6602 also includes software, which is stored in or accessible by the host 6602 and executable by the processing circuitry. The software includes a host application that may be operable to provide a service to a remote user, such as the UE 6606 connecting via an over-the-top (OTT) connection 6650 extending between the UE 6606 and host 6602. In providing the service to the remote user, a host application may provide user data which is transmitted using the OTT connection 6650.[000129] The network node 6604 includes hardware enabling it to communicate with the host 6602 and UE 6606. The connection 6660 may be direct or pass through a core network (like core network 2106 of Figure 5) and / or one or more other intermediate networks, such as one or more public, private, or hosted networks. For example, an intermediate network may be a backbone network or the Internet.[000130] The UE 6606 includes hardware and software, which is stored in or accessible by UE 6606 and executable by the UE’s processing circuitry. The software includes a client application, such as a web browser or operator-specific “app” that may be operable to provide a service to a human or non-human user via UE 6606 with the support of the host 6602. In the host 6602, an executing host application may communicate with the executing clientapplication via the OTT connection 6650 terminating at the UE 6606 and host 6602. In providing the service to the user, the UE's client application may receive request data from the host's host application and provide user data in response to the request data. The OTT connection 6650 may transfer both the request data and the user data. The UE's client application may interact with the user to generate the user data that it provides to the host application through the OTT connection 6650.[000131] The OTT connection 6650 may extend via a connection 6660 between the host 6602 and the network node 6604 and via a wireless connection 6670 between the network node 6604 and the UE 6606 to provide the connection between the host 6602 and the UE 6606. The connection 6660 and wireless connection 6670, over which the OTT connection 6650 may be provided, have been drawn abstractly to illustrate the communication between the host 6602 and the UE 1606 via the network node 6604, without explicit reference to any intermediary devices and the precise routing of messages via these devices.[000132] As an example of transmitting data via the OTT connection 6650, in step 6608, the host 6602 provides user data, which may be performed by executing a host application. In some embodiments, the user data is associated with a particular human user interacting with the UE 6606. In other embodiments, the user data is associated with a UE 6606 that shares data with the host 6602 without explicit human interaction. In step 6610, the host 6602 initiates a transmission carrying the user data towards the UE 6606. The host 6602 may initiate the transmission responsive to a request transmitted by the UE 6606. The request may be caused by human interaction with the UE 6606 or by operation of the client application executing on the UE 6606. The transmission may pass via the network node 6604, in accordance with the teachings of the embodiments described throughout this disclosure. Accordingly, in step 6612, the network node 6604 transmits to the UE 6606 the user data that was carried in the transmission that the host 6602 initiated, in accordance with the teachings of the embodiments described throughout this disclosure. In step 6614, the UE 6606 receives the user data carried in the transmission, which may be performed by a client application executed on the UE 6606 associated with the host application executed by the host 6602.[000133] In some examples, the UE 6606 executes a client application which provides user data to the host 6602. The user data may be provided in reaction or response to the data received from the host 6602. Accordingly, in step 6616, the UE 6606 may provide user data,which may be performed by executing the client application. In providing the user data, the client application may further consider user input received from the user via an input / output interface of the UE 6606. Regardless of the specific manner in which the user data was provided, the UE 6606 initiates, in step 6618, transmission of the user data towards the host 6602 via the network node 6604. In step 6620, in accordance with the teachings of the embodiments described throughout this disclosure, the network node 6604 receives user data from the UE 6606 and initiates transmission of the received user data towards the host 6602. In step 6622, the host 6602 receives the user data carried in the transmission initiated by the UE 6606.[000134] One or more of the various embodiments improve the performance of OTT services provided to the UE 6606 using the OTT connection 6650, in which the wireless connection 6670 forms the last segment. More precisely, the teachings of these embodiments may improve the data rate, latency, and / or power consumption and thereby provide benefits such as reduced user waiting time, relaxed restriction on file size, improved content resolution, better responsiveness, and / or extended battery lifetime.[000135] In an example scenario, factory status information may be collected and analyzed by the host 6602. As another example, the host 6602 may process audio and video data which may have been retrieved from a UE for use in creating maps. As another example, the host 6602 may collect and analyze real-time data to assist in controlling vehicle congestion (e.g., controlling traffic lights). As another example, the host 6602 may store surveillance video uploaded by a UE. As another example, the host 6602 may store or control access to media content such as video, audio, VR or AR which it can broadcast, multicast or unicast to UEs. As other examples, the host 6602 may be used for energy pricing, remote control of non-time critical electrical load to balance power generation needs, location services, presentation services (such as compiling diagrams etc. from data collected from remote devices), or any other function of collecting, retrieving, storing, analyzing and / or transmitting data.[000136] In some examples, a measurement procedure may be provided for the purpose of monitoring data rate, latency and other factors on which the one or more embodiments improve. There may further be an optional network functionality for reconfiguring the OTT connection 6650 between the host 6602 and UE 6606, in response to variations in the measurement results. The measurement procedure and / or the network functionality for reconfiguring the OTT connection may be implemented in software and hardware of the host 6602 and / or UE 6606. Insome embodiments, sensors (not shown) may be deployed in or in association with other devices through which the OTT connection 6650 passes; the sensors may participate in the measurement procedure by supplying values of the monitored quantities exemplified above, or supplying values of other physical quantities from which software may compute or estimate the monitored quantities. The reconfiguring of the OTT connection 6650 may include message format, retransmission settings, preferred routing etc.; the reconfiguring need not directly alter the operation of the network node 6604. Such procedures and functionalities may be known and practiced in the art. In certain embodiments, measurements may involve proprietary UE signaling that facilitates measurements of throughput, propagation times, latency and the like, by the host 6602. The measurements may be implemented in that software causes messages to be transmitted, in particular empty or ‘dummy’ messages, using the OTT connection 6650 while monitoring propagation times, errors, etc.[000137] Although the computing devices described herein (e.g., UEs, network nodes, hosts) may include the illustrated combination of hardware components, other embodiments may comprise computing devices with different combinations of components. It is to be understood that these computing devices may comprise any suitable combination of hardware and / or software needed to perform the tasks, features, functions and methods disclosed herein. Determining, calculating, obtaining or similar operations described herein may be performed by processing circuitry, which may process information by, for example, converting the obtained information into other information, comparing the obtained information or converted information to information stored in the network node, and / or performing one or more operations based on the obtained information or converted information, and as a result of said processing making a determination. Moreover, while components are depicted as single boxes located within a larger box, or nested within multiple boxes, in practice, computing devices may comprise multiple different physical components that make up a single illustrated component, and functionality may be partitioned between separate components. For example, a communication interface may be configured to include any of the components described herein, and / or the functionality of the components may be partitioned between the processing circuitry and the communication interface. In another example, non-computationally intensive functions of any of such components may be implemented in software or firmware and computationally intensive functions may be implemented in hardware.[000138] In certain embodiments, some or all of the functionality described herein may be provided by processing circuitry executing instructions stored on in memory, which in certain embodiments may be a computer program product in the form of a non-transitory computer-readable storage medium. In alternative embodiments, some or all of the functionality may be provided by the processing circuitry without executing instructions stored on a separate or discrete device-readable storage medium, such as in a hard-wired manner. In any of those particular embodiments, whether executing instructions stored on a non-transitory computer- readable storage medium or not, the processing circuitry can be configured to perform the described functionality. The benefits provided by such functionality are not limited to the processing circuitry alone or to other components of the computing device, but are enjoyed by the computing device as a whole, and / or by end users and a wireless network generally.[000139] It will be appreciated that computer systems are increasingly taking a wide variety of forms. In this description and in the claims, the terms “controller,” “computer system,” or “computing system” are defined broadly as including any device or system — or combination thereof — that includes at least one physical and tangible processor and a physical and tangible memory capable of having thereon computer-executable instructions that may be executed by a processor. By way of example, not limitation, the term “computer system” or “computing system,” as used herein is intended to include personal computers, desktop computers, laptop computers, tablets, hand-held devices (e.g., mobile telephones, PDAs, pagers), microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, multi-processor systems, network PCs, distributed computing systems, datacenters, message processors, routers, switches, and even devices that conventionally have not been considered a computing system, such as wearables (e.g., glasses).[000140] The computing system also has thereon multiple structures often referred to as an “executable component.” For instance, the memory of a computing system can include an executable component. The term “executable component” is the name for a structure that is well understood to one of ordinary skill in the art in the field of computing as being a structure that can be software, hardware, or a combination thereof. For instance, when implemented in software, one of ordinary skill in the art would understand that the structure of an executable component may include software objects, routines, methods, and so forth, that may be executed by one or more processors on the computing system, whether such an executable component exists in the heap ofa computing system, or whether the executable component exists on computer-readable storage media. The structure of the executable component exists on a computer-readable medium in such a form that it is operable, when executed by one or more processors of the computing system, to cause the computing system to perform one or more functions, such as the functions and methods described herein. Such a structure may be computer-readable directly by a processor — as is the case if the executable component were binary. Alternatively, the structure may be structured to be interpretable and / or compiled — whether in a single stage or in multiple stages — so as to generate such binary that is directly interpretable by a processor.[000141] The terms “component,” “service,” “engine,” “module,” “control,” “generator,” or the like may also be used in this description. As used in this description and in this case, these terms — whether expressed with or without a modifying clause — are also intended to be synonymous with the term “executable component” and thus also have a structure that is well understood by those of ordinary skill in the art of computing.[000142] In terms of computer implementation, a computer is generally understood to comprise one or more processors or one or more controllers, and the terms computer, processor, and controller may be employed interchangeably. When provided by a computer, processor, or controller, the functions may be provided by a single dedicated computer or processor or controller, by a single shared computer or processor or controller, or by a plurality of individual computers or processors or controllers, some of which may be shared or distributed. Moreover, the term “processor” or “controller” also refers to other hardware capable of performing such functions and / or executing software, such as the example hardware recited above.[000143] In general, the various exemplary embodiments may be implemented in hardware or special purpose chips, circuits, software, logic, or any combination thereof. For example, some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software which may be executed by a controller, microprocessor, or other computing device, although the disclosure is not limited thereto. While various aspects of the exemplary embodiments of this disclosure may be illustrated and described as block diagrams, flow charts, or using some other pictorial representation, it is well understood that these blocks, apparatus, systems, techniques, or methods described herein may be implemented in, as nonlimiting examples, hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controller or other computing devices, or some combination thereof.[000144] While not all computing systems require a user interface, in some embodiments a computing system includes a user interface for use in communicating information from / to a user. The user interface may include output mechanisms as well as input mechanisms. The principles described herein are not limited to the precise output mechanisms or input mechanisms as such will depend on the nature of the device. However, output mechanisms might include, for instance, speakers, displays, tactile output, projections, holograms, and so forth. Examples of input mechanisms might include, for instance, microphones, touchscreens, projections, holograms, cameras, keyboards, stylus, mouse, or other pointer input, sensors of any type, and so forth.Abbreviations and Defined Terms[000145] To assist in understanding the scope and content of this written description and the appended claims, a select few terms are defined directly below. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present disclosure pertains.[000146] The terms “approximately,” “about,” and “substantially,” as used herein, represent an amount or condition close to the specific stated amount or condition that still performs a desired function or achieves a desired result. For example, the terms “approximately,” “about,” and “substantially” may refer to an amount or condition that deviates by less than 10%, or by less than 5%, or by less than 1%, or by less than 0.1%, or by less than 0.01% from a specifically stated amount or condition.[000147] Various aspects of the present disclosure, including devices, systems, and methods may be illustrated with reference to one or more embodiments or implementations, which are exemplary in nature. As used herein, the term “exemplary” means “serving as an example, instance, or illustration,” and should not necessarily be construed as preferred or advantageous over other embodiments disclosed herein. In addition, reference to an “implementation” of the present disclosure or embodiments includes a specific reference to one or more embodiments thereof, and vice versa, and is intended to provide illustrative examples without limiting the scope of the present disclosure, which is indicated by the appended claims rather than by the present description.[000148] As used in the specification, a word appearing in the singular encompasses its plural counterpart, and a word appearing in the plural encompasses its singular counterpart, unless implicitly or explicitly understood or stated otherwise. Thus, it will be noted that, as used in this specification and the appended claims, the singular forms “a,” “an” and “the” include plural referents unless the context clearly dictates otherwise. For example, reference to a singular referent (e.g., “a widget”) includes one, two, or more referents unless implicitly or explicitly understood or stated otherwise. Similarly, reference to a plurality of referents should be interpreted as comprising a single referent and / or a plurality of referents unless the content and / or context clearly dictate otherwise. For example, reference to referents in the plural form (e.g., “widgets”) does not necessarily require a plurality of such referents. Instead, it will be appreciated that independent of the inferred number of referents, one or more referents are contemplated herein unless stated otherwise.[000149] References in the specification to "one embodiment," "an embodiment," "an example embodiment," and the like indicate that the embodiment described may include a particular feature, structure, or characteristic, but it is not necessary that every embodiment includes the particular feature, structure, or characteristic. Moreover, such phrases are not necessarily referring to the same embodiment. Further, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is submitted that it is within the knowledge of one skilled in the art to affect such feature, structure, or characteristic in connection with other embodiments whether or not explicitly described.[000150] It shall be understood that although the terms "first" and "second" etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first element could be termed a second element, and similarly, a second element could be termed a first element, without departing from the scope of example embodiments. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed terms.[000151] It will be further understood that the terms "comprises", "comprising", "has", "having", "includes" and / or "including", when used herein, specify the presence of stated features, elements, and / or components etc., but do not preclude the presence or addition of one or more other features, elements, components and / or combinations thereof.Conclusion[000152] The present disclosure includes any novel feature or combination of features disclosed herein either explicitly or any generalization thereof. Various modifications and adaptations to the foregoing exemplary embodiments of this disclosure may become apparent to those skilled in the relevant arts in view of the foregoing description, when read in conjunction with the accompanying drawings. However, any and all modifications will still fall within the scope of the non-limiting and exemplary embodiments of this disclosure.[000153] It is understood that for any given component or embodiment described herein, any of the possible candidates or alternatives listed for that component may generally be used individually or in combination with one another, unless implicitly or explicitly understood or stated otherwise. Additionally, it will be understood that any list of such candidates or alternatives is merely illustrative, not limiting, unless implicitly or explicitly understood or stated otherwise.[000154] In addition, unless otherwise indicated, numbers expressing quantities, constituents, distances, or other measurements used in the specification and claims are to be understood as being modified by the term “about,” as that term is defined herein. Accordingly, unless indicated to the contrary, the numerical parameters set forth in the specification and attached claims are approximations that may vary depending upon the desired properties sought to be obtained by the subject matter presented herein. At the very least, and not as an attempt to limit the application of the doctrine of equivalents to the scope of the claims, each numerical parameter should at least be construed in light of the number of reported significant digits and by applying ordinary rounding techniques. Notwithstanding that the numerical ranges and parameters setting forth the broad scope of the subject matter presented herein are approximations, the numerical values set forth in the specific examples are reported as precisely as possible. Any numerical values, however, inherently contain certain errors necessarily resulting from the standard deviation found in their respective testing measurements.[000155] Any headings and subheadings used herein are for organizational purposes only and are not meant to be used to limit the scope of the description or the claims. The terms and expressions which have been employed herein are used as terms of description and not of limitation, and there is no intention in the use of such terms and expressions of excluding any equivalents of the features shown and described or portions thereof, but it is recognized that various modifications are possible within the scope of the present disclosure. Thus, it should be understoodthat although the present disclosure has been specifically disclosed in part by certain embodiments, and optional features, modification and variation of the concepts herein disclosed may be resorted to by those skilled in the art, and such modifications and variations are considered to be within the scope of this present description.[000156] It will also be appreciated that systems, devices, products, kits, methods, and / or processes, according to certain embodiments of the present disclosure may include, incorporate, or otherwise comprise properties or features (e.g., components, members, elements, parts, and / or portions) described in other embodiments disclosed and / or described herein. Accordingly, the various features of certain embodiments can be compatible with, combined with, included in, and / or incorporated into other embodiments of the present disclosure. Thus, disclosure of certain features relative to a specific embodiment of the present disclosure should not be construed as limiting application or inclusion of said features to the specific embodiment. Rather, it will be appreciated that other embodiments can also include said features, members, elements, parts, and / or portions without necessarily departing from the scope of the present disclosure.[000157] Moreover, unless a feature is described as requiring another feature in combination therewith, any feature herein may be combined with any other feature of a same or different embodiment disclosed herein. Furthermore, various well-known aspects of illustrative systems, methods, apparatus, and the like are not described herein in particular detail in order to avoid obscuring aspects of the example embodiments. Such aspects are, however, also contemplated herein.[000158] It will be apparent to one of ordinary skill in the art that methods, devices, device elements, materials, procedures, and techniques other than those specifically described herein can be applied to the practice of the described embodiments as broadly disclosed herein without resort to undue experimentation. All art-known functional equivalents of methods, devices, device elements, materials, procedures, and techniques specifically described herein are intended to be encompassed by this present disclosure.[000159] When a group of materials, compositions, components, or compounds is disclosed herein, it is understood that all individual members of those groups and all subgroups thereof are disclosed separately. When a Markush group or other grouping is used herein, all individual members of the group and all combinations and sub-combinations possible of the group are intended to be individually included in the disclosure.[000160] The above-described embodiments are examples only. Alterations, modifications, and variations may be effected to the particular embodiments by those of skill in the art without departing from the scope of the description, which is defined solely by the appended claims.

Claims

CLAIMSWhat is claimed is:

1. A method for optimizing performance of one or more microservices being run in one or more nodes (280, 290) comprising one or more hardware architectures, the method comprising: generating (410) a representation model of one or more microservices based at least in part on one or features of one or more service chains related to the one or more microservices; extracting (415) one or more statuses of the one or more service chains, the one or more statuses identifying one or more nodes (280, 290) hosting the one or more microservices; collecting (420) one or more metrics related to the one or more nodes (280, 290) and the one or more microservices; extracting (425) one or more capabilities of the one or more hardware architectures; collecting (430) one or more resource usage metrics related to the one or more nodes (280, 290) or one or more microservices; estimating (435) service chain performance based at least in part on the one or more performance metrics; estimating (440) energy consumption of the one or more microservices based at least in part on the one or more resource usage metrics; allocating (445, 450), based at least in part on the one or more capabilities, one or more cores (289, 299) comprising the one or more nodes (280, 290) for performing the one or more microservices; mapping (455) the one or more microservices to the one or more cores (289, 299), the mapping comprising a configuration; and applying (460) the configuration to the one or more nodes (280, 290) and the one or more microservices.

2. The method of claim 1, further comprising continuously repeating (465) one or more steps in a closed-loop process.

3. The method of claim 1 or 2, wherein the one or more metrics comprise at least one of: usage by the one or more microservice-based application of central processing unit, CPU, memory, or disk resources; usage by the one or more nodes of CPU, memory, or disk resources.

4. The method of any of claims 1 to 3, wherein the one or more capabilities comprise one or more of: per-core Dynamic Voltage and Frequency Scaling, DVFS; per-chip DVFS; CPU capabilities; random access memory, RAM, capabilities.

5. The method of any of claims 1 to 4, wherein the one or more resource usage metrics comprise at least one of: latency; throughput; CPU usage.

6. The method of any of claims 1 to 5, wherein the estimating service chain performance comprises profiling one or more characteristics of the one or more microservices in terms of sensitivity to collocation with one or more other microservices running on one or more identical cores and variabilities of performance of microservice overtime.

7. The method of any of claims 1 to 6, wherein the estimating energy consumption is based at least in part on one or more of: CPU usage; CPU frequency; memory resources; disk resources; and network resources.

8. The method of any of claims 1 to 7, wherein the allocating comprises at least one of: allocating fractional cores; setting or scaling CPU frequency; performing per-core DVFS; performing per-chip DVFS.

9. The method of claim 8, wherein in case of per-core DVFS, the frequency of the one or more cores is decided by assessing a minimal frequency and number of cores or fractional cores that could fulfil a chain performance requirement and reduce energy usage.

10. The method of claim 8, wherein in case of per-chip DVFS, a lowest frequency is selected for all cores comprising a chip and a number of cores are scaled for individual microservices that run on the chip to fulfil requirements.

11. The method of claim 10, wherein in case of fractional core assignment for microservice of a time-critical service chain, a remaining fraction of a core (289, 299) can be assigned to besteffort microservice type to facilitate higher utilization.

12. The method of any of claims 1 to 11, wherein the mapping comprises use of non-uniform memory access topology.

13. The method of any of claims 1 to 12, wherein the applying the new configuration comprises instantiating at least one of: a core scaler (262); a CPU frequency scaler (264); and a microservice- to-core mapper (266).

14. A system (200) for modeling and monitoring one or more nodes (280, 290) running one or more microservices, the one or more microservices associated with one or more service chains and the one or more nodes (280, 290) comprising one or more hardware architectures (288, 298, 289, 299), comprising: an application modeler (215) configured to generate a representation model of the one or more microservices based at least in part on the one or more service chains; a monitor (230) configured to monitor the one or more nodes (280, 290) and to perform one or more of, detect a status of the one or more service chains, collect one or more metrics related to the one or more nodes (280, 290) or one or more microservices, collect one or more capabilities of the one or more hardware architectures (288, 289, 298, 299); a chain performance estimator (220) coupled to the monitor (230) and the application modeler (215) and configured to estimate a performance of the one or more service chains based at least in part on the one or more metrics;a microservice energy estimator (250) coupled to the monitor (230) and configured to estimate energy consumption of the one or more microservices based at least in part on the one or more metrics; and a central processing unit, CPU, resource allocator (240) coupled to the application modeler (215), the chain performance estimator (220), and the microservice energy estimator (250) and configured to allocate, based at least in part on the one or more capabilities, one or more cores (289, 299) comprising the one or more nodes (280, 290) for performing the one or more microservices, and further configured to map the one or more microservices to the one or more cores (289, 299), the mapping comprising a configuration, the resource allocator further configured to provide the configuration to an actuator (265) configured to apply the configuration to the one or more nodes (280, 290), the CPU resource allocator further configured to set CPU frequency for the one or more cores.

15. The system of claim 14, further comprising the actuator (265).

16. The system of claim 14 or 15, further comprising the one or more nodes (280, 290).

17. The system of any of claims 14 to 16, wherein actuator (265) comprises a Kubernetes cluster (260).

18. A method (600) for modeling and monitoring one or more nodes (280, 290) running one or more microservices, the one or more microservices associated with one or more service chains and the one or more nodes (280, 290) comprising one or more hardware architectures (288, 298, 289, 299), comprising: receiving (601) a service description describing one or more microservices to be performed in one or more nodes (280, 290); generating (602) a representation model based at least in part on the service description, the representation model describing one or more service chains for performing the one or more microservices in the one or more nodes (280, 290);collecting (603) one or more energy usage metrics, one or more central processing unit, CPU, metrics, and one or more performance metrics related to the one or more service chains or one or more nodes (280, 290); detecting (604) one or more performance violations related to at least one of the one or more energy usage metrics, the one or more CPU metrics, or one or more performance metrics; requesting (605a / b) one or more additional performance metrics and one or more additional energy usage metrics; detecting (606) one or more statuses related to the one or more nodes; extracting (607) one or more microservice statuses related to the one or more microservices; estimating (608) a performance level of the one or more service chains based at least in part on the one or more microservice statuses; estimating (609) an energy usage level of the one or more microservices; detecting (610) one or more capabilities of the one or more nodes or one or more hardware architectures; creating (611 ) a new resource configuration for the one or more service chains, one or more microservices, one or more nodes, and / or one or more hardware architectures; translating (613) the new resource configuration to one or more policies, the one or more policies formatted to be used by a cloud orchestrator (655) to apply the new resource configuration to the one or more nodes (280, 290); and transmitting (614) the one or more policies to the cloud orchestrator (655).

19. The method of claim 18, further comprising applying, by the cloud orchestrator (655), the one or more policies.

20. The method of claim 18 or 19, wherein the one or more policies comprises one or more of: CPU scaling; CPU frequency scaling; CPU pinning; CPU allocation; controlling container CPU shares and quota allocations in a Linux Cgroup; CPU frequency setting; setting a CPU power governor; setting user-controlled CPU frequencies in Linux power and / or Frequency control); CPU mapping.

21. The method of any of claims 18 to 20, wherein the service description is described using modeling language.

22. The method of any of claims 18 to 21, wherein the service description illustrates one or more of: one or more dependencies between the one or more microservices; one or more types of service chains; one or more performance requirements of the one or more service chains.

23. The method of any of claims 18 to 22, wherein the one or more statuses comprise one or more of: deployment status; non-uniform memory access, NUMA, topology of the one or more nodes; resource usage; CPU usage; memory usage; disk usage; network usage; CPU frequency of the one or more cores.

24. The method of any of claims 18 to 23, wherein the one or more microservice statuses comprise one or more of: latency; CPU usage; CPU frequency.

25. The method of any of claims 18 to 24, wherein the estimating an energy usage level is performed by one or more of: Scaphandre; Smartwatt; Kepler; a microservice energy estimator 630; a regression analysis method.

26. The method of any of claims 18 to 25, wherein the energy usage level comprises one or more of: CPU frequency; CPU memory; disk usage; network usage; power consumption by one or more components; power consumption by one or more CPUs; power consumption by one or more random access memories, RAMs.

27. The method of any of claims 18 to 26, wherein the one or more capabilities comprise one or more of: Dynamic Voltage and Frequency Scaling, DVFS, capabilities of the one or more nodes; per-core DVFS; per-chip DVFS.

28. The method of any of claims 18 to 27, wherein the new resource configuration comprises at least one of: scaling a number of cores; CPU frequency; mapping of the one or more microservices to physical cores;29. The method of any of claims 18 to 28, wherein the one or more policies comprise at least one of: a CPU scaling policy for the scaling of cores of the one or more microservices; a CPU frequency scaling policy for the scaling of CPU frequency of cores used by the one or more microservices; a CPU pinning policy for the mapping of microservices to specific physical cores.

30. A system for modeling and monitoring one or more nodes (280, 290) running one or more microservices, the one or more microservices associated with one or more service chains and the one or more nodes (280, 290) comprising one or more hardware architectures (288, 298, 289, 299), comprising: processing circuitry; and a memory, the memory containing instructions executable by the processing circuitry whereby the system is operative to: receive (601) a service description describing one or more microservices to be performed in one or more nodes (280, 290); generate (602) a representation model based at least in part on the service description, the representation model describing one or more service chains for performing the one or more microservices in the one or more nodes (280, 290); collect (603) one or more energy usage metrics, one or more central processing unit, CPU, metrics, and one or more performance metrics related to the one or more service chains or one or more nodes (280, 290); detect (604) one or more performance violations related to at least one of the one or more energy usage metrics, the one or more CPU metrics, or one or more performance metrics; request (605a / b) one or more additional performance metrics and one or more additional energy usage metrics; detect (606) one or more statuses related to the one or more nodes; extract (607) one or more microservice statuses related to the one or more microservices; estimate (608) a performance level of the one or more service chains based at least in part on the one or more microservice statuses;estimate (609) an energy usage level of the one or more microservices; detect (610) one or more capabilities of the one or more nodes or one or more hardware architectures; create (611) a new resource configuration for the one or more service chains, one or more microservices, one or more nodes, and / or one or more hardware architectures; translate (613) the new resource configuration to one or more policies, the one or more policies formatted to be used by a cloud orchestrator (655) to apply the new resource configuration to the one or more nodes (280, 290); and transmit (614) the one or more policies to the cloud orchestrator (655).

31. The system of claim 30, further comprising at least one of: a user equipment, UE; a network node; a host; a virtualization environment.

Images