Methods and systems for fault mitigation with additional state transitions in cloud systems

Abstract

Embodiments include methods, electronic device, storage medium, and computer program to mitigate faults in a cloud system. In one embodiment, a method comprises moving a first work unit from a running state to a first state upon determining that the first work unit is faulty; responsively causing a set of work units to be started to provide the service based on a probability of a work unit in the first state being moved to a second state, where in the second state the work unit is standby for task execution; moving a second work unit from the first to second state upon determining within a time period that the second work unit is ready for the service; and causing a third work unit to perform task execution by moving it from the second state to the running state upon determining that one or more additional work units are needed.

Description

SPECIFICATIONMETHODS AND SYSTEMS FOR FAULT MITIGATION WITH ADDITIONAL STATE TRANSITIONS IN CLOUD SYSTEMSTECHNICAL FIELD

[0001] Embodiments of the invention relate to the field of networking; and more specifically, to providing fault mitigation in a cloud system.BACKGROUND ART

[0002] In cloud systems, containers or pods offer a lightweight form of virtualization at the operating system level, as opposed to traditional virtualization that relies on a hypervisor. These work units can be deployed as part of a service that handles client requests.

[0003] A cloud system also provides mechanisms to identify and replace unresponsive or nonfunctioning work units. For example, a pod is the smallest deployable unit in Kubernetes that can be managed individually. It represents a single instance of a running application / service in a cluster and encapsulates one or more containers. Pods within a ReplicaSet or Deployment definition can be replaced upon termination.

[0004] When a work unit (e g., a pod / container) fails, the failure can be either temporary or permanent, and determining its status in real-time (or with low latency) is often challenging. This complexity arises from the need to avoid prematurely replacing work units, which would result in a lengthy startup delay for a new work unit. Cloud systems generally adopt a standard approach of waiting for a specific time period, a certain number of failed health tests, or a specific number of consecutive failed requests before replacing a work unit. Although these configurations can react quickly, they are typically not sufficient for latency sensitive applications in practice due to the long recovery time. The typical time horizon for replacement ranges from tens of seconds to minutes (which aligns with the startup and initialization time of a container).

[0005] Yet the time horizon of tens of seconds to minutes for the replacement is unacceptable for running latency sensitive applications in the cloud. These latency sensitive applications often require a much quicker fault detection response. Additionally, a latency sensitive application may be deployed in dedicated or highly distributed edge locations, which have a higher resource contention and hence it is difficult and more costly to under-utilize compute resources and it is important to keep the resource usage at the correct level. Thus, existing fault mitigationmethods, being reactive to failed tests / requests, are insufficient to run latency sensitive applications in a cloud system.SUMMARY OF THE INVENTION

[0006] Embodiments include methods, electronic device, storage medium, and computer program for providing fault mitigation in a cloud system. In one embodiment, a method comprises: moving a first work unit in the cloud system from a running state to a first state upon determining that the first work unit is faulty, wherein in the running state, the first work unit performed task execution to provide a service of the cloud system; responsive to the first work unit being moved to the first state, causing a set of work units to be started to provide the service based on a probability of a work unit in the first state being moved to a second state, wherein in the second state the work unit is standby for task execution to provide the service; moving a second work unit from the first state to the second state upon determining within a time period that the second work unit is ready for the service; and causing a third work unit to perform task execution by moving the third work unit from the second state to the running state upon determining that one or more additional work units are needed to provide the service.

[0007] Embodiments include electronic devices for providing fault mitigation in a cloud system. In one embodiment, an electronic device comprises a processor and machine-readable storage medium that provides instructions that, when executed by the processor, are capable of causing the electronic device to perform: moving a first work unit in the cloud system from a running state to a first state upon determining that the first work unit is faulty, wherein in the running state, the first work unit performed task execution to provide a service of the cloud system; responsive to the first work unit being moved to the first state, causing a set of work units to be started to provide the service based on a probability of a work unit in the first state being moved to a second state, wherein in the second state the work unit is standby for task execution to provide the service; moving a second work unit from the first state to the second state upon determining within a time period that the second work unit is ready for the service; and causing a third work unit to perform task execution by moving the third work unit from the second state to the running state upon determining that one or more additional work units are needed to provide the service.

[0008] Embodiments include machine-readable storage media for providing fault mitigation in a cloud system. In one embodiment, a machine-readable storage medium that provides instructions that, when executed, are capable of causing the electronic device to perform: moving a first work unit in the cloud system from a running state to a first state upon determining that the first work unit is faulty, wherein in the running state, the first work unitperformed task execution to provide a service of the cloud system; responsive to the first work unit being moved to the first state, causing a set of work units to be started to provide the service based on a probability of a work unit in the first state being moved to a second state, wherein in the second state the work unit is standby for task execution to provide the service; moving a second work unit from the first state to the second state upon determining within a time period that the second work unit is ready for the service; and causing a third work unit to perform task execution by moving the third work unit from the second state to the running state upon determining that one or more additional work units are needed to provide the service.

[0009] By implementing embodiments as described, a cloud system may proactively start a set of work units to maintain a sufficient number of work units in the second state, which then can be transitioned to the running state quickly without waiting on the whole duration of new work unit creation. By proactively starting the set of work units, these embodiments mitigate faults faster than the reactively creating new work units upon the determination that a running work unit has a permanent fault and needs to be replaced.BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The invention may best be understood by referring to the following description and accompanying drawings that are used to illustrate embodiments of the invention. In the drawings:

[0011] Figure 1 illustrates state transitions of a work unit per some embodiments.

[0012] Figure 2 illustrates functional blocks and operations of work unit life cycle management coordination per some embodiments.

[0013] Figure 3 illustrates components of a cloud system in which fault mitigation may be performed per some embodiments.

[0014] Figure 4 illustrates fault mitigation with additional states at the pod level in a cloud system per additional embodiments.

[0015] Figure 5 is a flow diagram illustrating operations of fault mitigation with additional states in a cloud system per some embodiments.

[0016] Figure 6 illustrates an electronic device to mitigate fault in a cloud system per some embodiments.

[0017] Figure 7 illustrates an example of a communication system per some embodiments.

[0018] Figure 8 illustrates a user equipment (UE) per some embodiments.

[0019] Figure 9 illustrates a network node per some embodiments.

[0020] Figure 10 is a block diagram of a host, which may be an embodiment of the host of Figure 7, per various aspects described herein.

[0021] Figure 11 is a block diagram illustrating a virtualization environment in which functions implemented by some embodiments may be virtualized.

[0022] Figure 12 illustrates a communication diagram of a host communicating via a network node with a user equipment (UE) over a partially wireless connection per some embodiments.DETAILED DESCRIPTION

[0023] Generally, all terms used herein are to be interpreted according to their ordinary meaning in the relevant technical field, unless a different meaning is clearly given and / or is implied from the context in which it is used. All references to a / an / the element, apparatus, component, means, step, etc. are to be interpreted openly as referring to at least one instance of the element, apparatus, component, means, step, etc., unless explicitly stated otherwise. The steps of any methods disclosed herein do not have to be performed in the exact order disclosed, unless a step is explicitly described as following or preceding another step and / or where it is implicit that a step must follow or precede another step. Any feature of any of the embodiments disclosed herein may be applied to any other embodiment, wherever appropriate. Likewise, any advantage of any of the embodiments may apply to any other embodiments, and vice versa. Other objectives, features, and advantages of the enclosed embodiments will be apparent from the following description.Work Unit State Machine with Additional States

[0024] Embodiments include methods, electronic device, storage medium, and computer program for providing timely fault mitigation in a cloud system. The fault mitigation includes additional state transitions in the state machine of a work unit.

[0025] A work unit is a deployable unit in the cloud system that can be managed as an individual virtual resource. The work unit is referred to as virtual because it is created and managed within a virtualized environment rather than being a physical entity. The work unit may be a virtual machine (VM), a container or pod, a virtual storage / database, a virtual load balancer, or another virtual resource in a cloud system such as Kubemetes. While pods / containers are often used as examples of the work units for illustration herein, other types of work units may use embodiments of the inventions as well.

[0026] Figure 1 illustrates state transitions of a work unit per some embodiments. The transitions of a work unit are presented as a work unit life cycle management (LCM) state machine at reference 100. The state machine is maintained by a life cycle management (LCM) coordinator (e.g., shown as LCM coordinator 290), which may be a software module or a hardware logic / circuit of an electronic device (e.g., electronic device 602).

[0027] The state transitions within the state machine include known states of a work unit for fault mitigation, and transitions between these states are self-explanatory. For example, a work unit may be initiated at Starting state 102, operate normally at Running state 104 to perform task execution to provide a service, and transition to Terminated state 116 when a fault is detected within the work unit. Once a work unit is terminated, it may be replaced or restored, so the repl ced / restored work unit can be initiated at Starting state 102, and the state machine for that work unit is started. These known states are shown with shaded cycles in Figure 1.

[0028] While three known states are shown in this embodiment, other existing approaches may have different states or give states different names. For example, the Starting state 102 may also be named as Pending state in that the work unit is being prepared but not ready to provide the service. Additionally, these state transitions are for fault mitigation, and state transitions during normal operations are omitted, e.g., in Running state 104, a work unit may transition from performing task execution to provide a service, to complete the task execution, and then to starting task execution to provide another service (or another service / application instance of the same service / application). Furthermore, some approaches include an Unknown state indicating the status of the work unit is to be determined (e.g., when the work unit has lost connectivity with the manager of the work unit).

[0029] Yet all of these known fault mitigation state transitions are reactive to fault detection. In the reaction, the work unit’s transition from Running state to Terminated state and subsequently replacing / restoring the work unit takes a relatively long time, since both of detecting that the fault is not recoverable and replacing / restoring a work unit can take substantial time and resources, and a cloud system needs to avoid prematurely replacing a work unit (1) that is temporarily at fault, and / or (2) that is determined as faulty by mistake. Such transition time is often tens of seconds to minutes, too long for some applications, particularly latency sensitive applications for consumers and enterprise.

[0030] Thus, two new states are introduced for fault mitigation in some embodiments. These states are referred to as Unhealthy state 110 and Backup state 108, which are also referred to as the first and second states, respectively. With the two new states, fault detection / mitigation can be examined in further granularity with different / additional transitions based on a fault being temporary or permanent.

[0031] A work unit may start at Starting state 102, where the work unit is being prepared for providing a service. Once the preparation is done, the work unit transitions to Backup state 108, where the work unit is standby for task execution to provide the service. In Backup state 108, the work unit does not execute tasks for the service but is ready to do so. In some embodiments, a work unit in Backup state 108 consumes storage resources (e.g., memory / cache) but no / fewexecution resources (e.g., central / graphics / tensor / infrastructure processing units (CPUs / GPUs / TPUs / IPUs)) or bandwidth / timeslot sources.

[0032] Once it is determined necessary, a work unit for the service transitions from Backup state 108 to Running state 104 to perform task execution to provide the service. The transition may be caused by notification of an event, e.g., the number of currently running work units to provide the service reaching a threshold (e.g., below a particular number to provide a given Quality-of-Service (QoS) for the service, the number of running work units < threshold ni) as shown at reference 154 in some embodiments.

[0033] At Running state 104, a work unit performs task execution to provide the service that the work unit is assigned. Upon a fault is detected on the work unit, the work unit is transitioned from Running state 104 to Unhealthy state 110. The transition to Unhealthy state 110 is speedy, and without determining whether the fault is temporary or permanent, or even confirming that the fault detection is accurate, either of which takes time and may extend the duration that an insufficient number of work units are in Running state 104 to provide the service. Instead, the work unit with detected fault (deemed unhealthy) is moved to Unhealthy state 110 promptly, and when such transition triggers an event (e.g., not enough work units to provide the service), another work unit in Backup state 108 transitions into Running state 104 (e.g., a work unit is added from Backup state 108 to Running state 104 when the number of running work units is less than a threshold m).

[0034] When a work unit is in Unhealthy state 110, it does not contribute to the service, and further examination is required to determine the nature of fault detection. The fault detection may be incorrect and the work unit actually operates properly, in which case (false positive) the work unit was put in Unhealthy state 110 unnecessarily. Alternatively, the fault detection may be correct but the fault is temporary / transient (e g., occasional insufficient bandwidth / execution resources caused fault), in which case the LCM coordinator has caused the transition properly, but the work unit should be returned to provide the service once the work unit is recovered from the temporary fault.

[0035] In a cloud system, the distinction between false positive and temporary fault may be blurry, and some scenarios can be counted in either category, e.g., fault detection due to network connectivity issues, resource exhaustion, address resolution failure, software / configuration errors, security / access control issues, and / or external service outage. For fault mitigation, the distinction between false positive and temporary fault is less important than the commonality of the two, which is that the work unit should be transitioned to Backup state 108 in both cases - transitioning upon determining that the fault is not permanent at reference 156 and that the work unit has recovered to healthy.

[0036] Alternatively, the fault that caused the work unit to transition to Unhealthy state 110 is permanent. The fault is permanent when the work unit can not be automatically or quickly rectified. In that case, the faulty work unit needs to be replaced and the state of the work unit is transitioned to Terminated state 116. For the permanently faulty work unit, the state transition is complete; and a new work unit needs to be initiated at Starting state 102 and the state machine operates again for the new work unit through transitions among the five states. Yet starting the work unit when the fault is determined to be permanently faulty may be too late for timesensitive services. Ideally a new work unit should be started (shown at reference 158) proactively prior to such permanent fault determination.

[0037] It is noted that in the transitions, a work unit at Backup state 108 is standby for task execution to provide the service. It consumes some resources yet not providing the service, it is thus preferrable to keep the number of work units at Backup state 108 to be at a level that is low enough to keep the resource usage down; at the same time, to maintain a required QoS of the service, the number needs to be high enough to replace work units that are detected to be faulty and transitioned to Unhealthy state 110.

[0038] The optimal number of work units at Backup state 108 is thus based on the likelihood and / or pattern of work units transitioning between different states. In some embodiments, the number of work units at Backup state 108 may be controlled by the likelihood of unhealthy work units that are to be determined not to be permanently faulty within a time period, and that likelihood may be used to start a new work unit at Starting state 102 when a work unit at Running state 104 is changed to Unhealthy state 110.

[0039] To characterize the likelihood, the term return probability (RP) is used to refer to the probability of an unhealthy work unit that recovers, where the work unit is to be determined not to be permanently faulty within a time period, as such work unit will be returned to service (transitioning from Unhealthy state 110, to Backup state 108 and then Running state 104).

[0040] In a simple form, the return probability may be determined through dividing (1) the number of work units that recovers to Backup state 108 from Unhealthy state 110 in a time period by (2) the number of work units entering Backup state 108 from Running state 104 in the same time period. Since some work units will be transitioned from Unhealthy state 110 to Terminated state 116, the resulting value should be in the range of zero to one, and the opposite probability 1 - RP is the probability of an unhealthy work unit that is to be determined to be permanently faulty within the time period. The return probability is calculated per work unit type, where for work unit type i, the return probability is RPi.

[0041] With the known return probability RPi, instead of waiting for the determination that a work unit is at permanent fault to create the new replacement work unit, a new work unit can becreated proactively at 158 based on (1) the return probability RPi and (2) a sufficient number of work units have transitioned from Running state 104 to Unhealthy state 110 so that the optimal number of work units are maintained in Backup state 108. In some embodiment, for every 1 / (1 - RPi) occurrences of a work unit of type i entering Unhealthy state, a new work unit of type i is started. The state transition from Unhealthy state 110 to Starting state 102 is shown with a dotted line to indicate that the transition happens only under a certain condition (thus optional), which is based on (1) the return probability RPi and (2) a sufficient number of work units have transitioned from Running state 104 to Unhealthy state 110. While the formula of !(\-RPi) is shown as a threshold example, alternative embodiment may use a different threshold

[0042] In such approach, the new work unit is started proactively only when necessary, and a set of work units is kept in the backup state, so when a running work unit becomes faulty, one or more work units may be transitioned from backup state to running state without enduring the delay of starting new work units. By proactively building up the work units in the backup state, this approach allows a faster response time over prior methods, which are reactive to a permanent fault, determination of which may take a long time and not be quick enough to provide a time sensitive service.Work Unit Life Cycle Management (LCM) Coordination

[0043] As shown in the state machine of a work unit, the work unit life cycle management (LCM) coordination includes determining whether a work unit is permanently faulty or recoverable to provide service. Without knowing the nature of the fault detection and based on the return probability, work units are started proactively, so that a sufficient number of work units are maintained in Backup state 108 to replace work units in Running state 104 when one or more running work units are detected faulty. The number of work units in the various states is thus dynamic and changes based on various factors. A work unit LCM coordinator may manage the transitions of work units to mitigate faults and provide services for a cloud system.

[0044] Figure 2 illustrates functional blocks and operations of work unit life cycle management (LCM) coordination per some embodiments. A life cycle management (LCM) coordinator 290 coordinates the operations of work unit LCM. LCM coordinator 290 operates on per work unit type basis so that each work unit type to provide a service may be managed separately. Unless noted otherwise, the operations of LCM coordinator 290 are discussed herein for one work unit type.

[0045] LCM coordinator 290 includes several entities, such as a return probability estimation module 202 to estimate the return probability of a work unit in Unhealthy state 110, a work unit autoscaler 204 to start new work units to Starting state 102, a work unit tester 206 to determine the nature of fault detection (leading to permanent fault or not at reference 160 / 156), and astarting time measurer 208 to determine the time that a work unit spends in the Starting state 102. While each entity is shown separately in the figure, two or more entities may be integrated into a single entity. Additionally, one or more entities may be implemented outside of LCM coordinator 290 (e.g., integrated with another function for a cloud system). Furthermore, the entities within LCM coordinator 290 may communicate with each other to share the input and output information.

[0046] Operations of each entity are discussed in further detail below.Estimate Return Probability

[0047] Return probability estimation module 202 takes as input notifications of work unit state transition events at reference 220, including (1) notification of fault, permanent fault, nonpermanent fault, (2) timing of event transitions (e.g., the time of transitions from Running state 104 to Unhealthy state 110, from Unhealthy state 110 to Backup state 108, from Unhealthy state 110 to Terminated state 116, and from Backup state 108 to Running state 104), and / or (3) fault status code / messages providing more details of the state transitions. Based on the input, return probability estimation module 202 provides the return probability RPi at reference 232 for work unit type z, which is the probability of a work unit recovering (transitioning from Unhealthy state 110 to Backup state 108) within a time period, TRi. The estimated return probability is then used by other entities / operations for fault mitigation.

[0048] The time period TRi for the probability determination may be configured based on one or more of the following factors: (1) After TRi, only a small fraction of any work units may recover, e.g., 95% of work units would recover within TRi,' (2) the time it takes to start a new work unit, Si, since that is the break-even time between waiting for recovery of the work unit at Unhealthy state 110 and creating a new replacement work unit; and (3) the work unit type, features of which corresponds to the time period selection.

[0049] A variety of features of the work unit type may determine the time period TRi. For example, the features may be the start command and the version of the work unit (e.g., the work unit may have the software image with specific version / size / compatibility that affects the time period). Some work units may be able to support several services and be within a common shared set of work units. These work units are bound to a particular service / work unit type only when they are configured in Running state 104. The versatility is accomplished by their software image containing multiple service software and common setup such as authentication, licensing, and configuration. Such work units can represent one of several work unit types, and the representation is based on what features each service utilizes. Once a work unit is initiated with the correct operative service (e.g., selected from several services it supports), the return probability may be determined.

[0050] The return probability may be set initially using one of a few options: (1) 0%, i.e., no Unhealthy work unit will be transitioned to Backup state 108, a pessimistic yet safe starting point; (2) 100%, optimistic one that may take too few resources to maintain sufficient work units to be in Running state 104; (3) the average probability of all work unit types’ return probabilities, which may be a realistic one; (4) the average probability of work units with most features in common; (5) a value based on a heuristic algorithm or machine learning obtained using historical return probabilities of the particular work unit types, all work unit types, or a subset of work unit types; and (6) a weighted combination of one or more of (1) to (5).

[0051] The return probability is then updated, e g., at regular intervals or based on when new event transition notifications are received. The update mechanism can be based on the same or different heuristic algorithm (e g., running averaging, which derives the average of the recent return probabilities) or machine learning as used to set the initial return probability.

[0052] When machine learning is used to set or update the return probability, one or more machine learning models may be deployed for the learning. The machine learning models may use supervised learning, unsupervised learning, semi-supervised learning, or other types of learning to predict the return probability. It can use artificial neural networks, decision trees, support-vector machines, regression analysis, Bayesian networks, genetic algorithms, or any other framework. When artificial neural networks are used, the first layer may receive as input data the work unit features, event transition notifications, and / or current return probabilities, the later hidden layers (potentially fully connected) may further process the data, and the output layer may then provide the determined return probability.

[0053] The machine learning models may be trained with the historical return probability of work units in the present cloud system or other cloud systems, and verified subsequent data of unhealthy work units being transitioned Backup state 108 (or not) to calibrate the machine learning model and provide more accurate initial and / or updated return probabilities for the work unit type.Measure Starting Time

[0054] Starting time measurer 208 determines the starting time, Si, of work units of each type, z. A work unit is triggered to start at some point, and with the notification of a work unit entering Starting state at reference 228, starting time measurer 208 determines the time until the work unit becomes ready for operations (transitioning from Starting state 102 to Backup state 108). Such determined time is provided as starting time measurement 238.

[0055] When a new service is set up, faults often do not happen before M work units are created, and the initial Si of pod type Pi can be determined based on previous starting times, e.g. by mean, median, max or other weighted value over them. Some services are updated duringoperation, which results in that a new work unit image version is to be used, typically replaced by few components within (e.g., replacing one or more containers within a pod where the pod is the work unit). During such update, the Si value could gradually be weighted from value corresponding to previous type i to the new type j. After the initial starting time is set, starting time measurer 208 adjusts Si based on measuring the time that work units stay within Starting state 102.Test Work Units

[0056] Work units in Unhealthy state 110 are tested to determine the nature of the detected fault. Work unit tester 206 provides the task in some embodiments, where work unit tester 206 receives the notification of a work unit entering Unhealthy state at reference 222, and determines and indicates the nature of the fault at fault determination 236.

[0057] Work units of type i are tested during an interval TRi. These tests can use standard liveness or readiness probes in some embodiments. For example, work unit tester 206 may replay of a previous request to a work unit, and see if the work unit responds correctly and in time, based on dropping the workload on the host of the work unit outside of the work unit (e.g., reducing the CPU load, the network load, and / or Input / output (I / O) load), and / or fixing or replacing hardware / software components with partial fault, overheating, configuration error through automatic management. These tests can also be combined with knowledge about the fault that occurred - whether the fault was due to timeout, incorrect response, or other more detailed errors based on internal operations. Knowledge about the fault can also lead to automatic declaring a work unit not faulty after an interval. Another indication could be how many times, how frequent or how long since the last fault was detected on the work unit, which could lead to the automatic decision of permanent fault when the fault occurs on the work unit at too high levels or within too short interval (e g., crossing the threshold of frequency or duration, respectively).Create New Work Units

[0058] As discussed, one or more new work units are created for a service based on the return probability and the number of work units entering Unhealthy state 110 in some embodiments. The creation is done by the work unit autoscaler 204 in some embodiments, where the name autoscaler indicates the automatic nature of work unit creation without manual intervention.

[0059] Work unit autoscaler 204 takes input of return probability 232 and the fault event notification 222 for the work unit type and produces a trigger to start a new work unit for that work unit type at reference 234. Optionally it takes a number of work units in Backup state 108 as shown at reference 224. The optional input can be useful to compensate for when the return probability RPi has been too optimistic or pessimistic to build a bias over time.

[0060] At the work unit startup stage, work unit autoscaler 204 may create a set of work units to perform task execution and provide a service, and such creation may be triggered by an external work unit creation request at reference 221. These work units are created at reference 234, transitioned to Backup state 108 and then Running state 104 to provide the service.

[0061] Then the additional work units may be created based on state transitions. In some embodiments, when the notification of a work unit entering Unhealthy state at reference 222 is received, work unit autoscaler 204 may perform the following operations: (1) updating an event counter 252, (2) determining whether enough fault events have occurred to indicate that a new work unit is needed, (3) triggering the creation of the new work unit if the determination is affirmative and resetting the event counter 252.

[0062] The event counter 252 counts the number of times that a work unit i entering Unhealthy state and the count is used to determine whether enough fault events have occurred.A number of heuristic algorithms and / or machine learning methods may be used for the determination, and these heuristic algorithms and corresponding machine learning methods may operate and / or be trained similar to the ones discussed herein above relating to ones determining the return probability. As noted earlier, for every l / (l-A z) times a work unit of type i enters Unhealthy state 110, a new work unit of type i is started in some embodiments. In these embodiments, when the count indicated by event counter 252 crossed the threshold ofa new work unit is started. Once the work unit begins starting up, the event counter 252 is reset to the default value (e.g., zero).

[0063] LCM coordinator 290 aims at maintaining the correct number of work units at Backup state 108, so that the flow of work units out of Backup state 108 is equal to the flow of work units into Backup state 108 over time. Such objective may be achieved through selecting a proper return probability RPi. When the return probability RPi is incorrect, a bias on the number of work units in Backup state 108 may be built up, and the number of work units in Backup state 108 at reference 224 can be used to offset the bias. Over time, the bias can be estimated and rectified for by terminating work units in Backup state 108 and / or transitioning work units to Backup state 108 periodically in some embodiments (such transition is not shown in the state machine in Figure 1). Additionally, when a work unit in Backup state 108 is shared among multiple services and can be recovered, the work unit may be reset to be ready to perform task for another service as needed. Such reset operation may benefit other work units as the reset allows the work unit to be transitioned to Running state 104 quicker and the resulting less resources it consumes while at standy can be reallocated to the other work units.

[0064] LCM coordinator 290 may set the initial number of work units at Backup state 108 in one of few ways. The initial backup number may be set to be based on one or more of (1) thenumber of work units to be configured in Running state 104 for the service, e.g., setting the initial backup number to be a fixed percentage of a work unit number (x %) of a threshold number of work unit type i, where the threshold number m is the one to provide a service at a given QoS for the corresponding service; (2) the minimum number of work units to continue meeting the requirements during fault free operation (i.e., the number of work units feasible to provide the service when no further fault work units being detected); (3) the number of faulty work units allowed for the service (e.g., the high the fault tolerance, the less initial backup number is set), (4) how long it takes for a work unit to transition from Starting state 104 to Backup state 108 (measured starting time); (5) the duration of time period, TRi.

[0065] Through proactively triggering the creation of new work units (at reference 234) when necessary (based on the return probability as discussed) and maintaining a set of work units in Backup state 108, LCM coordinator 290 does not wait for the fault determination to identify the nature of a work unit fault detection for work unit creation, Instead, by maintaining a sufficient number of work units in Running state 104 through starting new work units proactively without waiting on the whole duration of new work unit creation, LCM coordinator 290 provides timely and sufficient responses to maintain QoS of the service served by the working units in Running state 104.Exemplary Fault Mitigation with Additional States

[0066] Figure 3 illustrates components of a cloud system in which fault mitigation may be performed per some embodiments. In some embodiments, Cloud system 300 implements Kubemetes architecture.

[0067] The work units in Cloud system 300 are pods, each being the smallest deployment unit. Controller 364 creates pods, monitors their health and numbers, and performs management actions. LCM coordinator 290 is implemented for fault mitigation in some embodiments.

[0068] In Cloud system 300, a pod is an abstraction entity that hosts one or more containers, each of which packs the necessary libraries, dependencies, and other resources necessary for an application to function independently. The pod creates a wrapper with dependencies to allow a controller to manage the container. A node may run one or more pods, and it refers to bare metal or virtual machine that are responsible for hosting pods. The pods reside on work nodes, while another type of nodes is the master node, which features a control plane (with software modules implementing in hardware logic / circuit) that controls pod scheduling of the pods on the worker nodes (e.g., selecting a suitable worker node and / or initiating the creation / termination of the pods on the worker node). The exampalary control plane operations are shown at Figure 4 with control plane logic / circuit 414. A cluster is a group of nodes with at least one master node and multiple worker nodes.

[0069] LCM coordinator 290 may reside in or couple with controller 364 to mitigate faults in Cloud system 300, which include multiple clusters 360 to 362. Each cluster may include multiple nodes; for example cluster 360 has nodes 332 to 334. Each node may include one or more pods. As shown, node 332 has pods 322 and 324, while node 334 has pods 326 and 328. The pods may be single-container pods like pods 322 and 328 that contain containers 302 and 312, respectively; and they may be multi-container pods like 324 and 326 that contains 304 to 306 and 314 to 316, respectively.

[0070] While LCM coordinator 290 can be used to mitigate faults detected at the pod level, and replace pods through operations in the state machine shown of Figure 1 and functional entities of Figure 2, it may also be used to mitigate faults at the container level, when containers implement similar state transitions.

[0071] Figure 4 illustrates fault mitigation with additional states at the pod level in a cloud system per additional embodiments. The LCM coordinator 290 performs the fault mitigation, with return probability estimation module 202, work unit autoscaler 204, and work unit tester 206 shown within. The starting time measurer 208 is not shown as its operation of providing the starting time is done only once, for pod 1 at reference 410 in the pod startup stage 416.

[0072] The workflow is divided into several stages, where the initial stage is the pod startup stage at reference 416, where a control plane logic / circuit 414 causes pod 1 (Pl) at reference 410 to start. The starting may be based on an external work unit creation request (e.g., the one at reference 221) to work unit autoscaler 204, which causes Pl 410 to transition from Starting state 102 to Backup state 108 and then Running state 104.

[0073] Then the flow enters to pod fault stage 418, when a fault is detected by fault detector 408, which is a software module or hardware logic / circuit coupled to LCM coordinator 290 and the pods. The fault detection causes Pl 410 to transition to Unhealthy state 110 and an event notification to be sent to return probability estimation module 202, which determines the return probability of Pl 410. Event counter 252 counts the number of fault detection since last reset, and when the count exceeds a threshold (e.g., the threshold of l / (l-RPi), the flow goes to pod fault count exceeded stage 420, and LCM coordinator 290 initiates creation of new pod, and control plane logic / circuit 414 receives the initiation and causes Pod 2 (P2) 412 to be started. Note the creation of pod is initiated based on the threshold crossing, and whether the pod fault being permanent or not has not been determined yet.

[0074] Work unit tester 206 then determines the nature of the fault detection. In the first alternative, the pod is recovered within a time period thus not faulty as shown at reference 422. Such determination is provided to return probability estimation module 202, which updates thereturn probability for the pod type, RP (as noted, LCM coordinator 290 operates on a per pod type basis, and operations in Figure 4 shows two pods, Pl and P2, of the same pod type).

[0075] In the second alternative, the pod is determined to be at permanent fault as shown at reference 424. Such determination is provided to return probability estimation module 202, which also updates the return probability RPi. And Pl 410 is terminated by control plane logic / circuit 414.Operations per some Embodiments

[0076] Figure 5 is a flow diagram illustrating operations of fault mitigation with additional states in a cloud system per some embodiments. The operations of method 500 may be implemented in an electronic device (e.g., electronic device 602) that includes LCM coordinator 290.

[0077] At reference 502, a first work unit in the cloud system is moved from a running state to a first state upon determining that the first work unit is faulty, where in the running state, the first work unit performed task execution to provide a service of the cloud system. The running state is shown as Running state 104 herein above.

[0078] At reference 504, responsive to the first work unit being moved to the first state, a set of work units is caused to be started to provide the service based on a probability of a work unit in the first state being moved to a second state, where in the second state the work unit is standby for task execution to provide the service. The first and second states are shown in Figure 1 at references 110 and 108 as Unhealthy state and Backup state, respectively.

[0079] At reference 506, a second work unit is moved from the first state to the second state upon determining within a time period that the second work unit is ready for the service. That is, when the second work unit is determined not having a permanent fault within the time period, it is transitioned into the second state (Backup state 108).

[0080] At reference 508, a third work unit is caused to perform task execution by moving the third work unit from the second state to the running state upon determining that one or more additional work units are needed to provide the service.

[0081] The first, the second, and the third work units are the same type of work units to provide the service in some embodiments. One or more of the work units may be able to support multiple services and can be reset so it switches from supporting one service to another. In some embodiments, each of the first, second, and third work units is a pod in a Kubemetes architecture.

[0082] In some embodiments, the method further includes, upon determining within the time period that the second work unit remains faulty, terminating the second work unit at reference 510. That is, the second work unit enters Terminated state 116.

[0083] In some embodiments, a default number of work units are to be placed into the second state, and wherein the default number is determined based on one or more of: a number of work units to be configured in the running state for the service; a number of work units feasible to provide the service when no further fault work units being detected; a number of faulty work units allowed for the service; how long it takes for a work unit to transition from the starting state to the second state; and duration of the time period.

[0084] In some embodiments, the duration of the time period is determined based on one or more of: likelihood that a work unit to provide the service is determined to be ready for the service after the time period (e g., after the time period, only a small fraction of any work units may recover as noted herein above); how long it takes for the work unit to transition from the starting state to the running state; and the work unit’s type.

[0085] In some embodiments, the work unit’s type is based on one or more of services to be supported by the work unit and the version of the work unit.

[0086] In some embodiments, the probability of a work unit in the first state being moved to the second state is learned based on state transition events of a plurality of work units to provide the service. The probability is return probability, which may be set and / or updated based on machine learning as discussed herein above.

[0087] In some embodiments, the probability of a work unit in the first state being moved to the second state is learned based on counting a first number of work units for the service transitioning from the first state to the second state within in an observation period and a second number of work units for the service in the first state within the observation period. For example, the probability may be set to be the division of the first number by the second number, as the first number represents the number of work units recovers to Backup state 108 from Unhealthy state 110 in a time period while the second number represents the number of work units entering Backup state 108 from Running state 104 in the same time period.

[0088] In some embodiments, starting the set of work units based on the probability of a work unit in the first state being moved to the second state comprises counting, by a counter, how many work units have been moved to the first state since the counter is set to a default value. The event counter 252 performs the counting as discussed herein above.

[0089] In some embodiments, the set of work units is caused to be started when a threshold is crossed by a value that is derived based on: the counter value, and the probability of a work unit being moved to the first state and then being determined within the time period to be ready for the service. For example, when the counter value is over l / (l-RPi) as discussed herein above, the set of work units is to be started.

[0090] In some embodiments, upon starting the set of work units, the counter is reset to the default value, wherein a further work unit being determined to be faulty causes the counter to start counting. The reset and counting are discussed relating to event counter 252.

[0091] In some embodiments, starting the set of work units is further based on how many work units are in the second state standby already for task execution to provide the service. The number of work units in the second state (Backup state 108) is discussed in more detail relating to work unit autoscaler 204.

[0092] Through these embodiments, a cloud system may proactively start a set of work units under a certain condition thus maintaining a sufficient number of work units in a backup state (second state) so that work units in the backup state may be transitioned to the running state quickly without waiting on the whole duration of new work unit creation. The proactivecreation of the work unit allows these embodiments to mitigate faults faster than the reactive creation of new work units upon the determination that a running work unit has a permanent fault and needs to be replaced.Devices and Environments for Implementing Embodiments of the Invention

[0093] Figure 6 illustrates an electronic device to mitigate fault in a cloud system per some embodiments. The electronic device may be a host in a cloud system, or a network node in a wireless / wireline network, and the operating environment and further embodiments the host and the network node are discussed in more details herein below. The electronic device 602 may be implemented using custom application-specific integrated-circuits (ASICs) as processors and a special-purpose operating system (OS), or common off-the-shelf (COTS) processors and a standard OS. In some embodiments, the electronic device 602 implements the LCM coordinator 290 that performs operations discussed herein relating to Figures 1 to 5.

[0094] The electronic device 602 includes hardware 640 comprising a set of one or more processors 642 (which are typically COTS processors or processor cores or ASICs) and physical NIs 646, as well as non-transitory machine-readable storage media 649 having stored therein software 650. During operation, the one or more processors 642 may execute the software 650 to instantiate one or more sets of one or more applications 664A-R. While one embodiment does not implement virtualization, alternative embodiments may use different forms of virtualization. For example, in one such alternative embodiment, the virtualization layer 654 represents the kernel of an operating system (or a shim executing on a base operating system) that allows for the creation of multiple instances 662A-R called software containers that may each be used to execute one (or more) of the sets of applications 664A-R. The multiple software containers (also called virtualization engines, virtual private servers, or jails) are user spaces (typically a virtual memory' space) that are separate from each other and separate from the kernel space in which theoperating system is run. The set of applications running in a given user space, unless explicitly allowed, cannot access the memory of the other processes. In another such alternative embodiment, the virtualization layer 654 represents a hypervisor (sometimes referred to as a virtual machine monitor (VMM)) or a hypervisor executing on top of a host operating system, and each of the sets of applications 664A-R run on top of a guest operating system within an instance 662A-R called a virtual machine (which may in some cases be considered a tightly isolated form of software container) that run on top of the hypervisor - the guest operating system and application may not know that they are running on a virtual machine as opposed to running on a “bare metal” host electronic device, or through para-virtualization the operating system and / or application may be aware of the presence of virtualization for optimization purposes. In yet other alternative embodiments, one, some, or all of the applications are implemented as unikernel(s), which can be generated by compiling directly with an application only a limited set of libraries (e.g., from a library operating system (LibOS) including drivers / libraries of OS services) that provide the particul r OS services needed by the application. As a unikernel can be implemented to run directly on hardware 640, directly on a hypervisor (in which case the unikernel is sometimes described as running within a LibOS virtual machine), or in a software container, embodiments can be implemented fully with unikemels running directly on a hypervisor represented by virtualization layer 654, unikemels running within software containers represented by instances 662A-R, or as a combination of unikemels and the above-described techniques (e.g., unikemels and virtual machines both run directly on a hypervisor, unikemels, and sets of applications that are run in different software containers).

[0095] The software 650 contains LCM coordinator 290 that performs operations described with reference to operations as discussed relating to Figures 1 to 5. The LCM coordinator 290 may be instantiated within the applications 664A-R. The instantiation of the one or more sets of one or more applications 664A-R, as well as virtualization if implemented, are collectively referred to as software instance(s) 652. Each set of applications 664A-R, corresponding virtualization construct (e g., instance 662A-R) if implemented, and that part of the hardware 640 that executes them (be it hardware dedicated to that execution and / or time slices of hardware temporally shared), forms a separate virtual electronic device 660A-R.

[0096] A network interface (NI) may be physical or virtual. In the context of Internet Protocol (IP), an interface address is an IP address assigned to an NI, be it a physical NI or virtual NI. A virtual NI may be associated with a physical NI, with another virtual interface, or stand on its own (e.g., a loopback interface, a point-to-point protocol interface). A NI (physical or virtual) may be numbered (a NI with an IP address) or unnumbered (a NI without an IP address). The NIis shown as network interface card (NIC) 644. The physical network interface 646 may include one or more antenna of the electronic device 602. An antenna port may or may not correspond to a physical antenna. The antenna comprises one or more radio interfaces.A Wireless Network per Some Embodiments

[0097] Figure 7 illustrates an example of a communication system 700 per some embodiments. In the example, the communication system 700 includes a telecommunication network 702 that includes an access network 704, such as a radio access network (RAN), and a core network 706, which includes one or more core network nodes 708. The access network 704 includes one or more access network nodes, such as network nodes 710A and 710B (one or more of which may be generally referred to as network nodes 710), or any other similar 3rdGeneration Partnership Project (3 GPP) access nodes or non-3GPP access points. Moreover, as will be appreciated by those of skill in the art, a network node is not necessarily limited to an implementation in which a radio portion and a baseband portion are supplied and integrated by a single vendor. Thus, it will be understood that network nodes include disaggregated implementations or portions thereof. For example, in some embodiments, the telecommunication network 702 includes one or more Open-RAN (ORAN) network nodes. An ORAN network node is a node in the telecommunication network 702 that supports an ORAN specification (e.g., a specification published by the O-RAN Alliance, or any similar organization) and may operate alone or together with other nodes to implement one or more functionalities of any node in the telecommunication network 702, including one or more network nodes 710 and / or core network nodes 708.

[0098] Examples of an ORAN network node include an open radio unit (O-RU), an open distributed unit (O-DU), an open central unit (O-CU), including an O-CU control plane (O-CU- CP) or an O-CU user plane (O-CU-UP), a RAN intelligent controller (near-real time or non-real time) hosting software or software plug-ins, such as a near-real time control application (e.g., xApp) or a non-real time control application (e.g., rApp), or any combination thereof (the adjective “open” designating support of an ORAN specification). The network node may support a specification by, for example, supporting an interface defined by the ORAN specification, such as an Al, Fl, Wl, El, E2, X2, Xn interface, an open fronthaul user plane interface, or an open fronthaul management plane interface. Moreover, an ORAN access node may be a logical node in a physical node. Furthermore, an ORAN network node may be implemented in a virtualization environment (described further below) in which one or more network functions are virtualized. For example, the virtualization environment may include an O-Cloud computing platform orchestrated by a Service Management and Orchestration Framework via an O-2 interface defined by the O-RAN Alliance or comparable technologies.The network nodes 710 facilitate direct or indirect connection of user equipment (UE), such as by connecting UEs 712A, 712B, 712C, and 712D (one or more of which may be generally referred to as UEs 712) to the core network 706 over one or more wireless connections.

[0099] Example wireless communications over a wireless connection include transmitting and / or receiving wireless signals using electromagnetic waves, radio waves, infrared waves, and / or other types of signals suitable for conveying information without the use of wires, cables, or other material conductors. Moreover, in different embodiments, the communication system 700 may include any number of wired or wireless networks, network nodes, UEs, and / or any other components or systems that may facilitate or participate in the communication of data and / or signals whether via wired or wireless connections. The communication system 700 may include and / or interface with any type of communication, telecommunication, data, cellular, radio network, and / or other similar type of system.

[0100] The UEs 712 may be any of a wide variety of communication devices, including wireless devices arranged, configured, and / or operable to communicate wirelessly with the network nodes 710 and other communication devices. Similarly, the network nodes 710 are arranged, capable, configured, and / or operable to communicate directly or indirectly with the UEs 712 and / or with other network nodes or equipment in the telecommunication network 702 to enable and / or provide network access, such as wireless network access, and / or to perform other functions, such as administration in the telecommunication network 702.

[0101] In the depicted example, the core network 706 connects the network nodes 710 to one or more hosts, such as host 716. These connections may be direct or indirect via one or more intermediary networks or devices. In other examples, network nodes may be directly coupled to hosts. The core network 706 includes one more core network nodes (e.g., core network node 708) that are structured with hardware and software components. Features of these components may be substantially similar to those described with respect to the UEs, network nodes, and / or hosts, such that the descriptions thereof are generally applicable to the corresponding components of the core network node 708. Example core network nodes include functions of one or more of a Mobile Switching Center (MSC), Mobility Management Entity (MME), Home Subscriber Server (HSS), Access and Mobility Management Function (AMF), Session Management Function (SMF), Authentication Server Function (AUSF), Subscription Identifier De-concealing function (SIDF), Unified Data Management (UDM), Security Edge Protection Proxy (SEPP), Network Exposure Function (NEF), and / or a User Plane Function (UPF).

[0102] The host 716 may be under the ownership or control of a service provider other than an operator or provider of the access network 704 and / or the telecommunication network 702, and may be operated by the service provider or on behalf of the service provider. The host 716may host a variety of applications to provide one or more service. Examples of such applications include live and pre-recorded audio / video content, data collection services such as retrieving and compiling data on various ambient conditions detected by a plurality of UEs, analytics functionality, social media, functions for controlling or otherwise interacting with remote devices, functions for an alarm and surveillance center, or any other such function performed by a server.

[0103] As a whole, the communication system 700 of Figure 7 enables connectivity between the UEs, network nodes, and hosts. In that sense, the communication system may be configured to operate according to predefined rules or procedures, such as specific standards that include, but are not limited to: Global System for Mobile Communications (GSM); Universal Mobile Telecommunications System (UMTS); Long Term Evolution (LTE), and / or other suitable 2G, 3G, 4G, 5G standards, or any applicable future generation standard (e g., 6G); wireless local area network (WLAN) standards, such as the Institute of Electrical and Electronics Engineers (IEEE) 802.11 standards (WiFi); and / or any other appropriate wireless communication standard, such as the Worldwide Interoperability for Microwave Access (WiMax), Bluetooth, Z-Wave, Near Field Communication (NFC) ZigBee, LiFi, and / or any low-power wide-area network (LPWAN) standards such as LoRa and Sigfox.

[0104] In some examples, the telecommunication network 702 is a cellular network that implements 3GPP standardized features. Accordingly, the telecommunication network 702 may support network slicing to provide different logical networks to different devices that are connected to the telecommunication network 702. For example, the telecommunication network 702 may provide Ultra Reliable Low Latency Communication (URLLC) services to some UEs, while providing Enhanced Mobile Broadband (eMBB) services to other UEs, and / or Massive Machine Type Communication (mMTC) / Massive loT services to yet further UEs.

[0105] In some examples, the UEs 712 are configured to transmit and / or receive information without direct human interaction. For instance, a UE may be designed to transmit information to the access network 704 on a predetermined schedule, when triggered by an internal or external event, or in response to requests from the access network 704. Additionally, a UE may be configured for operating in single- or multiple radio access technology (multi-RAT) or multistandard mode. For example, a UE may operate with any one or combination of Wi-Fi, NR (New Radio) and LTE, i.e., being configured for multi-radio dual connectivity (MR-DC), such as E-UTRAN (Evolved-UMTS Terrestrial Radio Access Network) New Radio - Dual Connectivity (EN-DC).

[0106] In the example, the hub 714 communicates with the access network 704 to facilitate indirect communication between one or more UEs (e.g., UE 712C and / or 712D) and networknodes (e.g., network node 710B). In some examples, the hub 714 may be a controller, router, content source and analytics, or any of the other communication devices described herein regarding UEs. For example, the hub 714 may be a broadband router enabling access to the core network 706 for the UEs. As another example, the hub 714 may be a controller that sends commands or instructions to one or more actuators in the UEs. Commands or instructions may be received from the UEs, network nodes 710, or by executable code, script, process, or other instructions in the hub 714. As another example, the hub 714 may be a data collector that acts as temporary storage for UE data and, in some embodiments, may perform analysis or other processing of the data. As another example, the hub 714 may be a content source. For example, for a UE that is a virtual reality (VR) headset, display, loudspeaker or other media delivery device, the hub 714 may retrieve VR assets, video, audio, or other media or data related to sensory information via a network node, which the hub 714 then provides to the UE either directly, after performing local processing, and / or after adding additional local content. In still another example, the hub 714 acts as a proxy server or orchestrator for the UEs, in particular if one or more of the UEs are low energy loT devices.

[0107] The hub 714 may have a constant / persistent or intermittent connection to the network node 710b. The hub 714 may also allow for a different communication scheme and / or schedule between the hub 714 and UEs (e.g., UE 712C and / or 712D), and between the hub 714 and the core network 706. In other examples, the hub 714 is connected to the core network 706 and / or one or more UEs via a wired connection. Moreover, the hub 714 may be configured to connect to a machine-to-machine (M2M) service provider over the access network 704 and / or to another UE over a direct connection. In some scenarios, UEs may establish a wireless connection with the network nodes 710 while still connected via the hub 714 via a wired or wireless connection. In some embodiments, the hub 714 may be a dedicated hub - that is, a hub whose primary function is to route communications to / from the UEs from / to the network node 710B. In other embodiments, the hub 714 may be a non-dedicated hub - that is, a device which is capable of operating to route communications between the UEs and network node 710B, but which is additionally capable of operating as a communication start and / or end point for certain data channels. In some embodiments, the electronic device that implements the LCM coordinator 290 may be or coupled to network nodes 708, 710A-B, or host 717.UE per Some Embodiments

[0108] Figure 8 illustrates a UE 800 per some embodiments. As used herein, a UE refers to a device capable, configured, arranged and / or operable to communicate wirelessly with network nodes and / or other UEs. Examples of a UE include, but are not limited to, a smart phone, mobile phone, cell phone, voice over IP (VoIP) phone, wireless local loop phone, desktop computer,personal digital assistant (PDA), wireless cameras, gaming console or device, music storage device, playback appliance, wearable terminal device, wireless endpoint, mobile station, tablet, laptop, laptop-embedded equipment (LEE), laptop-mounted equipment (LME), smart device, wireless customer-premise equipment (CPE), vehicle, vehicle-mounted or vehicle embedded / integrated wireless device, etc. Other examples include any UE identified by the 3rd Generation Partnership Project (3GPP), including a narrow band internet of things (NB-IoT) UE, a machine type communication (MTC) UE, and / or an enhanced MTC (eMTC) UE.

[0109] A UE may support device-to-device (D2D) communication, for example by implementing a 3 GPP standard for sidelink communication, Dedicated Short-Range Communication (DSRC), vehicle-to-vehicle (V2V), vehicle-to-infrastructure (V2I), or vehicle- to-everything (V2X). In other examples, a UE may not necessarily have a user in the sense of a human user who owns and / or operates the relevant device. Instead, a UE may represent a device that is intended for sale to, or operation by, a human user but which may not, or which may not initially, be associated with a specific human user (e.g., a smart sprinkler controller).Alternatively, a UE may represent a device that is not intended for sale to, or operation by, an end user but which may be associated with or operated for the benefit of a user (e.g., a smart power meter).

[0110] The UE 800 includes processing circuitry 802 that is operatively coupled via a bus 804 to an input / output interface 806, a power source 808, a memory 810, a communication interface 812, and / or any other component, or any combination thereof. Certain UEs may utilize all or a subset of the components shown in Figure 8. The level of integration between the components may vary from one UE to another UE. Further, certain UEs may contain multiple instances of a component, such as multiple processors, memories, transceivers, transmitters, receivers, etc.

[0111] The processing circuitry 802 is configured to process instructions and data and may be configured to implement any sequential state machine operative to execute instructions stored as machine-readable computer programs in the memory 810. The processing circuitry 802 may be implemented as one or more hardware-implemented state machines (e.g., in discrete logic, field- programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), etc.); programmable logic together with appropriate firmware; one or more stored computer programs, general -purpose processors, such as a microprocessor or digital signal processor (DSP), together with appropriate software; or any combination of the above. For example, the processing circuitry 802 may include multiple central processing units (CPUs).

[0112] In the example, the input / output interface 806 may be configured to provide an interface or interfaces to an input device, output device, or one or more input and / or outputdevices. Examples of an output device include a speaker, a sound card, a video card, a display, a monitor, a printer, an actuator, an emitter, a smartcard, another output device, or any combination thereof. An input device may allow a user to capture information into the UE 800. Examples of an input device include a touch-sensitive or presence-sensitive display, a camera (e.g., a digital camera, a digital video camera, a web camera, etc.), a microphone, a sensor, a mouse, a trackball, a directional pad, a trackpad, a scroll wheel, a smartcard, and the like The presence-sensitive display may include a capacitive or resistive touch sensor to sense input from a user. A sensor may be, for instance, an accelerometer, a gyroscope, a tilt sensor, a force sensor, a magnetometer, an optical sensor, a proximity sensor, a biometric sensor, etc , or any combination thereof. An output device may use the same type of interface port as an input device. For example, a Universal Serial Bus (USB) port may be used to provide an input device and an output device.

[0113] In some embodiments, the power source 808 is structured as a battery or battery pack. Other types of power sources, such as an external power source (e.g., an electricity outlet), photovoltaic device, or power cell, may be used. The power source 808 may further include power circuitry for delivering power from the power source 808 itself, and / or an external power source, to the various parts of the UE 800 via input circuitry or an interface such as an electrical power cable. Delivering power may be, for example, for charging of the power source 808. Power circuitry may perform any formatting, converting, or other modification to the power from the power source 808 to make the power suitable for the respective components of the UE 800 to which power is supplied.

[0114] The memory 810 may be or be configured to include memory such as random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable readonly memory (EEPROM), magnetic disks, optical disks, hard disks, removable cartridges, flash drives, and so forth. In one example, the memory 810 includes one or more application programs 814, such as an operating system, web browser application, a widget, gadget engine, or other application, and corresponding data 816. The memory 810 may store, for use by the UE 800, any of a variety of various operating systems or combinations of operating systems.

[0115] The memory 810 may be configured to include a number of physical drive units, such as redundant array of independent disks (RAID), flash memory, USB flash drive, external hard disk drive, thumb drive, pen drive, key drive, high-density digital versatile disc (HD-DVD) optical disc drive, internal hard disk drive, Blu-Ray optical disc drive, holographic digital data storage (HDDS) optical disc drive, external mini-dual in-line memory module (DIMM), synchronous dynamic random access memory (SDRAM), external micro-DIMM SDRAM,smartcard memory such as tamper resistant module in the form of a universal integrated circuit card (UICC) including one or more subscriber identity modules (SIMs), such as a Universal Subscriber Identity Module (USIM) and / or IP Multimedia Services Identity Module (ISIM), other memory, or any combination thereof. The UICC may for example be an embedded UICC (eUICC), integrated UICC (iUICC) or a removable UICC commonly known as ‘ SIM card. ’ The memory 810 may allow the UE 800 to access instructions, application programs and the like, stored on transitory or non-transitory memory media, to off-load data, or to upload data. An article of manufacture, such as one utilizing a communication system may be tangibly embodied as or in the memory 810, which may be or comprise a device-readable storage medium.

[0116] The processing circuitry 802 may be configured to communicate with an access network or other network using the communication interface 812. The communication interface 812 may comprise one or more communication subsystems and may include or be communicatively coupled to an antenna 822. The communication interface 812 may include one or more transceivers used to communicate, such as by communicating with one or more remote transceivers of another device capable of wireless communication (e.g., another UE or a network node in an access network). Each transceiver may include a transmitter 818 and / or a receiver 820 appropriate to provide network communications (e.g., optical, electrical, frequency allocations, and so forth). Moreover, the transmitter 818 and receiver 820 may be coupled to one or more antennas (e.g., antenna 822) and may share circuit components, software or firmware, or alternatively be implemented separately.

[0117] In the illustrated embodiment, communication functions of the communication interface 812 may include cellular communication, Wi-Fi communication, LPWAN communication, data communication, voice communication, multimedia communication, short- range communications such as Bluetooth, near-field communication, location-based communication such as the use of the global positioning system (GPS) to determine a location, another like communication function, or any combination thereof. Communications may be implemented in according to one or more communication protocols and / or standards, such as IEEE 802.11, Code Division Multiplexing Access (CDMA), Wideband Code Division Multiple Access (WCDMA), GSM, LTE, New Radio (NR), UMTS, WiMax, Ethernet, transmission control protocol / internet protocol (TCP / IP), synchronous optical networking (SONET), Asynchronous Transfer Mode (ATM), Quick UDP Internet Connections (QUIC), Hypertext Transfer Protocol (HTTP), and so forth.

[0118] Regardless of the type of sensor, a UE may provide an output of data captured by its sensors, through its communication interface 812, via a wireless connection to a network node. Data captured by sensors of a UE can be communicated through a wireless connection to anetwork node via another UE. The output may be periodic (e.g., once every 15 minutes if it reports the sensed temperature), random (e g., to even out the load from reporting from several sensors), in response to a triggering event (e.g., when moisture is detected an alert is sent), in response to a request (e.g., a user initiated request), or a continuous stream (e.g., a live video feed of a patient).

[0119] As another example, a UE comprises an actuator, a motor, or a switch, related to a communication interface configured to receive wireless input from a network node via a wireless connection. In response to the received wireless input the states of the actuator, the motor, or the switch may change For example, the UE may comprise a motor that adjusts the control surfaces or rotors of a drone in flight according to the received input or to a robotic arm performing a medical procedure according to the received input.

[0120] A UE, when in the form of an Internet of Things (loT) device, may be a device for use in one or more application domains, these domains comprising, but not limited to, city wearable technology, extended industrial application and healthcare. Non-limiting examples of such an loT device are a device which is or which is embedded in: a connected refrigerator or freezer, a TV, a connected lighting device, an electricity meter, a robot vacuum cleaner, a voice controlled smart speaker, a home security camera, a motion detector, a thermostat, a smoke detector, a door / window sensor, a flood / moisture sensor, an electrical door lock, a connected doorbell, an air conditioning system like a heat pump, an autonomous vehicle, a surveillance system, a weather monitoring device, a vehicle parking monitoring device, an electric vehicle charging station, a smart watch, a fitness tracker, a head-mounted display for Augmented Reality (AR) or Virtual Reality (VR), a wearable for tactile augmentation or sensory enhancement, a water sprinkler, an animal- or item-tracking device, a sensor for monitoring a plant or animal, an industrial robot, an Unmanned Aerial Vehicle (UAV), and any kind of medical device, like a heart rate monitor or a remote controlled surgical robot. A UE in the form of an loT device comprises circuitry and / or software in dependence of the intended application of the loT device in addition to other components as described in relation to the UE 800 shown in Figure 8.

[0121] As yet another specific example, in an loT scenario, a UE may represent a machine or other device that performs monitoring and / or measurements, and transmits the results of such monitoring and / or measurements to another UE and / or a network node. The UE may in this case be an M2M device, which may in a 3 GPP context be referred to as an MTC device. As one particular example, the UE may implement the 3GPP NB-IoT standard. In other scenarios, a UE may represent a vehicle, such as a car, a bus, a truck, a ship and an airplane, or other equipment that is capable of monitoring and / or reporting on its operational status or other functions associated with its operation.

[0122] In practice, any number of UEs may be used together with respect to a single use case. For example, a first UE might be or be integrated in a drone and provide the drone’s speed information (obtained through a speed sensor) to a second UE that is a remote controller operating the drone. When the user makes changes from the remote controller, the first UE may adjust the throttle on the drone (e.g., by controlling an actuator) to increase or decrease the drone’s speed. The first and / or the second UE can also include more than one of the functionalities described above. For example, a UE might comprise the sensor and the actuator, and handle communication of data for both the speed sensor and the actuators.Network Node per Some Embodiments

[0123] Figure 9 illustrates a network node 900 per some embodiments. As used herein, network node refers to equipment capable, configured, arranged and / or operable to communicate directly or indirectly with a UE and / or with other network nodes or equipment, in a telecommunication network. Examples of network nodes include, but are not limited to, access points (APs) (e.g., radio access points), base stations (BSs) (e.g., radio base stations, Node Bs, evolved Node Bs (eNBs) and NR NodeBs (gNBs)), O-RAN nodes or components of an O-RAN node (e.g., radio base stations, Node Bs, evolved Node Bs (eNBs) and NR NodeBs (gNBs)).

[0124] Base stations may be categorized based on the amount of coverage they provide (or, stated differently, their transmit power level) and so, depending on the provided amount of coverage, may be referred to as femto base stations, pico base stations, micro base stations, or macro base stations. A base station may be a relay node or a relay donor node controlling a relay. A network node may also include one or more (or all) parts of a distributed radio base station such as centralized digital units, distributed units (e g., in an O-RAN access node) and / or remote radio units (RRUs), sometimes referred to as Remote Radio Heads (RRHs). Such remote radio units may or may not be integrated with an antenna as an antenna integrated radio. Parts of a distributed radio base station may also be referred to as nodes in a distributed antenna system (DAS).

[0125] Other examples of network nodes include multiple transmission point (multi-TRP) 5G access nodes, multi -standard radio (MSR) equipment such as MSR BSs, network controllers such as radio network controllers (RNCs) or base station controllers (BSCs), base transceiver stations (BTSs), transmission points, transmission nodes, multi-cell / multicast coordination entities (MCEs), Operation and Maintenance (O&M) nodes, Operations Support System (OSS) nodes, Self-Organizing Network (SON) nodes, positioning nodes (e.g., Evolved Serving Mobile Location Centers (E-SMLCs)), and / or Minimization of Drive Tests (MDTs).

[0126] The network node 900 includes a processing circuitry 902, a memory 904, a communication interface 906, and a power source 908. The network node 900 may be composedof multiple physically separate components (e.g., aNodeB component and a RNC component, or a BTS component and a BSC component, etc.), which may each have their own respective components. In certain scenarios in which the network node 900 comprises multiple separate components (e.g., BTS and BSC components), one or more of the separate components may be shared among several network nodes. For example, a single RNC may control multiple NodeBs. In such a scenario, each unique NodeB and RNC pair, may in some instances be considered a single separate network node. In some embodiments, the network node 900 may be configured to support multiple radio access technologies (RATs). In such embodiments, some components may be duplicated (e.g., separate memory 904 for different RATs) and some components may be reused (e.g., a same antenna 910 may be shared by different RATs). The network node 900 may also include multiple sets of the various illustrated components for different wireless technologies integrated into network node 900, for example GSM, WCDMA, LTE, NR, WiFi, Zigbee, Z-wave, LoRaWAN, Radio Frequency Identification (RFID) or Bluetooth wireless technologies. These wireless technologies may be integrated into the same or different chip or set of chips and other components within network node 900.

[0127] The processing circuitry 902 may comprise a combination of one or more of a microprocessor, controller, microcontroller, central processing unit, digital signal processor, application-specific integrated circuit, field programmable gate array, or any other suitable computing device, resource, or combination of hardware, software and / or encoded logic operable to provide, either alone or in conjunction with other network node 900 components, such as the memory 904, to provide network node 900 functionality.

[0128] In some embodiments, the processing circuitry 902 includes a system on a chip (SOC). In some embodiments, the processing circuitry 902 includes one or more of radio frequency (RF) transceiver circuitry 912 and baseband processing circuitry 914. In some embodiments, the radio frequency (RF) transceiver circuitry 912 and the baseband processing circuitry 914 may be on separate chips (or sets of chips), boards, or units, such as radio units and digital units. In alternative embodiments, part or all of RF transceiver circuitry 912 and baseband processing circuitry 914 may be on the same chip or set of chips, boards, or units.

[0129] The memory 904 may comprise any form of volatile or non-volatile computer- readable memory including, without limitation, persistent storage, solid-state memory, remotely mounted memory, magnetic media, optical media, random access memory (RAM), read-only memory (ROM), mass storage media (for example, a hard disk), removable storage media (for example, a flash drive, a Compact Disk (CD) or a Digital Video Disk (DVD)), and / or any other volatile or non-volatile, non-transitory device-readable and / or computer-executable memory devices that store information, data, and / or instructions that may be used by the processingcircuitry 902. The memory 904 may store any suitable instructions, data, or information, including a computer program, software, an application including one or more of logic, rules, code, tables, and / or other instructions capable of being executed by the processing circuitry 902 and utilized by the network node 900. The memory 904 may be used to store any calculations made by the processing circuitry 902 and / or any data received via the communication interface 906. In some embodiments, the processing circuitry 902 and memory 904 is integrated.

[0130] The communication interface 906 is used in wired or wireless communication of signaling and / or data between a network node, access network, and / or UE. As illustrated, the communication interface 906 comprises port(s) / terminal(s) 916 to send and receive data, for example to and from a network over a wired connection. The communication interface 906 also includes radio front-end circuitry 918 that may be coupled to, or in certain embodiments a part of, the antenna 910. Radio front-end circuitry 918 comprises filters 920 and amplifiers 922. The radio front-end circuitry 918 may be connected to an antenna 910 and processing circuitry 902. The radio front-end circuitry may be configured to condition signals communicated between antenna 910 and processing circuitry 902. The radio front-end circuitry 918 may receive digital data that is to be sent out to other network nodes or UEs via a wireless connection. The radio front-end circuitry 918 may convert the digital data into a radio signal having the appropriate channel and bandwidth parameters using a combination of filters 920 and / or amplifiers 922. The radio signal may then be transmitted via the antenna 910. Similarly, when receiving data, the antenna 910 may collect radio signals which are then converted into digital data by the radio front-end circuitry 918. The digital data may be passed to the processing circuitry 902. In other embodiments, the communication interface may comprise different components and / or different combinations of components.

[0131] In certain alternative embodiments, the network node 900 does not include separate radio front-end circuitry 918, instead, the processing circuitry 902 includes radio front-end circuitry and is connected to the antenna 910. Similarly, in some embodiments, all or some of the RF transceiver circuitry 912 is part of the communication interface 906. In still other embodiments, the communication interface 906 includes one or more ports or terminals 916, the radio front-end circuitry 918, and the RF transceiver circuitry 912, as part of a radio unit (not shown), and the communication interface 906 communicates with the baseband processing circuitry 914, which is part of a digital unit (not shown).

[0132] The antenna 910 may include one or more antennas, or antenna arrays, configured to send and / or receive wireless signals. The antenna 910 may be coupled to the radio front-end circuitry 918 and may be any type of antenna capable of transmitting and receiving data and / orsignals wirelessly. In certain embodiments, the antenna 910 is separate from the network node 900 and connectable to the network node 900 through an interface or port.

[0133] The antenna 910, communication interface 906, and / or the processing circuitry 902 may be configured to perform any receiving operations and / or certain obtaining operations described herein as being performed by the network node. Any information, data and / or signals may be received from a UE, another network node and / or any other network equipment. Similarly, the antenna 910, the communication interface 906, and / or the processing circuitry 902 may be configured to perform any transmitting operations described herein as being performed by the network node. Any information, data and / or signals may be transmitted to a UE, another network node and / or any other network equipment.

[0134] The power source 908 provides power to the various components of network node 900 in a form suitable for the respective components (e.g., at a voltage and current level needed for each respective component). The power source 908 may further comprise, or be coupled to, power management circuitry to supply the components of the network node 900 with power for performing the functionality described herein. For example, the network node 900 may be connectable to an external power source (e g., the power grid, an electricity outlet) via an input circuitry or interface such as an electrical cable, whereby the external power source supplies power to power circuitry of the power source 908. As a further example, the power source 908 may comprise a source of power in the form of a battery or battery pack which is connected to, or integrated in, power circuitry. The battery may provide backup power should the external power source fail.

[0135] Embodiments of the network node 900 may include additional components beyond those shown in Figure 9 for providing certain aspects of the network node’s functionality, including any of the functionality described herein and / or any functionality necessary to support the subject matter described herein. For example, the network node 900 may include user interface equipment to allow input of information into the network node 900 and to allow output of information from the network node 900. This may allow a user to perform diagnostic, maintenance, repair, and other administrative functions for the network node 900. In some embodiments, the network node 900 may implement the LCM coordinator 290 and perform operations described herein.Host per Some Embodiments

[0136] Figure 10 is a block diagram of a host 1000, which may be an embodiment of the host 716 of Figure 7, per various aspects described herein. As used herein, the host 1000 may be or comprise various combinations hardware and / or software, including a standalone server, a blade server, a cloud-implemented server, a distributed server, a virtual machine, container, orprocessing resources in a server farm. The host 1000 may provide one or more services to one or more UEs. In some embodiments, the host 1000 may implement LCM coordinator 290 and perform operations described herein.

[0137] The host 1000 includes processing circuitry 1002 that is operatively coupled via a bus 1004 to an input / output interface 1006, a network interface 1008, a power source 1010, and a memory 1012. Other components may be included in other embodiments. Features of these components may be substantially similar to those described with respect to the devices of previous figures, such as Figures 8 and 9, such that the descriptions thereof are generally applicable to the corresponding components of host 1000.

[0138] The memory 1012 may include one or more computer programs including one or more host application programs 1014 and data 1016, which may include user data, e.g., data generated by a UE for the host 1000 or data generated by the host 1000 for a UE. Embodiments of the host 1000 may utilize only a subset or all of the components shown. The host application programs 1014 may be implemented in a container-based architecture and may provide support for video codecs (e.g., Versatile Video Coding (VVC), High Efficiency Video Coding (HEVC), Advanced Video Coding (AVC), Moving Picture Experts Group (MPEG), VP9) and audio codecs (e.g., Free Lossless Audio Codec (FLAC), Advanced Audio Coding (AAC), MPEG, G.711), including transcoding for multiple different classes, types, or implementations of UEs (e.g., handsets, desktop computers, wearable display systems, heads-up display systems). The host application programs 1014 may also provide for user authentication and licensing checks and may periodically report health, routes, and content availability to a central node, such as a device in or on the edge of a core network. Accordingly, the host 1000 may select and / or indicate a different host for over-the-top services for a UE. The host application programs 1014 may support various protocols, such as the HTTP Live Streaming (HLS) protocol, Real-Time Messaging Protocol (RTMP), Real-Time Streaming Protocol (RTSP), Dynamic Adaptive Streaming over HTTP (MPEG-DASH), etc.Virtualization Environment per Some Embodiments

[0139] Figure 11 is a block diagram illustrating a virtualization environment 1100 in which functions implemented by some embodiments may be virtualized. In the present context, virtualizing means creating virtual versions of apparatuses or devices which may include virtualizing hardware platforms, storage devices and networking resources. As used herein, virtualization can be applied to any device described herein, or components thereof, and relates to an implementation in which at least a portion of the functionality is implemented as one or more virtual components. Some or all of the functions described herein may be implemented as virtual components executed by one or more virtual machines (VMs) implemented in one ormore virtual environments 1100 hosted by one or more of hardware nodes, such as a hardware computing device that operates as a network node, UE, core network node, or host. Further, in embodiments in which the virtual node does not require radio connectivity (e.g., a core network node or host), then the node may be entirely virtualized. In some embodiments, the virtualization environment 1100 includes components defined by the 0-RAN Alliance, such as an O-Cloud environment orchestrated by a Service Management and Orchestration Framework via an 0-2 interface. The virtualization environment may also implement pods and containers as shown in Figure 3.

[0140] Applications 1102 (which may alternatively be called software instances, virtual appliances, network functions, virtual nodes, virtual network functions, etc.) are run in the virtualization environment Q500 to implement some of the features, functions, and / or benefits of some of the embodiments disclosed herein. In some embodiments, Applications 1102 may implement the LCM coordinator 290 and perform operations described herein.

[0141] Hardware 1104 includes processing circuitry, memory that stores software and / or instructions executable by hardware processing circuitry, and / or other hardware devices as described herein, such as a network interface, input / output interface, and so forth. Software may be executed by the processing circuitry to instantiate one or more virtualization layers 1106 (also referred to as hypervisors or virtual machine monitors (VMMs)), provide VMs 1108A and 1108B (one or more of which may be generally referred to as VMs 1108), and / or perform any of the functions, features and / or benefits described in relation with some embodiments described herein. The virtualization layer 1106 may present a virtual operating platform that appears like networking hardware to the VMs 1108.

[0142] The VMs 1108 comprise virtual processing, virtual memory, virtual networking or interface and virtual storage, and may be run by a corresponding virtualization layer 1106. Different embodiments of the instance of a virtual appliance 1102 may be implemented on one or more of VMs 1108, and the implementations may be made in different ways. Virtualization of the hardware is in some contexts referred to as network function virtualization (NFV). NFV may be used to consolidate many network equipment types onto industry standard high volume server hardware, physical switches, and physical storage, which can be located in data centers, and customer premise equipment.

[0143] In the context of NFV, a VM 1108 may be a software implementation of a physical machine that runs programs as if they were executing on a physical, non-virtualized machine. Each of the VMs 1108, and that part of hardware 1104 that executes that VM, be it hardware dedicated to that VM and / or hardware shared by that VM with others of the VMs, forms separate virtual network elements. Still in the context of NFV, a virtual network function isresponsible for handling specific network functions that run in one or more VMs 1108 on top of the hardware 1104 and corresponds to the application 1102.

[0144] Hardware 1104 may be implemented in a standalone network node with generic or specific components. Hardware 1104 may implement some functions via virtualization. Alternatively, hardware 1104 may be part of a larger cluster of hardware (e.g., such as in a data center or CPE) where many hardware nodes work together and are managed via management and orchestration 1110, which, among others, oversees lifecycle management of applications 1102. In some embodiments, hardware 1104 is coupled to one or more radio units that each include one or more transmitters and one or more receivers that may be coupled to one or more antennas. Radio units may communicate directly with other hardware nodes via one or more appropriate network interfaces and may be used in combination with the virtual components to provide a virtual node with radio capabilities, such as a radio access node or a base station. In some embodiments, some signaling can be provided with the use of a control system 1112 which may alternatively be used for communication between hardware nodes and radio units.Communication among host, network node, and UE per Some Embodiments

[0145] Figure 12 illustrates a communication diagram of a host 1202 communicating via a network node 1204 with a UE 1206 over a partially wireless per some embodiments. Example implementations, in accordance with various embodiments, of the UE (such as a UE 712A of Figure 7 and / or UE 800 of Figure 8), network node (such as a network node 710 of Figure 7 and / or network node 900 of Figure 9), and host (such as host 716 of Figure 7 and / or host 1000 of Figure 10) discussed in the preceding paragraphs will now be described with reference to Figure 12. Either host 1202 or network node 1204 may implement the LCM coordinator 290 and performs the operations discussed herein.

[0146] Like host 1000, embodiments of host 1202 include hardware, such as a communication interface, processing circuitry, and memory. The host 1202 also includes software, which is stored in or accessible by the host 1202 and executable by the processing circuitry. The software includes a host application that may be operable to provide a service to a remote user, such as the UE 1206 connecting via an over-the-top (OTT) connection 1250 extending between the UE 1206 and host 1202. In providing the service to the remote user, a host application may provide user data which is transmitted using the OTT connection 1250.

[0147] The network node 1204 includes hardware enabling it to communicate with the host 1202 and UE 1206. The connection 1260 may be direct or pass through a core network (like core network 706 of Figure 7) and / or one or more other intermediate networks, such as one or more public, private, or hosted networks. For example, an intermediate network may be a backbone network or the Internet.

[0148] The UE 1206 includes hardware and software, which is stored in or accessible by UE 1206 and executable by the UE’s processing circuitry. The software includes a client application, such as a web browser or operator-specific “app” that may be operable to provide a service to a human or non-human user via UE 1206 with the support of the host 1202. In the host 1202, an executing host application may communicate with the executing client application via the OTT connection 1250 terminating at the UE 1206 and host 1202. In providing the service to the user, the UE's client application may receive request data from the host's host application and provide user data in response to the request data. The OTT connection 1250 may transfer both the request data and the user data. The UE's client application may interact with the user to generate the user data that it provides to the host application through the OTT connection 1250.

[0149] The OTT connection 1250 may extend via a connection 1260 between the host 1202 and the network node 1204 and via a wireless connection 1270 between the network node 1204 and the UE 1206 to provide the connection between the host 1202 and the UE 1206. The connection 1260 and wireless connection 1270, over which the OTT connection 1250 may be provided, have been drawn abstractly to illustrate the communication between the host 1202 and the UE 1206 via the network node 1204, without explicit reference to any intermediary devices and the precise routing of messages via these devices.

[0150] As an example of transmitting data via the OTT connection 1250, in step 1208, the host 1202 provides user data, which may be performed by executing a host application. In some embodiments, the user data is associated with a particular human user interacting with the UE 1206. In other embodiments, the user data is associated with a UE 1206 that shares data with the host 1202 without explicit human interaction. In step 1210, the host 1202 initiates a transmission carrying the user data towards the UE 1206. The host 1202 may initiate the transmission responsive to a request transmitted by the UE 1206. The request may be caused by human interaction with the UE 1206 or by operation of the client application executing on the UE 1206. The transmission may pass via the network node 1204, in accordance with the teachings of the embodiments described throughout this disclosure. Accordingly, in step 1212, the network node 1204 transmits to the UE 1206 the user data that was carried in the transmission that the host 1202 initiated, in accordance with the teachings of the embodiments described throughout this disclosure. In step 1214, the UE 1206 receives the user data carried in the transmission, which may be performed by a client application executed on the UE 1206 associated with the host application executed by the host 1202.

[0151] In some examples, the UE 1206 executes a client application which provides user data to the host 1202. The user data may be provided in reaction or response to the data received from the host 1202. Accordingly, in step 1216, the UE 1206 may provide user data, which maybe performed by executing the client application. In providing the user data, the client application may further consider user input received from the user via an input / output interface of the UE 1206. Regardless of the specific manner in which the user data was provided, the UE 1206 initiates, in step 1218, transmission of the user data towards the host 1202 via the network node 1204. In step 1220, in accordance with the teachings of the embodiments described throughout this disclosure, the network node 1204 receives user data from the UE 1206 and initiates transmission of the received user data towards the host 1202. In step 1222, the host 1202 receives the user data carried in the transmission initiated by the UE 1206.

[0152] In an example scenario, factory status information may be collected and analyzed by the host 1202. As another example, the host 1202 may process audio and video data which may have been retrieved from a UE for use in creating maps. As another example, the host 1202 may collect and analyze real-time data to assist in controlling vehicle congestion (e.g., controlling traffic lights). As another example, the host 1202 may store surveillance video uploaded by a UE. As another example, the host 1202 may store or control access to media content such as video, audio, VR or AR which it can broadcast, multicast or unicast to UEs. As other examples, the host 1202 may be used for energy pricing, remote control of non-time critical electrical load to balance power generation needs, location services, presentation services (such as compiling diagrams etc. from data collected from remote devices), or any other function of collecting, retrieving, storing, analyzing and / or transmitting data.

[0153] In some examples, a measurement procedure may be provided for the purpose of monitoring data rate, latency and other factors on which the one or more embodiments improve. There may further be an optional network functionality for reconfiguring the OTT connection 1250 between the host 1202 and UE 1206, in response to variations in the measurement results. The measurement procedure and / or the network functionality for reconfiguring the OTT connection may be implemented in software and hardware of the host 1202 and / or UE 1206. In some embodiments, sensors (not shown) may be deployed in or in association with other devices through which the OTT connection 1250 passes; the sensors may participate in the measurement procedure by supplying values of the monitored quantities exemplified above, or supplying values of other physical quantities from which software may compute or estimate the monitored quantities. The reconfiguring of the OTT connection 1250 may include message format, retransmission settings, preferred routing etc.; the reconfiguring need not directly alter the operation of the network node 1204. Such procedures and functionalities may be known and practiced in the art. In certain embodiments, measurements may involve proprietary UE signaling that facilitates measurements of throughput, propagation times, latency and the like, by the host 1202. The measurements may be implemented in that software causes messages to betransmitted, in particular empty or ‘dummy’ messages, using the OTT connection 1250 while monitoring propagation times, errors, etc.

[0154] Although the computing devices described herein (e g., UEs, network nodes, hosts) may include the illustrated combination of hardware components, other embodiments may comprise computing devices with different combinations of components. It is to be understood that these computing devices may comprise any suitable combination of hardware and / or software needed to perform the tasks, features, functions and methods disclosed herein. Determining, calculating, obtaining or similar operations described herein may be performed by processing circuitry, which may process information by, for example, converting the obtained information into other information, comparing the obtained information or converted information to information stored in the network node, and / or performing one or more operations based on the obtained information or converted information, and as a result of said processing making a determination. Moreover, while components are depicted as single boxes located within a larger box, or nested within multiple boxes, in practice, computing devices may comprise multiple different physical components that make up a single illustrated component, and functionality may be partitioned between separate components. For example, a communication interface may be configured to include any of the components described herein, and / or the functionality of the components may be partitioned between the processing circuitry and the communication interface. In another example, non-computationally intensive functions of any of such components may be implemented in software or firmware and computationally intensive functions may be implemented in hardware.

[0155] In certain embodiments, some or all of the functionality described herein may be provided by processing circuitry executing instructions stored on in memory, which in certain embodiments may be a computer program product in the form of a non-transitory computer- readable storage medium. In alternative embodiments, some or all of the functionality may be provided by the processing circuitry without executing instructions stored on a separate or discrete device-readable storage medium, such as in a hard-wired manner. In any of those particular embodiments, whether executing instructions stored on a non-transitory computer- readable storage medium or not, the processing circuitry can be configured to perform the described functionality. The benefits provided by such functionality are not limited to the processing circuitry alone or to other components of the computing device, but are enjoyed by the computing device as a whole, and / or by end users and a wireless network generally.Terms

[0156] References in the specification to “one embodiment,” “an embodiment,” “an example embodiment,” and so forth, indicate that the embodiment described may include a particularfeature, structure, or characteristic, but every embodiment may not necessarily include the particular feature, structure, or characteristic. Moreover, such phrases are not necessarily referring to the same embodiment. Further, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is submitted that it is within the knowledge of one skilled in the art to affect such feature, structure, or characteristic in connection with other embodiments whether or not explicitly described.

[0157] The description and claims may use the terms “coupled” and “connected,” along with their derivatives. These terms are not intended as synonyms for each other. “Coupled” is used to indicate that two or more elements, which may or may not be in direct physical or electrical contact with each other, co-operate or interact with each other. “Connected” is used to indicate the establishment of wireless or wireline communication between two or more elements that are coupled with each other. A “set,” as used herein can refer to any whole number of items including one item.

[0158] An electronic device (such as the electronic device 602) stores and transmits (internally and / or with other electronic devices over a network) code (which is composed of software instructions and which is sometimes referred to as a computer program code or a computer program) and / or data using machine-readable media (also called computer-readable media), such as machine-readable storage media (e.g., magnetic disks, optical disks, solid state drives, read only memory (ROM), flash memory devices, phase change memory) and machine- readable transmission media (also called a carrier) (e.g., electrical, optical, radio, acoustical, or other form of propagated signals - such as carrier waves, infrared signals). Thus, an electronic device (e.g., a computer) includes hardware and software, such as a set of one or more processors (e.g., of which a processor is a microprocessor, controller, microcontroller, central processing unit, digital signal processor, application specific integrated circuit (ASIC), field programmable gate array (FPGA), other electronic circuitry, or a combination of one or more of the preceding) coupled to one or more machine-readable storage media to store code for execution on the set of processors and / or to store data. For instance, an electronic device may include non-volatile memory containing the code since the non-volatile memory can persist code / data even when the electronic device is turned off (when power is removed). When the electronic device is turned on, that part of the code that is to be executed by the processor(s) of the electronic device is typically copied from the slower non-volatile memory into volatile memory (e.g., dynamic random-access memory (DRAM), static random-access memory (SRAM)) of the electronic device. Typical electronic devices also include a set of one or more physical network interface(s) (NI(s)) to establish network connections (to transmit and / or receive code and / or data using propagating signals) with other electronic devices. For example,the set of physical NIs (or the set of physical NI(s) in combination with the set of processors executing code) may perform any formatting, coding, or translating to allow the electronic device to send and receive data whether over a wired and / or a wireless connection. In some embodiments, a physical NI may comprise radio circuitry capable of (1) receiving data from other electronic devices over a wireless connection and / or (2) sending data out to other devices through a wireless connection. This radio circuitry may include transmitter(s), receiver(s), and / or transceiver(s) suitable for radio frequency communication. The radio circuitry may convert digital data into a radio signal having the proper parameters (e.g., frequency, timing, channel, bandwidth, and so forth). The radio signal may then be transmitted through antennas to the appropriate recipient(s). In some embodiments, the set of physical NI(s) may comprise network interface controller(s) (NICs), also known as a network interface card, network adapter, or local area network (LAN) adapter. The NIC(s) may facilitate in connecting the electronic device to other electronic devices allowing them to communicate with wire through plugging in a cable to a physical port connected to an NIC. One or more parts of an embodiment of the invention may be implemented using different combinations of software, firmware, and / or hardware.

[0159] The terms “module,” “logic,” and “unit” used in the present application, may refer to a circuit for performing the function specified. In some embodiments, the function specified may be performed by a circuit in combination with software such as by software executed by a general -purpose processor.

[0160] Any appropriate steps, methods, features, functions, or benefits disclosed herein may be performed through one or more functional units or modules of one or more virtual apparatuses. Each virtual apparatus may comprise a number of these functional units. These functional units may be implemented via processing circuitry, which may include one or more microprocessor or microcontrollers, as well as other digital hardware, which may include digital signal processors (DSPs), special-purpose digital logic, and the like. The processing circuitry may be configured to execute program code stored in memory, which may include one or several types of memory such as read-only memory (ROM), random-access memory (RAM), cache memory, flash memory devices, optical storage devices, etc. Program code stored in memory includes program instructions for executing one or more telecommunications and / or data communications protocols as well as instructions for carrying out one or more of the techniques described herein. In some implementations, the processing circuitry may be used to cause the respective functional unit to perform corresponding functions according one or more embodiments of the present disclosure.

[0161] The term unit may have conventional meaning in the field of electronics, electrical devices, and / or electronic devices and may include, for example, electrical and / or electronic circuitry, devices, modules, processors, memories, logic solid state and / or discrete devices, computer programs or instructions for carrying out respective tasks, procedures, computations, outputs, and / or displaying functions, and so on, as such as those that are described herein.

Claims

CLAIMSWhat is claimed is:

1. A method for fault mitigation in a cloud system, the method comprising: moving (502) a first work unit in the cloud system from a running state to a first state upon determining that the first work unit is faulty, wherein in the running state, the first work unit performed task execution to provide a service of the cloud system; responsive to the first work unit being moved to the first state, causing (504) a set of work units to be started to provide the service based on a probability of a work unit in the first state being moved to a second state, wherein in the second state the work unit is standby for task execution to provide the service; moving (506) a second work unit from the first state to the second state upon determining within a time period that the second work unit is ready for the service; and causing (508) a third work unit to perform task execution by moving the third work unit from the second state to the running state upon determining that one or more additional work units are needed to provide the service.

2. The method of claim 1, further comprising: upon determining within the time period that the second work unit remains faulty, terminating (510) the second work unit.

3. The method of claim 1 or 2, wherein a default number of work units are to be placed into the second state, and wherein the default number is determined based on one or more of: a number of work units to be configured in the running state for the service; a number of work units feasible to provide the service when no further fault work units being detected; a number of faulty work units allowed for the service; how long it takes for a work unit to transition from the starting state to the second state; and duration of the time period.

4. The method of any of claims 1 to 3, wherein duration of the time period is determined based on one or more of:likelihood that a work unit to provide the service is determined to be ready for the service after the time period; how long it takes for the work unit to provide the service to transition from the starting state to the running state; and the work unit’ s type.

5. The method of claim 4, wherein the work unit’s type is based on one or more of: services to be supported by the work unit, and version of the work unit.

6. The method of any of claims 1 to 5, wherein the probability of a work unit in the first state being moved to the second state is learned based on state transition events of a plurality of work units to provide the service.

7. The method of claim 6, wherein the probability of a work unit in the first state being moved to the second state is learned based on counting a first number of work units for the service transitioning from the first state to the second state within in an observation period and a second number of work units for the service in the first state within the observation period.

8. The method of any of claims 1 to 7, wherein starting the set of work units based on the probability of a work unit in the first state being moved to the second state comprises counting, by a counter, how many work units have been moved to the first state since the counter is set to a default value.

9. The method of claim 8, wherein the set of work units is caused to be started when a threshold is crossed by a value that is derived based on: the counter value, and the probability of a work unit being moved to the first state and then being determined within the time period to be ready for the service.

10. The method of claim 8, wherein upon starting the set of work units, the counter is reset to the default value, wherein a further work unit being determined to be faulty causes the counter to start counting.

11. The method of any of claims 1 to 10, wherein starting the set of work units is further based on how many work units are in the second state standby already for task execution to provide the service.

12. The method of any of claims 1 to 11, wherein each of the first, second, and third work units is a pod in a Kubernetes architecture.

13. An electronic device (602) for fault mitigation in a cloud system, comprising: a processor (642) and non-transitory machine-readable storage medium (649) that provides instructions that, when executed by the processor, are capable of causing the processor (642) to perform: moving (502) a first work unit in the cloud system from a running state to a first state upon determining that the first work unit is faulty, wherein in the running state, the first work unit performed task execution to provide a service of the cloud system; responsive to the first work unit being moved to the first state, causing (504) a set of work units to be started to provide the service based on a probability of a work unit in the first state being moved to a second state, wherein in the second state the work unit is standby for task execution to provide the service; moving (506) a second work unit from the first state to the second state upon determining within a time period that the second work unit is ready for the service; and causing (508) a third work unit to perform task execution by moving the third work unit from the second state to the running state upon determining that one or more additional work units are needed to provide the service.

14. The electronic device of claim 13, wherein the processor is caused to further perform: upon determining within the time period that the second work unit remains faulty, terminating (510) the second work unit.

15. The electronic device of claim 13 or 14, wherein a default number of work units are to be placed into the second state, and wherein the default number is determined based on one or more of: a number of work units to be configured in the running state for the service; a number of work units feasible to provide the service when no further fault work units being detected; a number of faulty work units allowed for the service; how long it takes for a work unit to transition from the starting state to the second state; andduration of the time period.

16. The electronic device of any of claims 13 to 15, wherein duration of the time period is determined based on one or more of: likelihood that a work unit to provide the service is determined to be ready for the service after the time period; how long it takes for the work unit to provide the service to transition from the starting state to the running state; and the work unit’ s type.

17. The electronic device of any of claims 13 to 16, wherein the probability of a work unit in the first state being moved to the second state is learned based on state transition events of a plurality of work units to provide the service.

18. The electronic device of any of claims 13 to 17, wherein starting the set of work units based on the probability of a work unit in the first state being moved to the second state comprises counting, by a counter, how many work units have been moved to the first state since the counter is set to a default value.

19. The electronic device of any of claims 13 to 18, wherein starting the set of work units is further based on how many work units are in the second state standby already for task execution to provide the service.

20. A machine-readable storage medium (649) that provides instructions that, when executed by a processor, are capable of causing an electronic device to perform any of methods 1 to 12.

Images