Method and system to mitigate faults in distributed system

Abstract

Embodiments include a method, an electronic device, a storage medium, and a computer program for fault mitigation in a distributed system. In one embodiment, a method includes obtaining measurements relating to one or more unidirectional data streams from one or more source service instances and to be distributed to one of at least two destination service instances in a distributed system; determining an obtained measurement indicating that distribution of a one-way data stream within the one or more one-way data streams to a destination service instance of the at least two destination service instances fails to meet a quality of service requirement; and causing the unidirectional data stream to be distributed to be rerouted to another destination service instance rather than the destination service instance.

Description

Technical Field

[0001] Embodiments of the invention relate to the field of networking; and more particularly, to mitigating failures in distributed systems. Background Art

[0002] The fundamental principle of cloud-native applications is to decompose software into smaller and more manageable loosely coupled parts. This concept is not new. It has always been a best practice to separate code into more manageable parts; however, what is new is that each part has a well-defined scope and can now be deployed, scaled, and upgraded independently. In addition, those parts communicate through well-defined and versioned network-based interfaces. These communicating parts form a distributed system.

[0003] Cloud native is about how to create and deploy applications, and it uses the concepts of building and running applications to take advantage of one or more distributed systems provided by the cloud delivery model. Those applications are designed and built to take advantage of the scale, elasticity, resilience, and flexibility provided by the cloud. For example, fifth-generation (5G) use cases drive the need for cloud-native applications such as 5G core network functions standardized by the Third Generation Partnership Project (3GPP). This increases the speed of application development and the efficiency of distributed systems.

[0004] In order to make applications in distributed systems reliable even during failures on individual components, the failed components need to be quickly identified and isolated, and other parts of the distributed system take over the task. Remote procedure calls (RPCs) are traditionally used in distributed systems, and requests and their responses are available at the same node of the RPC-driven system, allowing to mitigate failures from local response information. Still in other distributed systems, operations are often distributed to different nodes, the response of the call may not be returned to the caller node, and that makes implementing failure mitigation from local information inappropriate in other distributed systems. Summary of the invention

[0005] Embodiments include methods, electronic devices, storage media, and computer programs for fault mitigation in distributed systems. In one embodiment, the method includes: obtaining measurements related to one or more unidirectional data flows, the one or more unidirectional data flows coming from one or more source service instances and to be distributed to one of at least two destination service instances in the distributed system; determining the obtained measurements, the obtained measurements indicating that the distribution of the unidirectional data flows within the one or more unidirectional data flows to a destination service instance of the at least two destination service instances fails to meet quality of service requirements; and causing the unidirectional data flows to be distributed to be rerouted to another destination service instance instead of the destination service instance.

[0006] Embodiments include an electronic device for fault mitigation in a distributed system. In one embodiment, the electronic device includes a processor and a machine-readable storage medium, the machine-readable storage medium providing instructions that, when executed by the processor, can cause the electronic device to perform: obtaining measurements related to one or more unidirectional data flows, the one or more unidirectional data flows coming from one or more source service instances and to be distributed to one of at least two destination service instances in the distributed system; determining the obtained measurements, the obtained measurements indicating that distribution of the unidirectional data flows within the one or more unidirectional data flows to a destination service instance of the at least two destination service instances fails to meet quality of service requirements; and causing the unidirectional data flows to be distributed to be rerouted to another destination service instance instead of the destination service instance.

[0007] Embodiments include machine-readable storage media for fault mitigation in a distributed system. In one embodiment, the machine-readable storage medium provides instructions that, when executed, can cause an electronic device to perform: obtaining measurements related to one or more unidirectional data flows, the one or more unidirectional data flows coming from one or more source service instances and to be distributed to one of at least two destination service instances in a distributed system; determining the obtained measurements, the obtained measurements indicating that distribution of the unidirectional data flows within the one or more unidirectional data flows to a destination service instance of the at least two destination service instances fails to meet quality of service requirements; and causing the unidirectional data flows to be distributed to be rerouted to another destination service instance instead of the destination service instance.

[0008] By implementing the embodiments as described, faulty entities within a distributed system can be quickly identified and actions can be taken against the faulty entities within the distributed system based on violations of QoS requirements, where such violations can be derived using collected observation information. Such fault mitigation works well when distributing data flows in a communication system with multiple service instances for one or more services. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The invention may best be understood by referring to the following description and accompanying drawings which are used to illustrate embodiments of the invention. In the drawings:

[0010] Figure 1 An architecture for fault mitigation in a distributed system according to some embodiments is described.

[0011] Figure 2A Traces and spans in a distributed system according to some embodiments are described.

[0012] Figure 2B Updated traces and spans for fault mitigation in a distributed system according to some embodiments are described.

[0013] Figure 3 A list of parameters that may be included in collected observations from a service instance according to some embodiments is illustrated.

[0014] Figure 4 A list of parameters indicative of performance of a source or destination service instance based on collected observations according to some embodiments is illustrated.

[0015] Figure 5 An implementation of fault mitigation in a distributed system according to some embodiments is described.

[0016] Figure 6 is a flow chart illustrating operations for fault mitigation according to some embodiments.

[0017] Fig. 7A Connectivity between network devices (NDs) within an exemplary network and three exemplary implementations of NDs are illustrated in accordance with some embodiments.

[0018] Figure 7B Exemplary approaches to implementing a dedicated network device in accordance with some embodiments are described.

[0019] Figure 7C Various exemplary ways in which virtual network elements (VNEs) may be coupled according to some embodiments are described.

[0020] Fig.7D A network with a single network element (NE) on each ND in accordance with some embodiments is described, and within this direct approach, a traditional distributed approach (typically used by traditional routers) is contrasted with a centralized approach for maintaining reachability and forwarding information (also referred to as network control).

[0021] Fig. 7E A simple case is illustrated according to some embodiments, where each of the NDs implements a single NE, but a centralized control plane has abstracted multiple NEs in different NDs into (to represent) a single NE in one of (one or more) virtual networks.

[0022] Figure 7F A situation is described in which multiple VNEs are implemented on different NDs and coupled to each other and in which a centralized control plane has abstracted the multiple VNEs so that they appear as a single VNE within one of the virtual networks according to some embodiments. DETAILED DESCRIPTION

[0023] Generally, unless different meanings are implied by the context in which it is used and / or different meanings are clearly given, they will be interpreted according to the common meaning of all terms used in this article in the relevant technical field. Unless otherwise clearly stated, all references to one / an / the element, equipment, assembly, part, step, etc. will be openly interpreted as referring to at least one instance of the element, equipment, assembly, part, step, etc. Unless a step is clearly described as being followed by another step or before another step and / or it is implied that a step must be followed by another step or before another step, it is not necessary to perform the steps of any method disclosed herein in the precise order disclosed. Where appropriate, any feature of any embodiment in the embodiments disclosed herein may be applicable to any other embodiment. Similarly, any advantage of any embodiment in the embodiment may be applicable to any other embodiment, and vice versa. By the following description, other purposes, features and advantages of the attached embodiments will be obvious. Fault Mitigation Architecture

[0024] In a service grid system, a fault may be discovered when a remote procedure call (RPC) response times out or a standard status in the response indicates a fault in the service. To mitigate such faults, a remote procedure call (RPC) driven approach may be implemented, in which the request and the response to the request used for mitigation are available at the same network node. However, such an RPC-based implementation may be impractical for event / message driven distributed systems, in which the observability of the success / timeliness of the operation needs to be distributed rather than potentially local, because the response indicating a fault in the service may be returned to the caller at another network node. Therefore, for event / message driven systems, the fault mitigation problem becomes distributed. Moreover, messages in such systems do not have standard status indications and timed-out / missing messages may be delivered one or two orders of magnitude slower than data transmission in real-time applications using distributed systems, so event / message driven fault mitigation requires fast operation.

[0025] Embodiments of the invention can quickly identify and isolate a faulty component in a distributed system, and other components of the distributed system can take over the tasks performed by the identified faulty component. Figure 1 An architecture for fault mitigation in a distributed system according to some embodiments is described. While the architecture can be used for a wide range of applications, examples are discussed below in some embodiments for its use in real-time applications, and the system may be referred to as a real-time system.

[0026] The system 100 includes a collection of service instances 102 to 106 and 122 to 124, a publish / subscribe proxy or load balancer module 132, and a health monitoring module 152 as shown. Each service instance is implemented with observability instrumentation that collects observations of the service instances in the distributed system (e.g., information about messages and states). The collected observations can be used to derive measurements about data units of data streams processed by the service instances, including timing information, as that is highly relevant in real-time systems. The system directs traffic toward service instances for redundancy requirements as well as performance, as multiple alternative service instances can keep the utilization of computing, memory, network, and other resources at a level that can take over tasks from a failed service instance. Note that in some embodiments, service instances may also be referred to as application instances or software instances.

[0027] In some embodiments, each service instance can be a virtual machine (VM) that executes an application / service in a virtualized or simulated computing system. Alternatively, each service instance can be a container group (pod) in a Kubernetes cluster, which is part of an open source container orchestration system for automated software deployment, scaling, and management, where the container group includes one or more containers to be co-located on the same node. In addition, in other embodiments, each service instance can be a device in a cyber-physical system (CPS) or an intelligent system, which includes a computing system in which a computer-based algorithm controls or monitors the mechanism.

[0028] Data streams are distributed from service A instances 102 to 106 to service B instances 122 to 124, and the former may be referred to as source service instances and the latter may be referred to as destination service instances. The distribution is coordinated by a publish / subscribe proxy or load balancer module 132. In a publish / subscribe model (also referred to as a producer / consumer model, a producer / subscriber model, etc.), a publish / subscribe proxy manages publications by source service instances and subscriptions to publications by destination service instances. In a load balancing model, a load balancer distributes data streams from source service instances to destination service instances to maintain appropriate load distribution between destination service instances based on the respective capabilities of the destination service instances. Note that in a publish / subscribe model, a publish / subscribe proxy may also perform load balancing operations.

[0029] Although the publish / subscribe proxy or load balancer module 132 can be an independent distribution logic 134 (for example, implemented in the hardware or software of the electronic device) in some embodiments, in other embodiments, the publish / subscribe proxy or load balancer module 132 is virtualized on a shared resource of the system 100 (for example, a container in a container group, a distribution service in a cloud, a module in a source / destination service instance / a module related to a source / destination service instance).

[0030] As shown at reference numeral 190, data units of a data stream are transmitted unidirectionally (one-way) from a source service instance to a destination service instance. Each data stream can be identified by a set of attributes embedded in one or more data units of the data stream. An exemplary set of attributes includes a 5-tuple (source and destination IP addresses, protocol type, source and destination TCP / UDP ports); and as discussed in more detail below, another set of attributes includes data stream identification information used in fault mitigation partition keys, trace / span IDs. A data stream may also be referred to as a business flow or flow, and it carries an application payload (e.g., a payload of an end-user application) from a source service instance to a destination service instance.

[0031] The data unit of the data stream may include a packet, a frame or another protocol data unit (PDU) to carry the payload of the corresponding data stream (data plane service of the data stream, payload of the end-user application); and in addition / alternatively, the data unit may include a control message, such as metadata of the data stream and / or additional information for fault mitigation, both of which may be included in the header or payload of the data unit (control plane service of the data stream) to manage the data stream in the distributed system. That is, the data unit may include an application payload (e.g., a payload of the end-user application) and / or the control message itself; and when the data unit includes a control message without an application payload, it corresponds to / maps to a data unit with a payload. In the figure, the payload (representing the application payload) and the trace context (a type of control message) are shown as being transmitted while distributing the data stream to the destination service instance in the distributed system. Although only a unidirectional service flow is shown in the figure, a bidirectional service flow is the sum of two unidirectional service flows in which the source and destination service instances are opposite, so the fault mitigation mechanism in the embodiment of the invention can also be implemented for a bidirectional service flow.

[0032] The health monitoring module 152 analyzes the observed observations to discover faults, quickly avoid fault instances, and recycle fault instances. As explained in further detail below, observations in the form of traces provide information to check for quality of service (QoS) requirement violations. In this example, observations from source and destination service instances are obtained through observability collection 162, which provide information for the requirement violation check at reference numeral 164. In this example, the requirement violation check for destination instance #2 (B#2) of service B is the data unit latency (also referred to as delay) compared to a threshold of 20 milliseconds (ms). The comparison result is provided at reference numeral 166 to record the fault instance, which shows that the latency measurement indicates that the latency is below 20 milliseconds four times and above 20 milliseconds three times during the monitoring period. The record is provided to the health decision at reference numeral 168, where the decision is that B#2 is unhealthy because the configured threshold for the monitoring period is that the QoS requirement violation must be below three to be considered healthy.

[0033] Once it is determined that the destination instance is unhealthy, the circuit breaking reconfiguration module 169 will issue a configuration message 172 to cause the publish / subscribe proxy or load balancer module 132 to reroute to avoid the failed destination instance B#2. Recycling / reconfiguration is called circuit breaking. Additionally / alternatively, the destination instance can be recycled (also referred to as removing / deleting / discarding) at reference numeral 170, and thus B#2 is recycled.

[0034] Circuit breaking is a technique used to stop using an endpoint of a data stream in a distributed system, causing the data stream to go to another endpoint. Circuit breaking is required to prompt the Publish / Subscribe agent or load balancer module 132 to reconfigure so that the incoming data unit can be immediately directed toward another service instance. The Publish / Subscribe agent can support circuit breaking by issuing a "cancel monitoring subscriber" command (explicitly or implicitly specifying the unhealthy subscriber client), which removes the unhealthy subscriber client (B#2 in this example). The Publish / Subscribe agent will then redistribute the partition key (explained in further detail below in relation to FIG. 2) to other existing subscribers in the group (B#l in this example), so that the recycling of the unhealthy service instance no longer affects the monitored data stream.

[0035] Afterwards, the system can instantiate another destination service instance that provides similar functionality as the identified unhealthy service instance, and once the new destination service instance is ready, the publish / subscribe broker can add the newly instantiated destination service instance as a client to potentially cause another rerouting of the data flow to the new destination service instance. Similarly, the load balancer can also remove the identified unhealthy destination service instance from its load balancing operations and add a new destination service instance that provides similar functionality as the identified unhealthy service instance once it is ready. Different systems use different mechanisms to start service instances. For example, in a system based on a Kubernetes cluster, the orchestration module can keep the replica count the same by creating a new service instance.

[0036] Although latency is used as an example of a QoS requirement to be checked, other QoS requirements may also be checked using the architecture. For example, the measurements provided by the trace may include one or more of data unit jitter (latency variation of data units within the same data stream), data loss / degradation, out-of-order delivery, throughput, corrupted data, incomplete data, undecodable / unreadable data, and data processing anomalies. Any of the measurements may result in a disconnect to mitigate an identified fault that results in a violation of one or more QoS requirements required by a service level agreement (SLA), specified by a service operator, or otherwise deemed necessary for the monitored data stream. Checking compliance with QoS requirements may include comparing the measurement to a threshold, and a measurement (or multiple measurements during a time period) that exceeds the threshold results in a determination of a QoS requirement violation. In addition, machine learning techniques (such as support vector machines, decision trees, Bayesian networks, and neural networks) may be used in the determination of QoS requirement violations. Traces and spans

[0037] like Figure 1 As shown in , traces from source and destination service instances are obtained by the health monitoring module 152. The concepts of traces and spans are used in open telemetry, observation specifications and open source libraries. A trace may have multiple spans, and a trace may be viewed as a directed acyclic graph (DAG) of spans (also referred to as a span graph), where the edges between spans are defined as parent / child relationships. A directed acyclic graph (DAG) is a directed graph that does not have directed cycles. The directed acyclic graph of spans includes vertices and edges of spans (edges are also referred to as arcs and they represent data units that may include control messages), where each edge is directed from one vertex to another vertex.

[0038] In an RPC-based implementation, the parent span has a duration that covers all child spans, since the parent ends only when a response is returned. For event / message-driven distributed systems, the parent span typically ends shortly after the last data unit is sent. Figure 2A Traces and spans in a distributed system according to some embodiments are illustrated. The distributed system includes multiple traces, represented by traces 250 and 252, each starting with a root. Although trace 250 is shown in detail with spans / data units, other traces such as trace 252 include similar / different spans / data units. An illustration of spans for service instances, data units, and tasks is shown at reference numeral 290.

[0039] Each service instance can have many different operations that the service instance performs on incoming data, so service instances can be modularized into one or more tasks. A trace is an observation of the flow of data through the tasks within a service instance, through which the data flow is processed in a distributed system. Trace 250, for example, includes an observation of the flow of data through the tasks in service instance 262 (root) and through a directed acyclic graph (DAG) of spans to service instance 264 (leaves). A span is an observation of the execution of a task, and the span duration includes the time period used to process a data unit and potentially send and / or receive a data unit.

[0040] The figure shows the span observation of task processing and sending / receiving data units. The same task can be performed across multiple service instances and these tasks can be referred to as task groups. Tasks within the same task group are shown as boxes with the same pattern fill. A service instance can have multiple types of tasks, each shown with a pattern fill. Since the same task in a service instance generates two span observations, the service instance can include the same task multiple times because the task is triggered twice by a separate data unit. Service instance 202 shows an example of a service instance with multiple spans of the same task (two types of tasks are shown, each with two tasks).

[0041] Each edge shows data flow distribution. Data flow distribution can use publish and subscribe or load balancing mechanism and process corresponding data units in different service instances. In some embodiments, configuration messages (e.g., configuration message 172) can prompt distribution logic (e.g., distribution logic 134) to perform routing / partitioning (or rerouting) of data flows.

[0042] In these embodiments, additional information for fault mitigation included in a data unit (e.g., a packet, frame, or other PDU) of a data stream can be used to consistently select a routing endpoint from a plurality of alternative endpoints. Additional information can be provided to a corresponding load balancer or publish / subscribe agent by a source service instance. The use of additional information can be configured in a configuration message so that based on the additional information, the corresponding load balancer or publish / subscribe agent uses consistent hashing to route the data stream. The additional information in the control message may include a hash value generated by a hash function for the data stream (included in the header or payload of the PDU), and the hash value may be referred to as the partition key of the data stream. Independent of which source service instance is the publisher (also referred to as the producer) of the data unit, as long as they use the same partition key, the data unit will end at the same destination service instance. Since the partition key is mapped to a route to a specific destination service instance and the partition key mapping can be reconfigured (e.g., by distribution logic) when an event (e.g., a configuration message for disconnecting or adding / removing subscribers is received from a health monitoring module), such a route can be referred to as semi-static (static until reconfigured, and also referred to as semi-fixed). Such partition key based routing is superior to previous approaches for providing scaled services in distributed systems because it enables linearizability, reduces contention, and allows having related data units in local cache memories.

[0043] The parent / child relationships indicated by the edges can also indicate source and destination service instance relationships. For example, Figure 2A The source service instance 212 in may be similar to one of the source service instances 102 to 106, and Figure 2A Destination service instances 214 and 216 in can be similar to destination service instances 122 and 124.

[0044] Figure 2B An updated trace and span are illustrated for fault mitigation in a distributed system according to some embodiments. Observations at the span of the trace are collected and used to identify the faulty entity. In this example, the faulty entity corresponds to the destination service instance 214 as shown in the directed acyclic graph of the span. The identification of the faulty entity causes a circuit break to reroute one or more data streams away from the destination service instance 214, and the configuration message may cause the distribution logic (e.g., the distribution logic 134) to perform routing / partitioning of the data stream to the destination service instance 216 under the fault condition. In some embodiments, the routing / partitioning is based on a hash value that was mapped to the destination service instance 214 and is now mapped to the destination service instance 216. The hash value that can be generated by the hash function is an updated partition key that causes one or more data streams to be consistently rerouted to another destination service instance (in this example, the destination service instance 216).

[0045] Although the partition key can be used for rerouting of the data flow, other mechanisms can also be used. For example, a mapping / routing table (such as a flow table in a software defined network (SDN) system) can be used, and when a failed entity is identified, the mapping of the data flow can be switched from the destination service instance 214 to the destination service instance 216 by updating the corresponding flow table entry.

[0046] Note that in Figure 2B In the example of , a failed entity may not only cause rerouting at one layer in the directed acyclic graph, but also cause rerouting at the next layer. That is, rerouting the data flow to service instance 216 as the destination service instance may also cause a change in the source service instance at the next layer. The routing at the next layer may be determined based on measurements of data units transmitted from the new source service instance to its destination service instance, and such rerouting may be performed at the distribution logic as discussed above.

[0047] Note also that in order to maintain trace context in a distributed system, in some embodiments, trace context can be injected / extracted into data units sent between tasks. The trace context includes a trace identifier (ID) and a span ID. Traces can also have local spans; for example, a receiving span can call an internal task with its own span (local span).

[0048] A span in a directed acyclic graph (DAG) of spans gives a view of how data propagates and which tasks are executed, typically from an initial source that creates the root trace context. At any time, it is possible (even likely) to create many of these spans simultaneously using different or shared instances and tasks for the data, so multiple traces can be included in the DAG of spans. Some embodiments may use indirect links (provided by open telemetry) toward spans in the same trace or between traces, especially when the observed task uses previously received data to process new data units. This will then allow the span to have multiple incoming links and not be limited to a single parent. For example, a periodic task may have a link to a span received during the last period, which will then allow analysis across the span graph even if it is not a direct parent-child relationship. Such analysis can be timely handling of previously received messages. For example, QoS requirements (e.g., regarding latency) can be evaluated for each linked span and periodic task span as if they had a direct parent-child relationship. Another example is evaluating the quorum of periodic task spans and link spans in order not to violate QoS requirements. Observations and measurements derived from observations

[0049] Figure 1 Fault mitigation in a distributed system is shown, wherein each service instance is implemented with observability instrumentation that collects observations of messages and states in the distributed system, including information about service instances that process data units. Such collected observations may include traces, spans, instances, and other identifiers to identify the task that processes the corresponding data unit.

[0050] Figure 3 A list of parameters that may be included in collected observations from a service instance according to some embodiments is illustrated. The list of parameters is shown as a table that includes a plurality of entries, and each entry includes a parameter type at reference numeral 302 and a brief description of the parameter in the collected observation information at reference numeral 304. The parameters include (1) parameters defined in the open telemetry specification and customized to observation collection in a distributed system, and (2) parameters implemented specifically for embodiments of the invention, the latter of which are shown in bold and italic font in the table. Not all parameter values ​​need to be included in a given collected observation from a service instance, and in some embodiments, collected observations from a service instance may include values ​​for other parameters. The values ​​of these parameters are provided as observations / traces to a health monitoring module (e.g., health monitoring module 152) to derive measurements. For example, the values ​​of trace ID 322 and span ID 324 are added to a data unit (included in a header or payload of the data unit) as metadata to allow a health monitoring module (e.g., health monitoring module 152) to derive measurements.

[0051] In some embodiments, parameters defined in the open telemetry specification are adapted to embodiments of the invention to identify collected observations in the corresponding DAG of a span. Such parameters include: (i) a trace ID at reference numeral 322, which is a unique identifier of a trace having a common root and including all spans following the root in the trace graph (which may be predetermined or randomly generated); (ii) a span ID at reference numeral 324, which uniquely identifies the corresponding span in the trace; (iii) a parent span ID at reference numeral 326, which identifies the span in which the corresponding task was called or a control message was sent; (iv) a name at reference numeral 330, which is the name of a task associated with the operation of the service, and a service may have several tasks and subtasks, and a service may be referred to by a unique name across a complete application; (v) a parent span ID at reference numeral 327, which is a parent span ID at reference numeral 328, which is a parent span ID at reference numeral 329, which is a parent span ID at reference numeral 330, which is a parent span ID at reference numeral 331, which is a parent span ID at reference numeral 332, which is a parent span ID at reference numeral 333, which is a parent span ID at reference numeral 334, which is a parent span ID at reference numeral 335, which is a parent span ID at reference numeral 336, which is a parent span ID at reference numeral 337, which is a parent span ID at reference numeral 338, which is a parent span ID at reference numeral 339, which is a parent span ID at reference numeral 340, which is a parent span ID at reference numeral 341, which is a parent span ID at reference numeral 342, which is a parent span ID at reference numeral 343, which is a parent span ID at reference numeral 344, which is a parent span ID at reference numeral 345 (i) a start at reference numeral 334, which indicates the start time of the corresponding span (for example, the start time can be recorded using a timestamp identifying when processing of a data unit in the span starts); (vi) an end at reference numeral 336, which indicates the end time of the corresponding span (for example, the end time can be recorded using a timestamp identifying when processing of a data unit in the span is completed); and (vii) a type at reference numeral 338, which indicates the observation type of the collected observations, for example, it can be unknown, producer (source service instance in a publish / subscribe model), subscriber (destination service instance in a publish / subscribe model), client, server, and internal (for internal tasks / local spans).

[0052] Parameters implemented specifically for embodiments of the invention include an instance at reference numeral 332, which is a unique identifier of the entity (e.g., a VM, a container group in a Kubernetes cluster, a device in a CPS) that generated the observation and is to be identified as faulty or non-faulty.

[0053] The parameters may further include outgoing application (app) IDs at reference numeral 340 (app ID and application ID are used interchangeably herein), which in some embodiments are a list of application-defined identities corresponding to the partition key in the corresponding outgoing control message during the span. A source service instance may generate an observation and issue multiple control messages, and potentially several destination service instances will receive messages with the same trace ID and parent span ID. Therefore, from these IDs alone, it may not be possible to track the data flow or know how many and which control messages should be received and handled. However, the combination of the trace ID, parent span ID, and application ID can identify data units belonging to the same data flow. By adding a set of outgoing application IDs to the observation, it is possible to derive which application ID or IDs are expected to be received and handled at any destination service instance in the destination service instance, which in turn generates an observation including the handled incoming application ID. As discussed above, hash-based semi-static routing decisions can be used to distribute data flows with corresponding application IDs. For example, a routing decision can be based on a hash of a portion of a data unit, and a hash value for the routing decision can be derived based on the application ID. This then causes any data unit from any of the source service instances containing a certain application ID to be routed to the same destination service instance (as long as the destination service instance is healthy), following the corresponding partition key as discussed above.

[0054] Additionally or alternatively, the parameters of the collected observations may include a list of incoming application IDs at reference numeral 342, which identifies a list of application-defined identities corresponding to partition keys in incoming messages during the span. Note that the outgoing application IDs of all parent spans of a task should be equal to the incoming application IDs in all spans of the task. That is, if there are no missing control messages, the outgoing application IDs from all parent spans (the corresponding outgoing application IDs) match the incoming application ID to the task of the current span (the corresponding incoming application ID). Missing observations can be identified based on historical data of earlier observations with matching outgoing and incoming application IDs and mismatches. The historical data can be stored in a log or another data structure (e.g., a table, a graph of observations) in a database, and in some embodiments the historical data can be searched and compared to the current application ID.

[0055] For example, the collected observations for the source service instance include values ​​for {Trace ID: aa, Span ID: 1, Start: 100, Outgoing App IDs: [a1, a2, a3]}, but within the allocated 15 time units (the monitoring duration can be determined by QoS requirements), the collected observations for the corresponding destination service instance only include (1) {Trace: aa, Span ID: 2, Parent Span ID: 1, Start: 110, Incoming App IDs: [al]} and (2) {Trace: aa, Span ID: 3, Parent Span ID: 1, Start: 111, Incoming App IDs: [a2]}. The health monitoring module, having collected these observations, can conclude that App ID a3 was not handled within the required 15 time units. Based on the previous observations, it can be found that the mapping for a3 and this service / task is to destination instance B#2, and the missing observations then indicate that destination instance B#2 may be faulty. Note that late observations (eg, if an a3 observation is received after 15 time units) may be considered missing because the corresponding data unit was not processed within the given time period.

[0056] Although application IDs may be assigned to observations and used to derive partition keys for routing, some data units may not have known application IDs assigned to them. As shown at reference numeral 344, the collected parameters of the observation may include the number of outgoing control messages that do not correspond to any known outgoing application ID during the span. As shown at reference numeral 346, in the incoming direction, the collected parameters of the observation may include the number of incoming control messages that do not correspond to any known incoming application ID during the span. Similar to the incoming and outgoing application ID lists, the number of outgoing control messages that do not correspond to any known outgoing application ID of the parent span of the task should be equal to the number of incoming control messages that do not have an application ID in all spans of the task.

[0057] The values ​​of these parameters in the collected observations from the service instance provide information about the service instance processing data units of one or more data streams while performing a task, and they can be used to derive measurements about the performance of itself and / or its peer service instances (e.g., corresponding source / destination service instances).

[0058] Figure 4A list of parameters that indicate the performance of a source or destination service instance based on collected observations according to some embodiments is illustrated. The list of parameters is shown as a table that includes multiple entries, and each entry includes a parameter type at reference numeral 402 and a brief description of the parameter at reference numeral 404. Not all values ​​of the parameters are included to determine the health of a particular service instance, and in some embodiments, the health of a particular service instance may include values ​​of other parameters. The values ​​of the parameters may be derived by a health monitoring module (e.g., health monitoring module 152), a source / destination service instance (e.g., one of source / destination service instances 102 to 106 and 122 to 124).

[0059] The parameters may include a task duration at reference numeral 412 , which indicates an execution duration of a single task, and the execution duration may be determined based on a start time and an end time of a span (ie, execution duration of task=end time of span−start time of span).

[0060] The parameters may also include a delay (or delays) for a message with an application ID at reference numeral 414. A production task group in potentially multiple instances may send a control message with an application ID, which is received by a subscribing task group. The application ID needs to be matched between the producer and the subscriber. Mismatches may be used to identify any missing application IDs. In addition, the corresponding delays may be calculated: the receiving delay is calculated based on the end time of the production task span and the start time of the subscription task span (receiving delay = start time of subscription task span - end time of production task span); and the processing delay is based on the end time of the production task span and the end time of the start time of the subscription task span (processing delay = end time of subscription task span - end time of production task span). This may also be applied to longer chains with intermediate tasks to derive the latency of the chain.

[0061] The parameters may further include the latency (or latency) for messages without an application ID at reference numeral 416. A producing task group in potentially multiple service instances may send messages that are received by a subscribing task group. The number of control messages without an app ID needs to be matched between the producer and subscriber, and any mismatch in the number of control messages may be detected. Similar to the calculation of the latency (or latency) for messages with an application ID, the inter-task group latency may be calculated using the end time of the producing task span for either receiving latency or processing latency and the start or end time of the subscribing task span, respectively.

[0062] The parameters may additionally include a mapping instance of task processing based on the application ID at reference numeral 418. Due to the semi-static routing of the data stream and the partition key, a mapping between which instance of the task that processes the application ID can be established. This then enables identification of the instance when the span graph indicates that a control message with the application ID is either late or missing.

[0063] Additionally, the parameters may include a mapping instance of a task processing message without an application ID at reference numeral 420. For control messages that do not have a known application ID, it is possible to track which task instances exist. Then, when the number of control messages sent and received does not match, it is possible to identify unhealthy instances that are not in the span graph. Instances that are not used to process control messages will also be identified as unhealthy, but other span graphs that utilize that instance will then indicate that they are healthy.

[0064] The values ​​of these parameters are derived from information in the collected observations from the source / destination service instances. These values ​​indicating latency and / or missing control messages identify unhealthy service instances and trigger circuit breaking so that data flow distribution can be rerouted and the failed service instance can be reclaimed.

[0065] The health monitoring module in the distributed system can continuously collect observations and derive measurements about the operation at the service instance. Since the monitored applications are distributed in the system, the span observations arrive out of order and the span graph is continuously built and verified. And the faulty entity is identified by finding observations that violate the QoS requirements for (one or more) applications. Therefore, in the distributed system, the QoS requirements of the application are defined (such as constraints on one or more requirements for inter-task latency and task duration), and observations are collected from the source / destination service instances (such as including information about Figure 3 Parameters / values ​​explained), deriving measurements based on observations (e.g. including Figure 4 The measurements are compared to the QoS requirements to identify violation(s), which are used to identify faulty service instances(s) (e.g., Figure 1 Once the failed service instance is identified, the data flow destined for the service instance is rerouted to the healthy service instance (circuit breaking discussed above), and the failed service instance is recycled by removing it and recreating a new service instance with similar / identical functionality.

[0066] QoS requirements for applications include the timing, reliability and correctness of tasks and their relationship to other tasks when processing the application. A task may have a requirement for the duration of processing, and any span observations may then be compared directly to this duration requirement.

[0067] In addition, timing-related QoS requirements may include QoS requirements between tasks that are first based on the maximum delay allowed for any data unit. When such maximum delay is reached, the correctness of the operation is analyzed (for example, by a health monitoring module). All span observations for a specific task in each trace are collected, and the maximum delay added to the latest end time is used as the verification time. The correctness of the operation is then checked, for which all application IDs issued need to have been received and processed accordingly by the subscriber. Moreover, all parent-child span delays are verified to not violate the maximum delay. Similarly, at such similar times (but for another task), it is possible to verify that any message counts correspond between instances of the sending and receiving tasks. Such timing-related QoS requirements can be listed in the following format according to the task:

[0068] <task-name>:

[0069] duration:<>

[0070] children:<>

[0071] <task-name>:

[0072] latency_max:<>

[0073] type:<app_id or countbased>

[0074] When an observation is determined to violate the corresponding QoS requirement, the correct violating service instance needs to be identified. The main difficulty is that when the span observations for the subscribing task are missing due to arriving late or potentially never arriving, the health monitoring cannot just read the instance from the span that violated the requirement. Instead, that one is derived from previously received spans in other span graphs, as shown in the example above with respect to trace aa (where the control message with App ID a3 was determined to be missing). Such absence is then used to degrade the reliability of that instance, which can be compensated for by receiving observations that the instance is healthy. When the health record of a service instance is downgraded to a certain level (e.g., in Figure 1 The latency observed for data flows to B#2 during the monitoring period shown in was above 20 ms three out of seven times), redirecting any data flows towards that instance and reclaiming the service instance.

[0075] The measurements and determinations made are now described to provide a practical example as tested in one embodiment. For example, measurements can be made in standard Kubernetes deployed on a standard Openstack virtual machine (no real-time tuning is required). At the time of the unhealthy decision, after being requested, the new data unit is routed to a healthy existing container group (service instance in Kubernetes) within approximately 307 milliseconds. Starting from the introduction of additional latency, the detection of the unhealthy container group takes approximately 239 milliseconds. For the eight tests performed, the difference in these measurements is approximately 10-15 milliseconds. It should be noted that the requirement is 100 milliseconds of latency and therefore the violation of the requirement can be detected first when 100 milliseconds have passed, and the exact time of the unhealthy decision is highly dependent on how to tune the detection. Due to the large latency jitter in the standard cloud that executes applications when (standard) CPython is running, the detection needs to tolerate some jitter. In this case, it is configured to require three failure / missing observations within the last six observations from the container group. It can also be seen from the description that although it is potentially late, the faulty container group will continue to serve the data unit until rerouting is in place. The failed container group will be terminated gracefully (for example, within 30 seconds) and a replacement container group will be created to take over the business at the next failure.

[0076] Although timing-related QoS requirements are explained in detail above, examination of other QoS requirements, including data unit jitter, data loss / degradation, out-of-order delivery, throughput, corrupted data, incomplete data, undecodable / unreadable data, and data processing anomalies, are similar to those discussed above. Exemplary Implementations According to Some Embodiments

[0077] Figure 5 An implementation of fault mitigation in a distributed system according to some embodiments is illustrated. A distributed system 500 is used to monitor a location (e.g., a factory) and manage robots operating at the location. Content sources 518 include video and audio content, and they include cameras and other sensors to monitor activity at the location. Content sources 518 are distributed to content analysis modules 55 via a real-time publish / subscribe proxy or load balancer module 520, which includes multiple workers, each of which can be considered a destination service instance. The module real-time publish / subscribe proxy or load balancer module 520 can include distribution logic (e.g., Figure 1 Distribution logic 134 described in .

[0078] Based on the analysis results by the content analysis module 522, the real-time messaging module 526 is to cause the execution of multiple services, each of which has multiple instances, as shown at the reference numeral 524. For example, the services include path planning (to plan the route for the robot to travel), safety operation, scheduling (when / where the robot is to move), trajectory generation and collision avoidance (to prevent the robots from colliding with each other and other obstacles at the location). In addition, the real-time messaging module 526 causes the robots (each of which is regarded as a service instance within the autonomous transport robot 529) to operate at the location. These multiple service instances 524 and robot instances 529 can be reviewed as destination service instances, and the real-time messaging module can be a distribution logic (e.g. Figure 1 Distribution logic 134 described in .

[0079] Note that health monitoring module 552 can monitor the operation of workers within content analysis module 522 to identify faulty workers. Similarly, health monitoring module 554 can monitor the operation of multiple service instances 524 and robot instances 529 and identify faulty services and robots.

[0080] In one embodiment, a standard Kubernetes cluster deployment is used to implement Figure 5 The various real-time components like the real-time publish / subscribe proxy or load balancer module 520 and the real-time messaging module 526 are deployed in the container group, as are the workers in the content analysis module 522. Operation according to some embodiments

[0081] Figure 6 is a flow chart illustrating operations for fault mitigation according to some embodiments. The operations of method 600 may be implemented in an electronic device implementing health monitoring module 152 , 552 , and / or 554 .

[0082] At reference numeral 602, an electronic device is to obtain measurements related to one or more unidirectional data flows from one or more source service instances and to be distributed to one of at least two destination service instances in a distributed system.

[0083] At reference numeral 604, the electronic device determines obtained measurements indicating that the distribution of one-way data flows within one or more one-way data flows to the destination service instances of at least two destination service instances fails to meet quality of service (QoS) requirements. In some embodiments, as described above with Figure 1 As explained in relation, the determination of a QoS requirement violation comprises comparing the obtained measurements with a threshold value.

[0084] At reference numeral 606, the electronic device is to cause the unidirectional data flow to be distributed to be rerouted to another destination service instance instead of the destination service instance.

[0085] Optionally at reference numeral 608, the electronic device is to cause removal of the destination service instance and creation of a new destination service instance to service the unidirectional data flow.

[0086] In some embodiments, the obtained measurements indicate latency of distribution of a unidirectional data flow from one or more source service instances to a destination service instance.

[0087] In some embodiments, the latency is derived based on a start time and an end time of processing a data unit within a unidirectional data flow in at least one of a source service instance and a destination service instance. The data unit may include a packet, a frame, or another protocol data unit (PDU) to carry a payload of the corresponding data flow; and additionally / alternatively, it may include metadata and / or control messages for the data flow, both of which may be included in a header or payload of the data unit. The duration between the start time and the end time of processing the data unit in the service instance indicates a task duration in the service instance.

[0088] In some embodiments, the latency is further derived based on an end time of processing a data unit in the unidirectional data flow at a source service instance and a start time of processing a data unit in the unidirectional data flow at a destination service instance. The duration between (1) the end time of processing the data unit at the source service instance and (2) the start time of processing the data unit at the destination service instance indicates a reception latency of a data unit containing a control message with an application ID, or an inter-task group latency of a data unit containing a control message without an application ID, as described above with Figure 4 The two are discussed in relation.

[0089] In some embodiments, the measurement indicates that one or more data units within a unidirectional data flow from one or more source service instances to a destination service instance are missing. In some embodiments, the data unit missing is derived based on matching outgoing data units from the one or more source service instances with incoming data units to at least two destination service instances. In some embodiments, the data unit missing is derived based on matching outgoing data units from the one or more source service instances with incoming data units to at least two destination service instances. Figure 3 and Figure 4 The determination of missing data units, such as control messages, is discussed in related detail.

[0090] In some embodiments, rerouting the unidirectional data flow to another destination service instance instead of the destination service instance includes issuing a configuration message to change load balancing or subscription of at least two destination service instances.Rerouting caused by circuit breaking is discussed above.

[0091] In some embodiments, each of the source service instance and the destination service instance is one of a virtual machine, a container group in a Kubernetes cluster, and a device in a cyber-physical system as discussed above.

[0092] By means of embodiments of the invention, faulty entities within a distributed system can be quickly identified and actions can be taken against the faulty entities within the distributed system based on violations of QoS requirements, wherein collected observation information can be used to derive such violations. Such fault mitigation works well when distributing data flows in a communication system having multiple service instances for one or more services. Device for implementing the embodiment of the invention

[0093] Fig. 7A Connectivity between network devices (NDs) within an exemplary network and three exemplary implementations of NDs are illustrated in accordance with some embodiments. Fig. 7A NDs 700A-H are shown, and their connectivity by way of lines between 700A-700B, 700B-700C, 700C-700D, 700D-700E, 700E-700F, 700F-700G, and 700A-700G, and their connectivity by way of lines between 700H and each of 700A, 700C, 700D, and 700G. These NDs are physical devices, and the connectivity between these NDs may be wireless or wired (often referred to as links). Additional lines extending from NDs 700A, 700E, and 700F illustrate that these NDs act as entry and exit points for the network (and therefore, these NDs are sometimes referred to as edge NDs; while other NDs may be referred to as core NDs).

[0094] Fig. 7A Two of the exemplary ND implementations in are: 1) a dedicated network device 702 using a custom application specific integrated circuit (ASIC) and a dedicated operating system (OS); and 2) a general purpose network device 704 using a common off-the-shelf (COTS) processor and a standard OS.

[0095] The dedicated network device 702 includes networking hardware 710, which includes a collection of one or more processors 712, forwarding resource(s) 714 (which typically include one or more ASICs and / or network processors), and physical network interfaces (NIs) 716 (through which network connections are made, such as those shown by the connectivity between NDs 700A-H), and a non-transitory machine-readable storage medium 718 having stored therein networking software 720. During operation, the networking software 720 may be executed by the networking hardware 710 to instantiate a collection of one or more networking software instances 722. Each of the networking software instance(s) 722 and that portion of the networking hardware 710 that executes that networking software instance (whether hardware dedicated to that networking software instance and / or a time slice of hardware temporarily shared by that networking software instance with others of the networking software instance(s) 722) form a separate virtual network element 730A-R. Each of the virtual network element(s) (VNE) 730A-R includes a control communications and configuration module 732A-R (sometimes referred to as a local control module or control communications module) and forwarding table(s) 734A-R, such that a given virtual network element (e.g., 730A) includes a control communications and configuration module (e.g., 732A), a set of one or more forwarding tables (e.g., 734A), and that portion of the networking hardware 710 that executes the virtual network element (e.g., 730A). In some embodiments, the networking software 720 includes a health monitoring module 152 that can be instantiated in the networking software instance 722 and perform operations for fault mitigation as discussed above.

[0096] The dedicated network device 702 is often physically and / or logically considered to include: 1) an ND control plane 724 (sometimes referred to as a control plane), which includes (one or more) processors 712 that execute (one or more) control communications and configuration modules 732A-R; and 2) an ND forwarding plane 726 (sometimes referred to as a forwarding plane, data plane, or media plane), which includes (one or more) forwarding resources 714 that utilize (one or more) forwarding tables 734A-R and a physical NI 716. By way of example, where the ND is a router (or is implementing routing functionality), the ND control plane 724 (processor(s) 712 executing control communications and configuration modules(s) 732A-R) is generally responsible for participating in controlling how data (e.g., packets) is to be routed (e.g., the next hop for the data and the outgoing physical NI for that data) and storing that routing information in forwarding table(s) 734A-R, and the ND forwarding plane 726 is responsible for receiving that data on the physical NI 716 and forwarding that data out to the appropriate physical NI in the physical NI 716 based on the forwarding table(s) 734A-R.

[0097] Figure 7B An exemplary manner for implementing dedicated network device 702 according to some embodiments is described. Figure 7B A dedicated network device is shown that includes a card 738 (typically hot-swappable). Although in some embodiments the card 738 has two types (one or more operating as the ND forwarding plane 726 (sometimes referred to as a line card) and one or more operating to implement the ND control plane 724 (sometimes referred to as a control card)), alternative embodiments may combine functionality onto a single card and / or include additional card types (e.g., an additional type of card is referred to as a service card, resource card, or multi-application card). Service cards may provide specialized processing (e.g., Layer 4 to Layer 7 services (e.g., firewall, Internet Protocol Security (IPsec), Secure Sockets Layer (SSL) / Transport Layer Security (TLS), Intrusion Detection System (IDS), Point-to-Point (P2P), Voice over IP (VoIP) session border controller, mobile wireless gateway (Gateway General Packet Radio Service (GPRS) Support Node (GGSN), Evolved Packet Core (EPC) Gateway)). By way of example, a service card may be used to terminate an IPsec tunnel and perform the accompanying authentication and encryption algorithms. The cards are coupled together (e.g., a first full mesh coupling line cards and a second full mesh coupling all cards in the card) via one or more interconnect mechanisms as illustrated by backplane 736.

[0098] Return to Fig. 7A , the general network device 704 includes hardware 740, which includes a collection of the following: one or more processors 742 (often COTS processors) and a physical NI 746 and a non-transitory machine-readable storage medium 748 in which software 750 is stored. During operation, the (one or more) processors 742 execute the software 750 to instantiate one or more collections of one or more applications 764A-R. Although one embodiment does not implement virtualization, alternative embodiments may use different forms of virtualization. For example, in one such alternative embodiment, the virtualization layer 754 represents the kernel of an operating system (or a shim executing on top of a base operating system) that allows for the creation of multiple instances 762A-R, referred to as software containers, each of which can be used to execute one (or more) of a set of applications 764A-R; wherein the multiple software containers (also referred to as virtualization engines, virtual private servers, or jails) are user spaces (typically virtual memory spaces) that are separate from each other and from the kernel space in which the operating system is running; and wherein the set of applications running in a given user space cannot (unless explicitly allowed) access the memory of other processes. In another such alternative embodiment, the virtualization layer 754 represents a hypervisor (sometimes referred to as a virtual machine monitor (VMM)) or a hypervisor executing on top of a host operating system, and each of the set of applications 764A-R runs on top of a guest operating system within an instance 762A-R called a virtual machine (which in some cases can be considered a tightly isolated form of software container), running on top of the hypervisor - the guest operating system and applications may not be aware that they are running on a virtual machine instead of a "bare metal" host electronic device, or through paravirtualization, the operating system and / or application may be aware of the presence of virtualization for optimization purposes. In still other alternative embodiments, one, some or all of the applications are implemented as (one or more) monolithic kernels, which can be generated by directly compiling with the application only a limited set of libraries that provide specific OS services required by the application (e.g., from a library operating system (LibOS) that includes drivers / libraries of OS services). Since the single kernel can be implemented to run directly on hardware 740, directly on a hypervisor (in which case the single kernel is sometimes described as running within a LibOS virtual machine), or within a software container, embodiments can be implemented entirely utilizing a single kernel running directly on a hypervisor represented by virtualization layer 754, a single kernel running within a software container represented by instances 762A-R, or as a combination of a single kernel and the techniques described above (e.g., a single kernel and a virtual machine both running directly on a hypervisor, a collection of applications running in different software containers and a single kernel).

[0099] The instantiations of one or more sets of one or more applications 764A-R and virtualization (if implemented) are collectively referred to as software instance(s) 752. Each set of applications 764A-R, the corresponding virtualization constructs (e.g., instances 762A-R) (if implemented), and the portion of hardware 740 on which they execute (whether dedicated to that execution and / or a time slice of temporarily shared hardware) form separate virtual network element(s) 760A-R.

[0100] (One or more) virtual network elements 760A-R perform functionality similar to (one or more) virtual network elements 730A-R - for example, similar to (one or more) control communication and configuration modules 732A and (one or more) forwarding tables 734A (such virtualization of hardware 740 is sometimes referred to as network function virtualization (NFV)). Therefore, NFV can be used to integrate multiple network device types onto industry-standard high-capacity server hardware, physical switches, and physical storage devices that can be located in data centers, NDs, and customer premises equipment (CPE). Although embodiments of the invention are described using each instance 762A-R corresponding to one VNE 760A-R, alternative embodiments may implement such correspondence at a finer level of granularity (e.g., line card virtual machines virtualize line cards, control card virtual machines virtualize control cards, etc.); it should be understood that the techniques described herein with reference to the correspondence of instances 762A-R to VNEs are also applicable to embodiments in which such finer levels of granularity and / or a single kernel are used.

[0101] In some embodiments, the virtualization layer 754 includes a virtual switch that provides forwarding services similar to a physical Ethernet switch. Specifically, such a virtual switch forwards traffic between instances 762A-R and (one or more) physical NIs 746 and optionally between instances 762A-R; in addition, such a virtual switch can implement network isolation (e.g., by honoring virtual local area networks (VLANs)) between VNEs 760A-R that are not allowed to communicate with each other by policy. In some embodiments, the networking software 750 includes a health monitoring module 152, which can be instantiated in the software instance 752, and the health monitoring module 152 performs fault mitigation operations as discussed above. In some embodiments, the health monitoring module 152 includes a computer program, which includes instructions that can cause an electronic device to perform fault mitigation operations when the computer program is executed by a network device storing such a computer program.

[0102] Fig. 7A A third exemplary ND implementation in is a hybrid network device 706 that includes both a custom ASIC / dedicated OS and a COTS processor / standard OS in a single ND or a single card within the ND. In certain embodiments of such a hybrid network device, a platform VM (i.e., a VM that implements the functionality of dedicated network device 702) can provide paravirtualization to the networking hardware present in hybrid network device 706.

[0103] Regardless of the above exemplary implementations of the ND, when a single VNE among multiple VNEs implemented by the ND is being considered (e.g., only one of the VNEs is part of a given virtual network) or where only a single VNE is currently being implemented by the ND, the shortened term network element (NE) is sometimes used to refer to that VNE. Moreover, in all of the above exemplary implementations, each of the VNEs (e.g., those in (one or more) VNEs 730A-R, VNEs 760A-R, and hybrid network device 706) receives data on a physical NI (e.g., 716, 746) and forwards that data out of an appropriate physical NI among the physical NIs (e.g., 716, 746). For example, a VNE implementing IP router functionality forwards IP packets based on some of the IP header information in the IP packets; wherein the IP header information includes a source IP address, a destination IP address, a source port, a destination port (wherein, as opposed to the physical port of the ND, "source port" and "destination port" in this document refer to protocol ports), a transport protocol (e.g., User Datagram Protocol (UDP), Transmission Control Protocol (TCP), and a Differentiated Services Code Point (DSCP) value.

[0104] Figure 7C Various exemplary ways in which VNEs may be coupled according to some embodiments are described. Figure 7C VNEs 770A.1-770A.P (and optional VNEs 770A.Q-770A.R) implemented in ND 700A and VNE 770H.1 in ND 700H are shown. Figure 7C , VNEs 770A.1-P are separated from each other in the sense that they can receive packets from outside ND 700A and forward packets to outside ND 700A; VNE 770A.1 is coupled to VNE 770H.1, and so they pass packets between their respective NDs; VNEs 770A.2-770A.3 can optionally forward packets between themselves without forwarding them outside ND 700A; and VNE 770A.P can optionally be the first in a chain of VNEs, the chain of VNEs including VNE 770A.Q followed by VNE 770A.R (this is sometimes referred to as dynamic service chaining, where each VNE in the series of VNEs provides a different service - for example, one or more Layer 4-7 network services). Although Figure 7C Various exemplary relationships between VNEs are described, but alternative embodiments may support other relationships (e.g., more / fewer VNEs, more / fewer dynamic service chains, multiple different dynamic service chains with some common VNEs and some different VNEs).

[0105] A virtual network is a physical network (such as a Fig. 7A A virtual network can be implemented as an overlay network (sometimes referred to as a network virtualization overlay) that provides network services (e.g., Layer 2 (L2, data link layer) and / or Layer 3 (L3, network layer) services) on an underlying network (e.g., an L3 network such as an Internet Protocol (IP) network that uses tunnels (e.g., Generic Routing Encapsulation (GRE), Layer 2 Tunneling Protocol (L2TP), IPSec) to create an overlay network).

[0106] The network virtualization edge (NVE) is located at the edge of the underlying network and participates in implementing network virtualization; the network-facing side of the NVE uses the underlying network to tunnel frames to and from other NVEs; the external-facing side of the NVE sends data to and receives data from systems outside the network. A virtual network instance (VNI) is a specific instance of a virtual network on the NVE (e.g., a NE / VNE on the ND, a part of a NE / VNE on the ND, where that NE / VNE is divided into multiple VNEs through simulation); one or more VNIs can be instantiated on the NVE (e.g., as different VNEs on the ND). A virtual access point (VAP) is a logical connection point on the NVE for connecting external systems to a virtual network; a VAP can be a physical port or a virtual port identified by a logical interface identifier (e.g., a VLAN ID).

[0107] Examples of network services include: 1) Ethernet LAN emulation service (similar to Ethernet-based multipoint services of Internet Engineering Task Force (IETF) Multiprotocol Label Switching (MPLS) or Ethernet VPN (EVPN) services), where external systems are interconnected across networks over the underlying network through a LAN environment (e.g., NVE provides separate L2VNIs (virtual switching instances) for different such virtual networks and provides L3 (e.g., IP / MPLS) tunnel encapsulation across the underlying network); and 2) virtualized IP forwarding service (similar to IETF IP VPN (e.g., Border Gateway Protocol (BGP) / MPLSIPVPN) from a service definition perspective), where external systems are interconnected across networks over the underlying network through an L3 environment (e.g., NVE provides separate L3 VNIs (forwarding and routing instances) for different such virtual networks and provides L3 (e.g., IP / MPLS) tunnel encapsulation across the underlying network). Network services may also include quality of service capabilities (e.g., traffic classification marking, traffic conditioning and scheduling), security capabilities (e.g., filters to avoid malformed routing advertisements to protect customer premises from network-initiated attacks), and management capabilities (e.g., comprehensive detection and processing).

[0108] Fig.7D Described are some embodiments having Fig. 7A In this direct approach, the traditional distributed approach (typically used by traditional routers) is contrasted with the centralized approach for maintaining reachability and forwarding information (also known as network control). Specifically, Fig.7D Explains the Fig. 7A The ND 700A-H has identical connectivity to the Network Element (NE) 770A-H.

[0109] Fig.7D A distributed method 772 is illustrated that distributes the responsibility for generating reachability and forwarding information across the NEs 770A-H; in other words, the process of neighbor discovery and topology discovery is distributed.

[0110] For example, where a dedicated network device 702 is used, the control communication and configuration modules 732A-R of the ND control plane 724 typically include reachability and forwarding information modules to enable one or more routing protocols (e.g., exterior gateway protocols such as Border Gateway Protocol (BGP), interior gateway protocol (IGP) (e.g., Open Shortest Path First (OSPF), Intermediate System to Intermediate System (IS-IS), Routing Information Protocol (RIP), Label Distribution Protocol (LDP), Resource Reservation Protocol (RSVP) (including RSVP-Traffic Engineering (TE): an extension of RSVP for LSP tunnels and Generalized Multi-Protocol Label Switching (GMPLS) signaling RSVP-TE)) to communicate with other NEs to exchange routes and then select those routes based on one or more routing metrics. Thus, the NE 770A-H (e.g., processor(s) 712 executing(s) control communication and configuration modules 732A-R) perform their responsibilities for participating in controlling how data (e.g., packets) are to be routed (e.g., the next hop for the data and the outgoing physical NI for that data) by distributed determination of reachability within the network and calculation of its corresponding forwarding information. Routes and adjacencies are stored in one or more routing structures (e.g., routing information base (RIB), label information base (LIB), one or more adjacency structures) on the ND control plane 724. The ND control plane 724 utilizes information (e.g., based on the routing structure(s)) to determine reachability within the network and calculate its corresponding forwarding information. The ND control plane 724 programs the ND forwarding plane 726 with the adjacency and routing information (e.g., adjacency and routing information). For example, the ND control plane 724 programs the adjacency and routing information into one or more forwarding tables 734A-R (e.g., forwarding information base (FIB), label forwarding information base (LFIB), and one or more adjacency structures) on the ND forwarding plane 726. For Layer 2 forwarding, the ND may store one or more bridging tables used to forward that data based on the Layer 2 information in the data. Although the above example uses a dedicated network device 702, the same distributed method 772 may be implemented on a general purpose network device 704 and a hybrid network device 706.

[0111] Fig.7D A centralized approach 774 (e.g., software defined networking (SDN)) is described that separates the system that makes decisions about where to send traffic from the underlying system that forwards the traffic to the selected destination. The described centralized approach 774 has the responsibility of generating reachability and forwarding information in a centralized control plane 776 (sometimes referred to as an SDN control module, controller, network controller, OpenFlow controller, SDN controller, control plane node, network virtualization mechanism, or management control entity), and thus the process of neighbor discovery and topology discovery is centralized. The centralized control plane 776 has a southbound interface 782 with a data plane 780 (sometimes referred to as an infrastructure layer, network forwarding plane, or forwarding plane (which should not be confused with the ND forwarding plane)) including NEs 770A-H (sometimes referred to as switches, forwarding elements, data plane elements, or nodes). The centralized control plane 776 includes a network controller 778 including a centralized reachability and forwarding information module 779 that determines reachability within the network and distributes forwarding information to the NEs 770A-H of the data plane 780 through southbound interfaces 782 (which may use the OpenFlow protocol). Thus, network intelligence is centralized in the centralized control plane 776, which executes on an electronic device that is typically separate from the ND.

[0112] For example, where the dedicated network device 702 is used in the data plane 780, each of the (one or more) control communication and configuration modules 732A-R of the ND control plane 724 typically includes a control agent on the VNE side that provides a southbound interface 782. In this case, the ND control plane 724 (the (one or more) processors 712 executing the (one or more) control communication and configuration modules 732A-R) perform its responsibilities of participating in communicating with the centralized control plane 776 through the control agent to receive forwarding information (and in some cases, reachability information) from the centralized reachability and forwarding information module 779 to control how data (e.g., packets) are to be routed (e.g., the next hop for the data and the outgoing physical NI for that data) (it should be understood that in some embodiments of the invention, in addition to communicating with the centralized control plane 776, the (one or more) control communication and configuration modules 732A-R may also play a role in determining reachability and / or calculating forwarding information - although not as much as in the case of a distributed approach; such embodiments are generally considered to belong to the centralized approach 774, but may also be considered a hybrid approach).

[0113] While the above example uses a dedicated network device 702, the same centralized approach 774 may be implemented using a general purpose network device 704 (e.g., each of the VNEs 760A-R performs its responsibilities for controlling how data (e.g., packets) are to be routed (e.g., the next hop for the data and the outgoing physical NI for that data) by communicating with a centralized control plane 776 to receive forwarding information (and, in some cases, reachability information) from a centralized reachability and forwarding information module 779; it should be understood that in some embodiments of the invention, in addition to communicating with the centralized control plane 776, the VNEs 760A-R may also play a role in determining reachability and / or calculating forwarding information - although not as much as in the case of a distributed approach) and a hybrid network device 706. In fact, the use of SDN technology may enhance the NFV technology typically used in a general purpose network device 704 or hybrid network device 706 implementation, because NFV is able to support SDN by providing an infrastructure on which SDN software may run, and both NFV and SDN are intended to utilize commodity server hardware and physical switches. Note that the health monitoring module 152 may be included in the network controller 778, for example as a module in the centralized reachability and forwarding information module 779. That allows the electronics implementing the centralized control plane 776 to perform the fault mitigation discussed above on the data flows in the data plane 780.

[0114] Fig.7D The centralized control plane 776 is also shown to have a northbound interface 784 to the application layer 786, where resides (one or more) applications 788. The centralized control plane 776 has the ability to form a virtual network 792 (sometimes referred to as a logical forwarding plane, network service, or overlay network (where the NEs 770A-H of the data plane 780 are the underlying network)) for the (one or more) applications 788. Thus, the centralized control plane 776 maintains a global view of all NDs and configured NEs / VNEs, and it efficiently maps virtual networks to underlying NDs (including maintaining these mappings when the physical network changes, either through hardware (NDs, links, or ND components) failures, additions, or removals).

[0115] Although Fig.7D A distributed approach 772 is shown separate from a centralized approach 774, but in certain embodiments of the invention, the network control effort may be distributed differently or the two may be combined. For example: 1) an embodiment may generally use a centralized approach (SDN) 774, but delegate certain functions to the NE (e.g., a distributed approach may be used to implement one or more of fault monitoring, performance monitoring, protection switching, and primitives for neighbor and / or topology discovery); or 2) an embodiment of the invention may perform neighbor discovery and topology discovery via both a centralized control plane and a distributed protocol, and compare the results to raise an exception if they are inconsistent. Such embodiments are generally considered to belong to the centralized approach 774, but may also be considered a hybrid approach.

[0116] Although Fig.7D A simple case is described in which each of the NDs 700A-H implements a single NE 770A-H, but it should be understood that reference Fig.7D The described network control method also works for a network in which one or more of the NDs 700A-H implement multiple VNEs (e.g., those in the VNEs 730A-R, VNEs 760A-R, hybrid network devices 706). Alternatively or in addition, the network controller 778 may also simulate the implementation of multiple VNEs in a single ND. Specifically, instead of (or in addition to) implementing multiple VNEs in a single ND, the network controller 778 may also present the implementation of the VNE / NE in a single ND as multiple VNEs in a virtual network 792 (all in the same virtual network of (one or more) virtual networks 792, each in a different virtual network of (one or more) virtual networks 792, or some combination). For example, the network controller 778 may cause the ND to implement a single VNE (NE) in the underlying network, and then logically divide the resources of that NE within the centralized control plane 776 to present different VNEs in (one or more) virtual networks 792 (where these different VNEs in the overlay network are sharing the resources of a single VNE / NE implementation on the ND in the underlying network).

[0117] on the other hand, Fig. 7E and 7F Exemplary abstractions of NEs and VNEs, respectively, that the network controller 778 may present as part of different virtual networks in the virtual network 792. Fig. 7E Each of the ND 700A-H implements a single NE 770A-H (see Fig.7D ), but according to some embodiments, the centralized control plane 776 has abstracted multiple NEs (NE 770A-C and GH) in different NDs into (represented by) Fig.7D A single NE 770I within one of the (one or more) virtual networks 792. Fig. 7E It is shown that in this virtual network, NE 770I is coupled to NE 770D and 770F, both of which are still coupled to NE 770E.

[0118] Figure 7F The present invention illustrates a method according to some embodiments in which a plurality of VNEs (VNE 770A.1 and VNE 770H.1) are implemented on different NDs (ND 700A and ND 700H) and the plurality of VNEs (VNE 770A.1 and VNE 770H.1) are coupled to each other and a centralized control plane 776 has abstracted the plurality of VNEs so that they are interconnected. Fig.7D In the case of a single VNE 770T within one of the virtual networks 792. Therefore, the abstraction of NE or VNE can span multiple NDs.

[0119] A network interface (NI) can be physical or virtual. In the context of IP, an interface address is an IP address assigned to a NI, whether it is a physical NI or a virtual NI. A virtual NI can be associated with a physical NI, associated with another virtual interface, or exist independently (e.g., a loopback interface, a Point-to-Point Protocol interface). NIs (physical or virtual) can be numbered (NIs with IP addresses) or unnumbered (NIs without IP addresses).

[0120] Some of the embodiments contemplated above are more fully described with reference to the accompanying drawings. However, other embodiments are included within the scope of the subject matter disclosed herein, and the disclosed subject matter should not be construed as being limited to the embodiments set forth herein; rather, these embodiments are provided by way of example to convey the scope of the subject matter to those skilled in the art. the term

[0121] Generally, unless different meanings are implied by the context in which it is used and / or different meanings are clearly given, they are to be interpreted according to the common meaning of all terms used in this article in the relevant technical field. Unless otherwise clearly stated, all references to one / an / the element, equipment, assembly, part, step, etc. are to be openly interpreted as referring to at least one instance of the element, equipment, assembly, part, step, etc. Unless a step is clearly described as being followed by another step or before another step and / or it is implied that a step must be followed by another step or before another step, it is not necessary to perform the steps of any method disclosed herein in the precise order disclosed. Where appropriate, any feature of any embodiment in the embodiments disclosed herein may be applicable to any other embodiment. Similarly, any advantage of any embodiment in the embodiment may be applicable to any other embodiment, and vice versa. By the following description, other purposes, features and advantages of the attached embodiments will be obvious.

[0122] References in the specification to "one embodiment," "an embodiment," "an example embodiment," and the like indicate that the described embodiment may include a particular feature, structure, or characteristic, but every embodiment may not necessarily include the particular feature, structure, or characteristic. Furthermore, such phrases do not necessarily refer to the same embodiment. Furthermore, when a particular feature, structure, or characteristic is described in connection with an embodiment, whether or not explicitly described, it is considered to be within the knowledge of those skilled in the art to affect such feature, structure, or characteristic in connection with other embodiments.

[0123] The description and claims may use the terms "coupled" and "connected," along with their derivatives. These terms are not intended to be synonymous with each other. "Coupled" is used to indicate that two or more elements that may or may not be in direct physical or electrical contact with each other cooperate or interact with each other. "Connected" is used to indicate the establishment of wireless or wired communication between two or more elements that are coupled to each other. As used herein, "set" refers to any positive integer number of items that includes an item.

[0124] Electronic devices use machine-readable media (also referred to as computer-readable media) to store and transmit code (which consists of software instructions and which is sometimes referred to as computer program code or computer program) and / or data (internally and / or over a network with other electronic devices), such as machine-readable storage media (e.g., magnetic disks, optical disks, solid-state drives, read-only memory (ROM), flash memory devices, phase-change memory) and machine-readable transmission media (also referred to as carriers) (e.g., electrical, optical, radio, acoustic or other forms of propagated signals - such as carrier waves, infrared signals). Thus, an electronic device (e.g., a computer) includes hardware and software, such as a collection of one or more processors (e.g., where the processor is a microprocessor, controller, microcontroller, central processing unit, digital signal processor, application specific integrated circuit (ASIC), field programmable gate array (FPGA), other electronic circuit, or a combination of one or more of the foregoing) coupled to one or more machine-readable storage media, the one or more machine-readable storage media being used to store code for execution on the collection of processors and / or to store data. For example, an electronic device may include non-volatile memory containing code, because non-volatile memory can persist code / data even when the electronic device is turned off (when power is off). When the electronic device is turned on, the portion of the code to be executed by the (one or more) processors of the electronic device is typically copied from the slower non-volatile memory into the volatile memory (e.g., dynamic random access memory (DRAM), static random access memory (SRAM)) of the electronic device. A typical electronic device also includes a collection of one or more physical network interfaces ((one or more) NIs) used to establish a network connection with other electronic devices (to transmit and / or receive code and / or data using propagated signals). For example, the collection of physical NIs (or the collection of (one or more) physical NIs combined with the collection of processors that execute code) can perform any formatting, encoding, or translation to allow the electronic device to send and receive data, whether via a wired and / or wireless connection. In some embodiments, the physical NI may include a radio circuit capable of (1) receiving data from other electronic devices via a wireless connection and / or (2) sending data out to other devices via a wireless connection. This radio circuitry may include (one or more) transmitters, (one or more) receivers and / or (one or more) transceivers suitable for radio frequency communications. The radio circuitry may convert digital data into radio signals having appropriate parameters (e.g., frequency, timing, channel, bandwidth, etc.). The radio signals may then be transmitted via an antenna to (one or more) appropriate receivers. In some embodiments, the set of (one or more) physical NIs may include (one or more) network interface controllers (NICs), network adapters, or local area network (LAN) adapters, also referred to as network interface cards.(One or more) NICs can facilitate connecting electronic devices to other electronic devices, allowing them to communicate wired by plugging cables into physical ports connected to the NICs. One or more parts of embodiments of the invention may be implemented using different combinations of software, firmware, and / or hardware.

[0125] A network device (ND) (also referred to as a network node or simply a node) is an electronic device that communicatively interconnects other electronic devices on a network (e.g., other network devices, end-user devices). Some network devices are "multi-service network devices" that provide support for multiple networking functions (e.g., routing, bridging, switching, layer 2 aggregation, session border control, quality of service, and / or subscriber management) and / or provide support for multiple application services (e.g., data, voice, and video).

[0126] The terms "module", "logic" and "unit" used in this application may refer to circuits for performing specified functions. In some embodiments, the specified functions may be performed by circuits in conjunction with software, such as by software executed by a general-purpose processor.

[0127] Any appropriate steps, methods, features, functions or benefits disclosed herein may be performed by one or more functional units or modules of one or more virtual devices. Each virtual device may include a plurality of these functional units. These functional units may be implemented with the aid of processing circuits and other digital hardware, the processing circuits may include one or more microprocessors or microcontrollers, the other digital hardware may include digital signal processors (DSPs), dedicated digital logic, and the like. The processing circuit may be configured to execute program codes stored in a memory, the memory may include one or more types of memory, such as a read-only memory (ROM), a random access memory (RAM), a cache memory, a flash memory device, an optical storage device, and the like. The program code stored in the memory includes program instructions for executing one or more telecommunications and / or data communication protocols and instructions for executing one or more of the technologies described herein. In some implementations, according to one or more embodiments of the present disclosure, a processing circuit may be used to cause the corresponding functional unit to perform the corresponding function.

[0128] The term unit may have the conventional meaning in the field of electronic devices, electrical apparatuses and / or electronic devices and may include, for example, electrical and / or electronic circuits, devices, modules, processors, memories, logical solid-state and / or discrete devices, computer programs or instructions for performing corresponding tasks, processes, calculations, output and / or display functions, etc., such as those described in this document.

Claims

1. A method (600) for mitigating failures in a distributed system, the method comprising: obtaining (602) measurements related to one or more unidirectional data flows originating from one or more source service instances and to be distributed to one of at least two destination service instances in the distributed system; determining (604) obtained measurements indicating that distribution of unidirectional data flows within the one or more unidirectional data flows to a destination service instance of the at least two destination service instances fails to meet a quality of service requirement; as well as The unidirectional data flow to be distributed is caused (606) to be rerouted to another destination service instance instead of the destination service instance.

2. The method of claim 1, wherein: The obtained measurements indicate a latency of the distribution of the unidirectional data flow from the one or more source service instances to the destination service instance.

3. The method according to claim 1 or 2, wherein: The latency is derived based on a start time and an end time of processing a data unit within the unidirectional data flow in at least one of a source service instance and the destination service instance.

4. The method according to any one of claims 1 to 3, wherein: The latency is further derived based on an end time of processing a data unit within the unidirectional data flow at a source service instance and a start time of processing the data unit within the unidirectional data flow at the destination service instance.

5. The method according to any one of claims 1 to 4, wherein: The measurement indicates that one or more data units within the unidirectional data flow from the one or more source service instances to the destination service instance are missing.

6. The method according to any one of claims 1 to 5, wherein: The data unit missing is derived based on matching outgoing data units from the one or more source service instances with incoming data units to the at least two destination service instances.

7. The method according to any one of claims 1 to 6, wherein Matching the outgoing data units from the one or more source service instances with the incoming data units to the at least two destination service instances includes comparing application identifiers of the outgoing data units and the incoming data units.

8. The method according to any one of claims 1 to 7, wherein: Rerouting the unidirectional data flow to another destination service instance instead of the destination service instance includes issuing a configuration message to change a load balance or subscription of the at least two destination service instances.

9. The method according to any one of claims 1 to 8, further comprising: Removal of the destination service instance and creation of a new destination service instance to service the unidirectional data flow is caused (608).

10. The method according to any one of claims 1 to 9, wherein: Each of the source service instance and the destination service instance is one of a virtual machine, a container group in a Kubernetes cluster, and a device in an information-physical system.

11. An electronic device (702, 704) for mitigating failures in a distributed system, comprising: A processor (712, 742) and a non-transitory machine-readable storage medium (718, 748) providing instructions that, when executed by the processor (712, 742), can cause the processor (712, 742) to perform: obtaining (602) measurements related to one or more unidirectional data flows originating from one or more source service instances and to be distributed to one of at least two destination service instances in the distributed system; determining (604) obtained measurements indicating that distribution of unidirectional data flows within the one or more unidirectional data flows to a destination service instance of the at least two destination service instances fails to meet a quality of service requirement; as well as The unidirectional data flow to be distributed is caused (606) to be rerouted to another destination service instance instead of the destination service instance.

12. The electronic device (702, 704) of claim 11, wherein: The obtained measurements indicate a latency of the distribution of the unidirectional data flow from the one or more source service instances to the destination service instance.

13. The electronic device (702, 704) of claim 11 or 12, wherein: The latency is derived based on a start time and an end time of processing a data unit within the unidirectional data flow in at least one of a source service instance and the destination service instance.

14. The electronic device (702, 704) according to any one of claims 11 to 13, wherein: The latency is further derived based on an end time of processing a data unit within the unidirectional data flow at a source service instance and a start time of processing the data unit within the unidirectional data flow at the destination service instance.

15. The electronic device (702, 704) of any one of claims 11 to 14, wherein: The measurement indicates that one or more data units within the unidirectional data flow from the one or more source service instances to the destination service instance are missing.

16. The electronic device (702, 704) of any one of claims 11 to 15, wherein: The data unit missing is derived based on matching outgoing data units from the one or more source service instances with incoming data units to the at least two destination service instances.

17. The electronic device (702, 704) of any one of claims 11 to 16, wherein: The data unit missing is derived based on matching outgoing data units from the one or more source service instances with incoming data units to the at least two destination service instances.

18. The electronic device (702, 704) of any one of claims 11 to 17, rerouting the unidirectional data flow to another destination service instance instead of the destination service instance comprises issuing a configuration message to change load balancing or subscription of the at least two destination service instances.

19. The electronic device (702, 704) of any one of claims 11 to 18, wherein: When the instructions are executed by the processor, they can cause the electronic device (702, 704) to further execute: This causes (608) removal of the destination service instance and re-creation of a new destination service instance to serve the unidirectional data flow.

20. The electronic device (702, 704) of any one of claims 11 to 19, wherein: Each of the source service instance and the destination service instance is one of a virtual machine, a container group in a Kubernetes cluster, and a device in an information-physical system.

21. A non-transitory machine-readable storage medium (718, 748) providing instructions that, when executed by a processor (712, 742), can cause the processor (712, 742) to perform: obtaining (602) measurements related to one or more unidirectional data flows originating from one or more source service instances and to be distributed to one of at least two destination service instances in the distributed system; determining (604) obtained measurements indicating that distribution of unidirectional data flows within the one or more unidirectional data flows to a destination service instance of the at least two destination service instances fails to meet quality of service requirements; and The unidirectional data flow to be distributed is caused (606) to be rerouted to another destination service instance instead of the destination service instance.

22. The non-transitory machine-readable storage medium (718, 748) of claim 21, wherein: The obtained measurements indicate a latency of the distribution of the unidirectional data flow from the one or more source service instances to the destination service instance.

23. The non-transitory machine-readable storage medium (718, 748) of claim 21 or 22, wherein: The latency is derived based on a start time and an end time of processing a data unit within the unidirectional data flow in at least one of a source service instance and the destination service instance.

24. The non-transitory machine-readable storage medium (718, 748) of any one of claims 21 to 23, wherein: The latency is further derived based on an end time of processing a data unit within the unidirectional data flow at a source service instance and a start time of processing the data unit within the unidirectional data flow at the destination service instance.

25. The non-transitory machine-readable storage medium (718, 748) of any one of claims 21 to 24, wherein: The measurement indicates that one or more data units within the unidirectional data flow from the one or more source service instances to the destination service instance are missing.

26. The non-transitory machine-readable storage medium (718, 748) of any one of claims 21 to 25, wherein: The data unit missing is derived based on matching outgoing data units from the one or more source service instances with incoming data units to the at least two destination service instances.

27. The non-transitory machine-readable storage medium (718, 748) of any one of claims 21 to 26, wherein: Matching the outgoing data units from the one or more source service instances with the incoming data units to the at least two destination service instances includes comparing application identifiers of the outgoing data units and the incoming data units.

28. A non-transitory machine-readable storage medium (718, 748) as described in any one of claims 21 to 27, rerouting the unidirectional data flow to another destination service instance instead of the destination service instance includes issuing a configuration message to change the load balancing or subscription of the at least two destination service instances.

29. The non-transitory machine-readable storage medium (718, 748) of any one of claims 21 to 28, wherein: When the instructions are executed by the processor, they can cause the processor to further execute: Removal of the destination service instance and re-creation of a new destination service instance to service the unidirectional data flow is caused (608).

30. The non-transitory machine-readable storage medium (718, 748) of any one of claims 21 to 29, wherein: Each of the source service instance and the destination service instance is one of a virtual machine, a container group in a Kubernetes cluster, and a device in an information-physical system.

31. A computer program, comprising instructions, which can cause an electronic device (702, 704) to execute any one of methods 1 to 10 when the computer program is executed by the electronic device.

Images