Orchestration device for distributed processing system

Abstract

The invention relates to an orchestration apparatus for a distributed processing system. According to various embodiments, there is described an orchestration apparatus for a distributed processing system, comprising: an input interface configured to receive a specification of a data processing task to be performed by the distributed processing system and a specification of one or more failure types that the data processing system should be able to process when performing the data processing task; and a command interface configured to instruct each of a plurality of processing nodes of the distributed processing system to perform at least one respective sub-task of the data processing task, and instruct each of at least some of the plurality of processing nodes to implement one or more failed processing software modules, the failure handling software modules are configured to handle failures of a specified failure type.

Description

Technical Field

[0001] The present disclosure relates to orchestration devices, techniques, and methods for distributed processing systems. Background Art

[0002] Applications in real-time / safety-critical systems are often subject to strict timing constraints and must adhere to stringent reliability and safety requirements. For example, cyber-physical systems (CPS) from the automotive industry and industrial control / automation are examples where applications are not only required to operate within strict end-to-end latency bounds (e.g., for control loops that include sensing, processing, and actuation tasks), but also need to be resilient to faults and failures, relying on mechanisms to detect faults and failures (integrity) and, when detected, enable uninterrupted correct operation of the affected applications (fault tolerance) or switch these affected applications into a safe mode (safety).

[0003] Real-time / safety-critical systems typically encompass multiple applications, each with different requirements. In a dynamic distributed system, applications are dynamically added to / removed from the system, and the deployed applications are distributed across a collection of potentially heterogeneous computing nodes interconnected via a communication network. In such a system, the distribution and deployment of applications is a complex task. Meeting the individual requirements of each application based on the availability of resources in the system requires careful analysis and consideration of software / hardware aspects such as real-time, reliability, safety, and domain-specific (e.g., control engineering) aspects.

[0004] Existing approaches for dynamically distributing and deploying applications in distributed systems rely on a central component responsible for managing the system's applications and resources in light of potential system dynamics. This component, typically called an orchestrator, undertakes tasks such as deploying incoming applications, monitoring the status of deployed applications and resources in the system, and reacting to system dynamics (e.g., resource failures or workload changes) by adapting application deployment accordingly. Existing orchestration frameworks (e.g., Kubernetes, AWS Elastic Container Service (ECS), and AWS Step Functions) are designed for best-effort applications and systems (primarily cloud-based systems) that are not subject to strict timing requirements. They lack the necessary mechanisms to enforce the real-time, safety-critical requirements of applications and cannot provide the necessary Quality of Service (QoS) guarantees required by these systems.

[0005] Therefore, task distribution methods are desirable that are able to establish the QoS guarantees required by such systems with respect to aspects such as real-time requirements for real-time / safety-critical systems, reliability, fault tolerance, and safety. Summary of the Invention

[0006] According to various embodiments, there is provided an orchestration device for a distributed processing system, comprising:

[0007] an input interface configured to receive a specification of a data processing task to be performed by the distributed processing system and a specification of one or more failure types that the data processing system should be able to handle when performing the data processing task, and

[0008] A command interface configured to instruct each of a plurality of processing nodes of a distributed processing system (which may be a subset of all processing nodes of the distributed processing system, i.e., the plurality of processing nodes may be some of the processing nodes of the distributed system)

[0009] At least one corresponding subtask in the data processing task is executed, and each of at least some of the plurality of processing nodes is instructed to implement one or more failure handling software modules configured to handle failures of a specified failure type.

[0010] The orchestration appliance allows deploying (additional) capabilities for real-time applications to react to system dynamics in real time and autonomously, without the involvement of an orchestrator. It can be used for distributed processing systems with real-time and / or safety-critical applications, such as CPS related to industrial control and automation, automotive control and automated / autonomous driving, and other real-time systems targeting distributed infrastructure, for example in the context of Industry 4.0, the Industrial Internet of Things (IIoT), or automotive regional architectures.

[0011] In the following, various examples are given.

[0012] Example 1 is an orchestration device as described above.

[0013] Example 2 is the orchestration device according to example 1, wherein the one or more failure handling software modules are configured to handle failures of a specified failure type without communicating with the orchestration device.

[0014] In other words, once the failure handling software module is deployed on a processing node, it can handle failures of various types independently of the orchestration device. This allows the use of a central orchestration device that is remote from the processing node and still handles failures in real time.

[0015] Example 3 is an orchestration device according to example 1 or 2, wherein the input interface is configured to receive specifications of a plurality of software modules, wherein each software module implements a corresponding one of the subtasks, and wherein the orchestration device includes a software generator configured to supplement the plurality of software modules with one or more failure handling software modules.

[0016] In other words, the orchestration device adds the ability to handle one or more failure types to a given software. Therefore, the user does not need to be aware of providing code to handle the failure type, but the failure handling is transparent to the user and also transparent to the data processing task.

[0017] Example 4 is an orchestration device according to any one of Examples 1 to 3, wherein the input interface is configured to receive specifications on how to divide the data processing task into subtasks and / or specifications on requirements for how to perform the data processing task, and the orchestration device is configured to distribute the data processing task to multiple processing nodes according to the specifications on how to divide the data processing task into subtasks and / or the specifications on the requirements.

[0018] For example, the orchestration device may take into account that data processing tasks should be performed in such a way that real-time requirements are met.

[0019] Example 5 is the orchestration device according to any one of Examples 1 to 4, wherein the data processing task is a (eg, real-time) control task of a technical system.

[0020] For example, one of the plurality of processing nodes is a controller connected to the technical system (and, for example, is arranged adjacent to the technical system or is installed in the technical system), and at least one of the plurality of processing nodes is arranged remotely (for example is an edge node or is arranged in the cloud) and takes over data processing in order to control the technical system.

[0021] Example 6 is an orchestration device according to any one of Examples 1 to 5, wherein the command interface is configured to, upon a failure (which may or may not be a failure type) that impairs the ability of the distributed processing system to handle failures of a specified failure type, instruct one or more of the plurality of processing nodes of the distributed processing system and / or one or more additional processing nodes to implement one or more failure handling software modules and / or one or more additional failure handling modules that are configured to handle failures of the specified failure type (to restore the ability of the processing system to handle the one or more specified failure types).

[0022] In other words, the orchestration device can dynamically control the failure handling capabilities of the distributed processing system, and in particular reconfigure the distributed processing system in the event that the distributed processing system loses the ability to handle failures of a particular failure type (for example, because a processing node has been disconnected).

[0023] Example 7 is a method for orchestrating a distributed processing system, comprising: receiving a specification of a data processing task to be performed by the distributed processing system and a specification of one or more failure types that the data processing system should be able to handle when performing the data processing task, and instructing each of a plurality of processing nodes of the distributed processing system (e.g., a subset of all processing nodes of the distributed system) to perform at least one corresponding subtask of the data processing task, and instructing each of at least some of the plurality of processing nodes to implement one or more failure handling software modules that are configured to handle failures of the specified failure types.

[0024] Example 8 is a computer program comprising instructions which, when executed by a computer, cause the computer to carry out the method according to Example 7.

[0025] Example 9 is a computer-readable medium comprising instructions that, when executed by a computer, cause the computer to perform the method according to Example 7.

[0026] It should be noted that examples may be combined and that features described in the context of orchestrating an apparatus analogously apply to the method. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In the drawings, like reference characters generally refer to the same parts throughout the different views. The drawings are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of the invention. In the following description, various aspects are described with reference to the following drawings, in which:

[0028] Figure 1 A data processing arrangement is shown.

[0029] Figure 2 An example of a processing system including three processing nodes and deployed by an orchestrator is shown.

[0030] Figure 3 The diagram shows Figure 2 Adaptation of deployment.

[0031] Figure 4 A flow chart illustrating a method for orchestrating a distributed processing system according to an embodiment is shown. DETAILED DESCRIPTION

[0032] The following detailed description is made with reference to the accompanying drawings, which illustrate, by way of illustration, specific details and aspects of the present disclosure in which the present invention may be practiced. Other aspects may be utilized, and structural, logical, and electrical changes may be made, without departing from the scope of the present invention. Various aspects of the present disclosure are not necessarily mutually exclusive, as some aspects of the present disclosure may be combined with one or more other aspects of the present disclosure to form new aspects.

[0033] Hereinafter, various examples will be described in more detail.

[0034] Figure 1 A data processing arrangement 100 is shown.

[0035] The data processing arrangement 100 comprises a controller 101 arranged to control a controlled system 102 , such as a robot, a vehicle, a machine or the like.

[0036] The data processing arrangement 100 further includes (additional) processing devices 103, which are connected to the controller 101 (and possibly to each other) via a communication connection 104, which can be implemented by various interconnect technologies, such as a wireless and / or wired communication network 108 bus, etc. For example, at least some of the processing devices 103 are arranged in an edge cloud. For example, the data processing arrangement 100 implements a cyber-physical system (CPS).

[0037] Each processing device 103 may implement one or more processing nodes. For example, a processing device 103 may correspond to a single processing node, but a processing device 103 may also include sufficient resources (eg, a server computer) such that it implements multiple processing nodes.

[0038] According to various embodiments, control software for controlling the controlled system 102 (and designed to run on the controller 101) is executed in a distributed manner, i.e., distributed across multiple processing nodes (e.g., the controller 101 and one or more processing devices 103). This distribution is executed, for example, by the computer 106.

[0039] To this end, according to various embodiments, an automated distribution tool (or "distribution orchestrator") 107 (and a corresponding distribution method) is provided (e.g., implemented on a computer 106, which can accordingly be regarded as an orchestration device), which distributes the data processing tasks of the control software (e.g., control tasks) 110 into subtasks 109, which can then be executed by processing nodes, such as multiple processing devices (e.g., the controller 101 itself and at least some of the (additional) processing devices 103, such as edge devices).

[0040] The dynamic distributed deployment and management of real-time / safety-critical applications (i.e., applications that need to perform real-time and / or safety-critical data processing tasks 109) requires an orchestration solution (e.g., data processing arrangement 100) that is able to establish QoS (Quality of Service) guarantees required by the (distributed) system to perform the data processing tasks 110 of these applications. These QoS guarantees are regarding aspects such as real-time requirements, reliability, fault tolerance, security, etc.

[0041] A straightforward approach to building a suitable orchestration solution for such a system involves equipping the orchestrator 107 with supplementary analysis and resource management / configuration capabilities to enable it to (i) deploy applications with suitable configurations to establish real-time guarantees in accordance with their timing requirements, and (ii) continuously monitor system dynamics with respect to specific aspects that are critical to the correct operation of the deployed applications to detect critical events or scenarios, and react to them accordingly (e.g., by adapting the deployment of the affected applications) to maintain or restore the correct operation of the applications in compliance with their real-time requirements.

[0042] This direct approach can establish correct operation of applications according to the real-time requirements of each application both when the application is deployed and after its deployment is adapted in response to critical runtime events (such as resource failures). However, when reacting to critical events, orchestrator 107 is drawn into a reaction loop, making the correct operation of the affected applications strictly dependent on the orchestrator's ability to react to events within the timing constraints of the affected applications. This dependency creates the following problem: to react to system dynamics, orchestrator 107 uses a tool set of (potentially) computationally intensive mapping, analysis, and optimization routines that lack the timing predictability and responsiveness required to provide timely reactions according to the real-time requirements of the applications. On the other hand, applications affected by critical events are required to react in real time within their timing constraints to ensure that their correct operation is preserved or properly re-established.

[0043] In fact, any process (and therefore any data processing task) that is essential for the correct operation of a real-time / safety-critical application is inherently required to adhere to the same real-time constraints as the application itself. Therefore, the runtime risks violating the application's real-time requirements by involving the orchestrator 107 in a reaction loop to critical runtime events in the face of events and system dynamics that could jeopardize the correct operation of the application.

[0044] Furthermore, given the typically limited computational power of resources in a CPS on the one hand, and the high computational power requirements of orchestrator 107 on the other, it is generally desirable that orchestrator 107 be deployed on a processing node (computer 106) with high computational power, potentially remote from the system it controls (e.g., remote from processing devices 103 and controller 101). Given the typically limited communication bandwidth in a CPS system, this spatial distance and the associated added communication "hops" introduce higher communication latency between orchestrator 107 and the rest of the system, which can be significant. Consequently, the resulting communication latency can significantly increase the end-to-end reaction time to critical events. Therefore, it is imperative for time-critical decisions and reactions to occur as close as possible to the affected parts of the system. Finally, having orchestrator 107 involved in the reaction loop for all critical events would make it a single point of failure for the entire system, which is highly undesirable, especially in a safety-critical context.

[0045] According to various embodiments, a method for distributing data processing tasks (and a corresponding distribution tool 107) is provided. In view of the above, the method establishes a real-time reaction mechanism close to the corresponding application, and solves the single point of failure problem caused by the orchestrator 107 by eliminating the orchestrator 107 from the reaction loop of critical runtime events.

[0046] Therefore, according to various embodiments, a method for task distribution in a distributed processing system is provided that includes a dynamically provisioned real-time response mechanism. This method enables processing critical system dynamics in real time within the timing constraints of the affected application(s), without requiring the orchestrator to participate in the response process. This method establishes a decoupled design paradigm, in which the post-deployment operability of applications is decoupled from the orchestrator.

[0047] The key aspect of this distributed approach may be that it involves a priori provisioning of real-time applications with additional services and mechanisms that are able to detect critical runtime events and react to them accordingly. The choice of reaction mechanism depends on the specific event and the requirements of the application, for example in terms of reliability, security, and fault tolerance. The reaction may involve adapting the application's deployment, changing the configuration of its components or the resources they use, or switching its operating mode, for example, switching to a fail-safe mode in the event of an unrecoverable failure. These additional services and mechanisms are deployed with the application or dynamically added to the application, and once established, they ensure that their corresponding reactions are provided within the application's real-time constraints.

[0048] The distribution method is implemented, for example, by computer 106, for example, by automated distribution tool 107. Therefore, according to various embodiments, an orchestration device (e.g., a server computer) is provided, which is configured to execute the distribution method and includes components (especially interfaces, etc.) configured accordingly.

[0049] This distribution method allows data processing tasks to be distributed to multiple processing nodes in a manner that provides

[0050] Ability to react in real-time to critical runtime events: Real-time applications are equipped with (e.g., transient) services and mechanisms that allow reacting in real-time to critical runtime events, freeing the application from dependency on non-real-time orchestrators after deployment.

[0051] Robustness to Orchestrator Unavailability: The distributed approach enables applications (and the data processing tasks performed for them) to be independent of the orchestrator after deployment. This means that the processing system (i.e., the system of processing nodes performing data processing) can operate autonomously, regardless of the responsiveness of the orchestrator or even its presence. This makes the processing system resilient to the (temporary) unavailability of the orchestrator.

[0052] Potential for remote orchestrator deployment: Making the processing system independent of the real-time responsiveness of the orchestrator creates opportunities to deploy the orchestrator on powerful servers that may be located remotely, such as in the cloud, to better accommodate its high computational demands or to enable the use of an orchestrator with high computational demands.

[0053] Dynamic Provisioning of Real-Time Reaction Mechanisms: While distributed approaches remove the dependency of deployed real-time applications on the orchestrator, the potential for the orchestrator to monitor system dynamics remains. In response, it can introduce system adaptations that do not disrupt the established real-time operation of deployed applications, but instead aim to enhance their resilience to critical runtime events. In this context, distributed approaches enable the dynamic enhancement of deployed applications with complementary real-time reaction mechanisms and capabilities to achieve this.

[0054] According to various embodiments, in order to clearly separate the processes and interactions involved in the operation of real-time applications from those not involved, two operational planes are considered in the processing system: a control plane and a data plane. Each process / interaction is associated with one of these planes based on the real-time requirements to which it is subject. Those processes / interactions associated with the control plane are not bound by real-time constraints. The orchestrator and its associated services, such as monitoring services that report to the orchestrator, are part of the control plane. In contrast, the data plane covers real-time processes / interactions. In the distributed approach according to various embodiments, the processes / interactions within each real-time application and all the supplementary services and mechanisms associated with them (regarding real-time reaction capabilities) are part of the data plane.

[0055] like Figure 1 As illustrated in FIG, a distributed processing system is considered below, which consists of a collection of possibly heterogeneous computing nodes (processing nodes) interconnected via a communication network (possibly formed by multiple sub-networks). The processing system includes a component responsible for managing applications and resources in the system, which is called an orchestrator (implemented by the orchestration device 106, in Figure 1 In the example of , orchestration device 106 implements distribution tool 107. Generally speaking, the orchestrator's scope of management does not necessarily encompass deploying all applications in the system. Depending on the use case, it is possible that the orchestrator only oversees a subset of applications. Nevertheless, while the orchestrator does not necessarily manage all applications, it can configure system resources to control interference between existing (possibly externally managed) applications and incoming applications. Furthermore, the orchestrator's scope of management can cover a wide range, including adapting the deployment or configuration of existing applications based on system dynamics, dispatching incoming requests to deploy new applications, dispatching requests to adapt existing applications (e.g., to build fault tolerance into already deployed applications), and so on.

[0056] An application to be hosted in a processing system (i.e., whose data processing tasks are to be performed by the processing system) is represented as a collection of communicating software components, referred to as a (software) module (e.g., corresponding to subtask 109). An incoming request (e.g., at orchestrator 107) to deploy an application or adapt an existing application also provides a description of the application's non-functional requirements, such as real-time constraints, fault tolerance, etc. This information is provided to orchestrator 107 and used to make dynamic deployment and system management decisions. The orchestrator's decisions may require deploying, configuring, and terminating modules across computing nodes and / or configuring resources in the system.

[0057] Figure 2An example of a processing system comprising three processing nodes 201-203 and deployed by orchestrator 204 is shown. Orchestrator 204 can send commands to processing nodes 201-203 and receive confirmations and status information from processing nodes 201-203. In this example, the data processing task is controlling a technical system 205 (e.g., a factory). The connection to technical system 205 is via second processing node 202, which corresponds to controller 101, for example. Orchestrator 204 corresponds to computer 106, for example (or is implemented by computer 106). Each processing node 201-203 executes a corresponding subtask 206-208.

[0058] According to one embodiment, the decision to execute the orchestrator on processing nodes 201-203 (e.g., to deploy or terminate modules on processing nodes or to change the configuration of resources and modules) is facilitated by a software service resident on each processing node, which has sufficient system privileges to enact the commands of the orchestrator. This service has various implementation possibilities, for example, a server implemented as a kernel module or / and a service process with system privileges or / and a runtime-based execution environment for a control module. Herein, this service is referred to as the runtime 209-211 implemented on each processing node 201-203. In addition to formulating the commands of the orchestrator to start, configure and terminate modules on its nodes or configure the resources of its nodes, the runtime can also provide feedback to the orchestrator about the status of its managed modules, the status of node resources, or details about the occurrence of certain events.

[0059] This distributed approach enables provisioning of real-time reactive mechanisms for distributed applications. The following steps outline a general approach for implementing this distributed approach in conjunction with (i.e., for example, performed by) an orchestrator (e.g., orchestrator 204), describing the basic analysis and procedures involved. However, other implementations are possible.

[0060] i. Requirements Analysis: Analyze the application and system resources to identify critical scenarios (such as resource failures) that require real-time responses to the application's requirements. This analysis may involve specialized tools and / or expertise regarding system reliability, fault tolerance, security, resources / techniques, etc. Furthermore, it may introduce additional deployment requirements, such as redundant module (or subtask) deployment or module-node affinity.

[0061] ii. Deployment selection: Based on the deployment requirements of the application modules and the availability of resources, determine the initial deployment of these application modules (i.e., subtasks) on the processing system resources. This can be achieved using the mapping and optimization services of the orchestrator.

[0062] iii. Determine the required failure handling capabilities. Based on the initial deployment and the identified critical scenarios (i.e., failure types), determine the set of failure handling capabilities required for the deployment to effectively respond to these scenarios (failure types). These failure handling capabilities typically involve detecting failure scenarios and responding appropriately to them, such as changing the deployment configuration or switching the application's operating mode.

[0063] iv. Determine the required failure handling services and mechanisms. Determine the specific failure handling mechanisms and services to implement the required capabilities identified. There may be several options for implementing this capability, such as additional software components (to be linked with the application modules) or additional modules (to be deployed with the application modules).

[0064] v. Synthesis: Synthesize the identified failure handling services and mechanisms, such as by linking them to application modules, synthesizing them into separate modules, etc., to provide additional failure handling capabilities identified for the initial deployment equipment.

[0065] vi. Evaluation: The resulting deployment is evaluated with respect to whether it adheres to the real-time constraints of the application. If necessary, the resources involved in the execution of the application (i.e., processing nodes and their components and possibly the communication network connecting the processing nodes) are determined.

[0066] If the deployment fails the assessment, an alternative deployment is selected (return to step ii).

[0067] vii. Deployment Execution: Upon successfully passing the previous evaluation, the deployment is executed by distributing the corresponding modules to the corresponding system resources and applying the corresponding resource / module configuration.

[0068] In the general approach outlined above, steps (i), (iii), (iv) and (v) can be considered the core components of the distribution approach. These steps cover aspects of identifying the required failure handling capabilities in steps (i) and (iii), and incorporating the corresponding failure handling mechanisms into a given application deployment in steps (iv) and (v). Meanwhile, for the remaining steps, established methods and methodologies can be used. For example, for step (ii), specialized techniques such as constraint-based mapping and design space exploration (DSE) can be used to determine the deployment of the application. For step (vi), techniques such as real-time admission testing or timing analysis can be used to assess the compliance of a given deployment (including the mechanisms provided) with the real-time requirements of the application.

[0069] In the following, about Figure 2In the example illustrated in FIG, an example of a distributed method according to items i to vii as described above is given, wherein in this example, the (single) failure type that the processing system should be able to handle is the disconnection of a node (the first processing node 201 is used as an example). Therefore, in the following example, the distributed method provides real-time failover support for a real-time control application to establish the robustness of the real-time control application against node disconnections in the distributed system.

[0070] As mentioned above, the application (i.e., the overall data processing task) is a control application responsible for plant control and consists of two modules: a "sensing and action" module and a "processing" module. The sensing / action (S / A) module periodically collects sensor data from the plant 205 and sends this data to the processing module. Based on this data, the processing module calculates appropriate control data to be applied to the plant 205 and sends this appropriate control data to the sensing / action module, which applies the control signals to the plant 205.

[0071] The processing system consists of a collection of interconnected processing (or computing) nodes, each hosting a runtime instance (or "runtime service") that acts as an agent for the orchestrator. In this example, the second processing node 202 is directly connected to the plant 205, is equipped with sensing / actuation mechanisms for plant interaction, has low computing power (intended only for plant interactions with minimal computing requirements), and is hardened against failures and disconnections.

[0072] Assume that the deployment requirements include that the application must adhere to a given end-to-end latency constraint (from sensor reading to actuation of the control signal). Further, assume that the sensing / action module must be deployed exclusively on the second processing node 202 because it interacts directly with the plant, and assume that the processing module must be deployed on a processing node with high computational power, i.e., different from the second processing node 202. Further, as mentioned above, given the required failure types, the processing system should be able to process the application (data processing task) in a manner that can tolerate the potential disconnection of one processing node.

[0073] In this example, steps i to vii are as follows:

[0074] i. Requirements Analysis: The application's fault tolerance requirements necessitate that the application be robust against potential node disconnections. To this end, it was decided to deploy its modules with real-time failover support, requiring redundant instances (replicas) of each module to be deployed on different nodes. Since the sensing / action module must be deployed on a processing node 202 that is hardened against failure, the sensing / action module is excluded from the redundant deployment. Therefore, redundancy is planned only for the processing module. In order to have the required failure handling capabilities, two instances of the processing module are deployed: one as the failover master module and the other as its hot standby backup module.

[0075] ii. Deployment Selection: Based on the results of the requirements analysis, a total of three modules (subtasks) are considered for deployment: one instance of the sensing / action module on the second processing node 202, and two instances of the processing module on two nodes different from the second processing node. Specifically, the mapping program decides to deploy the primary processing module (P1, subtask 206) on the first processing node 201, and a backup instance (P2, subtask 208) on the third processing node 203. It should be noted that these two subtasks 206 and 208 correspond to (i.e., are copies of) the same subtask 109 of the corresponding data processing task 110, but only one is active: both receive data from the sensing / action module 207 and calculate control signals. Initially, the primary subtask 206 is configured to be active, so that its output data is sent to the sensing / action module 207, and the backup subtask 208 is configured to be inactive, so that its output is discarded, so that for each sensor reading, only one control signal is sent to the sensing / action module 207.

[0076] iii. Determine the required capabilities: Given the deployment from the previous step, the application is required to maintain disconnection of the first processing node 201 hosting the main processing module 206. This requires the ability to detect failure of the first processing node 201. In this event, the main processing module 206 must be deactivated (an action referred to as "silencing" in the following), and

[0077] The backup processing module 208 must be activated (hereinafter referred to as the "unsilencing" action). Therefore, corresponding capabilities are required:

[0078] a. detecting the disconnection on the first processing node 201 and subsequently silencing the local main processing module 206, and

[0079] b. Detecting the disconnection of the first processing node 201 on the third processing node 203 and subsequently unsilencing the backup processing module 208 .

[0080] iv. Determine the required mechanism: the ability to detect disconnection of the first processing node,

[0081] A watchdog-based mechanism is used, which is implemented by a special module. In addition, to formulate failover, two special modules are planned, one for silencing the primary processing module 206 on the first processing node 201 and the other for unsilencing the backup processing module 208 on the third processing node 203. For clarity, these special modules are called "sidecars":

[0082] a. Detector sidecar D1: This sidecar is a watchdog responsible for detecting the disconnection of the first processing node 201. It uses sensor data from the sensing / action module 207. If there is no new sensor data within a predefined interval, it concludes that the first processing node 201 is disconnected and issues a control command indicating that the first processing node 201 is disconnected.

[0083] b. Silencer Sidecar S: This sidecar listens for commands from D1 and, upon receiving the command, silences the main processing module 206 by requesting the runtime 209 of the first processing node to disable the output of the main processing module. This ensures that the main processing module 206 does not generate unwanted data if the first processing node 201 reconnects.

[0084] c. Detector Sidecar D2: This sidecar is a watchdog responsible for detecting disconnection of the first processing node 201. It uses control data from the main processing module 206 and sensor data from the sensing / action module 202. If it does not receive any control data within a predefined interval while still receiving sensor data, it concludes that the first processing node 201 is disconnected. Upon detecting that the first processing node 201 is disconnected, it issues a control command reflecting the disconnection of the first processing node 201.

[0085] d. Unsilencer sidecar US: This sidecar listens for commands from D2 and, upon receiving the command, unsilences the backup processing module 208 by requesting the runtime 211 of the third processing node to activate the output of the backup processing module 208 .

[0086] v. Synthesis, evaluation, and deployment: Add the sidecar as a failure handling module (implementing real-time detection of disconnection of the first processing node of the processing module and failover mechanism) to the set of application modules, resulting in a total of seven modules to be deployed (three modules for subtasks of the data processing task and four failure handling modules).

[0087] Evaluate and validate the deployment to meet the real-time requirements of the application.

[0088] Although orchestrator 204 is not involved in the execution of the application or the real-time failover of the processing modules in the event that first processing node 201 is disconnected, it can still make beneficial adaptations to the application without jeopardizing its real-time guarantees. For example, after one of processing nodes 201 or 203 hosting one of its replicas 206 or 208 is disconnected, orchestrator 204 can provision a new backup and the necessary sidecar for the processing module. This strengthens the application's resilience to potential subsequent node failures, thereby improving its availability. This provisioning is performed by orchestrator 204 and occurs orthogonally (i.e., transparently) to the real-time execution of the application. Therefore, it does not affect the operational correctness or real-time guarantees of the application.

[0089] Figure 3 The diagram illustrates such a deployment adaptation in response to the disconnection of the first processing node 201, encompassing a new backup instance 313 of the processing module (P3) on the fourth processing node 312 and the corresponding sidecar. The recently deployed sidecars D3, D4, S2, and US2 implement the necessary disconnection detection and failover enactment mechanisms. Here, D3 and D4 implement the mechanism for detecting the disconnection of the third processing node 203, similar to the operation of D1 and D2 in the initial deployment described in detail above ( Figure 2 ). Sidecars S2 and US2 also operate similarly to the S and US sidecars in the initial deployment: S2 listens for the output of D3 and, upon receiving this output, silences its local processing instance (P2, which is the current primary processing module after the first processing node 201 is disconnected). US2 listens for the output of D4 and, upon receiving this output, unsilences P3 (which is the new backup processing module). Thus, a real-time failover from P2 to P3 is enacted when the third processing node is disconnected.

[0090] In addition to the deployment scenarios described in the examples above, other scenarios are possible to implement real-time detection and response mechanisms for fault tolerance:

[0091] For disconnect detection: Instead of having the detector sidecar monitor application messages, an alternative approach is to deploy a dedicated heartbeat sidecar on each processing node to send periodic heartbeat messages into the system. The detector sidecar then listens for the heartbeat messages emitted by the heartbeat sidecar. The advantage of this approach over listening to application messages is that it allows for configurable disconnect detection speed by (dynamically) adjusting the heartbeat period of the heartbeat sidecar and the watchdog timeout of the detector sidecar.

[0092] For silencing a primary replica when the node hosting it is disconnected: Instead of having a silencer sidecar send requests to its runtime to change the output configuration of its local (hitherto primary) replica, an alternative approach is to have the silencer act as a relay for the output messages of its local replica. That is, the application's data flow is preconfigured so that the replica's output passes through the silencer. With this scheme, the silencer can act as a switch to relay or block the replica's output from entering the network. If a disconnect occurs, the silencer can discard messages from the replica and thereby prevent them from entering the network without requiring runtime involvement or reconfiguration of the primary replica. A similar scheme can be used for non-silencer sidecars and backup replicas.

[0093] Notifying the orchestrator about node disconnects: Instead of having the runtime provide feedback to the orchestrator about the detection of node disconnects, an alternative approach is to deploy a notifier sidecar on a different node, listening for disconnect messages generated by a local detector sidecar, and relaying that information to the orchestrator. In general, sidecars provide an opportunity to dynamically deploy monitors and notifiers within a system; in response to system dynamics, the orchestrator (or one of the sidecars) can deploy sidecars that are dedicated to monitoring specific events, modules, interactions, or resources within the system. These sidecars can then notify the orchestrator (or other entities interested in the monitored events) when observed events occur, or provide the orchestrator with periodic status updates about specific aspects of the state of the entities they monitor. This feedback can be provided proactively and / or reactively in response to queries from the orchestrator.

[0094] It should be noted that protection may similarly be provided against other types of failures than processing node disconnection, such as stop failures (i.e., a processing node (or processing (sub)task) stops (at least partially) performing its function, such as stopping generating (e.g., control) data), processing errors (output is incorrect or has wrong timing), environmental failures such as buffer overflows (not specific to a processing node or process), or failures of the controlled technical system.

[0095] Another example is a network partition / segmentation failure, such as the failure of a link or switch in a network, which results in the network being divided into disconnected sub-networks (or partitions) that were previously connected via the failed link / switch. Due to the partition failure, communication between different network partitions becomes impossible, while pre-existing communication options within each partition remain intact.

[0096] The case with a node disconnection failure can be considered a special case of this class of failures. There, the network is partitioned into a sub-network containing only the disconnected node and a second sub-network containing the rest of the network.

[0097] To achieve resilience against network partition failures, backup instances of subtasks can be strategically deployed on different nodes across the network, along with additional sidecars to detect the occurrence of network partition failures and activate (one or more) appropriate backup instances in the event that the primary instance is no longer reachable due to a partition on the network.

[0098] It should be noted that this approach to achieving resilience is only applicable due to the distribution of the sidecars that make failovers, so that in the event of a partition failure, the sidecars in each partition still have the necessary connectivity within that partition to detect the failure and react to it (e.g., by activating local backup replicas). On the other hand, a centralized orchestrator, regardless of its computing power, would not be able to make any actions on nodes located in parts of the network that are no longer reachable to the orchestrator (due to a partition failure).

[0099] like Figure 3 As mentioned in the example of , the orchestration device can dynamically adjust the failure handling mechanism or other aspects of the configuration of the processing system, for example, in response to failures (as described above) and changes in system parameters or requirements. Adjustments can include, for example, killing a process (subtask), deploying additional subtasks or subtask instances, instructing a processing node to unblock its output, or starting a subtask. The processing node can notify the orchestration device of detected failures or changes in the environment.

[0100] In summary, according to various embodiments, an orchestration apparatus for a distributed processing system is provided, such as Figure 1 Corresponding to the computer 106.

[0101] The orchestration device includes an input interface that is configured to receive a specification of a data processing task to be performed by the distributed processing system, and a specification of one or more (distributed processing) failure types (i.e., specifications of fault tolerance requirements) that the data processing system should be able to handle when performing the data processing task (e.g., the orchestration device receives a request including this information (specification) via the input interface).

[0102] The orchestration device also includes a command interface (and, for example, a corresponding controller (or system manager) configured to generate commands output by the command interface), which command interface is configured to instruct each of the multiple processing nodes of the distributed processing system to perform at least one corresponding subtask of the data processing task, and instruct each of at least some of the multiple processing nodes to implement one or more failure handling software modules, which are configured to handle failures of specified failure types.

[0103] A processing node may be a separate device, but this is not necessary (ie, a processing device such as a (server) computer may implement multiple processing nodes).

[0104] For example, to handle a failure, for at least one of the processing nodes ("node 1"), a detection module is provided in the processing node (node ​​1), the detection module being configured to detect a failure of the processing node (e.g., silence, lack of output (at least temporarily) of the process of node 1), and a second detection module is provided in one of the other processing nodes, the second detection module being configured to detect a disconnection of the processing node ("node 1"). In response to detecting the failure by the first detection module, the output of the processing node (node ​​1) is blocked (i.e., the processing node is silenced), and in response to detecting the failure by the second detection module, a replacement processing node (which may be a processing node including the second detection module) is activated for the disconnected processing node. Thus, for example, a processing node having a redundant process instance may be deployed and / or activated to handle the failure of node 1. Alternatively, a processing node that implements a safety mechanism may be activated.

[0105] Figure 4 A flow chart 400 is shown illustrating a method for orchestrating a distributed processing system.

[0106] In 401, a specification of a data processing task to be performed by a distributed processing system and a specification of one or more (distributed processing) failure types (i.e., fault tolerance requirements) that the data processing system should be able to handle when performing the data processing task are received (e.g., the orchestration device receives a request including this information (specification) via an input interface).

[0107] In 402, each of a plurality of processing nodes of a distributed processing system is instructed to perform at least one corresponding subtask of a data processing task, and each of at least some of the plurality of processing nodes is instructed to implement one or more failure handling software modules configured to handle failures of a specified failure type.

[0108] Figure 4 The orchestration device and method can be used to execute control software in a distributed manner, which calculates control signals for controlling technical systems, such as, for example, computer-controlled machines such as robots, vehicles, household appliances, power tools, manufacturing machines, personal assistants or access control systems.

[0109] The control software may, for example, process sensor data from various sensors (cameras), such as video, radar, LiDAR, ultrasound, thermal imaging, motion, sonar, temperature, pressure, etc. sensors.

[0110] The orchestration device may be implemented by one or more data processing devices (such as computers or microcontrollers) having one or more data processing units, and may be executed by the one or more data processing devices. Figure 4 The method of (and as described above, for example, implementing an automated distribution tool, such as distribution tool 107, which, for example, performs the method in an automatic manner in response to user input). The term "data processing unit" may be understood to mean any type of entity that implements processing of data or signals. For example, data or signals may be processed according to at least one (i.e., one or more) specific functions performed by the data processing unit. The data processing unit may include or be formed by the following: an analog circuit, a digital circuit, a logic circuit, a microprocessor, a microcontroller, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a field programmable gate array (FPGA), or any combination thereof. Any other apparatus for implementing the corresponding functions described in more detail herein may also be understood to include a data processing unit or a logic circuit. One or more of the method steps described in more detail herein may be performed (e.g., implemented) by the data processing unit through one or more specific functions performed by the data processing unit.

[0111] Thus, according to one embodiment, the method is computer-implemented.

Claims

1. An orchestration device (106, 204) for a distributed processing system (100), comprising: an input interface configured to receive a specification of a data processing task (110) to be executed by the distributed processing system (100) and a specification of one or more failure types that the data processing system should be able to handle when executing the data processing task (110); A command interface configured to instruct each of a plurality of processing nodes (101, 103, 201-203) of the distributed processing system (100) to perform at least one corresponding subtask (109) of the data processing task (110), and to instruct each of at least some of the plurality of processing nodes (101, 103, 201-203) to implement one or more failure handling software modules, wherein the failure handling software modules are configured to handle failures of a specified failure type.

2. The orchestration device (106, 204) of claim 1, wherein the one or more failure handling software modules are configured to handle failures of the specified failure type without communicating with the orchestration device (106, 204).

3. The orchestration device (106, 204) of claim 1 or 2, wherein the input interface is configured to receive specifications of a plurality of software modules, wherein each software module implements a corresponding one of the subtasks (109), and wherein the orchestration device (106, 204) includes a software generator configured to supplement the plurality of software modules with the one or more failure handling software modules.

4. The orchestration device (106, 204) according to any one of claims 1 to 3, wherein the input interface is configured to receive specifications on how to divide the data processing task (110) into the subtasks (109) and / or specifications on requirements for how to execute the data processing task (110), and the orchestration device (106, 204) is configured to distribute the data processing task (110) to the multiple processing nodes (101, 103, 201-204) according to the specifications on how to divide the data processing task (110) into the subtasks (109) and / or the specifications on the requirements.

5. The orchestration device (106, 204) according to any one of claims 1 to 4, wherein the data processing task (110) is a control task of a technical system (102).

6. The orchestration device (106, 204) according to any one of claims 1 to 5, wherein the command interface is configured to, upon a failure that impairs the ability of the distributed processing system to handle failures of the specified failure type, instruct one or more of the multiple processing nodes (101, 103, 201-203) and / or one or more additional processing nodes (312) of the distributed processing system (100) to implement the one or more failure handling software modules and / or one or more additional failure handling modules configured to handle failures of the specified failure type.

7. A method for orchestrating a distributed processing system (100), comprising: receiving (401) a specification of a data processing task (110) to be executed by the distributed processing system (100) and a specification of one or more failure types that the data processing system should be able to handle when executing the data processing task (110); and Instructing (402) each of a plurality of processing nodes (101, 103, 201-203) of the distributed processing system (100) to perform at least one corresponding subtask (109) of the data processing task (110), and instructing each of at least some of the plurality of processing nodes (101, 103, 201-203) to implement one or more failure handling software modules configured to handle failures of a specified failure type.

8. A computer program comprising instructions which, when executed by a computer, cause said computer to carry out the method according to claim 7.

9. A computer-readable medium comprising instructions which, when executed by a computer, cause the computer to carry out the method according to claim 7.

Images