Orchestration device for distributed processing system

Abstract

To provide an orchestration device for a distributed processing system.SOLUTION: According to various embodiments, an orchestration device for a distributed processing system comprises: an input interface configured to receive a specification of a data processing task to be performed by the distributed processing system and of one or more failure types that the data processing system should be able to handle when performing the data processing task; and a command interface configured to instruct each of a plurality of processing nodes of the distributed processing system to perform at least one respective sub-task of the data processing task and instruct each of at least some of the plurality of processing nodes to implement one or more failure handling software modules which are configured to handle failures of specified failure types.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present disclosure relates to orchestration devices, techniques, and methods for distributed processing systems. [Background technology]

[0002] Applications in real-time / safety-critical systems are often bound by strict timing constraints and must comply with strict reliability and safety requirements, such as cyber-physical systems (CPS) from the automotive industry and industrial control / automation, where applications not only need to operate within strict end-to-end latency constraints (e.g., for control loops with sensing, processing, and actuation tasks), but also need to be resilient to faults and failures, relying on mechanisms to detect faults and failures (integrity) and, upon detecting them, enable uninterrupted correct operation of affected applications (fault tolerance) or switch them into a safe mode (safety).

[0003] Real-time / safety-critical systems typically encompass multiple applications, each with unique requirements. In dynamically distributed systems, applications are dynamically added and removed from the system, and deployed applications are sometimes distributed across a heterogeneous set of computing nodes interconnected through a communication network. Distributing and deploying applications in such systems is a complex and involved task. Meeting the individual requirements of each application, according to the availability of resources within the system, demands careful analysis and software / hardware considerations, consisting of real-time, reliability, safety, and domain-specific (e.g., control engineering) aspects. Summary of the Invention [Problem to be solved by the invention]

[0004] State-of-the-art approaches to dynamic distribution and deployment of applications in distributed systems use a central component responsible for managing applications and the system's resources in terms of possible system dynamics. This component, typically called an orchestrator, undertakes tasks such as deploying incoming applications, monitoring the state of deployed applications and resources within the system, and reacting to system dynamics, such as resource failures or changes in workload, by adapting application deployments accordingly. Existing orchestration frameworks, such as Kubernetes, AWS Elastic Container Service (ECS), and AWS Step Functions, are designed for best-effort applications and systems that are not subject to strict timing requirements, primarily cloud-based systems. They lack the mechanisms necessary to enforce application requirements in real-time, safety-critical domains and cannot provide the necessary quality-of-service (QoS) guarantees required for these systems.

[0005] Therefore, a task distribution approach for real-time / safety-critical systems is desirable that has the ability to establish the required QoS guarantees of such systems, which relate to aspects such as real-time requirements, reliability, fault tolerance, and safety. [Means for solving the problem]

[0006] According to various embodiments, an orchestration device for a distributed processing system is provided, comprising: an input interface configured to receive a specification of a data processing task to be performed by the distributed processing system and of one or more fault types that the data processing system should be able to address when performing the data processing task; and a command interface configured to instruct each of a plurality of processing nodes of the distributed processing system (which may be a subset of all processing nodes of the distributed processing system, i.e., the plurality of processing nodes may be some of the processing nodes of the distributed system) to perform at least one respective subtask of the data processing task, and to instruct each of at least some of the plurality of processing nodes to implement one or more fault handling software modules configured to handle faults of a specified fault type;

[0007] Orchestration devices allow real-time applications to deploy (additional) capabilities to react to system dynamics in real time, autonomously, and without orchestrator involvement. This can be used for distributed processing systems with real-time and / or safety-critical applications, such as industrial control and automation, automotive control and automated / autonomous driving, and other real-time systems targeting distributed infrastructure, for example, in the context of Industry 4.0, Industrial Internet of Things (IIoT), or automotive zone architectures.

[0008] Below, various examples are given. Example 1 is an orchestration device as described above. Example 2 is the orchestration device of example 1, wherein the one or more fault handling software modules are configured to handle faults of the specified fault type without requiring communication with the orchestration device.

[0009] In other words, once deployed on a processing node, the fault handling software module can handle faults of that fault type independently of the orchestration device, making it possible to use a central orchestration device that is remote to the processing node and still handle faults in real time.

[0010] Example 3 is the orchestration device of Examples 1 or 2, wherein the input interface is configured to receive specifications of a plurality of software modules, each software module performing a respective one of the subtasks, and the orchestration device comprises a software generator configured to supplement the plurality of software modules with one or more fault handling software modules.

[0011] In other words, the orchestration device adds the ability to handle one or more fault types to a given piece of software, so that the user does not have to deal with preparing code to handle the fault types, but the fault handling is transparent to the user and transparent to the data processing task.

[0012] Example 4 is the orchestration device of any one of Examples 1-3, wherein the input interface is configured to receive a specification of how the data processing task may be separated into sub-tasks and / or a specification of requirements for how the data processing task is to be performed, and wherein the orchestration device is configured to distribute the data processing task to a plurality of processing nodes in accordance with the specification and / or requirements for how the data processing task may be separated into sub-tasks.

[0013] For example, the orchestration device may take into account that data processing tasks should be performed in a manner that meets real-time requirements. Example 5 is the orchestration device of any one of Examples 1-4, wherein the data processing task is a control task of a (eg, real-time) technical system.

[0014] For example, one of the multiple processing nodes is a controller connected to the technical system (and, e.g., located near or installed within the technical system), and at least one of the multiple processing nodes is located remotely (e.g., is an edge node or is located in the cloud) and takes over data processing for the control of the technical system.

[0015] Example 6 is the orchestration device of any one of Examples 1-5, wherein the command interface is configured, upon occurrence of a fault (which may or may not be a fault type) that impairs the ability of the distributed processing system to address faults of the specified fault type, to instruct one or more of the plurality of processing nodes and / or one or more additional processing nodes of the distributed processing system to implement one or more fault handling software modules and / or one or more additional fault handling modules configured to address faults of the specified fault type (so as to restore the processing system's ability to address the one or more specified fault types).

[0016] In other words, the orchestration device can dynamically control the failure handling capabilities of the distributed processing system and can reconfigure the distributed processing system in particular if it loses the ability to handle failures of a particular failure type (e.g., because one processing node is disconnected).

[0017] Example 7 is a method of orchestrating a distributed processing system, comprising receiving a specification of a data processing task to be performed by the distributed processing system and of one or more fault types that the data processing system should be able to handle when performing the data processing task; instructing each of a plurality of processing nodes of the distributed processing system (e.g., a subset of all processing nodes of the distributed system) to perform at least one respective subtask of the data processing task, and instructing each of at least some of the plurality of processing nodes to implement one or more fault handling software modules configured to handle failures of the specified fault types.

[0018] Example 8 is a computer program comprising instructions that, when executed by a computer, cause the computer to perform the method according to Example 7. Example 9 is a computer-readable medium comprising instructions that, when executed by a computer, cause the computer to perform a method according to Example 7.

[0019] The examples can be combined and features described in the context of an orchestration device are analogously applicable to the method. In the drawings, like reference characters generally refer to the same parts throughout the different views. The drawings are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of the invention. In the following description, various aspects are described with reference to the following drawings: [Brief explanation of the drawings]

[0020] [Figure 1] FIG. 1 illustrates a data processing device. [Figure 2] FIG. 1 illustrates an example processing system that includes three processing nodes and is deployed by an orchestrator. [Figure 3] FIG. 3 shows an adaptation of the development of FIG. 2. [Figure 4]FIG. 1 is a flow diagram illustrating a method for orchestrating a distributed processing system, according to one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0021] The following detailed description refers to the accompanying drawings, which show, by way of example, specific details and embodiments of the present disclosure in which the invention may be practiced. Other embodiments may be utilized, and structural, logical, and electrical changes may be made without departing from the scope of the present disclosure. Various aspects of the present disclosure are not necessarily mutually exclusive, as some aspects of the present disclosure may be combined with one or more other aspects of the present disclosure to form new aspects.

[0022] Various examples are described in more detail below. FIG. 1 shows a data processing apparatus 100 . The data processing apparatus 100 includes a controller 101 arranged to control a controlled system 102, such as a robot, a vehicle, a machine or the like.

[0023] The data processing apparatus 100 further comprises (additional) processing devices 103 connected to the controller 101 (and possibly to each other) by communication connections 104 which may be realized by various interconnection technologies, such as a wireless and / or wired communication network 108, a bus, etc. For example, at least some of the processing devices 103 are located in an edge cloud. The data processing apparatus 100 for example implements a cyber-physical system (CPS).

[0024] Each processing device 103 may implement one or more processing nodes. For example, a processing device 103 may correspond to a single processing node, but a processing device 103 may also include a sufficient amount of resources (e.g., a server computer) to implement multiple processing nodes.

[0025] According to various embodiments, the control software that controls the controlled system 102 (and is designed to run on the controller 101) is executed in a distributed manner, i.e., distributed across multiple processing nodes, e.g., the controller 101 and one or more processing devices 103. The distribution is performed, for example, by the computer 106.

[0026] To this end, according to various embodiments, an automated distribution tool (or “distribution orchestrator”) 107 (and corresponding distribution methodology) is provided (e.g., implemented on a computer 106, which may accordingly be seen as an orchestration device) that distributes the data processing tasks (e.g., control tasks) 110 of the control software into subtasks 109, which may then be executed by processing nodes, e.g., multiple processing devices (e.g., the controller 101 itself and at least some of the (additional) processing devices 103, e.g., edge devices).

[0027] Dynamic distributed deployment and management of real-time / safety-critical applications (i.e., applications in which real-time and / or safety-critical data processing tasks 109 need to be performed) requires an orchestration solution that can establish the required QoS (Quality of Service) guarantees of the (distributed) systems performing the data processing tasks 110 for these applications (e.g., data processing devices 100) related to real-time requirements, reliability, fault tolerance, safety, etc.

[0028] A straightforward approach to establishing an appropriate orchestration solution for such systems involves equipping the orchestrator 107 with complementary analysis and resource management / configuration capabilities to be able to: (i) deploy applications with a configuration suitable for establishing real-time guarantees according to their timing requirements; and (ii) continuously monitor the system dynamics with respect to certain aspects that are critical for the correct operation of the deployed applications to detect important events and scenarios and react to them accordingly (e.g., by adapting the deployment of the affected applications) to maintain or resume the correct operation of the applications in compliance with their real-time requirements.

[0029] This straightforward approach can potentially establish correct operation of applications according to their real-time requirements when deploying applications, and after those deployments are adapted in response to critical runtime events such as resource failures. However, having the orchestrator 107 in the reaction loop involves ensuring the correct operation of the affected applications, strictly relying on the orchestrator's ability to react to the event within the timing constraints of the affected applications. This dependency poses a problem: to react to system dynamics, the orchestrator 107 employs a (potentially) computationally intensive toolset of mapping, analysis, and optimization procedures, lacking the timing predictability and responsiveness necessary to provide a timely response according to the applications' real-time requirements. Meanwhile, for applications affected by critical events, real-time reaction within their timing constraints is necessary to ensure their correct operation is preserved or appropriately re-established.

[0030] Indeed, any process (and therefore any data processing task) that is essential to the correct operation of a real-time / safety-critical application is inherently required to adhere to the same real-time constraints as the application itself. Thus, having an orchestrator 107 involved in the reaction loop for critical runtime events necessarily carries the risk of violating the real-time requirements of the application in the face of system dynamics and events that may jeopardize their correct operation.

[0031] Furthermore, given the typically limited computational power of resources within a CPS on the one hand and the high computational power requirements of the orchestrator 107 on the other, it is often desirable for the orchestrator 107 to be deployed on a processing node (computer 106) with high computational power and potentially remote from the system it controls (e.g., remote from the processing device 103 and controller 101). This spatial distance and the associated additional communication “hops” lead to high communication latency between the orchestrator 107 and the rest of the system, which can be significant given the typically limited communication bandwidth in CPS systems. The resulting communication latency can therefore significantly increase the end-to-end reaction time to critical events. Therefore, it is essential that time-critical decisions and reactions occur as close to the affected parts of the system as possible. Finally, involving the orchestrator 107 in the reaction loop of all critical events makes it a single point of failure for the entire system, which is highly undesirable, especially in safety-critical contexts.

[0032] Various embodiments provide a data processing task distribution approach (and corresponding distribution tools 107) that, in view of the above, establishes a real-time reaction mechanism in close proximity to each application, which solves the problem of a single point of failure created by the orchestrator 107 by removing the orchestrator 107 from the reaction loop of critical runtime events.

[0033] Thus, according to various embodiments, a task distribution method is provided that includes dynamic provisioning of real-time reactive mechanisms in a distributed processing system, which allows the reactive process to address critical system dynamics in a real-time manner, within the timing constraints of the affected applications, without orchestrator involvement. This method establishes a paradigm of decoupled design, in which the post-deployment operability of applications is decoupled from the orchestrator.

[0034] A key aspect of the distributed approach can be seen in that it involves provisioning a real-time application a priori with additional services and mechanisms that can detect important runtime events and react to them accordingly. The choice of reaction mechanism depends on the concrete event and the requirements of the application, e.g., in terms of reliability, safety, and fault tolerance. The reaction can involve adapting the application's deployment, modifying its components or the resources they use, or switching its operation mode, e.g., to a fail-safe mode in case of an unrecoverable failure. These additional services and mechanisms are deployed together with the application or dynamically added to it, and once established, they are guaranteed to provide the respective reaction within the application's real-time constraints.

[0035] The distribution method is performed, for example, by computer 106, for example, by automated distribution tool 107. Thus, according to various embodiments, an orchestration device (e.g., a server computer) is provided that is configured to perform the distribution method and that comprises components (e.g., interfaces, among others) that are configured accordingly.

[0036] This distribution method allows data processing tasks to be distributed across multiple processing nodes in a manner that results in: · Ability to react in real time to critical runtime events Real-time applications are equipped with (eg, interim) services and mechanisms that enable real-time reactions to critical runtime events and remove the application's dependency on non-real-time orchestrators after deployment.

[0037] Robustness against orchestrator unavailability The distributed approach achieves independence of applications (and the data processing tasks performed on them) from the orchestrator after deployment. This means that the processing system (i.e. the system of processing nodes performing data processing) can operate autonomously regardless of the responsiveness, or even the existence, of the orchestrator. This makes the processing system resilient to the (temporary) unavailability of the orchestrator.

[0038] Possibility of remote orchestrator deployment The independence of the processing system from the real-time responsiveness of the orchestrator potentially creates the opportunity to deploy the orchestrator remotely, for example on a powerful server located in the cloud, to ameliorate its high computational demands or enable the use of orchestrators with high computational demands.

[0039] Dynamic provisioning of real-time reaction mechanisms Although the distributed approach removes the dependency of deployed real-time applications from the orchestrator, there is still the possibility for the orchestrator to monitor system dynamics. In response, it can introduce system adaptations that enhance the resilience of deployed applications to critical runtime events without disrupting their established real-time behavior. In this regard, the distributed approach makes it possible to dynamically enhance already deployed applications with additional real-time reaction mechanisms and capabilities to achieve this.

[0040] According to various embodiments, two operational planes in a processing system are considered: a control plane and a data plane, to clearly isolate processes and interactions involved in the operation of real-time applications from those that are not. Each process / interaction is associated with one of these planes based on the real-time requirements governing it. Those associated with the control plane are not bound by real-time constraints. The orchestrator and its associated services, e.g., monitoring services that report to the orchestrator, are part of the control plane. Conversely, the data plane encompasses real-time processes / interactions. In a distributed method according to various embodiments, the processes / interactions within each real-time application, as well as all supplemental services and mechanisms associated with them related to real-time reaction capabilities, are part of the data plane.

[0041] As shown in FIG. 1 , a distributed processing system is hereinafter considered to consist of a set of possibly heterogeneous computing nodes (processing nodes) interconnected through a communication network (possibly formed of multiple subnetworks). The processing system includes a component, called an orchestrator (implemented in the example of FIG. 1 by orchestration device 106 implementing distribution tool 107), that is responsible for managing applications and resources within the system. In general, the orchestrator's management scope does not necessarily encompass deploying all applications within the system. Depending on the use case, it is possible for the orchestrator to oversee only a subset of applications. Nevertheless, while not necessarily managing all applications, the orchestrator can configure system resources to control interference between existing (possibly externally managed) and upcoming applications. Furthermore, the orchestrator's management scope may cover a wide area, including deploying or configuring existing applications based on system dynamics, servicing upcoming requests to deploy new applications, and servicing requests to adapt existing applications (e.g., to establish fault tolerance for already deployed applications).

[0042] Applications to be hosted in the processing system (i.e., the data processing tasks to be performed by the processing system) are represented as communicating software components called (software) modules (e.g., corresponding to subtasks 109). An incoming request (e.g., to orchestrator 107) to deploy an application or adapt an existing one also results in a description of the application's non-functional requirements with respect to real-time constraints, fault tolerance, etc. This information is fed to orchestrator 107 and used to make dynamic deployment and system management decisions. The orchestrator's decisions entail deploying, configuring, and terminating modules across compute nodes and / or configuring resources within the system.

[0043] 2 illustrates an example of a processing system including three processing nodes 201-203 and deployed by an orchestrator 204. The orchestrator 204 can send commands to the processing nodes 201-203 and can receive acknowledgements and status information from the processing nodes 201-203. In this example, the data processing task is the control of a technical system 205 (e.g., a plant). The connection to the technical system 205 is through a second processing node 202, which corresponds, for example, to the controller 101. The orchestrator 204 corresponds, for example, to (or is implemented by) the computer 106. Each processing node 201-203 performs a respective subtask 206-208.

[0044] According to one embodiment, orchestrator decisions for processing nodes 201-203 (e.g., deploying or terminating modules on a processing node, or changing resource or module configurations) are facilitated by a software service resident on each processing node with appropriate system privileges to fulfill the orchestrator's commands. Various implementation possibilities exist for this service, such as a server implemented as a kernel module, or / and a service process with system privileges, or / and a runtime-based execution environment that controls the modules. This service is referred to herein as a runtime 209-211, implemented on each processing node 201-203. In addition to fulfilling the orchestrator's commands to start, configure, and terminate modules on its node or configure its node's resources, the runtime can also provide feedback to the orchestrator about details regarding the state of its hosted modules, the state of the node's resources, or the occurrence of certain events.

[0045] The distributed methodology enables provisioning of distributed applications with real-time responsiveness. The following steps outline an overall approach to performing this distributed methodology in conjunction with (e.g., performed by) an orchestrator (e.g., orchestrator 204) and describe the essential analyses and procedures involved. However, other implementations are possible.

[0046] i.Requirements analysis Application and system resources are analyzed to identify critical scenarios, such as resource failures, that require real-time responses according to application requirements. Such analysis may involve specialized tools and / or expertise on reliability, fault tolerance, safety, system resources / technologies, etc. Furthermore, additional deployment requirements may be introduced, such as redundant module (or subtask) deployment or module-node affinity relationships.

[0047] ii. Deployment Selection The pre-deployment of modules (i.e., subtasks) of an application on processing system resources is determined based on their deployment requirements and resource availability, which can be achieved using the orchestrator's mapping and optimization services.

[0048] iii. Determining the necessary disaster response capabilities Based on the pre-deployment and the identified critical scenarios (i.e., failure types), a set of failure capabilities required for the deployment is determined to react effectively to these scenarios (failure types). These failure capabilities typically involve detecting failure scenarios and reacting appropriately to them, for example, changing the deployment configuration or switching the operating mode of the application.

[0049] iv. Determining the necessary troubleshooting services and mechanisms Specific fault handling mechanisms and services for realizing the determined required capabilities are determined. Several options may be possible for realizing the capabilities, for example, additional software components (to be linked to the modules of the application) or additional modules (to be deployed together with the application modules).

[0050] v.Synthesis The determined fault handling services and mechanisms are composed by linking them to application modules, by composing them as separate modules, etc., to equip the pre-deployment with the determined additional fault handling capabilities.

[0051] vi. Evaluation The resulting deployment is evaluated for compliance with the application's real-time constraints. If necessary, an appropriate configuration of the resources involved in running the application (i.e., processing nodes and their components, and possibly the communication network connecting the processing nodes) is determined. If the deployment fails the evaluation, an alternative deployment is selected (return to step ii).

[0052] vii. Deployment execution Upon passing the preceding evaluation, deployment is performed by distributing each module to the corresponding system resources and applying the respective resource / module configuration.

[0053] In the overall approach outlined above, steps (i), (iii), (iv), and (v) can be seen as the core components of a distributed methodology. These steps cover aspects of identifying the required fault handling capabilities in steps (i) and (iii) and integrating the corresponding fault handling mechanisms into a given deployment of an application in steps (iv) and (v). Meanwhile, established approaches and methodologies can be used for the remaining steps. For example, for step (ii), specialized techniques such as constraint-based mapping and design space exploration (DSE) can be used to determine the deployment of the application. For step (vi), techniques such as real-time acceptance testing or timing analysis can be used to evaluate the compliance of a given deployment (including the provisioned mechanisms) with the application's real-time requirements.

[0054] In the following, an example of the distributed method according to i-vii as described above is given for the example shown in Figure 2, where the (single) failure type that the processing system should be capable of handling is, in this example, a node disconnection (first processing node 201 is used as an example). Thus, in the following example, the distributed method provides real-time failover support for real-time control applications to establish robustness against node disconnections in the distributed system.

[0055] As mentioned above, the application (i.e., the overall data processing task) is a control application responsible for plant control and consists of two modules: a "sense and act" and a "process" module. The sense / act (S / A) module periodically collects sensor data from the plant 205 and sends them to the process module. Based on this data, the process module calculates the appropriate control data to be applied to the plant 205 and sends this to the sense / act module, which applies a control signal to the plant 205.

[0056] The processing system consists of a set of interconnected processing (or computational) nodes, each hosting an instance of the runtime (or "runtime service") that acts as an agent for the orchestrator. In this current example, the second processing node 202 is directly connected to the plant 205, is equipped with sensing / actuation mechanisms for plant interaction, has low computational power (only plant interaction with minimal computational demands is intended), and is hardened against faults and disconnections.

[0057] Assume that the deployment requirements include that the application must comply with a given end-to-end latency constraint (from sensor readout to control signal actuation). Further assume that the sensing / actuation modules, with direct interaction with the plant, must be deployed exclusively on the second processing node 202, and that the process modules must be deployed on processing nodes with high computational power, i.e., other than the second processing node 202. Furthermore, as stated above, the required failure type is that the processing system should be able to handle the application (data processing tasks) in a way that the potential disconnection of one processing node can be tolerated.

[0058] Steps i to vii are as follows in this example: i.Requirements analysis The fault tolerance requirements of the application entail that it must be robust against potential node outages. To this end, it is decided to deploy with real-time failover support for its modules, requiring that redundant instances (replicas) of each module be deployed on different nodes. The detection / action module is excluded from the redundant deployment because it must be deployed on a processing node 202 that is hardened against failures. Therefore, redundancy is planned only for the process module. To have the required fault tolerance, two instances of the process module are deployed, one as a failover primary and the other as a hot standby backup.

[0059] ii. Deployment Selection According to the results of the requirements analysis, a total of three modules (subtasks) are expected to be deployed: one instance of the sensing / actuation module on the second processing node 202 and two instances of the process module on the two nodes other than the second processing node. Specifically, the mapping procedure determines to deploy a primary process module (P1, subtask 206) on the first processing node 201 and a backup instance (P2, subtask 208) on the third processing node 203. It should be noted that these two subtasks 206, 208 correspond to (i.e., are replicas of) the same subtasks 109 of the respective data processing tasks 110, but only one is active; both receive data from the sensing / actuation module 207 and calculate control signals. Initially, the primary subtask 206 is configured to be active such that its output data is sent to the sensing / actuation module 207, and the backup subtask 208 is configured to be inactive such that its output is discarded and only a control signal is sent to the sensing / actuation module 207 for each sensor reading.

[0060] iii. Determine the required competencies Given the evolution from the previous step, the application is required to maintain the disconnection of the first processing node 201, which hosts the primary process module 206. This requires the ability to detect a failure of the first processing node 201. When that event occurs, the primary process module 206 must be deactivated (an action hereinafter referred to as "silencing"), and the backup process module 208 must be activated (an action hereinafter referred to as "unsilencing"). Therefore, a corresponding capability is required.

[0061] a. on the first processing node 201, detecting the disconnection and subsequently silencing the local primary process module 206; and b. On the third processing node 203, detecting the disconnection of the first processing node 201 and subsequently unsilencing the backup process module 208.

[0062] iv. Determine the mechanisms needed A watchdog-based mechanism implemented by special modules is used for the ability to detect disconnection of the first processing node. Also, two special modules are planned, one for silencing the primary process module 206 on the first processing node 201 and the other for unsilencing the backup process module 208 on the third processing node 203. For clarity, these special modules are referred to as "sidecars."

[0063] a. Detector Sidecar D1 This sidecar is a watchdog responsible for detecting the disconnection of the first processing node 201. It consumes sensor data from the detection / action module 207. If there is no new sensor data for a predefined time interval, it concludes that the first processing node 201 has disconnected and issues a control command indicating that the first processing node 201 has disconnected.

[0064] b. Silencer Sidecar S This sidecar listens for commands from D1 and, upon receiving one, silences the primary process module 206 by requesting the runtime 209 of the first processing node to deactivate the output of the primary process module 206. This ensures that in the event of a reconnection of the first processing node 201, the primary process module 206 does not generate unwanted data.

[0065] c. Detector Sidecar D2 This sidecar is a watchdog responsible for detecting the disconnection of the first processing node 201. It consumes control data from the primary process module 206 and sensor data from the sensing / actuation module 202. If it does not receive any control data for a predefined time interval, it concludes that the first processing node 201 has disconnected, while sensor data is still being received. Upon detecting the disconnection of the first processing node 201, it issues a control command reflecting that the first processing node 201 has disconnected.

[0066] d.Unsilencer Sidecar US This sidecar listens for commands from D2 and, upon receiving one, unsilences the backup process module 208 by requesting the runtime 211 of the third processing node to activate the output of the backup process module 208.

[0067] v. Synthesis, Evaluation, and Development A sidecar is added to the set of modules of the application as a fault handling module (realizing real-time detection of the disconnection of the first processing node and a failover mechanism for the process module), resulting in a total of seven modules being deployed (three modules for subtasks of the data processing task and four fault handling modules).The deployment is then evaluated to verify that it meets the real-time requirements of the application.

[0068] Although orchestrator 204 is not involved in the execution of the application or the real-time failover of the process module in the event of the loss of the first processing node 201, it can still be an advantageous fit for the application without compromising its real-time guarantees. For example, orchestrator 204 can provision a new backup and necessary sidecar for the process module after one of processing nodes 201, 203, which hosts one of its replicas 206, 208, is lost. This hardens the application against potential subsequent node failures, thereby increasing its availability. This provisioning is done by orchestrator 204 and occurs orthogonally (i.e., transparently) to the real-time execution of the application. Therefore, it does not affect the application's operational correctness or real-time guarantees.

[0069] FIG. 3 illustrates such a deployment adaptation in response to the disconnection of the first processing node 201, involving a new backup instance 313 of a process module (P3) on a fourth processing node 312 and its corresponding sidecar. The necessary disconnection detection and failover execution mechanisms are implemented by newly deployed sidecars D3, D4, S2, and US2. Here, D3 and D4 implement mechanisms for detecting the disconnection of the third processing node 203, similar to the operation of D1 and D2 in the initial deployment detailed above (and illustrated in FIG. 2). Sidecars S2 and US2 also operate similarly to the S and US sidecars in the initial deployment, with S2 listening to the output of D3 and, upon receiving it, silencing its local process instance (P2, the current primary process module after the disconnection of the first processing node 201). US2 listens to the output of D4 and, upon receiving it, unsilencing P3 (the new backup process module). Disconnecting the third processing node thereby accomplishes real-time failover from P2 to P3. In addition to the deployment schemes described in the above examples, other schemes are possible to realize real-time detection and reaction mechanisms for fault tolerance.

[0070] -For disconnection detection Instead of having a detector sidecar monitor application messages, an alternative approach is to deploy a dedicated heartbeat sidecar on each processing node to send periodic heartbeat messages into the system. The detector sidecar then listens for heartbeat messages emitted by the heartbeat sidecar. The advantage of this approach over listening to application messages is that it makes it possible to configure the speed of disconnection detection by (dynamically) adjusting the heartbeat period of the heartbeat sidecar and the watchdog timeout of the detector sidecar.

[0071] To silence a primary replica if the node that hosts it disconnects Instead of having the silencer sidecar send a request to its runtime to change the output configuration of its local (previously primary) replica, an alternative approach is to have the silencer act as a relay for its local replica's output messages. That is, pre-configure the application's data flow so that the replica's output passes through the silencer. In this manner, the silencer can act as a switch to relay or block the replica's output from entering the network. In the event of a disconnection, the silencer can discard messages from the replica, thereby preventing them from entering the network without requiring the primary's runtime or configuration to be involved. A similar approach can be used for unsilencer sidecars and backup replicas.

[0072] Notify the orchestrator about node disconnections Instead of having runtime provide feedback to the orchestrator regarding the detection of node disconnection, an alternative approach is to deploy an annunciator sidecar on a different node, listening for disconnection messages generated by the local detector sidecar and relaying this information to the orchestrator. In general, sidecars provide an opportunity for dynamic deployment of monitors and annunciators within a system; in response to system dynamics, the orchestrator (or one of the sidecars) can deploy sidecars, modules, interactions, or resources within the system specialized to monitor specific events. These sidecars can then notify the orchestrator (or other entities interested in the monitored events) when the observed event occurs, or provide regular state updates to the orchestrator for specific aspects of the state of the objects they monitor. This feedback can be provided proactively, in response to queries from the orchestrator, and / or passively.

[0073] It should be noted that protection against types of failures other than processing node disconnection can be provided in a similar manner, for example a stop failure (i.e. a processing node (or processing (sub)task) stops its function (at least partially), e.g. stops producing (e.g. controlling) data), a processing error (output is incorrect or has incorrect timing), a buffer overflow (not specific to a processing node or process) or an environmental failure such as a failure of the technical system being controlled.

[0074] A further example is a network partition / segmentation failure, e.g., the failure of a link or switch within the network, which results in the network being partitioned into disconnected sub-networks (or partitions) that were previously connected through the failed link / switch. As a result of the partition failure, communication between different network partitions becomes impossible, while existing communication options within each partition remain intact.

[0075] A case with a disconnected node failure can be seen as a special case of this failure category, where the network is partitioned into a sub-network containing only the disconnected node and a second sub-network containing the rest of the network.

[0076] To achieve resilience against network partition failures, backup instances of subtasks are strategically deployed on different nodes across the network, along with additional sidecars, to detect the occurrence of a network partition failure and activate the appropriate backup instance in case the primary instance is no longer reachable due to network partitioning.

[0077] It should be noted that this scheme for achieving resilience is only applicable due to the distribution of sidecars that perform failover, so that in the event of a partition failure, the sidecars in each partition still have the necessary connectivity within that partition to detect the failure and react to it (e.g., by activating a local backup replica). On the other hand, the centralized orchestrator, regardless of its computational power, cannot perform any actions on nodes located in parts of the network that are no longer reachable by the orchestrator (due to the partition failure).

[0078] As noted in the example of Figure 3, the orchestration device may dynamically adjust fault handling mechanisms or other aspects of the processing system's configuration, for example, in response to faults (as described above) as well as changes in system parameters or requirements. Adjustments may include, for example, disabling processes (subtasks), deploying additional subtasks or subtask instances, or instructing processing nodes to unblock their outputs or start subtasks. Processing nodes may notify the orchestration device about detected faults or environmental changes.

[0079] In summary, according to various embodiments, an orchestration device for a distributed processing system is provided corresponding to, for example, computer 106 of FIG. The orchestration device has an input interface configured to receive specifications of data processing tasks to be performed by the distributed processing system and of one or more (distributed processing) fault types that the data processing system should be able to handle when performing the data processing tasks (i.e., fault tolerance requirements) (e.g., the orchestration device receives a request including this information (specification) through the input interface).

[0080] The orchestration device further comprises a command interface (and, e.g., a corresponding controller (or system manager) configured to generate command output via the command interface) configured to instruct each of a plurality of processing nodes of the distributed processing system to perform at least one respective subtask of the data processing task, and to instruct each of at least some of the plurality of processing nodes to implement one or more fault handling software modules configured to handle faults of a specified fault type.

[0081] A processing node may be, but is not required to be, a separate device (i.e., a processing device (e.g., a (server) computer) may implement multiple processing nodes).

[0082] For example, to address a failure, for at least one of the processing nodes (“Node 1”), a detection module is provided within the processing node (Node 1) configured to detect a failure of the processing node (e.g., process silencing of Node 1, lack of output (at least temporarily)), and a second detection module is provided within one of the other processing nodes configured to detect a disconnection of the processing node (“Node 1”). In response to detection of the failure by the first detection module, the output of the processing node (Node 1) is blocked (i.e., the processing node is silenced), and in response to detection of the failure by the second detection module, an alternative processing node (which may include the second detection module) is activated for the disconnected processing node. Thus, for example, a processing node having a redundant process instance may be deployed and / or activated to address a failure of Node 1. Alternatively, a processing node implementing a safety mechanism may be activated.

[0083] FIG. 4 shows a flow diagram 400 illustrating a method for orchestrating a distributed processing system. At 401, a specification of a data processing task to be performed by a distributed processing system and of one or more (distributed processing) fault types that the data processing system should be able to handle when performing the data processing task (i.e., fault tolerance requirements) is received (e.g., the orchestration device receives a request including this information (specification) through an input interface).

[0084] At 402, each of a plurality of processing nodes of the distributed processing system is instructed to perform at least one respective subtask of the data processing task and to instruct each of at least some of the plurality of processing nodes to implement one or more fault handling software modules configured to handle faults of a specified fault type.

[0085] The orchestration device and approach of FIG. 4 can be used to execute control software in a distributed manner that calculates control signals for controlling technical systems, such as computer-controlled machines, robots, vehicles, home appliances, power tools, manufacturing machines, personal assistants, or access control systems.

[0086] The control software may process sensor data from various sensors (cameras), such as video, radar, LiDAR, ultrasound, thermal imaging, motion, sonar, temperature, pressure, and others.

[0087] The orchestration device may be implemented, and the method of FIG. 4 may be performed, by one or more data processing devices (e.g., computers or microcontrollers) having one or more data processing units (and, as described above, implementing, for example, an automated distributed tool, e.g., distributed tool 107, which performs the method in an automated manner, e.g., in response to user input). The term “data processing unit” may be understood to mean any type of entity that enables processing of data or signals. For example, data or signals may be addressed according to at least one (i.e., one, or more) specific function performed by the data processing unit. The data processing unit may include or be formed from analog circuitry, digital circuitry, logic circuitry, microprocessor, microcontroller, central processing unit (CPU), graphics processing unit (GPU), digital signal processor (DSP), field programmable gate array (FPGA), or any combination thereof. Any other means for performing the respective functions described in more detail herein may also be understood to include a data processing unit or logic circuitry. One or more of the method steps described in more detail herein may be performed (e.g., implemented) by a data processing unit through one or more specific functions performed by the data processing unit.

[0088] Thus, according to one embodiment, the method is computer-implemented. [Explanation of symbols]

[0089] 100 Data processing device 101 Controller 102 System 103 Processing Device 104 Communication Connections 106 Computer 107 Orchestrator 108 Communication Network 109 Subtasks 110 Data Processing Tasks 201 processing nodes 202 processing nodes 203 processing nodes 204 Orchestrator 205 Technical Systems 206 subtasks 207 subtasks 208 subtasks 209 Runtime 210 Runtime 211 Runtime 312 processing nodes 313 Backup Instance 401 Steps 402 Step

Claims

1. An orchestration device (106, 204) for a distributed processing system (100), comprising: an input interface configured to receive a specification of a data processing task (110) to be performed by the distributed processing system (100) and of one or more fault types that the data processing system should be able to handle when performing the data processing task (110); a command interface configured to instruct each of a plurality of processing nodes (101, 103, 201-203) of the distributed processing system (100) to perform at least one respective subtask (109) of the data processing task (110), and to instruct each of at least some of the plurality of processing nodes (101, 103, 201-203) to implement one or more fault handling software modules configured to handle faults of the specified fault type; An orchestration device (106, 204) comprising:

2. The orchestration device (106, 204) of claim 1, wherein the one or more fault handling software modules are configured to handle faults of the specified fault type without requiring communication with the orchestration device (106, 204).

3. 3. The orchestration device (106, 204) of claim 1 or 2, wherein the input interface is configured to receive specifications of a plurality of software modules, each software module performing a respective one of the subtasks (109), and the orchestration device (106, 204) comprises a software generator configured to supplement the plurality of software modules with one or more fault handling software modules.

4. The orchestration device (106, 204) of any one of claims 1 to 3, wherein the input interface is configured to receive specifications on how the data processing task (110) can be separated into the subtasks (109) and / or specifications of requirements on how the data processing task (110) is to be performed, and the orchestration device (106, 204) is configured to distribute the data processing task (110) to the plurality of processing nodes (101, 103, 201-203) in accordance with the specifications and / or requirements on how the data processing task (110) can be separated into the subtasks (109).

5. The orchestration device (106, 204) according to any one of claims 1 to 4, wherein the data processing task (110) is a control task of a technical system (102).

6. The orchestration device (106, 204) of any one of claims 1 to 5, wherein the command interface is configured to instruct one or more of the plurality of processing nodes (101, 103, 201-203) and / or one or more additional processing nodes (312) of the distributed processing system (100) to implement the one or more fault handling software modules and / or one or more additional fault handling modules configured to handle faults of the specified fault type, upon the occurrence of a fault that impairs the ability of the distributed processing system to handle faults of the specified fault type.

7. A method for orchestrating a distributed processing system (100), comprising: receiving (401) a specification of a data processing task (110) to be performed by said distributed processing system (100) and of one or more fault types that said data processing system should be able to handle when performing said data processing task (110); instructing each of a plurality of processing nodes (101, 103, 201-203) of the distributed processing system (100) to perform at least one respective subtask (109) of the data processing task (110), and instructing each of at least some of the plurality of processing nodes (101, 103, 201-203) to implement one or more fault handling software modules configured to handle failures of the specified fault type; A method comprising:

8. A computer program comprising instructions which, when executed by a computer, cause the computer to perform the method of claim 7.

9. A computer-readable medium comprising instructions that, when executed by a computer, cause the computer to perform the method of claim 7.

Images