Architecture in industrial production plant

Abstract

Embodiments of the present disclosure relate to orchestration in an industrial production plant. In the field of industrial automation, manual task arrangement processing in a global layer is labor-intensive, and errors and compatibility problems are easy to occur at the same time. A federated orchestration system is provided herein. A joint orchestration system includes: a plurality of in-plant orchestrators assigned to respective ones of a plurality of production plants, where each in-plant orchestrator is operable to perform one or more in-plant orchestration tasks with respect to the respective production plant; and an inter-plant orchestrator operable to perform one or more inter-plant orchestration tasks. The inter-plant orchestrator and the in-plant orchestrator are operable to cooperate to orchestrate resources and services of a plurality of production plants.

Description

Technical Field

[0001] The present invention relates to a joint orchestration system, an inter-factory orchestrator for a joint orchestration system, an intra-factory orchestrator for a joint orchestration system, and corresponding methods. Background Art

[0002] In the field of industrial automation, factory owners typically use corresponding orchestration systems to operate multiple factories. Although the orchestration system of each factory undertakes specific tasks of locally managing and optimizing the IT / OT infrastructure at the factory level, multi-factory operators face tasks at the global level that cannot be handled by the current orchestration systems. Manually handling these tasks requires a large amount of manpower and is prone to errors and compatibility issues. Summary of the Invention

[0003] To better solve one or more of these problems, in a first aspect of the present invention, a joint orchestration system is provided, including:

[0004] A plurality of intra-factory orchestrators, assigned to corresponding production factories among a plurality of production factories, wherein each intra-factory orchestrator is operable to execute one or more intra-factory orchestration tasks regarding the corresponding production factory; and

[0005] An inter-factory orchestrator, operable to execute one or more inter-factory orchestration tasks,

[0006] wherein the inter-factory orchestrator and the intra-factory orchestrators are operable to cooperate to orchestrate the resources and services of the plurality of production factories.

[0007] In a second aspect, an inter-factory orchestrator for a joint orchestration system is provided, wherein the joint orchestration system includes a plurality of intra-factory orchestrators, the plurality of intra-factory orchestrators are assigned to corresponding production factories among a plurality of production factories, wherein each intra-factory orchestrator is operable to execute one or more intra-factory orchestration tasks regarding the corresponding production factory, wherein the inter-factory orchestrator is operable to execute one or more inter-factory orchestration tasks, and wherein the inter-factory orchestrator and the intra-factory orchestrators are operable to cooperate to orchestrate the resources and services of the plurality of production factories.

[0008] In a third aspect, an intra-factory orchestrator for a joint orchestration system is provided, wherein the intra-factory orchestrator is operable to be assigned to a corresponding production factory among a plurality of production factories, wherein the joint orchestration system further includes an inter-factory orchestrator, the inter-factory orchestrator is operable to execute one or more inter-factory orchestration tasks, wherein the plurality of intra-factory orchestrators are operable to execute one or more intra-factory orchestration tasks regarding the corresponding production factories, and wherein the intra-factory orchestrator and the inter-factory orchestrator are operable to cooperate to orchestrate the resources and services of the plurality of production factories.

[0009] In one example, the in-factory orchestrator forms part of the first layer of the hierarchy, while the inter-factory orchestrator forms part of the second layer of the hierarchy, with the second layer being higher than the first layer, such that the combined orchestration system orchestrates factory resources and services using multiple layers of orchestrators. The present disclosure further contemplates that the combined orchestration system may include more than two layers of orchestrators, the system including, for example, another orchestrator forming part of the third layer of the hierarchy, with the third layer being higher than the second layer. For example, factories may be grouped into sites (i.e., one site may include multiple factories), where the combined orchestration system includes a site orchestrator for each site, with the site orchestrator being on a layer above the factory (inter)orchestrators of the factories within the corresponding site, and a corporate orchestrator on a layer above the multiple site orchestrators.

[0010] Orchestration tasks (whether inter-factory or in-factory) involve the automatic configuration, orchestration, and management of resources and services. In-factory orchestration tasks involve the automatic configuration, orchestration, and / or management of resources and services within the above-mentioned production factory. Inter-factory orchestration tasks involve the automatic configuration, orchestration, and / or management of resources and services between two or more of the above-mentioned production factories. Orchestration tasks may include, for example, one or more of the following: scheduling; workload allocation; deployment; resource monitoring; rollout of updates. More specifically, inter-factory orchestration tasks may include one or more of the following: cross-factory workload allocation; application of central updates across factories; application of higher-level instructions; execution of failover or disaster handling; initial deployment; transfer of proven deployment configurations between factories; optimization of global resource usage for workloads; early response to global security issues. In-factory orchestration tasks may include one or more of the following: deployment; restart; removal of components of the factory; service discovery; load balancing; storage orchestration; secret management; network management; automatic update rollout and rollback; pod horizontal autoscaling.

[0011] Each in-factory orchestrator may be operable to perform in-factory scheduling for allocating one or more resources of the corresponding factory to perform one or more resources required by the factory. The inter-factory orchestrator may be operable to perform inter-factory scheduling for allocating one or more services of one factory among multiple factories to perform one or more services required by another factory among the multiple factories. In this way, resource utilization and optimization (''across the border'') can be achieved both within individual factories and between factories.

[0012] Scheduling (whether inter-factory scheduling or intra-factory scheduling) can be performed according to one or more policies (such as a combined policy) that govern scheduling decisions. Such policies may relate to one or more of the following: criticality; quality of service (e.g., latency); priority; failover handling; access restrictions; security; limitations (e.g., geographical or legal). In one particular example, the above policy requires certain services (e.g., control services) to meet more stringent latency requirements than those required by other services (e.g., supervisory services). In other words, the policy can prioritize certain services over others. In such a case, the policy can specify that certain services are hosted locally at the above factory, while other services can be transferred to other factories.

[0013] Inter-factory scheduling decisions can be made at least in part based on factory data collected by the inter-factory orchestrator from one or more of the production factories. The factory data can include status data representing the status of resources, services, and / or equipment of the respective factory based on monitoring performed by that factory. For example, the factory data may relate to one or more of the following: resource utilization; service status; equipment availability and downtime due to faults or maintenance. The factory data can be used to identify problems that cannot be resolved locally at the respective factory. The factory data can be obtained from one or more distributed control systems or services. Additionally or alternatively, the factory data can be obtained from one or more manufacturing execution systems or services. In this way, the systems and methods disclosed herein enable the consideration of MES-level information in scheduling decisions. Examples of MES data that can be used by the combined orchestration system described herein may relate to one or more of the following: production scheduling; work order status; start and end times of production batches; materials used; production quantity; quality data from raw material inspections; equipment data, such as machine status, usage, maintenance records; material data, such as inventory levels, material consumption; energy and utility data, such as energy consumption, utility usage; logistics and supply chain data; environmental data, such as temperature, humidity in the production area; labor data, e.g., upcoming training courses, shift schedules. For example, upcoming operator training can be combined with software updates to try out new features. Production downtime can be used to run diagnostic functions or restart servers for updates.

[0014] The inter-factory orchestrator can be operable to propagate inter-factory scheduling decisions to the intra-factory orchestrators. The inter-factory orchestrator can be operable to propagate inter-factory scheduling decisions according to a propagation configuration (hereinafter referred to as a cluster configuration) that specifies which intra-factory orchestrators form part of the combined orchestration system, and / or how to contact these intra-factory orchestrators, and / or the type of one or more of the intra-factory orchestrators. The inter-factory orchestrator can be operable to generate a combined configuration representing the inter-factory scheduling decisions. The inter-factory orchestrator can be operable to derive local configurations for each of a plurality of production factories from the combined configuration, where each local configuration instructs the intra-factory orchestrator of the corresponding factory to implement at least part of the inter-factory scheduling decisions. More specifically, the local configuration can instruct the intra-factory orchestrator to deploy one or more specified services to one or more specified resources in order to implement at least part of the inter-factory scheduling decisions. At least part of the inter-factory scheduling decisions may relate to resources and / or services associated with the corresponding factory. The local orchestrator or its operator can be operable or capable of rejecting at least part of the corresponding local configuration. Rejection of at least part of the local configuration by the local orchestrator or its operator may trigger the inter-factory orchestrator to modify the inter-factory scheduling decisions.

[0015] The intra-factory orchestrator can be operable to determine whether it has resources available to solve a problem by performing local measures. The intra-factory orchestrator can be operable to prioritize performing local measures over requesting global measures. For example, the problem may be related to excessive resource utilization, and / or resource failure, and / or service failure affecting the intra-factory orchestrator. In response to determining that the resources are available, the intra-factory orchestrator can be operable to allocate these resources to solve the problem. The intra-factory orchestrator can be operable to forward the problem to the inter-factory orchestrator for global measures to be taken in response to determining that the intra-factory orchestrator has no available resources. In response to receiving a problem forwarded by one of the intra-factory orchestrators, the inter-factory orchestrator can be operable to update the combined configuration to solve the problem and propagate the updated local configuration derived from the updated combined configuration to one or more of the intra-factory orchestrators.

[0016] A combined configuration may include at least one resource template that defines the allocation of at least one resource to at least one service according to an inter-plant scheduling decision. A resource template may be defined for each factory. The combined configuration may also include a placement configuration that defines which resource template is to be propagated to which in-plant orchestrator. The at least one resource and the at least one service specified in the resource template may belong to the same factory or different factories. The resource template for at least one factory may correspond to the type of the respective factory. The inter-plant orchestrator may be operable to use different resource templates corresponding to different types of factories. The resource template may specify the resources to be used by the respective in-plant orchestrator with respect to one or more predefined resource types. The inter-plant orchestrator may be operable to access a resource type library storing one or more predefined resource types for selection. The resource template and / or the predefined resource types may relate to resources that are commonly used between factories, i.e., plant-agnostic resources or resource types. The combined configuration may also include at least one override configuration that overrides at least part of one or more of the resource templates. The override configuration may relate to at least one resource or resource type that is not commonly used between factories, i.e., factory-specific resources or resource types. The combined configuration may include one or more policies as described elsewhere herein. The inter-plant orchestrator may be operable to derive a local configuration for a respective factory in the factory from the combined configuration by allocating resources to the services specified in the respective resource template, optionally modified according to at least one of the respective override configuration and / or policies of the factory.

[0017] The inter-plant orchestrator may be operable to roll out a central update that affects multiple factories. Additionally or alternatively, the inter-plant orchestrator may be operable to perform one or more maintenance tasks that affect multiple factories.

[0018] The combined orchestration system may be operable to automatically merge the topology specification of each site into the inter-plant topology specification. Additionally or alternatively, the combined orchestration system may be operable to automatically derive the topology specification of each site from the inter-plant topology specification. In a non-limiting example, merging the topology specification of each site into the inter-plant topology specification may include testing a common service (e.g., for security auditing) in one factory and then, if successful, promoting it to the global cluster layer and requiring each factory to run the security auditing service. In a non-limiting example, to derive the topology specification of each site from the inter-plant topology specification, each factory cluster may have special attributes (e.g., user credentials, resource limitations, spatial characteristics, different hardware versions) that may require factory-specific adaptation. For example, when instantiating a global deployment template in a specific factory cluster, the password used in the specific factory may be inserted into the global deployment template.

[0019] The joint orchestration system can be implemented as a cluster joint system, where each in-plant orchestrator forms part of a corresponding member cluster, and where the inter-plant orchestrator is implemented as a host cluster for coordinating the federation of member clusters.

[0020] In a fourth aspect, there is provided a joint orchestration method, comprising:

[0021] Allocating a plurality of in-plant orchestrators to respective ones of a plurality of industrial factories;

[0022] Performing, by each in-plant orchestrator, one or more in-plant orchestration tasks regarding the respective factory; and

[0023] Performing, by the inter-plant orchestrator, one or more inter-plant orchestration tasks,

[0024] wherein the inter-plant orchestrator and the in-plant orchestrators cooperate to orchestrate resources and services of the plurality of factories.

[0025] In a fifth aspect, there is provided a method performed by an inter-plant orchestrator for a joint orchestration system, wherein the joint orchestration system includes a plurality of in-plant orchestrators allocated to respective ones of a plurality of factories, where each in-plant orchestrator is operable to perform one or more in-plant orchestration tasks regarding the respective factory, the method comprising: performing, by the inter-plant orchestrator, one or more inter-plant orchestration tasks, wherein the inter-plant orchestrator and the in-plant orchestrators cooperate to orchestrate resources and services of the plurality of factories.

[0026] In a sixth aspect, there is provided a method performed by an in-plant orchestrator for a joint orchestration system, wherein the in-plant orchestrator is allocated to a respective one of a plurality of factories, and wherein the joint orchestration system further includes an inter-plant orchestrator operable to perform one or more inter-plant orchestration tasks, the method comprising: performing, by the in-plant orchestrator, one or more in-plant orchestration tasks regarding the respective factory, wherein the in-plant orchestrator and the inter-plant orchestrator cooperate to orchestrate resources and services of the plurality of factories.

[0027] The method of any one of the fourth to sixth aspects may include any optional features or sub-aspects of any one of the first to third aspects (with necessary modifications).

[0028] Any method described herein may include additional steps of orchestrating resources and services during operation of one or more of the plurality of factories, such as performing an industrial process or manufacturing a product.

[0029] In a seventh aspect, a method is provided that includes a joint operator generating a joint configuration as described herein. In the case where the joint orchestration system is a greenfield project, the method includes the joint operator generating the joint configuration from the start. In the case where the joint orchestration system is a brownfield project, the method includes the joint operator generating the joint configuration at least in part based on one or more legacy plant resource templates.

[0030] The method of any of the fourth to sixth aspects may be computer-implemented.

[0031] According to an eighth aspect, a computing system is provided that is configured to perform the method of any of the fourth to sixth aspects.

[0032] According to a ninth aspect, a computer program (product) including instructions is provided that, when executed by a computing system, cause the computing system to be able to or cause the computing system to perform the method of any of the fourth to sixth aspects.

[0033] According to a tenth aspect, a computer-readable (storage) medium including instructions is provided that, when executed by a computing system, cause the computing system to be able to or cause the computing system to perform the method of any of the fourth to sixth aspects. The computer-readable medium may be transient or non-transient, volatile or non-volatile.

[0034] The systems and methods disclosed herein take into account individual as well as shared characteristics and interests and provide for the orchestration of multiple control systems with cross-border resource management. The systems and methods disclosed herein enable the updating and maintenance of components available for multiple plants, resource utilization between plants, dependencies between plants, and high-level instructions affecting multiple plants. This is in contrast to prior orchestrators that only manage resources and services within a single plant, causing the orchestrators of multiple plants run by a single operator to work independently of each other, and more advanced orchestration tasks (e.g., rollout updates across plants) having to be applied through manual interaction.

[0035] In addition to enabling the shared use of resources, the systems and methods described herein provide a novel way of considering higher-level attributes (especially MES data) in orchestration decisions.

[0036] The systems and methods disclosed herein also provide a reduced workload due to centralized control system management, an integrated representation of the states of multiple plants, and management of optimized control systems and processes due to cross-border management.

[0037] By combining the individual system specifications of the factory IT / OT infrastructure, which are currently used by the orchestrator to deploy and manage a single factory, into an inter-factory system specification, the systems and methods disclosed herein enable a policy to be defined that enables the orchestrator to reach a target state with different policies. This can include placing software components on available infrastructure (cluster scheduling) and / or changing the configuration of the components. The inter-factory system specification can include a declarative description of the target IT and OT topologies.

[0038] By synchronizing the state representation of its control system with the inter-factory orchestrator during runtime, the in-factory orchestrator enables the inter-factory orchestrator to react to changes in the factory to derive cross-border optimizations. A factory that is not synchronized with the inter-factory orchestrator retains its ability to stably execute the control system managed by the in-factory orchestrator. Once synchronization is re-established, any emerging tasks can be executed.

[0039] The term "resource" as used herein is used to denote any resource that can be allocated to perform services within or across factories. This term can refer to computing resources, including, for example, CPUs, memory, storage devices, or redundancy. A resource can be referred to as a part or component, such as a virtual part or component, belonging to the IT / OT infrastructure of, for example, one or more factories. That is, a resource can include hardware and / or software that detects or causes changes by monitoring and / or controlling factory equipment, processes, and events. A resource can be included in or included within one or more clusters that include one or more (computing) nodes. In a Kubernetes implementation, a resource can include one or more of the following: Pods; Services; Volumes; Namespaces; ConfigMaps; Secrets; Deployments; StatefulSets; DaemonSets; ReplicaSets; Jobs; CronJobs; PersistentVolumeClaims; PersistentVolumes; ServiceAccounts; Roles; RoleBindings; ClusterRoles; ClusterRoleBindings; Ingress; NetworkPolicies; HorizontalPodAutoscalers; CustomResourceDefinitions; ResourceQuotas; LimitRanges; Endpoints; PodTemplates; StorageClasses; Node; Events; ComponentStatuses; APIResourceLists. However, it should be understood that the present disclosure is not limited to Kubernetes or any particular orchestration tool using the above terms.

[0040] The "service" as used herein refers to a software service or application to which resources can be allocated to perform services. Although a service is described herein as being deployed by an orchestrator, it should be understood that the orchestrator itself can be considered a service. The services described herein can include stateful and / or containerized services. One or more services can execute control logic to automate processes.

[0041] As used herein, the service-related term "update" refers to the deployment of a new version of an existing service or application. An update may include performing a rolling update. An update may include shutting down one or more old service instances and starting one or more new service instances. The federated orchestration system may utilize one or more load balancers to ensure that user requests are directed to one or more new service instances. An update may include maintaining the state of the service on an external storage capacity accessible, for example, by the new service instances.

[0042] "Deploy" means that a service or a part thereof is installed to run on an execution engine of a resource (such as a node), or the resource is instructed to execute the service or a part thereof.

[0043] "Manufacturing Execution System (MES)" refers to any system in the industry used to track and record the transformation of raw materials or components into finished or assembled products. MES data may include any data that helps production decision-makers optimize or increase production. MES data may include data from one or more real-time monitoring systems that enable the control of factory equipment. Typical MES applications may include any one or more of the following: production planning and scheduling; work order management; inventory management; quality management; performance analysis and reporting; resource allocation and utilization; process monitoring and control; maintenance management; document management; material requirements planning (MRP); traceability and pedigree; labor management; energy management; product lifecycle management (PLM) integration; supply chain management; data collection and acquisition; shop floor production control; asset management; regulatory compliance management; analytics and business intelligence; machine integration and Internet of Things; collaboration and communication tools; environmental, health and safety management; integration with enterprise resource planning (ERP); change management; customer relationship management (CRM) integration.

[0044] "(Industrial) plant" herein refers to any system for process automation, factory automation, or warehouse automation. A plant may include a production plant and / or a processing plant for performing industrial processes. Industrial processes may be continuous, batch, or discrete processes. A plant may include one or more pipelines for transforming one or more isolates or raw materials into products. Additionally or alternatively, a plant may include one or more assembly lines for assembling one or more components into products. A plant may be modular or monolithic (i.e., non-modular).

[0045] The term "inter-plant" as used herein is used to denote aspects related to information or measures involving multiple plants. Depending on the context, this term may be used interchangeably with "higher-level" or "global".

[0046] The term "intra-plant" is used herein to denote aspects related to information or measures involving only a single plant. Depending on the context, this term may be used interchangeably with "lower-level" or "local".

[0047] As used herein, the term "obtain" may include, for example, receiving from another system, device, or process; receiving through interaction with a user; loading or retrieving from a storage device or memory; measuring or collecting using a sensor or other data acquisition device.

[0048] As used herein, the term "determine" includes a variety of measures and may include, for example, calculating, computing, processing, deriving, investigating, looking up (e.g., looking up in a table, database, or another data structure), ascertaining, etc. Additionally, "determine" may include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), etc. Additionally, "determine" may include resolving, selecting, choosing, establishing, etc.

[0049] The indefinite article "a" does not exclude a plural. Additionally, unless otherwise specified in the context or explicitly indicated as a singular form, the terms "a" and "an" as used herein shall generally be construed to mean "one or more".

[0050] Unless otherwise specified or the context clearly indicates, the phrases "one or more of A, B, and C", "at least one of A, B, and C", and "A, B, and / or C" as used in this agreement are intended to represent all possible permutations of one or more of the listed items. That is, "A and / or B" means (A), (B), or (A and B), and "A, B, and / or C" means (A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C).

[0051] The term "comprising" does not exclude other elements or steps. Additionally, the terms "comprising", "including", "having", etc. may be used interchangeably herein.

[0052] The present invention may include one or more aspects, examples, or features, either alone or in combination, whether specifically disclosed in such combination or separately. Any optional feature or sub - aspect of any of the above - mentioned aspects applies to any other aspect.

[0053] The above - mentioned aspects will become apparent and be elucidated with reference to the detailed description provided below. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] A detailed description will now be given with reference to the drawings, by way of example only, in which:

[0055] Figure 1 illustrates a joint orchestration system;

[0056] Figure 2 is illustrated in more detail Figure 1 of the joint orchestration system;

[0057] Figure 3 is a flowchart of a method for initial deployment in a Figure 2 joint orchestration system;

[0058] Figure 4 is a flowchart of an operation method of a Figure 2 joint orchestration system;

[0059] Figure 5 describes a Figure 4 method in an exemplary use case involving disaster handling; and

[0060] Figure 6 illustrates a computing system that can be used according to the systems and methods disclosed herein. DETAILED DESCRIPTION

[0061] Figure 1 illustrates a joint orchestration system 10 that is operable to orchestrate the IT / OT (Information Technology / Operational Technology) infrastructure and services of multiple production factories 50 (labeled as factories 50-A, 50-B, 50-C in Figure 1 ) using a multi-layer orchestrator. In addition to multiple low-level, in-plant orchestrators 100 (labeled as 100-A, 100-B, 100-C in Figure 1 ), the joint orchestration system 10 in this non-limiting example includes a higher-level inter-plant orchestrator 150.

[0062] Each in-plant orchestrator 100 performs in-plant orchestration tasks regarding the corresponding factory 50, while the inter-plant orchestrator 150 performs inter-plant tasks, which include, for example, centrally updating across multiple factories 50, distributing workloads across multiple factories 50, and applying higher-level instructions.

[0063] Figure 2 illustrates in more detail the joint orchestration system 10, which is implemented as a clustered joint system in this non-limiting example, in which each in-plant orchestrator 100 acts as a local orchestrator 100 that forms part of a corresponding member cluster 102 (labeled as member clusters 102-A, 102-B,..., 102-n in Figure 2 ), while the inter-plant orchestrator 150 acts as a host cluster 150 for coordinating the union of the member clusters 102.

[0064] The host cluster 150 includes a cluster configurator 152, a resource type configurator 154, a cluster propagator 156, a joint scheduler 158, and a policy configurator 160.

[0065] The cluster configurator 152 can be used by the federated operator to create a cluster configuration 162 that includes all the information required for the cluster propagator 156 to identify and communicate with the federated member clusters 102. More specifically, the cluster configurator 152 can be used to specify which member clusters 102 exist, how to access them (e.g., IP address, VPN, or SSH credentials), and the corresponding types of the local orchestrator 100 (e.g., TOSCA or Kubernetes) to determine the API to be used to communicate with the member clusters 102. The federated orchestration system 10 supports both homogeneous and heterogeneous federated clusters (i.e., different types of local orchestrators, such as Kubernetes and TOSCA).

[0066] The resource type configurator 154 is operable to access a resource type library 164 that stores resource types 166 for specifying a federation configuration 168 that indicates the resources to be federated. The federation configuration 168 in this non-limiting embodiment includes at least one resource template 170, at least one federation policy 172, a placement configuration 174, and at least one override configuration 176.

[0067] The resource template 170 specifies the resources to be used by a particular member group 102. Resource templates can be specified for different types of factories, such as oil and gas refineries or pharmaceutical enterprises.

[0068] The federation policy 172 enables policies to be specified regarding the placement of resources in different member clusters 102. Exemplary policies can relate to one or more of the following: failover handling; access restrictions; security checks; restrictions based on geography or legislation.

[0069] The placement configuration 174 specifies which resource template 170 is to be propagated by the cluster propagator 156 to which member cluster 102.

[0070] The override configuration 176 specifies optional per-cluster, factory-specific changes to the resource template 170 that will be applied when the resource template 170 is propagated to the member clusters 102 (through the local configuration described below).

[0071] The federated scheduler 158 additionally uses MES (Manufacturing Execution System) data 178 to represent the data 180 obtained from the member clusters 102 to make scheduling decisions. The federated scheduler 158 calculates the placement configuration 174 and the override configuration 176 based on the federation policy 172, the current system state 180, and the MES data 178 of the member clusters 102. If beneficial or even necessary, the federated scheduler 158 can decide to move resources between factories 50 (e.g., due to resource bottlenecks, unexpected high demand, or component failures in any of the factories 50). This enables factories 50 to use resources from other factories 50.

[0072] The policy configurator 160 receives MES data 178 from the member clusters 102 and enables the policy designers responsible for designing the joint policy 172 involving multiple factories 50 to make informed decisions and / or propose optimizations to the existing joint policy 172. The policy designers can also consider external sources, such as the intentions communicated by a third party (thus enabling cooperating companies to align their production).

[0073] Whenever the placement configuration 174 or the override configuration 176 changes, the cluster propagator 156 automatically or semi-automatically propagates the updated local configuration 182 to the local orchestrator 100 with the assistance of the joint operator. This can be accompanied by an increase in the version number of the services running in one or more member clusters 102 (i.e., an upgrade). The cluster propagator 156 receives the cluster configuration 162 (containing information about the member clusters 102) and the joint configuration 168 (containing the resource template 170, the joint policy 172, the placement configuration 174, and the override configuration 176) as inputs. The cluster propagator 156 applies the joint policy 172, the placement configuration 174, and the override configuration 176 to the resource template 170 included in the joint configuration 168 to derive the local configuration 182 for each factory 50 and uses the information contained in the cluster configuration 162 to propagate the local configuration 182 to the member clusters 102. As further discussed below, these operations can also be performed during the initial deployment phase of the joint orchestration.

[0074] Each member cluster 102 serves a corresponding production factory 50 and thus has resources at its disposal for operating that factory 50. In Figure 2 the non-limiting example shown, each member cluster 102 provides a MES service 104 and a DCS (Distributed Control System) service 106 (correspondingly illustrated as MES services 104-A, 104-B, …, 104-n and DCS services 106-A, 106-B, …, 106-n in Figure 2 ), as well as its local orchestration service. Any of these services can be specified in the resource template 170 for the factory 50, possibly overridden by the corresponding per-cluster override configuration 176. Each member cluster 102 deploys the services according to its local configuration 182 propagated by the cluster propagator 156. Details of the factory, such as the type of DCS, can be specified in the override configuration 176.

[0075] Each local orchestrator 100 manages low-level, in-plant tasks such as the deployment, restart, and removal of components of the factory 50. The local orchestrator 100 monitors the status of its production factory 50, such as resource utilization and issues that cannot be resolved locally, and communicates them in the form of status data 180 received via a status update to the federated scheduler 158. As described above, the federated scheduler 158 makes orchestration decisions at the federated layer, such as updating the local configuration 182 to a new version (e.g., running a particular component with 3 rather than 2 replicas, or changing the user authentication configuration of all OPC UA servers to require a longer password), applying a central update across all factories 50, or instructing one factory 50 to take over parts of another factory 50 in the case of a critical system failure that cannot be remedied locally. For example, the regulatory user interface backend can be hosted by another factory 50, and the user interface can be transmitted to the user on-site, or the Historian database can take over collecting process data for long-term storage when the historical database in the current factory 50 is shut down. In any case, the local cluster operator can instruct the local orchestrator 100 to reject the local configuration 182. In the case where the federated scheduler 158 is not working, the local orchestrator 100 is able to operate autonomously.

[0076] The hardware resources of the member clusters 102 (such as distributed control nodes) can be automatically discovered (e.g., via a service such as Redfish) or manually configured using the cluster configurator 152.

[0077] Figure 3 is a flowchart showing a method 300 for initial deployment in the federated orchestration system 10. The method implements different strategies for adopting federated orchestration depending on whether the system 10 is a greenfield project or a brownfield project, determined in step 302.

[0078] In the case where the factory operator is already running multiple factories 50 without federated orchestration (step 302 - Greenfield? - No), the method proceeds to step 304, in which a resource template 170 is generated based on the legacy factory resource template 306 for the existing factories 50. More specifically, the common part of the legacy factory resource template 306 is identified, associated with the resource type 166, and designated as a federated resource in the resource template 170 and the federated policy 172 (i.e., as part of the inter-factory system specification). The non-common part of the legacy factory resource template 306, i.e., the factory-specific part, is extracted into the overlay configuration 176.

[0079] In the case where the factory operator has not yet operated any factory (step 302 - greenfield? - yes), the method proceeds to step 308, in which the federation is first configured by a federation operator that creates a federation resource template 170 as well as a federation policy 172, a placement configuration 174, and a factory - specific override configuration 176.

[0080] Subsequently, the placement configuration 174 and the override configuration 176 are applied (step 310) to the resource template 170 to derive a per - factory local configuration 182, and the local configuration 182 is then propagated (step 312) by the cluster propagator 156 to the individual member clusters for deployment by the local orchestrator 100, as described above.

[0081] Figure 4 FIG. 7 is a flowchart showing the operational method 400 of the federated orchestration system 10 at runtime. Figure 4 The interactions shown occur between the host cluster 150 and the individual member clusters 102 - n, but it should be understood that these operations can be repeated for other member clusters 102.

[0082] In step 402, the cluster propagator 156 propagates the local configuration 182 to the member cluster 102 - n. If accepted by the local cluster operator (step 404 - accepted? - yes), then in step 406 the local configuration 182 is deployed by the responsible local orchestrator 100 - n. If rejected by the local cluster operator (step 404 - accepted? - no), which can also occur during initial deployment, the host cluster 150 can further optimize the federation configuration based on the reason for rejection in step 416, as described below.

[0083] In step 408, components of the member cluster 102 - n (e.g., the DCS service 106 - n and the MES service 104 - n) continuously monitor their current state, including system bottlenecks or failures, and communicate them to the host cluster 150 in step 410. If optimization is not possible or not required (step 412 - optimize? - no), then no further action is taken. If optimization is possible or required (step 412 - optimize? - yes), for example, in the case where a service pushes its hosting compute node to full CPU or memory utilization, or a failure must be recovered (e.g., due to a partial power outage, service crash, or compute node failure), the local orchestrator 100 - n determines (in step 414) whether it has the means to solve the problem by performing cluster - local measures. If the local orchestrator 100 - n has the necessary resources at its disposal, it updates the local configuration 182 in step 416 to allocate these resources to solve the problem, which may involve, for example, executing a second instance of the same service on another compute node to balance the load, or executing the service running on the failing compute node on another compute node that still has the necessary remaining capacity.

[0084] If the local orchestrator 100-n does not have the necessary means (step 414 - global measure? - yes), it forwards the problem to the host cluster 150 to take global measures. In step 418, the host cluster 150 optimizes the joint configuration 168 to implement the global measure, for example, by temporarily deploying the services required by factory 50-A to the resources of factory 50-B. In step 420, the host cluster 150 updates the joint configuration 168 (where the deployment configuration and the override configuration now reflect the deployment of the services of factory 50-A to the resources of factory 50-B). The method finally returns to step 402, in which the host cluster 150 propagates the local configuration 182 to the corresponding member clusters 102 again.

[0085] Figure 5 The operation method 400 in an exemplary usage scenario involving disaster handling via runtime failover is described. Figure 4 In steps 402 - 406, the initial propagation of the local configuration 182 from the host cluster 150 to the member cluster 102-n and its subsequent deployment by the local orchestrator 100-n are successful. At some point during the status monitoring 408, a local computing node in the member cluster 102-n fails. This node hosts the virtual controller service and the historical service. To keep the factory running, the local orchestrator 100-n decides in step 412 to optimize its resource utilization: there is another computing node still running in the local member cluster 102-n, which has some remaining capacity (i.e., CPU and memory) sufficient to host either a replacement virtual controller or a replacement historical instance, but not both. Since there are existing policies (such as the joint policy 172) that require the control service to meet certain latency requirements, and these latency requirements do not apply to the monitoring service (such as the historical record), the local orchestrator 100-n decides (in step 414) to locally host the replacement virtual controller on its remaining computing node (in step 416). Then, it communicates its current state and the lost historical service to the host cluster 150. In step 418, the joint scheduler 158 finds suitable computing resources in another factory 50 to take over the hosting of the historical service of the troubled factory and updates the joint configuration 168 (specifically, the resource template 170, the placement configuration 174, and the override configuration 176). The cluster propagator 156 propagates the updated local configuration 182 to the affected member cluster 102-n, which accepts and deploys the updated local configuration 182. The interaction between local and global orchestration (i.e., joint orchestration) keeps all factories 50 running at all times.

[0086] Accordingly, described herein is a more advanced orchestrator that allows for the joint management of multiple factories, each served by a local member cluster that includes an IT / OT infrastructure. The local orchestrator communicates its local state data and MES data to the more advanced host cluster orchestrator and relies on the more advanced orchestrator for central updates and maintenance, as well as load balancing and disaster handling. However, even if communication with the more advanced orchestrator is interrupted, the local orchestrator can continue to act autonomously and make local orchestration decisions. The more advanced orchestrator supports the plant operator by automatically maintaining a global resource view across multiple factories based on the monitored system state and MES data and automatically making informed decisions on global measures related to multiple clusters in the federation, enabling the scheduling and rollout of cross-border resource usage or central updates.

[0087] The proposed system and method envision several use cases: - Updates and maintenance: The more advanced orchestrator performs global maintenance tasks such as software updates and upgrades on common resources across all factories. It takes into account the state of the DCS service (such as system load) and the state of the MES service (such as machine availability and downtime due to failures or maintenance) to make informed decisions on rolling out centralized maintenance tasks.

[0088] Load balancing and disaster handling: The more advanced orchestrator performs global resource scheduling by monitoring system load and moving virtual components across factories if necessary to (i) overcome resource bottlenecks, (ii) scale out in the event of unexpectedly high demand (while maintaining the advantages of in-house hosting), or (iii) overcome hosted resource failures in the event of a disaster.

[0089] Figure 6 An exemplary computing system 800 that can be used in accordance with the systems and methods disclosed herein is illustrated. The computing system 800 can form part of or include any desktop computer, laptop computer, server, or cloud-based computing system. The computing system 800 includes at least one processor 802 that executes instructions stored in a memory 804. The instructions can be, for example, instructions for implementing functions described as being performed by one or more components described herein, or instructions for implementing one or more methods described herein. The processor 802 can access the memory 804 via a system bus 806. In addition to storing executable instructions, the memory 804 can also store dialogue inputs, scores assigned to the dialogue inputs, and the like.

[0090] The computing system 800 additionally includes a data store 808 accessible by the processor 802 via the system bus 806. The data store 808 can include executable instructions, log data, and the like. The computing system 800 also includes an input interface 810 that allows external devices to communicate with the computing system 800. For example, the input interface 810 can be used to receive instructions from external computer devices, users, and the like. The computing system 800 also includes an output interface 812 that interfaces the computing system 800 with one or more external devices. For example, the computing system 800 can display text, images, and the like via the output interface 812.

[0091] External devices that are expected to communicate with the computing system 800 via the input interface 810 and the output interface 812 can be included in an environment that provides substantially any type of user interface with which a user can interact. Examples of user interface types include graphical user interfaces, natural user interfaces, and the like. For example, a graphical user interface can accept input from a user using (multiple) input devices such as a keyboard, a mouse, a remote control, etc., and provide output on an output device such as a display. Additionally, a natural user interface can enable a user to interact with the computing system 800 in a manner that is not restricted by the limitations imposed by input devices such as a keyboard, a mouse, a remote control, etc. Instead, a natural user interface can rely on speech recognition, touch and stylus recognition, on-screen and near-screen gesture recognition, air gestures, head and eye tracking, voice and speech, vision, touch, gesture, machine intelligence, and the like.

[0092] Additionally, although shown as a single system, it should be understood that the computing system 800 can be a distributed system. Thus, for example, several devices can communicate via a network connection and can jointly perform tasks described as being performed by the computing system 800.

[0093] The various functions described herein can be implemented by hardware, software, or any combination thereof. If implemented in software, the functions can be stored or transmitted on a computer-readable medium as one or more instructions or code. The computer-readable medium includes computer-readable storage media. The computer-readable storage media can be any available storage media accessible by a computer. By way of example and not limitation, such computer-readable storage media can include flash memory (FLASH) storage media, random access memory (RAM), read-only memory (ROM), programmable read-only memory (EEPROM), optical disk (CD-ROM) or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and that can be accessed by a computer. Disks and discs used herein include compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and Blu-ray discs (BDs), where disks typically reproduce data magnetically and discs typically reproduce data optically with a laser. In addition, propagated signals can be included within the scope of computer-readable storage media. The computer-readable medium also includes communication media, including any medium that facilitates the transfer of a computer program from one place to another. For example, a connection can be a communication medium. For example, if software is transferred from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technology (such as infrared, radio, and microwave), then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technology (such as infrared, radio, and microwave) is included in the definition of communication media. Combinations of the above should also be included within the scope of computer-readable media.

[0094] Alternatively, or in addition, the functions described herein can be performed, at least in part, by one or more hardware logic components. By way of example, and not limitation, exemplary types of hardware logic components that can be used include field programmable gate arrays (FPGA), application specific integrated circuits (ASIC), application specific standard products (ASSP), system on a chip (SOC), complex programmable logic devices (CPLD), and the like.

[0095] The applicant hereby separately discloses each individual feature described herein and any combination of two or more such features, provided that such features or combinations can be implemented based on the general knowledge of those skilled in the art from the whole of this specification, regardless of whether such features or combinations of features solve any of the problems disclosed herein, and are not limited to the scope of the claims. The applicant points out that aspects of the present invention can be composed of any such individual feature or combination of features.

[0096] It must be noted that embodiments of the present invention are described with reference to different categories. Specifically, some examples are described with reference to methods, while other examples are described with reference to devices. However, those skilled in the art will understand from the description that, unless otherwise stated, any combination of features belonging to one category, and any combination between features associated with different categories are considered to be disclosed in this application. However, all functions can be combined to provide synergistic effects, rather than just a simple superposition of functions.

[0097] Although the present invention has been illustrated and described in detail in the drawings and the foregoing description, such illustration and description should be considered illustrative rather than restrictive. The present invention is not limited to the disclosed embodiments. Other variations of the disclosed embodiments may be understood and effected by those skilled in the art by studying the drawings, the disclosure, and the appended claims.

[0098] The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

[0099] Any reference signs in the claims should not be construed as limiting the scope.

Claims

1. A joint orchestration system, comprising: a plurality of in-plant orchestrators assigned to respective ones of the plurality of production plants, wherein each in-plant orchestrator is operable to perform one or more in-plant orchestration tasks with respect to the respective production plant; as well as an inter-plant scheduler operable to perform one or more inter-plant schedule tasks, The inter-plant orchestrator and the intra-plant orchestrator are operable to cooperate to orchestrate resources and services of the plurality of production plants.

2. The joint orchestration system of claim 1, wherein the intra-plant orchestrator forms part of a first layer of a hierarchy, and wherein the inter-plant orchestrator forms part of a second layer of the hierarchy, the second layer of the hierarchy being higher than the first layer.

3. A joint orchestration system according to claim 1 or 2, wherein the inter-plant orchestration tasks include one or more of the following: distributing workload across plants; applying central updates across plants; applying higher-level instructions; performing failover or disaster handling; and initial deployment.

4. A joint orchestration system according to any one of the preceding claims, wherein each intra-plant orchestrator is operable to perform intra-plant scheduling for allocating one or more resources of the corresponding factory to perform one or more services required by the factory, and wherein the inter-plant orchestrator is operable to perform inter-plant scheduling for allocating one or more resources of one of the multiple factories to perform one or more services required by another of the multiple factories.

5. The joint orchestration system of claim 4, wherein the inter-plant orchestrator is operable to perform the inter-plant scheduling according to one or more policies governing scheduling decisions.

6. A joint orchestration system according to claim 4 or 5, wherein the inter-plant orchestrator is operable to perform the inter-plant scheduling based at least in part on plant data collected by the inter-plant orchestrator from one or more of the production plants. 7 . The joint orchestration system according to claim 6 , wherein the plant data comprises status data, the status data representing the status of resources, services and / or equipment of the corresponding plant.

8. The joint orchestration system of claim 6 or 7, wherein the factory data comprises data obtained from one or more manufacturing execution systems or services.

9. A joint orchestration system according to any one of the preceding claims, wherein the inter-plant orchestrator is operable to generate a joint configuration representing an inter-plant scheduling decision, and is operable to derive a local configuration for each of the multiple production plants from the joint configuration, wherein each local configuration instructs the in-plant orchestrator of the corresponding plant to implement at least part of the inter-plant scheduling decision.

10. The joint orchestration system of claim 9, wherein the joint configuration comprises at least one resource template, wherein the at least one resource template defines allocation of at least one resource to at least one service according to the inter-plant scheduling decision, and wherein the joint configuration further comprises at least one overlay configuration, wherein the at least one overlay configuration overlays at least a portion of the at least one resource template.

11. A joint orchestration system according to any of the preceding claims, wherein at least one of the in-plant orchestrators is operable to determine whether the at least one in-plant orchestrator has resources available to resolve the problem by performing local measures, and is operable to allocate these resources to resolve the problem in response to determining that the resources are available.

12. The federated orchestration system of claim 11, wherein the in-plant orchestrator is operable to forward the problem to the inter-plant orchestrator for global action to be taken in response to determining that the in-plant orchestrator does not have the available resources, and wherein the inter-plant orchestrator is operable to update a federated configuration to resolve the problem, and is operable to propagate an updated local configuration derived from the updated federated configuration to one or more of the in-plant orchestrators.

13. A federated orchestrator according to any preceding claim, wherein the inter-plant orchestrator is operable to push out a central update affecting a plurality of plants and / or is operable to perform one or more maintenance tasks affecting a plurality of plants.

14. A computer-implemented joint orchestration method, comprising: assigning a plurality of in-plant orchestrators to respective industrial plants among a plurality of industrial plants; executing, by each in-plant orchestrator, one or more in-plant orchestration tasks with respect to the corresponding plant; and Execute one or more inter-plant orchestration tasks through the inter-plant orchestrator, The inter-plant orchestrator and the intra-plant orchestrator cooperate to orchestrate resources and services of the plurality of plants.

15. A computer readable medium comprising instructions which, when executed by a computing system, cause the computing system to perform the method according to claim 14.

Images