Using container orchestration systems to modernize operational technology devices

Abstract

Disclosed is updating an operational technology device using a container orchestration system. A method may include receiving a first pod from a second computing node via a first computing node. The method may also include retrieving a first image file that may include a first set of containers from a registry based on the first pod. The first set of containers may cause a control system to suspend operation. The method may then involve generating a first package based on the first set of containers and storing the first package in a file system, receiving a second pod from a second computing node, and retrieving a second image file with a second set of containers from a registry. The second pod may include a second set of containers, and the second set of containers may cause the control system to update software components. The method may also involve generating a second package based on the second set of containers and storing the second package in a file system.

Description

Technical Field

[0001] The present disclosure generally relates to systems and methods for implementing a container orchestration system in an operational technology (OT) or industrial platform. More specifically, embodiments of the present disclosure relate to systems and methods for utilizing a container orchestration system to coordinate the operation of OT devices. Background Art

[0002] In particular, in an industrial automation environment, automated control and monitoring systems (e.g., industrial control systems) are used to manage and operate industrial automation systems. Such applications may include controlling various components such as valves, electric motors, etc., and collecting data via sensors. A typical industrial control system may include one or more components, such as programming terminals, automation controllers, input / output (I / O) modules, communication networks, human-machine interface (HMI) terminals, etc.

[0003] Typically, industrial control systems operating in an OT environment are used to control industrial equipment accessible via an OT network. Although industrial control systems can be used to manage the operation of equipment within an OT network, improved systems and methods for operating equipment within an OT network are desired. For example, certain technologies available in an information technology (IT) environment can provide certain industrial control system users with additional management tools that they use to operate their IT assets.

[0004] This section is intended to introduce the reader to various aspects of the field that may be related to various aspects of the present technology described and / or claimed below. This discussion is considered to be helpful in providing the reader with background information to facilitate a better understanding of various aspects of the present disclosure. Therefore, it should be understood that these statements are to be understood from this perspective and are not intended to be an admission of the prior art. Summary of the invention

[0005] The following is an overview of the specific embodiments disclosed herein. It should be understood that these aspects are presented only to provide the reader with a brief overview of these specific embodiments, and these aspects are not intended to limit the scope of the present disclosure. In fact, the present disclosure may include multiple aspects that may not be described below.

[0006] In one embodiment, a system may include multiple control systems for controlling multiple operations of multiple operational technology (OT) devices. The system may also include a first computing node in a computing node cluster as part of a container orchestration system. The first computing node may receive update data for a first control system in multiple control systems, a first OT device in multiple OT devices, or both the first control system and the first OT device. The first control system, the first OT device, or both the first control system and the first OT device are configured to perform one or more operations. The update data may update one or more software components executed by the first control system, the first OT device, or both the first control system and the first OT device. The first computing node may retrieve multiple machine state data sets from multiple control systems via a portion of the computing node cluster, so that each of the multiple machine state data sets corresponds to an operational state associated with each control system in multiple control systems, each OT device in multiple OT devices, or each control system in multiple control systems and each OT device in multiple OT devices. The first computing node may also identify a second control system as a suitable host to perform one or more operations in place of the first control system based on the multiple machine state data sets and store a first container set (pod) in a first file system accessible to the second computing node. The first pod may cause the second computing node to suspend one or more operations. The first computing node can then store the second pod in a second file system accessible to a third computing node associated with the second control system, so that the second pod can cause the third computing node to perform one or more operations via the second control system, a second OT device associated with the second control system, or both the second control system and the second OT device.

[0007] In another embodiment, a method may include: receiving a first pod from a second computing node in a computing node cluster in a container orchestration system via a first computing node in the computing node cluster. The method may also include retrieving a first image file that may include a first set of containers from a registry based on the first pod, so that the first pod may include an indication of the location of the first image file in the registry. The first set of containers may cause a control system in a plurality of control systems to suspend one or more operations. Then, the method may involve generating a first package based on the first set of containers and storing the first package in a file system shared with the control system, receiving a second pod from a second computing node, and retrieving a second image file that may include a second set of containers from a registry based on the second pod. The second pod may include a second indication of a second location of the second image file in the registry, so that the second set of containers may enable the control system to update one or more software components. The method may also involve generating a second package based on the second set of containers and storing the second package in the file system.

[0008] In another embodiment, a non-transitory computer-readable medium may include computer-executable instructions that, when executed, cause a processor to receive update data for a first control system in a plurality of control systems, a first OT device in a plurality of OT devices, or both the first control system and the first OT device. The first control system, the first OT device, or both the first control system and the first OT device are configured to perform one or more operations so that the update data can update one or more software components executed by the first control system, the first OT device, or both the first control system and the first OT device. The processor may then retrieve a plurality of machine state data sets from the plurality of control systems via a portion of a computing node cluster as part of a container orchestration system, so that each of the plurality of machine state data sets corresponds to an operational state associated with each of the plurality of control systems, each of the plurality of OT devices, or each of the plurality of control systems and each of the plurality of OT devices. The processor may then identify a second control system in the plurality of control systems as a suitable host to perform one or more operations in place of the first control system based on the plurality of machine state data sets. The processor may then generate a first pod and store the first pod in a first file system accessible to a first computing node in the node cluster, such that the first pod may cause the first computing node to suspend one or more operations. The processor may also generate a second pod and store the second pod in a second file system accessible to a second computing node in the node cluster, such that the second computing node is associated with a second control system. The second pod may cause the second computing node to perform one or more operations via a second control system, a second OT device associated with the second control system, or both the second control system and the second OT device. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] These and other features, aspects and advantages of the present embodiments will become better understood when the following detailed description is read with reference to the accompanying drawings, in which like reference numerals represent like parts throughout the drawings, wherein:

[0010] Figure 1 is a perspective view of an example industrial automation system according to an embodiment;

[0011] Figure 2 is a block diagram of an example industrial control system according to an embodiment;

[0012] Figure 3 is a block diagram of an example operational technology (OT) network coordinating with a container orchestration system, according to an embodiment;

[0013] Figure 4 According to the embodiment, the pod is deployed to Figure 3 Data flow diagram of industrial control systems in OT networks;

[0014] Figure 5 is a flow chart of a method for causing an industrial control system to passively implement commands received via a control orchestration system according to an embodiment;

[0015] Figure 6 is a flow chart of a method for causing an industrial control system to proactively implement commands received via a control orchestration system, according to an embodiment;

[0016] Figure 7 is a flow chart of a method for coordinating analytical operations in an industrial control system using a container orchestration system according to an embodiment; and

[0017] Figure 8 is a flow chart of a method for coordinating rolling updates in an industrial control system using a container orchestration system, according to an embodiment. DETAILED DESCRIPTION

[0018] One or more specific embodiments of the present disclosure will be described below. In order to provide a concise description of these embodiments, all features of the actual implementation may not be described in the specification. It should be understood that in the development process of any such actual implementation, as in any engineering project or design project, a large number of implementation-specific decisions must be made to achieve the developer's specific goals, such as meeting system-related constraints and business-related constraints, which may vary with the implementation. In addition, it should be understood that such development efforts may be complex and time-consuming, but for those of ordinary skill who benefit from the present disclosure, such development efforts will still be routine tasks of design, manufacture and production.

[0019] When introducing elements of various embodiments of the present disclosure, the articles "a," "an," "the," and "said" are intended to mean that there are one or more elements. The terms "comprising," "including," and "having" are intended to be inclusive and mean that there may be additional elements in addition to the listed elements.

[0020] The present disclosure generally relates to using information technology (IT) systems to coordinate the operation of devices as part of an operational technology (OT) system. As mentioned above, industrial control systems can be used to control and manage the operation of devices as part of an OT system. However, operators of these industrial automation systems may benefit from using similar processing provided by information technology systems to manage assets such as programmable logic controllers (PLCs) as part of an OT network. As an example, a container orchestration system can be used in an IT system to manage IT assets. That is, some IT systems can utilize software containers (e.g., operating system-level virtualization) and container orchestration systems (e.g., Docker, Kubernetes) to coordinate the construction and deployment of various containers across multiple computing resources. In fact, a container can include a standard unit of software that packages code and its dependencies so that a container node can execute an application stored in a container, regardless of the computing environment or infrastructure. As a result, multiple containers can run on the same machine and share an operating system kernel with other containers, so that each container runs as an independent process in the corresponding machine. In this way, container orchestration systems operating in an IT environment build application services that operate across multiple computing resources, making it possible to automatically deploy, scale, and manage certain applications (e.g., packaged as software containers) in the same machine or across multiple machines in different computing environments.

[0021] With this in mind, using a container orchestration system to manage the operation of OT assets can achieve many advantages, including large-scale application deployment, providing updates from a managed registry, using spare and backup container copies for different OT assets to provide high availability, etc. However, container orchestration systems that rely on certain operating systems that are different from the operating system that controls the OT assets may not be able to access OT assets. Alternatively, OT assets can be individually programmed and managed through corresponding design tools operating in the OT space. For many industrial applications, particularly in the process automation space, distributed industrial control systems can provide multiple remote control nodes that can access or interface with these OT assets via an OT network. However, the container orchestration system cannot access the remote control nodes via the OT network to manage these distributed OT assets in the same way as the container orchestration system manages its networked IT assets in terms of provisioning, deploying, operating, and maintaining assets during their respective life cycles.

[0022] With the above in mind, in certain embodiments, specialized hardware and / or software control systems may be integrated into industrial control systems operating in the OT space so that control systems native to OT assets may participate in orchestration operations. For example, in one embodiment, the control system may be configured as a worker node that may support a limited number of operations or commands provided by the container orchestration system. That is, the worker node may include an application programming interface (API) that supports receiving certain communications from a master node of the container orchestration system, parsing those communications to determine requested commands, mapping the requested commands to OT asset commands, and sending the OT asset commands to the corresponding OT assets.

[0023] In another embodiment, a separate computing system may be operated using an operating system (e.g., Linux) that enables the computing system to operate as an agent node (that is part of a container orchestration system cluster). In this way, the agent node may be programmed to execute a container daemon that enables the agent node to receive containers stored in a container registry and deploy these containers at a certain scale to one or more control systems that control the operation of one or more corresponding OT assets. When deployed in this manner, the control system (e.g., a controller) of the corresponding OT asset may unpack the container content so that the container content may be verified and loaded for execution by the corresponding control system. In addition, the agent node may provide a bidirectional bridge for coordination between the OT asset and the master node of the container orchestration system. That is, the agent node may share state data associated with the control system of the OT asset with the master node of the container orchestration system. In addition, since the agent node provides the master node with an observation of the operating state of the corresponding OT asset, the master node may include the OT asset in its scheduling operation to maintain the desired state of the OT asset set.

[0024] In any case, the embodiments described herein provide systems and methods for using an industrial control system (e.g., a controller) that is capable of controlling the operation of OT assets in an industrial automation system and that is capable of participating in a container orchestration system as a worker node or agent node. In this way, more efficient use of OT assets can be coordinated through a container orchestration system that can automatically control the processes that make up one or more applications across multiple OT assets. Therefore, the container orchestration system can provide services such as automatic updates, health monitoring, failover processes, system resource coordination, etc. to the OT asset manager while ensuring that the OT assets continue to perform their respective operations in the industrial automation system. The following will refer to Figures 1 to 8 Discuss additional details about coordinating the operations of the container orchestration system with the operations of the industrial control systems that control the OT assets.

[0025] As an introduction, Figure 1 1 is a perspective view of an example industrial automation system 10 controlled by one or more industrial control systems 12. As depicted, the industrial automation system 10 includes stations 14 having machine components and / or machines for performing functions in an automated process, such as silicon wafer manufacturing. The automated process may begin at station 14A, which is used to load an object, such as a substrate, into the industrial automation system 10 via a conveyor portion 16. The conveyor portion 16 may transport the object to station 14B to perform a first action, such as printing solder paste onto a substrate via a stencil. As the object leaves station 14B, conveyor section 16 can transport the object to station 14C for solder paste inspection (SPI) to check the printing results, to stations 14D, 14E and 14F for placement of surface mount technology (SMT) components, to station 14G so that a convection reflow oven melts the solder to form an electrical coupling, and finally to station 14H for automatic optical inspection (AOI) to inspect the manufactured object (e.g., a manufactured printed circuit board). After the object travels through the various stations, the object can be removed from station 14H for storage in a warehouse or for shipment, for example. Obviously, for other applications, the specific systems, machine components, machines, stations and / or conveyors may be different or specifically adapted for the application.

[0026] For example, the industrial automation system 10 may include machinery that performs various operations in compressor stations, refineries, batch operations for making food, chemical processing operations, brewery operations, mining operations, mechanized assembly lines, etc. Thus, the industrial automation system 10 may include various operating components, such as electric motors, valves, actuators, temperature elements, pressure sensors, or a large number of machines or equipment for manufacturing, processing, material handling, and other applications. The industrial automation system 10 may also include electrical equipment, hydraulic equipment, compressed air equipment, steam equipment, machine tools, protective equipment, refrigeration equipment, power lines, hydraulic lines, steam lines, etc. Some example types of equipment may include mixers, machine conveyors, tanks, skids, dedicated original equipment manufacturer machines, etc. In addition to the above equipment, the industrial automation system 10 may also include motors, protective equipment, switchgear, compressors, etc. Each of these described operational components may correspond to and / or generate various operational technology (OT) data regarding operations, status, sensor data, operating modes, alarm conditions, etc., which may be expected to be output for analysis using IT data from an IT network, stored in the IT network, analyzed using expected operational set points (e.g., thresholds), etc.

[0027] In some embodiments, one or more properties of equipment in the industrial automation system 10, such as a workstation 14, can be monitored and controlled by the industrial control system 12 to adjust the control variables. For example, a sensing device (e.g., a sensor 18) can monitor various properties of the industrial automation system 10 and can be used at least in part by the industrial control system 12 to adjust the operation of the industrial automation system 10 (e.g., as part of a control loop). In some cases, the industrial automation system 10 can be associated with devices used by other equipment. For example, scanners, instruments, valves, flow meters, etc. can be arranged on or inside the industrial automation system 10. Here, the industrial control system 12 can receive data from associated devices and use the data to perform their respective operations more efficiently. For example, a controller associated with a motor drive of the industrial automation system 10 can receive data about the temperature of the connected motor and can adjust the operation of the motor drive based on the data.

[0028] The industrial control system 12 may be communicatively coupled to a display / operator interface 20 (e.g., a human-machine interface (HMI)) and devices of the industrial automation system 10. It should be understood that any suitable number of industrial control systems 12 may be used in a particular implementation of the industrial automation system 10. The industrial control system 12 may facilitate representing components of the industrial automation system 10 through programming objects that may be instantiated and executed to provide simulated functionality similar to or identical to actual components and visualization of these components, or both, on the display / operator interface 20. The programming objects may include code and / or instructions stored in the industrial control system 12 and executed by processing circuitry of the industrial control system 12. The processing circuitry may communicate with memory circuitry to allow visualization of stored components.

[0029] As shown, the display / operator interface 20 depicts a representation 22 of each component in the industrial automation system 10. The industrial control system 12 can use the data transmitted by the sensor 18 to update the visualization of the component by changing one or more states, conditions and / or indications of the current operation of the component. These sensors 18 can be any suitable device suitable for providing information about the process status. In fact, the sensor 18 can be used in a process loop (e.g., a control loop) that can be monitored and controlled by the industrial control system 12. In this way, the process loop can be activated based on a process input (e.g., an input from the sensor 18) or a direct input by a person via the display / operator interface 20. A person operating and / or monitoring the industrial automation system 10 can refer to the display / operator interface 20 to determine the various states, conditions and / or current operations of the industrial automation system 10 and / or a specific component. In addition, a person operating and / or monitoring the industrial automation system 10 can adjust the various components to start, stop, power off, power on or otherwise adjust the operation of one or more components of the industrial automation system 10 by interacting with a control panel or various input devices.

[0030] The industrial automation system 10 can be considered as a data-rich environment with several processes and operations that each generate various data. For example, the industrial automation system 10 can be associated with material data (e.g., data corresponding to properties or characteristics of a substrate or raw material), parameter data (e.g., data corresponding to the performance of a machine and / or workstation during operation of the industrial automation system 10, for example), test result data (e.g., data corresponding to various quality control tests performed on final or intermediate products of the industrial automation system 10), etc., which can be organized and classified as OT data. In addition, the sensor 18 can collect OT data indicating one or more operations of the industrial automation system 10 or the industrial control system 12. In this way, the OT data can be analog data or digital data indicating measurement results, status, alarms, etc. associated with the operation of the industrial automation system 10 or the industrial control system 12.

[0031] The industrial control system 12 described above can operate in an OT space where OT data is used to monitor and control OT assets, such as the equipment shown in the workstation 14 of the industrial automation system 10 or other industrial equipment. The OT space, environment, or network typically includes direct monitoring and control operations coordinated by the industrial control system 12 and corresponding OT assets. For example, a programmable logic controller (PLC) can operate in an OT network to control the operation of OT assets (e.g., drives, motors). The industrial control system 12 can be specifically programmed or configured to communicate directly with the corresponding OT assets.

[0032] On the other hand, the container orchestration system 24 can operate in an information technology (IT) environment. That is, the container orchestration system 24 may include a cluster of multiple computing devices that coordinates the automatic process of managing or scheduling the work of each container of the application within the computing device of the cluster. In other words, the container orchestration system 24 can be used to automate various tasks at a certain scale across multiple computing devices. As an example, the container orchestration system 24 can automate the following tasks: the tasks such as configuring and scheduling containers, supplying and deploying containers, determining the availability of containers, configuring applications according to the containers in which the applications are running, extending containers to evenly balance the application workload on the infrastructure, allocating resources between containers, performing load balancing, business routing and service discovery of containers, performing health monitoring of containers, protecting interactions between containers, etc. In any case, the container orchestration system 24 can use configuration files to determine network protocols that facilitate communication between containers, storage locations for saving logs, etc. The container orchestration system 24 can also schedule the deployment of containers to clusters and identify the host (e.g., node) that is most suitable for executing the container. After identifying the host, the container orchestration system 24 can manage the life cycle of the container based on a predetermined specification.

[0033] With the above in mind, it should be noted that containers refer to a technology for packaging applications and their runtime dependencies. That is, containers include applications that are decoupled from the infrastructure (e.g., an operating system) of the underlying host. By including runtime dependencies with the container, the container can be executed in the same manner regardless of the host in which the container operates. In some embodiments, the container can be stored in a container registry 26 as a container image 28. The container registry 26 can be any suitable data storage device or database accessible to the container orchestration system 24. The container image 28 can correspond to an executable software package that includes tools and data for executing the corresponding application. In other words, the container image 28 can include relevant code for operating the application, application libraries, system libraries, runtime tools, default values ​​for various settings, etc.

[0034] As an example, a user may use an integrated development environment (IDE) tool to create a deployment profile that specifies the desired state of a set of nodes of the container orchestration system 24. The deployment profile may be stored in the container registry 26 together with the various container images 28 associated with the deployment profile. The deployment profile may include a list of different pods and multiple copies of each pod that should be operated within the container orchestration system 24 at any given time. Each pod may correspond to a logical unit of an application that may be associated with one or more containers. The container orchestration system 24 may coordinate the distribution and execution of the pods listed in the deployment profile so that the desired state can be continuously met. In some embodiments, the container orchestration system 24 may include a master node that retrieves the deployment profile from the container registry 26, schedules the deployment of the pod to the connected node, and ensures that the desired state specified in the deployment profile is met. For example, if a pod stops operating on a node, the master node may receive a notification from the corresponding working node that no longer executes the pod, and deploys the pod to another working node to ensure that the desired state is presented through the node cluster.

[0035] As mentioned above, container orchestration system 24 may include a cluster of computing devices, computing systems, or container nodes that can work together to achieve certain specifications or states as specified in corresponding containers. In some embodiments, container node 30 may be integrated into Figure 1 In other words, the container nodes 30 can be implemented by the industrial control system 12 so that they appear as working nodes relative to the master node in the container orchestration system 24. In this way, the master node of the container orchestration system 24 can send commands to the container nodes 30, which are also configured to execute applications and operations for the corresponding industrial equipment.

[0036] With this in mind, the container nodes 30 can be integrated with the industrial control system 12 so that they act as passive-indirect participants, passive-direct participants, or active participants of the container orchestration system 24. As a passive-indirect participant, the container node 30 can respond to a subset of all commands that can be issued by the container orchestration system 24. In this way, the container node 30 can support limited container lifecycle features, such as receiving pods, executing pods, updating the corresponding file system to include software packages for execution by the industrial control system 12, and reporting the status of the pod to the master node of the container orchestration system 24. The limited features that can be implemented by the container node 30 operating in passive-indirect mode can be limited to commands that the corresponding industrial control system 12 can implement using local commands, which are directly mapped to commands received by the master node of the container orchestration system 24. In addition, the container node 30 operating in the passive-indirect operation mode may not be able to push packages or directly control the operation of the industrial control system 12 to execute the package. Alternatively, the industrial control system 12 can periodically check the file system of the container node 30 and retrieve new packages for execution at that time.

[0037] As a passive-direct participant, the container node 30 can operate as a node that is part of a node cluster of the container orchestration system 24. In this way, the container node 30 can support all container lifecycle features. That is, the container node 30 operating in passive-direct mode can unpack the container image and push the resulting package to the industrial control system 12, so that the industrial control system 12 executes the package in response to receiving the package from the container node 30. In this way, the container orchestration system 24 can access the worker node that can implement the commands received from the master node directly on the industrial control system 12.

[0038] In active participant mode, the container node 30 may include a computing module or system hosting an operating system (e.g., Linux) that can continuously operate a container host daemon that can participate in the management of container operations. In this way, the active participant container node 30 can perform any operation that the master node of the container orchestration system 24 can perform. By including a container node 30 operating in the OT space, the container orchestration system 24 is able to extend its management operations to the OT space. That is, the container node 30 can supply equipment in the OT space, serve as an agent node 32 that provides bidirectional coordination between the IT space and the OT space, and the like. For example, a container node 30 operating as an agent node 32 can intercept an orchestration command and, based on the orchestration command, cause the industrial control system 12 to implement an appropriate machine control routine. The industrial control system 12 can confirm the machine status to the agent node 32, and then the agent node 32 can reply to the master node of the container orchestration system 24 on behalf of the industrial control system 12.

[0039] In addition, the industrial control system 12 can share the OT device tree via the proxy node 32. In this way, the proxy node 32 can provide the master node with status data, address data, descriptive metadata, version data, certificate data, key information, and other relevant parameters related to the industrial control system 12. In addition, the proxy node 32 can issue requests to other industrial control systems 12 to control other OT devices. For example, the proxy node 32 can use one or more OT communication protocols to convert commands and forward them to the target OT device, can receive replies from the OT device and convert them, etc. In this way, the proxy node 32 can perform health checks, provide configuration updates, send firmware patches, perform key refreshes, and other OT operations on other OT devices.

[0040] Keeping the above in mind, Figure 2 is a block diagram of an example industrial control system 12 that can be used with the embodiments described herein. The industrial control system 12 may include a communication component 42, a processor 44, a memory 46, a storage device 48, an input / output (I / O) port 50, a display 20, etc. The communication component 42 may be a wireless or wired communication component that facilitates communication between the container orchestration system 24 and the industrial control system 12 or any other suitable electronic device. The processor 44 may be any type of computer processor or microprocessor capable of executing computer executable code. The processor 44 may also include multiple processors that can perform the operations described below.

[0041] Memory 46 and storage 48 may be any suitable article of manufacture that may be used as a medium for storing processor-executable code, data, etc. These articles of manufacture may represent computer-readable media (i.e., any suitable form of memory or storage) that may store processor-executable code used by processor 44 to perform the presently disclosed techniques. Memory 46 and storage 48 may represent non-transitory computer-readable media (e.g., any suitable form of memory or storage) that may store processor-executable code used by processor 44 to perform the various techniques described herein. It should be noted that non-transitory merely indicates that the medium is tangible and not a signal.

[0042] The I / O port 50 may be coupled to one or more sensors 18, one or more input devices, one or more displays, etc. to facilitate human or machine interaction with the industrial control system 12. For example, based on notifications provided to the user via the display 20, the user may use an input device to indicate adjustments to the OT device.

[0043] As discussed above, the display 20 can be operated to depict visualizations associated with the software or executable code being processed by the processor 44. In one embodiment, the display 20 can be a touch display capable of receiving input from a user of the industrial control system 12. For example, the display 20 can be any suitable type of display, such as a liquid crystal display (LCD), a plasma display, or an organic light emitting diode (OLED) display. Additionally, in one embodiment, the display 20 can be provided in conjunction with a touch-sensitive mechanism (e.g., a touch screen) that can be used as part of a control interface for the industrial control system 12.

[0044] Although the industrial control system 12 is described Figure 2 , but it should be noted that the container orchestration system 24, the container node 30, and the proxy node 32 may also include the same or similar components to perform or facilitate the various techniques described herein. Figure 2 The depicted components are exemplary illustrations, and the industrial control system 12 and other suitable computing systems may include more or fewer components as described in detail above.

[0045] Keeping the above in mind, Figure 3 A block diagram depicting the relative positions of container nodes 30 and proxy nodes 32 relative to container orchestration system 24 is shown. As mentioned above, container orchestration system 24 may include a collection of nodes for implementing a desired state of one or more containers through multiple nodes. Figure 3 As shown, the container orchestration system 24 may include a master node 62, which may execute control plane processes for the container orchestration system 24. The control plane processes may include processes that enable the container orchestration system 24 to coordinate the operations of the container nodes 30 to meet the desired state. In this way, the master node can execute application programming interfaces (APIs) for the container orchestration system 24, scheduler components, core resource controllers, etc. As an example, the master container node 62 can coordinate all interactions between the nodes that make up the cluster of the container orchestration system 24. In fact, the master container node 62 can be responsible for determining the operations that will be run on the container node 30, including scheduling workloads (e.g., containerized applications), managing the life cycle of the workload, expansion and upgrades, managing the network and storage resources of the workload, etc. The master container node 62 can run an API server to handle requests and status updates received from the container node 30.

[0046] By way of operation, an integrated development environment (IDE) tool 64 may be used by an operator to develop a deployment configuration file 65. As mentioned above, the deployment configuration file 65 may include details about containers, pods, constraints for operating containers / pods, and other information describing the desired state of the container specified in the deployment configuration file 65. In some embodiments, the deployment configuration file 65 may be generated in a YAML file, a JSON file, or other suitable file format compatible with the container orchestration system 24. After the IDE tool 64 generates the deployment configuration file 65, the IDE tool 64 may transmit the deployment configuration file 65 to the container registry 26, which may store the file together with the container image 28, which represents the container stored in the deployment configuration file 65.

[0047] In some embodiments, the master container node 62 may receive a deployment configuration file 65 directly from an IDE tool 64 or the like via a container registry 26. The master container node 62 may use the deployment configuration file 65 to determine where to collect container images 28, determine a communication protocol for establishing networking between container nodes 30, determine a location for mounting storage volumes, a location for storing logs about containers, and the like.

[0048] Based on the desired state provided in the deployment configuration file 65, the master container node 62 may deploy the container to the container host node 30. That is, the master container node 62 may schedule the deployment of the container based on the constraints (e.g., CPU or memory availability) provided in the deployment configuration file 65. After the container is operating on the container node 30, the master container node 62 may manage the life cycle of the container to ensure that the container specified by the deployment configuration file 65 is operating according to the specified constraints and desired state.

[0049] With the above in mind, the industrial control system 12 may not use an operating system (OS) that is compatible with the container orchestration system 24. That is, the container orchestration system 24 may be configured to operate in an IT space involving digital information flows. In contrast, the industrial control system 12 may operate in an OT space involving the management of physical processes and the operation of machinery used to perform those physical processes. For example, the OT space may involve communications formatted according to an OT communication protocol, such as FactoryTalk Live Data, EtherNet / IP, Common Industrial Protocol (CIP), OPC direct access (e.g., a machine-to-machine communication protocol for industrial automation developed by the OPC Foundation), or any suitable OT communication protocol (e.g., DNP3, Modbus, Profibus, LonWorks, DALI, BACnet, KNX, EnOcean). Because the industrial control system 12 operates in the OT space, the industrial control system cannot implement commands received via the container orchestration system 24.

[0050] In some embodiments, the container node 30 may be programmed or implemented in the industrial control system 12 to act as a node agent that can register the industrial control system 12 with the main container node 62. For example, the industrial control system 12 may include a programmable logic controller (PLC), which cannot support an operating system (e.g., Linux) for receiving and / or implementing request operations issued by the container orchestration system 24. However, the PLC can perform certain operations that can be mapped to certain container events. In this way, the container node 30 may include software and / or hardware components that can map certain events or commands received from the main container node 62 to actions that can be performed by the PLC. After converting the received command into a command that can be interpreted by the PLC, the container node 30 can forward the mapped command to the PLC that can implement the mapped command. In this way, the container node 30 can operate as part of a node cluster that constitutes the container orchestration system 24, and the control system 66 (e.g., PLC) coordinates OT operations for OT devices 67 in the industrial control system 12. Control system 66 may include a controller, such as a programmable logic controller (PLC), a programmable automation controller (PAC), or any other controller that can monitor, control, and operate industrial automation equipment or components.

[0051] Industrial automation equipment or components may correspond to OT devices 67. OT devices 67 may include any suitable industrial equipment operating in the OT space. In this way, OT devices 67 may involve adjusting the physical process implemented by the industrial system 10. In some embodiments, OT devices 67 may include motor control centers, motors, human-machine interfaces (HMIs), operator interfaces, contactors, starters, sensors, drives, relays, protection devices, switchgear, compressors, network switches (e.g., Ethernet switches, modular managed switches, fixed managed switches, service router switches, industrial switches, unmanaged switches, etc.), etc. In addition, OT devices 67 may also involve various industrial equipment, such as mixers, machine conveyors, tanks, skids, specialized OEM machines, etc. OT devices 67 may also be associated with equipment used by the equipment, such as scanners, meters, valves, flow meters, etc. In one embodiment, each aspect of OT devices 67 may be controlled or operated by a control system 66.

[0052] In the present embodiment described herein, the control system 66 can therefore perform actions based on commands received from the container node 30. By mapping certain container lifecycle states to appropriate corresponding actions that can be implemented by the control system 66, the container node 30 enables the program content for the industrial control system 12 to be containerized, published to certain registries, and deployed using the master container node 62, thereby establishing a bridge between the IT-based container orchestration system 24 and the OT-based industrial control system 12.

[0053] Figure 4 The above reference shows Figure 3 80 of the master container node 62 and the container node 30 described in the embodiment of the present invention to track the deployment of the container. At step 82, the IDE tool 64 can create a deployment profile 65 to the master container node 62. After receiving the deployment profile 65, the master container node 62 can identify the container node 30 that can meet the constraint specifications (e.g., memory, CPU availability) provided in the deployment profile 65. In other words, the master container node 62 performs a scheduling operation, which involves checking the status data of each node of the container orchestration system 24, determining whether there is a suitable node for the constraints provided in the deployment profile 65, etc.

[0054] After identifying the appropriate container node, at step 84, the master container node 62 can schedule the deployment of the container to the corresponding container node 30. At step 86, the master container node 62 can deploy the pod to the container node 30, which can cause the container node 30 to retrieve the container image 28 from the container registry 26 at step 88. In this way, the container node 30 is configured to receive the pod from the master container node 62 and execute the pod, although the control system 66 of the industrial control system 12 may not be able to execute the pod. After pulling the container image 28 from the container registry 26, at step 90, the container node 30 can run the container image 28 or unpack the container image 28 and store the application or operation executable by the control system 66 in the file system. At step 92, the control system 66 can check the file system of the container node 30 for updated or newly stored packages. At step 94, the control system 66 can retrieve the stored package, which may include the container scheduled by the master container node 62 at step 84 for the operation performed by the control system 66. At step 96, the control system 66 may execute the package retrieved from the file system of the container node 30. The container node 30 may then monitor the operation of the package executed on the control system 66, the state of the control system 66 (e.g., memory, CPU usage), and send updates to the container node 62. If the container node 30 sends an update indicating that the control system 66 is no longer executing the package, the master container node 62 may schedule the deployment of another pod to another container node 30 for execution to maintain the desired state.

[0055] Keeping the above in mind, Figure 5 A method 110 is shown that the container node 30 may implement via the master container node 62 of the container orchestration system 24 to coordinate activities. Although the following description of the method 110 is described as being performed by the container node 30, it should be understood that any suitable container node configured to interface with the industrial control system 12 and the node cluster of the container orchestration system 24 may perform the operations described herein. In addition, although the method 110 is described in a particular order, it should be understood that the method 110 may be performed in any suitable order.

[0056] Now refer to Figure 5, at box 112, the container node 30 may receive a pod from the master container node 62. As mentioned above, a pod may include one or more containers deployed to a host (e.g., a container node 30). In some embodiments, a pod may operate to specify coordinated operations between a set of containers. A pod may include an indication or reference to a storage location of one or more container images 28 associated with the pod. In some embodiments, the container image 28 may be stored in a container registry 26. In this way, the pod may provide an indication of a network address or other address for accessing a storage location in a container registry 26.

[0057] After receiving the pod from the master container node 62, at box 114, the container node 30 can download the relevant container image 28 from the container registry 26. As mentioned above, the container image 28 represents data that encapsulates the application and its software dependencies. The container image 28 can be an executable software bundle that can be executed as independent software regardless of the operating system being used by the corresponding container node 30. In some embodiments, the container node 30 can receive the container image directly from the container registry 26 via the master container node 62 or any other suitable communication scheme.

[0058] After receiving the container image 28, at box 116, the container node 30 can run the container image 28 or unpack it, and determine commands that can be executed by the control system 66 based on the container image 28. That is, the container image 28 may include software applications that are executable by the container node 30. However, these software applications may not be executable by the control system 66. In this way, the container node 30 can determine commands for the control system 66 that correspond to the software applications encapsulated in the container image 28. In some embodiments, certain operations in the software application can be mapped to certain operations that can be performed by the control system 66. For example, software application commands for create, run, start, pause, stop, and delete can be mapped to download, download / run, run, idle, and uninstall commands that can be executed by the control system 66, respectively.

[0059] After determining the commands that can be implemented by the control system 66 based on the container image 28, at block 118, the container node 30 may generate a package that can be retrieved and executed by the control system 66. That is, the container node 30 may organize or structure the determined commands into a software package that can be used by the control system 66. For example, if the control system 66 corresponds to a PLC, the package may be organized according to a programming structure (e.g., ladder logic) that the PLC itself may use for programming.

[0060] At block 120, the container node 30 may store the package in a memory or file system accessible to the control system 66. In some implementations, the container node 30 may not be able to interface directly with the control system 66. However, the container node 30 may provide the control system 66 with access to its memory or storage components so that the control system 66 can retrieve the stored packages. The control system 66 may be programmed to periodically (e.g., daily, hourly) check the file system for updates or new packages available for download.

[0061] It should be noted that the method 110 may describe a passive-indirect mode of operation of a container node 30 as part of a container orchestration system 24. That is, the container node 30 is limited to executing a subset of the commands that can be provided by the master container node 62, rather than executing all commands that a node that is part of the container orchestration system 24 can execute. In addition, the container node 30 may not be able to perform all lifecycle operations provided in the container. For example, as described above, the container node 30 can facilitate a package download operation of the control system 66, but if the control system 66 does not have a corresponding operation that can be mapped to a corresponding container lifecycle operation, it may not be able to perform every function specified in the container.

[0062] In some embodiments, the container node 30 can operate in a passive-direct operation mode, in which the container node 30 can participate as a node in the container orchestration system 24. In this way, the container node 30 can support the full set of container lifecycle operations. That is, since the control system 66 is capable of implementing a limited set of commands provided by the main container node 62, the container node 30 can be implemented or executed by a sidecar computing module, which can host a container host daemon that can perform a full set of operations that a node in the container orchestration system 24 can perform. In this way, the sidecar computing module can be any suitable computing system capable of executing an operating system (OS) so that commands received from the main container node 62 can be implemented by the corresponding sidecar computing module.

[0063] By implementing the container node 30 with a sidecar computing module, the container node 30 can operate as a node that is part of the container orchestration system 24 but operates in the OT space. As a result, the container node 30 can extend the functionality available via the container orchestration system 24 to OT devices 67 that are generally invisible to the main container node 62 of the container orchestration system 24. In order to operate in passive-direct mode, the container node 30 may include applications and / or APIs that directly interface with the control system 66 and the main container node 62. In this way, the container node 30 can provide a two-way communication bridge between the control system 66 and the main container node 62. In some embodiments, the container node 30 may include an API that converts OT data received from the control system 66 into IT data that can be interpreted by the main container node 62. In this way, the container node 30 can provide visibility to the operation and status of the OT devices 67 operating in the OT space to the main container node 62.

[0064] Remember this, Figure 6 A method 130 for a container node 30 to send packets directly to a control system 66 and machine state data directly to a master container node 62 is shown in accordance with an embodiment described herein. Similar to the method 110 described above, the following description of the method 130 may be performed by any suitable computing system and in any suitable order. It should be noted that the method 130 described below corresponds to operating a container node 30 in a passive-direct operation mode. In this way, the container node 30 may receive and interpret orchestration commands received from a master container node 62, etc., and directly interface with the control system 66 to verify that the control system 66 is operating in the desired state.

[0065] Reference Figure 6 At block 132, container node 30 may receive an orchestration command from master container node 62 or some other node that is part of container orchestration system 24. An orchestration command may include instructions regarding one or more containers that control system 66 should currently be executing. As such, an orchestration command may be part of a monitoring function that causes container node 30 to verify that machine state data of control system 66 corresponds to an expected machine state as specified by deployment configuration file 65.

[0066] At block 134, the container node 30 may retrieve machine state data from the control system 66. The machine state data may include the current operational state of each OT device controlled by the control system 66 (e.g., active, inactive), available processing resources (e.g., CPU availability), available memory resources (e.g., storage, RAM), etc. The machine state data may also indicate whether any containers are being executed by the control system 66. In this way, the machine state data may be reported back to the master container node 62 to ensure that the desired state specified by the deployment configuration file 65 exists.

[0067] To receive machine status data, the container node 30 may send a request to the control system 66 via an appropriate OT communication protocol. In response to receiving the request, the control system 66 may query a database, storage unit, or other suitable storage device that may include information about the requested data. After retrieving the requested information, the control system 66 may send the requested data to the container node 30 using the same OT communication protocol as the OT communication protocol it received the request.

[0068] In some embodiments, the container node 30 may be embedded or integrated into one or more cores of the control system 66. In this way, the container node 30 may communicate data with portions of the control system 66 using onboard communication methods. Alternatively, the container node 30 may retrieve the machine state data directly from a corresponding memory location.

[0069] After receiving the machine state data from the control system 66, at block 136, the container node 30 may determine whether the control system 66 is operating in an expected state based on the deployment configuration file 65. In this embodiment, the container node 30 may evaluate whether the control system 66 is executing the container as specified in the deployment configuration file 65. That is, since the container node 30 may execute the container host daemon, the container node 30 may participate in the management of containers distributed throughout the container scheduling system 24 by monitoring the machine state data of the control system 66.

[0070] If the control system 66 is operating in the desired state, the container node 30 may proceed to block 138 and send the machine state data to the master container node 62. The master container node 62 may then check whether the other container nodes 30 are achieving the desired state.

[0071] However, if the container node 30 determines that the control system 66 is not operating in the desired state, the container node 30 can proceed to box 140 and generate a packet that can cause the control system 66 to modify its operation to execute the corresponding pod and the container therein. After generating the packet, the container node 30 can send the packet directly to the control system 66 for execution. In this way, the container node 30 operates in a passive-direct mode because the container node 30 can directly send commands that cause the control system 66 to change operation. For example, instead of the control system 66 periodically checking the file system (as described above with respect to Figure 5 In some embodiments, the control system 66 may interface directly with the container node 30 and receive commands that may cause it to adjust the operation of a connected device (e.g., an OT device), execute received packets, etc. (e.g., passive-indirect mode). In this way, the container node 30 may push packets, firmware updates, OT device credential updates, security updates, encryption keys, and other data to the control system 66 at any given time. That is, the container system 30 may generate an event notification that causes other components or APIs of the control system 66 to react in response to detecting the event notification. In this way, the container node 30 may actively participate in the coordination of the container with the corresponding control system 66 based on orchestration commands passively received from the master container node 62, etc.

[0072] By employing container nodes 30 to enable the container orchestration system 24 to implement software containers on the control system 66, the present embodiment described herein enables coordination of control of multiple control systems 66 and multiple OT devices 67 to control operations in the industrial automation system 10. That is, the desired machine state may include desired operating parameters for the industrial equipment, and the container orchestration system 24 may monitor available industrial equipment resources to ensure that the desired machine state is continuously achieved by coordinating activities via the container nodes 30 communicatively coupled to the control system 66.

[0073] In addition to operating in passive-indirect mode and passive-direct mode, the container node 30 may operate in active mode so that the container node may invoke container orchestration commands for other container nodes 30. For example, referring back to Figure 3, the proxy node 32 can act as a proxy or gateway node as part of the container orchestration system 24. The proxy node 32 can be implemented as a sidecar computing module with an operating system (OS) that supports a container host daemon. In another embodiment, the proxy node 32 can be implemented directly on a configured (e.g., partitioned) core of the control system 66, so that the control system 66 can operate using an operating system that enables the container node 30 to execute orchestration commands and act as part of the container orchestration system 24. In any case, the proxy node 32 can act as a bidirectional bridge for IT / OT orchestration, which enables the execution of automation functions in IT devices based on OT data and the execution of automation functions in OT devices 67 based on IT data. For example, the proxy node 32 can obtain OT device tree data, status data of OT devices, descriptive metadata associated with corresponding OT data, version data of OT devices 67, certificate / key data of OT devices, and other relevant OT data via the OT communication protocol. The proxy node 32 can then convert the OT data into IT data, which can be formatted so that the master container node 62 can extract relevant data (e.g., machine status data) to perform scheduling operations and ensure that the container orchestration system 24 and the connected control system 66 operate in the desired state. Based on the results of its scheduling operations, the master container node 62 can issue supervisory control commands to the target OT device via the proxy node 32, and the proxy node 32 can convert these commands via the appropriate OT communication protocol and forward the converted commands to the corresponding control system 66.

[0074] In addition, the proxy node 32 may also perform certain supervisory operations based on its analysis of the machine status data of the corresponding control system 66. As a result of its analysis, the proxy node 32 may issue commands and / or pods to other nodes that are part of the container orchestration system 24. For example, referring to Figure 3, the proxy node 32 can send instructions or pods to other worker container nodes 68 that can be part of the container orchestration system 24. The worker container node 68 can correspond to other container nodes 30 that are communicatively coupled to other control systems 70 to control other OT devices 71. In this way, the proxy node 32 can convert the command or directly forward the command to the other control system 70 via some OT communication protocol, or indirectly forward the command to the other control system 70 via other worker container nodes 68 associated with other control systems 70. In addition, the proxy node 32 can receive replies from the control system 70 via the OT communication protocol and convert these replies so that the nodes in the container orchestration system 24 can interpret these replies. In this way, the container orchestration system 24 can effectively perform health checks, send configuration updates, provide firmware patches, perform key refreshes, and provide other services to OT devices 71 in a coordinated manner. That is, the proxy node 32 can enable the container orchestration system to coordinate the activities of multiple control systems 66 and 70 to achieve a set of desired machine states for the connected OT devices 67 and 71. Implementing serverless functions in industrial control systems using container orchestration systems

[0075] In addition to coordinating communications between IT systems and OT systems, the container orchestration system 24 can also be used to implement function as a service (FaaS) operations or serverless functions using the embodiments described herein. More specifically, the master container node 62 can distribute FaaS operations on one or more control systems 70 using corresponding worker container nodes, etc. In this way, the container orchestration system 24 can utilize the available resources of the control system 70 in the OT space to coordinate various OT operations.

[0076] By way of introduction, a serverless function or FaaS operation corresponds to a computing scheme that enables execution of certain logic or applications (e.g., packages, containers) without regard to the available computing resources available on the corresponding device. In other words, regardless of the current state of the control system 66 (e.g., CPU availability, memory availability, line availability, device availability), the master container node 62 can identify the working container node 68 and / or the control system 70 to host and / or execute the application. In some embodiments, the FaaS operation can be triggered based on an asynchronous event occurring in another system (e.g., the control system 70). For example, the master container node 62 monitors incoming messages from other container nodes of the container orchestration system 24 to obtain events, triggers, or notifications that may cause the FaaS operation to be executed. In addition, the master container node 62 can monitor the shared file system to obtain new files, rows added to a database table, data added to a database, etc. The master container node 62 can also listen to API endpoint calls (e.g., GET, POST, PUT, Delete) or any other suitable triggering events. In any case, the master container node 62 may employ a function (eg, an executing application) of continuously (eg, periodically) monitoring an external system for a state change that corresponds to causing another application to execute.

[0077] Remember this, Figure 7 A method 150 for performing an example FaaS operation using the embodiments described herein is shown. As an example, the method 150 described below will be described in detail as being performed by the master container node 62. However, it should be noted that any suitable container node capable of coordinating the activities of nodes in the container orchestration system 24, such as the agent node 32, can perform the method 150. In addition, although the following description of the method 150 is described in a particular order, it should be understood that the method 150 can be performed in any suitable order.

[0078] Now refer to Figure 7At block 152, the master container node 62 may receive event-based notifications from the container node 30, the worker container node 68, or other suitable nodes in the container orchestration system 24. In some embodiments, the event-based notifications may be related to conditions or states corresponding to the control system 66, the OT device 67, and the like. Thus, the event-based notifications may correspond to states that satisfy a certain condition, exceed a certain threshold, fall below a certain threshold, and the like. For example, the event-based notifications may be related to conditions that are related to computing properties associated with the control system 66 (e.g., CPU availability), machine availability of the OT device 67 (e.g., available, unavailable), or other variables detected by the control system 66. Thus, the event-based notifications may be generated by the control system 66 and reported to the container node 30, which may convert the event-based notifications into a format that the master container node 62 can interpret.

[0079] At box 154, the master container node 62 can determine an analytical operation or logical operation to be performed based on the event-based notification. That is, the master container node 62 can monitor the event-based notification, which can cause the master container node 62 to execute an application or function (e.g., a container) in response to receiving the event-based notification. In some embodiments, the called function may involve performing certain analytical operations on data stored on various devices (e.g., OT devices, control systems). For example, if the event-based notification involves an overheating of the OT device 67 or the corresponding control system 66 detects that the temperature associated with the OT device 67 exceeds a threshold, the master container node 62 can perform an analytical operation to determine the average temperature increase of multiple OT devices over a period of time (e.g., 24 hours).

[0080] To perform an analysis operation, the master container node 62 may identify one or more functions (e.g., equations, processes), one or more variables, and other data elements that may be involved in performing the analysis. In some embodiments, the master container node 62 may execute a container in response to receiving an event-based notification, and the container may include relevant information about the analysis operation to be performed, the function used to perform the analysis operation, the variables or data involved in completing the analysis operation, and the like. A portion of the information or data that may be used to perform the analysis operation may include pre-analysis data. Pre-analysis data may include certain data sets that have been pre-processed or collected to facilitate higher-level analysis operations. Referring again to the example provided above, in order to determine the average temperature increase of multiple OT devices, the pre-analysis data may include the average temperature increase of an OT device over a period of time.

[0081] With this in mind, a container for performing an analysis operation may enable master container node 62 to determine whether other nodes of container orchestration system 24 may perform certain pre-analysis operations. Alternatively, the container may include a list of commands or instructions for other container nodes 30 to perform corresponding pre-analysis operations. In any case, at block 156, based on the instructions provided by the corresponding container, master container node 62 may determine a pre-analysis operation related to the analysis operation determined at block 154.

[0082] After determining the pre-analysis operations to be performed by the various container nodes 30 or other industrial control systems 66, at block 158, the master container node 62 may generate or retrieve one or more pods for deployment to one or more container nodes 30. Figure 4 As described, a pod may include executable instructions that cause a corresponding container node 30 to retrieve a corresponding container image 28 associated with the pod. In some embodiments, a pod may be stored as part of a container that specifies a container node 30 and / or control system 66 that may perform a corresponding pre-analysis operation.

[0083] In some embodiments, before deploying the pod, at box 160, the master container node 62 can identify the control system 66 that can perform the pre-analysis operation. That is, the master container node 62 can schedule the deployment of the pod based on the constraints associated with the container orchestration system 24. For example, the master container node 62 can confirm that deploying the pod and causing the corresponding control system 66 to perform the pre-analysis operation will still enable the control system 66 to maintain the machine state corresponding to the constraints defined in the deployment configuration file 65. In this way, the master container node 62 can schedule the deployment of the pod at a specific time or cause the pod to call the control system 66 at the following time: the control system 66 can perform the pre-analysis operation while maintaining the desired machine state. If the corresponding container node 30 indicates to the master container node 62 that the control system 66 may not be able to perform the pre-analysis operation, the master container node 62 can deploy other pods to one or more other container nodes 68 to retrieve raw data from the unavailable control system 66 (e.g., via one or more intermediate container nodes 30) so that other control systems 70 can perform the pre-analysis operation. In fact, the master container node 62 can confirm that the other control systems 70 can perform corresponding pre-analysis operations based on the corresponding machine state data received via the corresponding other container nodes 68. In this way, the coordination of pre-analysis operation commands can be facilitated by the master container node 62 as part of the container orchestration system 24 and its local scheduling operation.

[0084] After identifying the appropriate container node 30, the master container node 62 may send the pod to the corresponding container node 30. The container node 30 may then pull the container image associated with the received pod and store the associated package on a file system shared with the control system 66 or send the package directly to the control system 66 (e.g., via passive-direct mode, active mode operation) for implementation.

[0085] After the control system 66 performs the pre-analysis operation and obtains the pre-analysis data, at box 164, the master container node 62 can receive the pre-analysis data from the container node 30. As discussed above, the container node 30 can receive the pre-analysis data via the OT communication protocol and convert the received data into a format that can be interpreted by the master container node 62. Because the pre-analysis data is processed near the data source and organized as pre-analysis data, the amount of data sent to the master container node 62 is less than the amount of raw data analyzed to obtain the pre-analysis data. In this way, the container orchestration system 24 can reduce the amount or capacity of network traffic transmitted across the nodes of the container orchestration system 24 by using the control system 70 to process the raw data and transmit smaller pre-analysis data results.

[0086] With this in mind, the present embodiment enables the container orchestration system 24 to implement serverless functions in an industrial automation environment by coordinating various data processing operations on different control systems 70. As a result, users and / or developers from different domains (e.g., the IT domain and the OT domain) can create a joint solution using their respective tools in their respective environments. For example, a control engineer working with the control system 60 may use a language such as ladder logic or function blocks to create a robust control program, while an IT engineer may use a language such as Python or JavaScript to create an analysis or integration solution. The container orchestration system 24 may coordinate the deployment of tasks across different domains while maintaining the desired machine state across the IT domain and the OT domain. In this way, the container orchestration system 24 enables each developer to build their part of the solution without sacrificing the advantages of their respective tools or having excessive knowledge of the domains of other developers. Furthermore, by employing a control system 70 (e.g., a controller in a chassis or other compute surface / sidecar) to perform certain functions (e.g., IT operations), the present embodiments described herein can reduce the complexity associated with networking connections between various devices (e.g., OT devices and IT devices) and providing security throughout the communications by utilizing certain OT communication protocols. Furthermore, by coordinating operations on the control system 70 of the industrial control system 12, the container orchestration system 24 can improve the reliability and availability of the generated data by scheduling operations based on available resources and desired machine states.

[0087] Since the main container node 62 deploys functions in containers (e.g., making them serverless), the container orchestration system 24 eliminates the need for IT developers to understand the infrastructure on which the functions run. In addition, since the logic of the functions runs in containers, control engineers are not concerned with other processing that changes the performance or behavior of the implemented control programs. As a result, the present embodiment described herein provides a robust computing architecture that utilizes the control system 70 available in the industrial control system 12 and the lifecycle management operations of the container orchestration system 24 to effectively coordinate data analysis operations, machine control operations, line control operations, etc. As an example, the container orchestration system 24 can push the analysis operations to be performed to a location close to the data input source, and make the results of the analysis operations available for downstream analysis. In addition, the container orchestration system 24 can shape the data for use by other services, generate user-friendly patterns for graphical representation using GraphQL, provide access to third-party systems (e.g., calling web services, accessing databases), and perform other operations in the IT field. In addition, the container orchestration system 24 can coordinate OT operations, such as obtaining recipe data from the control system 66 for batch processing, logging information to MES applications, reporting detected OT device anomalies, tracking inventory material consumption and quality, etc. In fact, OT data can be made available to IT devices via the container orchestration system 24 for data processing, thereby utilizing the data processing capabilities of IT devices for OT data.

[0088] Providing rolling updates in industrial control systems using container orchestration systems

[0089] In addition to pushing analytics operations to the control system, the present embodiment can also be used to provide rolling updates to the control system 66, OT devices 67, and other suitable components that may exist in the industrial system 10, industrial control system 12, etc. In an industrial environment, software patches, firmware updates, and other updates are implemented by the control system 66 when the corresponding OT devices 67 are offline. That is, in order to update the control system 66 or OT devices 67, the corresponding devices can be taken offline before the updates are implemented to the control system 66 and / or OT devices 67. In the industrial automation system 10, taking components offline can directly affect production because fewer online OT devices 67 can reduce the productivity or efficiency of the industrial system 10.

[0090] With this in mind, the container orchestration system 24 described herein can coordinate the distribution of updates across the various components of the industrial system 10 to ensure that the components are updated while maintaining the desired state of the industrial system 10. That is, the deployment profile 65 for distributing updates can include desired state data indicating the number of control systems 66 and / or OT devices 67 that can remain operational at any given time. In some embodiments, in order to implement the following method for providing updates, redundant control systems 66 and / or OT devices 67 can be used to allow the container orchestration system 24 to transfer operations between components to update certain components when they are offline and keep other components online to perform their respective operations. In addition, the container orchestration system 24 can schedule the deployment of updates based on the expected operational schedule for each control system 66 and / or OT device 67. That is, the container orchestration system 24 can monitor the operating mode of each control system 66 and / or OT device 67, and schedule the deployment of the corresponding pods to update these components during the time period in which the components are planned to be offline. In any case, the present embodiments described herein enable updates to components in the industrial system 10 to be provided in a coordinated manner based on available resources and desired state to ensure that the industrial system 10 continues to operate according to the desired protocol.

[0091] As an example, Figure 8 A flow chart is shown of a method 180 that the master container node 62 in the container orchestration system 24 may use to distribute updates to the control system 66 and / or the OT device 67. Similar to the methods described above, although the method 180 is described as being performed by the master container node 62 and in a particular order, it should be understood that the method 180 may be performed by any suitable computing system or node in the container orchestration system 24 in any suitable order.

[0092] Now refer to Figure 8At box 182, the master container node 62 may receive update data for the target control system 66. The update data may be provided via a deployment configuration file 65 or via a command input by a user of the container orchestration system 24. The update data may include software updates, controller updates, firmware updates, or other types of updates that may modify the operation of the target control system 66 and / or OT device 67. In addition, the update data may include an indication of the target control system 66 and / or target OT device 67 to which the update may be implemented. In some embodiments, the update data may also include desired machine state data for a collection of control systems 66 and / or OT devices of the industrial system 10. The desired machine state data may include the number of control systems 66 and / or OT devices and the corresponding desired operating states of the various components. In this way, the master container node 62 may analyze the components of the industrial system 10 to ensure that the desired machine state is maintained while providing updates to the target control system 66 and / or OT device 67.

[0093] At box 184, the master container node 62 can receive machine state data of the target control system 66 and other control systems 70 and / or associated OT devices 67 that are part of the industrial control system 12 and visible to the container orchestration system 24. In this way, the master container node 62 can send a request for machine state data of the corresponding component to the proxy node 32 and / or the target container node 30, container node 68 that are communicatively coupled to the target control system 66 and other control systems 70. The machine state data may include data about the availability of certain computing resources (e.g., memory, processing power) of the corresponding control systems 66 and 70. In addition, the machine state data may also include information about processes and / or containers executed by the control systems 66 and 70. In addition, the machine state data may also include information about the operating state (e.g., operating mode, state) of the connected OT device 67.

[0094] Based on the machine state data of the control systems 66 and 70 and / or the OT devices 67, the master container node 62 can schedule the deployment of the pod to the target control system 66 and the redundant control system 70. The redundant control system 70 can include any suitable control system 70 that can interface with and communicate with the target OT device 67. In this way, the pod can be associated with suspending the operation of the corresponding control system 70 and / or the corresponding OT device 67. In addition, the pod can be associated with using the redundant control system 70 to perform the suspension operation of the corresponding control system 70 and / or the corresponding OT device 71.

[0095] In some embodiments, the master container node 62 can monitor the machine status data of the control systems 66 and 70, and can maintain a list of OT devices 67 present in the industrial system 10. Using this list, the master container node 62 can identify a control system 79 or OT device 67 that can replace the operation of the target control system 66 and / or the target OT device 67. After identifying a suitable control system 70 and / or OT device 67, the master container node 62 can analyze the corresponding machine status data to identify a suitable control system 70 and / or OT device 67 to be used as a replacement control system 70 and / or OT device 71. In some embodiments, the replacement control system 70 and / or OT device 71 can correspond to a redundant control system and / or OT device that can be reserved for performing operations in the event that the target control system 66 and / or the target OT device 67 is unavailable. In this way, the replacement control system 70 and / or OT device 71 can be identified as a component that can be used to perform the corresponding operation of the target control system 66 and / or OT device 67 when the target control system 66 and / or OT device 67 is updated.

[0096] In addition, the replacement control system 70 and / or OT device 71 may correspond to other control systems and / or OT devices that may currently be used for certain operations but may also be configured to perform additional operations. That is, some control systems may include sufficient processing and computing resources to control multiple OT devices. In this way, the master container node 62 can receive machine status data from these control systems via the corresponding container nodes to determine whether they will be able to perform the operations of the target control system 66 and / or OT device 67.

[0097] After identifying the replacement control system 70 and / or OT device 71, at box 186, the master container node 62 can schedule the deployment of the pod to the identified component to continue the operation currently being performed by the target control system 66 and / or OT device 67. In addition to sending the pod to the identified component, at box 188, the master container node 62 can also deploy the updated pod to the target control system 66 and / or OT device 67 to cause the target control system 66 and / or OT device 67 to suspend operation and receive one or more corresponding updates. In some embodiments, the master container node 62 can schedule the deployment of the pod so that the replacement control system 70 and / or OT device 71 assumes the responsibilities of the target control system 66 and / or OT device 67 after the target control system 66 and / or OT device 67 suspends operation. That is, the master container node 62 can coordinate the operation of the target control system 66 and / or OT device 67 to stop, while the replacement control system 70 and / or OT device 71 takes over the corresponding operation. The master container node 62 can coordinate the transfer to ensure that the OT device continues to perform the functions of the industrial system 10.

[0098] After the pods are deployed to the replacement control system 70 and / or OT device 71 and the target control system 66 and / or OT device 67, at block 190, the master container node 62 may again receive machine state data of the control system and / or OT device accessible to the master container node 62. In this way, the master container node 62 may receive updated machine state data after the pods are deployed to the redundant control system 70 and / or OT device 71 and the target control system 66 and / or OT device 67. The master container node 62 may verify that the industrial system 10 is achieving the desired machine state of the various components herein.

[0099] Based on the updated machine state data, at box 200, the master container node 62 can schedule the deployment of the pod to the redundant control system 70 and / or OT device 71 and the target control system 66 and / or OT device 67 again. That is, if the machine state data of the industrial system 10 does not correspond to the desired state, the master container node 62 can send additional pods to the redundant control system 70 and / or OT device 71 to modify their respective operations to achieve the desired machine state. However, if the machine state data indicates that the update is complete, the master container node 62 can schedule the deployment of the pod to the redundant control system 70 and / or OT device 71 and the target control system 66 and / or OT device 67, so that the redundant control system 70 and / or OT device 71 suspends its respective operations, and the target control system 66 and / or OT device 67 resumes the operation previously performed before receiving the updated pod. In some embodiments, the master container node 62 can enable the redundant control system 70 and / or OT device 71 to maintain its current machine state and keep the target control system 66 and / or OT device 67 available as a redundant component of the industrial system 10. In any case, the master container node 62 can update the component lists in the industrial system 10 and the industrial control system 12 to represent the current configuration of each component.

[0100] By employing a container scheduling system 24 to perform update operations on components in the industrial system 10, the present embodiment described herein enables the industrial system 10 to continue operating without reducing uptime or decreasing production efficiency. In fact, other update methods involve powering off components, thereby reducing the production or operational efficiency of the industrial system 10. In addition, updates may involve performing tasks in a design-time environment when the corresponding components (e.g., control systems) are disconnected from the coordinated operations performed by the corresponding OT devices. With this in mind, the container orchestration system 24 coordinates the update process in the runtime environment while the components continue to perform their corresponding operations. In fact, the operations of various components can be transferred between available resources in the industrial system 10, but the operational tasks of the industrial system 10 remain online, and therefore the productivity of the industrial system 10 remains unchanged.

[0101] In addition to providing rolling updates, the container orchestration system 24 can also perform other high availability operations, such as periodically testing the health of the industrial control system 12 or any suitable control system 66 / 70 and transmitting the detected health status back to the master container node 62. If the health check fails, the master container node 62 can initiate a new control process pod launched via the agent node 32, which corresponds to the unhealthy control system 66 / 70, OT device 67 / 71, etc. Alternatively, the master container node 62 can identify another suitable agent node 32 and initiate a new control process pod via the agent node 32, thereby replacing the failed component and maintaining the availability of the control system of the corresponding OT device 67 / 71.

[0102] Although only certain features of the presently disclosed embodiments are shown and described herein, many modifications and changes will occur to those skilled in the art. It is therefore to be understood that the appended claims are intended to cover all such modifications and changes that fall within the true spirit of the embodiments described herein.

Claims

1. A system comprising: Multiple control systems for controlling multiple operations of multiple operational technology (OT) devices; A first computing node in a computing node cluster, the computing node being part of a container orchestration system, wherein the first computing node is configured to: receiving update data for a first control system among the plurality of control systems, a first OT device among the plurality of OT devices, or both the first control system and the first OT device, wherein the first control system, the first OT device, or both the first control system and the first OT device are configured to perform one or more operations, wherein the update data is configured to update one or more software components executed by the first control system, the first OT device, or both the first control system and the first OT device; retrieving, via a portion of the cluster of computing nodes, a plurality of machine state data sets from the plurality of control systems, wherein each of the plurality of machine state data sets corresponds to an operational state associated with each of the plurality of control systems, each of the plurality of OT devices, or both each of the plurality of control systems and each of the plurality of OT devices; identifying, based on the plurality of machine status data sets, a second control system as a suitable host for performing the one or more operations in place of the first control system; storing a first set of containers in a first file system accessible to a second computing node, wherein the first set of containers is configured to cause the second computing node to suspend the one or more operations; and A second set of containers is stored in a second file system accessible to a third computing node associated with the second control system, wherein the second set of containers is configured to cause the third computing node to perform the one or more operations via the second control system, a second OT device associated with the second control system, or both the second control system and the second OT device.

2. The system according to claim 1, wherein: The first computing node is configured to coordinate an automated process for managing or scheduling deployment of a plurality of containers for execution across the plurality of control systems via the cluster of computing nodes.

3. The system according to claim 2, wherein: Each container of the plurality of containers includes one or more applications and one or more runtime dependencies associated with each container of the plurality of containers.

4. The system according to claim 1, wherein: The first control system, the first OT device, or both the first control system and the first OT device are configured to update the one or more software components using the update data when the first control system, the first OT device, or both the first control system and the first OT device are offline.

5. The system according to claim 1, wherein: The first computing node is configured to identify the second control system as the suitable host based on the plurality of machine state data sets compared to a plurality of expected machine states.

6. The system according to claim 1, wherein: The second control system, the second OT device, or both the second control system and the second OT device correspond to a redundant control system, a redundant OT device, or both for performing the one or more operations.

7. The system according to claim 1, wherein: The first computing node is configured to transmit the first set of containers and the second set of containers during a runtime environment.

8. The system according to claim 7, wherein: The first computing node is configured to: transmitting the first set of containers directly to the second computing node; and The second set of containers is transmitted directly to the third computing node.

9. A method comprising: Receiving, via a first computing node in a computing node cluster in a container orchestration system, a first set of containers from a second computing node in the computing node cluster; retrieving, via the first computing node, a first image file comprising a first set of containers from a registry based on the first set of containers, wherein the first set of containers comprises an indication of a location of the first image file in the registry, and wherein the first set of containers is configured to cause a control system of a plurality of control systems to suspend one or more operations; generating, via the first computing node, a first packet based on the first set of containers; storing, via the first computing node, the first packet in a file system shared with the control system; receiving, via the first computing node, a second set of containers from the second computing node; retrieving, via the first computing node, a second image file including a second set of containers from the registry based on the second set of containers, wherein the second set of containers includes a second indication of a second location of the second image file in the registry, and wherein the second set of containers is configured to cause the control system to update one or more software components; generating, via the first computing node, a second package based on the second set of containers; and The second package is stored in the file system via the first computing node.

10. The method of claim 9, comprising sending the first packet and the second packet directly to the control system via an operational technology (OT) communication protocol.

11. The method according to claim 10, wherein: The first set of containers and the second set of containers are received via an information technology (IT) communication protocol.

12. The method according to claim 9, comprising: receiving, via the first computing node, a third set of containers from the second computing node after the one or more software components are updated; retrieving, via the first computing node, from the registry a third image file including a third set of containers based on the third set of containers, wherein the third set of containers includes a third indication of a third location of the third image file in the registry, and wherein the third set of containers is configured to cause the control system to resume the one or more operations; generating, via the first computing node, a third packet based on the third set of containers; and The third package is stored in the file system via the first computing node.

13. The method according to claim 12, comprising: Prior to receiving the third set of containers, machine state data associated with the control system, an operational technology (OT) device communicatively coupled to the control system, or both the control system and the OT device is sent to the second computing node.

14. The method according to claim 9, wherein: The one or more operations are performed by the control system, an operations technology (OT) device communicatively coupled to the control system, or both the control system and the OT device.

15. The method according to claim 9, comprising: When the control system is operating in a runtime environment, the first packet and the second packet are sent directly to the control system.

16. A non-transitory computer readable medium comprising computer executable instructions which, when executed, are configured to cause a processor to perform operations comprising: receiving update data for a first control system among a plurality of control systems, a first operational technology (OT) device among a plurality of OT devices, or both the first control system and the first OT device, wherein: The first control system, the first OT device, or both the first control system and the first OT device are configured to perform one or more operations, wherein the update data is configured to update one or more software components executed by the first control system, the first OT device, or both the first control system and the first OT device; retrieving, via a portion of a cluster of computing nodes that is part of a container orchestration system, a plurality of machine state data sets from the plurality of control systems, wherein each of the plurality of machine state data sets corresponds to an operational state associated with each of the plurality of control systems, each of the plurality of OT devices, or both each of the plurality of control systems and each of the plurality of OT devices; identifying, based on the plurality of machine status data sets, a second control system among the plurality of control systems as a suitable host to perform the one or more operations in place of the first control system; generating a first container set and storing the first container set in a first file system accessible to a first computing node in the computing node cluster, wherein the first container set is configured to cause the first computing node to suspend the one or more operations; and A second set of containers is generated and stored in a second file system accessible to a second computing node in the computing node cluster, wherein the second computing node is associated with the second control system, wherein the second set of containers is configured to cause the second computing node to perform the one or more operations via the second control system, a second OT device associated with the second control system, or both the second control system and the second OT device.

17. The non-transitory computer readable medium of claim 16, wherein: Each machine state data set of the plurality of machine state data sets corresponds to an amount of computing resources available on a corresponding control system of the plurality of control systems.

18. The non-transitory computer readable medium of claim 16, wherein: The computer-executable instructions are configured to cause the processor to perform operations including generating a third set of containers and storing the third set of containers in the first file system, wherein the third set of containers is configured to cause the first computing node to update the one or more software components.

19. The non-transitory computer readable medium of claim 18, wherein: The computer executable instructions are configured to cause the processor to perform operations including: receiving an indication that the one or more software components are updated; A fourth container set is generated and stored in the second file system, wherein the fourth container set is configured to cause the second computing node to suspend the one or more operations.

20. The non-transitory computer readable medium of claim 19, wherein: The computer-executable instructions are configured to cause the processor to perform operations including generating a fifth set of containers and storing the fifth set of containers in the first file system, wherein the fifth set of containers is configured to cause the first computing node to resume the one or more operations.

Images