Preventive controller switching

Abstract

The present invention relates to performing a preventive switchover from a primary controller to a secondary controller even before the primary controller fails. A server collects log files in real-time from the primary controller including operational parameters of the primary controller. Further, the server determines abnormal patterns or features in the operational parameters of the primary controller by comparing the operational parameters with reference patterns or features. The reference patterns or features are generated by training one or more artificial intelligence (AI) based models. Upon determining the abnormal patterns or features, the server predicts an event that will lead to a switchover from the primary controller to the secondary controller. Thereafter, the server provides a signal to the primary controller to perform a preventive switchover to the secondary controller before the primary controller fails. Thus, preventive measures are taken so that no abnormality occurs in the secondary controller when it takes over.

Description

TECHNICAL FIELD

[0001] The present invention relates generally to controller redundancy in process / industrial facilities. More specifically, the present invention relates to predicting a master controller failure event and performing a switchover to a secondary controller. BACKGROUND

[0002] Process / industrial facilities include a plurality of devices such as pumps, drives, air compressors, machinery, electrical appliances, and the like. The industrial devices have to be operated continuously to ensure maximum output of the process / industrial facility. Current technology enables the devices to be operated continuously to reduce downtime of the device operation. Typically, the devices are controlled by a controller and due to controller failure, the device operation stops. Typically, a redundant controller is configured as a hot standby controller which takes over the control of the device when the master controller fails. The redundant controller can store all the configuration settings and parameters related to operating the device when the switchover from the master controller takes place. Thus, the redundant controller facilitates smooth operation of the process / industrial facility.

[0003] In the existing controller redundancy systems, the controller switchover takes place after the master controller has failed. However, the cause of the master controller failure can also occur in the redundant controller, thereby causing the redundant controller to fail. Further, the switchover takes place at a critical time when the master controller has actually failed. Many times, the master controller needs to transfer the state and operational parameters of the master controller to the redundant controller. However, when the master controller fails, the master controller is unable to transfer all the required data to the redundant controller to effectively control the device. Further, the delay in the switchover from the master controller to the redundant controller during the critical time can increase the operational failure.

[0004] Therefore, there is a need to provide a preventive switchover from the master controller to the redundant controller before the master controller fails. SUMMARY

[0005] In one embodiment, the present invention relates to a method, server, and system for performing redundant switching in a process / industrial facility. In one embodiment, the process / industrial facility includes a primary controller and a secondary controller. The primary controller is configured to operate one or more devices. The secondary controller is redundant with the primary controller. The primary controller and the secondary controller are connected to a server. The server is configured to receive a log file from the primary controller in real time, wherein the log file includes operating parameters of the primary controller. In addition, the server determines an abnormal pattern or feature in the operating parameters of the primary controller by comparing the operating parameters with a reference pattern or feature. In one embodiment, the reference pattern or feature is generated based on one or more trained models. The server also predicts one or more events that lead to switching from the primary controller to the secondary controller based on the abnormal pattern or feature in the operating parameters. Thereafter, the server provides a signal to the primary controller based on the predicted one or more events to perform switching from the primary controller to the secondary controller. Therefore, switching occurs even before the one or more events occur, thereby achieving preventive controller switching.

[0006] In one embodiment, the operating parameters include at least one of hardware parameters, software parameters, firmware parameters, and network parameters.

[0007] In one embodiment, the one or more trained models are artificial intelligence (AI) based models.

[0008] In one embodiment, the server generates a reference pattern or signature by performing the following steps. The server receives a training log file including historical operating parameters of the primary controller. In addition, the server uses the historical operating parameters to detect one or more events that lead to a switch from the primary controller to the secondary controller. In addition, the server identifies patterns or signatures in the historical parameters in the training log file that correspond to the one or more events. Thereafter, the server classifies normal patterns or signatures and abnormal patterns or signatures based on the identified patterns or signatures, wherein the classified patterns are stored as reference patterns. In one embodiment, one or more trained models determine abnormal patterns or signatures and predict one or more events in real time.

[0009] In one embodiment, the predicted one or more events are displayed on a display unit.

[0010] Systems of varying scope are described herein. In addition to the aspects and advantages described in this summary, further aspects and advantages will become apparent by reference to the drawings and by reference to the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The subject matter of the invention will be explained in more detail hereinafter with reference to preferred exemplary embodiments shown in the accompanying drawings, in which:

[0012] Figure 1 Controller redundancy in a process / industrial facility according to some embodiments of the present invention is illustrated;

[0013] Figure 2 illustrates the internal architecture of a server for performing preventive switching in a process / industrial facility according to some embodiments of the present invention;

[0014] FIG3 illustrates the training and inference phases of a model for generating reference patterns or features according to some embodiments of the present invention;

[0015] Figure 3b shows an example diagram of generating reference patterns or features according to some embodiments of the present invention;

[0016] Figure 4a An exemplary flow chart for performing redundancy switching in a process / industrial facility according to some embodiments of the present invention is shown;

[0017] Figure 4b An example diagram illustrating predicting events in a master controller according to some embodiments of the present invention is shown;

[0018] Figure 5 An exemplary scenario illustrating normal operation of a master controller in a process / industrial facility according to some embodiments of the present invention; and

[0019] Figure 6 Exemplary scenarios of abnormal operation of a master controller in a process / industrial facility according to some embodiments of the present invention are illustrated. DETAILED DESCRIPTION

[0020] Embodiments of the present invention relate to performing a preventive switch from a primary controller to a secondary controller even before the primary controller fails. A server collects a log file including operating parameters of the primary controller from the primary controller in real time. In addition, the server determines an abnormal pattern or signature in the operating parameters of the primary controller by comparing the operating parameters with a reference pattern or signature. The reference pattern or signature is generated by training one or more artificial intelligence (AI) based models. After determining the abnormal pattern or signature, the server predicts an event that will result in a switch from the primary controller to the secondary controller. Thereafter, the server provides a signal to the primary controller to perform a preventive switch to the secondary controller before the primary controller fails. Therefore, when the secondary controller takes over, the abnormality in the operating parameters is notified and preventive measures are taken so that the abnormality does not occur in the secondary controller.

[0021] Figure 1Controller redundancy in a process / industrial facility is shown. The process / industrial facility includes one or more devices (104). Examples of devices include, but are not limited to, motors, drives, pumps, machinery, etc. The one or more devices (104) are controlled by a master controller (102a). In an example embodiment, the master controller (102a) can be connected to the one or more devices (104) via an industry standard communication protocol (examples include, but are not limited to, Ethernet, RS232, and RS485). Further, the master controller (102a) can be connected to a server (101) via a network (103). The network (103) can support Ethernet LAN, WAN, Wi-Fi, etc. In one embodiment, the server (101) can be part of a Distributed Control System (DCS) or a Supervisory Control and Data Acquisition (SCADA) system, or a standalone system. In one embodiment, the server (101) is configured to acquire data of the process / industrial facility and perform analysis on the acquired data. For example, the server (101) can acquire data of the process, the one or more devices (104), and the master controller (102a). In one embodiment, the server (101) can be replaced by a personal computer, a laptop, a mobile device, or any other electronic device capable of analyzing a plurality of operating parameters of the master controller (102a) and predicting a fault condition in the master controller (102a).

[0022] In an embodiment, the secondary controller (102a) is commissioned as a redundant controller in the process / industrial facility. To reduce the downtime due to failure of the primary controller (102a), the secondary controller (102b) is used as a redundant controller. In an embodiment, the secondary controller (102b) is connected to one or more devices (104) and the server (101). In an embodiment, the primary controller (102a) and the secondary controller (102b) can be connected via a redundant link. The redundant link can support an Ethernet protocol. Other protocols known in the art can also be used as the redundant link. Typically, when the primary controller (102a) fails, a switchover occurs from the primary controller (102a) to the secondary controller (102b). The primary controller (102a) initiates the redundant switchover to the secondary controller (102b). Typically, the primary controller (102a) transmits a plurality of operational parameters and the state of one or more devices to the secondary controller at the time of initiating the redundant switchover. Thus, when the secondary controller (102b) takes over, the operation of the process / industrial facility can be resumed from where the primary controller (102a) failed. In an embodiment, the cause of failure of the primary controller (102a) can include, but is not limited to, a hardware failure, a software failure, a firmware failure, and a network failure. In an embodiment, the secondary controller (102b) can be connected to the server (101) via a different network (103) than the network used to connect the primary controller (102a) to the server (101). Thus, when the primary controller (102a) fails due to a network failure, the secondary controller (102b) is not affected by the network failure.

[0023] Reference is now made to Figure 2 . Figure 2The internal architecture of the server (101) is shown. The server comprises one or more processors (201), a memory (202), and an input / output (I / O) interface (203). The I / O interface (203) provides connectivity between the server (101) and the primary controller (102a) and the secondary controller (102b). The I / O interface (203) can also provide connectivity between the server (101) and the DCS or SCADA. In an embodiment, the memory (202) stores a plurality of operational parameters of the primary device (102a). In an embodiment, the memory (202) can be a database. In an embodiment, the one or more processors (201) are configured to receive the plurality of operational parameters of the primary controller (102a) in real-time. In an embodiment, the one or more processors (201) can receive the operational parameters at regular intervals. In an embodiment, the one or more parameters can be time series data. The one or more processors (201) determine abnormal patterns or features in the plurality of operational parameters by comparing the plurality of operational parameters with reference patterns or features. The reference patterns or features are generated by one or more trained models (AI models (204)). Further, the one or more models (204) predict one or more events leading to a failure of the primary controller (102a) which eventually leads to a switchover from the primary controller (102a) to the secondary controller (102b). Thereafter, the one or more processors (201) provide a signal to the primary controller (102a) to perform the switchover from the primary controller (102a) to the secondary controller (102b) based on the predicted one or more events. Thus, the switchover is performed even before the primary controller (102a) fails, which results in a preventive controller switchover. The preventive controller switchover enables a smooth switchover. Further, since the preventive controller switchover occurs before the primary controller (102a) fails, the criticality of the switchover is reduced, thereby providing a more efficient switchover to the secondary controller (102a). These advantages can be achieved considering other embodiments explained along with the examples.

[0024] Figure 3a The training and inference phase of the one or more models (204) for generating the reference patterns or features is shown. The one or more models (204) are implemented in the server (101) and the one or more models (204) are trained to generate the reference patterns or features.

[0025] At step (301), the one or more models (204) receive a training log file comprising historical operational parameters of the primary controller (102a). The training log file can be obtained from one or more historians associated with the process / industrial facility. The training log can be selected which comprises historical operational parameters leading to a switchover from the primary controller (102a) to the secondary controller (102b).

[0026] At step (302), one or more models (204) detect one or more historical events that caused a switchover from the primary controller (102a) to the secondary controller (102b). In one embodiment, a domain expert can label a small set of historical operating parameters with one or more historical events. The one or more events are failure events in the primary controller (102a). In one embodiment, a domain expert can label a small set of historical operating parameters and train one or more models (204) to label a large set of historical operating parameters. For example, a domain expert can label a first set of historical operating parameters with the label "memory failure," a second set of historical operating parameters with the label "communication port failure," a third set of historical parameters with the label "null zero failure," and a fourth set of historical operating parameters with the label "communication failure." In one embodiment, each type of controller failure (historical event) can be diagnosed and a root cause can be associated. For example, the failure "memory failure" can be associated with the root cause "insufficient memory," the failure "null zero failure" can be associated with the root cause "incorrect dynamic memory allocation," the failure "communication failure" can be associated with the root cause "incompatible firmware," and the failure "communication port failure" can be associated with the root cause "port burnout." The above examples should not be considered limiting, as other examples known to those skilled in the art can also be used with the present application.

[0027] At step (303), the one or more models (204) identify patterns or features in the historical operating parameters corresponding to the one or more historical events. In an embodiment, the one or more models (204) can include, but are not limited to, a classification model, a regression model, or any other type of model capable of determining patterns or features in the historical operating parameters. In some embodiments, a pattern can refer to a change in the historical operating parameters preceding the occurrence of the one or more historical events in the host controller (102a). For example, there can be a particular pattern in the historical operating parameters when a “memory failure” occurs. There can be a particular change in the values of the historical operating parameters that led to the “memory failure” of the host controller. The one or more models (204) can require time series data to identify the pattern. In some embodiments, features can also be identified. Not always a pattern can lead to a failure of the host controller (102a). A particular change in the historical operating parameters can also lead to the one or more historical events. Such a particular change can be considered a feature, which can not require time series data, but data at a particular time (e.g., a value of an operating parameter). In an embodiment, a domain expert can have labeled a small amount of historical operating parameters, and the one or more models (204) can use these labels to further label the historical operating parameters, which can include a vast dataset. An auto-labeller can be used to label the historical operating parameters. The auto-labeller can be an AI-based model. In an embodiment, the one or more models (204) can also determine the time taken for the failure of the host controller (102a) based on the identified patterns or features. In some embodiments, the one or more models (204) can map the time values for each of the one or more events.

[0028] At step (304), the one or more models (204) classify the identified patterns or features as normal patterns or features and abnormal patterns or features. In an embodiment, a domain expert can also label a small number of historical operating parameters with normal patterns and abnormal patterns. The one or more models (204) can be trained to label the historical operating parameters with normal patterns and abnormal patterns using the labels provided by the domain expert. In an embodiment, an abnormal pattern can be an entire region after a certain point in the time series data. In an embodiment, an abnormal feature can be a point value in the historical operating parameters. It would be clear to a person skilled in the art that abnormal features in the main controller (102a) can be observed using point values or time series data. It would also be clear to a person skilled in the art how to train and use an AI model to infer abnormal behavior of the main controller (102a). In an embodiment, a classification model can be used to classify into normal patterns and abnormal patterns. In an embodiment, the normal patterns or abnormal patterns can be stored as reference patterns. For example, in a process / industrial facility where one or more historical events are deterministic and known to the domain expert, the abnormal patterns can be stored as reference patterns. In a process / industrial facility where one or more historical events are non-deterministic and not generally known to the domain expert, the normal patterns can be stored as reference patterns. The reference patterns are compared with the real-time operating parameters of the main controller (102a) to predict one or more events. In an embodiment, the training of the one or more models (204) ends when the one or more models (204) have a defined efficiency.

[0029] Figure 3b An example diagram showing generation of reference patterns or features is shown. As shown, the one or more models (204) receive historical operating parameters as input. Optionally, the one or more models (204) can also receive labels as input. In an embodiment, the one or more models (204) can cluster different types of events and label the clusters without the need for a domain expert to provide labels as input. Further, the one or more models (204) generate reference patterns or features using the historical operating parameters as described in Figure 3a

[0030] Figure 4a An example flow diagram for performing redundant switchover in a process / industrial facility is shown. The following steps are performed in real-time.

[0031] ​At step (401), the server (101) receives, in real-time, log files comprising operational parameters of the primary controller (102a). In an embodiment, the operational parameters can be received at regular time intervals (e.g., every 5 minutes). In an embodiment, the operational parameters are received during normal operation of the primary controller (102a). While the primary controller (102a) is normally operating, the secondary controller (102b) is in a hot standby state. The operational parameters include hardware parameters, software parameters, firmware parameters, and network parameters.

[0032] At step (402), the server (101) determines abnormal patterns or features in the operational parameters by comparing the operational parameters to reference patterns or features. In an embodiment, the server (101) can compare the operational parameters to reference abnormal patterns or features or to reference normal patterns or features. When comparing the operational parameters to normal patterns or features, an abnormal pattern or feature in the operational parameters is determined when the value of the operational parameter deviates from the normal pattern or feature. When comparing the operational parameters to abnormal patterns or features, an abnormal pattern or feature in the operational parameters is determined when the value of the operational parameter matches the abnormal pattern or feature.

[0033] At step (403), the server (101) predicts one or more events that lead to a switchover from the primary controller (102a) to the secondary controller (102b). The one or more events are events that lead to a failure of the primary controller (102a). In an embodiment, one or more trained models are used to predict the one or more events. The one or more trained models can have associated events with reference patterns or features. Using the association, the one or more models (204) predict, in real-time, the one or more events. For example, the one or more models (204) can identify an abnormal pattern in the operational parameters related to a memory of the primary controller (102a) and predict that a memory failure can occur. In an embodiment, the one or more models (204) can also predict a time of the failure of the primary controller (012a). For example, the one or more models (204) can predict that the primary controller (102a) can fail 5 minutes after detecting the abnormal pattern in the operational parameters related to the memory. Figure 5 An exemplary scenario is shown in which the primary controller (102a) is normally operating and the secondary controller (102b) is in a hot standby state while the server (101) receives operational parameters of the primary controller (102a).

[0034] At step (404), the server (101) provides a signal to the primary controller (102a) to perform the switchover to the secondary controller (102b). In an embodiment, the server (101) provides the signal to the primary controller (102a) even before the primary controller (102a) fails. Thus, the switchover from the primary controller (102a) to the secondary controller (102b) occurs before the primary controller (102a) fails. Thus, the problems associated with the switchover after the primary controller (102a) fails are avoided. When the primary controller (102a) receives the signal from the server (101), the primary controller (102a) transmits its operating parameters and the status associated with the device (104). Further, the secondary controller (102b) controls the device (104) and the primary controller (102a) is inactive and can be scheduled for maintenance. In an embodiment, when the one or more events are predicted by the one or more models (204), a notification can be provided to indicate the one or more events. The operator can attend the notification before the switchover and resolve the one or more events such that the one or more events do not occur in the secondary controller (102b). In an embodiment, the operating parameters of the secondary controller (102b) can be modified to avoid the one or more events from occurring in the secondary controller (102b). For example, in case of a "memory failure", the operator can provide additional memory for the secondary controller (102b). Thus, the predicted event of "memory failure" does not occur in the secondary controller (102b).

[0035] As shown in FIG. 2, the one or more models (204) are provided with the real-time operating parameters of the primary controller (102a). Since the one or more models (204) are trained and have generated the reference patterns or features, the one or more models (204) compare the operating parameters with the reference patterns or features. Based on the comparison, the one or more models (204) predict one or more events (failure events) in the primary controller (102a) and even before the primary controller (102a) fails, a signal is provided to the primary controller (102a) to perform a preventive switchover to the secondary controller (102b). Figure 4b Thus, the signal provided by the server (101) enables a preventive switchover from the primary controller (102a) to the secondary controller (102b). In an embodiment, since the failure in the primary controller (102a) does not occur, the logs including the failure events are reduced and such logs can be closed.

[0036] The scenario is shown in which the secondary controller (102b) controls the device (104) and the primary controller (102a) is inactive. Figure 6

[0037] ​This written description uses examples to describe the subject matter in this document, including the best mode, and also to enable any person skilled in the art to make and use the subject matter. The patentable scope of the subject matter is defined by the claims, and can include other examples that occur to those skilled in the art. Such other examples are intended to be within the scope of the claims if they have structural elements in common with the words of the claims or if they include equivalent

[0038] Reference Signs:

[0039] 101: server

[0040] 102a: main controller

[0041] 102b: sub controller

[0042] 103: network

[0043] 104: device

[0044] 201: processor

[0045] 202: memory

[0046] 203: I / O interface

[0047] 204: AI model

Claims

1. A method for performing redundancy switching in a process / industrial facility, wherein the process / industrial facility includes a primary controller and a secondary controller, wherein the secondary controller is redundant with the primary controller, wherein the primary controller is configured to operate one or more devices, wherein the primary controller and the secondary controller are connected to a server, wherein the method is performed by the server, the method comprising: receiving a log file from the main controller in real time, wherein the log file includes a plurality of operating parameters of the main controller; determining abnormal patterns or signatures in the plurality of operating parameters of the master controller by comparing the plurality of operating parameters to reference patterns or signatures, wherein the reference patterns or signatures are generated based on one or more trained models; predicting one or more events that will cause a switch from the primary controller to the secondary controller based on the abnormal patterns or characteristics in the plurality of operating parameters of the primary controller; as well as performing a preventative controller switch from the primary controller to the secondary controller by providing a signal to the primary controller to perform the switch based on the predicted one or more events, Wherein generating the reference pattern or feature comprises: receiving a training log file, the training log file including historical operating parameters of the master controller; detecting one or more historical events that resulted in a switch from the primary controller to the secondary controller; identifying patterns or features in the historical operating parameters in the training log file that correspond to the one or more historical events that led to the switching; Normal patterns or features and abnormal patterns or features are classified based on the identified patterns or features, wherein the reference pattern or feature is one of the normal pattern or feature or the abnormal pattern or feature. 2 . The method of claim 1 , wherein the plurality of operating parameters comprises at least one of hardware parameters, software parameters, firmware parameters, and network parameters.

3. The method of claim 1, wherein the one or more trained models are artificial intelligence-based models. 4 . The method of claim 1 , wherein the abnormal pattern or signature is determined and the one or more events are predicted by the one or more trained models. 5 . The method of claim 1 , wherein prior to performing the switch from the primary controller to the secondary controller, operating parameters of the secondary controller are modified based on the predicted one or more events.

6. A server for performing redundancy switching in a process / industrial facility, wherein the process / industrial facility includes a primary controller and a secondary controller, wherein the secondary controller is redundant with the primary controller, wherein the primary controller is configured to operate one or more devices, wherein the primary controller and the secondary controller are connected to the server, wherein the server comprises: Memory; as well as One or more processors configured to: receiving a log file from the main controller in real time, wherein the log file includes a plurality of operating parameters of the main controller; determining abnormal patterns or signatures in the plurality of operating parameters of the master controller by comparing the plurality of operating parameters to reference patterns or signatures, wherein the reference patterns or signatures are generated based on one or more trained models; predicting one or more events that will cause a switch from the primary controller to the secondary controller based on the abnormal patterns or characteristics in the plurality of operating parameters of the primary controller; as well as performing a preventative controller switch from the primary controller to the secondary controller by providing a signal to the primary controller to perform the switch based on the predicted one or more events, Wherein, in order to generate the reference pattern or signature, the one or more processors are configured to: receiving a training log file, the training log file including historical operating parameters of the master controller; detecting one or more historical events that resulted in a switch from the primary controller to the secondary controller; identifying patterns or features in the historical operating parameters in the training log file that correspond to the one or more historical events that led to the switching; Normal patterns or features and abnormal patterns or features are classified based on the identified patterns or features, wherein the reference pattern or feature is one of the normal pattern or feature or the abnormal pattern or feature.

7. The server of claim 6, wherein the one or more trained models are artificial intelligence-based models.

8. The server of claim 6, wherein the one or more processors are associated with a display unit for displaying a notification regarding the predicted one or more events.

9. A system for performing redundancy switching in a process / industrial facility, the system comprising: a main controller configured to operate one or more devices in the process / industrial facility; a secondary controller, redundant to the primary controller; as well as The server is configured as: receiving a log file from the main controller in real time, wherein the log file includes a plurality of operating parameters of the main controller; determining abnormal patterns or signatures in the plurality of operating parameters of the master controller by comparing the plurality of operating parameters to reference patterns or signatures, wherein the reference patterns or signatures are generated based on one or more trained models; predicting one or more events that will cause a switch from the primary controller to the secondary controller based on the abnormal patterns or characteristics in the plurality of operating parameters; as well as performing a preventative controller switch from the primary controller to the secondary controller by providing a signal to the primary controller to perform the switch based on the predicted one or more events, In order to generate the reference pattern or feature, the server is configured to: receiving a training log file, the training log file including historical operating parameters of the master controller; detecting one or more historical events that resulted in a switch from the primary controller to the secondary controller; identifying patterns or features in the historical operating parameters in the training log file that correspond to the one or more historical events that led to the switching; Normal patterns or features and abnormal patterns or features are classified based on the identified patterns or features, wherein the reference pattern or feature is one of the normal pattern or feature or the abnormal pattern or feature.

10. The system of claim 9, wherein the one or more trained models are artificial intelligence based models.

11. The system of claim 9, further comprising a display unit for displaying a notification regarding the predicted one or more events.

12. The system of claim 9, wherein prior to performing the switch from the primary controller to the secondary controller, operating parameters of the secondary controller are modified based on the predicted one or more events.

Images