System and method for monitoring software application processes - Patents.com
The system with compute nodes and monitoring components addresses delays in detecting and restarting software failures, reducing downtime by continuous monitoring and automatic recovery.
Patent Information
- Application Number
- JP2019228066
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2018-12-20
- Filing Date
- 2019-12-18
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2039-12-18
AI Technical Summary
Existing systems face delays in detecting software application failures and restarting them, leading to extended downtime due to inconvenient and costly continuous monitoring by system administrators.
A system with multiple compute nodes, each equipped with process monitors, status data analyzers, and peer monitors to continuously check the status of software applications, automatically restart failed instances, and manage incomplete tasks across nodes.
Reduces downtime by promptly detecting and addressing software application failures, ensuring efficient recovery and management of incomplete tasks, thereby minimizing service disruptions.
Smart Images

Figure 0007584217000001 
Figure 0007584217000002 
Figure 0007584217000003
Abstract
Description
[Technical field]
[0001] The present invention relates generally to monitoring software application processes. [Background technology]
[0002] Computation servers may execute software applications to provide services. The software applications may fail for a variety of reasons, such as stack overflow, network or power outages, or other unexpected disruptions. There may be a delay between the software application failing and a system administrator determining that the software application has failed. There may also be a delay between the system administrator detecting that the software application has failed and the system administrator restarting the software application. The delay in restarting the software application extends the downtime during which the service is unavailable. Continuous monitoring of the software applications by system administrators may be inconvenient and expensive. Summary of the Invention
[0003] In one particular implementation, a system for monitoring a process includes a first computing node of a plurality of computing nodes configured to execute a plurality of instances of a software application. The system also includes a second computing node of the plurality of computing nodes. The first computing node includes a process monitor, a status data analyzer, a process monitor checker, and a peer monitor. The process monitor is configured to generate process status data for the first instance of the software application executing on the first computing node. The status data analyzer is configured to determine, based on the process status data, whether a scheduled process of the first instance of the software application has stopped executing on the first computing node. The status data analyzer is configured to selectively restart the first instance of the software application based on a determination that the scheduled process has stopped executing on the first computing node. The process monitor checker is configured to determine whether a process monitor is executing on the first computing node. The process monitor checker is also configured to restart the process monitor in response to determining that the process monitor has stopped executing on the first computing node. The peer monitor is configured to determine whether a second instance of the software application has failed on a second compute node of the plurality of compute nodes. The peer monitor is also configured to perform an action based on an incomplete task associated with the second instance of the software application in response to determining that the second instance has failed on the second compute node.
[0004] In another particular implementation, a method of monitoring a process includes generating, at a first computing node of a plurality of computing nodes configured to execute a plurality of instances of a software application, process status data related to a first instance of the software application executing on the first computing node. The method also includes determining whether a scheduled process of the first instance of the software application has stopped executing on the first computing node based on the process status data. The method further includes selectively restarting the first instance of the software application on the first computing node based on determining that the scheduled process has stopped executing on the first computing node. The method also includes determining whether a second instance of the software application has failed on a second computing node of the plurality of computing nodes. The method further includes performing an action based on an incomplete task associated with the second instance of the software application in response to determining that the second instance has failed on the second computing node.
[0005] In another particular implementation, a computer-readable storage device stores instructions that, when executed by a processor, cause the processor to perform operations including generating, at a first computing node of a plurality of computing nodes configured to execute a plurality of instances of the software application, process status data related to a first instance of the software application executing on the first computing node. The operations also include determining whether a scheduled process of the first instance of the software application has stopped executing on the first computing node based on the process status data. The operations further include selectively restarting the first instance of the software application on the first computing node based on determining that the scheduled process has stopped executing on the first computing node. The operations also include determining whether a second instance of the software application has failed on a second computing node of the plurality of computing nodes. The operations further include performing an action based on an incomplete task associated with the second instance of the software application in response to determining that the second instance has failed on the second computing node.
[0006] The features, functions, and advantages described herein can be realized alone in various implementations or can be combined in other implementations, further details of which can be understood with reference to the following description and accompanying drawings. [Brief description of the drawings]
[0007] [Figure 1] FIG. 1 is a block diagram illustrating a system operable to perform process monitoring of software applications. [Diagram 2] FIG. 1 is a flow diagram illustrating an embodiment of a method for monitoring the processes of a software application. [Diagram 3] FIG. 1 is a flow diagram illustrating an embodiment of a method for monitoring the processes of a software application. [Figure 4]FIG. 1 is a flow diagram illustrating an embodiment of a method for monitoring the processes of a software application. [Figure 5A] FIG. 2 is a diagram illustrating one embodiment of a status data analyzer and memory of the system of FIG. 1. [Figure 5B] 2 is a flow diagram illustrating one embodiment of a method performed by a status data analyzer of the system of FIG. 1. [Figure 6] FIG. 1 is a flow diagram illustrating an embodiment of a method for monitoring the processes of a software application. [Figure 7] FIG. 1 is a flow diagram illustrating an embodiment of a method for monitoring the processes of a software application. [Figure 8] FIG. 1 is a block diagram illustrating a computational environment including a computing device configured to support aspects of computer-implementable methods and computer-executable program instructions (or codes) according to the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0008] Implementations described herein are directed to monitoring processes of a software application. For example, a system includes a plurality of computing nodes configured to execute a plurality of instances of the software application. For example, a first computing node is configured to execute a first instance of the software application, and a second computing node is configured to execute a second instance of the software application. Each of the plurality of computing nodes includes a service group configured to monitor whether the software application is running as expected on the node. For example, the service group of the first computing node includes a process monitor, a process monitor checker, or both. The process monitor generates process status data indicative of processes of the software application running on the first computing node. The process monitor provides the process status data to a status data analyzer of the first computing node. The process monitor checker restarts the process monitor in response to determining that the process monitor has stopped running on the first computing node.
[0009] One or more components of the service group are configured to monitor one or more other components of the service group. In one particular embodiment, the process monitor and the process monitor checker are configured to monitor each other. Specifically, the process monitor restarts the process monitor checker in response to determining that the process monitor checker is not running (e.g., has stopped running) on the first compute node, and the process monitor checker restarts the process monitor in response to determining that the process monitor is not running (e.g., has stopped running) on the first compute node. In this embodiment, either the process monitor or the process monitor checker that is not running (e.g., has stopped running) can be restarted by the other of the process monitor or the process monitor checker. In this manner, the service group provides multiple levels of checking to provide close monitoring of processes on the first compute node.
[0010] The status data analyzer is configured to perform one or more actions based on the analysis of the process status data, such as generating a notification, restarting the software application, restarting a process of the software application, preventing the software application from restarting, shutting down a service group, updating log data, cleaning up temporary files, or combinations thereof. The status data analyzer has access to a list of processes to monitor. Specifically, the monitored processes are scheduled to execute when the software application is running normally. The status data analyzer performs one or more actions in response to determining that the process status data indicates that a scheduled process has stopped executing. In some embodiments, the status data analyzer performs different actions for different types of scheduled processes. In one particular embodiment, the status data analyzer restarts a first instance of the software application in response to determining that a first scheduled process of a first process type is not executing (e.g., has stopped executing). As another embodiment, the status data analyzer starts (or restarts) a second scheduled process in response to determining that a second scheduled process of a second process type is not executing (e.g., has stopped executing). In this manner, the status data analyzer reduces the downtime associated with the first instance of the software application.
[0011] Some of the plurality of computing nodes also include a peer alert service group configured to monitor whether the software application is running as expected on the other nodes. The peer alert service group includes a peer monitor, a peer monitor checker, or both. The peer monitor accesses cluster process data indicative of processes of the software application running on the plurality of computing nodes. In one embodiment, a first computing node includes the peer alert service group, and the peer monitor provides the cluster process data to a status data analyzer of the first computing node. The peer monitor checker restarts the peer monitor in response to determining that the peer monitor is not running (e.g., has stopped running) on the first computing node.
[0012] One or more components of the peer alert service group are configured to monitor one or more components of the peer alert service group. In one particular embodiment, the peer monitor and the peer monitor checker are configured to monitor each other. Specifically, the peer monitor restarts the peer monitor checker in response to determining that the peer monitor checker is not running (e.g., has stopped running) on the first compute node, and the peer monitor checker restarts the peer monitor in response to determining that the peer monitor is not running (e.g., has stopped running) on the first compute node. In this embodiment, either the peer monitor or the peer monitor checker that is not running (e.g., has stopped running) can be restarted by the other of the peer monitor or the peer monitor checker. In this manner, the peer alert service group provides multiple levels of checking to provide close monitoring of processes on multiple compute nodes.
[0013] The peer monitor is configured to notify the cluster managing device, one or more other computing nodes, or a combination thereof, based on the analysis of the cluster process data. For example, the peer monitor notifies the cluster managing device, one or more other computing nodes, or a combination thereof, in response to determining that the cluster process data indicates that the second instance of the software application has failed on the second computing node. The peer monitor (or cluster managing device) designates a computing node to handle any incomplete tasks associated with the second instance of the software application. For example, the peer monitor (or cluster managing device) identifies, based on the cluster process data, that a first computing node has the fewest running processes among a plurality of computing nodes having active (e.g., non-failed) instances of the software application. The peer monitor (or cluster managing device) designates the first computing node to handle any incomplete tasks associated with the second instance of the software application in response to identifying the first computing node as running the fewest processes.
[0014] The peer monitor, in response to determining that the first computing node is designated, accesses log data associated with the second node. For example, the system includes a storage device (e.g., a network storage device or other shared storage) that maintains log data for the multiple computing nodes. The peer monitor identifies one or more incomplete tasks based on the log data. The peer monitor performs an action associated with the incomplete task. For example, the peer monitor undoes the incomplete task (e.g., rolls back an incomplete database update) or completes the incomplete task (e.g., performs a database update). In this manner, the peer monitor enables the computing nodes to reach a valid state of the software application.
[0015] 1 is a block diagram of a system 100 operable to monitor processes of a software application. The system 100 includes a plurality of computing nodes. For example, the system 100 includes a computing node 102 coupled to a computing node 104, a computing node 106, a storage device 108, a cluster management device 110, a device 112, or a combination thereof via a network 190. In a particular embodiment, the cluster management device 110 includes a physical device, a virtual device, a process running on the computing node 102, a process running on the computing node 104, a process running on the computing node 106, or a combination thereof. In a particular embodiment, the computing node 102, the computing node 104, or the computing node 106 includes the cluster management device 110. For example, one of the computing nodes 102, the computing node 104, or the computing node 106 is designated as the cluster management device 110.
[0016] It should be understood that the system 100 including three computing nodes is described as an example. In other examples, the system 100 may include fewer than three computing nodes or more than three computing nodes. In one particular aspect, the number of computing nodes in the system 100 changes dynamically as various computing nodes move in and out of the coverage area of the network 190. One or more of the computing nodes 102, 104, 106, and the devices 112 include, as an example, a user device, a mobile device, a fixed location device, a communication device, a computing device, a server, a desktop, a network node, a virtual device, or a combination thereof. In one particular aspect, the devices 112 include an automated device, a mobile robot, a mobile robotic platform, or a combination thereof.
[0017] Network 190 includes a wired network, a wireless network, or both. Storage device 108, as an example, includes a network storage device, a shared storage device, a distributed storage system, a data server, or a combination thereof. Cluster management device 110, as an example, includes a server, a mobile device, a fixed location device, a communication device, a computing device, a desktop, a network node, a virtual device, or a combination thereof. Cluster management device 110 is configured to execute a cluster management application. The cluster management application is configured to generate cluster process data 143 indicative of the status of processes running on one or more of compute nodes 102, 104, and 106.
[0018] The compute node 102 includes a memory 132 coupled to a processor 134. The processor 134 includes a single-core processor or a multi-core processor. In one particular aspect, the processor 134 includes multiple processors. In one particular aspect, the processor 134 is configured to implement multi-threaded program execution. For example, a single-core processor is configured to appear to implement multi-threaded program execution by interleaving execution of instructions of multiple threads. A multi-core processor is configured to execute multiple threads simultaneously. In one particular aspect, the processor 134 includes an application server configured to execute a first instance 160 of the software application 122.
[0019] Computing node 102 includes a service group including process monitor 140, process monitor checker 142, or a combination thereof. Process monitor 140 includes process monitor checker verifier 144. Process monitor 140 is configured to generate process status data 105 indicative of processes 124 executing on processor 134 (e.g., by executing a process status (ps) command). In one particular aspect, process status data 105 indicative of a status of a subset of processes 124 associated with a first instance 160 of software application 122 executing on the processes 124. Process monitor checker 142 is configured to determine whether process monitor 140 is executing on computing node 102. In response to determining that process monitor 140 is not executing on computing node 102, process monitor checker 142 is configured to start (e.g., restart) process monitor 140. In one particular aspect, process monitor checker 142 is configured to determine whether process monitor 140 has stopped executing on computing node 102. For example, process monitor checker 142 is configured to determine whether process monitor 140 was previously running but is no longer running on compute node 102. In such an aspect, process monitor checker 142 is configured to restart process monitor 140 in response to determining that process monitor 140 has stopped running on compute node 102.
[0020] The process monitor checker verifier 144 is configured to determine whether the process monitor checker 142 is running on the compute node 102. The process monitor checker verifier 144 is configured to start (e.g., restart) the process monitor checker 142 in response to determining that the process monitor checker 142 is not running on the compute node 102. In one particular aspect, the process monitor checker verifier 144 is configured to determine whether the process monitor checker 142 has stopped running on the compute node 102. For example, the process monitor checker verifier 144 is configured to determine whether the process monitor checker 142 was previously running on the compute node 102 but is no longer running. In such an aspect, the process monitor checker verifier 144 is configured to restart the process monitor checker 142 in response to determining that the process monitor checker 142 has stopped running on the compute node 102. In this manner, the process monitor checker 142 and the process monitor 140 are configured to monitor each other.
[0021] It should be understood that a service group including two levels of checks (e.g., process monitor 140 and process monitor checker 142) is described as one embodiment, in which a service group includes more than two levels of checks with one or more components of the service group monitoring one or more other components of the service group.
[0022] Compute node 102 includes a status data analyzer 146. Status data analyzer 146 is configured to perform one or more actions based on an analysis of process status data 105, such as generating a notification, restarting first instance 160 of software application 122, starting a process of software application 122, preventing restart of first instance 160 of software application 122, shutting down a service group (e.g., process monitor 140, process monitor checker 142, or both), updating log data 131, or a combination thereof.
[0023] In one particular implementation, the compute node 102 includes a peer alert service group. The peer alert service group includes a peer monitor 150, a peer monitor checker 152, or both. The peer monitor 150 includes a peer monitor checker verifier 154. The peer monitor 150 is configured to receive cluster process data 143 from the cluster management device 110. The peer monitor checker 152 is configured to start (e.g., restart) the peer monitor 150 in response to determining that the peer monitor 150 is not running on the compute node 102. The peer monitor checker verifier 154 is configured to start (e.g., restart) the peer monitor checker 152 in response to determining that the peer monitor checker 152 is not running on the compute node 102. In one particular implementation, the peer monitor checker 152 is configured to restart the peer monitor 150 in response to determining that the peer monitor 150 has stopped running on the compute node 102. The peer monitor checker verifier 154 is configured to restart the peer monitor checker 152 in response to determining that the peer monitor checker 152 has stopped executing on the compute node 102. In this manner, the peer monitor checker 152 and the peer monitor 150 are configured to monitor each other.
[0024] It should be appreciated that a peer alert service group including two levels of checks (e.g., peer monitor 150 and peer monitor checker 152) is described as one embodiment, in which the peer alert service group includes more than two levels of checks with one or more components of the peer alert service group monitoring one or more other components of the peer alert service group.
[0025] Peer monitor 150 is configured to determine, based on analysis of cluster process data 143, whether second instance 162 of software application 122 has failed on compute node 104. In response to determining that second instance 162 of software application 122 has failed on compute node 104, peer monitor 150 is configured to notify cluster managing device 110, one or more other compute nodes, or a combination thereof. In one particular aspect, peer monitor 150 is configured to designate computing node 102 or computing node 106 as cluster managing device 110 in response to determining that computing node 104 corresponds to (e.g., includes) cluster managing device 110. For example, peer monitor 150 starts an instance of a cluster managing application on computing node 102 or sends a request to compute node 106 to start an instance of a cluster managing application. Peer monitor 150 is configured to, in response to determining that compute node 102 is designated to handle any incomplete tasks of the second instance, access log data 141 associated with compute node 104. Peer monitor 150 is configured to, in response to determining that log data 141 indicates one or more incomplete tasks 109 associated with second instance 162 of software application 122, perform action 128 associated with incomplete task 109.
[0026] In an active / passive configuration, one computing node is maintained in an idle (e.g., passive) state as a backup computing node in case another computing node running an instance of software application 122 fails. When a failure of software application 122 is detected, an instance of software application 122 is started on the backup computing node to take over the operation of the failed computing node. In an active / active configuration, resources are used more efficiently by running instances of software application 122 on all available computing nodes. In FIG. 1, cluster management device 110, computing node 102, computing node 104, computing node 106, or a combination thereof, are set up to support an active / active configuration. For example, cluster management device 110 is configured to start instances of software application 122 on computing node 102, computing node 104, and computing node 106 (e.g., all available computing nodes of system 100). Compute node 102 is configured to address incomplete tasks 109 associated with compute node 104 in response to detecting a failure of software application 122 on compute node 104. Cluster management device 110 is configured to select one of the other compute nodes (e.g., an already active compute node) in response to detecting a failure of software application 122 on compute node 104 and update the load balancer to redirect future traffic intended for compute node 104 to the selected compute node (e.g., compute node 106). The active / active configuration increases efficiency by making all available compute nodes active and allowing recovery to a valid state by addressing incomplete tasks 109.
[0027] Computing node 102 includes a display interface 126, a network interface 130, or both. In Figure 1, display interface 126 is coupled to display 114. Network interface 130 is coupled to network 190. In a particular aspect, network interface 130 includes a network interface controller, a network interface device, a network socket, a network interface card, a serial network interface, a wireless interface, or a combination thereof.
[0028] The memory 132 is configured to store scheduled process data indicative of a scheduled process associated with the software application 122. For example, the data may be indicative of scheduled process 107, scheduled process 113, or both. The memory 132 is also configured to store application restart criteria 111. In one particular aspect, the scheduled process data, the application restart criteria 111, or both are based on user input, default data, configuration settings, or a combination thereof. The memory 132 is configured to store instructions 125. The instructions 125, when executed by the processor 134, perform one or more operations described herein.
[0029] It should be noted that in the following description, various functions performed by the system 100 of FIG. 1 are described as being performed by particular components or modules. However, such division of components and modules is for illustrative purposes only. In alternative aspects, functions described herein as being performed by particular components or modules are divided into multiple components or modules. Furthermore, in alternative aspects, two or more components or modules of FIG. 1 are integrated into a single component or module. Each component or module illustrated in FIG. 1 may be implemented using hardware (e.g., a field programmable gate array (FPGA) device, an application specific integrated circuit (ASIC), a digital signal processor (DSP), a controller, etc.), software (e.g., instructions executable by a processor), or any combination thereof.
[0030] During operation, the cluster management device 110 starts instances and service groups of the software application 122 on one or more nodes of the system 100. For example, the cluster management device 110 starts instances and service groups of the software application 122 in response to user input from a system administrator. Specifically, the cluster management device 110 sends a first request to the compute node 102. The first request indicates that an instance of the software application 122 and a corresponding service group should be started on the compute node 102. In response to receiving the first request, the status data analyzer 146 starts (e.g., starts) the first instance 160 of the software application 122 on the compute node 102. In response to receiving the first request, the status data analyzer 146 also starts (e.g., starts) the service groups (e.g., the process monitor 140, the process monitor checker 142, or both) on the compute node 102. It should be understood that two components of a service group are provided as one example. In other embodiments, a service group includes fewer than two components or more than two components. In some embodiments, two or more components of a service group are combined into a single component. In some embodiments, functionality described herein that is performed by a particular component of a service group is performed by multiple components of a service group.
[0031] In a particular embodiment, the cluster managing device 110 selects at least one computing node to monitor other computing nodes. For example, the cluster managing device 110 selects the computing node 102 based on the cluster process data 143. In a particular implementation, the cluster managing device 110 receives the process status data 105 from the computing node 102, the second process status data from the computing node 104, the process status data 165 from the computing node 106, or a combination thereof. For example, the cluster managing device 110 receives the process status data 105, the second process status data, and the process status data 165 in response to sending the first request to the computing node 102, the second request to the computing node 104, and the request 163 to the computing node 106. The cluster managing device 110 generates the cluster process data 143 based on the process status data 105, the second process status data, the process status data 165, or a combination thereof. The cluster process data 143 indicates a first number of a subset of processes 124 associated with the software applications 122 executing on the compute node 102. The cluster managing device 110 selects the compute node 102 in response to determining that the first number of the subset of processes 124 is less than or equal to a second number of processes executing on the compute node 104, a third number of processes executing on the compute node 106, or both. In one particular implementation, the cluster managing device 110 selects the compute node 102 independent of the cluster process data 143. For example, the cluster managing device 110 selects the compute node 102 based on a round robin selection, user input, default data, configuration settings, or a combination thereof.
[0032] The cluster management device 110 sends the second request to the compute node 102 (e.g., the selected compute node). In response to receiving the second request, the status data analyzer 146 initiates (e.g., starts) a peer service group (e.g., a peer monitor 150, a peer monitor checker 152, or both) on the compute node 102.
[0033] In one particular embodiment, the first thread, the second thread, the third thread, the fourth thread, and the fifth thread of the processor 134 correspond to the first instance 160 of the software application 122, the process monitor 140, the process monitor checker 142, the peer monitor 150, and the peer monitor checker 152, respectively. It should be understood that two components of the service group are provided as one example. In other embodiments, the service group includes fewer than two components or more than two components. In some embodiments, two or more components of the service group are combined into a single component. In some embodiments, the functionality described herein that is performed by a particular component of the service group is performed by multiple components of the service group.
[0034] The process monitor checker verifier 144 determines whether the process monitor checker 142 is running on the compute node 102. For example, the process monitor checker verifier 144 executes a first command (e.g., a ps command) to generate first process data indicative of at least a subset of the processes 124 associated with the process monitor checker 142. In a particular aspect, the first command is indicative of the process monitor checker 142. In a particular embodiment, an identifier of the process monitor checker 142 is provided as an argument to the first command (e.g., a ps command). In response to determining that the first process data does not indicate any processes associated with the process monitor checker 142 or does not indicate a particular scheduled process associated with the process monitor checker 142, the process monitor checker verifier 144 determines that the process monitor checker 142 is not running (e.g., not validly running) on the compute node 102. The process monitor checker verifier 144 starts (eg, restarts) the process monitor checker 142 in response to determining that the process monitor checker 142 is not running on the compute node 102 .
[0035] In one particular aspect, the process monitor checker verifier 144 is configured to determine whether the process monitor checker 142 has stopped executing on the computing node 102. For example, in response to determining that the process monitor checker 142 is not executing on the computing node 102, the process monitor checker verifier 144 determines whether a previous version of the first process data indicates that the process monitor checker 142 previously ran on the computing node 102. Specifically, the process monitor checker verifier 144 generates a first version of the first process data by executing a first command at a first particular time and generates a second version of the first process data by executing the first command at a second particular time. The process monitor checker verifier 144 determines that the process monitor checker 142 has stopped executing on the compute node 102 in response to determining that the first version of the first process data indicates that the process monitor checker 142 was running at a first particular time and the second version of the first process data indicates that the process monitor checker 142 is not running at a second particular time. The process monitor checker verifier 144 stores the second version of the first process data in the memory 132. The process monitor checker verifier 144 restarts the process monitor checker 142 in response to determining that the process monitor checker 142 has stopped executing on the compute node 102. In one particular aspect, the process monitor checker verifier 144 executes the first command in response to detecting expiration of a timer (e.g., every five minutes), receiving user input, receiving a request from the status data analyzer 146, or a combination thereof.
[0036] The process monitor checker 142 determines whether the process monitor 140 is running on the compute node 102. For example, the process monitor checker 142 executes a second command (e.g., a ps command) to generate second process data indicative of at least a subset of the processes 124 associated with the process monitor 140. In response to determining that the second process data does not indicate any processes associated with the process monitor 140 or does not indicate any particular scheduled processes associated with the process monitor 140, the process monitor checker 142 determines that the process monitor 140 is not running (e.g., not validly running) on the compute node 102. In response to determining that the process monitor 140 is not running on the compute node 102, the process monitor checker 142 starts (e.g., restarts) the process monitor 140.
[0037] In one particular aspect, the process monitor checker 142 determines whether the process monitor 140 has stopped running on the compute node 102. For example, in response to determining that the process monitor 140 is not running on the compute node 102, the process monitor checker 142 determines whether a previous version of the second process data indicates that the process monitor 140 was previously running on the compute node 102. Specifically, the process monitor checker 142 generates a first version of the second process data by executing a second command at a first particular time and generates a second version of the second process data by executing the second command at a second particular time. In response to determining that the first version of the second process data indicates that the process monitor 140 was running at the first particular time and the second version of the second process data indicates that the process monitor 140 is not running at the second particular time, the process monitor checker 142 determines that the process monitor 140 has stopped running on the compute node 102. The process monitor checker 142 stores a second version of the second process data in the memory 132. The process monitor checker 142 restarts the process monitor 140 in response to determining that the process monitor 140 has stopped executing on the compute node 102. In one particular aspect, the process monitor checker 142 executes the second command in response to detecting the expiration of a timer (e.g., every five minutes), receiving user input, receiving a request from the status data analyzer 146, or a combination thereof.
[0038] The process monitor 140 generates process status data 105 for the first instance 160 of the software application 122. For example, the process monitor 140 executes a third command (e.g., a ps command) to generate process status data 105 indicative of at least a subset of the processes 124 associated with the software application 122. In a particular aspect, the third command is indicative of the software application 122. In a particular embodiment, an identifier of the software application 122 is provided as an argument to the third command (e.g., the ps command). The process monitor 140 stores the process status data 105 in the memory 132, provides the process status data 105 to the status data analyzer 146, or both. In a particular aspect, the process monitor 140 executes the third command in response to detecting the expiration of a timer (e.g., every five minutes), receiving user input, receiving a request from the status data analyzer 146, or a combination thereof.
[0039] The status data analyzer 146 analyzes the process status data 105 to check the status of a scheduled process associated with the software application 122. In one particular embodiment, the status data analyzer 146 analyzes the process status data 105 in response to detecting the expiration of a timer (e.g., every five minutes), receiving user input, receiving the process status data 105 from the process monitor 140, determining that the process status data 105 has been updated in the memory 132, or a combination thereof.
[0040] The status data analyzer 146 determines a process count 127 based on the process status data 105 (e.g., during a first stage of analysis of the process status data 105). The process count 127 indicates the number of a subset of the processes 124 that includes processes of a particular process type (e.g., level 1 processes, level 2 processes, level 3 processes, or a combination thereof). For example, the scheduled process data indicates a corresponding process type for each of the scheduled processes associated with the software application 122. The status data analyzer 146 determines a process count 127 based on the process status data 105 and the scheduled process data. For example, the process count 127 indicates the number of a subset of the processes 124 indicated by the process status data 105. The subset includes processes indicated by the scheduled process data as having a particular process type (e.g., level 3 processes).
[0041] The status data analyzer 146 determines whether the process number 127 meets the process number threshold 119. For example, the status data analyzer 146 determines that the process number 127 does not meet the process number threshold 119 in response to determining that the process number 127 is greater than or equal to a first threshold (e.g., 10) and less than or equal to a second threshold (e.g., 20). As another example, the status data analyzer determines that the process number 127 does not meet the process number threshold 119 in response to determining that the process number 127 is greater than or equal to the second threshold (e.g., 20). In a particular aspect, the process number threshold 119 indicates a first threshold, a second threshold, or both.
[0042] Status data analyzer 146 performs one or more actions in response to determining that process count 127 does not meet process count threshold 119. For example, the one or more actions include generating notification 151, notification 153, update 155, notification 157, notification 159, or a combination thereof. One or more of notification 151, notification 153, update 155, notification 157, or notification 159 indicate that process count 127 of a particular process type (e.g., level 3 processes) does not meet the process count threshold. For example, one or more of notification 151, notification 153, update 155, notification 157, or notification 159 indicate that process count 127 of a particular process type (e.g., level 3 processes) is greater than or equal to a first threshold (e.g., 10) and less than or equal to a second threshold (e.g., 20). As another example, one or more of notification 151, notification 153, update 155, notification 157, or notification 159 may indicate that the process count 127 of a particular process type (e.g., level 3 processes) exceeds a second threshold (e.g., 20).
[0043] The status data analyzer 146 sends, via the network interface 130, a notification 151 to the cluster management device 110, a notification 153 to the device 112, an update 155 to the storage device 108, a notification 157 to the compute node 106, a notification 159 to the compute node 104, or a combination thereof. In a particular embodiment, the cluster management device 110, in response to receiving the notification 151, updates the dashboard data 121 to indicate that the process count 127 of the particular process type (e.g., level 3 processes) on the compute node 102 does not meet the process count threshold 119. The cluster management device 110 provides the dashboard data 121 to a display, the compute node 102, the compute node 104, the compute node 106, the device 112, or a combination thereof. In a particular embodiment, the compute node 102 generates an output 123 based on the dashboard data 121 and provides the output 123 to the display 114. For example, a user can monitor the operation of the system 100 by viewing the output 123. In one particular aspect, the dashboard data 121 is associated with a dashboard (eg, a visual information display and management interface).
[0044] In one particular aspect, the device 112 displays a message (e.g., an email message or an alert) on the device 112 in response to receiving the notification 153. In one particular aspect, the device 112 sends a message to a dashboard (e.g., a visual information display and management interface), updates a status log file, or both in response to receiving the notification 153. The message indicates that the process count 127 of the particular process type (e.g., level 3 processes) on the computing node 102 does not meet the process count threshold 119. In one particular aspect, the storage device 108 updates the log data 131 to indicate that the process count 127 of the particular process type (e.g., level 3 processes) on the computing node 102 does not meet the process count threshold 119 in response to receiving the update 155. The log data 131 is associated with the computing node 102, the software application 122, the first instance 160 of the software application 122, or a combination thereof. In one particular aspect, in response to receiving notification 157 (or notification 159), computing node 106 (or computing node 104) updates its dashboard data to indicate that the process count 127 of a particular process type (e.g., level 3 processes) on computing node 102 does not meet process count threshold 119.
[0045] The status data analyzer 146 determines (e.g., during the second stage of analysis of the process status data 105) whether one or more of the scheduled processes indicated by the scheduled process data are executing on the compute node 102. For example, the status data analyzer 146 determines whether the scheduled process 107 of the first instance 160 of the software application 122 is executing on the first compute node 102 based on the process status data 105. Specifically, the status data analyzer 146 determines that the scheduled process 107 is not executing on the compute node 102 in response to the process status data 105 indicating that the scheduled process 107 has stopped or determining that the processes 124 indicated by the process status data 105 do not include the scheduled process 107.
[0046] In one particular aspect, the status data analyzer 146 determines whether the scheduled process 107 has stopped executing on the compute node 102. For example, in response to determining that the scheduled process 107 is not executing on the compute node 102, the status data analyzer 146 determines whether a previous version of the process status data 105 indicates that the scheduled process 107 was previously executing on the compute node 102. Specifically, the status data analyzer 146 generates a first version of the process status data 105 at a first particular time (e.g., by executing a ps command) and generates a second version of the process status data 105 at a second particular time (e.g., by executing a ps command). In response to determining that the first version of the process status data 105 indicates that the scheduled process 107 was executing at the first particular time and that the second version of the process status data 105 indicates that the scheduled process 107 is not executing at the second particular time, the status data analyzer 146 determines that the scheduled process 107 has stopped executing on the compute node 102. The status data analyzer 146 stores the second version of the process status data 105 in the memory 132 .
[0047] The status data analyzer 146 performs one or more actions in response to determining that the scheduled process 107 is not running (or has stopped running) on the compute node 102. The one or more actions include starting (or restarting) the scheduled process 107, starting (or restarting) the first instance 160 of the software application 122, generating a notification 151, generating a notification 153, generating an update 155, generating a notification 157, generating a notification 159, shutting down (e.g., stopping) a service group (e.g., process monitor 140, process monitor checker 142, or both), shutting down a peer alert service group (e.g., peer monitor 150, peer monitor checker 152, or both), shutting down the first instance 160 of the software application 122, or a combination thereof. For example, the status data analyzer 146 determines that the scheduled process data indicates that the scheduled process 107 is of process type 117 and selects one or more actions based on the process type 117.
[0048] In one particular aspect, the status data analyzer 146 determines whether the application restart criteria 111 are met in response to determining that the process type 117 matches the first process type (e.g., a level 1 process), as further described with reference to FIG. 5B. The status data analyzer 146 restarts the first instance 160 of the software application 122 on the compute node 102 in response to determining that the application restart criteria 111 are met. The status data analyzer 146 generates a notification 151, a notification 153, an update 155, a notification 157, a notification 159, or a combination thereof, indicating that a failure of the scheduled process 107 of the process type 117 has been detected and that the first instance 160 of the software application 122 has been restarted on the compute node 102.
[0049] Alternatively, the status data analyzer 146 inhibits the restart of the first instance 160 of the software application 122 in response to determining that the application restart criteria 111 have not been satisfied. For example, the status data analyzer 146 shuts down (e.g., stops) the first instance 160 of the software application 122, the process monitor 140, the process monitor checker 142, the peer monitor 150, the peer monitor checker 152, or a combination thereof, on the compute node 102. The status data analyzer 146 generates a notification 151, a notification 153, an update 155, a notification 157, a notification 159, or a combination thereof, indicating that a failure of the scheduled process 107 of the process type 117 has been detected and that the first instance 160 of the software application 122 has been shut down on the compute node 102. To indicate that a failure of scheduled process 107 of process type 117 has been detected and that software application 122 will be restarted (or shut down) on compute node 102, cluster management device 110 updates dashboard data 121 based on notification 151, device 112 displays a message based on notification 153, storage device 108 updates log data 131 based on update 155, compute node 106 updates dashboard data based on notification 157, compute node 104 updates dashboard data based on notification 159, or a combination of these.
[0050] In one particular aspect, the status data analyzer 146 starts (e.g., restarts) the scheduled process 107 in response to determining that the process type 117 matches a second process type (e.g., a level 2 process). The status data analyzer 146 generates a notification 151, a notification 153, an update 155, a notification 157, a notification 159, or a combination thereof, indicating that a failure of the scheduled process 107 of the process type 117 has been detected and that the scheduled process 107 has been started (e.g., restarted) on the compute node 102. The cluster management device 110 updates the dashboard data 121 based on the notification 151, the device 112 displays a message based on the notification 153, the storage device 108 updates the log data 131 based on the update 155, the compute node 106 updates the dashboard data based on the notification 157, the compute node 104 updates the dashboard data based on the notification 159, or a combination thereof, to indicate that a failure of the scheduled process 107 of the process type 117 has been detected and the scheduled process 107 is started (e.g., restarted) on the compute node 102. The status data analyzer 146 and the service group (e.g., the process monitor 140, the process monitor checker 142, or both) may selectively restart the first instance 160, start a scheduled process 107 (e.g., a level 2 process) that corresponds to the valid state of the software application 122, or both, thus reducing the downtime of the first instance 160 of the software application 122.
[0051] The peer monitor checker verifier 154 determines whether the peer monitor checker 152 is running on the compute node 102. For example, the peer monitor checker verifier 154 executes a first command (e.g., a ps command) to generate first process data indicative of at least a subset of the processes 124 associated with the peer monitor checker 152. In a particular aspect, the first command is indicative of the peer monitor checker 152 (e.g., an identifier for the peer monitor checker 152 is provided to the first command). In response to determining that the first process data does not indicate any processes associated with the peer monitor checker 152 or does not indicate a particular scheduled process associated with the peer monitor checker 152, the peer monitor checker verifier 154 determines that the peer monitor checker 152 is not running (e.g., not validly running) on the compute node 102. The peer monitor checker verifier 154 , in response to determining that the peer monitor checker 152 is not running on the compute node 102 , starts (eg, restarts) the peer monitor checker 152 .
[0052] In one particular aspect, the peer monitor checker verifier 154 is configured to determine whether the peer monitor checker 152 has stopped running on the computing node 102. For example, in response to determining that the peer monitor checker 152 is not running on the computing node 102, the peer monitor checker verifier 154 determines whether a previous version of the first process data indicates that the peer monitor checker 152 was previously running on the computing node 102. Specifically, the peer monitor checker verifier 154 generates a first version of the first process data at a first particular time and generates a second version of the first process data at a second particular time. In response to determining that the first version of the first process data indicates that the peer monitor checker 152 was running at the first particular time and the second version of the first process data indicates that the peer monitor checker 152 is not running at the second particular time, the peer monitor checker verifier 154 determines that the peer monitor checker 152 has stopped running on the computing node 102. The peer monitor checker verifier 154 stores the second version of the first process data in the memory 132. The peer monitor checker verifier 154 restarts the peer monitor checker 152 in response to determining that the peer monitor checker 152 has stopped running on the compute node 102. In one particular aspect, the peer monitor checker verifier 154 executes the first command in response to detecting the expiration of a timer (e.g., every five minutes), receiving user input, receiving a request from the status data analyzer 146, or a combination thereof.
[0053] Peer monitor checker 152 determines whether peer monitor 150 is running on compute node 102. For example, peer monitor checker 152 executes a second command (e.g., a ps command) to generate second process data indicative of at least a subset of processes 124 associated with peer monitor 150. In response to determining that the second process data does not indicate any processes associated with peer monitor 150 or does not indicate any particular scheduled processes associated with peer monitor 150, peer monitor checker 152 determines that peer monitor 150 is not running (or not validly running) on compute node 102. In response to determining that peer monitor 150 is not running on compute node 102, peer monitor checker 152 starts (or restarts) peer monitor 150.
[0054] In one particular aspect, the peer monitor checker 152 is configured to determine whether the peer monitor 150 has stopped running on the computing node 102. For example, in response to determining that the peer monitor 150 is not running on the computing node 102, the peer monitor checker 152 determines whether a previous version of the second process data indicates that the peer monitor 150 was previously running on the computing node 102. Specifically, the peer monitor checker 152 generates a first version of the second process data at a first particular time and generates a second version of the second process data at a second particular time. In response to determining that the first version of the second process data indicates that the peer monitor 150 was running at the first particular time and the second version of the second process data indicates that the peer monitor 150 is not running at the second particular time, the peer monitor checker 152 determines that the peer monitor 150 has stopped running on the computing node 102. The peer monitor checker 152 stores a second version of the second process data in the memory 132. The peer monitor checker 152 restarts the peer monitor 150 in response to determining that the peer monitor 150 has stopped executing on the compute node 102. In one particular aspect, the peer monitor checker 152 executes the second command in response to detecting the expiration of a timer (e.g., every five minutes), receiving user input, receiving a request from the status data analyzer 146, or a combination thereof.
[0055] The peer monitor 150 collects data indicative of the status of scheduled processes associated with the software application 122 on other compute nodes. In one particular implementation, the peer monitor 150 receives the cluster process data 143 from the cluster management device 110. In one particular implementation, the peer monitor 150 sends a request to the cluster management device 110 and receives the cluster process data 143 in response to the request. In an alternative implementation, the cluster management device 110 sends the cluster process data 143 to the peer monitor 150 independent of receiving a request. For example, the cluster management device 110 sends the cluster process data 143 to the peer monitor 150 in response to detecting the expiration of a timer, receiving a user input, or both. The peer monitor 150 stores the cluster process data 143 in the memory 132, provides the cluster process data 143 to the status data analyzer 146, or both.
[0056] In one particular implementation, the peer monitor 150 receives the log data 141 from the storage device 108. In one particular embodiment, the peer monitor 150 sends a log request 161 to the storage device 108 and receives the log data 141 in response to the log request 161. In an alternative embodiment, the storage device 108 sends the log data 141 to the peer monitor 150 independent of receiving the request 161. For example, the storage device 108 sends the log data 141 to the peer monitor 150 in response to detecting the expiration of a timer, receiving user input, or both. The peer monitor 150 stores the log data 141 in the memory 132, provides the log data 141 to the status data analyzer 146, or both.
[0057] The peer monitor 150 checks the status of a scheduled process associated with the software application 122 on the compute node 104. For example, the peer monitor 150 determines whether a second instance 162 of the software application 122 has failed on the compute node 104. In one particular implementation, the peer monitor 150 determines whether the second instance 162 has failed on the compute node 104 based on the cluster process data 143, the log data 141, or both. For example, the peer monitor 150 determines that the cluster process data 143 (or the log data 141) indicates that one or more second processes are running on the compute node 104, that one or more third processes are stopped on the compute node 104, or both. In response to determining that the second processes do not include the scheduled process 113, that the third processes include the scheduled process 113, or both, the peer monitor 150 determines that the second instance 162 is not running in a valid state. In one particular aspect, the peer monitor 150 determines that the second instance 162 of the software application 122 has failed on the compute node 104 in response to determining that the second instance 162 is not running in a valid state.
[0058] In one particular aspect, the peer monitor 150 determines whether the second instance 162 has stopped validly executing. For example, in response to determining that the second instance 162 is not validly executing, the peer monitor 150 determines whether a previous version of the cluster process data 143 (or a previous version of the log data 141) indicates that the second instance 162 previously ran validly on the compute node 104. Specifically, the peer monitor 150 determines that a first version of the cluster process data 143 (or a first version of the log data 141) is generated (or received) at a first particular time and a second version of the cluster process data 143 (or a second version of the log data 141) is generated (or received) at a second particular time. Peer monitor 150 determines that second instance 162 has stopped validly executing on compute node 104 in response to determining that the first version of cluster process data 143 (or the first version of log data 141) indicates that second instance 162 was validly executing at a first particular time and the second version of cluster process data 143 (or the second version of log data 141) indicates that second instance 162 is not validly executing at a second particular time. Peer monitor 150 determines that second instance 162 has failed on compute node 104 in response to determining that second instance 162 has stopped validly executing on compute node 104. Peer monitor 150 stores the second version of cluster process data 143 (or the second version of log data 141) in memory 132.
[0059] In one particular aspect, the peer monitor 150 determines that the second instance 162 of the software application 122 has failed on the compute node 104 in response to determining that the cluster process data 143 (or the log data 141) indicates that the software application 122 has been shut down on the compute node 104. In one particular aspect, the peer monitor 150 analyzes the cluster process data 143, the log data 141, or both in response to detecting the expiration of a timer (e.g., every five minutes), receiving a user input, receiving the cluster process data 143 from the peer monitor 150, determining that the cluster status data 143 has been updated in the memory 132, receiving the log data 141 from the storage device 108, determining that the log data 141 has been updated in the memory 132, or a combination thereof.
[0060] Peer monitor 150 performs one or more actions in response to determining that second instance 162 of software application 122 has failed on second computing node 104. The one or more actions may include generating notification 151, notification 153, update 155, notification 157, notification 159, designating a cluster managing device 110, or a combination thereof, indicating that a failure of second instance 162 of software application 122 has been detected on computing node 104. For example, peer monitor 150 designates computing node 102 or computing node 106 as cluster managing device 110 in response to determining that computing node 104 corresponds to (e.g., includes) cluster managing device 110. Specifically, peer monitor 150 starts an instance of a cluster managing application on computing node 102 or sends a request to computing node 106 to start an instance of a cluster managing application. In one particular aspect, in response to determining that the second instance 162 of the software application 122 has failed on the second computing node 104, the peer monitor 150 generates process status data 105, sends the process status data 105 to the cluster management device 110, or both.
[0061] In one particular aspect, the cluster management device 110, in response to receiving the notification 151 and determining that the notification 151 indicates a failure of the second instance 162 on the compute node 104, updates the load balancer to redirect future requests intended for the compute node 104 to another compute node. In one particular aspect, the cluster management device 110, in response to receiving the notification 151 and determining that the notification 151 indicates a failure of the second instance 162 on the compute node 104, generates (or updates) cluster process data 143, selects a compute node to perform any incomplete tasks associated with the second instance 162, or both. For example, the cluster management device 110, in response to receiving the notification 151, sends a request 163 to the compute node 106. The compute node 106, in response to receiving the request 163, sends process status data 165 to the cluster management device 110. Cluster management device 110 generates (or updates) cluster process data 143 based on process status data 105, process status data 165, or both.
[0062] The cluster management device 110 determines that the process status data 105 indicates a first number of a subset of the processes 124 associated with the software application 122 executing on the compute node 102. The cluster management device 110 selects the compute node 102 in response to determining, based on the cluster process data 143, that the first number is less than or equal to a second number of processes executing on the compute node 106. For example, the second number indicates a subset of the processes associated with the software application 122 executing on the compute node 106. The cluster management device 110 sends an assignment 145 to the compute node 102 to indicate that the compute node 102 (e.g., the selected compute node) should perform any incomplete tasks associated with the second instance 162.
[0063] The peer monitor 150 identifies an incomplete task 109 associated with the second instance 162 of the software application 122. In one particular implementation, the peer monitor 150 sends a log request 161 to the storage device 108 in response to determining that the second instance 162 of the software application 122 has failed on the compute node 104, receiving an assignment 145 from the cluster management device 110, or both. The peer monitor 150 determines whether the log data 141 indicates an incomplete task 109 associated with the second instance 162 of the software application 122.
[0064] Peer monitor 150 performs one or more actions associated with incomplete task 109. For example, peer monitor 150 performs action 128 associated with incomplete task 115. In a particular aspect, peer monitor 150 performs action 128 to undo incomplete task 115. For example, peer monitor 150 performs action 128 to roll back database updates in response to determining that incomplete task 115 includes database updates. In an alternative aspect, peer monitor 150 performs action 128 to complete incomplete task 115. For example, peer monitor 150 performs action 128 to make database updates in response to determining that incomplete task 115 includes database updates.
[0065] In this manner, the system 100 reduces downtime of the software application 122 on the compute node 102 by selectively restarting the first instance 160, by starting a scheduled process 107 (e.g., a level 2 process) that corresponds to a valid state of the software application 122, or both. The system 100 can restore a valid state by having the compute node 102 address incomplete tasks 109 associated with a failed instance of the software application 122 on the compute node 104.
[0066] 2 illustrates one embodiment of a method 200 for monitoring processes of a software application. In a particular embodiment, the method 200 is performed by the cluster management device 110, the compute node 102, the compute node 104, the compute node 106, the system 100, or a combination thereof of FIG.
[0067] The method 200 includes, at 202, starting an instance of a software application on a node. For example, the cluster managing device 110 of FIG. 1 sends a first request to the computing node 102, the computing node 104, the computing node 106, or a combination thereof. The computing node 102 starts the first instance 160 of the software application 122 in response to receiving the first request from the cluster managing device 110, as described with reference to FIG. 1. In a particular embodiment, the other computing nodes of the system 100 also start instances of the software application 122 in response to receiving the first request. For example, the computing node 104 starts the second instance 162 of the software application 122.
[0068] Method 200 also includes starting the service group of the node, at 204. For example, cluster management device 110 of FIG. 1 sends the second request to compute node 102, compute node 104, compute node 106, or a combination thereof. Computing node 102 starts the service group associated with software application 122 (e.g., process monitor 140, process monitor checker 142, or both) in response to receiving the second request from cluster management device 110, as described with reference to FIG. 1. In a particular embodiment, other computing nodes of system 100 also start instances of the service group in response to receiving the second request. In a particular embodiment, computing node 102 starts first instance 160 in response to receiving the first request and starts the service group in response to receiving a second request separate from the first request. In an alternative embodiment, the compute node 102 starts the first instance 160 and the service group (eg, the process monitor 140, the process monitor checker 142, or both) in response to receiving a single request.
[0069] The method 200 further includes starting a peer alert service group at 206. For example, the cluster management device 110 of FIG. 1 selects at least one computing node to monitor other computing nodes and sends the third request to the selected computing node. Specifically, the cluster management device 110 selects the computing node 104, as described with reference to FIG. 1. The cluster management device 110 sends the third request to the computing node 104 (e.g., the selected computing node). The computing node 102 starts a peer alert service group (e.g., the peer monitor 150, the peer monitor checker 152, or both) in response to receiving the third request from the cluster management device 110, as described with reference to FIG. 1.
[0070] The method 200 enables the cluster management device 110 to start instances of the software application 122 and corresponding service groups on multiple computing nodes. The method 200 also enables the cluster management device 110 to select at least one computing node and start a peer alert service group on the selected computing node to monitor the other computing nodes.
[0071] 3 illustrates one embodiment of a method 300 for monitoring processes of a software application. In one particular embodiment, the method 300 is performed by the status data analyzer 146, the process monitor 140, the process monitor checker 142, the process monitor checker verifier 144, the compute node 102, the compute node 104, the compute node 106, the cluster management device 110, the system 100, or a combination thereof, of FIG.
[0072] In one particular aspect, method 300 continues from 204 of Figure 2. For example, cluster management device 110 sends a second request to compute node 102, compute node 104, compute node 106, or a combination thereof, to start the service group corresponding to the instance of software application 122. Method 300 is performed on compute node 102, compute node 104, compute node 106, or a combination thereof in response to receiving the second request.
[0073] Method 300 includes starting a process monitor checker and a process monitor, at 302. For example, status data analyzer 146 of Figure 1 starts process monitor checker 142, process monitor 140, or both, as described with reference to Figure 1. In one particular embodiment, status data analyzer 146 starts first and second threads of execution in processor 134 of Figure 1 that correspond to process monitor checker 142 and process monitor 140, respectively.
[0074] The method 300 also includes checking a status of the process monitor, at 310. For example, the process monitor checker 142 of FIG. 1 generates first process status data indicative of the subset, if any, of the processes 124 that correspond to the process monitor 140, as described with reference to FIG.
[0075] The method 300 further includes determining whether the process monitor has stopped executing, at 312. For example, the process monitor checker 142 of FIG. 1 determines whether the process monitor 140 has stopped executing based on the first process status data, as described with reference to FIG.
[0076] 1 , in response to determining that the process monitor 140 is running or that the process monitor 140 was not previously running, generates an updated version of the first process status data at a second time. Specifically, the process monitor checker 142 generates the first process status data in response to detecting the expiration of a timer.
[0077] Method 300 includes restarting the process monitor at 314 in response to determining that the process monitor has stopped executing at 312. For example, process monitor checker 142 of Figure 1 restarts process monitor 140 in response to determining that the process monitor 140 has stopped executing based on the first process status data. Method 300 continues at 310.
[0078] The method 300 also includes checking the status of the scheduled processes, at 316. For example, the process monitor 140 of FIG. 1 generates process status data 105 indicative of processes 124 running on the compute nodes 102, as described with reference to FIG.
[0079] The method 300 further includes determining whether the process count exceeds a threshold, at 318. For example, the status data analyzer 146 of Figure 1 generates the process count 127 based on the process status data 105, as described with reference to Figure 1. The status data analyzer 146 determines whether the process count 127 exceeds the process count threshold 119.
[0080] Method 300 includes, in response to determining that the process count exceeds the threshold, performing 320 a notification of the process count, updating the log data, and updating the dashboard data. For example, status data analyzer 146 of FIG. 1 generates notification 151, notification 153, update 155, notification 157, notification 159, or a combination thereof, indicating that process count 127 does not meet process count threshold 119, as described with reference to FIG. 1. In one particular embodiment, status data analyzer 146 updates dashboard data 121 to indicate that process count 127 does not meet process count threshold 119, in response to determining that process count 127 exceeds process count threshold 119.
[0081] The method 300 includes determining whether any scheduled processes have stopped executing, at 322. For example, the status data analyzer 146 of FIG. 1 determines whether any scheduled processes associated with the software application 122 have stopped executing on the compute node 102 based on the process status data 105. In response to determining, at 322, that no scheduled processes have stopped executing, the method 300 includes continuing to 326.
[0082] In response to determining at 322 that at least one scheduled process has stopped executing, the method 300 includes performing an analysis on the scheduled process that has stopped executing, at 324. For example, the status data analyzer 146 of FIG. 1 performs an analysis of the scheduled process that has stopped executing, as further described with reference to FIG.
[0083] The method 300 also includes checking the status of the process monitor checker, at 326. For example, the process monitor checker verifier 144 of FIG. 1 generates second process status data, as described with reference to FIG. 1, indicating the subset, if any, of the processes 124 that correspond to the process monitor checker 142 of FIG.
[0084] The method 300 further includes determining whether the process monitor checker has stopped executing, at 328. For example, the process monitor checker verifier 144 of FIG. 1 determines whether the process monitor checker 142 has stopped executing based on the second process status data, as described with reference to FIG.
[0085] The method 300 includes returning to 316 in response to determining at 328 whether the process monitor checker has stopped executing. For example, the process monitor 140 of FIG. 1 generates a first version of the process status data 105 at a first time. The status data analyzer 146 of FIG. 1 determines whether there are any scheduled processes associated with the software application 122 that have stopped executing based on the first version of the process status data 105 and performs an analysis of the scheduled processes that have stopped executing. The process monitor checker verifier 144 generates a second process status at a second time after the first time. The process monitor checker verifier 144 determines whether the process monitor checker 142 has stopped executing based on the second process status data and restarts the process monitor checker 142 if the process monitor checker 142 has stopped executing. In one particular embodiment, the process monitor 140 generates a second version of the process status data 105 at a third time after the second time. In one particular embodiment, the status data analyzer 146 sends a request to the process monitor 140 to provide an updated version of the process status data 105. Specifically, the status data analyzer 146 sends the request in response to detecting the expiration of a timer.
[0086] 1 restarts the process monitor checker 142 in response to determining that the second process data indicates that the process monitor checker 142 has stopped executing. The method 300 returns to 316.
[0087] In this manner, method 300 provides multiple levels of checks to enable close monitoring of scheduled processes associated with software application 122. Figure 3 illustrates two levels of checks provided by process monitor checker 142 and process monitor 140.
[0088] Figure 4 illustrates one embodiment of a method for monitoring software application processes. The method corresponds to 324 of Figure 3. In a particular embodiment, the method 324 is performed by the status data analyzer 146, the computing node 102, the computing node 104, the computing node 106, the system 100, or a combination thereof of Figure 1.
[0089] The method 324 includes determining 402 a process type of the scheduled process that has stopped executing. For example, the status data analyzer 146 of FIG. 1 determines that the scheduled process 107 has a process type 117 based on the scheduled process data, as described with reference to FIG.
[0090] Method 324 also includes, in response to determining 402 that the process type matches process type 401, determining 404 whether application restart criteria are met. For example, status data analyzer 146 of FIG. 1 determines whether application restart criteria 111 are met in response to determining that process type 117 matches process type 401 (e.g., a level 1 process), as further described with reference to FIG. 5B.
[0091] The method 324 includes, in response to determining that the application restart criterion is met at 404, increasing a value of a hazard counter, at 406. For example, the status data analyzer 146 of Figure 1 increases a value of the hazard counter 517 of Figure 5B (e.g., by one) in response to determining that the application restart criterion 111 is met.
[0092] The method 324 further includes restarting 408 the software application. For example, the status data analyzer 146 of FIG.
[0093] The method 324 also includes performing 410 a process failure restart notification, updating the log data, and updating the dashboard data. For example, the status data analyzer 146 of Figure 1 generates notification 151, notification 153, update 155, notification 157, notification 159, or a combination thereof, as described with reference to Figure 1, indicating that the software application 122 is restarted on the compute node 102. In one particular aspect, the status data analyzer 146 updates the dashboard data 121 of Figure 1 to indicate that the software application is restarted on the compute node 102. The method 324 returns to 326.
[0094] The method 324 includes, in response to determining that the application restart criteria have not been met at 404, shutting down the software application, at 412. For example, the status data analyzer 146 of FIG. 1, in response to determining that the application restart criteria 111 have not been met, shuts down (e.g., stops) the first instance 160 of the software application 122.
[0095] The method 324 also includes stopping 414 the service group on the compute node. For example, the status data analyzer 146 of FIG.
[0096] The method 324 further includes performing 416 a stop process notification, updating the log data, and updating the dashboard data. For example, the status data analyzer 146 of FIG. 1 generates notification 151, notification 153, update 155, notification 157, notification 159, or a combination thereof, to indicate that the software application 122 is being shut down on the compute node 102. In one particular aspect, the status data analyzer 146 updates the dashboard data 121 to indicate that the software application 122 is being shut down on the compute node 102.
[0097] The method 324 includes, in response to determining at 402 that the process type matches the process type 403, restarting 418 the scheduled process. For example, the status data analyzer 146 of FIGURE 1 restarts the scheduled process 107 in response to determining that the process type 117 matches the process type 403 (e.g., a level 2 process).
[0098] The method 324 includes determining whether another scheduled process should be analyzed, at 420. For example, the status data analyzer 146 of FIG. 1 determines, based on the process status data 105, that another scheduled process is not running on the compute node 102.
[0099] Method 324 includes returning to 402 in response to determining at 420 that another scheduled process should be analyzed. For example, status data analyzer 146 of FIG. 1 performs an analysis on the second scheduled process in response to determining based on process status data 105 that the second scheduled process has stopped executing and analysis has not been performed on the second scheduled process. Method 324 includes returning to 326 in response to determining that there are no other scheduled processes to be analyzed. For example, status data analyzer 146 of FIG. 1 sends a request to process monitor 140 to check the status of process monitor checker 142 of FIG. 1 in response to determining that analysis of all scheduled processes that have stopped executing has been completed.
[0100] 1 and perform various actions based on the type of scheduled process that stopped executing on the compute node 102. For example, the status data analyzer 146 selectively restarts the first instance 160 of the software application 122 for a scheduled process 107 of process type 401. As another example, the status data analyzer 146 restarts a scheduled process 107 of process type 403.
[0101] 5A, a diagram is shown and generally designated by reference numeral 500. Diagram 500 illustrates a status data analyzer 146 and a memory 132. Memory 132 is configured to store process status data 105, start times 511, stop times 513, fault count thresholds 519, time thresholds 515, application restart criteria 111, fault counters 517, or combinations thereof. In one particular aspect, fault count thresholds 519, time thresholds 515, or both, are based on user input, default data, configuration settings, or combinations thereof.
[0102] 1. Start time 511 indicates the time when scheduled process 107 was last started on compute node 104. Stop time 513 indicates the time when scheduled process 107 is detected as not running on compute node 104. For example, stop time 513 indicates the time when scheduled process 107 is determined to not be running on compute node 104 (or the time when data indicating not running is generated). In one particular aspect, stop time 513 corresponds to a timestamp of process status data 105 (e.g., indicating the time when process status data 105 was generated).
[0103] Figure 5B illustrates an embodiment of a method corresponding to 404 of Figure 4. In a particular embodiment, method 404 is performed by the status data analyzer 146, the computing node 102, the computing node 104, the computing node 106, the system 100, or a combination thereof of Figure 1.
[0104] The method 404 includes determining 502 whether a difference between a stop time and a start time is greater than or equal to a time threshold. For example, the status data analyzer 146 determines the difference between a start time 511 and a stop time 513. The status data analyzer 146 determines whether the difference between the start time 511 and the stop time 513 meets (e.g., is greater than or equal to) a time threshold 515.
[0105] The method 404 includes, in response to determining at 502 that the difference between the stop time and the start time is greater than or equal to the time threshold, resetting a failure counter at 504. For example, the status data analyzer 146 resets a failure counter 517 (e.g., to 0) in response to determining that the difference between the stop time 513 and the start time 511 meets (e.g., is greater than or equal to) the time threshold 515. In one particular aspect, the failure counter 517 indicates the number of consecutive failures of the scheduled process that occurred relatively soon after the start of the scheduled process.
[0106] The method 404 also includes determining 506 whether the current value of the failure counter is less than a failure count threshold. For example, the status data analyzer 146 of FIG. 1 determines whether the value of the failure counter 517 meets (e.g., is less than) the failure count threshold 519.
[0107] The method 404 includes determining, at 508, that the application restart criterion is met in response to determining that the value of the failure counter is less than the failure count threshold at 506. For example, the status data analyzer 146 determines that the application restart criterion 111 is met in response to determining that the value of the failure counter 517 meets (e.g., is less than) the failure count threshold 519.
[0108] The method 404 includes, in response to determining 506 that the value of the failure counter is greater than or equal to the failure count threshold, determining 510 that the application restart criterion is not met. For example, the status data analyzer 146 determines that the application restart criterion 111 is not met in response to determining that the value of the failure counter 517 does not meet (e.g., is greater than or equal to) the failure count threshold 519. Specifically, if the number of consecutive failures of the scheduled process occurring relatively soon after start of the scheduled process is greater than or equal to the failure count threshold 519, the status data analyzer 146 inhibits the restart 122 of the software application on the compute node 102.
[0109] In this manner, the status data analyzer 146 enables a determination of whether the application restart criteria 111 are met based on the time between the start of the scheduled process and detection of the scheduled process stopping. For example, the status data analyzer 146 may shut down a software application 122 on a compute node 102 in response to the scheduled process stopping relatively soon after starting and detecting a number of faults greater than a threshold.
[0110] 6 illustrates one embodiment of a method 600 for monitoring the processes of a software application. In a particular aspect, the method 600 is performed by the peer monitor 150, the peer monitor checker 152, the peer monitor checker verifier 154, the computing node 102, the computing node 104, the computing node 106, the system 100, or a combination thereof, of FIG.
[0111] In one particular aspect, method 600 continues from 206 of Figure 2. For example, cluster management device 110 sends a request to compute node 102 (e.g., a selected compute node) to start a peer alert service group. Method 600 is performed at compute node 102 in response to receiving the request.
[0112] Method 600 includes starting 602 a peer monitor checker and a peer monitor. For example, status data analyzer 146 of Figure 1 starts peer monitor checker 152, peer monitor 150, or both, as described with reference to Figure 1. In one particular aspect, status data analyzer 146 starts first and second threads of execution in processor 134 of Figure 1 that correspond to peer monitor checker 152 and peer monitor 150, respectively.
[0113] The method 600 also includes checking the status of the peer monitor, at 610. For example, the peer monitor checker 152 of FIG.
[0114] The method 600 further includes determining whether the peer monitor has stopped running, at 612. For example, the peer monitor checker 152 of FIG. 1 determines whether the peer monitor 150 has stopped running on the compute node 102 based on the first process data, as described with reference to FIG.
[0115] The method 600 also includes returning to 610 in response to determining whether the peer monitor has stopped running at 612. For example, the peer monitor checker 152 of FIG. 1 generates an updated version of the first process data at a second time in response to determining at a first time that the peer monitor 150 is running on the computing node 102 or that the peer monitor 150 was not previously running on the computing node 102.
[0116] Method 600 further includes restarting the peer monitor at 614 in response to determining that the peer monitor has stopped executing at 612. For example, peer monitor checker 152 restarts peer monitor 150 in response to determining that peer monitor 150 has stopped executing on compute node 102.
[0117] Method 600 includes checking 616 the status of the service group at other nodes. For example, peer monitor 150 of Figure 1 collects data indicative of the status of the service group at other compute nodes. In particular, peer monitor 150 receives cluster process data 143 from cluster management device 110, log data 141 corresponding to compute node 104 from storage device 108, third log data corresponding to compute node 106 from storage device 108, or a combination thereof, as described with reference to Figure 1.
[0118] Method 600 includes, at 618, determining whether any service groups have stopped running. For example, peer monitor 150 determines whether any service groups have stopped running on other compute nodes. Specifically, peer monitor 150 determines whether cluster process data 143 (or log data 141) indicates that a second service group (e.g., a second process monitor, a second process monitor checker, or both) has stopped running on compute node 104. In one particular aspect, peer monitor 150 determines whether cluster process data 143 (or third log data) indicates that a third service group (e.g., a third process monitor, a third process monitor checker, or both) has stopped running on compute node 106. Method 600 includes, in response to determining, at 618, that no service groups have stopped running, continuing to 622.
[0119] In response to determining 618 that the at least one service group has stopped executing, method 600 includes performing 620 a stopped node notification and updating dashboard data. For example, peer monitor 150 determines that the second service group has stopped executing in response to determining that at least one of the second process monitor or the second process monitor checker has stopped executing on compute node 104. In response to determining that the second service group has stopped executing, peer monitor 150 generates notification 151, notification 153, notification 157, notification 159, or a combination thereof, indicating that a failure (or shutdown) of software application 122 has been detected on compute node 104. In a particular aspect, peer monitor 150 updates dashboard data 121 to indicate that a failure (or shutdown) of software application 122 has been detected on compute node 104.
[0120] The method 600 also includes checking the status of the peer monitor checker, at 622. For example, the peer monitor checker verifier 154 of FIG.
[0121] The method 600 further includes determining whether the peer monitor checker has stopped executing, at 624. For example, the peer monitor checker verifier 154 of FIG. 1 determines whether the peer monitor checker 152 has stopped executing based on the second process status data, as described with reference to FIG.
[0122] The method 600 includes returning to 616 in response to determining at 624 whether the peer monitor checker has stopped running. For example, the peer monitor 150 of FIG. 1 generates a first version of cluster process data 143, a first version of log data (e.g., log data 141, third log data, or both), or a combination thereof, at a first time. The peer monitor 150 determines whether service groups have stopped running on other compute nodes based on the first version of cluster process data 143, the first version of log data, or both, and performs a stopped node notification for any service groups that have stopped running. The peer monitor checker verifier 154 generates second process data at a second time after the first time. The peer monitor checker verifier 154 determines whether the peer monitor checker 152 has stopped running based on the second process data, and restarts the peer monitor checker 152 if the peer monitor checker 152 has stopped running. Peer monitor 150 generates a second version of cluster process data 143, a second version of the log data (e.g., log data 141, the third log data, or both), or a combination thereof, at a third time. In one particular aspect, the third time is after the second time. In one particular aspect, peer monitor 150 sends a first request to cluster managing device 110 for an updated version of cluster process data 143, sends a second request to storage device 108 for an updated version of the log data (e.g., log data 141, the third log data, or both), or a combination thereof.
[0123] Method 600 includes restarting the peer monitor checker at 626 in response to determining that the peer monitor checker has stopped executing at 624. For example, peer monitor checker verifier 154 of Figure 1 restarts peer monitor checker 152 in response to determining that the second peer data indicates that peer monitor checker 152 has stopped executing. Method 600 returns to 616.
[0124] In this manner, method 600 allows for close monitoring of processes on other compute nodes using multiple levels of checks. Figure 6 illustrates two levels of checks provided by peer monitor checker 152 and peer monitor 150.
[0125] 7 is a flow diagram of a method 700 for monitoring software application processes. The method 700 may be implemented by the status data analyzer 146, the process monitor 140, the peer monitor 150, the compute node 102, the compute node 104, the compute node 106, the system 100, or any combination thereof of FIG.
[0126] The method 700 includes, at 702, generating, at a first computing node of a plurality of computing nodes configured to execute a plurality of instances of the software application, process status data relating to a first instance of the software application executing on the first computing node. For example, the process monitor 140 of Figure 1 generates process status data 105 relating to a first instance 160 of the software application 122 executing on the computing node 102. In one particular aspect, the computing node 102, the computing node 104, and the computing node 106 are configured to execute the first instance 160, the second instance 162, and the third instance of the software application 122, respectively.
[0127] The method 700 also includes determining 700 whether a scheduled process of the first instance of the software application has stopped executing on the first computing node based on the process status data, at 704. For example, the status data analyzer 146 of FIG. 1 determines whether a scheduled process 107 of the first instance 160 of the software application 122 has stopped executing on the computing node 102 based on the process status data 105, as described with reference to FIG.
[0128] The method 700 further includes selectively restarting the first instance of the software application on the first computing node based on a determination that the scheduled process has stopped executing on the first computing node, at 706. For example, the status data analyzer 146 of FIG. 1 selectively restarts the first instance 160 of the software application 122 on the computing node 102 based on a determination that the scheduled process 107 has stopped executing on the computing node 102, as described with reference to FIG.
[0129] The method 700 also includes determining 700 whether a second instance of the software application has failed on a second compute node of the plurality of compute nodes, at 708. For example, the peer monitor 150 of FIG. 1 determines whether the second instance 162 of the software application 122 has failed on the compute node 104, as described with reference to FIG.
[0130] Method 700 further includes, at 710, performing an action based on the incomplete task associated with the second instance of the software application in response to determining that the second instance has failed on the second compute node. For example, peer monitor 150 of FIG. 1 performs action 128 based on incomplete task 109 associated with second instance 162 of software application 122 in response to determining that the second instance 162 of software application 122 has failed on compute node 104, as described with reference to FIG.
[0131] In this manner, the method 700 reduces downtime of the software application 122 on the compute node 102 by selectively restarting the first instance 160. The method 700 also enables the compute node 102 to address incomplete tasks 109 associated with a failed instance of the software application 122 on the compute node 104, thereby enabling recovery to a valid state.
[0132] 8 illustrates a block diagram of a computing environment 800 including a computing device 810 configured to support aspects of computer-implemented methods and computer-executable program instructions (or code) according to the present disclosure. For example, the computing device 810, or a portion thereof, is configured to execute instructions to initiate, perform, or control one or more of the operations described with reference to FIGS. 1-7. In one particular aspect, the computing device 810 corresponds to a computing node 102, a computing node 104, a computing node 106, a cluster managing device 110, or a combination thereof.
[0133] The computing device 810 includes a transceiver 822. The computing device 810 includes a processor 820. In one particular aspect, the processor 820 corresponds to the processor 134 of FIG. 1. In one particular aspect, the processor 820 includes a process monitor 140, a process monitor checker 142, a status data analyzer 146, a peer monitor 150, a peer monitor checker 152, or a combination thereof. The processor 820 is configured to communicate with a system memory 830, one or more storage devices 840, one or more input / output interfaces 850, one or more communication interfaces 860, or a combination thereof. The system memory 830 includes a volatile memory device (e.g., a random access memory (RAM) device), a non-volatile memory device (e.g., a read-only memory (ROM) device, a programmable read-only memory, and a flash memory), or both. System memory 830 stores operating system 832, which may include a basic input / output system for booting computing device 810, as well as a full operating system for allowing computing device 810 to interact with users, other programs, and other devices. System memory 830 stores system (program) data 836. In a particular aspect, memory 132 of FIG. 1 includes system memory 830, one or more storage devices 840, or a combination thereof.
[0134] System memory 830 includes one or more applications 834 executable by processor 820. As one example, one or more applications 834 include instructions 125 of FIGURE 1 executable by processor 820 to initiate, control, or perform one or more operations described with reference to FIGURES 1-7. In particular, one or more applications 834 include instructions 125 of FIGURE 1 executable by processor 820 to initiate, control, or perform one or more operations described with respect to process monitor 140, process monitor checker 142, status data analyzer 146, peer monitor 150, peer monitor checker 152, or combinations thereof.
[0135] The processor 820 is configured to communicate with one or more storage devices 840. For example, the one or more storage devices 840 include non-volatile storage devices such as magnetic disks, optical disks, or flash memory devices. In a particular embodiment, the storage devices 840 include both removable and non-removable memory devices. The storage devices 840 are configured to store an operating system, operating system images, applications, and program data. In a particular aspect, the system memory 830, the storage devices 840, or both include tangible computer readable media. In a particular aspect, one or more of the storage devices 840 are external to the computing device 810.
[0136] The processor 820 is configured to communicate with one or more input / output interfaces 850 that enable the computing device 810 to communicate with one or more input / output devices 870 to facilitate interaction with a user. The processor 820 is configured to detect an interaction event based on a user input received via the input / output interface 850. In one particular aspect, the input / output device 870 includes the display 114 of FIG. 1. The processor 820 is configured to send a display to the display 114 of FIG. 1 via the input / output interface 850. In one particular aspect, the input / output interface 850 includes the display interface 126 of FIG. 1. The processor 820 is configured to communicate with a device or controller 880 via one or more communication interfaces 860. For example, the one or more communication interfaces 860 include the network interface 130 of FIG. 1, and the device or controller 880 includes the cluster management device 110, the storage device 108, the computing node 104, the computing node 106, the network 190, or a combination thereof. In one embodiment, a non-transitory computer-readable storage medium (e.g., system memory 830) includes instructions that, when executed by a processor (e.g., processor 820), cause the processor to initiate, perform or control operations, including one or more of the operations described with reference to Figures 1-7.
[0137] Furthermore, the present disclosure includes embodiments according to the following clauses:
[0138] Clause 1. A system (100) for monitoring a process, comprising: The system includes a first computing node (102) and a second computing node (104) of a plurality of computing nodes (102, 104, 106) configured to execute a plurality of instances (160, 162) of a software application (122), the first computing node (102) comprising: a process monitor (140) configured to generate process status data (105) regarding a first instance (160) of a software application (122) executing on a first computing node (102); determining whether a scheduled process (107) of a first instance (160) of a software application (122) has stopped executing on the first computing node (102) based on the process status data (105); Selectively restarting a first instance (160) of a software application (122) based on a determination that a scheduled process (107) has stopped executing on a first computing node (102). A status data analyzer (146) configured to determining whether the process monitor (140) on the first compute node (102) has stopped executing; restarting the process monitor (140) in response to determining that the process monitor (140) has stopped executing on the first compute node (102); A process monitor checker (142) configured to determining whether a second instance (162) of the software application (122) has failed on a second computing node (104) of the plurality of computing nodes (102, 104, 106); In response to determining that the second instance (162) has failed on the second compute node (104), perform an action (128) based on an incomplete task (109) associated with the second instance (162) of the software application (122). Peer monitors configured to (150) Including, the system.
[0139] Clause 2. The process monitor (140) determining whether a process monitor checker (142) has stopped executing on the first compute node (102); In response to determining that the process monitor checker (142) has stopped executing on the first compute node (102), restarting the process monitor checker (142). 13. The system (100) of claim 1, further comprising a process monitor checker verifier (144) configured to:
[0140] Clause 3. The first computing node (102) determining whether the peer monitor (150) on the first compute node (102) has stopped executing; In response to determining that the peer monitor (150) has stopped executing on the first compute node (102), restarting the peer monitor (150). The system (100) of clause 1 or 2, further comprising a peer monitor checker (152) configured to:
[0141] Article 4. Peer monitors (150) determining whether a peer monitor checker (152) on the first compute node (102) has stopped executing; In response to determining that the peer monitor checker (152) has stopped executing on the first compute node (102), restarting the peer monitor checker (152). The system (100) of claim 2 or 3, further comprising a peer monitor checker verifier (154) configured to:
[0142] Clause 5. In response to the status data analyzer (146) determining that the scheduled process (107) has stopped executing on the first compute node (102), determining whether application restart criteria (111) are met; In response to determining that the application restart criteria (111) are satisfied, restarting the first instance (160) of the software application (122) on the first computing node (102). The system (100) according to any one of clauses 1-4,
[0143] Article 6. Status Data Analyzer (146) determining, based on the process status data (105), that a scheduled process (107) is detected as not running on a first computing node (102) at a first time (513); determining whether a difference between the first time (513) and a start time (511) indicating a time when the scheduled process (107) was previously started on the first computing node (102) is greater than or equal to a time threshold (515); resetting a fault counter (517) in response to determining that the difference is greater than or equal to the time threshold (515); Determine whether the application restart criteria (111) are met based on the failure counters (517). The system (100) according to any one of clauses 1-5,
[0144] Clause 7. After the status data analyzer (146) determines that a scheduled process (107) of a first instance (160) of a software application (122) has stopped executing on the first compute node (102), determining whether the fault counter (517) is greater than or equal to a fault count threshold (519); In response to determining that the failure counter (517) is less than the failure count threshold (519), determine that the application restart criteria (111) are satisfied. The system (100) according to any one of clauses 1-6,
[0145] Clause 8. The system (100) of any one of clauses 1-7, wherein the status data analyzer (146) is configured to determine that the application restart criterion (111) is not satisfied in response to determining that the failure counter (517) is greater than or equal to the failure count threshold (519).
[0146] Clause 9. The system (100) of any one of clauses 1-8, wherein the status data analyzer (146) is configured to inhibit a restart of the first instance (160) of the software application (122) on the first computing node (102) in response to determining that the application restart criteria (111) are not satisfied.
[0147] Clause 10. The system (100) of any one of clauses 1-9, further comprising a cluster management device (110) and a storage device (108), and configured to, in response to the status data analyzer (146) determining that the application restart criteria (111) are not satisfied, send a first notification (153) to the user device (112), send a second notification (151) to the cluster management device (110), send a third notification (157) to a third computing node (106) of the plurality of computing nodes (102, 104, 106), update log data (131) in the storage device (108) to indicate that the software application (122) has been shut down on the first computing node (102), or a combination thereof.
[0148] Clause 11. The system (100) of any one of clauses 1-10, wherein the status data analyzer (146) is configured to stop execution of the process monitor (140), the process monitor checker (142), and the peer monitor (150) on the first computing node (102) in response to determining that the application restart criteria (111) are not satisfied.
[0149] Clause 12. At a first computing node (102) of a plurality of computing nodes (102, 104, 106) configured to execute a plurality of instances (160, 162) of a software application (122), generating process status data (105) relating to a first instance (160) of the software application (122) executing on the first computing node (102); determining whether a scheduled process (107) of a first instance (160) of a software application (122) has stopped executing on the first computing node (102) based on the process status data (105); selectively restarting the first instance (160) of the software application (122) on the first computing node (102) based on a determination that the scheduled process (107) has stopped executing on the first computing node (102); determining whether a second instance (162) of the software application (122) has failed on a second computing node (104) of the plurality of computing nodes (102, 104, 106); and performing an action (128) based on an incomplete task (109) associated with the second instance (162) of the software application (122) in response to determining that the second instance (162) has failed on the second compute node (104); The method includes:
[0150] Clause 13. Receiving, at the first computing node (102), log data (141) associated with the second computing node (104) from the storage device (108); and Identifying incomplete tasks (109) based on log data (141) The method of clause 12 further comprising:
[0151] Clause 14. The method of clause 12 or 13, wherein performing the action (128) includes undoing the incomplete task (115) or completing the incomplete task (115).
[0152] Clause 15. The method of any one of clauses 12-14, wherein the incomplete task (109) includes a database update (115), and the action (128) includes performing a rollback of the database update (115) or performing the database update (115).
[0153] Clause 16. Receiving cluster process data (143) from a cluster management device (110); and determining that a second instance (162) of the software application (122) has failed on a second compute node (104) based on the cluster process data (143); 16. The method of any one of clauses 12-15, further comprising:
[0154] Clause 17. The method of any one of clauses 12-16, further comprising determining that the second instance (162) of the software application (122) has failed on the second compute node (104) in response to determining that the cluster process data (143) indicates that the second scheduled process (113) of the second instance (162) of the software application (122) has stopped executing on the second compute node (104).
[0155] Clause 18. Receiving, at the first computing node (102), log data (141) associated with the second computing node (104) from the storage device (108); and determining that a second instance (162) of the software application (122) has failed on the second computing node (104) in response to determining that the log data (141) indicates that the software application (122) has been shut down on the second computing node (104); 17. The method of any one of clauses 12-17, further comprising:
[0156] Clause 19. A computer-readable storage device (132) storing instructions, the instructions, when executed by the processor, causing the processor to: generating, at a first computing node (102) of a plurality of computing nodes (102, 104, 106) configured to execute a plurality of instances (160, 162) of a software application (122), process status data (105) relating to the first instance (160) of the software application (122) executing on the first computing node (102); determining whether a scheduled process (107) of a first instance (160) of a software application (122) has stopped executing on the first computing node (102) based on the process status data (105); selectively restarting the first instance (160) of the software application (122) on the first computing node (102) based on a determination that the scheduled process (107) has stopped executing on the first computing node (102); determining whether a second instance (162) of the software application (122) has failed on a second computing node (104) of the plurality of computing nodes (102, 104, 106); and performing an action (128) based on an incomplete task (109) associated with the second instance (162) of the software application (122) in response to determining that the second instance (162) has failed on the second compute node (104); A computer-readable storage device that causes operations to be performed, including:
[0157] Clause 20. The operation, in response to determining that the second instance (162) has failed on the second compute node (104), comprises: sending a notification (151) from the first computing node (102) to the cluster management device (110) indicating that the second instance (162) has failed on the second computing node (104); and receiving, at the first compute node (102), an assignment (145) for the incomplete task (109) from a cluster management device (110), wherein an action (128) is performed in response to receiving the assignment (145); The computer readable storage device (132) of clause 19, further comprising:
[0158] Clause 21. The operation In response to receiving the assignment from the cluster management device, sending a request to the storage device for log data associated with the second computing node; and Identifying incomplete tasks (109) based on log data (141) The computer readable storage device (132) of clause 19 or 20, further comprising:
[0159] Clause 22. The computer-readable storage device (132) of any one of clauses 19-21, wherein the operations further include sending process status data (105) of the first computing node (102) to a cluster management device (110) in response to determining that the second instance (162) has failed on the second computing node (104), and a notification (151) sent to the cluster management device (110) initiates sending a request (163) for the second process data (165) from the cluster management device (110) to a third computing node (106) of the plurality of computing nodes (102, 104, 106), and the allocation (145) is based on the process status data (105) and the second process status data (165).
[0160] Although one or more of Figures 1-8 illustrate systems, devices, and / or methods according to the teachings of the present disclosure, the present disclosure is not limited to these illustrated systems, devices, and / or methods. One or more functions or components shown in any of Figures 1-8 or described herein may be implemented in combination with other parts of one or more of the other figures of Figures 1-8. For example, one or more elements of method 700 of Figure 7 may be combined with other operations described herein. Thus, a single implementation described herein should not be construed as limiting, and implementations of the present disclosure may be appropriately combined without departing from the teachings of the present disclosure. As an example, one or more operations described with reference to Figures 1-8 are optional and may be implemented at least partially simultaneously and / or in a different order than that shown or described.
[0161] The above-described embodiments are illustrative and do not limit the present disclosure, and it will be understood that numerous modifications and variations are possible in accordance with the principles of the present disclosure.
[0162] The description of the embodiments described herein is intended to provide a general understanding of the structure of various implementations. Such description is not intended to exhaustively describe all of the elements and features of apparatus and systems that utilize the structures or methods described herein. Many other implementations will be apparent to those skilled in the art upon review of this disclosure. Other implementations can be utilized and derived from this disclosure, such that structural and logical substitutions and changes can be made without departing from the disclosure. For example, method operations can be performed in an order different from that shown in the figures, or one or more method operations can be omitted. Thus, the disclosure and drawings are illustrative rather than limiting.
[0163] Moreover, although specific embodiments have been illustrated and described herein, it should be recognized that subsequent configurations designed to accomplish the same or similar results may be substituted for the specific implementations illustrated. The present disclosure is intended to include any subsequent adaptations or variations of the various implementations. Combinations of the above implementations, as well as other implementations not specifically described herein, will be apparent to those of skill in the art upon review of the present disclosure.
[0164] The Abstract of the Disclosure is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. In addition, in the above Detailed Description, various features may be grouped together or described in a single implementation for the purpose of brevity of the disclosure. The above-described embodiments are illustrative and do not limit the disclosure. It should also be understood that numerous modifications and variations are possible in accordance with the principles of the disclosure. As set forth in the claims, the claimed subject matter may be directed to less than all features of any disclosed embodiment. The scope of the disclosure is therefore defined by the claims and their equivalents.
Claims
1. A system (100) for monitoring a process, comprising: A system comprising: a first computing node (102) and a second computing node (104) of a plurality of computing nodes (102, 104, 106) configured to execute a plurality of instances (160, 162) of a software application (122), said first computing node (102) comprising: a process monitor (140) configured to generate process status data (105) regarding a first instance (160) of the software application (122) executing on the first computing node (102); determining whether a scheduled process (107) of the first instance (160) of the software application (122) has stopped executing on the first computing node (102) based on the process status data (105); Selectively restarting the first instance (160) of the software application (122) based on a determination that the scheduled process (107) has stopped executing on the first computing node (102). a status data analyzer (146) configured to determining whether the process monitor (140) has stopped executing on the first compute node (102); restarting the process monitor (140) in response to determining that the process monitor (140) has stopped executing on the first compute node (102). A process monitor checker (142) configured to determining whether a second instance (162) of the software application (122) has failed on the second computing node (104) of the plurality of computing nodes (102, 104, 106); In response to determining that the second instance (162) has failed on the second computing node (104), perform an action (128) based on an incomplete task (109) associated with the second instance (162) of the software application (122). A peer monitor (150) configured to Including, the system.
2. The process monitor (140) determining whether the process monitor checker (142) has stopped executing on the first computing node (102); In response to determining that the process monitor checker (142) has stopped executing on the first compute node (102), restarting the process monitor checker (142). The system (100) of claim 1, further comprising a process monitor checker verifier (144) configured to:
3. The first computing node (102), determining whether the peer monitor (150) has stopped executing on the first compute node (102); restarting the peer monitor (150) in response to determining that the peer monitor (150) has stopped executing on the first compute node (102). The present invention further comprises a peer monitor checker (152) configured to: The peer monitor (150), determining whether the peer monitor checker (152) has stopped executing on the first computing node (102); restarting the peer monitor checker (152) in response to determining that the peer monitor checker (152) has stopped executing on the first computing node (102). The system (100) of claim 1 or 2, further comprising a peer monitor checker verifier (154) configured to:
4. In response to the status data analyzer (146) determining that the scheduled process (107) has stopped executing on the first computing node (102), determining whether the application restart criteria (111) are met; In response to determining that the application restart criteria (111) are satisfied, restarting the first instance (160) of the software application (122) on the first computing node (102). It is structured as follows: The status data analyzer (146) determining, based on the process status data (105), that the scheduled process (107) is detected as not running on the first computing node (102) at a first time (513); determining whether a difference between the first time (513) and a start time (511) indicating a time when the scheduled process (107) was previously started on the first computing node (102) is greater than or equal to a time threshold (515); resetting a fault counter (517) in response to determining that the difference is greater than or equal to the time threshold (515); determining whether the application restart criteria (111) are met based on the failure counters (517); determining whether said fault counter (517) is greater than or equal to a fault count threshold (519); In response to determining that the failure counter (517) is less than the failure count threshold (519), determining that the application restart criterion (111) is satisfied. The system (100) of claim 1 or 2, configured to:
5. 5. The system of claim 4, wherein the status data analyzer is configured to determine that the application restart criterion is not satisfied in response to determining that the failure counter is greater than or equal to the failure count threshold.
6. 6. The system (100) of claim 4 or 5, wherein the status data analyzer (146) is configured to, in response to determining that the application restart criterion (111) is not satisfied, inhibit a restart of the first instance (160) of the software application (122) on the first computing node (102) or stop execution of the process monitor (140), the process monitor checker (142), and the peer monitor (150) on the first computing node (102).
7. 7. The system (100) of claim 4, further comprising a cluster management device (110) and a storage device (108), and configured to, in response to the status data analyzer (146) determining that the application restart criterion (111) is not satisfied, send a first notification (153) to a user device (112), send a second notification (151) to the cluster management device (110), send a third notification (157) to a third computing node (106) of the plurality of computing nodes (102, 104, 106), update log data (131) on the storage device (108) to indicate that the software application (122) has been shut down on the first computing node (102), or a combination thereof.
8. generating, at a first computing node (102) of a plurality of computing nodes (102, 104, 106) configured to execute a plurality of instances (160, 162) of a software application (122), process status data (105) relating to a first instance (160) of the software application (122) executing on the first computing node (102); determining whether a scheduled process (107) of the first instance (160) of the software application (122) has stopped executing on the first computing node (102) based on the process status data (105); a status data analyzer (146) of the first computing node (102) selectively restarting the first instance (160) of the software application (122) on the first computing node (102) based on a determination that the scheduled process (107) has stopped executing on the first computing node (102); determining whether a second instance (162) of the software application (122) has failed on a second computing node (104) of the plurality of computing nodes (102, 104, 106); and performing an action (128) based on an incomplete task (109) associated with the second instance (162) of the software application (122) in response to determining that the second instance (162) has failed on the second computing node (104); A method comprising: Determining whether the second instance (162) of the software application (122) has failed on the second computing node (104) comprises: receiving cluster process data (143) from a cluster management device (110); and determining that the second instance (162) of the software application (122) on the second computing node (104) has failed based on the cluster process data (143); A method comprising:
9. receiving, at the first computing node (102), log data (141) associated with the second computing node (104) from a storage device (108); and Identifying the incomplete task (109) based on the log data (141). The method of claim 8 further comprising:
10. 10. The method of claim 8 or 9, wherein performing the action (128) comprises undoing an incomplete task (115) or completing the incomplete task (115), the incomplete task (109) comprising a database update (115), and the action (128) comprises performing a rollback of the database update (115) or making the database update (115). determining that the second instance (162) of the software application (122) has failed on the second compute node (104) in response to determining that the cluster process data (143) indicates that a second scheduled process (113) of the second instance (162) of the software application (122) has stopped executing on the second compute node (104). The method of any one of claims 8 to 10, further comprising:
12. receiving, at the first computing node (102), log data (141) associated with the second computing node (104) from a storage device (108); and determining that the second instance (162) of the software application (122) has failed on the second computing node (104) in response to determining that the log data (141) indicates that the software application (122) has been shut down on the second computing node (104); The method of any one of claims 8 to 11, further comprising:
13. A computer-readable storage device (132) storing instructions that, when executed by a processor, cause the processor to: generating, at a first computing node (102) of a plurality of computing nodes (102, 104, 106) configured to execute a plurality of instances (160, 162) of a software application (122), process status data (105) relating to a first instance (160) of the software application (122) executing on the first computing node (102); determining whether a scheduled process (107) of the first instance (160) of the software application (122) has stopped executing on the first computing node (102) based on the process status data (105); selectively restarting, by a status data analyzer (146) of the first computing node (102), the first instance (160) of the software application (122) on the first computing node (102) based on a determination that the scheduled process (107) has stopped executing on the first computing node (102); determining whether a second instance (162) of the software application (122) has failed on a second computing node (104) of the plurality of computing nodes (102, 104, 106); and In response to determining that the second instance (162) has failed on the second computing node (104), sending a notification (151) from the first computing node (102) to a cluster management device (110) indicating that the second instance (162) has failed on the second computing node (104); receiving, at the first computing node (102), from the cluster management device (110) an assignment (145) of an incomplete task (109) associated with the second instance (162) of the software application (122); and In response to receiving the assignment, performing an action based on the incomplete task. A computer readable storage device (132) for performing operations including:
14. The operation In response to receiving the assignment from the cluster management device, sending a request to a storage device for log data associated with the second computing node; and Identifying the incomplete task (109) based on the log data (141). The computer readable storage device (132) of claim 13, further comprising:
15. 15. The computer-readable storage device of claim 14, wherein the operations further include sending the process status data of the first computing node to the cluster management device in response to determining that the second instance has failed on the second computing node, and the notification sent to the cluster management device is for initiating sending a request for second process status data from the cluster management device to a third computing node of the plurality of computing nodes, and wherein the allocation is based on the process status data and the second process status data.
Citation Information
Patent Citations
Task monitoring system and its method
JP2003140925A
State monitoring device, state monitoring method, and state monitoring program
JP2015138512A
Electronic equipment and reboot program
JP2016207121A
Managing extrinsic processes
US20110041009A1
Fault monitor for restarting failed instances of the fault monitor
US6718486B1
