An application exception automatic isolation method, system, medium and device

Abstract

The application discloses an application exception automatic isolation method, system, medium and equipment, and belongs to the technical field of power grid dispatching. The method comprises the following steps: if any exception exists in the basic hardware and software exception flag, the node platform layer and application layer exception flag, and the cross-node platform layer and application layer exception flag, the application is set to a fault state and is automatically isolated; when the application returns to normal, the application priority is reduced to the lowest priority; when the application data of the current node and the application host node data are inconsistent or the operating system basic hardware and software index of the node where the application is located is in a warning state, the application priority is reduced to the lowest priority; and when the application stably runs for a preset time interval, the application data of the current node and the application host data are consistent, and the operating system basic hardware and software index is not in the warning state, the initial preset priority is restored. The application solves the problems that the application fault isolation is not complete, the system is prone to jitter, and the system cannot be smoothly recovered in the prior art.

Description

Technical Field

[0001] This invention relates to an application of an automatic isolation method, system, medium, and equipment for anomalies, belonging to the field of power grid dispatching technology. Background Technology

[0002] Currently, in the field of power grid dispatching, traditional application primary / backup management mainly relies on single-dimensional anomaly detection techniques such as network connectivity and process status. This method typically only performs detection at the local node, determining the application's running status by monitoring basic indicators such as whether the application process is alive and whether the network is reachable, and performing primary / backup failover based on statically configured priorities. When an application failure is detected at the local node, the system will trigger a backup-to-primary process, where the backup machine with the highest pre-configured priority will take over the services.

[0003] However, existing technologies have significant drawbacks in practical applications. First, due to their singular detection dimension, relying solely on the local node's processes and network status, they often fail to accurately detect fundamental operating system hardware and software anomalies such as disk failures, CPU exhaustion, and memory leaks. This results in applications being considered "normally running" even when the underlying environment is unreliable, leading to incomplete fault isolation. Second, anomaly detection is limited to the local node's perspective. When a serious failure occurs in the local node's application, such as kernel hangs or network partitions, preventing the transmission of heartbeats or anomaly information, the system cannot perform cross-node detection from a third-party perspective, thus missing the opportunity for isolation. Finally, the standby priority is statically configured and does not support dynamic adjustments based on comprehensive indicators such as data consistency and hardware health. This can lead to the highest-priority standby machine taking over services during primary / standby switching due to outdated data or hardware warnings on its node, potentially causing secondary system failures, service interruptions, or data errors, severely impacting the reliable and stable operation of the power grid dispatching system. Summary of the Invention

[0004] The purpose of this invention is to provide an application anomaly automatic isolation method, system, medium, and device. By monitoring for any anomaly among the basic hardware and software anomaly flags, the local node platform layer and application layer anomaly flags, and the cross-node platform layer and application layer anomaly flags, the application is set to a fault state and automatically isolated. Applications recovering from faults or network outages are downgraded in priority and smoothly restored, thereby solving the problems of incomplete application fault isolation, system jitter, and inability to smoothly restore in the prior art.

[0005] To solve the above-mentioned technical problems, the present invention is implemented using the following technical solution:

[0006] In a first aspect, the present invention provides an application anomaly automatic isolation method, comprising:

[0007] Real-time monitoring of operating system basic software and hardware metrics is performed to detect abnormal states of the basic software and hardware.

[0008] Based on the abnormal state of the basic software and hardware, the abnormal type is determined to obtain the abnormal flag of the basic software and hardware.

[0009] Perform offline and abnormal monitoring of the critical processes of this node to obtain abnormal flags at the platform layer of this node;

[0010] Monitor the application functions and application data of this node for anomalies and obtain the application layer anomaly flags of this node.

[0011] Anomalies are detected in key cross-node process functions, data acquisition functions, and data non-refresh functions to obtain cross-node platform layer and application layer anomaly flags.

[0012] If any of the following anomalies are present: basic hardware / software anomaly flag, local node platform layer anomaly flag, local node application layer anomaly flag, or cross-node platform layer and application layer anomaly flag, the application is set to a fault state and automatically isolated.

[0013] Based on the preset application priority, when an application recovers from a failure or network outage, the application priority is reduced to the lowest priority in the corresponding priority range within the system, resulting in the first priority adjustment result.

[0014] Based on the first priority adjustment result, when the application data of the current node and the application host node are inconsistent, or when the operating system basic software and hardware indicators of the application node are in a warning state, the application priority is reduced to the lowest priority of the application in the corresponding priority range within the system, and the second priority adjustment result is obtained.

[0015] Based on the second priority adjustment result, when the application runs stably for a configurable preset time interval, and the application data of the current node is consistent with the application host data and there are no operating system basic software and hardware indicators in a warning state, the application priority is restored to the initial preset priority.

[0016] Furthermore, the basic hardware and software indicators of the operating system include at least one of the following: system CPU, system memory, system average load, disk I / O, disk utilization, disk failure, network packet loss rate, and node D state.

[0017] Furthermore, real-time monitoring of the operating system's basic hardware and software metrics is performed to identify abnormal states of the basic hardware and software, including:

[0018] Real-time monitoring of the operating system's basic hardware and software metrics is performed to detect anomalies, and warning and fault states are used as the basis for identifying abnormal hardware and software states.

[0019] Set the basic operating system hardware and software indicators that exceed the alarm threshold to a warning state;

[0020] Set operating system basic hardware and software metrics that exceed the maximum threshold or fail to a fault state.

[0021] Furthermore, when the basic hardware and software anomaly flags are abnormal, a method is used to set the application to a fault state and automatically isolate it through a preset basic hardware and software anomaly judgment framework, including:

[0022] Based on the anomaly monitoring program of the operating system's basic software and hardware indicators, periodically check the operating system's basic software and hardware indicators.

[0023] When an anomaly is detected in the basic hardware and software indicators of the operating system, the basic hardware and software anomaly flag is set in the shared memory of the basic hardware and software anomaly judgment framework.

[0024] Based on the preset basic hardware and software anomaly judgment framework, the basic hardware and software anomaly flag information in the shared memory of the basic hardware and software anomaly judgment framework is periodically read.

[0025] When any basic hardware or software fault flag is present:

[0026] Based on the basic hardware and software exception flag information, set the basic hardware and software exception flag in the application state shared memory, and periodically read the basic hardware and software exception flag based on the application state management program.

[0027] The application status management program sets all applications on this node to a fault state and automatically isolates them.

[0028] Furthermore, when the platform layer anomaly flag or the application layer anomaly flag of this node is abnormal, a method is used to set the application to a fault state and automatically isolate it through a preset platform layer and application layer anomaly judgment framework, including:

[0029] The detection process is found to have at least one of the following abnormalities: abnormal startup function, abnormal critical process function, or abnormal data quality; or there is an abnormality at the platform layer and application layer of this node.

[0030] When an anomaly is detected at the platform layer of this node, the anomaly flag of this node's platform layer is set in the shared memory of the anomaly judgment framework of this node's platform layer and application layer.

[0031] When an application layer anomaly is detected in this node, the application layer anomaly flag for this node is set in the shared memory of the platform layer and application layer anomaly judgment framework of this node.

[0032] Based on the preset anomaly detection framework for the local node platform layer and application layer, the system periodically reads the local node platform layer anomaly flag information or the local node application layer anomaly flag information from the shared memory of the local node platform layer and application layer anomaly detection framework:

[0033] When any platform-level or application-level exception flag exists on this node:

[0034] Set the platform layer exception flag or the application layer exception flag in the application state shared memory of this node, and periodically read the platform layer exception flag or the application layer exception flag based on the application state management program.

[0035] The application is set to a fault state and automatically isolated based on the application status management program.

[0036] Furthermore, when anomalies are detected in the cross-node platform layer and application layer anomaly flags, a method is employed to set the application to a fault state and automatically isolate it using a pre-defined cross-node platform layer and application layer anomaly judgment framework. This method includes:

[0037] Detect at least one cross-node platform layer and application layer anomaly in critical process functions, data acquisition functions, and data non-refresh functions;

[0038] When cross-node platform layer and application layer anomalies are detected, the cross-node platform layer and application layer anomaly flags are set in the shared memory of the cross-node platform layer and application layer anomaly judgment framework.

[0039] Based on the preset cross-node platform layer and application layer anomaly judgment framework, the cross-node platform layer and application layer anomaly flag information is read periodically;

[0040] Using a pre-defined cross-node platform layer and application layer exception detection framework, exception information from cross-node platform layer and application layer exception flags is distributed to all nodes in the system. When any cross-node platform layer or application layer exception flag exists:

[0041] Set cross-node platform layer and application layer exception flags in the application state shared memory, and periodically read the cross-node platform layer and application layer exception flags based on the application state management program;

[0042] The application status management program sets the corresponding application of each node to a fault state and automatically isolates it.

[0043] Furthermore, the preset basic hardware and software anomaly judgment framework also supports viewing the status of various basic hardware and software indicators of the operating system on all nodes, including normal and abnormal statuses.

[0044] In a second aspect, the present invention provides an automatic isolation system for application anomalies, comprising:

[0045] The basic hardware and software anomaly monitoring module is used to monitor real-time data of basic hardware and software indicators of the operating system and obtain the abnormal status of basic hardware and software.

[0046] The basic software and hardware anomaly flag determination module is used to determine the anomaly type based on the anomaly status of the basic software and hardware, and obtain the basic software and hardware anomaly flag.

[0047] The platform layer anomaly flag acquisition module of this node is used to perform offline and anomaly monitoring of the key processes of this node and obtain the platform layer anomaly flag of this node.

[0048] The application layer anomaly flag acquisition module of this node is used to monitor the application functions and application data of this node for anomalies and obtain the application layer anomaly flag of this node.

[0049] The cross-node platform layer and application layer exception flag acquisition module is used to perform exception judgment on key process functions, data acquisition functions, and data non-refresh functions across nodes, and obtain cross-node platform layer and application layer exception flags.

[0050] The application automatic isolation module is used to set the application to a fault state and automatically isolate it if any of the following abnormalities exist: basic software and hardware abnormality flags, local node platform layer abnormality flags, local node application layer abnormality flags, or cross-node platform layer and application layer abnormality flags.

[0051] The first priority adjustment module is used to reduce the application priority to the lowest priority in the corresponding priority range within the system when the application recovers from a failure or network outage, based on the preset application priority, and obtain the first priority adjustment result.

[0052] The second priority adjustment module is used to adjust the application priority to the lowest priority in the corresponding priority range within the system based on the first priority adjustment result, when the application data of the current node and the application host node are inconsistent, or when the operating system basic software and hardware indicators of the application node are in a warning state.

[0053] The priority recovery module is used to restore the application priority to the preset initial priority based on the second priority adjustment result. After the application has been running stably for a configurable preset time interval, and the application data of the current node is consistent with the application host data and there are no operating system basic software and hardware indicators in a warning state.

[0054] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the application anomaly automatic isolation method as described in the first aspect.

[0055] Fourthly, the present invention provides a computer device, comprising:

[0056] Memory, used to store instructions;

[0057] A processor for executing the instructions, causing the device to perform operations implementing the application exception automatic isolation method as described in the first aspect.

[0058] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:

[0059] 1. This invention achieves comprehensive, three-dimensional anomaly detection from the basic environment, single-node platform, single-node application to cross-node interaction by monitoring real-time data of basic operating system hardware and software indicators, monitoring key processes of the local node for both offline and anomalies, monitoring application functions and application data of the local node for anomalies, and judging anomalies in key process functions, data acquisition functions, and data non-refresh functions across nodes. This ensures that when an anomaly occurs in any link, the application can be set to a fault state in a timely manner and automatically isolated, thereby effectively preventing the spread of anomalies, significantly improving the overall stability and self-healing capability of the system, and solving the problems of incomplete application fault isolation, system jitter and inability to recover smoothly in the prior art.

[0060] 2. This invention constructs a complete detection system from infrastructure, single-machine platform, single-machine application to distributed collaboration by performing real-time anomaly monitoring of operating system basic hardware and software indicators, offline and anomaly monitoring of key processes of the local node, anomaly monitoring of application functions and application data of the local node, and anomaly judgment of key process functions, data acquisition functions, and data non-refresh functions across nodes. It can capture subtle faults at any level.

[0061] 3. This invention achieves refined perception of basic environmental issues by monitoring real-time data of operating system basic hardware and software indicators for anomalies and distinguishing between warning states and fault states. When a fault occurs, the collaborative work of the basic hardware and software anomaly judgment framework, the local node platform layer and application layer anomaly judgment framework, and the cross-node platform layer and application layer anomaly judgment framework can accurately locate the source of the anomaly and trigger isolation, avoiding misjudging warning jitter as fatal faults and improving system stability.

[0062] 4. After an application is isolated and returns to normal, this invention reduces the application priority to the lowest priority within the corresponding priority range in the system, and restores the application priority to the initial preset priority after data consistency and basic hardware and software indicators have stabilized. This effectively prevents the newly restored application from crashing again due to immediately bearing a high load, and provides a smooth recovery buffer period for the system. Attached Figure Description

[0063] Figure 1 This is a flowchart illustrating an application anomaly automatic isolation method provided in an embodiment of the present invention;

[0064] Figure 2This is a priority adjustment diagram provided in an embodiment of the present invention.

[0065] Figure 3 This is a schematic diagram of the operating system basic software and hardware anomaly monitoring service process provided in an embodiment of the present invention. Detailed Implementation

[0066] It should be noted that the "priority" mentioned in the embodiments of this invention is a numerical attribute of each application; the smaller the value, the higher the priority. When an application has no other host and the local application has the highest priority, the local application backup will automatically switch to the host state.

[0067] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations thereof. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0068] The term "and / or" simply describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0069] Example 1

[0070] like Figure 1 As shown in the figure, this embodiment introduces an application anomaly automatic isolation method, including:

[0071] Step 1: Monitor the real-time data of the operating system's basic hardware and software indicators to obtain the abnormal status of the basic hardware and software.

[0072] This embodiment incorporates underlying metrics such as system CPU, memory, disk I / O, and network packet loss rate into the monitoring scope, solving the problem that traditional methods only focus on application process status while ignoring the health of underlying hardware, and providing a comprehensive data foundation for subsequent accurate judgment.

[0073] Step 2: Determine the type of abnormality based on the abnormal state of the basic software and hardware to obtain the abnormality flag of the basic software and hardware.

[0074] This embodiment determines the anomaly type based on the abnormal state of basic software and hardware, obtains the anomaly flag of basic software and hardware, and specifically sets indicators that exceed the alarm threshold to the warning state, and indicators that exceed the maximum threshold and / or are faulty to the fault state. This realizes the fine classification of basic software and hardware anomalies and provides a clear basis for distinguishing between warnings that "require downgrade processing" and faults that "require immediate isolation".

[0075] Step 3: Perform offline and abnormal monitoring of the critical processes of this node to obtain the abnormal flags of the platform layer of this node.

[0076] This embodiment performs offline and anomaly monitoring on the key processes of this node. Offline monitoring of key processes can capture abnormal states such as zombie and exit of the core processes inside the node in real time. Anomaly monitoring of key processes is carried out by key processes under each application, such as message bus processes, to determine whether the functions are normal. When an anomaly is detected, an anomaly flag is accurately generated at the platform layer of this node, thereby ensuring the reliability and controllability of the node's operating environment and providing a solid platform guarantee for the stable operation of upper-layer applications.

[0077] Step 4: Monitor the application functions and application data of this node for anomalies and obtain the application layer anomaly flags of this node.

[0078] This embodiment monitors the application functions and data of this node for anomalies, enabling in-depth detection of faults at the business logic level, such as service unresponsiveness or data read / write errors. It accurately outputs application layer anomaly flags for this node, achieving precise perception of the application's own health status and effectively preventing defective applications from continuing to provide services.

[0079] Step 5: Perform anomaly checks on key cross-node process functions, data acquisition functions, and data refresh non-refresh functions to obtain cross-node platform layer and application layer anomaly flags.

[0080] This embodiment effectively detects complex faults in distributed systems, such as network partitions, data synchronization interruptions, or peer service failures, by performing anomaly detection on key cross-node process functions, data acquisition functions, and data non-refresh functions. It also generates cross-node platform layer and application layer anomaly flags, thereby overcoming the limitations of single-node monitoring and ensuring data consistency and service collaboration capabilities of the entire cluster system.

[0081] Step Six: If any of the following anomalies are present: basic hardware / software anomaly flag, local node platform layer anomaly flag, local node application layer anomaly flag, or cross-node platform layer and application layer anomaly flag, then set the application to a fault state and automatically isolate it.

[0082] This embodiment achieves logical aggregation of multi-dimensional monitoring results by determining whether any of the following anomalies exist: basic hardware and software anomaly flags, local node platform layer anomaly flags, local node application layer anomaly flags, and cross-node platform layer and application layer anomaly flags. Once an anomaly is detected, the application is immediately set to a fault state and automatically isolated, thereby cutting off the path of fault propagation and maximizing the availability of the overall system and the continuity of business.

[0083] Step 7: Based on the preset application priority, when the application recovers from a failure or network outage, the application priority is reduced to the lowest priority in the corresponding priority range within the system, thus obtaining the first priority adjustment result.

[0084] This embodiment, based on a preset application priority, reduces the application priority to the lowest priority within the corresponding priority range in the system when the application recovers from a fault or network outage, thus obtaining the first priority adjustment result. This introduces a "soft start" protection period for the newly recovered application, preventing it from crashing again due to immediately bearing high load, effectively reducing the probability of system jitter.

[0085] Step 8: Based on the first priority adjustment result, when the application data of the current node and the application host node are inconsistent, or when the operating system basic software and hardware indicators of the application node are in a warning state, the application priority is reduced to the lowest priority of the application in the corresponding priority range within the system, and the second priority adjustment result is obtained.

[0086] This embodiment, based on the first priority adjustment result, reduces the application priority to the lowest priority within the corresponding priority range of the system when the application data of the current node and the application host node are inconsistent, or when the operating system's basic hardware and software indicators on the application node are in a warning state. This yields the second priority adjustment result, which is further combined with data consistency and hardware health for a second downgrade. This ensures that the application can only be promoted to host when the data is up-to-date and the node is healthy, thus avoiding switching failures caused by data inconsistency or sub-healthy nodes.

[0087] Step 9: Based on the second priority adjustment result, when the application runs stably for a configurable preset time interval, and the application data of the current node is consistent with the application host data and there are no operating system basic software and hardware indicators in a warning state, restore the application priority to the initial preset priority.

[0088] This embodiment adjusts the application priority to the initial preset priority based on the second priority adjustment result. After the application has been running stably for a configurable preset time interval, and the application data of the current node is consistent with the application host data and there are no warning indicators of the operating system's basic software and hardware indicators, the application priority is restored to the initial preset priority. This sets a complete observation period for the application. Only after the application has been running stably for a long time and has met all health conditions will its initial priority be restored, which fundamentally improves the long-term operational reliability of the system.

[0089] Example 2

[0090] Based on the same inventive concept as Embodiment 1, this embodiment describes the implementation steps of an automatic anomaly isolation method, including:

[0091] Step 1: Monitor the real-time data of the operating system's basic hardware and software indicators to obtain the abnormal status of the basic hardware and software.

[0092] Step 1.1: Monitor the real-time data of the basic software and hardware indicators of the operating system for anomalies, and use the warning status and fault status as the abnormal status of the basic software and hardware.

[0093] Step 1.2: Set the basic operating system hardware and software indicators that exceed the alarm threshold to a warning state.

[0094] Step 1.3: Set the basic operating system hardware and software indicators that exceed the maximum threshold or are faulty to a fault state.

[0095] In this embodiment, the basic hardware and software indicators of the operating system include at least one of the following: system CPU, system memory, system average load, disk I / O, disk utilization, disk failure, network packet loss rate, and node D status.

[0096] The basic hardware and software anomaly monitoring service first reads the alarm threshold and maximum threshold of the corresponding anomaly indicator from the local configuration file during program initialization, and obtains the real-time data of the corresponding basic hardware and software indicator through the corresponding interface provided by the operating system. When the real-time data of the corresponding operating system basic hardware and software indicator exceeds the alarm threshold N times consecutively, the status of the corresponding operating system basic hardware and software indicator in the anomaly status information shared memory is set to warning and a corresponding indicator over-warning alarm is issued; when the real-time data of the corresponding operating system basic hardware and software indicator exceeds the maximum threshold N times consecutively, the status of the corresponding operating system basic hardware and software indicator in the anomaly status information shared memory is set to fault and a corresponding indicator anomaly alarm is issued. Figure 3 The diagram shown is a schematic of the operating system basic software and hardware anomaly monitoring service process provided in an embodiment of the present invention.

[0097] Step 2: Determine the type of abnormality based on the abnormal state of the basic software and hardware to obtain the abnormality flag of the basic software and hardware.

[0098] Step 3: Perform offline and abnormal monitoring of the critical processes of this node to obtain the abnormal flags of the platform layer of this node.

[0099] Step 4: Monitor the application functions and application data of this node for anomalies and obtain the application layer anomaly flags of this node.

[0100] Step 5: Perform anomaly checks on key cross-node process functions, data acquisition functions, and data non-refresh functions to obtain cross-node platform layer and application layer anomaly flags.

[0101] Specifically, the abnormal status information shared memory stores the abnormal information detected by the abnormal monitoring modules at the basic hardware and software, platform level, and application level. The abnormal flag bits occupy a total of 64 bits and are divided into 5 fault types: basic hardware and software abnormal flag, local node platform layer abnormal flag, local node application layer abnormal flag, and cross-node platform layer and application layer abnormal flag. The 5-bit fault flag bits of the application status information are set according to the 64-bit abnormal flag bits of the abnormal status information.

[0102] Step 6: If any of the following anomalies are present: basic hardware / software anomaly flag, local node platform layer anomaly flag, local node application layer anomaly flag, or cross-node platform layer and application layer anomaly flag, then set the application to a fault state and automatically isolate it.

[0103] The application status management program determines whether an application is normal or faulty based on a 5-bit fault flag in the application status information. The program retrieves basic hardware and software anomaly information from the shared memory of the application status, performs the final application fault setting, and switches or dynamically adjusts application priorities. When the basic hardware and software anomaly flag for the corresponding application in the shared memory is faulty, the application status of this node is set to faulty through the application status management program, thus obtaining the application fault status of this node.

[0104] The application status management program periodically checks the application status of each application, obtains the basic hardware and software fault flags, the local node application fault flags, and the cross-node application fault flags. If any fault flag is found, the application status is set to fault. When no fault flag is found, the application status is kept as master or standby.

[0105] Step 6.1: When there is an anomaly in the basic software and hardware anomaly flag, the application is set to a fault state and automatically isolated using the preset basic software and hardware anomaly judgment framework.

[0106] In this embodiment, the preset basic software and hardware anomaly judgment framework also supports viewing the status of various basic software and hardware indicators of the operating system of all nodes. The status of the basic software and hardware indicators includes normal status and abnormal status.

[0107] Step 6.1.1: Periodically check the basic software and hardware indicators of the operating system according to the anomaly monitoring program of the basic software and hardware indicators of the operating system.

[0108] Step 6.1.2: When an anomaly is detected in the basic hardware and software indicators of the operating system, the basic hardware and software anomaly flag is set in the shared memory of the basic hardware and software anomaly judgment framework.

[0109] Step 6.1.3: Based on the preset basic hardware and software anomaly judgment framework, periodically read the basic hardware and software anomaly flag information in the shared memory of the basic hardware and software anomaly judgment framework.

[0110] Step 6.1.4: When any basic hardware or software exception flag exists:

[0111] Based on the basic hardware and software exception flag information, the basic hardware and software exception flag in the application state shared memory is set, and the basic hardware and software exception flag is read periodically based on the application state management program; based on the application state management program, all applications of this node are set to a fault state and automatically isolated.

[0112] Step 6.2: When there is an anomaly in the platform layer anomaly flag or the application layer anomaly flag of this node, the application is set to a fault state and automatically isolated by using the preset platform layer and application layer anomaly judgment framework of this node.

[0113] Step 6.2.1: Detect at least one of the following abnormalities in the local node's platform layer and application layer: abnormal process startup function, abnormal critical process function, and abnormal data quality.

[0114] Step 6.2.2: When an anomaly is detected at the platform layer of this node, set the anomaly flag of the platform layer of this node to the shared memory of the anomaly judgment framework of the platform layer and the application layer of this node.

[0115] Step 6.2.3: When an application layer anomaly is detected in this node, set the application layer anomaly flag of this node to the shared memory of the platform layer and application layer anomaly judgment framework of this node.

[0116] Step 6.2.4: Based on the preset anomaly judgment framework of the local node platform layer and application layer, periodically read the local node platform layer anomaly flag information or local node application layer anomaly flag information in the shared memory of the local node platform layer and application layer anomaly judgment framework.

[0117] Step 6.2.4: When any platform-level or application-level anomaly flag exists on this node:

[0118] Set the local node platform layer exception flag or local node application layer exception flag in the application status shared memory; periodically read the local node platform layer exception flag or local node application layer exception flag based on the application status management program; set the application to a fault state and automatically isolate it based on the application status management program.

[0119] Step 6.3: When there are abnormalities in the cross-node platform layer and application layer exception flags, the application is set to a fault state and automatically isolated by using the preset cross-node platform layer and application layer exception judgment framework.

[0120] Step 6.3.1: Detect at least one cross-node platform layer and application layer anomaly in the following functions: critical process function, data acquisition function, and data non-refresh function;

[0121] Step 6.3.2: When an anomaly is detected across the platform layer and application layer, set the cross-node platform layer and application layer anomaly flags in the shared memory of the cross-node platform layer and application layer anomaly judgment framework;

[0122] Step 6.3.3: Based on the preset cross-node platform layer and application layer anomaly judgment framework, periodically read the cross-node platform layer and application layer anomaly flag information;

[0123] Step 6.3.4: Using the pre-defined cross-node platform layer and application layer exception judgment framework, distribute the exception information of the cross-node platform layer and application layer exception flags to all nodes in the system. When any cross-node platform layer or application layer exception flag exists:

[0124] Configure cross-node platform layer and application layer exception flags in the application state shared memory, and periodically read the cross-node platform layer and application layer exception flags based on the application state management program; set the corresponding application of each node to a fault state and automatically isolate it based on the application state management program.

[0125] Step 7: Based on the preset application priority, when the application recovers from a fault or network outage, the application priority is reduced to the lowest priority in the corresponding priority range within the system, thus obtaining the first priority adjustment result.

[0126] like Figure 2 The diagram shown is a priority adjustment schematic provided in an embodiment of the present invention.

[0127] Step 8: Based on the first priority adjustment result, when the application data of the current node and the application host node are inconsistent, or when the operating system basic software and hardware indicators of the application node are in a warning state, the application priority is reduced to the lowest priority of the application in the corresponding priority range within the system, and the second priority adjustment result is obtained.

[0128] When the operating system's basic hardware and software indicators on the node hosting the application are in a warning state, the application priority value range is increased by a second priority range N2 based on the first priority range. When the application data on the current node is inconsistent with the data on the application host node, the application priority value range is increased by a third priority range N3 based on the first priority range. When both of the above situations exist simultaneously, the application priority value range is increased by N2 + N3 based on the first priority range. Finally, the maximum application priority value across all nodes within the corresponding application priority value range is taken plus 1 as the current application priority, set in the application management shared memory, and synchronized to all other nodes.

[0129] Step 9: Based on the second priority adjustment result, after the application has been running stably for a configurable preset time interval, and the application data of the current node is consistent with the application host data and there are no operating system basic software and hardware indicators in a warning state, restore the application priority to the initial preset priority.

[0130] Once the application has been running stably for a configurable preset time interval, and simultaneously the application data on the current node and the application host data are consistent, and there are no operating system hardware or software indicators in a warning state, the application priority will be restored to the initial preset priority. If the above conditions are not met, the application's current priority will remain unchanged.

[0131] To more clearly illustrate the priority dynamic adjustment mechanism of the present invention, this embodiment is illustrated by the following specific scenarios:

[0132] Scenario 1: There are four nodes, A, B, C, and D, with initial application priorities of 1, 2, 3, and 4, respectively.

[0133] Node A has just recovered from a failure or network outage. At this time, the priority value range of Node A is adjusted to N1, set to 1000. The priority of Node A is then updated to 1001, which is the maximum value in the range, 1000 plus 1.

[0134] The application data on node C is inconsistent with the application host. At this time, the application priority value range of node C is adjusted to N3, set to 20000, and the application priority of node C is updated to 20001, which is the maximum value in the range 20000 plus 1.

[0135] Subsequently, node B recovers from the fault or network outage state. At this time, the application priority of node B is updated to 1002, which is the current maximum value in the interval, 1001, plus 1.

[0136] After node A has been running stably for half an hour, the application priority is restored to the initial value of 1.

[0137] After running stably for half an hour, node C's application priority was restored to its initial value of 3.

[0138] Scenario 2: There are four nodes, A, B, C, and D, with initial application priorities of 1, 2, 3, and 4, respectively.

[0139] If node A has basic hardware and software warning signs and the application data on node A is inconsistent with the application host, then the application priority value range of node A is adjusted to N2+N3, where N2 is set to 5000 and N3 is set to 20000. The application priority of node A is then updated to 25001.

[0140] The application data on node C is inconsistent with the application host. At this time, the application priority value range of node C is adjusted to N3, and the application priority of node C is updated to 20001.

[0141] Subsequently, node B also experienced inconsistencies between application data and the application host. The application priority value range of node B was adjusted to N3, and the application priority of node B was updated to 20002.

[0142] Subsequently, the application data on node A and the application host were restored to consistency, but basic software and hardware warning signs still existed. At this time, the application priority value range of node A was adjusted to N2, and the application priority of node A was updated to 5001.

[0143] Subsequently, once the basic hardware and software warning flags for node A are no longer present, the application priority is restored to its initial value of 1.

[0144] Example 3

[0145] Based on the same inventive concept as other embodiments, this embodiment introduces an application anomaly automatic isolation system, including:

[0146] The basic hardware and software anomaly monitoring module is used to monitor real-time data of basic hardware and software indicators of the operating system and obtain the abnormal status of basic hardware and software.

[0147] The basic software and hardware anomaly flag determination module is used to determine the anomaly type based on the anomaly status of the basic software and hardware, and obtain the basic software and hardware anomaly flag.

[0148] The platform layer anomaly flag acquisition module of this node is used to perform offline and anomaly monitoring of the key processes of this node and obtain the platform layer anomaly flag of this node.

[0149] The application layer anomaly flag acquisition module of this node is used to monitor the application functions and application data of this node for anomalies and obtain the application layer anomaly flag of this node.

[0150] The cross-node platform layer and application layer exception flag acquisition module is used to perform exception judgment on key process functions, data acquisition functions, and data non-refresh functions across nodes, and obtain cross-node platform layer and application layer exception flags.

[0151] The application automatic isolation module is used to set the application to a fault state and automatically isolate it if any of the following abnormalities exist: basic software and hardware abnormality flags, local node platform layer abnormality flags, local node application layer abnormality flags, or cross-node platform layer and application layer abnormality flags.

[0152] The first priority adjustment module is used to reduce the application priority to the lowest priority in the corresponding priority range within the system when the application recovers from a failure or network outage, based on the preset application priority, and obtain the first priority adjustment result.

[0153] The second priority adjustment module is used to adjust the application priority to the lowest priority in the corresponding priority range within the system based on the first priority adjustment result, when the application data of the current node and the application host node are inconsistent, or when the operating system basic software and hardware indicators of the application node are in a warning state.

[0154] The priority recovery module is used to restore the application priority to the preset initial priority based on the second priority adjustment result. After the application has been running stably for a configurable preset time interval, and the application data of the current node is consistent with the application host data and there are no operating system basic software and hardware indicators in a warning state.

[0155] The specific functions of each module described above are explained in the relevant content of Embodiment 1 or 2, and will not be repeated here.

[0156] Example 4

[0157] Based on the same inventive concept as other embodiments, this embodiment introduces a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the steps of the application anomaly automatic isolation method described in Embodiment 1 or 2 above.

[0158] Example 5

[0159] Based on the same inventive concept as other embodiments, this embodiment describes a computer device, including:

[0160] Memory, used to store instructions;

[0161] A processor is configured to execute the instructions, causing the device to perform operations that implement the application anomaly automatic isolation method as described in Embodiment 1 or 2.

[0162] In summary, this invention achieves comprehensive, three-dimensional anomaly detection from the basic environment, single-node platform, single-node application to cross-node interaction by monitoring real-time data of basic operating system hardware and software indicators, monitoring critical processes of the local node for both offline and anomalies, monitoring application functions and application data of the local node for anomalies, and judging anomalies in critical process functions, data acquisition functions, and data non-refresh functions across nodes. This ensures that when an anomaly occurs in any link, the application can be set to a fault state in a timely manner and automatically isolated, thereby effectively preventing the spread of anomalies, significantly improving the overall stability and self-healing capability of the system, and solving the problems of incomplete application fault isolation, system jitter, and inability to recover smoothly in the prior art.

[0163] This invention constructs a complete detection system from infrastructure, single-machine platform, single-machine application to distributed collaboration by performing real-time anomaly monitoring of basic operating system hardware and software indicators, offline and anomaly monitoring of key processes on the local node, anomaly monitoring of application functions and application data on the local node, and anomaly judgment of key process functions, data acquisition functions, and data non-refresh functions across nodes. It can capture subtle faults at any level.

[0164] This invention achieves refined perception of fundamental environmental issues by monitoring real-time data of basic operating system hardware and software metrics for anomalies and distinguishing between warning and fault states. When a fault occurs, the collaborative work of the basic hardware and software anomaly judgment framework, the local node platform layer and application layer anomaly judgment framework, and the cross-node platform layer and application layer anomaly judgment framework can accurately locate the source of the anomaly and trigger isolation, avoiding misjudging warning jitter as fatal faults and improving system stability.

[0165] After an application is isolated and returns to normal, this invention reduces the application priority to the lowest priority within the corresponding priority range in the system, and restores the application priority to the initial preset priority after data consistency and basic hardware and software indicators stabilize. This effectively prevents the newly restored application from crashing again due to immediately bearing a high load, and provides a smooth recovery buffer period for the system.

[0166] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0167] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0168] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0169] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0170] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. An application of automatic anomaly isolation method, characterized in that, include: Real-time monitoring of operating system basic software and hardware metrics is performed to detect abnormal states of the basic software and hardware. Based on the abnormal state of the basic software and hardware, the abnormal type is determined to obtain the abnormal flag of the basic software and hardware. Perform offline and abnormal monitoring of the critical processes of this node to obtain abnormal flags at the platform layer of this node; Monitor the application functions and application data of this node for anomalies and obtain the application layer anomaly flags of this node. Anomalies are detected in key cross-node process functions, data acquisition functions, and data non-refresh functions to obtain cross-node platform layer and application layer anomaly flags. If any of the following anomalies are present: basic hardware / software anomaly flag, local node platform layer anomaly flag, local node application layer anomaly flag, or cross-node platform layer and application layer anomaly flag, the application is set to a fault state and automatically isolated. Based on the preset application priority, when an application recovers from a failure or network outage, the application priority is reduced to the lowest priority in the corresponding priority range within the system, resulting in the first priority adjustment result. Based on the first priority adjustment result, when the application data of the current node and the application host node are inconsistent, or when the operating system basic software and hardware indicators of the application node are in a warning state, the application priority is reduced to the lowest priority of the application in the corresponding priority range within the system, and the second priority adjustment result is obtained. Based on the second priority adjustment result, when the application runs stably for a configurable preset time interval, and the application data of the current node is consistent with the application host data and there are no operating system basic software and hardware indicators in a warning state, the application priority is restored to the initial preset priority.

2. The application anomaly automatic isolation method according to claim 1, characterized in that, The basic hardware and software metrics of the operating system include at least one of the following: system CPU, system memory, system average load, disk I / O, disk utilization, disk failure, network packet loss rate, and node D status.

3. The application anomaly automatic isolation method according to claim 2, characterized in that, Real-time monitoring of operating system basic hardware and software metrics reveals abnormal states of these components, including: Real-time monitoring of the operating system's basic hardware and software metrics is performed to detect anomalies, and warning and fault states are used as the basis for identifying abnormal hardware and software states. Set the basic operating system hardware and software indicators that exceed the alarm threshold to a warning state; Set operating system basic hardware and software metrics that exceed the maximum threshold or fail to a fault state.

4. The application anomaly automatic isolation method according to claim 3, characterized in that, When an anomaly is detected in the basic hardware and software anomaly flags, the application is set to a fault state and automatically isolated using a preset basic hardware and software anomaly judgment framework, including: Based on the anomaly monitoring program of the operating system's basic software and hardware indicators, periodically check the operating system's basic software and hardware indicators. When an anomaly is detected in the basic hardware and software indicators of the operating system, the basic hardware and software anomaly flag is set in the shared memory of the basic hardware and software anomaly judgment framework. Based on the preset basic hardware and software anomaly judgment framework, the basic hardware and software anomaly flag information in the shared memory of the basic hardware and software anomaly judgment framework is periodically read. When any basic hardware or software fault flag is present: Based on the basic hardware and software exception flag information, set the basic hardware and software exception flag in the application state shared memory, and periodically read the basic hardware and software exception flag based on the application state management program. The application status management program sets all applications on this node to a fault state and automatically isolates them.

5. The application anomaly automatic isolation method according to claim 4, characterized in that, When an anomaly is detected in either the platform layer anomaly flag or the application layer anomaly flag of this node, the application is set to a fault state and automatically isolated using a pre-defined platform layer and application layer anomaly detection framework. This includes: The detection process is found to have at least one of the following abnormalities: abnormal startup function, abnormal critical process function, or abnormal data quality; or there is an abnormality at the platform layer and application layer of this node. When an anomaly is detected at the platform layer of this node, the anomaly flag of this node's platform layer is set in the shared memory of the anomaly judgment framework of this node's platform layer and application layer. When an application layer anomaly is detected in this node, the application layer anomaly flag for this node is set in the shared memory of the platform layer and application layer anomaly judgment framework of this node. Based on the preset anomaly detection framework for the local node platform layer and application layer, the system periodically reads the local node platform layer anomaly flag information or the local node application layer anomaly flag information from the shared memory of the local node platform layer and application layer anomaly detection framework: When any platform-level or application-level exception flag exists on this node: Set the platform layer exception flag or the application layer exception flag in the application state shared memory of this node, and periodically read the platform layer exception flag or the application layer exception flag based on the application state management program. The application is set to a fault state and automatically isolated based on the application status management program.

6. The application anomaly automatic isolation method according to claim 5, characterized in that, When anomalies are detected across the cross-node platform layer and application layer, a pre-defined cross-node platform layer and application layer anomaly detection framework is used to set the application to a fault state and automatically isolate it, including: Detect at least one cross-node platform layer and application layer anomaly in critical process functions, data acquisition functions, and data non-refresh functions; When cross-node platform layer and application layer anomalies are detected, the cross-node platform layer and application layer anomaly flags are set in the shared memory of the cross-node platform layer and application layer anomaly judgment framework. Based on the preset cross-node platform layer and application layer anomaly judgment framework, the cross-node platform layer and application layer anomaly flag information is read periodically; Using a pre-defined cross-node platform layer and application layer exception detection framework, exception information from cross-node platform layer and application layer exception flags is distributed to all nodes in the system. When any cross-node platform layer or application layer exception flag exists: Set cross-node platform layer and application layer exception flags in the application state shared memory, and periodically read the cross-node platform layer and application layer exception flags based on the application state management program; The application status management program sets the corresponding application of each node to a fault state and automatically isolates it.

7. The application anomaly automatic isolation method according to claim 6, characterized in that, The preset basic hardware and software anomaly judgment framework also supports viewing the status of various basic hardware and software indicators of the operating system on all nodes. The status of the basic hardware and software indicators includes normal status and abnormal status.

8. An application anomaly automatic isolation system, characterized in that, include: The basic hardware and software anomaly monitoring module is used to monitor real-time data of basic hardware and software indicators of the operating system and obtain the abnormal status of basic hardware and software. The basic software and hardware anomaly flag determination module is used to determine the anomaly type based on the anomaly status of the basic software and hardware, and obtain the basic software and hardware anomaly flag. The platform layer anomaly flag acquisition module of this node is used to perform offline and anomaly monitoring of the key processes of this node and obtain the platform layer anomaly flag of this node. The application layer anomaly flag acquisition module of this node is used to monitor the application functions and application data of this node for anomalies and obtain the application layer anomaly flag of this node. The cross-node platform layer and application layer exception flag acquisition module is used to perform exception judgment on key process functions, data acquisition functions, and data non-refresh functions across nodes, and obtain cross-node platform layer and application layer exception flags. The application automatic isolation module is used to set the application to a fault state and automatically isolate it if any of the following abnormalities exist: basic software and hardware abnormality flags, local node platform layer abnormality flags, local node application layer abnormality flags, or cross-node platform layer and application layer abnormality flags. The first priority adjustment module is used to reduce the application priority to the lowest priority in the corresponding priority range within the system when the application recovers from a failure or network outage, based on the preset application priority, and obtain the first priority adjustment result. The second priority adjustment module is used to adjust the application priority to the lowest priority in the corresponding priority range within the system based on the first priority adjustment result, when the application data of the current node and the application host node are inconsistent, or when the operating system basic software and hardware indicators of the application node are in a warning state. The priority recovery module is used to restore the application priority to the preset initial priority based on the second priority adjustment result. After the application has been running stably for a configurable preset time interval, and the application data of the current node is consistent with the application host data and there are no operating system basic software and hardware indicators in a warning state.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the application exception automatic isolation method as described in any one of claims 1-7.

10. A computer device, characterized in that, include: Memory, used to store instructions; A processor for executing the instructions, causing the device to perform operations implementing the application exception automatic isolation method as described in any one of claims 1-7.

Images