Fault self-check recovery method and system based on cloud deployment system

Abstract

The invention discloses a fault self-check recovery method and system based on a cloud deployment system, and the method comprises the steps: carrying out the multi-dimensional deep detection of the operation states of an operation system layer, a database layer and an application layer of a cloud platform through constructing an all-dimensional real-time monitoring system, and automatically triggering an adaptive recovery mechanism once an abnormality is found, and continuous and stable operation of the system is ensured. The matching tool integrates core modules of state acquisition, anomaly analysis, recovery execution, log recording and the like, collaboratively realizes real-time acquisition of operation data, accurate anomaly identification through an intelligent algorithm, rapid matching and implementation of a recovery scheme, and records logs in the whole process for troubleshooting and system optimization. According to the tool, server nodes, database states, service accessibility and webpage availability are checked in sequence, emergency processing is started immediately when abnormity is found, the stability and reliability of a cloud platform are remarkably improved, and the manual intervention cost is reduced.

Description

Technical Field

[0001] This invention belongs to the field of computer system software technology, specifically relating to a fault self-diagnosis and recovery method and system for cloud-based deployed systems. Background Technology

[0002] With the advent of the digital age, cloud platforms have been widely used across various industries and have become the first choice for many enterprises. Based on their high scalability, high security, and cost-effectiveness, cloud platforms have brought numerous conveniences to businesses and individuals, while also driving digital transformation and innovation. However, cloud platform systems are complex, involving hardware devices, software components, network architecture, and massive amounts of data. During operation, they are prone to various failures, such as server hardware failures, software program errors, network congestion or interruptions, and data corruption or loss.

[0003] Looking back at existing technologies, traditional minicomputer servers often rely excessively on manual operation when facing failures. Typical scenarios include system crashes, unresponsive operating systems, or inability to connect remotely. This necessitates on-site personnel to manually perform restarts, a method that is inefficient and can lead to prolonged business interruptions in emergencies, causing significant losses to enterprises. Even in the cloud platform system field, current detection and recovery methods still largely depend on manual inspection and operation. Manual inspection has many unavoidable drawbacks, such as untimely detection, difficulty in capturing instantaneous system anomalies, high false negative rates (easily overlooking some well-hidden faults), and slow recovery speeds (the time from fault detection to recovery operations is often considerable, severely impacting the normal operation of the cloud platform and failing to meet enterprises' stringent requirements for business continuity and stability).

[0004] While some existing automated testing tools have improved testing efficiency to some extent, their functions are relatively limited, typically only able to test a specific aspect of the cloud platform, failing to achieve comprehensive and integrated system monitoring. Furthermore, their recovery mechanisms lack sufficient flexibility and intelligence, making it difficult to quickly and accurately formulate and implement effective recovery strategies in the face of complex and ever-changing system failures, resulting in untimely and inadequate resolution of system malfunctions. Therefore, there is a pressing need for a method and corresponding tools capable of comprehensive self-testing of cloud platform deployed systems and rapid recovery in the event of anomalies or failures. Summary of the Invention

[0005] To address the problems existing in current technologies, this invention provides a method and tool for self-diagnosis and recovery of cloud-deployed operating system faults. It fundamentally overcomes many challenges faced by existing cloud platform fault detection and recovery technologies, such as incomplete detection, low recovery efficiency, and excessive reliance on manual intervention. This invention aims to provide an innovative self-diagnosis and fault recovery method and tool for cloud-deployed systems. Through this method and tool, comprehensive and real-time monitoring of the operating system, database, and application layers of the cloud platform system is achieved. When system anomalies occur, the recovery mechanism can be automatically, quickly, and accurately triggered, significantly improving the stability and reliability of the cloud platform system, greatly reducing operational costs, and providing a solid guarantee for the efficient and stable operation of the cloud platform.

[0006] To achieve the above objectives, the present invention provides a fault self-diagnosis and recovery system based on a cloud-deployed system, comprising: Status acquisition module: Connects to the operating system layer, database layer, and application layer, and is used to collect the operating parameters of each layer in real time, and transmit the raw operating parameter data to the anomaly analysis module. Anomaly Analysis Module: Connects to both the status acquisition module and the threshold configuration module to obtain operational parameter data and pre-set thresholds for various parameters. It performs in-depth comparative analysis between real-time data and thresholds to determine the anomaly type and level, synchronizes the anomaly information, and triggers subsequent recovery and processing procedures. Recovery Execution Module: After receiving the exception type and level information, it calls the recovery plan corresponding to the exception type and level information from the intelligent recovery strategy library maintained by the strategy update module and executes it; Log recording module: Records system running status, abnormal information, and every key step and data in the recovery process as log information; Alarm module: Upon receiving abnormal information, the alarm module responds by sending alarm information to relevant maintenance personnel; Threshold configuration module: Used to flexibly and independently configure normal thresholds for various parameters according to actual business needs; Strategy Update Module: Used to add new recovery strategies, adjust the execution priority or parameter settings of existing strategies.

[0007] Furthermore, the anomaly analysis module synchronizes anomaly information to the recovery execution module, alarm module, and log recording module.

[0008] Furthermore, during the execution process, the recovery execution module monitors the progress of the recovery operation in real time and feeds back the recovery results to the log recording module, recording and tracking the entire recovery process.

[0009] Furthermore, the alarm module sends alarm information to relevant maintenance personnel through various pre-set SMS, email, and instant messaging tools; the alarm information includes the time, location, anomaly type, and severity of the fault.

[0010] Furthermore, users can adjust the threshold ranges of various parameters at the operating system layer, database layer, and application layer through the threshold configuration module, based on the operating characteristics of the cloud platform system, business load, and their own requirements for system stability.

[0011] On the other hand, the present invention provides a fault self-diagnosis and recovery method for a cloud-based deployed system, comprising: Real-time collection of system operation status parameters, database operation status parameters, and application operation status parameters of the cloud platform deployment system; Each collected parameter is compared with its corresponding preset normal threshold range to determine whether the system has an abnormal operating status or a fault. When an abnormal operating status or system failure is detected, the type and level of the abnormality or failure are determined; based on the type and level of the abnormality or failure, the corresponding recovery strategy is called from the preset recovery strategy library and executed to achieve rapid system recovery; Record system operating status parameters, abnormal or fault information, and recovery process information; When a system malfunction occurs, an alarm message is immediately sent to the system administrator. The alarm message includes the malfunction type, the time of occurrence, the node, and the malfunction item. After recovery, the system will be tested three times consecutively. If the test passes, the system is considered to have recovered. If the test fails, the fault level will be automatically escalated, the corresponding policy will be invoked, and a high-level alarm will be triggered. If the system still fails to recover after three consecutive policy upgrades, a manual intervention mechanism will be triggered, the system will lock its current state, and the system will wait for maintenance personnel to operate.

[0012] Furthermore, for Windows systems, the program can be set to start automatically as a service through the service management console, or a scheduled task can be created and run as "system" with the trigger set to activate at "system startup"; for Linux systems, the service can be enabled using the `systemctl enable` command, or the `@reboot` command of crontab can be used to implement the application instance; for AIX systems, the `rc.local` script file can be added to the ` / etc / inittab` file through the init process, and the user applications that need to be started during system boot can be written into the `rc.local` script file in detail. For Linux systems, CPU load and memory usage details are obtained in real time through the proc file system. For AIX systems, the memory paging activity status is collected through the nmon command.

[0013] Furthermore, if the operating system is detected to be running normally, but users cannot access normal services, and the database connection status is abnormal, the database service status is checked. If the system does not automatically switch nodes within 3 cycles, then node switching, directory mounting, and service restart are performed on the database service to restore normal business operation.

[0014] Furthermore, when the number of database connections is detected to exceed 80% of the preset threshold for 10 consecutive minutes, the database connection pool expansion mechanism is automatically triggered, temporarily increasing the connection quota by 50%.

[0015] Furthermore, by conducting comprehensive and continuous monitoring of the server operating system, running databases, and applications, efficient fault self-checking and recovery operations are achieved, and detailed fault information is sent after the system is successfully recovered.

[0016] Compared with the prior art, the present invention has at least the following beneficial effects: This invention achieves comprehensive monitoring of the operating system layer, database layer, and application layer of a cloud platform deployment system, completely breaking the limitations of traditional tools that can only perform single-dimensional detection, and ensuring that any subtle anomalies in the system can be detected and handled in a timely manner; This invention, by leveraging an intelligent recovery strategy library, can automatically and accurately match and implement the most suitable recovery plan based on the type and level of the anomaly. At the same time, for complex faults, it supports fault level escalation and multi-round recovery attempt mechanisms, which significantly improves the success rate of recovery operations and ensures the stable operation of the cloud platform system to the greatest extent. This invention greatly reduces reliance on manual intervention. Especially during off-peak hours, the system can automatically and independently handle various faults, effectively reducing operation and maintenance costs. At the same time, the rapid fault recovery mechanism significantly shortens business interruption time, effectively ensuring the continuity of enterprise business, avoiding potential economic losses caused by business interruption, and significantly improving the enterprise's operational efficiency and competitiveness.

[0017] The system information processing tool of this invention has high flexibility. It not only supports extensive adaptation to multiple operating systems, including Linux, AIX, and Windows, but also allows users to customize the threshold range of various parameters according to actual business needs. It can also easily update and optimize the recovery strategy library, thus perfectly adapting to the cloud platform operation needs of different industries and scenarios, demonstrating strong versatility and adaptability.

[0018] Furthermore, by dynamically adjusting the detection frequency mechanism, during working hours, considering the characteristics of busy business and high system load, the detection frequency is set to once per minute to promptly capture any possible anomalies; during non-working hours, the detection frequency is appropriately reduced to once every 5 seconds, which not only ensures real-time monitoring of the system but also optimizes the utilization of system resources. This flexible detection frequency setting ensures that anomalies are detected in a timely manner, buying valuable time for subsequent rapid recovery.

[0019] Furthermore, for servers operating different operating systems, to ensure the continuous and stable operation of services, the startup methods for major business programs and database applications are carefully incorporated into the system startup applications. Specifically, for Windows systems, programs can be cleverly set to start automatically as services through the service management console, or scheduled tasks can be flexibly created and run as "system," with triggers precisely set to activate at "system startup," thus achieving automatic program startup at boot. For Linux systems, the `systemctl enable` service command or the `@reboot` command in crontab can be used to achieve fast and reliable startup of application instances during system restart, effectively ensuring business operation. For AIX systems, the `rc.local` script file is scientifically added to the ` / etc / inittab` file through the init process, detailing the user applications that need to be started during system boot. When the system restarts and calls the init process, the corresponding applications can be successfully launched, achieving automatic startup of application services at boot.

[0020] Furthermore, when it is detected that the operating system is running normally, but users cannot access normal business, and the database connection status is abnormal, the database service status is confirmed in a timely and accurate manner through the interface. If the system does not automatically switch nodes within 3 cycles, a series of critical operations such as node switching, directory mounting, and service restart are quickly performed on the database service to fully restore normal business operation.

[0021] Furthermore, when the operating system and database are detected to be running normally, but a certain node application fails to log in multiple times, the interface return value can quickly and accurately determine that the node is faulty. Then, the application running status on the application node is thoroughly checked. By comparing the number of application processes when it is normal and when it is faulty, the faulty process is accurately located and the corresponding program is started in a timely manner. If starting the program cannot solve the problem, the application on the node is restarted decisively, thereby ensuring that there are enough business nodes and the normal and stable operation of the business.

[0022] Furthermore, by conducting comprehensive and uninterrupted monitoring of the server operating system, running databases, and applications, efficient fault self-checking and recovery operations are achieved. After the system successfully recovers, logs are automatically and systematically organized, and detailed fault information is promptly sent to administrators via SMS. This allows administrators to conduct in-depth analysis and rectification of faults during appropriate downtime, effectively improving work efficiency and ensuring the long-term, stable, and normal operation of the business.

[0023] Furthermore, the alarm SMS message design is highly scientific and practical, not only including the precise time of the fault occurrence, but also covering key information such as the specific values ​​of abnormal parameters and preset normal threshold ranges. This allows managers to quickly and intuitively locate the severity of the fault after receiving the SMS, providing strong support for subsequent decision-making and handling.

[0024] Furthermore, the system features an intelligent early warning and response mechanism for database layer failures. When the number of database connections exceeds 80% of the preset threshold for 10 consecutive minutes, it will automatically and quickly trigger the database connection pool expansion mechanism, temporarily increasing the connection quota by 50% to alleviate database pressure and ensure stable database operation and normal business operations. Attached Figure Description

[0025] Figure 1 This is a flowchart of the self-detection and fault recovery process of the cloud platform-based deployment system of the present invention.

[0026] Figure 2 This is a flowchart illustrating the execution of the recovery strategy after a failure occurs. Detailed Implementation

[0027] The self-detection and fault recovery process of the cloud platform-based deployment system of this invention is as follows: Figure 1 As shown, this flowchart clearly and intuitively illustrates the complete process from system operation status monitoring, anomaly detection and identification, recovery strategy initiation and execution, to the final system recovery and log recording. Through this diagram, the core execution steps and logical relationships of the technical solution of this invention can be readily understood, providing an important reference for technical personnel to understand and implement it.

[0028] refer to Figure 2 This paper details the process of how, after a system detects a fault, it selects, matches, and executes appropriate recovery strategies from an intelligent recovery strategy library based on the fault type and severity. It also demonstrates how the recovery effect is monitored and verified in real time during the recovery process, and the decision-making process for escalating the fault severity and initiating a new recovery attempt if recovery fails. This flowchart provides crucial visual support for a deeper understanding of the intelligent recovery mechanism of this invention, helping technicians to accurately grasp the execution details and logic of the recovery strategies.

[0029] To more clearly illustrate the specific implementation schemes in the embodiments of the present invention or the prior art, the connection relationships and functions of the various modules of the system information processing tool are as follows, based on the collaborative working logic design: Status Acquisition Module: As the front end for system information acquisition, the status acquisition module establishes close connections with the operating system layer, database layer, and application layer through designed interfaces. Its core function is to collect the operating parameters of each layer in real time and quickly and accurately transmit these raw operating parameter data to the anomaly analysis module, providing a solid data foundation for subsequent anomaly detection and analysis. It collects operating parameters of the operating system (including key parameters such as CPU utilization, memory usage, and disk read / write speed), database (such as connection count, query response time, and transaction processing success rate), and applications (such as application response time, request success rate, and process running status) in real time and comprehensively, and accurately compares these parameters with pre-set scientific and reasonable thresholds to efficiently determine whether the system has experienced anomalies.

[0030] Anomaly Analysis Module: As the "intelligent brain" of the entire system, the anomaly analysis module connects to both the status acquisition module and the threshold configuration module. It receives real-time operational data from the status acquisition module and obtains pre-set threshold values ​​for various parameters from the threshold configuration module. Upon receiving data, the anomaly analysis module uses pre-defined algorithms and rules to perform in-depth comparative analysis between the real-time data and the thresholds to accurately determine the anomaly type and level. Once an anomaly is identified, relevant information is synchronized to the recovery execution module, alarm module, and log recording module, triggering subsequent recovery and processing procedures. Based on the accurate identification of the anomaly type and level (anomaly levels are divided into minor, moderate, and severe), the module intelligently invokes the corresponding recovery plan. For example, for minor anomalies, simple and effective operations such as reconnecting to the database and clearing the cache may be taken; for moderate anomalies, strategies such as restarting the application and migrating tasks to a backup node may be executed; and for severe anomalies, critical measures such as restarting the server and switching database master-slave nodes are initiated. Simultaneously, a multi-period verification mechanism continuously tracks and evaluates the recovery effect to ensure that the system truly recovers to a stable and normal operating state.

[0031] Recovery Execution Module: This module is the direct executor for restoring the system to normal operation. Collaborating with the anomaly analysis module, it receives anomaly type and severity information from the anomaly analysis module, retrieves the corresponding recovery plan from the intelligent recovery strategy library maintained by the strategy update module, and executes it. During execution, the recovery execution module monitors the progress of the recovery operation in real time and promptly feeds back the recovery results to the log recording module for detailed recording and tracking of the entire recovery process.

[0032] Log recording module: The log recording module is like a "recorder", recording the system's operating status, abnormal information, and every key step and data in the recovery process as log information. The log information not only provides data support for subsequent troubleshooting and system optimization, but also helps maintenance personnel to gain a deeper understanding of the system's operating status and continuously improve the system's stability and reliability.

[0033] Alarm Module: The alarm module serves as a bridge between the system and operations and maintenance (O&M) personnel. When the anomaly analysis module detects an anomaly and sends relevant information, the alarm module responds by sending detailed alarm information to relevant O&M personnel through various pre-defined channels (such as SMS, email, instant messaging tools, etc.). The alarm information includes key details such as the time and location of the fault, the type of anomaly, and its severity, ensuring that O&M personnel can obtain system anomaly information immediately and take timely countermeasures. When the system detects an anomaly, the alarm module can quickly send alarm information to relevant personnel through multiple channels (such as SMS, email, synchronized voice or video). The threshold configuration module allows users to flexibly and conveniently configure the normal threshold ranges for various parameters according to actual business needs. Threshold Configuration Module: This module allows users to flexibly and independently configure normal thresholds for various parameters based on actual business needs. Users can adjust the threshold ranges for key parameters at the operating system, database, and application layers according to the cloud platform system's operating characteristics, business load, and their own system stability requirements. This makes the system's anomaly detection more closely aligned with actual business scenarios, improving the accuracy and relevance of detection.

[0034] The strategy update module is used to adjust the system's continuous optimization and adaptability. It allows users to update and optimize the intelligent recovery strategy library in a timely manner based on experience accumulated during system operation, the latest technological trends, and changes in business needs. Through the strategy update module, users can add new recovery strategies, adjust the execution priority or parameter settings of existing strategies, and ensure that the system can always take the most effective recovery measures when facing constantly changing fault types and complex operating environments, thereby improving the system's stability and reliability.

[0035] This invention provides a self-detection and fault recovery method for a cloud platform-based deployment system, comprising the following steps: Self-testing steps: Real-time collection of system operation status parameters, database operation status parameters, and application operation status parameters of the cloud platform deployment system; comparison of each collected parameter with the corresponding preset normal threshold range to determine whether the system has any abnormal operation status or fault.

[0036] Fault recovery steps: When an abnormal operating status or system fault is determined, the type and level of the abnormality or fault are identified; based on the type and level of the abnormality or fault, the corresponding recovery strategy is called from the preset recovery strategy library and executed to achieve rapid system recovery.

[0037] The system operating status parameters include at least one of CPU utilization, memory usage, disk space utilization, and network bandwidth utilization; the database operating status parameters include database connection count, query response time, transaction commit success rate, and log growth rate; the application operating status parameters include application response time, request success rate, and number of error logs.

[0038] If the collected parameters exceed the preset normal threshold range during the self-testing step, they are marked as abnormal parameters. When the number of abnormal parameters reaches the preset quantity threshold or the degree to which a single abnormal parameter exceeds the threshold reaches the preset severity level, it is determined that the system has an abnormal operating status or a fault.

[0039] The detected anomalies or faults are classified into three levels: minor, moderate, and severe. Different levels correspond to different recovery strategies. Recovery strategies include restarting the application, reconnecting to the database, migrating the system to a standby node, expanding resources, rolling back database transactions, and so on.

[0040] After the recovery strategy is executed, the system performs a self-test again. If the system returns to normal, the process ends. If the system does not return to normal, the abnormality or fault level is increased, and the corresponding recovery strategy is called and re-executed until the system returns to normal or a manual intervention mechanism is triggered.

[0041] The present invention also provides a system for implementing the above-described method, comprising: Status acquisition module: Used to collect system operation status parameters, database operation status parameters and application operation status parameters of the cloud platform deployment system in real time.

[0042] Anomaly Analysis Module: Connected to the status acquisition module, it compares the acquired parameters with the corresponding preset normal threshold range to determine whether the system has any abnormal operating status or fault, and to determine the type and level of the anomaly or fault.

[0043] Recovery Execution Module: Connected to the anomaly analysis module, it is used to call the corresponding recovery strategy from the preset recovery strategy library according to the type and level of the anomaly or fault, and execute the recovery strategy.

[0044] Log recording module: Connected to the status acquisition module, anomaly analysis module and recovery execution module respectively, it is used to record system operating status parameters, anomaly or fault information and recovery process information.

[0045] It also includes a threshold configuration module, which is connected to the anomaly analysis module, and is used by users to configure the preset normal threshold range, the threshold for the number of abnormal parameters, and the severity of abnormal parameters exceeding the threshold for each running status parameter.

[0046] It includes a policy update module, connected to the recovery execution module, used by users to update the recovery policies in the preset recovery policy library. It also includes an alarm module, connected to the anomaly analysis module, which issues alarm messages when the system experiences abnormal operating conditions or faults. The alarm information includes the type, level, time of occurrence, and location information of the anomaly or fault; the alarm methods are primarily SMS alarms and email alarms.

[0047] This embodiment takes a coal truck information management system deployed on a cloud platform in a power plant as an example. The system involves multiple servers, which run Linux, Windows and AIX operating systems respectively, are equipped with a MySQL database, and have applications for coal truck information input and query.

[0048] I. System Initialization Configuration Threshold configuration: Parameters are set through the threshold configuration module. Operating system layer: The normal threshold range for CPU utilization is set to 0-70%, minor abnormalities to 75%-85%, general abnormalities to 85%-95%, and severe abnormalities to >95%. The normal threshold range for memory utilization is set to 0-75%. The normal threshold range for disk space utilization is 0-80%.

[0049] Database layer: The normal threshold range for the number of connections is set to 0-300, the minor abnormality is 300-400, the general abnormality is 400-500, and the severe abnormality is >500. The normal threshold range for query response time is set to 0-2 seconds.

[0050] Application layer: The normal threshold range for response time is set to 0-3 seconds, and the normal threshold for request success rate is set to ≥99%. The threshold for the number of abnormal parameters is also set to 3. The severity of a single abnormal parameter exceeding the threshold is categorized as follows: less than 10% exceeding the threshold is considered minor, 10%-30% is considered moderate, and more than 30% is considered severe.

[0051] Anomaly detection criteria: When there are ≥2 abnormal parameters or a single parameter reaches the level of severe anomaly, the system is judged as having a fault.

[0052] Recovery strategy library configuration: Configure recovery strategies through the strategy update module. Recovery strategies for minor anomalies include reconnecting to the database and clearing the application cache; recovery strategies for general anomalies include restarting the application and migrating some tasks to a standby node; and recovery strategies for severe anomalies include restarting the server and switching the database master-slave node.

[0053] Data collection frequency settings: During working hours (8:00 AM - 6:00 PM), the server status detection frequency is set to once per minute; during non-working hours (6:00 PM - 8:00 AM), the detection frequency is set to once every 5 seconds.

[0054] II. Self-testing process Data Acquisition: The status acquisition module collects the operating data of each server in real time.

[0055] Operating system layer: Linux server: CPU usage is collected through an agent program and the top command, memory usage is collected through the free command, and network bandwidth is collected through the iftop command.

[0056] Windows Server: CPU information is obtained through the Win32_Processor class of the WMI interface, and memory data is obtained through the Win32_PerfFormattedData_PerfOS_Memory class.

[0057] AIX server: Use the topas command to collect system resources and the netstat -i command to collect network status.

[0058] Data is acquired via system commands. Simultaneously, data including database connection counts, query response times, application response times, and request success rates are collected and sent to the anomaly analysis module.

[0059] Anomaly Detection: The anomaly analysis module compares the received parameters with preset thresholds. For example, at 3 PM one afternoon (during working hours), the status acquisition module detects that the CPU utilization of a Linux server reaches 85% (exceeding the normal threshold of 70%), the memory utilization reaches 85% (exceeding the normal threshold of 80%), and the application response time on the server reaches 5 seconds (exceeding the normal threshold of 3 seconds). The number of abnormal parameters is 3, reaching the preset threshold, and is therefore judged as a general anomaly, with the anomaly type being server resource strain and slow application response.

[0060] III. Fault Recovery Process Alarm: The anomaly analysis module sends the anomaly information to the alarm module, which immediately sends an alarm message to the system administrator. As an example, the alarm message includes the anomaly type as general anomaly, the occurrence time as 21:45, the location as a Linux server, and the anomalies in CPU, memory, and application response time.

[0061] Recovery strategy execution: After receiving the exception information, the recovery execution module calls the recovery strategy corresponding to the general exception from the recovery strategy library, which means restarting the coal truck information entry application on the server. The application service is restarted using the Linux systemctl command, and the application cache is cleared simultaneously.

[0062] Recovery Verification: After the recovery execution module performs the recovery operation, the status acquisition module collects the relevant parameters of the server again. For example, if the relevant parameters show that the CPU utilization rate has dropped to 60%, the memory utilization rate has dropped to 75%, and the application response time has dropped to 2 seconds, all of which are within the normal threshold range, the status acquisition module will immediately start three consecutive verifications (with an interval of 10 seconds).

[0063] If the verification passes, the system is considered to have returned to normal. The logging module generates a "Fault Recovery Report," which includes the fault time, processing steps, and recovery time (accurate to the second).

[0064] If verification fails (e.g., response time still exceeds the threshold after restarting the application), the fault level will be automatically escalated (e.g., from "normal" to "critical"), the corresponding strategy will be invoked (e.g., migration to a backup node), and a high-level alarm (telephone voice notification) will be triggered.

[0065] If the system fails to recover after three consecutive policy upgrades, a manual intervention mechanism will be triggered, locking the current state and waiting for maintenance personnel to take action.

[0066] Log recording: The log recording module records relevant information about this exception, including the time of the exception, parameter values, recovery strategy, and execution results.

[0067] IV. Special Scenario Handling Server failure during off-peak hours: As an example, at 2 AM one day (off-peak hours), the status acquisition module failed to ping a Windows server for three consecutive cycles (15 seconds) during a 5-second check. The anomaly analysis module determined it to be a serious anomaly, and the recovery execution module invoked the corresponding recovery strategy, restarting the Windows server via remote command. After restarting, the status acquisition module detected that the server was accessible again, the application started automatically (triggered at system startup via scheduled tasks), the system returned to normal, and the logging module recorded the relevant process.

[0068] Database connection error: At a certain point, the status acquisition module detects that the number of database connections remains at 0, and query response times out. The anomaly analysis module determines this to be a database connection error, and the recovery execution module checks and finds that the database service is not running, attempting to restart it. After restarting, the connection number returns to normal, and the application can access the database normally. If the restart fails, the recovery execution module performs a database node switch, transferring the business to the backup node and mounting the relevant directories to ensure normal data access.

[0069] Application process anomaly: A coal truck information query application on an AIX server failed to log in multiple times. The status acquisition module detected that the number of application processes was two fewer than normal. The anomaly analysis module determined that the application process was abnormal, and the recovery execution module found the missing process by comparison and started the corresponding program. If starting failed, the application was restarted (implemented through the rc.local script called in the / etc / inittab file) to ensure the application runs normally.

[0070] The technical solution of this invention demonstrates significant advantages in practical applications. Taking a power plant coal transport vehicle information system as an example, this system may face many complex situations during operation. If the system suddenly crashes, causing coal transport vehicle information to be unable to be entered in a timely manner, it will not only affect the normal production scheduling of the power plant but may also cause supply chain disruptions. However, with the technical solution of this invention, the system can automatically attempt to quickly restore business operations by restarting the application and switching nodes as soon as an anomaly is detected. Practical verification shows that this process can be completed in a very short time, effectively reducing economic losses caused by system failures and fully demonstrating the superior performance of this invention in ensuring business continuity.

[0071] Through the above specific implementation methods, the present invention can comprehensively and in real time detect anomalies in cloud platform deployment systems, quickly and accurately recover from faults, and effectively ensure the stable operation of the system. Especially when staff cannot respond in a timely manner, it can still efficiently handle faults and reduce economic losses, which fully demonstrates the advantages of the present invention in real-time performance, intelligence and economy.

[0072] This invention provides a highly intelligent and automated self-detection and fault recovery method for cloud platform-based deployment systems. By constructing a real-time monitoring system, it performs multi-dimensional and in-depth self-detection of the operating system, database, and application layers of the cloud platform deployment system. Once the system's operating status deviates from the normal range or a fault occurs, it can quickly and accurately trigger an appropriate recovery mechanism, thereby ensuring the continuous and stable operation of the cloud platform system. It integrates multiple core functional modules, including status acquisition, anomaly analysis, recovery execution, and log recording. These modules collaborate and cooperate with each other. The status acquisition module acquires real-time operating data from each level of the system. The anomaly analysis module uses advanced algorithms and rules to deeply analyze the data and accurately identify abnormal conditions. The recovery execution module quickly matches and implements corresponding recovery solutions from an intelligent recovery strategy library based on the anomaly type. The log recording module records the entire detection and recovery process in detail, providing crucial data support for subsequent troubleshooting and system optimization. In terms of the detection process, the tool meticulously checks the status of cloud server nodes, database operation status, service and application accessibility, and webpage availability. Based on the detection feedback instructions, it immediately initiates emergency handling procedures for any detected anomalies, striving to ensure the system can return to normal operation in the shortest possible time. This significantly improves the stability and reliability of the cloud platform and greatly reduces the cost of manual intervention.

[0073] The above content is only for illustrating the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solution based on the technical concept proposed in this invention shall fall within the scope of protection of the claims of this invention.

Claims

1. A fault self-diagnosis and recovery system based on a cloud-deployed system, characterized in that, include: Status acquisition module: Connects to the operating system layer, database layer, and application layer, and is used to collect the operating parameters of each layer in real time, and transmit the raw operating parameter data to the anomaly analysis module. Anomaly Analysis Module: Connects to both the status acquisition module and the threshold configuration module to obtain operational parameter data and pre-set thresholds for various parameters. It performs in-depth comparative analysis between real-time data and thresholds to determine the anomaly type and level, synchronizes the anomaly information, and triggers subsequent recovery and processing procedures. Recovery Execution Module: After receiving the exception type and level information, it calls the recovery plan corresponding to the exception type and level information from the intelligent recovery strategy library maintained by the strategy update module and executes it; Log recording module: Records system running status, abnormal information, and every key step and data in the recovery process as log information; Alarm module: Upon receiving abnormal information, the alarm module responds by sending alarm information to relevant maintenance personnel; Threshold configuration module: Used to flexibly and independently configure normal thresholds for various parameters according to actual business needs; Strategy Update Module: Used to add new recovery strategies, adjust the execution priority or parameter settings of existing strategies.

2. The fault self-diagnosis and recovery system based on a cloud-deployed system according to claim 1, characterized in that, The anomaly analysis module synchronizes anomaly information to the recovery execution module, alarm module, and log recording module.

3. The fault self-diagnosis and recovery system based on a cloud-deployed system according to claim 1, characterized in that, During the recovery process, the recovery execution module monitors the progress of the recovery operation in real time and feeds back the recovery results to the log recording module to record and track the entire recovery process.

4. The fault self-diagnosis and recovery system based on a cloud-deployed system according to claim 1, characterized in that, The alarm module sends alarm information to relevant maintenance personnel through various pre-set SMS, email, and instant messaging tools; the alarm information includes the time, location, anomaly type, and severity of the fault.

5. The fault self-diagnosis and recovery system based on a cloud-deployed system according to claim 1, characterized in that, Users can adjust the threshold ranges of various parameters at the operating system layer, database layer, and application layer through the threshold configuration module, based on the operating characteristics of the cloud platform system, business load, and their own requirements for system stability.

6. A fault self-diagnosis and recovery method for a cloud-based deployed system, characterized in that, include: Real-time collection of system operation status parameters, database operation status parameters, and application operation status parameters of the cloud platform deployment system; Each collected parameter is compared with its corresponding preset normal threshold range to determine whether the system has an abnormal operating status or a fault. When an abnormal operating status or system failure is detected, the type and level of the abnormality or failure are determined; based on the type and level of the abnormality or failure, the corresponding recovery strategy is called from the preset recovery strategy library and executed to achieve rapid system recovery; Record system operating status parameters, abnormal or fault information, and recovery process information; When a system malfunction occurs, an alarm message is immediately sent to the system administrator. The alarm message includes the malfunction type, the time of occurrence, the node, and the malfunction item. After recovery, the system will be tested three times consecutively. If the test passes, the system is considered to have recovered. If the test fails, the fault level will be automatically escalated, the corresponding policy will be invoked, and a high-level alarm will be triggered. If the system still fails to recover after three consecutive policy upgrades, a manual intervention mechanism will be triggered, the system will lock its current state, and the system will wait for maintenance personnel to operate.

7. The fault self-diagnosis and recovery method for a cloud-based deployment system according to claim 6, characterized in that, For Windows systems, you can set the program to start automatically as a service through the Service Management console, or create a scheduled task and choose to run it as "System" with the trigger set to start on "System Startup". For Linux systems, you can use the systemctlenable service command or the crontab @reboot command to implement the application instance. For AIX systems, you can use the init process and add the rc.local script file to the / etc / inittab file to write the user applications that need to be started during the system boot process into the rc.local script file. For Linux systems, CPU load and memory usage details are obtained in real time through the proc file system. For AIX systems, the memory paging activity status is collected through the nmon command.

8. The fault self-diagnosis and recovery method for a cloud-based deployment system according to claim 6, characterized in that, When it is detected that the operating system is running normally, but users cannot access normal business, and the database connection status is abnormal, the database service status is checked. If the system does not automatically switch nodes within 3 cycles, node switching, directory mounting, and service restart are performed on the database service to restore normal business operation.

9. The fault self-diagnosis and recovery method for a cloud-based deployment system according to claim 8, characterized in that, When the number of database connections exceeds 80% of the preset threshold for 10 consecutive minutes, the database connection pool expansion mechanism is automatically triggered, temporarily increasing the connection quota by 50%.

10. The fault self-diagnosis and recovery method for a cloud-based deployment system according to claim 6, characterized in that, By performing comprehensive and uninterrupted monitoring of the server operating system, running databases, and applications, efficient fault self-checking and recovery operations are achieved, and detailed fault information is sent after the system is successfully recovered.

Images