Distributed micro-service automated recovery method based on multi-dimensional health check

Abstract

The application relates to the technical field of distributed system operation and maintenance, and discloses a distributed micro-service automatic recovery method based on multi-dimensional health examination, which comprises the following steps: sequentially performing health examination on an infrastructure layer, a service component layer, a data storage layer and a business function layer in a fixed order to generate health examination results of each layer; collecting the results of each layer to calculate an overall health state; triggering a four-stage progressive automatic recovery operation according to the overall health state; generating an alarm notification and recording an audit log. The application solves the technical problems that traditional single-dimensional detection cannot perceive deep-layer abnormalities and lacks a hierarchical automatic recovery strategy.

Description

Technical Field

[0001] This invention relates to the field of distributed system operation and maintenance technology, specifically to an automated recovery method for distributed microservices based on multi-dimensional health checks. Background Technology

[0002] In the operation and maintenance scenario of a distributed microservice architecture, the system consists of multiple layers of components, including infrastructure, service components, data storage, and business functions, working together. An anomaly in any layer may affect the continuity of the overall business.

[0003] Existing operation and maintenance systems typically use a single-dimensional process liveness detection method to monitor the health status of microservices, only determining whether the service process exists.

[0004] The above approach has the following drawbacks: First, it cannot detect deep abnormal states such as database connection pool exhaustion or sudden drop in cache hit rate within the service, which may lead to the failure being missed or misjudged. Second, when an anomaly is detected, there is a lack of a systematic hierarchical automatic recovery strategy, and fault handling relies on manual intervention, resulting in a long recovery time and affecting business continuity. Summary of the Invention

[0005] This invention provides a distributed microservice automated recovery method based on multi-dimensional health checks, which solves the technical problems of single-dimensional fault detection, lack of progressive escalation capability in recovery operations, and imperfect alarm auditing in related technologies.

[0006] This invention discloses a distributed microservice automated recovery method based on multi-dimensional health checks, comprising the following steps: Health checks are performed sequentially in a fixed order: infrastructure layer, service component layer, data storage layer, and business function layer. For the infrastructure layer, raw data in the operating system's virtual file system is read to calculate resource utilization indicators. For the service component layer, process liveness detection, port connectivity detection, and HTTP interface detection are performed sequentially. For the data storage layer, connection percentage, cache hit rate, and cluster status are collected. For the business function layer, the availability of core business processes is verified in an end-to-end liveness detection manner, and health check results are generated for each layer. Summarize the health examination results at all levels, calculate the overall health status according to the predefined status judgment rules, and the overall health status can be one of healthy, degraded or unhealthy. Based on the overall health status and specific abnormal information at each level, automatic recovery operations are triggered according to a four-level progressive strategy. The four-level progressive strategy is as follows: automatic process restart, service failover, service degradation, and full restart. If a lower-level recovery operation fails, it will automatically escalate to the next level. Based on the results of the recovery operation, generate alarm notifications of the corresponding level and send them through multiple channels. Record all recovery operations in the audit log table in an append-only manner.

[0007] Furthermore, the health check of the infrastructure layer includes: calculating CPU utilization using the following formula: ,in This is the increment of the idle time slice between two samplings. The sum of all types of time slice increments between two samplings; memory usage is calculated using the following formula: ,in Total physical memory. This refers to the amount of free memory. This refers to the kernel buffer usage. Page cache usage To determine the amount of reclaimable Slab memory, calculate disk usage using the following formula: ,in The number of free blocks available to non-privileged processes. The total number of blocks in the file system; when , or An anomaly exceeding the first threshold is marked as non-fatal, and an anomaly exceeding the second threshold is marked as fatal, where the second threshold is greater than the first threshold.

[0008] Furthermore, health checks on the data storage layer include: calculating the percentage of active database connections using the following formula: ,in This represents the current number of active connections. The maximum number of connections allowed by the database, when When the number of connections exceeds the alarm threshold, it is marked as abnormal; the cache hit rate is calculated using the following formula: ,in To accumulate cache hits, To accumulate the number of cache misses, when When the hit rate falls below the alarm threshold, it is marked as abnormal; obtain the status of the search engine cluster and judge the health of the cluster according to three levels: green, yellow, and red.

[0009] Furthermore, the health check of the business function layer includes the following sub-steps executed sequentially: sending a request to the business interface to create a test plan with a special marker and recording the plan identifier; triggering the execution of the test task and waiting for the task to complete; verifying the readability of the task execution result record; calling the deletion interface to clean up the test data generated during the liveness detection process; if any sub-step returns an error code or fails to complete within the timeout period, the business function layer is determined to be in an unhealthy state and the failure sub-step identifier is attached to the result.

[0010] Furthermore, the automatic process restart in the four-level progressive strategy includes: sending a start command to the process manager of the target service, waiting for a preset time, and then re-executing the process liveness probe and port connectivity probe to verify the start result; if the verification fails, retrying after calculating the exponential backoff waiting time using the following formula: ,in For the first Waiting time before the next retry Based on waiting time, This is the current number of retries; if the maximum number of retries is reached and the service still fails, it will be upgraded to service failover.

[0011] Furthermore, the service failover in the four-level progressive strategy includes: setting a global failover status flag and writing it to a distributed lock through atomic operations; selecting a slave database from a pre-configured slave database list that meets the data synchronization delay condition as a candidate new master database; performing a master database promotion operation on the candidate slave database and verifying its ability to accept write requests; updating the global database connection configuration through the configuration center and notifying all service instances to reload; clearing the global failover lock after performing write tests and read verifications on the new master database; creating a checkpoint snapshot of the current system state before executing the above steps, and if any step fails, performing the compensation operations of each step in reverse order to roll back the system state to the checkpoint state.

[0012] Furthermore, the service degradation in the four-level progressive strategy includes: when the search engine is unavailable, suspend writing logs to it and temporarily store the data to be written in the local memory queue. When the queue capacity limit is exceeded, discard the oldest data according to the least recently used strategy. Interfaces that rely on its full-text search function return a predefined degradation response. After the search engine recovers, the temporarily stored data is automatically replenished in batches. When the caching service is unavailable, revert session management to the local memory caching mode, reduce the maximum number of sessions, shorten the session timeout time, and activate the session affinity strategy for request routing.

[0013] Furthermore, the full restart in the four-level progressive strategy includes: sending stop commands in reverse order of the service dependency graph topology, and forcibly terminating the process after a timeout; cleaning up the temporary file directories, process lock files, and local runtime cache directories of each service; starting each service in ascending order of the service dependency graph topology, waiting for its health check to pass after each service is started before starting the next service; performing a complete four-level health check after all services have started, and if any service still cannot be recovered, raising the alarm level to the highest level and triggering a manual intervention notification.

[0014] Furthermore, the generation of alarm notifications includes: alarm levels are divided into four levels: low, medium, high, and critical; when a fault that triggers an alarm fails again after a health check following automatic recovery operations, or when the same fault triggers alarms more than a preset number of times within a preset time window, the current alarm level is raised by one level and the alarm is resent; alarm notifications are sent through four channels: email, SMS, enterprise instant messaging platform Webhook, and general Webhook callback, with each channel independently configured with its enabled status and applicable minimum alarm level.

[0015] This invention discloses a distributed microservice automated recovery system based on multi-dimensional health checks, comprising: The health check data collection module is used to perform health checks in a fixed order of infrastructure layer, service component layer, data storage layer and business function layer, collect indicator data at each layer, and generate health check results at each layer. The health status calculation module is used to summarize the health examination results at all levels, calculate the overall health status according to the predefined status judgment rules, and the overall health status can be one of healthy, degraded or unhealthy. The overall health status and its basis are recorded in the system status table. The automatic recovery execution module is used to trigger and execute automatic recovery operations based on the overall health status and specific abnormal information at each level, according to a four-level progressive strategy of automatic process restart, service failover, service degradation and full restart. It automatically upgrades to the next level after a low-level recovery operation fails, and creates a system status checkpoint snapshot before the execution of service failover and higher-level recovery operations. The alarm notification and audit module is used to generate alarm notifications of corresponding levels based on the results of recovery operations and send them through multiple channels. When the fault persists or occurs repeatedly, the alarm level is automatically escalated, and all recovery operations are recorded in the audit log table in an append-only manner.

[0016] The beneficial effects of this invention are as follows: This invention establishes a four-layer health check indicator system covering the infrastructure layer, service component layer, data storage layer, and business function layer, and performs checks at each layer in a fixed order from bottom to top. This solves the technical problems of single-dimensional detection being unable to detect deep abnormal states and low-level anomalies being misjudged as independent upper-level faults. It achieves the technical effect of prioritizing the marking of low-level resource anomalies before upper-level service anomalies, thereby reducing fault misjudgments.

[0017] By dividing the overall health status into three states—healthy, degraded, and unhealthy—and triggering a four-level progressive automatic recovery operation based on the overall health status, the technical problems of lacking a graded automatic recovery strategy and relying on manual intervention for fault handling are solved. This achieves the technical effect of taking matching recovery measures for faults of different severity levels and avoiding over-response or under-response caused by a single strategy.

[0018] By introducing an exponential backoff retry mechanism in the first-level recovery, the technical effect of avoiding resource contention caused by intensive retries for abnormal services is achieved. By introducing a distributed lock in the second-level failover, the technical effect of preventing state inconsistencies caused by multiple health check nodes concurrently triggering failover is achieved. By introducing the Saga pattern to decompose the failover operation into local transaction steps with compensation operations, the technical effect of being able to execute compensation operations in reverse order and roll back the system state to the checkpoint state when any step fails is achieved. Attached Figure Description

[0019] Figure 1 This is a flowchart of a distributed microservice automated recovery method based on multi-dimensional health checks provided in an embodiment of the present invention; Figure 2 This is a sub-flowchart of the distributed microservice automated recovery method based on multi-dimensional health checks provided in this embodiment of the invention, which triggers automatic recovery operations of the corresponding level according to the overall health status. Detailed Implementation

[0020] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and implement the subject matter described herein, and changes may be made to the function and arrangement of the elements discussed without departing from the scope of this specification. Various processes or components may be omitted, substituted, or added as needed in the examples. Furthermore, some features described in the examples may be combined in other examples.

[0021] At least one embodiment of the present invention discloses a distributed microservice automated recovery method based on multi-dimensional health checks, see [link to relevant documentation]. Figures 1-2 This includes the following steps: Step 1: Collect four levels of health examination indicators and generate health examination results for each level.

[0022] The operation and maintenance management module performs health checks on the infrastructure layer, service component layer, data storage layer, and business function layer in the distributed microservice cluster in sequence. The checks are performed in a fixed order of infrastructure layer → service component layer → data storage layer → business function layer, with a complete inspection cycle every 30 seconds. The results of each layer's checks are recorded in the health check history table as three statuses: success, failure, or timeout.

[0023] For the infrastructure layer, the operation and maintenance management module reads the raw data in the virtual file system exposed by the operating system kernel, calculates CPU utilization, memory utilization, disk utilization, disk I / O wait time percentage, instantaneous network traffic and network packet loss rate, compares the values ​​of each indicator with the corresponding alarm thresholds, and generates the infrastructure layer health check results.

[0024] The CPU utilization rate is calculated as follows: Using a sampling period of 30 seconds, the ` / proc / stat` file is read twice consecutively, and the increment of the count value for each type of time slice is extracted and calculated using the following formula: ; in, For CPU utilization, This is the increment of the idle time slice between two samplings. This is the sum of all types of time slice increments between two samplings, i.e.: ; The lower bound for the summation in the above formula is: The upper boundary is This corresponds to 8 time slice types; , , , , , , , These correspond to user mode, low-priority user mode, kernel mode, idle, I / O wait, hard interrupt, soft interrupt, and the time slice increment occupied by the virtual machine, respectively.

[0025] Memory usage is calculated using the following formula: ; in, For memory usage, Total physical memory. This represents the current amount of free memory. The amount of memory used by the kernel buffer. Page cache usage To show the reclaimable Slab memory usage, all the above units are in kB, and all data are read from the / proc / meminfo file.

[0026] Disk usage is obtained by using the statfs system call to retrieve file system statistics for a specified mount point, and is calculated using the following formula: ; in, For disk usage, The number of free blocks available to non-privileged processes. This represents the total number of blocks in the file system.

[0027] when , or When the rate exceeds 85%, the corresponding indicator is marked as a non-fatal anomaly; when it exceeds 95%, it is marked as a fatal anomaly. Disk I / O latency is calculated by reading / proc / diskstats, and when it exceeds 100ms, it is marked as a non-fatal anomaly. Network packet loss rate is calculated by reading / proc / net / dev, and when it exceeds a preset threshold, it is marked as an anomaly. The above check results are summarized as the infrastructure layer health check results.

[0028] For the service component layer, the operations and maintenance management module performs three checks on each core microservice sequentially: process liveness detection, port connectivity detection, and HTTP interface detection. These three detection methods are executed in sequence, with the next detection only performed after the previous one passes, generating a health check result for the service component layer. Taking the Server core service as an example, the operations and maintenance management module first sends a signal 0 to the target process ID. If the system call returns 0, the process liveness detection passes. Then, a TCP connection is established to port 9527 on the local machine, with a connection timeout of 3 seconds, verifying that the port is in a listening state. Finally, an HTTP GET request is sent to the / health endpoint to verify that the returned HTTP status code is 200, with a request timeout of 5 seconds. If all three checks pass, the service is considered healthy. The Mohist Server distributed management platform checks port 45001 and verifies the heartbeat connection status with the management end; the SSO authentication service checks port 9988 and verifies the login interface response; the Python execution engine checks the engine process liveness status and verifies the availability of the scene loading function.

[0029] For the data storage layer, the operation and maintenance management module performs health checks on the MySQL database, Redis cache, and Elasticsearch search engine, and generates data storage layer health check results.

[0030] The percentage of active connections to a MySQL database is calculated using the following formula: ; in, The percentage of active connections. This represents the current number of active connections. The maximum number of connections allowed by the database, when If the percentage exceeds 80%, it is marked as abnormal. The MySQL master-slave replication status is determined by parsing the Slave_IO_Running and Slave_SQL_Running fields in the results returned by the SHOWSLAVE STATUS statement. If both fields are "Yes", it means that master-slave replication is normal.

[0031] Redis cache hit rate is calculated using the following formula: ; in, For cache hit rate, To accumulate cache hits, To accumulate the number of cache misses, when A value below 90% is marked as abnormal. Similarly, a Redis memory usage exceeding 80% of `maxmemory` or a current number of connections exceeding 80% of `maxclients` is also marked as abnormal.

[0032] The cluster status of the Elasticsearch search engine is obtained by calling the cluster health check REST API interface, which is judged by three levels: green, yellow, and red. Green means that all shards are normally allocated; yellow means that the primary shards are normal but some replica shards are not allocated; red means that some primary shards are not allocated and the cluster functionality is impaired.

[0033] For the business function layer, the operations and maintenance management module verifies the integrity and availability of core business processes through end-to-end liveness detection simulating real business requests, generating business function layer health check results. The liveness detection process consists of four sequential sub-steps: First, a POST request is sent to the plan management interface to create a test plan with a special marker, and the returned plan ID is recorded; then, the execution of the test task is triggered with the plan ID as a parameter, and the task is waited for completion, with a maximum waiting time of 30 seconds; next, the readability of the task execution result record is verified through the result query interface; finally, the deletion interface is called to clean up the test data generated during the above liveness detection process. If any sub-step returns an error code or fails to complete within the timeout period, the operations and maintenance management module determines that the business function layer is in an unhealthy state and attaches the failed sub-step identifier to the business function layer health check results. Business function layer liveness detection is performed every 5 minutes by default.

[0034] It should be noted that the four layers of health checks are performed in a fixed order: infrastructure layer → service component layer → data storage layer → business function layer. This is to ensure that anomalies in lower-level resources are identified before anomalies in upper-level services. If a fatal anomaly is detected in the infrastructure layer, the results of subsequent layers' checks will be marked as affected by the lower-level anomaly during status aggregation, thus avoiding misjudging upper-level service anomalies caused by insufficient basic resources as independent failures.

[0035] In this embodiment, to improve the timeliness of health status data, in addition to the scheduled health check in step 1, the operation and maintenance management module also integrates an indicator collection client library into each microservice component. This library registers and updates the following real-time indicators on the critical paths of the service code using a data entry method: the total number of HTTP requests uses a Counter type indicator, with three label dimensions: HTTP method, request path, and response status code; the HTTP request latency distribution uses a Histogram type indicator, calculated using quantile calculation. , and Delayed quantiles, quantiles Defined as satisfying the cumulative distribution function Minimum delay value ,in The values ​​are taken as 0.5, 0.95, and 0.99 respectively, with a default statistical time window of 5 minutes. The number of active tasks uses a Gauge-type metric, increasing upon task creation and decreasing upon task completion or failure. The number of database connections uses a Gauge-type metric with three connection status labels: active, idle, and wait. The above real-time metrics are collected every 15 seconds, with the metric collection service periodically pulling metric endpoints exposed by each service. The collected time-series data is stored in a time-series database.

[0036] Step 2: Summarize the health examination results at each level and calculate the overall health status.

[0037] The operation and maintenance management module summarizes the four levels of health check results generated in step 1, calculates the overall health status according to the predefined status judgment rules, and the overall health status can be one of healthy, degraded or unhealthy.

[0038] The status determination rules are as follows: If all four levels of checks pass and all indicators do not exceed the corresponding alarm thresholds, the overall health status is determined to be healthy. If there are non-fatal anomalies in the infrastructure layer or data storage layer (such as...), the overall health status is determined to be healthy. More than 85% but less than 95% If the health check of the Server core service or Mohist Server distributed management platform in the service component layer fails, or the MySQL master database in the data storage layer is inaccessible, or the core task execution function in the business function layer is completely unavailable, the overall health status is determined to be degraded.

[0039] The overall health status calculation results and their basis are recorded in the system status table and pushed to the operation and maintenance management interface via server push events or WebSocket.

[0040] It should be noted that the distinction between fatal and non-fatal exceptions is based on whether the exception directly renders core business functions unavailable. When the failure rate exceeds 85% but is below 95%, the system can still operate normally, so it is marked as a non-fatal exception and the overall status is downgraded to degraded. However, inaccessibility of the MySQL master database will directly lead to data write failure, so it is marked as a fatal exception and the overall status is set to unhealthy. The alarm thresholds for each indicator can be adjusted independently in the configuration items of the operation and maintenance management module.

[0041] Step 3: Trigger the corresponding level of automatic recovery operation based on the overall health status.

[0042] Based on the overall health status calculated in step 2 and the specific abnormal information in the health check results at each level, the operation and maintenance management module triggers automatic recovery operations according to a four-level progressive strategy, and automatically upgrades to the next level after the lower-level recovery operation fails.

[0043] Level 1 recovery is triggered automatically when a process check or port check of a service in the service component layer fails. The operations and maintenance management module, based on the process manager type of the target service, starts the target service process by calling the `supervisor.startProcess(group:name)` API interface of `supervisor` or executing the command `systemctl start service-name`. After the initial startup command is issued, the operations and maintenance management module waits for 5 seconds, then re-executes the process liveness probe and port connectivity probe to verify successful startup. If verification fails, the operations and maintenance management module uses an exponential backoff strategy to retry, with the retry waiting time calculated using the following formula: ; in, Indicates the first Waiting time before the next retry The base waiting time (initially 5 seconds). This is the current number of retries. The steps 1, 2, and 3 are taken sequentially, with corresponding waiting times of 5 seconds, 10 seconds, and 20 seconds, respectively. After three retries fail, the operations and maintenance management module records the first-level recovery failure event and escalates the recovery task to second-level processing.

[0044] Level 2 recovery is a service failover mechanism: When the MySQL master database in the data storage layer becomes inaccessible and Level 1 recovery fails, Level 2 recovery is triggered. The operations and maintenance management module first sets a global failover status flag and writes it to a distributed lock via atomic operations to prevent multiple health check nodes from concurrently triggering failover. Then, it queries the pre-configured slave database list for currently available slave nodes, selecting the slave with the highest priority and data synchronization latency of 0 or close to 0 as a candidate new master. Next, it executes STOP SLAVE and RESET MASTER statements sequentially on the selected candidate slave, making it run as the master and verifying that the candidate slave can accept write requests. Then, it updates the global database connection configuration through the configuration center, changing the master database address to the new master's address and triggering a configuration change notification. All registered service instances are notified to reload their database connection configurations. Finally, it initiates write tests and read verifications to the new master to confirm its writability and the integrity of critical data tables. After successful verification, the global failover lock is cleared. The entire failover process must be completed within 60 seconds; otherwise, a manual intervention alarm is triggered.

[0045] Level 3 recovery is a service degradation measure: When non-critical services are unavailable and Level 1 recovery cannot restore them to normal operation within a reasonable time, Level 3 recovery is triggered. For Elasticsearch unavailability, the operations management module suspends all log writing operations to Elasticsearch, temporarily storing pending log data in a local memory queue. The local memory queue has a maximum capacity limit; once this limit is exceeded, the oldest logs are discarded according to the least recently used strategy, and the number of discarded events is recorded. Simultaneously, interfaces relying on Elasticsearch's full-text search functionality return predefined degradation responses, while core interfaces not relying on Elasticsearch continue to process requests normally. Once a subsequent health check confirms Elasticsearch's availability, the operations management module automatically reads the temporarily stored logs from the local memory queue and batches them into Elasticsearch. For Redis cache unavailability, the operations management module reverts session management to a local memory cache-based mode, reduces the maximum number of sessions to 20% of the normal value, shortens the session timeout from the default 30 minutes to 10 minutes, and activates the session affinity strategy for request routing to ensure that requests from the same user are always routed to the same service instance.

[0046] Level 4 recovery is a full restart: Level 4 recovery is triggered when multiple service components are simultaneously in an unhealthy state (the default threshold is 3 or more services simultaneously failing), or when the same service triggers Level 3 or higher recovery operations more than 3 times within 1 hour. The operations and maintenance management module sends stop commands in reverse order of the service dependency graph topology, sets a stop waiting timeout for each service (default 30 seconds), and forcibly terminates the process after the timeout; then it cleans up the temporary file directory, process lock file, and local runtime cache directory of each service; then it starts each service in ascending order of the service dependency graph topology, waiting for its health check to pass after each service is started (a maximum wait of 60 seconds) before starting the next service; after all services have started, a complete Level 4 health check is performed. If the full check passes, a full restart success event is recorded; if there are still services that cannot be recovered, the alarm level is raised to the highest level and a manual intervention notification is triggered.

[0047] It should be noted that the upgrade conditions between the above-mentioned Level 1 to Level 4 recovery strategies can be adjusted in the configuration items of the operation and maintenance management module. The maximum number of retries for Level 1 recovery is 3 by default, which can be adjusted to 2 or 5 according to actual operation and maintenance needs; the trigger threshold for Level 4 recovery (the number of services that are abnormal at the same time) is 3 by default, which can be adjusted appropriately according to the cluster size.

[0048] In this embodiment, to ensure the atomicity and traceability of recovery operations at level two and above, the operation and maintenance management module automatically creates a checkpoint snapshot of the current system state before executing recovery operations at level two and above. The checkpoint data includes snapshots of the running status of all service components, the current binlog position of the database master-slave replication, status records of all ongoing plans and tasks, and connection configuration values ​​for each service in the current configuration center. The checkpoint data is serialized in JSON format and simultaneously written to the local disk's ` / data / checkpoints / ` directory and remote backup storage. The integrity of the written data is verified by calculating a SHA-256 checksum.

[0049] In this embodiment, the Saga pattern is used to ensure eventual consistency for critical operations involving the synchronous modification of multiple service states, such as database failover. The Saga pattern decomposes the failover operation into several independent local transaction steps and defines a corresponding compensation operation for each step: Compensation operation C1 for step T1 (updating slave configuration) restores the slave's master-slave replication configuration and restarts the replication process; compensation operation C2 for step T2 (updating the master address in the configuration center) restores the master address in the configuration center to its original value; compensation operation C3 for step T3 (notifying each service to reload configuration) sends a reload command to each service to reconnect to the original master. If a step fails, the Saga coordinator executes the compensation operations of the completed steps in reverse order, rolling back the system state to the checkpoint state before the recovery operation began. Every operation record in the entire Saga execution process is persistently stored in a dedicated log table.

[0050] Step 4: Generate alarm notifications and record recovery operation audit logs.

[0051] Based on the execution results of the recovery operations at each level in step 3, the operation and maintenance management module generates alarm notifications of the corresponding level and sends them through multiple channels, while recording all automatic recovery operations in the audit log table.

[0052] Alarm levels are divided into four levels: low, medium, high, and critical, corresponding to four scenarios: resource utilization exceeding the threshold but service unaffected, non-critical service anomalies, system entering degraded operation state, and core service completely unavailable. The operations and maintenance management module will raise the current alarm level by one level and resend the alarm when one of the following conditions is met: the fault that triggered the alarm is again determined to have failed after a health check following the completion of automatic recovery operations; or the same fault triggers more than 3 alarms within 1 hour.

[0053] Alarm notifications are sent via four channels: email (SMTP protocol), SMS (SMS gateway), enterprise instant messaging platform Webhook, and general Webhook callback. Each channel can be independently configured with its enabled status and applicable minimum alarm level. The general Webhook callback includes a signature generated based on the HMAC-SHA256 algorithm in the request header. The signature key is set by the user in the configuration, and the user can verify the legitimacy of the request source by verifying the signature.

[0054] All automatic recovery operations are recorded in the recovery operation audit log table. Each record includes a unique operation identifier, trigger timestamp, trigger reason, recovery level, target service name, specific operation command or API call executed, operation execution result, operation time, and reference path to the execution checkpoint. The audit log is written in append-only mode; modification and deletion of historical records are prohibited.

[0055] It's worth noting that the enterprise instant messaging platform's Webhook channel formats alarm messages into card message formats supported by each platform and sends them to the pre-configured Webhook address via HTTP POST requests. The messages include quick action buttons for operations and maintenance personnel to respond quickly on mobile devices. The card message formats for DingTalk, WeChat Work, and Slack are handled separately by the format adapters built into the operations and maintenance management module, eliminating the need for additional configuration by operations and maintenance personnel for different platforms.

[0056] In this embodiment, to support operations and maintenance personnel in actively verifying the actual effectiveness of automatic recovery capabilities, the operations and maintenance management module provides a fault drill function. Operations and maintenance personnel can trigger fault injection operations through the management interface or REST API. Optional fault injection methods include: service process crash (via kill command), port unavailability (via iptables rules), database connection pool exhaustion (via simulating high concurrency requests), and insufficient disk space (via creating large files to fill space). During the drill, the operations and maintenance management module compresses the health check interval to 10 seconds, continuously performs health checks, records the fault injection completion timestamp, the timestamp of the first anomaly detection, and the timestamp of the system recovering to a healthy state, and calculates the detection latency and actual recovery time (actual RTO value), as well as whether data loss occurred during the recovery process (actual RPO value). After the drill, the operations and maintenance management module automatically generates a drill report containing timeline data, key indicators, and drill conclusions. The drill report is persistently stored in both PDF and HTML formats and is provided to operations and maintenance personnel for review through the drill history query interface.

[0057] On the other hand, this invention also proposes a distributed microservice automated recovery system based on multi-dimensional health checks, including: The health check data collection module is used to perform health checks on the infrastructure layer, service component layer, data storage layer and business function layer in sequence, collect indicator data at each layer, and generate health check results at each layer. The health status calculation module is used to summarize the health check results at all levels, calculate the overall health status according to the predefined status judgment rules, record the overall health status and its basis in the system status table and push it to the operation and maintenance management interface. The automatic recovery execution module is used to trigger and execute automatic recovery operations according to a four-level progressive strategy based on the overall health status and specific abnormal information at each level. It automatically upgrades to the next level after a low-level recovery fails, and creates a system status checkpoint snapshot before executing recovery operations at the second level or above. The alarm notification and audit module is used to generate alarm notifications of the corresponding level based on the results of recovery operations and send them through multiple channels. When the fault persists or occurs repeatedly, the alarm level is automatically escalated, and all recovery operations are recorded to the audit log table in an append-only manner. The fault drill module is used to receive fault injection operations triggered by operation and maintenance personnel. During the drill, it continuously performs health checks and records the detection latency and actual recovery time. After the drill, it automatically generates a drill report and stores it persistently.

[0058] By way of example, the system proposed in this invention may also include various features and combinations thereof in the method embodiments, which will not be elaborated here.

[0059] This invention establishes a four-tiered health check indicator system covering the infrastructure layer, service component layer, data storage layer, and business function layer, and performs checks on each layer sequentially from bottom to top, thereby identifying and addressing anomalies in underlying resources (such as...). Exceeding the threshold More than 80% of the maximum number of connections can be flagged before upper-layer services fail, thus avoiding misjudging upper-layer service failures caused by insufficient basic resources as independent faults.

[0060] By dividing the overall health status into three states—healthy, degraded, and unhealthy—and triggering a four-level progressive automatic recovery operation based on the overall health status and specific abnormal information, this implementation method enables the operation and maintenance management module to take matching recovery measures for faults of different severities without relying on manual intervention. This avoids the problem of over-responding to minor faults and under-responding to severe faults caused by a single strategy.

[0061] By introducing an exponential backoff retry mechanism in the first-level recovery, resource contention caused by intensive retries of abnormal services is avoided. By introducing a distributed lock in the second-level failover, inconsistencies in state caused by multiple health check nodes concurrently triggering failover operations are prevented. By introducing the Saga pattern, the failover operation is decomposed into local transaction steps with compensation operations. If any step fails, the compensation operation can be executed in reverse order, rolling back the system state to the checkpoint state before the recovery operation, ensuring the traceability of critical recovery operations.

[0062] Through service degradation strategies, when the Elasticsearch search engine or Redis cache becomes unavailable, the operations and maintenance module can suspend or degrade non-core functions while maintaining the continuous availability of core business functions, rather than triggering a complete service outage. A dynamic alert level escalation mechanism automatically expands the notification scope when automatic recovery fails or faults recur, ensuring that faults exceeding the limits of automated handling receive timely manual responses. Through fault drill functionality, operations personnel can proactively verify the actual effectiveness of automatic recovery capabilities in isolation from the production environment and continuously optimize recovery strategy parameters based on actual RTO and RPO values.

[0063] The embodiments of the present invention have been described above. However, the embodiments are not limited to the specific implementation methods described above. The specific implementation methods described above are merely illustrative and not restrictive. Those skilled in the art can make more equivalent embodiments under the guidance of the present embodiments, and all of them are within the protection scope of the present embodiments.

Claims

1. A distributed microservice automated recovery method based on multi-dimensional health checks, characterized in that, Includes the following steps: Health checks are performed sequentially in a fixed order: infrastructure layer, service component layer, data storage layer, and business function layer. For the infrastructure layer, raw data in the operating system's virtual file system is read to calculate resource utilization indicators. For the service component layer, process liveness detection, port connectivity detection, and HTTP interface detection are performed sequentially. For the data storage layer, connection percentage, cache hit rate, and cluster status are collected. For the business function layer, the availability of core business processes is verified in an end-to-end liveness detection manner, and health check results are generated for each layer. Summarize the health examination results at all levels, calculate the overall health status according to the predefined status judgment rules, and the overall health status can be one of healthy, degraded or unhealthy. Based on the overall health status and specific abnormal information at each level, automatic recovery operations are triggered according to a four-level progressive strategy. The four-level progressive strategy is as follows: automatic process restart, service failover, service degradation, and full restart. If a lower-level recovery operation fails, it will automatically escalate to the next level. Based on the results of the recovery operation, generate alarm notifications of the corresponding level and send them through multiple channels. Record all recovery operations in the audit log table in an append-only manner.

2. The distributed microservice automated recovery method based on multi-dimensional health checks according to claim 1, wherein, Health checks on the infrastructure layer include: Calculate CPU utilization using the following formula: ,in This is the increment of the idle time slice between two samplings. This is the sum of all types of time slice increments between two samplings; Calculate memory usage using the following formula: ,in Total physical memory. This refers to the amount of free memory. This refers to the kernel buffer usage. Page cache usage This is for the reclaimable Slab memory usage; Calculate disk usage using the following formula: ,in The number of free blocks available to non-privileged processes. This represents the total number of blocks in the file system. when , or An anomaly exceeding the first threshold is marked as non-fatal, and an anomaly exceeding the second threshold is marked as fatal, where the second threshold is greater than the first threshold.

3. The distributed microservice automated recovery method based on multi-dimensional health checks according to claim 1, wherein, Health checks on the data storage layer include: Calculate the percentage of active database connections using the following formula: ,in This represents the current number of active connections. The maximum number of connections allowed by the database, when If the number of connections exceeds the alarm threshold, it is marked as abnormal; Calculate cache hit rate using the following formula: ,in To accumulate cache hits, To accumulate the number of cache misses, when When the hit rate falls below the alarm threshold, it is marked as abnormal; Get the status of the search engine cluster and judge the health of the cluster according to three levels: green, yellow, and red.

4. The distributed microservice automated recovery method based on multi-dimensional health checks according to claim 1, wherein, The health check of the business function layer includes the following sub-steps executed in sequence: sending a request to the business interface to create a test plan with a special tag and recording the plan identifier; triggering the execution of the test task and waiting for the task to complete; verifying the readability of the task execution result record; calling the deletion interface to clean up the test data generated during the liveness detection process; if any sub-step returns an error code or fails to complete within the timeout period, the business function layer is determined to be in an unhealthy state and the failed sub-step identifier is attached to the result.

5. The distributed microservice automated recovery method based on multi-dimensional health checks according to claim 1, wherein, The automatic process restart strategy in the four-level progressive approach includes: sending a start command to the process manager of the target service, waiting for a preset time, and then re-executing the process liveness probe and port connectivity probe to verify the start result; if the verification fails, the exponential backoff waiting time is calculated according to the following formula, and then retrying: ,in For the first Waiting time before the next retry Based on waiting time, This is the current number of retries; if the maximum number of retries is reached and the service still fails, it will be upgraded to service failover.

6. The distributed microservice automated recovery method based on multi-dimensional health checks according to claim 1, wherein, The service failover strategy in the four-level progressive approach includes: setting a global failover status flag and writing it to a distributed lock via atomic operations; selecting a slave database from a pre-configured slave database list that meets the data synchronization delay condition as a candidate new master database; performing a master database promotion operation on the candidate slave database and verifying its ability to accept write requests; updating the global database connection configuration through the configuration center and notifying all service instances to reload; clearing the global failover lock after performing write tests and read verifications on the new master database; and creating a checkpoint snapshot of the current system state before executing the above steps. If any step fails, the compensation operations of each step are executed in reverse order to roll back the system state to the checkpoint state.

7. The distributed microservice automated recovery method based on multi-dimensional health checks according to claim 1, wherein, The service degradation strategy in the four-level progressive approach includes: when the search engine is unavailable, suspend writing logs to it and temporarily store the data to be written in a local memory queue. When the queue capacity limit is exceeded, discard the oldest data according to the least recently used strategy. Interfaces that rely on its full-text search function return a predefined degradation response. After the search engine recovers, the temporarily stored data will be automatically replenished in batches. When the caching service is unavailable, revert session management to local memory caching mode, reduce the maximum number of sessions, shorten the session timeout time, and activate the session affinity strategy for request routing.

8. The distributed microservice automated recovery method based on multi-dimensional health checks according to claim 1, wherein, The full restart in the four-level progressive strategy includes: sending stop commands in reverse order of the service dependency graph topology, and forcibly terminating the process after a timeout; cleaning up the temporary file directories, process lock files, and local runtime cache directories of each service; starting each service in ascending order of the service dependency graph topology, waiting for its health check to pass after each service is started before starting the next service; and performing a complete four-level health check after all services have started. If any service still cannot be recovered, the alarm level will be raised to the highest level and a manual intervention notification will be triggered.

9. The distributed microservice automated recovery method based on multi-dimensional health checks according to claim 1, wherein, The generation of alarm notifications includes: alarm levels are divided into four levels: low, medium, high, and critical; when a fault that triggers an alarm fails again after a health check following automatic recovery operations, or when the same fault triggers an alarm more than a preset number of times within a preset time window, the current alarm level will be raised by one level and the alarm will be resent; alarm notifications are sent through four channels: email, SMS, enterprise instant messaging platform Webhook, and general Webhook callback, each of which is independently configured with its enabled status and applicable minimum alarm level.

10. A distributed microservice automated recovery system based on multi-dimensional health checks, characterized in that, include: The health check data collection module is used to perform health checks in a fixed order of infrastructure layer, service component layer, data storage layer and business function layer, collect indicator data at each layer, and generate health check results at each layer. The health status calculation module is used to summarize the health examination results at all levels, calculate the overall health status according to the predefined status judgment rules, and the overall health status can be one of healthy, degraded or unhealthy. The overall health status and its basis are recorded in the system status table. The automatic recovery execution module is used to trigger and execute automatic recovery operations based on the overall health status and specific abnormal information at each level, according to a four-level progressive strategy of automatic process restart, service failover, service degradation and full restart. It automatically upgrades to the next level after a low-level recovery operation fails, and creates a system status checkpoint snapshot before the execution of service failover and higher-level recovery operations. The alarm notification and audit module is used to generate alarm notifications of corresponding levels based on the results of recovery operations and send them through multiple channels. When the fault persists or occurs repeatedly, the alarm level is automatically escalated, and all recovery operations are recorded in the audit log table in an append-only manner.

Images