Backend system fault self-recovery method and device, storage medium and computer equipment

Abstract

The back-end system fault self-recovery method, device, storage medium and computer equipment provided by the application perform multi-dimensional state analysis on the running data collected by the system micro service in real time during system operation to obtain a running state; when the running state is abnormal, the running data is analyzed by using a fault analysis model to improve the efficiency of fault positioning; after obtaining the fault data output by the model, a self-recovery strategy corresponding to the fault data is matched from a dynamic knowledge base, and then a self-recovery operation is performed on the micro service to realize rapid response of fault processing. If the self-recovery result is a failure, self-recovery data in the self-recovery operation process is obtained, the self-recovery strategy of the micro service is adjusted to make it more suitable for the actual fault condition, and then the self-recovery operation is performed on the micro service based on the new self-recovery strategy until the self-recovery result meets the self-recovery end condition, so as to improve the flexibility of system fault self-recovery and minimize the influence of the fault on the stability of the system.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, storage medium and computer equipment for self-healing of back-end system faults. Background Technology

[0002] With the rapid development of internet technology, microservice architecture has been widely used in backend system development due to its high cohesion, low coupling, and ease of expansion. Microservice architecture breaks down a complex backend system into multiple independently running and deployable microservices, each focused on completing a specific business function. However, this architectural pattern also brings new challenges. Due to the large number of services and their complex dependencies, a failure in any microservice can lead to malfunctions or even crashes of the entire backend system.

[0003] Currently, when a microservice in a system fails, a simple automatic restart strategy can be adopted, which involves directly restarting the service when it becomes unresponsive to restore service. However, this strategy only applies to failures caused by temporary resource consumption or brief program anomalies. For complex failures such as code defects, configuration errors, or permanent resource damage, the failure will reappear after restarting the service. As a result, operations and maintenance personnel need to manually troubleshoot the problem and perform operations such as restarting and configuration adjustments. However, manual intervention relies on operational experience and is prone to further escalation of the failure due to slow response times and low troubleshooting efficiency. Summary of the Invention

[0004] The purpose of this application is to at least address one of the aforementioned technical defects, particularly the technical defect in the prior art where system fault handling requires manual intervention, relies on operational experience, and is prone to further escalation of faults due to slow response speed and low troubleshooting efficiency.

[0005] This application provides a backend system fault self-healing method, the method comprising:

[0006] The system collects operational data of microservices in real time and performs multi-dimensional state analysis on the operational data based on multi-dimensional detection rules to obtain the operational status of the system microservices.

[0007] When the operating state is abnormal, the operating data is analyzed using a preset fault analysis model to obtain fault data;

[0008] The system obtains a self-healing strategy corresponding to the fault data from the dynamic knowledge base, and performs self-healing operations on the system microservices based on the fault data and the self-healing strategy.

[0009] If the self-healing result fails, the self-healing data of the system microservice during the self-healing operation is obtained, and the self-healing strategy of the system microservice is adjusted based on the self-healing data. The new self-healing strategy is then used to re-execute the self-healing operation on the system microservice until the self-healing result meets the self-healing termination condition.

[0010] Optionally, the real-time acquisition system microservice operation data includes:

[0011] A lightweight probe program is used to collect real-time operational status data of system microservices; the lightweight probe program is deployed within the system microservices and at key nodes in the call chain.

[0012] Distributed tracing technology is used to track the runtime status data to obtain inter-service call chain information and request-response data.

[0013] The system microservice's runtime data is generated based on the runtime status data, the call chain information, and the request-response data.

[0014] Optionally, the step of performing multi-dimensional state analysis on the running data based on multi-dimensional detection rules to obtain the running state of the system microservices includes:

[0015] Extract the detection data corresponding to each detection dimension from the running data, and perform state detection on the detection data of each detection dimension based on the multi-dimensional detection rules to obtain the detection result of each detection dimension.

[0016] The system's microservices are comprehensively analyzed based on the various detection results to obtain their operational status.

[0017] Optionally, the detection dimensions include service process status, interface call status, database connection status, resource usage, request response time, and call chain anomalies;

[0018] The state detection is performed on the detection data of each detection dimension based on the multi-dimensional detection rules to obtain the detection result of each detection dimension, including:

[0019] Based on the detection data of the service process status, the process liveness status and restart frequency are detected to obtain the first detection result;

[0020] Based on the detection data of the interface call status, the request success rate, call frequency, and timeout count are detected to obtain a second detection result;

[0021] Based on the detection data of the database connection status, the connection success rate, the number of connections, and the number of connection timeouts are detected to obtain a third detection result;

[0022] Based on the detection data of the resource usage, CPU utilization, memory usage, disk I / O and network bandwidth are detected to obtain the fourth detection result;

[0023] The average response time and 95th percentile response time are detected based on the detection data of the request response time, and the fifth detection result is obtained;

[0024] Based on the detection data of the call chain anomalies, the service call failure rate and call timeout ratio are detected, and the sixth detection result is obtained.

[0025] Optionally, the step of using a preset fault analysis model to perform fault analysis on the operating data to obtain fault data includes:

[0026] The operational data is input into a preset fault analysis model so that the fault analysis model performs abnormal indicator trend analysis based on the operational data and outputs the fault type and impact range as fault data according to the analysis results.

[0027] Optionally, adjusting the self-healing strategy of the system microservices based on the self-healing data includes:

[0028] The self-healing data, the operational data, and the fault data are input into the fault analysis model for fault identification and data correction, resulting in the corrected fault data output by the fault analysis model.

[0029] Generate self-healing tags for the system microservices, and based on the corrected fault data, match the self-healing strategy with the highest similarity to the self-healing tag from the dynamic knowledge base;

[0030] The self-healing tag is used to define the self-healing result of the self-healing strategy that has been executed by the system microservice.

[0031] Optionally, the management process of the dynamic knowledge base includes:

[0032] A dynamic knowledge base is constructed based on historical fault data and the historical processing experience corresponding to the historical fault data; the historical processing experience includes self-healing strategies for multiple different fault scenarios.

[0033] Whenever a system microservice failure is successfully handled, the failure handling data of the system microservice is obtained, and the dynamic knowledge base is optimized based on the failure handling data.

[0034] This application also provides a backend system fault self-healing device, including:

[0035] The status analysis module is used to collect the running data of the system microservices in real time, and perform multi-dimensional status analysis on the running data based on multi-dimensional detection rules to obtain the running status of the system microservices.

[0036] The fault analysis module is used to perform fault analysis on the operating data using a preset fault analysis model when the operating state is abnormal, and to obtain fault data.

[0037] The service self-healing module is used to match the self-healing strategy corresponding to the fault data from the dynamic knowledge base, and to perform self-healing operations on the system microservices based on the fault data and the self-healing strategy.

[0038] The strategy adjustment module is used to obtain the self-healing data of the system microservice during the self-healing operation if the self-healing result fails, adjust the self-healing strategy of the system microservice based on the self-healing data, and re-execute the self-healing operation of the system microservice using the new self-healing strategy until the self-healing result meets the self-healing termination condition.

[0039] This application also provides a storage medium storing computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the back-end system fault self-healing method as described in any of the above embodiments.

[0040] This application also provides a computer device, including: one or more processors, and memory;

[0041] The memory stores computer-readable instructions, which, when executed by the one or more processors, perform the steps of the back-end system fault self-healing method as described in any of the above embodiments.

[0042] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0043] The backend system fault self-healing method, apparatus, storage medium, and computer equipment provided in this application can collect the operational data of system microservices in real time during the operation of the backend system, and perform multi-dimensional state analysis on the operational data based on multi-dimensional detection rules, thereby more accurately capturing complex operational characteristics and determining the operational status of system microservices. When an abnormal operational status of a microservice is detected, a pre-trained fault analysis model can be used to perform automated fault analysis on the operational data, thereby improving the efficiency of complex fault location. After obtaining the fault data output by the fault analysis model, a self-healing strategy corresponding to the fault data can be matched from a dynamic knowledge base. Then, based on the fault data and the self-healing strategy, self-healing operations can be performed on the system microservices to achieve rapid response in fault handling. Furthermore, if the current self-healing result fails, the self-healing data of the system microservice during this self-healing operation can be obtained, and the self-healing strategy of the system microservice can be adjusted based on the self-healing data to make it more adaptable to the actual fault situation. Then, the self-healing operation of the system microservice can be re-executed based on the new self-healing strategy until the self-healing result meets the self-healing termination condition. This can improve the flexibility of system fault self-healing and minimize the impact of faults on system stability. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 A flowchart illustrating a backend system fault self-healing method provided in an embodiment of this application;

[0046] Figure 2 A flowchart illustrating a data acquisition process provided in this application embodiment;

[0047] Figure 3 A schematic diagram of a backend system fault self-healing device provided in an embodiment of this application;

[0048] Figure 4 This is a schematic diagram of the internal structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0049] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0050] Currently, when a microservice in a system fails, a simple automatic restart strategy can be adopted, which involves directly restarting the service when it becomes unresponsive to restore service. However, this strategy only applies to failures caused by temporary resource consumption or brief program anomalies. For complex failures such as code defects, configuration errors, or permanent resource damage, the failure will reappear after restarting the service. As a result, operations and maintenance personnel need to manually troubleshoot the problem and perform operations such as restarting and configuration adjustments. However, manual intervention relies on operational experience and is prone to further escalation of the failure due to slow response times and low troubleshooting efficiency.

[0051] Based on this, this application proposes the following technical solution, as detailed below:

[0052] In some embodiments, the backend system fault self-healing method provided in this application can be applied to a service fault self-healing system, which consists of a lightweight probe program, a fault detection module, a self-healing execution module, and a knowledge base module. The lightweight probe program is deployed within each microservice and on key nodes of the service call chain, primarily used to collect relevant data for each microservice. The fault detection module is mainly used to detect and locate faults in the relevant data collected by the lightweight probe program. The self-healing execution module is mainly used to perform self-healing operations on faulty microservices using corresponding self-healing strategies, where the self-healing strategies are derived from a dynamic knowledge base in the knowledge base module. Therefore, through the service fault self-healing system, this application can quickly detect faulty microservices in the system and respond with timely self-healing.

[0053] In one embodiment, such as Figure 1 As shown, Figure 1 This is a flowchart illustrating a backend system fault self-healing method provided in an embodiment of this application. The present application provides a backend system fault self-healing method, specifically including the following:

[0054] S110: Collects real-time operational data of system microservices and performs multi-dimensional status analysis on the operational data based on multi-dimensional detection rules to obtain the operational status of system microservices.

[0055] In this step, during the operation of the backend system, the service fault self-healing system can collect the operation data of the system microservices in real time, and perform multi-dimensional status analysis on the operation data based on multi-dimensional detection rules, so as to more accurately capture complex operation characteristics and determine the current operation status of the system microservices in a timely manner.

[0056] It should be noted that, since there are multiple microservices in the system, this application can collect runtime data and analyze runtime status of each system microservice in parallel, so as to respond promptly when system microservice runtime anomalies occur.

[0057] Specifically, in the service fault self-healing device, a lightweight probe program continuously and in real time collects relevant operational data such as CPU utilization, memory usage, response time, and request success rate of each microservice in the system with extremely low overhead. The collected operational data is then standardized and transmitted to the fault detection module at a preset frequency, so that the fault detection module can use multi-dimensional detection rules to perform multi-dimensional state analysis on the operational data, thereby accurately perceiving the current operational status of the system's microservices.

[0058] It is understandable that the multi-dimensional detection rules here refer to a set of rules that comprehensively judge the operational status of system microservices based on multiple different angles and dimensions of indicators and logic. It can perform deep correlation and real-time analysis of operational data from various dimensions. Unlike traditional single-point threshold alarms, the multi-dimensional status analysis of this application can accurately capture complex operational characteristics and abnormal patterns such as slowly increasing latency, sporadic error surges, early signs of resource leaks, and cascading risks of service dependencies, thereby significantly improving the timeliness and accuracy of fault detection and diagnosis.

[0059] S120: When the operating status is abnormal, the operating data is analyzed using a preset fault analysis model to obtain fault data.

[0060] In this step, when the system microservice is detected to be in an abnormal running state through step S110, the fault detection module can use a pre-trained fault analysis model to automatically analyze the running data and obtain fault data, thereby improving the efficiency of complex fault location.

[0061] Of course, if the system microservices are running normally, the service failure self-healing system can continue to perform routine data monitoring on them.

[0062] Understandably, the fault analysis module can combine system architecture information and historical fault data, employing machine learning algorithms such as decision trees and Bayesian networks to construct a fault analysis model. Therefore, when further analysis of a malfunctioning system microservice is required, the fault analysis module can directly utilize this model to perform in-depth fault profiling of its operational data, including potential and actual faults. During the model-based fault analysis process, the model can trace the root cause of the fault by correlating and analyzing the dependencies between services, call chains, and anomaly indicators, thereby determining the fault type and scope of impact.

[0063] Furthermore, as the system continuously encounters new faults and completes repairs during actual operation, this application can obtain fault data after each fault is resolved, using it as input for retraining the fault analysis model to achieve continuous optimization of the model.

[0064] S130: Match the self-healing strategy corresponding to the fault data from the dynamic knowledge base, and perform self-healing operations on the system microservices based on the fault data and the self-healing strategy.

[0065] In this step, after generating the fault data of the system microservice through step S120, the self-healing execution module can obtain the fault data, and then match the self-healing strategy corresponding to the fault data from the dynamic knowledge base in the knowledge base module. Based on the fault data and the self-healing strategy, the self-healing operation can be performed on the system microservice to achieve a rapid response to fault handling.

[0066] The dynamic knowledge base refers to a knowledge management system that is constantly updated and evolved based on system operation and historical experience. It stores self-healing strategies for multiple fault scenarios and is a key foundation for achieving intelligent and automated system operation and maintenance. Through self-healing strategies, the self-healing execution module can perform a series of predefined or intelligently generated repair operations on faulty microservices to achieve rapid fault recovery, stable system operation, and continuous service availability.

[0067] Specifically, the self-healing execution module can call the knowledge base module based on key information such as the fault type and scope of impact in the fault data. It can then match the self-healing strategy that is highly corresponding to the current fault scenario from the dynamic knowledge base. Based on the operation steps, execution conditions and dependencies in the self-healing strategy, and combined with the current system running status, the module can perform self-healing operations on the system microservices, such as restarting instances, releasing resources, switching standby nodes, adjusting configuration parameters or repairing dependency links, to ensure that faults can be handled quickly without relying on manual intervention.

[0068] It is understandable that by automatically matching and executing the self-healing strategy, this application can effectively shorten the fault response and recovery time, avoid service interruption or system-level failure caused by fault propagation, and significantly improve the continuity and service availability of the system.

[0069] S140: If the self-healing result fails, obtain the self-healing data of the system microservice during the self-healing operation, adjust the self-healing strategy of the system microservice based on the self-healing data, and re-execute the self-healing operation of the system microservice using the new self-healing strategy until the self-healing result meets the self-healing termination condition.

[0070] In this step, after performing a self-healing operation on the system microservice in step S130, if the self-healing result fails, the self-healing execution module can obtain the self-healing data of the system microservice during this self-healing operation, and adjust the self-healing strategy of the system microservice based on the self-healing data to make it more adaptable to the actual fault situation. Then, the self-healing operation can be re-executed on the system microservice based on the new self-healing strategy until the self-healing result meets the self-healing termination condition. This can improve the flexibility of system fault self-healing and minimize the impact of faults on system stability.

[0071] Specifically, if the self-healing result fails, it indicates that the microservice's operational status has not yet returned to the expected normal level. At this point, the self-healing execution module can immediately initiate a backtracking mechanism to automatically acquire the self-healing data generated by the system microservice during this self-healing operation. This includes the specific steps of the operation, the execution status of each step, real-time trends of system metrics, side effect information triggered by the operation, and related log records, among other multi-dimensional data. Based on this self-healing data, the self-healing execution module can intelligently adjust the original self-healing strategy to obtain a more targeted strategy. Subsequently, the self-healing execution module can re-execute the self-healing operation on the system microservice according to the new strategy until the self-healing result meets the self-healing termination conditions. This process will continue to iterate until the self-healing operation is successfully completed or the protective termination conditions are met, thereby constructing a closed-loop, evolvable self-healing control process.

[0072] Understandably, the self-healing termination condition here can refer to successful self-healing, meaning the system microservices' operational status returns to the expected normal level, or it can be that the number of self-healing attempts exceeds the self-healing retry threshold. When the number of self-healing attempts exceeds the self-healing retry threshold, the self-healing execution module can automatically trigger a manual intervention process, such as sending alarms to operations and maintenance personnel, thereby allowing for manual troubleshooting and ensuring the stability of system operation.

[0073] In the above embodiments, during the operation of the backend system, the operational data of the system microservices can be collected in real time, and multi-dimensional state analysis can be performed on the operational data based on multi-dimensional detection rules. This allows for more accurate capture of complex operational characteristics and determination of the operational status of the system microservices. When an abnormal operational status of a microservice is detected, a pre-trained fault analysis model can be used to perform automated fault analysis on the operational data, thereby improving the efficiency of complex fault location. After obtaining the fault data output by the fault analysis model, a self-healing strategy corresponding to the fault data can be matched from a dynamic knowledge base. Then, based on the fault data and the self-healing strategy, a self-healing operation can be performed on the system microservices to achieve rapid response in fault handling. Furthermore, if the current self-healing result is a failure, the self-healing data of the system microservices during this self-healing operation can be obtained. Based on this self-healing data, the self-healing strategy of the system microservices can be adjusted to better adapt to the actual fault situation. Then, the self-healing operation can be re-executed on the system microservices based on the new self-healing strategy until the self-healing result meets the self-healing termination condition. This improves the flexibility of system fault self-healing and minimizes the impact of faults on system stability.

[0074] In one embodiment, such as Figure 2 As shown, Figure 2 A flowchart illustrating a data acquisition process provided in this application embodiment; Figure 2 In step S110, the process of collecting real-time operational data of system microservices may include:

[0075] S111: Real-time collection of system microservice runtime status data through a lightweight probe program; the lightweight probe program is deployed within the system microservice and at key nodes in the call chain.

[0076] S112: Distributed tracing technology is used to track runtime status data and obtain inter-service call chain information and request-response data.

[0077] S113: Generate the runtime data of the system microservices based on the runtime status data, call chain information, and request-response data.

[0078] In this embodiment, the runtime data may include runtime status data, call chain information, and request-response data. When collecting runtime status data, a lightweight probe program deployed within the system microservices and on key nodes of the call chain can collect relevant data on the system microservices in real time, including service process status, interface call status, database connection status, and resource usage, forming runtime status data. Then, distributed tracing technology can be used to trace the runtime status data to obtain call chain information and request-response data between services.

[0079] Specifically, lightweight probe programs can be embedded into the service execution environment in a non-intrusive and low-resource-consumption manner. They can collect core indicator data during system operation in real time and at high frequency without interrupting business processes. This includes the running status of service processes, the number of API calls and latency distribution, the activity and abnormal situations of database connections, the usage of computing resources such as memory and CPU, and detailed information such as thread pool and queue status. After unified standardization and structured processing, this data can form comprehensive running status data, which can then serve as the basic data support for the fault detection module to perceive and locate faults.

[0080] Subsequently, the lightweight probe program can introduce distributed tracing technology to perform end-to-end data tracing on the collected runtime status data. By fully restoring the request path, accurately marking cross-service calls, and visually modeling upstream and downstream dependencies, it can further generate detailed inter-service call chain information and request response data such as response latency, return code, and failure type for each request. These two data can be used to improve data correlation and temporal integrity, providing a contextual perspective for fault localization, so that the fault detection module can detect complex operational anomalies such as latency bottlenecks, dependency failures, and link blockages.

[0081] In one embodiment, the process of performing multi-dimensional state analysis on the running data based on preset multi-dimensional detection rules to obtain the running status of the system microservices in step S110 may include:

[0082] S114: Extract the detection data corresponding to each detection dimension from the running data, and perform state detection on the detection data of each detection dimension based on the multi-dimensional detection rules to obtain the detection result of each detection dimension.

[0083] S115: Based on the various detection results, a comprehensive analysis of the system microservices is performed to obtain the operating status.

[0084] In this embodiment, when performing multi-dimensional state analysis on the running data, the fault detection module can extract the detection data corresponding to each detection dimension from the running data, and perform state detection on the detection data of each detection dimension based on the multi-dimensional detection rules to obtain the detection results of each detection dimension. Then, the system microservices can be comprehensively analyzed based on the detection results to obtain the running status.

[0085] Among them, multi-dimensional detection rules refer to a set of judgment criteria set based on different detection dimensions in system operation data to evaluate the operating status of microservices. They are used to conduct independent and targeted status analysis on the operating indicators of each dimension in order to identify potential anomalies, performance degradation or fault symptoms.

[0086] Specifically, the fault detection module first performs structured parsing of the collected operational data, classifying the data according to preset detection dimensions, and extracting the detection data corresponding to each detection dimension, such as service process status, interface call status, database connection status, resource usage, request response time, and call chain anomalies. Subsequently, the fault detection module can independently assess the status of data within each detection dimension based on pre-set multi-dimensional detection rules, calculating the detection result for each dimension using techniques such as statistical threshold discrimination, trend analysis, behavioral pattern recognition, or anomaly distribution fitting. Building upon this, the fault detection module can further integrate the detection results from all dimensions, using comprehensive judgment mechanisms such as weighted scoring, logical reasoning, Bayesian inference, or decision tree analysis to form an accurate assessment of the overall operational status of the system's microservices, determining whether it is in a healthy, sub-healthy, or faulty state, and outputting the final assessment result.

[0087] Understandably, through multi-dimensional state analysis, this application can effectively improve the ability to identify complex operating states, no longer relying on the fluctuation of a single indicator to judge the health of the system, but capturing more comprehensive and hidden fault signs through multi-dimensional data fusion, thereby significantly improving the sensitivity and accuracy of system anomaly detection.

[0088] In one embodiment, the detection dimensions in step S114 may include service process status, interface call status, database connection status, resource usage, request response time, and call chain anomalies; wherein, the process of performing status detection on the detection data of each detection dimension based on multi-dimensional detection rules to obtain the detection result of each detection dimension may include:

[0089] S1141: Based on the detection data of the service process status, detect the process liveness status and restart frequency to obtain the first detection result.

[0090] S1142: Based on the detection data of interface call status, perform request success rate, call frequency and timeout count detection to obtain the second detection result.

[0091] S1143: Based on the database connection status detection data, perform connection success rate, number of connections, and number of connection timeouts to obtain the third detection result.

[0092] S1144: Based on the resource usage detection data, CPU utilization, memory usage, disk I / O, and network bandwidth are detected to obtain the fourth detection result.

[0093] S1145: Based on the detection data of the request response time, the average response time and 95th percentile response time are detected to obtain the fifth detection result.

[0094] S1146: Based on the detection data of call chain anomalies, the service call failure rate and call timeout ratio are detected, and the sixth detection result is obtained.

[0095] In this embodiment, the multi-dimensional detection rules include several detection dimensions such as service process status, interface call status, database connection status, resource usage, request response time and call chain anomalies, as well as the detection content corresponding to each detection dimension. Therefore, the fault detection module can obtain the detection data of each detection dimension and perform fault detection on it according to the corresponding detection content, thereby obtaining the detection results of each detection dimension.

[0096] Specifically, for service process status, the fault detection module can detect process liveness and restart frequency based on its detection data. For example, if a process is detected to have terminated or restarted more than 3 times within 5 minutes, it indicates abnormal data in the first detection result. For API call status, the fault detection module can detect request success rate, call frequency, and timeout count based on its detection data. For example, if a request success rate is less than 90%, the call volume increases / decreases by more than 50% from the baseline within 1 hour, or the number of API timeout calls exceeds 10, it indicates abnormal data in the second detection result. For database connection status, the fault detection module can detect connection success rate, number of connections, and number of connection timeouts based on its detection data. For example, if a database connection success rate is less than 85% for 30 seconds, the number of connections exceeds 90% of the maximum number of connections for 1 minute, or the number of connection timeouts exceeds 5 consecutive times, it indicates abnormal data in the third detection result. For resource usage, the fault detection module can detect CPU usage based on its detection data. Usage, memory usage, disk I / O, and network bandwidth are detected. For example, if CPU usage exceeds 80% for 5 consecutive minutes, memory usage exceeds 90% for 3 consecutive minutes, disk I / O read / write latency exceeds 500ms for 1 consecutive minute, or network bandwidth usage exceeds 90% for 2 consecutive minutes, it indicates abnormal data in the fourth detection result. For request response time, the fault detection module can perform average response time and 95th percentile response time detection based on its detection data. For example, if the average response time is greater than 500ms for 1 consecutive minute or the 95th percentile response time is greater than 1000ms for 30 seconds, it indicates abnormal data in the fifth detection result. For call chain anomalies, the fault detection module can perform average response time and 95th percentile response time detection based on its detection data. For example, if the call failure rate of a node in the chain exceeds 15% for 1 consecutive minute or the call timeout rate exceeds 20% for 30 seconds, it indicates abnormal data in the sixth detection result.

[0097] In one embodiment, the process of performing fault analysis on the operating data using a preset fault analysis model to obtain fault data in step S120 may include:

[0098] S121: Input the operating data into the preset fault analysis model so that the fault analysis model can perform abnormal indicator trend analysis based on the operating data, and output the fault type and impact range as fault data according to the analysis results.

[0099] In this embodiment, after determining that the current running state of the system microservice is abnormal, the fault detection system can input its running data into a preset fault analysis model, so that the fault analysis model can perform abnormal indicator trend analysis based on the running data, and output the fault type and impact range according to the analysis results as fault data, thereby achieving accurate fault location.

[0100] Specifically, since the fault analysis model is trained by combining historical fault data and system architecture information, it has the ability to extract high-dimensional features and identify abnormal patterns from operational data. Based on this, after the fault detection module inputs the operational data of the system microservices into the fault analysis model, the model can conduct in-depth analysis of the key indicator dimensions involved in the operational data, paying particular attention to the temporal trends, fluctuation patterns, and cross-dimensional correlations of abnormal indicators such as resource usage, call chain latency, interface error rate, and log anomaly frequency. Through trend analysis, change rate calculation, and anomaly window identification, the model can uncover the core data causing the current operational anomaly. Then, the fault analysis model can further comprehensively determine the fault type, such as resource shortage faults, code logic error faults, network faults, database connection faults, etc., and infer the impact range of the fault based on call chain tracing data and resource dependency graphs, including directly affected services, upstream dependencies, and downstream propagation paths, thereby outputting structured fault type and impact range information as standardized fault data.

[0101] For example, through in-depth analysis of operational data using a fault analysis model, if the analysis results indicate abnormal resource usage data, such as CPU utilization consistently exceeding 80%, memory usage exceeding the threshold of 90%, disk I / O read / write latency exceeding 500ms, or indirect abnormalities in interface calls, such as prolonged interface response time or decreased request success rate due to insufficient resources, then the fault analysis model can determine that the system microservice is experiencing a resource shortage fault. If the analysis results indicate abnormal interface call data, such as a sudden drop in the success rate of specific interface requests, failures concentrated on business logic errors, or indirect abnormalities in service process status, such as unresponsive service processes due to code infinite loops or frequent process crashes and restarts, then... The fault analysis model can determine that the system microservice has a code logic error fault. If the analysis result shows abnormal call chain information, such as a sudden increase in the proportion of inter-service call timeouts that only occurs on a specific network link, or abnormal network resource data, such as drastic fluctuations in network bandwidth and a packet loss rate exceeding 5%, then the fault analysis model can determine that the system microservice has a network fault. If the analysis result shows abnormal database connection status data, such as a database connection success rate dropping below 80% or the number of connections frequently reaching 95% of the maximum number of connections, or abnormal interface call behavior, such as a synchronous decrease in the success rate of interface requests that depend on the database and an extended response time, then the fault analysis model can determine that the system microservice has a database connection fault.

[0102] In one embodiment, the process of adjusting the self-healing strategy of the system microservices based on self-healing data in step S140 may include:

[0103] S141: Input the self-healing data, operational data, and fault data into the fault analysis model for fault identification and data correction, and obtain the corrected fault data output by the fault analysis model.

[0104] S142: Generate self-healing tags for system microservices, and based on the corrected fault data, match the self-healing strategy with the highest similarity to the self-healing tag from the dynamic knowledge base.

[0105] In this embodiment, when adjusting the self-healing strategy of the system microservice, the fault detection module can input self-healing data, running data, and fault data into the fault analysis model for fault identification and data correction, thereby obtaining the corrected fault data output by the fault analysis model. Then, a self-healing tag for the system microservice can be generated. This self-healing tag can be used to define the self-healing result of the self-healing strategy that the system microservice has already executed. Therefore, based on the corrected fault data, the fault detection module matches the self-healing strategy with the highest similarity to the self-healing tag from the dynamic knowledge base.

[0106] Specifically, during the adjustment of the self-healing strategy of the system's microservices, the fault detection module can input self-healing data, operational data, and initially obtained fault data into the fault analysis model. This allows the model to more deeply identify and evaluate the current fault based on the fusion of data from different dimensions, and to correct for potential misjudgments, incomplete information, or state deviations in the original fault data. The fault analysis model analyzes the actual execution path of the self-healing operation, the intervention results of the self-healing operation on the service state, and the changing trends of fault indicators. Combined with system state fluctuations, resource consumption changes, and dependency chain responses in the operational data, it performs multi-dimensional trend analysis and causal reasoning, thereby outputting corrected fault data.

[0107] Based on the corrected fault data, the fault detection module can further generate self-healing tags for the current running state of the system microservices. These tags detail the self-healing strategies, results, recovery time, and fault persistence experienced by the microservices, serving as an empirical summary of the self-healing process already performed by the microservices. Therefore, the fault detection module can use these self-healing tags as query vectors to retrieve the self-healing strategy with the highest feature matching degree from the dynamic knowledge base. Through mechanisms such as vector similarity calculation, semantic matching, or contextual structure comparison, it can accurately select the optimal self-healing solution that best matches the current fault scenario from the dynamic knowledge base, further improving the accuracy and success rate of the self-healing strategy.

[0108] For example, in a faulty microservice, the fault analysis model outputs a resource shortage fault as the fault data for that microservice. After the self-healing operation using the self-healing strategy corresponding to this fault type fails, the fault analysis model can perform in-depth fault identification and fault data correction on the microservice based on the self-healing data during this self-healing operation. The resource shortage fault can be refined into uneven resource allocation and abnormal instance scheduling. Then, when the fault analysis model matches the newly generated fault data with self-healing strategies from the dynamic database, it can prioritize matching fault scenarios that adjust strategies after self-healing failure based on their self-healing labels. This allows for the matching of self-healing strategies with high similarity to fault scenarios, further improving the self-healing success rate of the microservice.

[0109] In one embodiment, the management process of the dynamic knowledge base in step S130 may include:

[0110] S131: A dynamic knowledge base is constructed based on historical fault data and the historical processing experience corresponding to the historical fault data; the historical processing experience includes self-healing strategies for multiple different fault scenarios.

[0111] S132: Whenever a system microservice failure is successfully handled, obtain the failure handling data of the system microservice and optimize the dynamic knowledge base based on the failure handling data.

[0112] In this embodiment, the application can embed a dynamic knowledge base in the knowledge base module. This dynamic knowledge base is constructed based on historical fault data and corresponding historical processing experience, whereby the historical processing experience covers self-healing strategies for multiple different fault scenarios. Furthermore, whenever a system microservice fault is successfully handled, the application can also obtain the fault handling data of that system microservice and optimize the dynamic knowledge base based on the fault handling data.

[0113] Understandably, the self-healing strategies from historical processing experience not only include fault types, triggering conditions, and impact scope, but also record specific self-healing actions, operation sequences, execution feedback results, and applicable conditions and precautions, thus forming a structured strategy knowledge graph. This provides high-quality reference and strategy matching support for subsequent fault handling. As the system continuously encounters new faults and completes repair processing during actual operation, this application can also automatically acquire fault handling data of the corresponding microservice during each successful fault handling process. This data includes key information such as fault characteristics, self-healing strategies, processing results, and system state changes. This data is then used as new data samples and input into the knowledge base update mechanism to perform reinforcement learning, feature supplementation, or path optimization on existing strategies. This enables real-time correction and expansion of the dynamic knowledge base content, ensuring that the dynamic knowledge base continuously adapts to changes in the current system structure, business model, and operating environment.

[0114] Furthermore, this application can also periodically maintain and organize the dynamic knowledge base, delete invalid or outdated self-healing strategies, and ensure the accuracy and effectiveness of the dynamic knowledge base.

[0115] The back-end system fault self-healing device provided in the embodiments of this application is described below. The back-end system fault self-healing device described below and the back-end system fault self-healing method described above can be referred to in correspondence.

[0116] In one embodiment, such as Figure 3 As shown, Figure 3 This application provides a schematic diagram of a backend system fault self-healing device according to an embodiment of the present application. The present application also provides a backend system fault self-healing device, including a status analysis module 210, a fault analysis module 220, a service self-healing module 230, and a strategy adjustment module 240, specifically comprising the following:

[0117] The status analysis module 210 is used to collect the running data of the system microservices in real time, and perform multi-dimensional status analysis on the running data based on multi-dimensional detection rules to obtain the running status of the system microservices.

[0118] The fault analysis module 220 is used to perform fault analysis on the operating data using a preset fault analysis model when the operating status is abnormal, and obtain fault data.

[0119] The service self-healing module 230 is used to match the self-healing strategy corresponding to the fault data from the dynamic knowledge base, and to perform self-healing operations on the system microservices based on the fault data and the self-healing strategy.

[0120] The strategy adjustment module 240 is used to obtain the self-healing data of the system microservice during the self-healing operation if the self-healing result fails, adjust the self-healing strategy of the system microservice based on the self-healing data, and re-execute the self-healing operation of the system microservice using the new self-healing strategy until the self-healing result meets the self-healing termination condition.

[0121] In the above embodiments, during the operation of the backend system, the operational data of the system microservices can be collected in real time, and multi-dimensional state analysis can be performed on the operational data based on multi-dimensional detection rules. This allows for more accurate capture of complex operational characteristics and determination of the operational status of the system microservices. When an abnormal operational status of a microservice is detected, a pre-trained fault analysis model can be used to perform automated fault analysis on the operational data, thereby improving the efficiency of complex fault location. After obtaining the fault data output by the fault analysis model, a self-healing strategy corresponding to the fault data can be matched from a dynamic knowledge base. Then, based on the fault data and the self-healing strategy, a self-healing operation can be performed on the system microservices to achieve rapid response in fault handling. Furthermore, if the current self-healing result is a failure, the self-healing data of the system microservices during this self-healing operation can be obtained. Based on this self-healing data, the self-healing strategy of the system microservices can be adjusted to better adapt to the actual fault situation. Then, the self-healing operation can be re-executed on the system microservices based on the new self-healing strategy until the self-healing result meets the self-healing termination condition. This improves the flexibility of system fault self-healing and minimizes the impact of faults on system stability.

[0122] In one embodiment, the state analysis module 210 may include:

[0123] The data acquisition submodule is used to collect real-time runtime status data of system microservices through a lightweight probe program; the lightweight probe program is deployed inside the system microservices and at key nodes in the call chain.

[0124] The data tracing submodule is used to trace runtime status data using distributed tracing technology to obtain inter-service call chain information and request-response data.

[0125] The data generation submodule is used to generate runtime data for system microservices based on runtime status data, call chain information, and request-response data.

[0126] In one embodiment, the state analysis module 210 may further include:

[0127] The multi-dimensional detection submodule is used to extract the detection data corresponding to each detection dimension from the running data, and perform state detection on the detection data of each detection dimension based on the multi-dimensional detection rules to obtain the detection result of each detection dimension.

[0128] The comprehensive analysis submodule is used to perform comprehensive analysis of the system microservices based on the various detection results to obtain the running status.

[0129] In one embodiment, the detection dimensions in the multi-dimensional detection submodule may include service process status, API call status, database connection status, resource usage, request response time, and call chain anomalies; the multi-dimensional detection submodule may also include:

[0130] The first detection unit is used to detect the process liveness status and restart frequency based on the detection data of the service process status, and obtain the first detection result.

[0131] The second detection unit is used to detect the request success rate, call frequency, and timeout count based on the detection data of the interface call situation, and obtain the second detection result.

[0132] The third detection unit is used to detect the connection success rate, number of connections, and number of connection timeouts based on the detection data of the database connection status, and obtain the third detection result.

[0133] The fourth detection unit is used to detect CPU utilization, memory usage, disk I / O, and network bandwidth based on the resource usage detection data, and obtain the fourth detection result.

[0134] The fifth detection unit is used to detect the average response time and 95th percentile response time based on the detection data of the request response time, and obtain the fifth detection result.

[0135] The sixth detection unit is used to detect the service call failure rate and call timeout ratio based on the detection data of call link anomalies, and obtain the sixth detection result.

[0136] In one embodiment, the fault analysis module 220 may include:

[0137] The model analysis submodule is used to input operational data into a preset fault analysis model, so that the fault analysis model can perform abnormal indicator trend analysis based on the operational data, and output the fault type and impact range as fault data according to the analysis results.

[0138] In one embodiment, the strategy adjustment module 240 may include:

[0139] The data correction submodule is used to input self-healing data, operational data, and fault data into the fault analysis model for fault identification and data correction, and to obtain the corrected fault data output by the fault analysis model.

[0140] The strategy matching submodule is used to generate self-healing tags for system microservices and, based on the corrected fault data, to match the self-healing strategy with the highest similarity to the self-healing tag from the dynamic knowledge base.

[0141] The self-healing tag is used to define the self-healing result of the self-healing strategy that has been executed by the system microservice.

[0142] In one embodiment, the service self-healing module 230 may include:

[0143] The knowledge base construction submodule is used to build a dynamic knowledge base based on historical fault data and the historical processing experience corresponding to the historical fault data; the historical processing experience includes self-healing strategies for multiple different fault scenarios.

[0144] The knowledge base optimization submodule is used to obtain the fault handling data of the system microservice whenever a fault is successfully handled, and to optimize the dynamic knowledge base based on the fault handling data.

[0145] In one embodiment, this application also provides a storage medium storing computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the back-end system fault self-healing method as described in any of the above embodiments.

[0146] In one embodiment, this application also provides a computer device storing computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the back-end system fault self-healing method as described in any of the above embodiments.

[0147] Indicatively, such as Figure 4 As shown, Figure 4 This is a schematic diagram of the internal structure of a computer device 300 provided in an embodiment of this application. The computer device 300 can be provided as a server. (Refer to...) Figure 4 The computer device 300 includes a processing component 302, which further includes one or more processors, and memory resources represented by memory 301 for storing instructions, such as application programs, that can be executed by the processing component 302. The application programs stored in memory 301 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 302 is configured to execute instructions to perform the back-end system fault self-healing method of any of the above embodiments.

[0148] The computer device 300 may also include a power supply component 303 configured to perform power management of the computer device 300, a wired or wireless network interface 304 configured to connect the computer device 300 to a network, and an input / output (I / O) interface 305. The computer device 300 may operate on an operating system stored in memory 301, such as Windows Server™, Mac OS X™, Unix™, Linux™, Free BSD™, or similar.

[0149] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0150] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0151] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.

[0152] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for back-end system failure self-healing, the method comprising: The method includes: Real-time collection of system microservice operation data, and multi-dimensional state analysis of the operation data based on multi-dimensional detection rules to obtain the operation status of the system microservices; When the operating state is abnormal, the operating data is analyzed using a preset fault analysis model to obtain fault data; The system obtains a self-healing strategy corresponding to the fault data from the dynamic knowledge base, and performs self-healing operations on the system microservices based on the fault data and the self-healing strategy. If the self-healing result fails, the self-healing data of the system microservice during the self-healing operation is obtained, and the self-healing strategy of the system microservice is adjusted based on the self-healing data. The new self-healing strategy is then used to re-execute the self-healing operation on the system microservice until the self-healing result meets the self-healing termination condition. The adjustment of the self-healing strategy for the system microservices based on the self-healing data includes: The self-healing data, the operational data, and the fault data are input into the fault analysis model for fault identification and data correction, resulting in the corrected fault data output by the fault analysis model. Generate self-healing tags for the system microservices, and based on the corrected fault data, match the self-healing strategy with the highest similarity to the self-healing tag from the dynamic knowledge base; The self-healing tag is used to define the self-healing result of the self-healing strategy that has been executed by the system microservice.

2. The back-end system failure self-healing method of claim 1, wherein, The real-time acquisition system microservice operation data includes: A lightweight probe program is used to collect real-time operational status data of system microservices; the lightweight probe program is deployed within the system microservices and at key nodes in the call chain. Distributed tracing technology is used to track the runtime status data to obtain inter-service call chain information and request-response data. The system microservice's runtime data is generated based on the runtime status data, the call chain information, and the request-response data.

3. The backend system fault self-healing method according to claim 1, characterized in that, The multi-dimensional state analysis of the running data based on multi-dimensional detection rules is used to obtain the running state of the system microservices, including: Extract the detection data corresponding to each detection dimension from the running data, and perform state detection on the detection data of each detection dimension based on the multi-dimensional detection rules to obtain the detection result of each detection dimension. The system's microservices are comprehensively analyzed based on the various detection results to obtain their operational status.

4. The back-end system failure self-healing method of claim 3, wherein, The detection dimensions include service process status, interface call status, database connection status, resource usage, request response time, and call chain anomalies. The state detection is performed on the detection data of each detection dimension based on the multi-dimensional detection rules to obtain the detection result of each detection dimension, including: Based on the detection data of the service process status, the process liveness status and restart frequency are detected to obtain the first detection result; Based on the detection data of the interface call status, the request success rate, call frequency, and timeout count are detected to obtain a second detection result; Based on the detection data of the database connection status, the connection success rate, the number of connections, and the number of connection timeouts are detected to obtain a third detection result; Based on the detection data of the resource usage, CPU utilization, memory usage, disk I / O and network bandwidth are detected to obtain the fourth detection result; The average response time and 95th percentile response time are detected based on the detection data of the request response time, and the fifth detection result is obtained; Based on the detection data of the call chain anomalies, the service call failure rate and call timeout ratio are detected, and the sixth detection result is obtained.

5. The back-end system failure self-healing method of claim 1, wherein, The step of using a preset fault analysis model to perform fault analysis on the operating data to obtain fault data includes: The operational data is input into a preset fault analysis model so that the fault analysis model performs abnormal indicator trend analysis based on the operational data and outputs the fault type and impact range as fault data according to the analysis results.

6. The back-end system failure self-healing method of claim 1, wherein, The management process of the dynamic knowledge base includes: A dynamic knowledge base is constructed based on historical fault data and the historical processing experience corresponding to the historical fault data; the historical processing experience includes self-healing strategies for multiple different fault scenarios. Whenever a system microservice failure is successfully handled, the failure handling data of the system microservice is obtained, and the dynamic knowledge base is optimized based on the failure handling data.

7. A back-end system failure self-healing apparatus, characterized by, include: The status analysis module is used to collect the running data of the system microservices in real time, and perform multi-dimensional status analysis on the running data based on multi-dimensional detection rules to obtain the running status of the system microservices. The fault analysis module is used to perform fault analysis on the operating data using a preset fault analysis model when the operating state is abnormal, and to obtain fault data. The service self-healing module is used to match the self-healing strategy corresponding to the fault data from the dynamic knowledge base, and to perform self-healing operations on the system microservices based on the fault data and the self-healing strategy. The strategy adjustment module is used to obtain the self-healing data of the system microservice during the self-healing operation if the self-healing result fails, adjust the self-healing strategy of the system microservice based on the self-healing data, and re-execute the self-healing operation of the system microservice using the new self-healing strategy until the self-healing result meets the self-healing termination condition. The adjustment of the self-healing strategy for the system microservices based on the self-healing data includes: The self-healing data, the operational data, and the fault data are input into the fault analysis model for fault identification and data correction, resulting in the corrected fault data output by the fault analysis model. Generate self-healing tags for the system microservices, and based on the corrected fault data, match the self-healing strategy with the highest similarity to the self-healing tag from the dynamic knowledge base; The self-healing tag is used to define the self-healing result of the self-healing strategy that has been executed by the system microservice.

8. A storage medium characterized by: The storage medium stores computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the back-end system fault self-healing method as described in any one of claims 1 to 6.

9. A computer device, comprising: include: One or more processors, and memory; The memory has stored therein computer readable instructions that, when executed by the one or more processors, perform the steps of the method for backend system failure self-healing of any one of claims 1 to 6.

Images