Cloud computer end-to-end fault root cause analysis method and system

Abstract

The invention discloses a cloud computer end-to-end fault root cause analysis method and system, and relates to the technical field of operation and maintenance data analysis, and the method comprises the steps: deploying data collectors on a virtual resource pool, a multi-stage cloud management platform and a cloud computer terminal, and monitoring and collecting various operation and maintenance data in real time; a unified operation and maintenance management center is introduced to complete the following work: providing management and configuration services of the data collector; receiving and summarizing operation and maintenance data; various AI models are accessed, operation and maintenance data from different sources are distributed based on the types of the AI models, the AI models perform deep analysis and diagnosis on the operation and maintenance data, and potential fault modes or abnormal behaviors are identified; customizing an alarm strategy, an alarm processing flow and an alarm notification mode; and introducing an AIOps service to generate an alarm and fault analysis report according to the analysis and diagnosis results of the AI model. According to the method, the service quality of the end-to-end full link of the cloud computer is monitored in real time, so that faults can be quickly positioned, early warning can be performed in advance, and the operation and maintenance cost and the fault risk are reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of operation and maintenance data analysis, and in particular to a method and system for analyzing root causes of end-to-end faults in cloud computers. Background Art

[0002] Cloud computing, also known as virtual machines or cloud servers in cloud computing, refers to computing resources provided through the Internet. These resources can include processors, memory, storage, etc., which users can use on demand without owning physical hardware. With the development of the cloud computing industry, various business platforms and tools have been continuously introduced, making the cloud computing business environment increasingly complex, making monitoring and analysis difficult, and resulting in operational pain points such as complex environment and difficult monitoring and analysis. When a business failure occurs, it is often impossible to locate the cause of the failure in a short period of time.

[0003] There are some root cause analysis methods on the market, but none of them can meet this demand and have the following defects:

[0004] 1. The root cause analysis method based on rule engine and policy customization has weak analytical capabilities and cannot adapt to such a complex business environment.

[0005] 2. Existing AIOps-based analysis methods have failed to adapt to the cloud computer business environment and can only analyze business platforms or cloud computer terminals, but cannot achieve end-to-end root cause analysis. Summary of the invention

[0006] In view of the needs and shortcomings of current technological development, the present invention provides a cloud computer end-to-end fault root cause analysis method and system to quickly locate faults and provide early warnings, thereby effectively reducing operation and maintenance costs and fault risks.

[0007] In the first aspect, the present invention provides a cloud computer end-to-end fault root cause analysis method, and the technical solution adopted to solve the above technical problems is as follows:

[0008] A cloud computer end-to-end fault root cause analysis method comprises the following steps:

[0009] S1. Deploy data collectors on the virtualized resource pool, multi-level cloud management platform, and cloud computer terminals. The data collectors monitor and collect various operation and maintenance data in real time.

[0010] S2. Introduce a unified operation and maintenance management center for operation and maintenance administrators to complete the following tasks: provide management and configuration services for data collectors; receive and summarize operation and maintenance data collected by data collectors; access various AI models, allocate operation and maintenance data from different sources based on the type of AI model, and use AI models to conduct in-depth analysis and diagnosis of operation and maintenance data to identify potential failure modes or abnormal behaviors; customize alarm strategies, alarm processing procedures, and alarm notification methods;

[0011] S3. Introduce AIOps services. AIOps services generate alarms and fault analysis reports based on the analysis and diagnosis results of the AI ​​model, send alarm notifications to the operation and maintenance team, quickly locate the root cause of the fault, and send it to the operation and maintenance team for review through the unified operation and maintenance management center.

[0012] Optionally, the operation and maintenance data collected in step S1 includes system performance indicators, log information, network traffic and user operation records.

[0013] Optionally, the AI ​​models accessed by the unified operation and maintenance management center include the following four: time series analysis model LSTM or ARIMA model, natural language processing technology and large language model LLM, graph neural network or clustering algorithm, causal reasoning model or Bayesian network, among which:

[0014] Use time series analysis models such as LSTM or ARIMA to capture the evolution of performance indicators over time, identify abnormal fluctuations or trend changes, and thus predict potential system failures;

[0015] Use natural language processing technology to parse log text and extract key information and patterns. Use large language models (LLMs) to understand complex log content and provide analysis and explanation of new faults by learning historical fault information and diagnostic records.

[0016] Use graph neural networks to analyze dependencies between system components and quickly locate fault propagation paths; or use clustering algorithms to detect abnormal patterns in network traffic;

[0017] Use causal inference models or Bayesian networks to analyze the causal relationship between variables, infer the direct and indirect causes of the failure, and help operation and maintenance personnel trace back to the deep-seated causes of the problem.

[0018] Optionally, the AI ​​model involved conducts in-depth analysis and diagnosis of the operation and maintenance data to identify potential failure modes or abnormal behaviors. The specific process includes:

[0019] Data preprocessing: Clean, convert and standardize the collected operation and maintenance data to ensure data quality and consistency;

[0020] Feature extraction: extract useful features from preprocessed data;

[0021] Model training and optimization: Use annotated historical operation and maintenance data to train the AI ​​model and optimize model parameters through cross-validation methods;

[0022] Real-time monitoring and analysis: Deploy the trained AI model to the real-time monitoring system to perform real-time analysis and diagnosis on the newly collected operation and maintenance data.

[0023] Optionally, the execution step S3 involves the AIOps service generating an alarm before a fault occurs based on the analysis and diagnosis results of the AI ​​model. When a fault occurs, the service analyzes and finds the root cause of the fault and generates a fault analysis report.

[0024] In a second aspect, the present invention provides a cloud computer end-to-end fault root cause analysis system, and the technical solution adopted to solve the above technical problems is as follows:

[0025] A cloud computer end-to-end fault root cause analysis system, comprising:

[0026] The data collector module is deployed in the virtualized resource pool, multi-level cloud management platform, and cloud computer terminal, responsible for real-time monitoring and collection of various operation and maintenance data;

[0027] The unified operation and maintenance management center is for operation and maintenance administrators and is responsible for completing the following tasks: providing management and configuration services for data collector modules; receiving and summarizing operation and maintenance data collected by data collectors; accessing various AI models, allocating operation and maintenance data from different sources based on the type of AI model, and the AI ​​model conducts in-depth analysis and diagnosis of operation and maintenance data to identify potential failure modes or abnormal behaviors; customizing alarm strategies, alarm processing procedures, and alarm notification methods;

[0028] The AIOps service is responsible for generating alarms and fault analysis reports based on the analysis and diagnosis results of the AI ​​model, sending alarm notifications to the operation and maintenance team, quickly locating the root cause of the fault, and sending it to the operation and maintenance team for review through the unified operation and maintenance management center.

[0029] Optionally, the operation and maintenance data collected by the data collector module involved include system performance indicators, log information, network traffic and user operation records.

[0030] Optionally, the AI ​​models accessed by the unified operation and maintenance management center include the following four: time series analysis model LSTM or ARIMA model, natural language processing technology and large language model LLM, graph neural network or clustering algorithm, causal reasoning model or Bayesian network, among which:

[0031] Use time series analysis models such as LSTM or ARIMA to capture the evolution of performance indicators over time, identify abnormal fluctuations or trend changes, and thus predict potential system failures;

[0032] Use natural language processing technology to parse log text and extract key information and patterns. Use large language models (LLMs) to understand complex log content and provide analysis and explanation of new faults by learning historical fault information and diagnostic records.

[0033] Use graph neural networks to analyze dependencies between system components and quickly locate fault propagation paths; or use clustering algorithms to detect abnormal patterns in network traffic;

[0034] Use causal inference models or Bayesian networks to analyze the causal relationship between variables, infer the direct and indirect causes of the failure, and help operation and maintenance personnel trace back to the deep-seated causes of the problem.

[0035] Optionally, the AI ​​model involved conducts in-depth analysis and diagnosis of the operation and maintenance data to identify potential failure modes or abnormal behaviors. The specific process includes:

[0036] Data preprocessing: Clean, convert and standardize the collected operation and maintenance data to ensure data quality and consistency;

[0037] Feature extraction: extract useful features from preprocessed data;

[0038] Model training and optimization: Use annotated historical operation and maintenance data to train the AI ​​model and optimize model parameters through cross-validation methods;

[0039] Real-time monitoring and analysis: Deploy the trained AI model to the real-time monitoring system to perform real-time analysis and diagnosis on the newly collected operation and maintenance data.

[0040] Optionally, the AIOps service involved generates an alarm before a fault occurs based on the analysis and diagnosis results of the AI ​​model. When a fault occurs, it analyzes and finds the root cause of the fault and generates a fault analysis report.

[0041] Compared with the prior art, the cloud computer end-to-end fault root cause analysis method and system of the present invention have the following beneficial effects:

[0042] 1. The present invention can quickly locate faults and issue early warnings by real-time monitoring of the service quality of the entire end-to-end link of cloud computers, thereby improving the efficiency and accuracy of root cause analysis of end-to-end faults of cloud computers and reducing operation and maintenance costs and fault risks;

[0043] 2. The present invention can monitor and collect various operation and maintenance data in real time by deploying data collectors on virtualized resource pools, multi-level cloud management platforms, and cloud computer terminals, providing a rich source of information for subsequent analysis and diagnosis;

[0044] 3. The present invention introduces a unified operation and maintenance management center for operation and maintenance administrators, realizing centralized management and configuration services for data collectors. At the same time, the unified operation and maintenance management center is also responsible for receiving and summarizing operation and maintenance data from various data collectors, so that the operation and maintenance team can view and analyze all relevant data on a unified platform, thereby improving work efficiency and data utilization. By accessing various AI models and allocating operation and maintenance data from different sources based on the type of AI model, in-depth analysis and diagnosis of operation and maintenance data are realized. The AI ​​model can automatically identify potential fault modes or abnormal behaviors, greatly reducing the time and cost of manual troubleshooting and improving the accuracy and timeliness of fault discovery. The operation and maintenance team can customize alarm strategies, alarm processing procedures and alarm notification methods to ensure timely response when an abnormality occurs. This customized alarm mechanism can meet the needs of different business scenarios and improve the effectiveness and pertinence of alarms.

[0045] 4. After the introduction of AIOps service, the present invention can quickly generate alarm and fault analysis reports based on the analysis and diagnosis results of the AI ​​model, and send alarm notifications to the operation and maintenance team, which enables the operation and maintenance team to quickly locate the root cause of the fault and take corresponding measures to repair it. At the same time, the generation of reports also helps the operation and maintenance team to review and summarize the fault and improve future operation and maintenance efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Attached Figure 1 is a flow chart of a method according to Embodiment 1 of the present invention;

[0047] Attached Figure 2 This is a module connection block diagram of the second embodiment of the present invention. DETAILED DESCRIPTION

[0048] In order to make the technical solution, the technical problem solved and the technical effect of the present invention more clearly understood, the technical solution of the present invention is clearly and completely described below in conjunction with specific embodiments.

[0049] Embodiment 1:

[0050] Combined with Figure 1 This embodiment proposes a cloud computer end-to-end fault root cause analysis method, which includes the following steps:

[0051] S1. Data collectors are deployed on the virtualized resource pool, multi-level cloud management platform and cloud computer terminals. The data collectors monitor and collect various operation and maintenance data in real time, including system performance indicators, log information, network traffic and user operation records.

[0052] S2. Introduce a unified operation and maintenance management center for operation and maintenance administrators to complete the following tasks:

[0053] (a) Provide management and configuration services for data collectors, specifically: provide a unified interface and tools for managing and configuring various types of data collectors; support registration, deregistration, update and status monitoring of data collectors; allow administrators to define parameters such as collection frequency, data format, transmission protocol, etc. to ensure the efficiency and accuracy of data collection.

[0054] (b) Receive and summarize the operation and maintenance data collected by the data collectors, which is specifically reflected in: establishing a reliable data transmission mechanism to ensure that the data collected from each data collector can be safely and completely transmitted to the unified operation and maintenance management center; cleaning, organizing and storing the collected data to form a structured data set; providing data query, statistics and analysis functions to help administrators fully understand the system's operating status and performance indicators.

[0055] (c) Connect to various AI models, assign operation and maintenance data from different sources based on the type of AI model, and use AI models to conduct in-depth analysis and diagnosis of operation and maintenance data to identify potential failure modes or abnormal behaviors. For example, use LSTM or ARIMA models for time series analysis of system performance indicators, natural language processing technology and large language model LLM for log information, graph neural network or clustering algorithm for network traffic, and causal reasoning model or Bayesian network for user operation records, among which: ① Use LSTM or ARIMA models for time series analysis to capture the evolution of performance indicators over time, identify abnormal fluctuations or trend changes, and thus predict potential system failures; ② Use natural language processing technology to parse log texts and extract key information and patterns; Use large language model LLM to understand complex log content, and provide analysis and explanation of new failures by learning historical failure information and diagnostic records; ③ Use graph neural networks to analyze the dependencies between system components and quickly locate the failure propagation path; or use clustering algorithms to detect abnormal patterns in network traffic; ④ Use causal reasoning models or Bayesian networks to analyze the causal relationship between variables, infer the direct and indirect causes of the failure, and help operation and maintenance personnel trace back to the deep-seated causes of the problem.

[0056] It should be added that the AI ​​model conducts in-depth analysis and diagnosis of operation and maintenance data to identify potential failure modes or abnormal behaviors. The specific process includes:

[0057] Data preprocessing: Clean, convert and standardize the collected operation and maintenance data to ensure data quality and consistency;

[0058] Feature extraction: extract useful features from preprocessed data;

[0059] Model training and optimization: Use annotated historical operation and maintenance data to train the AI ​​model and optimize model parameters through cross-validation methods;

[0060] Real-time monitoring and analysis: Deploy the trained AI model to the real-time monitoring system to perform real-time analysis and diagnosis on the newly collected operation and maintenance data.

[0061] (d) Customize alarm strategies, alarm handling procedures, and alarm notification methods. Specifically, customize alarm strategies to allow administrators to set alarm rules and thresholds based on business needs and system characteristics. Customize alarm handling procedures, including alarm confirmation, location, repair, and feedback, to ensure that alarms can be handled promptly and effectively. Customize alarm notification methods, such as email, SMS, WeChat, etc., to ensure that relevant personnel can receive alarm information in a timely manner and take appropriate measures.

[0062] S3. Introduce AIOps services. Based on the analysis and diagnosis results of the AI ​​model, the AIOps service generates an alarm before a fault occurs. When a fault occurs, it analyzes and finds the root cause of the fault, generates a fault analysis report, sends an alarm notification to the operation and maintenance team, quickly locates the root cause of the fault, and sends it to the operation and maintenance team for review through the unified operation and maintenance management center. The report content includes the time of the fault, the scope of impact, the possible root cause, the recommended solution, and related operation and maintenance data and charts.

[0063] Embodiment 2:

[0064] Combined with Figure 2 , this embodiment provides a cloud computer end-to-end fault root cause analysis system, which includes:

[0065] The data collector module is deployed in the virtualized resource pool, multi-level cloud management platform, and cloud computer terminal, responsible for real-time monitoring and collection of various operation and maintenance data;

[0066] The unified operation and maintenance management center is for operation and maintenance administrators and is responsible for completing the following tasks: providing management and configuration services for data collector modules; receiving and summarizing operation and maintenance data collected by data collectors; accessing various AI models, allocating operation and maintenance data from different sources based on the type of AI model, and the AI ​​model conducts in-depth analysis and diagnosis of operation and maintenance data to identify potential failure modes or abnormal behaviors; customizing alarm strategies, alarm processing procedures, and alarm notification methods;

[0067] The AIOps service is responsible for generating alarms and fault analysis reports based on the analysis and diagnosis results of the AI ​​model, sending alarm notifications to the operation and maintenance team, quickly locating the root cause of the fault, and sending it to the operation and maintenance team for review through the unified operation and maintenance management center.

[0068] In this embodiment, the operation and maintenance data collected by the data collector module includes system performance indicators, log information, network traffic, and user operation records. Correspondingly, the AI ​​models accessed by the unified operation and maintenance management center include the following four: time series analysis model LSTM or ARIMA model, natural language processing technology and large language model LLM, graph neural network or clustering algorithm, causal reasoning model or Bayesian network, among which:

[0069] Use time series analysis models such as LSTM or ARIMA to capture the evolution of performance indicators over time, identify abnormal fluctuations or trend changes, and thus predict potential system failures;

[0070] Use natural language processing technology to parse log text and extract key information and patterns. Use large language models (LLMs) to understand complex log content and provide analysis and explanation of new faults by learning historical fault information and diagnostic records.

[0071] Use graph neural networks to analyze dependencies between system components and quickly locate fault propagation paths; or use clustering algorithms to detect abnormal patterns in network traffic;

[0072] Use causal inference models or Bayesian networks to analyze the causal relationship between variables, infer the direct and indirect causes of the failure, and help operation and maintenance personnel trace back to the deep-seated causes of the problem.

[0073] In this embodiment, the AI ​​model performs in-depth analysis and diagnosis on the operation and maintenance data to identify potential failure modes or abnormal behaviors. The specific process includes:

[0074] Data preprocessing: Clean, convert and standardize the collected operation and maintenance data to ensure data quality and consistency;

[0075] Feature extraction: extract useful features from preprocessed data;

[0076] Model training and optimization: Use annotated historical operation and maintenance data to train the AI ​​model and optimize model parameters through cross-validation methods;

[0077] Real-time monitoring and analysis: Deploy the trained AI model to the real-time monitoring system to perform real-time analysis and diagnosis on the newly collected operation and maintenance data.

[0078] In this embodiment, the AIOps service generates an alarm based on the analysis and diagnosis results of the AI ​​model before a fault occurs. When a fault occurs, it analyzes and finds the root cause of the fault and generates a fault analysis report. The report includes the time of the fault, the scope of impact, the possible root cause, the recommended solution, and related operation and maintenance data and charts.

[0079] In summary, the cloud computer end-to-end fault root cause analysis method and system of the present invention can quickly locate faults and provide early warnings by real-time monitoring of the service quality of the entire end-to-end link of the cloud computer, thereby improving the efficiency and accuracy of the cloud computer end-to-end fault root cause analysis and reducing operation and maintenance costs and fault risks.

[0080] The above specific examples are used to explain the principles and implementation methods of the present invention in detail. These examples are only used to help understand the core technical content of the present invention. Based on the above specific embodiments of the present invention, any improvements and modifications made by technicians in this technical field without departing from the principles of the present invention should fall within the scope of patent protection of the present invention.

Claims

1. A cloud computer end-to-end fault root cause analysis method, characterized in that: The steps include: S1. Deploy data collectors on the virtualized resource pool, multi-level cloud management platform, and cloud computer terminals. The data collectors monitor and collect various operation and maintenance data in real time. S2. Introduce a unified operation and maintenance management center for operation and maintenance administrators to complete the following tasks: provide management and configuration services for data collectors; receive and summarize operation and maintenance data collected by data collectors; access various AI models, allocate operation and maintenance data from different sources based on the type of AI model, and use AI models to conduct in-depth analysis and diagnosis of operation and maintenance data to identify potential failure modes or abnormal behaviors; customize alarm strategies, alarm processing procedures, and alarm notification methods; S3. Introduce AIOps services. AIOps services generate alarms and fault analysis reports based on the analysis and diagnosis results of the AI ​​model, send alarm notifications to the operation and maintenance team, quickly locate the root cause of the fault, and send it to the operation and maintenance team for review through the unified operation and maintenance management center.

2. A cloud computer end-to-end fault root cause analysis method according to claim 1, characterized in that: The operation and maintenance data collected in step S1 includes system performance indicators, log information, network traffic and user operation records.

3. A cloud computer end-to-end fault root cause analysis method according to claim 2, characterized in that: The AI ​​models accessed by the unified operation and maintenance management center include the following four: time series analysis model LSTM or ARIMA model, natural language processing technology and large language model LLM, graph neural network or clustering algorithm, causal reasoning model or Bayesian network, among which: Use time series analysis models such as LSTM or ARIMA to capture the evolution of performance indicators over time, identify abnormal fluctuations or trend changes, and thus predict potential system failures; Use natural language processing technology to parse log text and extract key information and patterns. Use large language models (LLMs) to understand complex log content and provide analysis and explanation of new faults by learning historical fault information and diagnostic records. Use graph neural networks to analyze dependencies between system components and quickly locate fault propagation paths; or use clustering algorithms to detect abnormal patterns in network traffic; Use causal inference models or Bayesian networks to analyze the causal relationship between variables, infer the direct and indirect causes of the failure, and help operation and maintenance personnel trace back to the deep-seated causes of the problem.

4. A cloud computer end-to-end fault root cause analysis method according to claim 3, characterized in that: The AI ​​model performs in-depth analysis and diagnosis of operation and maintenance data to identify potential failure modes or abnormal behaviors. The specific process includes: Data preprocessing: Clean, convert and standardize the collected operation and maintenance data to ensure data quality and consistency; Feature extraction: extract useful features from preprocessed data; Model training and optimization: Use annotated historical operation and maintenance data to train the AI ​​model and optimize model parameters through cross-validation methods; Real-time monitoring and analysis: Deploy the trained AI model to the real-time monitoring system to perform real-time analysis and diagnosis on the newly collected operation and maintenance data.

5. The method for analyzing root cause of end-to-end failures of cloud computers according to claim 1, characterized in that: Execute step S3. The AIOps service generates an alarm before a fault occurs based on the analysis and diagnosis results of the AI ​​model. When a fault occurs, it analyzes and finds the root cause of the fault and generates a fault analysis report.

6. A cloud computer end-to-end fault root cause analysis system, characterized in that: It includes: The data collector module is deployed in the virtualized resource pool, multi-level cloud management platform, and cloud computer terminal, responsible for real-time monitoring and collection of various operation and maintenance data; ④ The unified operation and maintenance management center is for operation and maintenance administrators and is responsible for completing the following tasks: providing management and configuration services for data collector modules; receiving and summarizing operation and maintenance data collected by data collectors; accessing various AI models, allocating operation and maintenance data from different sources based on the type of AI model, and the AI ​​model conducts in-depth analysis and diagnosis of operation and maintenance data to identify potential failure modes or abnormal behaviors; customizing alarm strategies, alarm processing procedures, and alarm notification methods; The AIOps service is responsible for generating alarms and fault analysis reports based on the analysis and diagnosis results of the AI ​​model, sending alarm notifications to the operation and maintenance team, quickly locating the root cause of the fault, and sending it to the operation and maintenance team for review through the unified operation and maintenance management center.

7. The cloud computer end-to-end fault root cause analysis system according to claim 6, characterized in that: The operation and maintenance data collected by the data collector module include system performance indicators, log information, network traffic and user operation records.

8. The cloud computer end-to-end fault root cause analysis system according to claim 7, characterized in that: The AI ​​models accessed by the unified operation and maintenance management center include the following four: time series analysis model LSTM or ARIMA model, natural language processing technology and large language model LLM, graph neural network or clustering algorithm, causal reasoning model or Bayesian network, among which: Use time series analysis models such as LSTM or ARIMA to capture the evolution of performance indicators over time, identify abnormal fluctuations or trend changes, and thus predict potential system failures; Use natural language processing technology to parse log text and extract key information and patterns. Use large language models (LLMs) to understand complex log content and provide analysis and explanation of new faults by learning historical fault information and diagnostic records. Use graph neural networks to analyze dependencies between system components and quickly locate fault propagation paths; or use clustering algorithms to detect abnormal patterns in network traffic; Use causal inference models or Bayesian networks to analyze the causal relationship between variables, infer the direct and indirect causes of the failure, and help operation and maintenance personnel trace back to the deep-seated causes of the problem.

9. The cloud computer end-to-end fault root cause analysis system according to claim 8, characterized in that: The AI ​​model performs in-depth analysis and diagnosis of operation and maintenance data to identify potential failure modes or abnormal behaviors. The specific process includes: Data preprocessing: Clean, convert and standardize the collected operation and maintenance data to ensure data quality and consistency; Feature extraction: extract useful features from preprocessed data; Model training and optimization: Use annotated historical operation and maintenance data to train the AI ​​model and optimize model parameters through cross-validation methods; Real-time monitoring and analysis: Deploy the trained AI model to the real-time monitoring system to perform real-time analysis and diagnosis on the newly collected operation and maintenance data.

10. The cloud computer end-to-end fault root cause analysis system according to claim 6, characterized in that: Based on the analysis and diagnosis results of the AI ​​model, the AIOps service generates an alarm before a fault occurs. When a fault occurs, it analyzes and finds the root cause of the fault and generates a fault analysis report.

Images