Intelligent fault detection system for Elasticsearch cluster

Abstract

The invention provides an Elasticsearch cluster fault intelligent detection system, which relates to the technical field of big data storage and retrieval, and comprises a data acquisition module, an Elasticsearch cluster fault analysis module, an expert system module and a model optimization and self-learning module, the current log data and the monitoring index data of the Elasticsearch cluster are analyzed through a fault detection model of the Elasticsearch cluster fault analysis module, the prediction fault type of the Elasticsearch cluster is determined, and automatic detection and prediction are achieved; a deep learning algorithm and an expert rule base are fused, so that complex and dynamic fault scenes can be efficiently processed, the false report rate and the missing report rate are reduced, and the detection accuracy is improved; and through the model optimization and self-learning module, self-learning and dynamic optimization of the fault detection model are realized, and the operation and maintenance complexity and cost are reduced.

Description

Technical Field

[0001] The present invention relates to the field of big data storage and retrieval technology, and in particular to an Elasticsearch cluster fault intelligent detection system. Background Art

[0002] Elasticsearch is a distributed, real-time, high-performance search and analytics engine that can process massive amounts of data and provide fast, accurate search results. In real-world applications, Elasticsearch clusters may experience various failures, such as node downtime, disk fullness, and network outages. Therefore, detecting Elasticsearch cluster failures is crucial.

[0003] In existing technologies, Elasticsearch cluster fault detection mainly relies on traditional monitoring tools and manual troubleshooting, which usually includes the following aspects: (1) Performance monitoring and log analysis: Evaluate the health of the Elasticsearch cluster by monitoring various performance indicators (such as central processing unit (CPU) usage, memory usage, query latency, etc.). Use log analysis tools to analyze Elasticsearch logs and detect abnormal behavior or error information. (2) Application of visualization tools: Use visualization tools such as Kibana and Grafana to display various performance indicators of the cluster, and deploy independent log analysis tools (such as Logstash) to process and analyze log data; (3) Real-time monitoring and historical records: Real-time monitoring of the running status of the Elasticsearch cluster allows for timely detection and resolution of cluster anomalies and failures. It also provides a historical record of cluster operation to assist in troubleshooting and performance optimization.

[0004] However, existing technologies rely on static monitoring and manual experience analysis, which cannot efficiently handle complex and dynamic failure scenarios, make it difficult to predict and prevent potential failures, are inefficient, and have high operation and maintenance complexity and costs. Summary of the Invention

[0005] In response to the problems existing in the prior art, an embodiment of the present invention provides an Elasticsearch cluster fault intelligent detection system.

[0006] The present invention provides an Elasticsearch cluster fault intelligent detection system, comprising a data acquisition module, an Elasticsearch cluster fault analysis module, an expert system module, and a model optimization and self-learning module; wherein: The data collection module is used to collect the current log data and monitoring indicator data of the Elasticsearch cluster; The Elasticsearch cluster fault analysis module is used to extract feature vectors from the current log data and monitoring indicator data using a natural language processing (NLP) algorithm, input the feature vectors into a fault detection model, obtain the probability distribution of each fault type output by the fault detection model, and determine the predicted fault type of the Elasticsearch cluster; wherein the fault detection model is a large-scale pre-trained language model based on deep learning, and is trained based on historical log data and monitoring indicator data marked as normal, historical log data and monitoring indicator data marked as fault, and associated fault types included in a training data set; The expert system module is configured to verify the predicted fault type based on the current log data, monitoring indicator data, and an expert rule base, and determine the target fault type of the Elasticsearch cluster based on the verification result; wherein the expert rule base includes a plurality of rules, each rule including a fault type, Elasticsearch cluster fault symptoms, and a solution; The model optimization and self-learning module is used to perform model training on the fault detection model based on the current log data and monitoring indicator data, the predicted fault type and the target fault type.

[0007] Optionally, the expert system module is specifically used to: Matching and verifying the current log data, monitoring indicator data, and predicted fault type with the rules of the expert rule base to obtain a verification result; When the verification result is that the target rule exists in the expert rule base, and the fault type and Elasticsearch cluster fault symptom of the target rule match the predicted fault type and the current log data and monitoring indicator data, determining that the target fault type of the Elasticsearch cluster is the predicted fault type; When the verification result is that there is no target rule in the expert rule base that matches the current log data, monitoring indicator data and the predicted fault type, based on the Elasticsearch cluster fault symptoms corresponding to the current log data and monitoring indicator data in the expert rule base, the fault type associated with the Elasticsearch cluster fault symptom in the expert rule base is determined as the target fault type of the Elasticsearch cluster.

[0008] Optionally, the fault detection model includes an input layer, a convolutional layer, a pooling layer, a fully connected layer and a Softmax layer; Among them, the input layer receives the feature vector; the convolution layer extracts local features in the feature vector through the convolution kernel to identify each fault type; the pooling layer downsamples the local features extracted by the convolution layer to obtain key features; the fully connected layer maps the key features sampled by the pooling layer to each fault type to obtain a feature mapping result; the Softmax layer generates a probability distribution of each fault type based on the feature mapping result output by the fully connected layer.

[0009] Optionally, the data acquisition module is further used to: The historical log data and monitoring indicator data, various fault types and solutions are collected and sent to the model optimization and self-learning module.

[0010] Optionally, the model optimization and self-learning module is further used to: According to the Elasticsearch cluster failure symptoms corresponding to the historical log data and monitoring indicator data, the historical log data and monitoring indicator data are labeled as corresponding failure types, and the occurrence time, triggering conditions, operation steps, and impact of each historical log data and monitoring indicator data labeled as a failure, as well as the impact of the failure on the Elasticsearch cluster and business are recorded to generate the training data set.

[0011] Optionally, the model optimization and self-learning module is further used to: The fault detection model is regularly retrained and updated based on newly added historical log data and monitoring indicator data.

[0012] Optionally, the model optimization and self-learning module is further used to: The fault detection model is retrained according to the updated rules and newly added rules in the expert rule base.

[0013] Optionally, the model optimization and self-learning module is further used to: A cross entropy loss function is used to measure the difference between the predicted fault type and the target fault type, and parameters of the fault detection model are updated.

[0014] Optionally, the fault type includes at least one of the following: Index fragmentation, query latency, CPU overload, memory overflow, master node failure, data node failure, network partition failure.

[0015] Optionally, the system further comprises: The system monitoring and feedback module, together with the data acquisition module, the Elasticsearch cluster fault analysis module, the expert system module and the model optimization and self-learning module, is used to monitor the operation of the data acquisition module, the Elasticsearch cluster fault analysis module, the expert system module and the model optimization and self-learning module.

[0016] The present invention also provides an Elasticsearch cluster fault intelligent detection method, comprising: Collect current log data and monitoring indicator data of the Elasticsearch cluster; Extracting feature vectors of the current log data and monitoring indicator data using a natural language processing (NLP) algorithm, inputting the feature vectors into a fault detection model, obtaining a probability distribution of each fault type output by the fault detection model, and determining the predicted fault type of the Elasticsearch cluster; wherein the fault detection model is a large-scale pre-trained language model based on deep learning, and is trained based on historical log data and monitoring indicator data marked as normal, historical log data and monitoring indicator data marked as fault, and associated fault types included in a training data set; Verifying the predicted fault type based on the current log data, monitoring indicator data, and an expert rule base, and determining a target fault type of the Elasticsearch cluster based on the verification result; wherein the expert rule base includes a plurality of rules, each rule including a fault type, Elasticsearch cluster fault symptoms, and a solution; The fault detection model is trained based on the current log data and monitoring indicator data, the predicted fault type and the target fault type.

[0017] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the intelligent Elasticsearch cluster fault detection method as described above is implemented.

[0018] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the intelligent detection method for Elasticsearch cluster failures as described above is implemented.

[0019] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the above-mentioned Elasticsearch cluster fault intelligent detection method.

[0020] The Elasticsearch cluster fault intelligent detection system provided by the present invention includes a data acquisition module, an Elasticsearch cluster fault analysis module, an expert system module and a model optimization and self-learning module. The fault detection model in the Elasticsearch cluster fault analysis module performs in-depth analysis on the current log data and monitoring indicator data of the Elasticsearch cluster, determines the predicted fault type of the Elasticsearch cluster, and thus predicts potential fault risks, thereby realizing automatic detection and prediction of Elasticsearch cluster faults, reducing reliance on manual experience, greatly improving detection efficiency, and helping operation and maintenance personnel take measures in advance to avoid faults; and, by integrating deep learning algorithms and expert rule libraries, it can efficiently handle complex and dynamic fault scenarios, reduce false alarm and missed alarm rates, and improve detection accuracy; through the model optimization and self-learning modules, the fault detection model can achieve self-learning and dynamic optimization, that is, it has continuous learning and dynamic adaptability, so that the fault detection model can continuously maintain a high level of fault detection capability, reducing operation and maintenance complexity and cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0022] Figure 1 This is a schematic diagram of the framework of the Elasticsearch cluster fault intelligent detection system provided by the present invention; Figure 2 This is a schematic diagram of the module relationship of the Elasticsearch cluster fault intelligent detection system provided by the present invention; Figure 3 This is a flow chart of the intelligent Elasticsearch cluster fault detection method provided by the present invention; Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0023] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0024] In order to facilitate a clearer understanding of the various embodiments of the present application, some relevant knowledge is first introduced as follows.

[0025] Currently, fault detection in Elasticsearch clusters mainly relies on traditional monitoring tools and manual troubleshooting. These methods generally include the following aspects: (1) performance monitoring and log analysis; (2) application of visualization tools; and (3) real-time monitoring and historical records.

[0026] These methods can provide real-time performance monitoring and exception response. Combined with visualization tools, users can intuitively view and analyze cluster status. They have strong scalability and can process large-scale log and monitoring data.

[0027] However, these methods also have shortcomings: (1) Reliance on manual analysis, low efficiency: For complex fault scenarios, relying on manual experience for analysis and troubleshooting is inefficient and prone to misjudgment. (2) Unable to predict and prevent potential failures: Traditional monitoring methods focus on existing faults and lack intelligent and automated predictive capabilities, making it difficult to prevent potential fault risks in a timely manner. (3) High complexity of operation and maintenance: The need to configure and maintain multiple independent monitoring and analysis tools increases system complexity and operation and maintenance costs; (4) Processing capacity bottleneck: In large-scale clusters, performance monitoring and log analysis may face bottlenecks in data volume and processing capacity, making it difficult to respond and handle failures in a timely manner.

[0028] To address the above problems, the present invention proposes an Elasticsearch cluster fault intelligent detection system to achieve intelligent, automated and accurate fault detection of the Elasticsearch cluster.

[0029] The following combination Figure 1 The Elasticsearch cluster fault intelligent detection system provided by the present invention is described in detail. Figure 1This is a schematic diagram of the framework of the Elasticsearch cluster fault intelligent detection system provided by the present invention. Figure 1 As shown, the system includes: Data collection module 101, Elasticsearch cluster fault analysis module 102, expert system module 103 and model optimization and self-learning module 104; wherein: The data collection module 101 is used to collect current log data and monitoring indicator data of the Elasticsearch cluster.

[0030] In an embodiment of the present invention, the data collection module 101 is connected to the Elasticsearch cluster fault analysis module 102, and is responsible for collecting current log data and monitoring indicator data from the Elasticsearch cluster, and passing them to the Elasticsearch cluster fault analysis module 102 for subsequent in-depth analysis and fault detection.

[0031] In actual applications, in order to provide a reliable data basis for the Elasticsearch cluster fault analysis module 102, the data collection module 101 needs to collect and organize the monitoring indicator data and log data of the Elasticsearch cluster in real time.

[0032] Among them, monitoring indicator data includes functional indicators and performance indicators; performance indicators include CPU usage, memory usage, query latency and other data; log data includes error logs, warning logs and other data.

[0033] The Elasticsearch cluster fault analysis module 102 is used to use a natural language processing (NLP) algorithm to extract feature vectors of the current log data and monitoring indicator data, input the feature vectors into a fault detection model, obtain the probability distribution of each fault type output by the fault detection model, and determine the predicted fault type of the Elasticsearch cluster; wherein the fault detection model is a large-scale pre-trained language model based on deep learning, and the fault detection model is trained based on historical log data and monitoring indicator data marked as normal, historical log data and monitoring indicator data marked as faulty, and associated fault types included in a training data set.

[0034] In this embodiment of the present invention, the Elasticsearch cluster fault analysis module 102 is used to implement intelligent fault detection and prediction, reducing manual intervention. Based on the fault detection model, this module extracts features and conducts in-depth analysis on the monitoring indicator data and log data provided by the data acquisition module 101 to identify potential faults and anomalies.

[0035] In actual applications, the fault detection model is a customized large-scale pre-trained language model, such as the Tongyi Qianwen Qwen2 model. This model deeply integrates the data of the Elasticsearch cluster, including log data, monitoring indicator data, and historical fault cases for training and optimization. It adapts to the log structure, monitoring indicator data, and common fault modes of the Elasticsearch cluster to ensure the effectiveness and practicality of the model.

[0036] The Elasticsearch cluster fault analysis module 102 is connected to the expert system module 103 and is used to transmit the predicted fault type to the expert system module 103 for verification and optimization by the expert system module 103 .

[0037] Optionally, the fault type includes at least one of the following: Index fragmentation, query latency, CPU overload, memory overflow, master node failure, data node failure, network partition failure.

[0038] The expert system module 103 is used to verify the predicted fault type based on the current log data, monitoring indicator data and expert rule base, and determine the target fault type of the Elasticsearch cluster based on the verification result; wherein, the expert rule base includes multiple rules, each rule includes a fault type, Elasticsearch cluster fault symptoms and solutions.

[0039] In this embodiment of the present invention, an expert system module 103 is introduced to improve the accuracy and practicality of fault detection. Based on an expert rule base, expert system module 103 verifies and optimizes the predicted fault types output by the fault detection model, thereby determining the target fault type and improving reliability in complex fault scenarios. Each rule in the expert rule base defines a specific fault type, fault symptoms, and corresponding solution. Deploying the rule base in a local environment fully considers the specificities of actual application scenarios, ensuring its effectiveness and applicability.

[0040] This module integrates the operation and maintenance experience of domain experts, reviews the detection results of the Elasticsearch cluster fault analysis module 102, and provides optimization suggestions, thereby improving the accuracy of detection, especially playing an important role in complex or new fault scenarios.

[0041] It should be noted that the expert system module 103 can transmit the optimized feedback information to the model optimization and self-learning module 104 to update the fault detection model.

[0042] The model optimization and self-learning module 104 is used to perform model training on the fault detection model based on the current log data and monitoring indicator data, the predicted fault type and the target fault type.

[0043] In order to maintain sensitivity to the latest fault modes and high detection accuracy, the Elasticsearch cluster fault intelligent detection system of the present invention is provided with a model optimization and self-learning module 104 to achieve continuous optimization and self-learning of the fault detection model.

[0044] Specific implementation steps: (1) Continuous data collection and preprocessing During system operation, new log data and monitoring indicator data are continuously collected. The newly collected data is standardized and feature extracted to ensure that it meets the requirements of model training; (2) Incremental learning and parameter optimization Leverage new data to incrementally learn the fault detection model, learn new fault characteristics, and improve the ability to detect new faults. Dynamically adjust model parameters based on the performance of the fault detection model on new data to improve the model's generalization and robustness. (3) Integration of expert feedback Incorporate the feedback and new fault rules provided by the expert system module 103 into the model training data set. Through retraining, correct the model's misjudgment of specific fault types and improve detection accuracy; (4) Model update and deployment The optimized fault detection model is updated to the Elasticsearch cluster fault analysis module 102 to improve its detection capabilities. This forms a closed loop of data collection, model optimization, and model deployment, continuously improving system performance.

[0045] The optimization and self-learning module 104 will be described in detail in subsequent embodiments and will not be repeated here.

[0046] In this embodiment of the present invention, the model optimization and self-learning module 104 is capable of continuous learning and dynamic adaptation, maintaining sensitivity to new faults and improving the long-term effectiveness of the system. This module continuously collects new data and expert feedback to incrementally learn and dynamically optimize the fault detection model, thereby forming a closed loop. The various modules in the aforementioned Elasticsearch cluster fault intelligent detection system achieve close collaboration through data transmission and feedback mechanisms, ensuring the efficient execution of the Elasticsearch cluster's fault detection and optimization workflow.

[0047] The Elasticsearch cluster fault intelligent detection system provided by the present invention includes a data acquisition module, an Elasticsearch cluster fault analysis module, an expert system module and a model optimization and self-learning module. The fault detection model in the Elasticsearch cluster fault analysis module can perform in-depth analysis on the current log data and monitoring indicator data of the Elasticsearch cluster, determine the predicted fault type of the Elasticsearch cluster, and thus predict potential fault risks, thereby realizing automatic detection and prediction of Elasticsearch cluster faults, reducing reliance on manual experience, greatly improving detection efficiency, and helping operation and maintenance personnel take measures in advance to avoid faults; and, by integrating deep learning algorithms and expert rule libraries, it can efficiently handle complex and dynamic fault scenarios, reduce false alarm and missed alarm rates, and improve detection accuracy; through the model optimization and self-learning modules, the fault detection model can achieve self-learning and dynamic optimization, that is, it has continuous learning and dynamic adaptability, so that the fault detection model can continuously maintain a high level of fault detection capability, reducing operation and maintenance complexity and cost.

[0048] Optionally, the fault detection model includes an input layer, a convolutional layer, a pooling layer, a fully connected layer and a Softmax layer; Among them, the input layer receives the feature vector; the convolution layer extracts local features in the feature vector through the convolution kernel to identify each fault type; the pooling layer downsamples the local features extracted by the convolution layer to obtain key features; the fully connected layer maps the key features sampled by the pooling layer to each fault type to obtain a feature mapping result; the Softmax layer generates a probability distribution of each fault type based on the feature mapping result output by the fully connected layer.

[0049] In an embodiment of the present invention, the fault detection model uses an architecture that combines convolutional neural networks (CNN) with a fully connected network. In practical applications, the fault detection model generates a probability distribution for each fault type, which can be achieved through the following process: (1) The input layer receives feature vectors: Before the monitoring indicator data and log data of the Elasticsearch cluster are input into the input layer of the fault detection model, the monitoring indicator data and log data must be standardized and vectorized.

[0050] Specifically, natural language processing models (such as the BRET model) are used to convert log data into vector representations. These log data contain characteristic information about various faults, enabling the fault detection model to capture the fault patterns implicit in the log data.

[0051] Monitoring metric data (such as CPU usage, memory usage, and query latency) requires standardization and dimensionality reduction to extract key features. These monitoring metric data are closely related to fault occurrences, and fault detection models can identify abnormal performance behavior through analysis.

[0052] (2) The convolution layer extracts local features in the feature vector through the convolution kernel to identify failure modes and abnormal behaviors in the system log.

[0053] (3) The pooling layer downsamples the output of the convolutional layer to reduce the data dimension and retain key features.

[0054] (4) The fully connected layer further processes the key features and maps them to different failure types, such as master node failure, data node failure, network partition, etc.

[0055] (5) The Softmax layer converts the feature mapping results output by the fully connected layer into the probability distribution of various types of faults, generating the probability distribution of each fault type.

[0056] This deep learning and prediction mechanism enables the fault detection model to not only accurately identify known faults, but also have a certain ability to detect unknown faults.

[0057] In order to ensure the efficiency and accuracy of the fault detection model, it is necessary to continuously optimize it using the model optimization and self-learning module 104 during the model training and application process.

[0058] Specifically, the model optimization and self-learning module 104 is also used to: A cross entropy loss function is used to measure the difference between the predicted fault type and the target fault type, and parameters of the fault detection model are updated.

[0059] The specific process includes the following: 1) Use the cross entropy loss function to measure the difference between the predicted results and the actual results to guide the update of model parameters; 2) Combining the Adam optimizer and adaptive learning rate strategy to accelerate model convergence and avoid overfitting; 3) As new fault data is collected, the model is regularly retrained and updated to maintain sensitivity to the latest fault types.

[0060] Optionally, the expert system module 103 is specifically configured to: Matching and verifying the current log data, monitoring indicator data, and predicted fault type with the rules of the expert rule base to obtain a verification result; When the verification result is that the target rule exists in the expert rule base, and the fault type and Elasticsearch cluster fault symptom of the target rule match the predicted fault type and the current log data and monitoring indicator data, determining that the target fault type of the Elasticsearch cluster is the predicted fault type; When the verification result is that there is no target rule in the expert rule base that matches the current log data, monitoring indicator data and the predicted fault type, based on the Elasticsearch cluster fault symptoms corresponding to the current log data and monitoring indicator data in the expert rule base, the fault type associated with the Elasticsearch cluster fault symptom in the expert rule base is determined as the target fault type of the Elasticsearch cluster.

[0061] In this embodiment of the present invention, to improve the accuracy and practicality of fault detection, an expert system module 103 is introduced into the Elasticsearch cluster fault intelligent detection system. By systematizing the long-term operational and maintenance experience accumulated by domain experts, an independent expert rule base is established. This expert rule base and the Elasticsearch cluster fault analysis module 102 jointly participate in the fault detection decision-making process. The specific implementation process is as follows: (1) Construction and localized deployment of expert rule base: Summarize and systematize the operational and maintenance experience of domain experts to build a rule library covering common fault types. Each rule defines a specific fault type, symptom, and corresponding solution. Deploy the rule library in the local environment, taking into account the specificities of actual application scenarios to ensure its effectiveness and applicability.

[0062] (2) Real-time verification and optimization: After the Elasticsearch cluster fault analysis module 102 outputs the predicted fault type, the expert system module 103 calls the expert rule library in real time to verify the predicted fault type and confirm the fault type through rule matching. The expert rule library includes multiple rules, each of which includes the fault type, Elasticsearch cluster fault symptoms, and corresponding solutions.

[0063] Scenario 1: The predicted fault type output by the previous log data, monitoring indicator data, and fault detection model is matched and verified with the rules of the expert rule base. When the verification result is that the target rule exists in the expert rule base, and the fault type and Elasticsearch cluster fault symptoms included in the target rule match the predicted fault type and the previous log data and monitoring indicator data, the target fault type of the Elasticsearch cluster is determined to be the predicted fault type.

[0064] Scenario 2: When the verification result shows that there is no target rule in the expert rule base that matches the previous log data, monitoring indicator data, and predicted fault type, based on the Elasticsearch cluster fault symptoms corresponding to the previous log data and monitoring indicator data in the expert rule base, the fault type associated with the Elasticsearch cluster fault symptoms in the expert rule base is determined as the target fault type of the Elasticsearch cluster.

[0065] (3) Rule base maintenance and update During system operation, if new fault types and expert feedback are discovered, the rule base needs to be continuously updated and improved to maintain its timeliness and effectiveness.

[0066] Optionally, the model optimization and self-learning module 104 is further configured to: The fault detection model is retrained according to the updated rules and newly added rules in the expert rule base. Through the above measures, the fault detection model can maintain its sensitivity to the latest fault types.

[0067] Optionally, the data acquisition module 101 is further configured to: The historical log data and monitoring indicator data, various fault types and solutions are collected and sent to the model optimization and self-learning module 104.

[0068] In an embodiment of the present invention, the data acquisition module 101 sends historical log data and monitoring indicator data, various fault types and solutions to the model optimization and self-learning module 104, so that the high-quality training data set generated by the model optimization and self-learning module 104 is used to train the fault detection model, ensuring that the fault detection model can learn the complex correlation between different fault types, and then accurately identify and predict various types of complex faults.

[0069] In actual applications, the training dataset includes historical log data and monitoring indicator data of the Elasticsearch cluster, various fault types and solutions, and multiple sample rules in the expert rule library. Each sample rule includes the fault type, Elasticsearch cluster fault symptoms, and corresponding solutions.

[0070] Specifically, fault types include index fragmentation, query latency, CPU overload, memory overflow, master node failure, data node failure, network partitioning, etc. The training dataset not only includes the conditions, symptoms, and impacts of the faults, but also incorporates rich manual experience to ensure the diversity and representativeness of the training dataset.

[0071] During the data collection process, the monitoring indicator data (such as CPU usage, memory usage, query latency, etc.) and log data (such as WARN and ERROR log entries) collected by the data acquisition module 101 are standardized, feature extracted, and dimensionally reduced to provide accurate input features for the subsequent fault detection model.

[0072] Optionally, the model optimization and self-learning module 104 is used to collect historical log data and monitoring indicator data of the Elasticsearch cluster, various fault types and solutions, and perform data annotation on each historical log data and monitoring indicator data to obtain a training data set.

[0073] Labeling historical log data and monitoring indicator data can be achieved through the following methods: According to the Elasticsearch cluster failure symptoms corresponding to the historical log data and monitoring indicator data, the historical log data and monitoring indicator data are labeled as corresponding failure types, and the occurrence time, triggering conditions, operation steps, and impact of each historical log data and monitoring indicator data labeled as a failure, as well as the impact of the failure on the Elasticsearch cluster and business are recorded to generate the training data set.

[0074] Specifically, during the annotation process of historical log data and monitoring indicator data, we first label them according to the characteristics of the failure, assigning them to specific failure types, including index fragmentation, query latency, CPU overload, memory overflow, master node failure, data node failure, and network partition. We then record in detail the occurrence time, triggering conditions, operation steps, and the impact of each failure on the cluster and business, forming a labeling of the failure conditions and impact. To ensure the consistency and accuracy of the annotations, the initially labeled data will be reviewed and corrected by domain experts. For difficult or controversial samples, consensus is reached through expert discussion.

[0075] Optionally, the model optimization and self-learning module 104 is further configured to: The fault detection model is regularly retrained and updated based on newly added historical log data and monitoring indicator data.

[0076] In the Elasticsearch cluster fault intelligent detection system provided in the embodiment of the present invention, by closely integrating with the model optimization and self-learning module 104, the fault detection model maintains high sensitivity to new fault types, realizes intelligent and automated detection and prediction of Elasticsearch cluster faults, and thus improves the efficiency of fault detection and the ability to accurately detect faults.

[0077] Figure 2 This is a schematic diagram of the module relationship of the Elasticsearch cluster fault intelligent detection system provided by the present invention. Figure 1 Based on the data collection module 101, Elasticsearch cluster fault analysis module 102, expert system module 103 and model optimization and self-learning module 104 shown in the figure, Figure 2 It also includes: a system monitoring and feedback module 105; wherein: The system monitoring and feedback module 105 is connected to the data acquisition module 101, the Elasticsearch cluster fault analysis module 102, the expert system module 103 and the model optimization and self-learning module 104, and is used to monitor the operation of the data acquisition module 101, the Elasticsearch cluster fault analysis module 102, the expert system module 103 and the model optimization and self-learning module 104.

[0078] In this embodiment of the present invention, the system monitoring and feedback module 105 runs throughout the entire system, monitoring the operating status and performance of each module, coordinating data and information flows, and responsible for monitoring the overall system status. It also controls the behavior of other modules through feedback mechanisms to ensure stable system operation. Through the system monitoring and feedback module 105, each module achieves close collaboration, ensuring efficient fault detection and optimization workflows for the Elasticsearch cluster.

[0079] In summary, the present invention addresses the problems of existing fault detection technologies, such as reliance on manual analysis, inability to predict potential faults, and high operational complexity. This invention proposes an intelligent Elasticsearch cluster fault detection system based on dataset construction, localized intelligent models, and expert experience integration. The system has the following advantages: 1. Improve the accuracy and efficiency of fault detection (1) Intelligence and automation. Utilizing customized large language models, we conduct in-depth analysis of large amounts of log and monitoring indicator data to achieve automatic fault detection and prediction, reducing reliance on manual experience and significantly improving detection efficiency.

[0080] (2) Accurately identify complex faults. By integrating deep learning with expert experience, the model can accurately identify complex and dynamic fault scenarios, reduce false positives and false negatives, and improve detection accuracy.

[0081] 2. Capable of predicting and preventing faults (1) Early warning of potential failures. The model can analyze historical data and real-time indicators to predict potential failure risks, helping operation and maintenance personnel take measures in advance to avoid failures.

[0082] (2) Dynamically adapt to environmental changes. Through self-learning and dynamic optimization of the model, the system can adapt to changes in cluster configuration and load, and continuously maintain a high level of detection capabilities.

[0083] 3. Reduce operation and maintenance complexity and costs (1) Functional integration. The system integrates functions such as data collection, intelligent analysis, and expert systems, reducing dependence on multiple independent monitoring and analysis tools and reducing the complexity of operation and maintenance.

[0084] (2) Reduce manual intervention. The automated detection and prediction mechanism significantly reduces the workload of manual investigation, reduces operation and maintenance costs, and improves operation and maintenance efficiency.

[0085] 4. Improve system stability and scalability (1) Efficient processing of large-scale data. Using advanced deep learning algorithms, the system can efficiently process log and performance data from large-scale clusters, overcoming the bottlenecks of traditional methods in terms of data volume and processing power.

[0086] (2) Modular design. The system has a clear structure and clear connections between modules. It is easy to deploy, maintain and expand, and can adapt to the growth of business needs.

[0087] The following describes the Elasticsearch cluster fault intelligent detection method provided by the present invention. The Elasticsearch cluster fault intelligent detection method described below and the Elasticsearch cluster fault intelligent detection system described above can be referenced to each other. Figure 3 FIG. 1 is a flow chart of the Elasticsearch cluster fault intelligent detection method provided by the present invention, as shown in FIG. Figure 3 As shown, the Elasticsearch cluster fault intelligent detection method is executed by the Elasticsearch cluster fault intelligent detection system, which specifically includes steps 301-304, wherein: Step 301: Collect current log data and monitoring indicator data of the Elasticsearch cluster.

[0088] Step 302: Use the NLP algorithm to extract the feature vectors of the current log data and monitoring indicator data, input the feature vectors into the fault detection model, obtain the probability distribution of each fault type output by the fault detection model, and determine the predicted fault type of the Elasticsearch cluster; wherein the fault detection model is a large-scale pre-trained language model based on deep learning, and the fault detection model is trained based on the historical log data and monitoring indicator data marked as normal, the historical log data and monitoring indicator data marked as fault, and the associated fault types included in the training data set.

[0089] Step 303: Verify the predicted fault type based on the current log data, monitoring indicator data, and the expert rule base, and determine the target fault type of the Elasticsearch cluster based on the verification result; wherein the expert rule base includes multiple rules, each rule includes a fault type, Elasticsearch cluster fault symptoms, and a solution.

[0090] Step 304: Perform model training on the fault detection model based on the current log data and monitoring indicator data, the predicted fault type and the target fault type.

[0091] The intelligent Elasticsearch cluster fault detection method provided by the present invention reduces the complexity of operation and maintenance. At the same time, the customized fault detection model can perform in-depth analysis of the current log data and monitoring indicator data of the Elasticsearch cluster, determine the predicted fault type of the Elasticsearch cluster, and thus predict potential failure risks, thereby realizing automatic detection and prediction of Elasticsearch cluster failures, reducing reliance on manual experience, greatly improving detection efficiency, and helping operation and maintenance personnel take measures in advance to avoid failures. Moreover, by integrating deep learning algorithms and expert rule libraries, it can efficiently handle complex and dynamic fault scenarios, reduce false alarm and missed alarm rates, and improve detection accuracy. At the same time, the fault detection model can achieve self-learning and dynamic optimization, that is, it has continuous learning and dynamic adaptability, so that the fault detection model can continuously maintain a high level of fault detection capability, reducing the complexity and cost of operation and maintenance.

[0092] Figure 4 An example of a physical structure diagram of an electronic device is shown below. Figure 4As shown, the electronic device may include: a processor (processor) 410 , a communication interface (Communications Interface) 420 , a memory (memory) 430 and a communication bus 440 , wherein the processor 410 , the communication interface 420 , and the memory 430 communicate with each other via the communication bus 440 . The processor 410 can call the logic instructions in the memory 430 to execute the Elasticsearch cluster fault intelligent detection method, which includes: collecting current log data and monitoring indicator data of the Elasticsearch cluster; using the NLP algorithm to extract the feature vectors of the current log data and monitoring indicator data, inputting the feature vectors into the fault detection model, obtaining the probability distribution of each fault type output by the fault detection model, and determining the predicted fault type of the Elasticsearch cluster; wherein the fault detection model is a large-scale pre-trained language model based on deep learning, and the fault detection model is trained based on the historical log data and monitoring indicator data marked as normal, the historical log data and monitoring indicator data marked as faults, and the associated fault types included in the training data set; the predicted fault type is verified according to the current log data and monitoring indicator data and the expert rule base, and the target fault type of the Elasticsearch cluster is determined according to the verification result; wherein the expert rule base includes multiple rules, each rule includes a fault type, Elasticsearch cluster fault symptoms and solutions; the fault detection model is trained according to the current log data and monitoring indicator data, the predicted fault type and the target fault type.

[0093] Furthermore, the logic instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product, stored in a storage medium, includes instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0094] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the Elasticsearch cluster fault intelligent detection method provided by the above methods, which includes: collecting current log data and monitoring indicator data of the Elasticsearch cluster; using the NLP algorithm to extract feature vectors of the current log data and monitoring indicator data, inputting the feature vectors into a fault detection model, obtaining the probability distribution of each fault type output by the fault detection model, and determining the predicted fault type of the Elasticsearch cluster; wherein the fault detection model The model is a large-scale pre-trained language model based on deep learning. The fault detection model is trained based on historical log data and monitoring indicator data marked as normal, historical log data and monitoring indicator data marked as fault, and associated fault types included in the training data set; the predicted fault type is verified based on the current log data and monitoring indicator data and the expert rule base, and the target fault type of the Elasticsearch cluster is determined based on the verification result; wherein the expert rule base includes multiple rules, each rule includes a fault type, Elasticsearch cluster fault symptoms and solutions; the fault detection model is trained based on the current log data and monitoring indicator data, the predicted fault type and the target fault type.

[0095] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the Elasticsearch cluster intelligent fault detection method provided by the above-mentioned methods, the method comprising: collecting current log data and monitoring indicator data of the Elasticsearch cluster; using an NLP algorithm to extract feature vectors of the current log data and monitoring indicator data, inputting the feature vectors into a fault detection model, obtaining a probability distribution of each fault type output by the fault detection model, and determining the predicted fault type of the Elasticsearch cluster; wherein the fault detection model is a large-scale pre-trained language model based on deep learning, and the fault detection model is trained based on historical log data and monitoring indicator data marked as normal, historical log data and monitoring indicator data marked as fault, and associated fault types included in a training data set; the predicted fault type is verified based on the current log data and monitoring indicator data and an expert rule base, and the target fault type of the Elasticsearch cluster is determined based on the verification result; wherein the expert rule base includes multiple rules, each rule includes a fault type, Elasticsearch cluster fault symptoms and solutions; the fault detection model is trained based on the current log data and monitoring indicator data, the predicted fault type and the target fault type.

[0096] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0097] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0098] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. An Elasticsearch cluster fault intelligent detection system, characterized in that: It includes data collection module, Elasticsearch cluster fault analysis module, expert system module and model optimization and self-learning module; among them: The data collection module is used to collect current log data and monitoring indicator data of the Elasticsearch cluster; The Elasticsearch cluster fault analysis module is used to extract feature vectors of the current log data and monitoring indicator data using a natural language processing (NLP) algorithm, input the feature vectors into a fault detection model, obtain the probability distribution of each fault type output by the fault detection model, and determine the predicted fault type of the Elasticsearch cluster; wherein the fault detection model is a large-scale pre-trained language model based on deep learning, and the fault detection model is trained based on historical log data and monitoring indicator data marked as normal, historical log data and monitoring indicator data marked as faults, and associated fault types included in a training data set; The expert system module is used to verify the predicted fault type according to the current log data, monitoring indicator data and expert rule base, and determine the target fault type of the Elasticsearch cluster according to the verification result; wherein the expert rule base includes multiple rules, each rule includes a fault type, Elasticsearch cluster fault symptoms and solutions; The model optimization and self-learning module is used to perform model training on the fault detection model based on the current log data and monitoring indicator data, the predicted fault type and the target fault type.

2. The Elasticsearch cluster fault intelligent detection system according to claim 1, characterized in that: The expert system module is specifically used for: Matching and verifying the current log data, monitoring indicator data and the predicted fault type with the rules of the expert rule base to obtain a verification result; When the verification result is that the target rule exists in the expert rule base, and the fault type and the Elasticsearch cluster fault symptom of the target rule match the predicted fault type and the current log data and the monitoring indicator data, determining that the target fault type of the Elasticsearch cluster is the predicted fault type; When the verification result is that there is no target rule in the expert rule base that matches the current log data and monitoring indicator data and the predicted fault type, based on the Elasticsearch cluster fault symptoms corresponding to the current log data and monitoring indicator data in the expert rule base, the fault type associated with the Elasticsearch cluster fault symptom in the expert rule base is determined as the target fault type of the Elasticsearch cluster.

3. The Elasticsearch cluster fault intelligent detection system according to claim 1, characterized in that: The fault detection model includes an input layer, a convolution layer, a pooling layer, a fully connected layer and a Softmax layer; Among them, the input layer receives the feature vector; the convolution layer extracts local features in the feature vector through the convolution kernel to identify each fault type; the pooling layer downsamples the local features extracted by the convolution layer to obtain key features; the fully connected layer maps the key features sampled by the pooling layer to each fault type to obtain a feature mapping result; the Softmax layer generates a probability distribution of each fault type according to the feature mapping result output by the fully connected layer.

4. The Elasticsearch cluster fault intelligent detection system according to claim 1, characterized in that: The data acquisition module is also used for: The historical log data and monitoring indicator data, various fault types and solutions are collected and sent to the model optimization and self-learning module.

5. The Elasticsearch cluster fault intelligent detection system according to claim 4, characterized in that: The model optimization and self-learning module is also used to: According to the Elasticsearch cluster failure symptoms corresponding to the historical log data and monitoring indicator data, the historical log data and monitoring indicator data are marked as corresponding failure types, and the occurrence time, triggering conditions, operation steps, and impact of the failure on the Elasticsearch cluster and business corresponding to each historical log data and monitoring indicator data marked as a failure are recorded to generate the training data set.

6. The Elasticsearch cluster fault intelligent detection system according to claim 4, characterized in that: The model optimization and self-learning module is also used to: According to the newly added historical log data and monitoring indicator data, the fault detection model is regularly retrained and updated.

7. The Elasticsearch cluster fault intelligent detection system according to claim 1, characterized in that: The model optimization and self-learning module is also used to: The fault detection model is retrained according to the updated rules and newly added rules in the expert rule base.

8. The Elasticsearch cluster fault intelligent detection system according to claim 1, characterized in that: The model optimization and self-learning module is also used to: A cross entropy loss function is used to measure the difference between the predicted fault type and the target fault type, and the parameters of the fault detection model are updated.

9. The Elasticsearch cluster fault intelligent detection system according to claim 1, characterized in that: The fault type includes at least one of the following: Index fragmentation, query delay, CPU overload, memory overflow, master node failure, data node failure, network partition failure.

10. The Elasticsearch cluster fault intelligent detection system according to any one of claims 4 to 6, characterized in that: The system further comprises: The system monitoring and feedback module, together with the data acquisition module, the Elasticsearch cluster fault analysis module, the expert system module and the model optimization and self-learning module, is used to monitor the operation of the data acquisition module, the Elasticsearch cluster fault analysis module, the expert system module and the model optimization and self-learning module.

Images