An intelligent inspection and early warning method based on an Elasticsearch system
By constructing multi-dimensional inspection projects and complex problem analysis models, the problem of difficulty in locating the root cause of faults in the Elasticsearch system by monitoring a single indicator was solved, enabling fast and accurate fault location and handling, and improving system stability and efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SUNING COM CO LTD
- Filing Date
- 2026-02-06
- Publication Date
- 2026-06-05
Smart Images
Figure CN122152571A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of system inspection, and in particular to an intelligent inspection and early warning method based on the Elasticsearch system. Background Technology
[0002] During the operation of an Elasticsearch system, it is common to encounter sudden, complex, high-impact problems with difficult-to-locate root causes (such as resource overload, service interruption, and recovery delay).
[0003] Currently, single-metric monitoring systems are commonly used for early warning. However, a single metric can only reflect surface problems and cannot trace the underlying root causes. For example, during read / write operations, resource consumption on some nodes may increase dramatically, affecting the operational efficiency of other data units. In severe cases, this can lead to the collapse of critical underlying components, node failures, or even system service interruptions. A single metric can only show an increase in JVM GC (Java Virtual Machine garbage collection) frequency and processing time, but cannot help staff quickly pinpoint the cause of the problem as a sudden surge in resource consumption. Similarly, after a cluster node experiences a failure and restart, a single metric may only show that some data units cannot be restored to a usable state for an extended period, failing to help staff quickly locate the cause as a few large data units occupy the recovery process. Furthermore, when creating new data units, a single metric may only show a significant increase in the time spent allocating cluster resources, failing to help staff quickly pinpoint the cause as a lack of reasonable evaluation and dynamic adjustment of resource allocation parameters in the data unit creation rules.
[0004] Relying on the professional experience of staff for manual operation and maintenance makes it difficult to respond quickly to such problems, often resulting in high post-event handling costs and poor system stability. Summary of the Invention
[0005] The technical problem to be solved by this invention is to address the aforementioned deficiencies in the prior art by resolving the issue that a single-metric monitoring system cannot quickly locate the root cause of Elasticsearch system malfunctions.
[0006] To address the aforementioned technical problems, the present invention proposes an intelligent inspection and early warning method based on the Elasticsearch system, comprising the following steps:
[0007] The inspection project is constructed in multiple dimensions. Each inspection project includes at least an inspection item, a value standard, and a processing suggestion. The dimensions, from top to bottom, include at least a cluster, a node, and an index.
[0008] Construct several complex problem analysis models, each of which corresponds to a problem type and the inspection items involved in that problem type;
[0009] Data is collected for each inspection item to obtain inspection item information; the inspection item information is compared with the value standard, and inspection item information that exceeds the value standard is regarded as abnormal information.
[0010] Each complex problem analysis model extracts abnormal information related to the problem type of this model; when the abnormal information is limited to the same dimension, the complex problem analysis model outputs risk warnings and handling suggestions for the corresponding problem type that occurs in that dimension; when the abnormal information involves multiple dimensions, the complex problem analysis model outputs risk warnings and handling suggestions for the corresponding problem type that occurs in the highest dimension.
[0011] The output of all complex problem analysis models is compiled and sent to the user.
[0012] In one implementation, when constructing a complex problem analysis model, a warning value corresponding to the inspection item is set simultaneously. Abnormal information is compared with the warning value. When the abnormal information exceeds the warning value, the complex problem analysis model outputs an increased risk of the corresponding problem type.
[0013] In one implementation, as the number of abnormal information items extracted by the complex problem analysis model increases, the risk of the corresponding problem type occurring increases.
[0014] In one implementation, the inspection items at the cluster level include at least the number of pending tasks and the CPU utilization range; the inspection items at the node level include at least the number of search rejects, the number of bulk rejects, the JVM heap memory utilization, and the number of shards per node; and the inspection items at the index level include at least the range of shards within the same index and the size of a single shard.
[0015] In one implementation, the complex problem analysis model includes a resource burst occupation problem analysis model, involving node-level inspection items such as the number of search rejects, the number of bulk rejects, and JVM heap memory usage, as well as index-level inspection items such as the size of a single shard. When abnormal information appears in the JVM heap memory usage and multiple single shard sizes of a node, the complex problem analysis model outputs a risk warning about the resource burst occupation problem and corresponding handling suggestions for the inspection items. When the number of search rejects and the number of bulk rejects of the node also show abnormal information, the complex problem analysis model outputs a risk warning about the occurrence of a resource burst occupation problem and handling suggestions.
[0016] In one implementation, the complex problem analysis model includes a state recovery blocking problem analysis model, involving the number of shards per node in the inspection items at the node level, the size of a single shard in the inspection items at the index level, and the number of pending tasks in the inspection items at the cluster level. When the number of shards and the size of multiple single shards of a node show abnormal information, the complex problem analysis model outputs a risk warning and handling suggestions that the node has experienced a state recovery blocking problem. When multiple nodes show the above abnormal information and the number of pending tasks also shows abnormal information, the complex problem analysis model outputs a risk warning and handling suggestions that the cluster has experienced a state recovery blocking problem.
[0017] In one implementation, the complex problem analysis model includes a resource allocation imbalance problem analysis model, involving the range of the index shard in the inspection items at the index dimension, and the range of CPU utilization in the inspection items at the cluster dimension. When anomalies are found in the range of the index shard in the same index, the complex problem analysis model outputs a risk warning and handling suggestions indicating that storage imbalance has occurred. When anomalies are found in the CPU utilization range at the same time, the complex problem analysis model outputs a risk warning and handling suggestions indicating that in addition to storage imbalance, system CPU and memory resources are also beginning to show imbalance.
[0018] In one implementation, when a new type of problem arises, additional inspection items related to that problem type are added to the existing inspection items, and a complex problem analysis model corresponding to that problem type is constructed.
[0019] In one implementation, an inspection control switch is provided, which shuts down the inspection when the number of inspections exceeds the preset number of inspections per day.
[0020] The beneficial effects of this invention are:
[0021] 1. This invention employs abstract modeling to address sudden, complex, high-impact, and difficult-to-analyze problems, mapping specific issues to general complex problem analysis models. This allows for rapid identification of the root cause of failures and reduces reliance on human experience. Compared to traditional solutions, this invention offers superior foresight, efficiency, and low cost.
[0022] 2. Establish a multi-dimensional inspection project system. Based on the differences in the impact scope of inspection projects in different dimensions, construct complex problem analysis models for the corresponding problem types in each dimension. Through multi-granularity periodic inspections, local problems and system problems can be quickly identified, further improving the efficiency of fault diagnosis and achieving a model upgrade from "passive firefighting" to "proactive fire prevention".
[0023] 3. When new types of problems emerge, new inspection items and corresponding complex problem analysis models can be added simultaneously, demonstrating the good flexibility and scalability of this invention. Attached Figure Description
[0024] The invention will now be further described with reference to the accompanying drawings.
[0025] Figure 1 This is a flowchart of an intelligent inspection and early warning method based on the Elasticsearch system according to an embodiment of the present invention. Detailed Implementation
[0026] like Figure 1 As shown in the figure, this embodiment of the invention provides an intelligent inspection and early warning method based on the Elasticsearch system, including the following steps:
[0027] S1, construct inspection projects by dimension, each inspection project includes at least inspection items, value standards, and processing suggestions; the dimensions from top to bottom include at least clusters, nodes, and indexes;
[0028] In one implementation, the inspection items at the cluster level include the number of pending tasks and the CPU utilization range; the inspection items at the node level include the number of search rejections, the number of bulk rejections, the JVM heap memory utilization, and the number of shards per node; and the inspection items at the index level include the range of shards within the same index and the size of a single shard.
[0029] Specifically, the inspection items constructed in this embodiment are shown in the table below:
[0030]
[0031] S2, Construct several complex problem analysis models, each complex problem analysis model corresponds to a problem type and the inspection items involved in that problem type; in this embodiment, each problem type involves inspection items in at least two dimensions;
[0032] In one implementation, an analysis model for sudden resource occupancy, a model for state recovery blocking, and a model for resource allocation imbalance were constructed.
[0033] S3: Collect data for each inspection item to obtain inspection item information; compare the inspection item information with the value standard, and treat the inspection item information that exceeds the value standard as abnormal information.
[0034] Specifically, when starting intelligent inspection, it is necessary to first use the Client to establish a connection with the cluster, use the Client to query different APIs to obtain information and indicators on the cluster, identify and judge the health status against the standards of the inspection items, and input the inspection item status into the complex problem analysis model for extraction and analysis by different analysis models.
[0035] S4, Each complex problem analysis model extracts abnormal information related to the problem type of this model; when the abnormal information is limited to the same dimension, the complex problem analysis model outputs risk warnings and handling suggestions for the corresponding problem type occurring in that dimension; when the abnormal information involves multiple dimensions, the complex problem analysis model outputs risk warnings and handling suggestions for the corresponding problem type occurring in the highest dimension.
[0036] In one implementation, when constructing a complex problem analysis model, a warning value is simultaneously set for each inspection item. Abnormal information is compared with the warning value. When the abnormal information exceeds the warning value, the complex problem analysis model outputs an increased risk of the corresponding problem type. For example, if the abnormal information does not exceed the warning value, it prompts attention to the problem; if it exceeds the warning value, it indicates that the problem has likely occurred.
[0037] In one implementation, as the number of anomalous information items extracted by the complex problem analysis model increases, the risk of the corresponding problem type increases. For example, if only one or two anomalous information items are extracted, a warning is issued to pay attention to the problem; if more than two anomalous information items are extracted, a warning is issued indicating that the problem has likely occurred.
[0038] S5 summarizes the output of all complex problem analysis models and sends it to the user.
[0039] In one implementation, the complex problem analysis model includes a resource burst consumption problem analysis model. This model involves node-level inspection items such as the number of search rejects, the number of bulk rejects, and JVM heap memory usage, as well as index-level inspection items such as single shard size. When anomalies are found in the JVM heap memory usage and multiple single shard sizes of a node, the complex problem analysis model outputs a risk warning indicating a resource burst consumption problem and corresponding handling suggestions for the relevant inspection items. Similarly, when anomalies are found in the number of search rejects and the number of bulk rejects for the same node, the model outputs a risk warning indicating a resource burst consumption problem has occurred and handling suggestions. In this embodiment, the warning value for JVM heap memory usage is 90%; the warning value for single shard size is 60GB; and the warning values for the number of search rejects and the number of bulk rejects are 200.
[0040] The inherent connection between these inspection projects and sudden resource consumption issues is as follows: sudden resource consumption mainly manifests at the memory and disk I / O levels. When performing query operations on an index with an excessively large single shard data volume, it can lead to a surge in short-term memory demand. The larger the shard data volume and the wider the query coverage, the higher the memory demand. Although the Elasticsearch system itself has a circuit breaker mechanism that rejects some search requests, when the JVM (Java Virtual Machine) utilization is already at a high level, if concurrent queries on multiple large shard indexes occur at this time, memory resources will be heavily consumed. This will not only cause other query requests to be rejected, but in more severe cases, it can also cause an OutOfMemoryError (OOM), making the service completely unavailable.
[0041] In one implementation, the complex problem analysis model includes a state recovery blocking problem analysis model, involving the number of shards per node in the inspection items at the node level, the size of a single shard in the inspection items at the index level, and the number of pending tasks in the inspection items at the cluster level. When anomalies occur in the number of shards and multiple single shard sizes of a node, the complex problem analysis model outputs a risk warning and handling suggestions indicating that the node has experienced a state recovery blocking problem. When multiple nodes exhibit the above-mentioned anomalies and the number of pending tasks also shows anomalies, the complex problem analysis model outputs a risk warning and handling suggestions indicating that the cluster has experienced a state recovery blocking problem. In this embodiment, the warning value for the number of shards per node is 1000; the warning value for the size of a single shard is 60GB; and the warning value for the number of pending tasks is 200.
[0042] The inherent connection between these inspection items and sudden resource consumption issues is as follows: After a node restarts, it will randomly load the original index shards on the node into memory one by one. Only after completing this operation can the node provide services. The number of shards loaded at one time is at most dozens. The main factors affecting the recovery of all indexes on a node are the number and size of the index shards. pending_tasks represent a backlog of tasks in the cluster. The more backlogged tasks there are, the weaker the service response capability of the cluster becomes. Specifically, the official recommended value for the number of shards on a node is 1000. However, in actual business scenarios, the number of shards on a node has reached as high as 2000. Because these shards are relatively small, the overall recovery speed is relatively fast. However, when there are large shards (such as a single shard reaching 50GB), the recovery process will be significantly slower, and the recovery process of such large shards will cause blockage, which will affect the normal recovery of other index shards and ultimately cause blocking problems in state recovery.
[0043] In one implementation, the complex problem analysis model includes a resource allocation imbalance problem analysis model, involving the range of index-level inspection items (such as the range of shards within the same index) and the range of cluster-level inspection items (such as the range of CPU utilization). When anomalies are found in the range of a certain index's shards, the complex problem analysis model outputs a risk warning and handling suggestions indicating that storage imbalance has occurred. When anomalies are also found in the CPU utilization range, the complex problem analysis model outputs a risk warning and handling suggestions indicating that, in addition to storage imbalance, system CPU and memory resource imbalances are also beginning to occur. In this embodiment, the shard range warning value is 5GB; the CPU utilization range warning value is 15.
[0044] The inherent connection between these inspection items and sudden resource consumption issues is as follows: Under normal circumstances, the shard sizes under the same index are roughly similar, data is evenly distributed, and the shards of that index are distributed across different nodes in the cluster, resulting in a load-balanced cluster. When some nodes in the cluster trigger resource alerts, while the resource utilization of other nodes remains at a low level, the CPU_util (central processing unit utilization) of each node is compared, and the relevant status of the index shards is checked. It was found that the larger the range of shard-related indicators, the more severe the resource imbalance problem in the cluster. Of course, due to the influence of different server configurations and cluster rules, the reasonable threshold for this range will also vary and needs to be flexibly adjusted according to the actual scenario.
[0045] In one implementation, when a new type of problem arises, additional inspection items related to that problem type are added to the existing inspection items, and a complex problem analysis model corresponding to that problem type is constructed.
[0046] In one implementation, a patrol control switch is set up. When the number of patrols exceeds a preset daily patrol count, patrols are shut down. Specifically, implementing a dynamic switch relies on the parameter transitivity of the scheduling platform. An executable flag is verified at the initial stage of patrols to determine whether to execute the specific patrol logic. During certain specific periods, the patrol function needs to be disabled, such as during cluster migration or scaling up / down. The core logic of reasonable scheduling is: due to high service usage density and peak business periods, the cluster is under significant pressure, making it unsuitable to conduct more patrol operations at these times, as this would affect the cluster's read / write efficiency. For different clusters, a customizable patrol execution time range can be defined. The system first determines whether the current time is within this execution time range; if it is, the specific patrol process is then executed.
Claims
1. A method for intelligent inspection and early warning based on the Elasticsearch system, characterized in that, Includes the following steps: The inspection project is constructed in multiple dimensions. Each inspection project includes at least an inspection item, a value standard, and a processing suggestion. The dimensions, from top to bottom, include at least a cluster, a node, and an index. Construct several complex problem analysis models, each of which corresponds to a problem type and the inspection items involved in that problem type; Data is collected for each inspection item to obtain inspection item information; Compare the inspection item information with the value standard, and treat the inspection item information that exceeds the value standard as abnormal information; Each complex problem analysis model extracts abnormal information related to the problem type of this model; When the abnormal information is limited to the same dimension, the complex problem analysis model outputs risk warnings and handling suggestions for the corresponding problem type in that dimension; When abnormal information involves multiple dimensions, this complex problem analysis model outputs a risk warning and handling suggestions for the problem type corresponding to the highest dimension. The output of all complex problem analysis models is compiled and sent to the user.
2. A method for intelligent inspection and early warning based on the Elasticsearch system, characterized in that: When constructing a complex problem analysis model, a warning value is set for each inspection item. Abnormal information is compared with the warning value. When the abnormal information exceeds the warning value, the complex problem analysis model outputs an increased risk of the corresponding problem type.
3. An intelligent inspection and early warning method based on the Elasticsearch system, characterized in that: As the number of abnormal information items extracted by the complex problem analysis model increases, the risk of the corresponding problem type occurring increases.
4. The intelligent inspection and early warning method based on the Elasticsearch system as described in claim 1, characterized in that: Cluster-level inspection items should include at least the number of pending tasks and the CPU utilization range; node-level inspection items should include at least the number of search rejects, the number of bulk rejects, JVM heap memory utilization, and the number of shards per node; index-level inspection items should include at least the range of shards within the same index and the size of a single shard.
5. The intelligent inspection and early warning method based on the Elasticsearch system as described in claim 4, characterized in that: The complex problem analysis model includes a resource burst consumption problem analysis model, which involves node-level inspection items such as the number of search rejections, the number of bulk rejections, and JVM heap memory usage, as well as index-level inspection items such as the size of a single shard. When abnormal information appears in the JVM heap memory usage of a node and multiple single shard sizes, this complex problem analysis model outputs risk warnings about resource burst consumption problems and corresponding handling suggestions for the inspection items. When the number of search rejects and bulk rejects of a node also shows abnormal information, the complex problem analysis model outputs a risk warning and handling suggestions for a sudden resource occupation problem.
6. The intelligent inspection and early warning method based on the Elasticsearch system as described in claim 4, characterized in that: The complex problem analysis model includes a state recovery blocking problem analysis model, which involves the number of shards per node in the inspection items at the node level, the size of a single shard in the inspection items at the index level, and the number of pending tasks in the inspection items at the cluster level. When abnormal information appears in the number of shards and multiple single shard sizes of a node, the complex problem analysis model outputs a risk warning and handling suggestions that the node has experienced a state recovery blocking problem. When multiple nodes exhibit the above-mentioned abnormal information and the number of pending tasks also shows abnormal information, the complex problem analysis model outputs a risk warning and handling suggestions indicating that the cluster has experienced a state recovery blocking problem.
7. The intelligent inspection and early warning method based on the Elasticsearch system as described in claim 4, characterized in that: The complex problem analysis model includes a resource allocation imbalance problem analysis model, which involves the range of index-level inspection items and the range of CPU utilization inspection items at the cluster level. When an abnormal information is found in the range of a certain index's shards, the complex problem analysis model outputs a risk warning and handling suggestions indicating that storage imbalance has occurred. When both CPU util values are extremely poor and abnormal information is displayed, the complex problem analysis model outputs a risk warning and handling suggestions, indicating that in addition to storage imbalance, the system's CPU and memory resources are also beginning to show imbalance.
8. The intelligent inspection and early warning method based on the Elasticsearch system as described in claim 4, characterized in that: When a new type of problem emerges, based on the existing inspection items, add inspection items related to that problem type and construct a complex problem analysis model corresponding to that problem type.
9. The intelligent inspection and early warning method based on the Elasticsearch system as described in claim 1, characterized in that: Set an inspection control switch so that the inspection is turned off when the number of inspections exceeds the preset number of inspections per day.