Thread pool protection system of Elasticsearch cluster

Abstract

The invention provides a thread pool protection system of an Elasticsearch cluster, and relates to the technical field of computers. The thread pool protection system comprises a flow monitoring module and a thread pool protection module, wherein the flow monitoring module is used for collecting first multi-dimensional data of an Elasticsearch cluster; the anomaly detection module is used for identifying an anomaly type corresponding to the first multi-dimensional data according to an anomaly detection strategy; the strategy decision execution module is used for acquiring and executing a first processing strategy matched with the exception type; the cluster state feedback module is used for continuously obtaining second multi-dimensional data of the Elasticsearch cluster after the first processing strategy is executed, and if it is determined that the execution result of the first processing strategy does not reach the standard on the basis of the currently obtained second multi-dimensional data, the Elasticsearch cluster is executed. If yes, the control range is expanded on the basis of the first processing strategy according to a preset retry strategy, strategy retry is carried out by keeping a preset core index in the first processing strategy unchanged, and the preset core index is an index associated with the service.

Description

TECHNICAL FIELD

[0001] The present application mainly relates to the technical field of computer technology, and particularly relates to a thread pool protection system of an Elasticsearch cluster. BACKGROUND

[0002] An Elasticsearch cluster (hereinafter referred to as an ES cluster) usually uses multi-threading to process various operations such as indexing, searching and aggregation. When encountering abnormal traffic (such as DDoS attack, sudden write peak), the thread pool may cause queue overflow and thread exhaustion due to task backlog, eventually triggering the rejection strategy, causing service interruption and business anomalies. The traditional solution generally adopts manual temporary adjustment of the thread pool size based on experience, and the business adopts a fuse mechanism to reduce the add, delete, modify and query operations on the Elasticsearch cluster. It can be seen that the traditional solution relies on manual troubleshooting-experience adjustment, and the efficiency of adjusting the thread pool is low and the accuracy cannot be guaranteed. SUMMARY

[0003] It would be advantageous to provide a mechanism that alleviates, mitigates or eliminates at least one of the above problems.

[0004] In a first aspect, the present application provides a thread pool protection system of an Elasticsearch cluster, comprising: a traffic monitoring module configured to collect first multi-dimensional data of the Elasticsearch cluster, the first multi-dimensional data comprising any one or more of network traffic, thread pool state data and host system resources; an anomaly detection module configured to identify an anomaly type corresponding to the first multi-dimensional data according to an anomaly detection strategy; a strategy decision execution module configured to obtain and execute a first processing strategy matched with the anomaly type; a cluster state feedback module configured to, after execution of the first processing strategy, continuously obtain second multi-dimensional data of the Elasticsearch cluster, and if it is determined that the execution result of the first processing strategy does not meet the standard based on the currently obtained second multi-dimensional data, perform strategy retry by expanding the control range based on the first processing strategy and keeping a preset core index in the first processing strategy unchanged according to a preset retry strategy, the preset core index being a business-related index; The cluster state feedback module is further configured to, when the execution result of the strategy retry does not meet the standard, upgrade the strategy level of the first processing strategy to obtain a second processing strategy, and execute the second processing strategy.

[0005] Compared with the prior art, the application has the following advantages: through the abnormality detection module and the policy decision execution module, automatic fault positioning, policy generation and execution feedback are completed, manual intervention cost is reduced, the fault processing period is shortened from manual troubleshooting-experience adjustment of the hour level response to real-time detection-automatic repair of the minute level closed loop, and the operation and maintenance efficiency and the accuracy of problem disposal are significantly enhanced. In addition, through the cluster state feedback module, the effect of the current policy execution is verified after the policy execution, when the effect of the current policy execution does not meet the standard, the policy retry is performed by expanding the control range on the basis of the original policy and keeping the preset core index unchanged, which is beneficial to balance the defense and business impact. And after the policy retry fails, the policy level of the current policy is upgraded, the defense is gradually enhanced step by step, and the stability of the business is ensured. BRIEF DESCRIPTION OF DRAWINGS

[0006] The accompanying drawings are included to provide a further understanding of the application, and they are collected and constitute a part of the application, which show the embodiments of the application, and together with the specification, play a role in explaining the principles of the application. In the drawings: Figure 1 is a structural schematic diagram of an Elasticsearch cluster thread pool protection system provided by an embodiment of the application. DETAILED DESCRIPTION

[0007] In order to more clearly illustrate the technical solutions of the embodiments of the application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some examples or embodiments of the application, and for those skilled in the art, the application can also be applied to other similar scenarios without creative labor. Unless the context clearly indicates otherwise or otherwise stated, the same reference numbers in the drawings represent the same structure or operation.

[0008] As shown in the application and claims, unless the context clearly indicates otherwise or otherwise stated, "one", "a", "an" and / or "the" do not refer to a single number, but also include plural. Generally speaking, the terms "include" and "contain" only indicate that the steps and elements explicitly identified are included, and these steps and elements do not constitute an exclusive list, and the method or device can also include other steps or elements.

[0009] The foregoing summary, as well as the following detailed description of the application, is better understood when read in conjunction with the appended drawings, in which: FIG. 1 illustrates an example of a system for providing a user interface for a user to interact with a virtual environment, according to an embodiment of the present application.

[0010] Flowcharts have been used herein to illustrate the operations performed by systems in accordance with embodiments of the present application. It should be understood that the operations previously or hereafter are not necessarily performed in the precise order described. Rather, various steps can be handled in reverse order, or simultaneously. Also, other operations can be added to, or removed from, these processes, or one or more steps can be omitted.

[0011] To better understand the present application, the following terms related to the present application are explained: Elasticsearch is a Lucene-based distributed search engine that uses multi-threading to handle various operations such as indexing, searching, and aggregating, supports multi-node cluster deployment, can handle PB-level data and achieve near real-time search. Unlike other search engines (such as Solr): Elasticsearch adopts a distributed architecture (Master / Slave or Cluster mode), supports automatic sharding (Shard) and replica mechanism (Replica), has higher availability and fault tolerance. Elasticsearch stores data shards (Shard) on multiple nodes in the Elasticsearch cluster, each shard is an independent Lucene index, supports horizontal expansion (adding nodes can increase storage and computing power). For example, in an enterprise log retrieval system, a distributed architecture can solve the storage problem of massive log data, by sharding the data to multiple nodes in the Elasticsearch cluster, improving retrieval efficiency. Elasticsearch index delay (from data writing to retrievable time) is usually within 1 second, meeting the real-time data processing demand (such as log analysis, real-time monitoring). For example, in an enterprise log retrieval system, near real-time indexing can allow operations personnel to retrieve the latest logs immediately after the logs are written, quickly locate problems.

[0012] Abnormal traffic: Request traffic that exceeds the system's normal processing capacity or violates preset patterns, including but not limited to DDoS attacks, sudden traffic peaks, slow query storms, etc.

[0013] Thread pool: A component in Elasticsearch used to manage task execution, containing parameters such as core thread count, maximum thread count, queue capacity, etc., responsible for coordinating request processing and resource allocation.

[0014] Self-protection mechanism: Automatically trigger adjustment strategies (such as dynamic expansion, throttling, and circuit breaking) by monitoring traffic and thread pool status in real time, preventing system outages or resource depletion due to overload.

[0015] Rejection policy: Handling method when the thread pool cannot handle new tasks, such as the default AbortPolicy (throwing exceptions) and DiscardPolicy (discarding tasks).

[0016] Dynamic adjustment: Automatically modify key parameters of the thread pool (such as queue size and core thread count) based on real-time monitoring indicators (such as queue blocking length, thread utilization, and memory occupancy).

[0017] Circuit breaking mechanism: Temporarily cut off part of the request when the anomaly persists, gradually release when the system recovers, preventing fault propagation.

[0018] To alleviate the problem of traditional Elasticsearch cluster thread pool protection, the present application proposes a thread pool protection system for Elasticsearch cluster, aiming to strengthen the dynamic linkage of business traffic characteristics and Elasticsearch system parameters, and to address abnormal large traffic concurrency scenarios such as sudden traffic peaks, DDoS attacks, and slow query storms. Through dynamic adjustment mechanisms (including real-time adaptation of queue capacity, thread count, and rejection policy) and hierarchical response protection systems (gradient triggering of early warning, throttling, and circuit breaking), the system-level overload defense capability is built to achieve at least one of the following goals: 1. Systematic risk prevention and control: Through deep coupling analysis of traffic characteristics and thread pool status, real-time overload risk perception is achieved to avoid system crashes and service interruptions caused by queue overflow and thread depletion, ensuring core business continuity. 2. Automatic elastic adaptation: Break through the limitations of traditional static configuration, dynamically optimize thread pool resources based on real-time monitoring data, achieve intelligent matching of traffic-resources, and improve the balance ability of system throughput and stability. 3. Operation and maintenance efficiency improvement: Through the abnormality detection module and the policy decision execution module, automatic fault positioning, policy generation and execution feedback are completed, manual intervention cost is reduced, the fault handling period is shortened from the hour-level response of "manual troubleshooting-experience adjustment" to the minute-level closed loop of "real-time detection-automatic repair", and the operation and maintenance efficiency and the accuracy of problem disposal are significantly enhanced.

[0019] Referring to Figure 1 In an embodiment of the present application, the thread pool protection system comprises: The traffic monitoring module 10 is configured to collect first multi-dimensional data of the Elasticsearch cluster, and the first multi-dimensional data comprises any one or more of network traffic, thread pool state data and host system resources.

[0020] In some embodiments, the network traffic refers to network traffic accessing the Elasticsearch cluster, and the network traffic is effective request traffic of a preset target port, including request rate, protocol type, source IP region distribution, port connection number and other characteristic indexes. For example, the preset target port is 9200 / 9300, and the specific process of collecting the network traffic can be as follows: the network traffic accessing the Elasticsearch cluster with the target port of 9200 / 9300 is captured through a log statistical interface of a front reverse proxy / load balancing device (such as Nginx), including indexes such as request rate (QPS), protocol type (HTTP / HTTPS / TCP), source IP region distribution, and port connection number.

[0021] In some embodiments, the thread pool state comprises thread pool indexes and network layer indexes, the thread pool indexes comprise queue length, active thread number, completed task number and thread rejection number, and the network layer indexes comprise inbound traffic rate and outbound traffic rate.

[0022] For example, the specific process of collecting the thread pool state can be as follows: the Elasticsearch built-in API interface is called to periodically (by default, 1 second) pull the thread pool state of the ES cluster, and the specific interface is as follows: the thread pool indexes of queue length, active thread number, completed task number and thread rejection number are obtained through / _nodes / stats / thread_pool, and the network layer indexes of inbound traffic rate and outbound traffic rate are obtained through / _nodes / stats / network.

[0023] In some embodiments, the host system resources comprise CPU utilization, heap memory usage and GC (Garbage Collection) pause time. For example, the host system resources can be collected by integrating JVM monitoring (using the java.lang.management interface).

[0024] The anomaly detection module 11 is configured to identify the type of anomaly corresponding to the first multi-dimensional data according to an anomaly detection strategy.

[0025] In some embodiments, the traffic monitoring module 10 is further configured to perform data standardization processing on the collected first multi-dimensional data, and perform uniform format processing on the collected first multi-dimensional data, such as converting the thread pool queue length into a usage rate (queue_size / max_queue_size), converting the request rate into an average value, and forming a standard data set D={t, text{QPS} request rate average, text{queue_ratio} thread pool queue usage rate, text{reject_rate} rejection rate, text{cpu_usage} CPU usage. Based on this, the anomaly detection module 11 is configured to identify the type of anomaly according to the standardized data.

[0026] In some embodiments, the anomaly detection strategy can include a traffic surge and overload detection strategy, an attack feature and malicious request identification strategy, and a composite rule identification strategy. Each anomaly detection strategy can specify specific strategy content, anomaly type, and risk level.

[0027] For example, the traffic surge and overload detection strategy can include a traffic surge detection strategy, which is: cluster-level QPS > historical peak value x 150% (the peak value is dynamically calculated by a 7-day sliding window), and does not fall back for 3 consecutive 30-second windows, marking a risk level of the fourth level and a traffic red surge anomaly (i.e., the anomaly type is a traffic red surge anomaly). For another example, the traffic surge detection strategy can also include: the current window QPS increases by >100% compared to the previous window, and exceeds the load test baseline (such as 2000 QPS), marking a risk level of the third level and a traffic yellow warning anomaly (i.e., the anomaly type is a traffic yellow warning anomaly).

[0028] The traffic surge and overload detection strategy can include a traffic overload detection strategy, which is: when the search thread pool queue usage rate >80% and the / _search interface QPS growth >50%, marking a risk level of the second level and a query queue backlog anomaly, i.e., the anomaly type is a query queue backlog anomaly; when the thread pool rejection rate >5% and the corresponding interface traffic / request ratio >70%, marking a risk level of the second level and a query rejection rate overload risk, i.e., the anomaly type is a query rejection rate overload risk.

[0029] The attack feature and malicious request identification strategy includes an attack feature identification strategy, and the attack feature identification strategy includes a DDoS attack detection strategy. The DDoS attack detection strategy can be, for example, that the number of SYN packets occupying TCP connections is greater than 80%, and the source IP region concentration degree is greater than 90%, and the risk level is marked as the third level and the SYN protocol is abnormal, that is, the abnormal type is the SYN protocol abnormality or the DDoS attack.

[0030] The attack feature and malicious request identification strategy includes a malicious request identification strategy. The malicious request identification strategy is, for example, that the request path contains _search and the size parameter is not specified in the query statement, and from+size>10000, the risk level is marked as the third level, and the full-scan abnormality, that is, the abnormal type is the full-scan abnormality.

[0031] The composite rule identification strategy is a strategy that combines the traffic surge and overload detection strategy and the attack feature and malicious request identification strategy. The composite rule identification strategy can improve the detection accuracy. For example, the composite rule identification strategy is that when the SYN protocol abnormality and the query rejection rate overload risk are met at the same time, the risk level is marked as the fourth level, and the abnormal type is the attack query; when the high-overhead query abnormality and the query queue backlog abnormality are met at the same time, the risk level is marked as the fourth level, and the abnormal type is the slow query storm; and when the traffic red surge abnormality and the query rejection rate overload risk are met at the same time, the risk level is marked as the fourth level, and the abnormal type is the high-concurrency query of the business.

[0032] It should be noted that the abnormality detection strategy can be pre-configured by relevant personnel. The specific content of the abnormality detection strategy described above is only an example. As a feasible way, the content of the abnormality detection strategy can be adjusted according to the actual situation. The basic principle is to ensure that each abnormal type can be covered by the corresponding abnormality detection strategy to ensure the comprehensiveness of the abnormality detection. The present application does not make a specific limitation in this regard.

[0033] The policy decision execution module 12 is configured to acquire and execute the first processing strategy matched with the abnormal type.

[0034] The policy decision execution module 12 is mainly responsible for dynamically adjusting the thread pool parameters, enabling the hierarchical protection strategy, etc. according to the exception type (such as traffic surge, DDoS attack, slow query storm, etc.) output by the exception detection module, and realizing the closed-loop control from detecting the exception to executing the defense. The policy decision execution module 12 is designed in accordance with the principle of “precise matching of exception types and standardized calling of strategy interfaces”, ensuring that each exception type has a corresponding technical solution. Based on this, the corresponding processing strategy is configured in advance based on different exception types, and the policy decision execution module 12 can subsequently filter a first processing strategy matching the exception type output by the exception detection module from the pre-stored multiple processing strategies and execute the first processing strategy. Exemplarily, the processing strategies corresponding to different exception types can be as shown in Table 1.

[0035] Table 1

[0036] Exemplarily, when the exception type output by the exception detection module is “traffic red surge exception”, the corresponding first processing strategy includes a thread pool elastic expansion mechanism, which can be: increasing the core thread number to CPU core number x (1 + IO waiting rate / 100) x 1.5, and expanding the maximum thread number to 2 times the core thread number, to realize thread pool elastic expansion and enhance processing capacity. Alternatively, when the exception type output by the exception detection module is “traffic red surge exception”, the corresponding first processing strategy includes a queue elastic expansion mechanism, which can be: expanding the queue capacity by 1.5 times the current value, but not more than 10% of the physical memory, to prevent memory overflow.

[0037] The index of Elasticsearch is composed of multiple shards (primary shard + replica shard), and the shard data is stored in the form of “inverted index” on the disk. If a shard has not been accessed for a long time (cold shard), the node will replace it from the memory to release resources. When a large number of requests suddenly access the cold shard, index shard reconstruction needs to be triggered, and the core process of shard reconstruction is: shard metadata loading → inverted index reading from disk to memory → query cache initialization → shard state from “unloaded” to “available”.

[0038] This reconstruction process needs to occupy the core threads of the search thread pool, and each shard loading occupies at least one thread. If a large number of shards trigger reconstruction at the same time, the following problems will occur: 1. The number of active threads in the thread pool is suddenly pulled full; 2. New query requests enter the queue backlog; 3. The thread pool rejection rate rises, eventually triggering the chain problem of “traffic surge + thread pool overload”.

[0039] The core of the traffic preheating mechanism is to proactively load cold shards into memory before traffic surges, avoiding a large number of requests triggering shard reconstruction at the same time, and reducing the transient load of the thread pool from the source. Based on this, the strategy decision execution module is also used to: when detecting that the exception type is a target exception type related to traffic overload, identifying the hot index of the Elasticsearch cluster and the cold shard in the hot index, and performing a preheating operation on the cold shard, which is equivalent to triggering the reconstruction of the cold shard in advance. Wherein, the target exception type includes traffic yellow warning exception or traffic red surge exception, when the target exception type is traffic yellow warning exception, the subsequent preheating operation is triggered to execute, which is equivalent to early warning, when the target exception type is traffic red surge exception, the subsequent preheating operation is triggered to execute, which is equivalent to completing the preheating before the traffic reaches the peak, avoiding preheating while overloading.

[0040] In some embodiments, identifying the hot index of the Elasticsearch cluster specifically includes: calling a preset hot index interface to query the hot index based on a screening rule, avoiding wasting resources without differentiated preheating. The screening rule may, for example, be to query the indexes with a TopN (N is a positive integer) access frequency within a recent time T1 (such as within 1 hour), and determine them as hot indexes. The core purpose is to focus on high-traffic indexes and ensure that preheating resources are invested in core business scenarios.

[0041] In some embodiments, identifying the cold shard in the hot index specifically can include: calling a shard state interface to identify shards in the hot index that are not loaded into memory to determine as cold shards. Illustratively, the cold shards in the hot index can be identified according to a cold shard determination standard, which can be: the shard state is in a startup state but the document load is < a preset proportion threshold (such as 10%), or no memory access record of the shard is confirmed through a node hot thread interface (such as / _nodes / hot_threads)). For example, the order_search index has 8 shards, of which shards 2, 5, and 7 are cold shards, then shards 2, 5, and 7 can be used as objects of subsequent preheating operations.

[0042] In some embodiments, when it is determined in the above steps that the cold shards in the hotspot index include multiple cold shards, the preheating operation is performed on the cold shards, including: using a batched and low-priority strategy to call a cluster index shard preheating interface to perform the preheating operation on the multiple cold shards; wherein the number of cold shards preheated in each batch does not exceed a preset threshold (for example, 3), a preset time interval (for example, 10 seconds) is set between batches, and the preheating task is marked as low priority to avoid occupying core business thread resources. If preheating is directly performed on all cold shards, a single API call will still occupy thread pool resources, therefore, in the embodiments of the present application, the strategy of batched shards and time interval is used to control the thread occupation amount of a single preheating (1 search thread is occupied for each shard preheating), so as to avoid temporary overload of the thread pool caused by preheating itself.

[0043] The following describes how to perform the preheating operation on a single cold shard, and the preheating operation of the remaining cold shards is the same. In some embodiments, the preheating operation is performed on the cold shard, including: configuring a preheating request parameter for the cold shard, transmitting a preheating request carrying the preheating request parameter to a cluster index shard preheating interface, marking the preheating request as low priority to ensure that core query requests are processed first, and the preheating request is used to trigger the Elasticsearch cluster to perform metadata loading and inverted index loading from disk to memory on the cold shard, and return an execution result. When the execution result indicating successful execution is detected, it is confirmed that the preheating of the cold shard is completed.

[0044] In some embodiments, after the preheating of the cold shard is completed, a cluster state monitoring interface is called to verify the memory loading state and the thread pool running state of the cold shard, to confirm that the preheating takes effect and does not cause thread pool exceptions.

[0045] The memory loading state of the cold shard includes the inverted index memory occupation of the cold shard, and as a feasible way, the strategy decision execution module 12 is further configured to confirm that the preheating takes effect when the inverted index memory occupation of the cold shard is updated from 0 to a stable value (such as 512 MB). Otherwise, when the preheating does not take effect, preheating failure alarm information can be output to remind relevant personnel to handle.

[0046] In some embodiments, the thread pool running state includes the number of active threads, the queue length, and the number of rejected tasks, and the strategy decision execution module 12 is further configured to confirm that no thread pool exception is caused according to the thread pool running state and the thread pool verification standard. The thread pool verification standard is, for example, that the number of active threads is less than or equal to 50% of the number of core threads, the queue length is 0, and the number of rejected tasks is 0, to confirm that the preheating process does not occupy too many thread pool resources. Otherwise, when a thread pool exception is caused, thread exception alarm information can be output to remind relevant personnel to handle.

[0047] In some embodiments, after the preheating of all cold shards in the hot index is completed, the policy decision execution module 12 can still detect that the abnormality related to the traffic overload has not been eliminated, such as the red burst abnormality or the yellow burst abnormality. Based on this, in order to provide continuous protection during the traffic peak period, the policy decision execution module 12 is further configured to periodically perform preheating inspection according to a shard inspection period (such as 30 minutes) after the preheating of all cold shards in the hot index is completed, if it is detected that the abnormality corresponding to the target abnormality type has not been eliminated. The preheating inspection specifically includes: repeatedly performing the above-mentioned identification of the hot index of the Elasticsearch cluster and the cold shard in the hot index, identifying whether a new cold shard is generated, and repeatedly performing the above-mentioned preheating operation, the operation of confirming that the preheating takes effect and does not cause a thread pool abnormality, and the like, until it is detected that the abnormality related to the traffic overload is eliminated, the preheating inspection is stopped, and invalid resource occupation is avoided.

[0048] In some embodiments, when the abnormality type is the SYN protocol abnormality, the corresponding first processing strategy includes IP blocking, and the specific process of the policy decision execution module 12 performing the first processing strategy includes: cooperating with the firewall to block the attack source IP and block malicious traffic from entering the cluster.

[0049] In some embodiments, when the abnormality type is the SYN protocol abnormality, the corresponding first processing strategy includes rejection policy switching, and the specific process of the policy decision execution module 12 performing the first processing strategy includes: based on the rejection policy switching, submitting a policy configuration request to the configuration management interface of the cluster, dynamically configuring the rejection policy of the specified thread pool as the caller execution policy (CallerRunsPolicy), so that the thread pool executes the task exceeding the processing capacity by the calling thread, rather than directly rejecting the task, to realize the transfer of processing pressure from the thread pool to the calling thread, and prevent the thread pool from being exhausted of resources by malicious requests.

[0050] In some embodiments, when the abnormality type is the full scan abnormality, the corresponding first processing strategy includes the query automatic rewriting strategy, and the specific process of the policy decision execution module 12 performing the first processing strategy includes: automatically adding a quantity limit parameter size to the full scan request through the query automatic rewriting strategy, without the need for the client to modify the code, to realize non-intrusive intervention. For example, the full scan abnormality is a full scan type query request without specifying a quantity limit parameter. Through the query automatic rewriting strategy, a limit condition of size = a preset limit value (such as 1000) can be automatically added to the full scan type query request, and a prompt information of the optimized query is output.

[0051] In some embodiments, for the type of composite type, for example, both "SYN protocol anomaly" and "query rejection rate overload risk" are met, the risk level is marked as the fourth level, and the anomaly type is attack query. In this case, the attack query countermeasure strategy can include both the processing strategy corresponding to the "SYN protocol anomaly" and the processing strategy corresponding to the "query rejection rate overload risk".

[0052] The cluster state feedback module 13 is configured to, after the execution of the first processing strategy, continuously acquire second multi-dimensional data of the Elasticsearch cluster, and if it is determined based on the currently acquired second multi-dimensional data that the execution result of the first processing strategy does not meet the standard, perform policy retry by expanding the control range based on the first processing strategy and keeping the preset core indicators in the first processing strategy unchanged according to a preset retry strategy.

[0053] The cluster state feedback module 13 is further configured to, when the execution result after the policy retry does not meet the standard, upgrade the policy level of the first processing strategy to obtain a second processing strategy, and execute the second processing strategy.

[0054] In some embodiments, the cluster state feedback module 13 can further include a real-time state monitoring module configured to, after the execution of the first processing strategy, continuously acquire second multi-dimensional data such as thread pool load, system resources, network traffic, query performance, and the like of the Elasticsearch cluster to provide data support for policy effect evaluation. The specific monitoring dimensions are shown in Table 2.

[0055] Table 2

[0056] In some embodiments, the cluster state feedback module is specifically configured to, after the execution of the first processing strategy, wait for a stable period of time (such as 5 minutes) to stabilize the cluster state area and avoid misjudgment caused by transient fluctuations. After the stable period of time, it is determined whether the second multi-dimensional data acquired in real time meets the standard condition, such as whether the queue usage rate needs to be reduced from 90% to 60%. If the standard condition is met, it is determined that the execution result of the first processing strategy meets the standard, the first processing strategy is marked as valid, and an observation period is entered for continuous monitoring. Otherwise, it is determined that the execution result of the first processing strategy does not meet the standard, the first processing strategy is marked as invalid, policy adjustment is triggered, and subsequent policy retry is performed.

[0057] Further, the specific process of policy retry includes performing policy retry by expanding the control range based on the first processing strategy and keeping the preset core indicators in the first processing strategy unchanged according to a preset retry strategy. By expanding the control range based on the original strategy and keeping the preset core indicators unchanged, it is beneficial to balance defense and business impact, wherein the preset core indicators are indicators in the first processing strategy that are strongly related to business.

[0058] For each processing strategy, a corresponding preset retry strategy can be configured in advance. In the execution of the above-mentioned strategy retry process, the preset retry strategy corresponding to the first processing strategy can stipulate the maximum number of retries, which adjustable control indicators, the adjustment logic of each retry (such as how much the control range corresponding to each adjustable control indicator is gradually expanded each time), the waiting time after each retry, and the like.

[0059] Exemplarily, assuming that the first processing strategy is a flow limiting related strategy, and the original strategy content of the first processing strategy is: limiting the flow of the first 10 high flow IPs. The preset retry strategy corresponding to the first processing strategy stipulates: a maximum of 3 retries, the adjustable control indicators include the number of high flow IPs for flow limiting, the adjustment logic of each retry is that the number of high flow IPs for flow limiting is increased by 10 each time, and the waiting time after each retry is 5 minutes. Based on this, the cluster state feedback module can first perform the first retry according to the preset retry strategy, and expand the flow limiting from the first 10 high flow IPs to the first 20, adjust and wait for 5 minutes, and then verify whether the expanded strategy is effective. If it is not effective, it is expanded to the first 30 and the first 40. A maximum of 3 retries, if the execution result is still not up to standard after 3 times, it is determined that the strategy level of the current processing strategy cannot match the load intensity. Based on this, the first processing strategy can be directly upgraded from the flow limiting strategy to a higher level of the fuse strategy, that is, the second processing strategy after the upgrade of the first processing strategy, and the execution of the fuse strategy is triggered.

[0060] In some embodiments, in order to avoid the risk of relapse after the strategy execution is temporarily up to standard, an observation period can be set, and during the observation period, the cluster state feedback module is also used to perform the following steps: Step S1, continuously collect the target indicator value associated with the thread pool pressure in at least one statistical time window, the statistical time length of each statistical time window is the same, and the statistical time points are continuous and non-overlapping. The target indicators associated with the thread pool pressure can include the thread pool rejection rate, the queue usage rate, and the cluster level QPS. The statistical time length of the statistical time window can be 30 seconds, for example, 10:00:00-10:00:30 is the first statistical time window, 10:00:30-10:01:00 is the second statistical time window, and so on. If the observation period is 10 minutes, then there are 20 corresponding statistical time windows.

[0061] In some embodiments, the target indicator values associated with thread pool pressure collected for each statistical time window can be pre-processed, including outlier filtering and normalization. The outlier filtering includes removing extreme values (e.g., 100% rejection rate) using the 3σ principle, and replacing them with the valid value of the previous statistical time window to avoid data distortion. The normalization includes mapping the indicator values to the 0-1 interval (e.g., 5% rejection rate → 0.05, 70% queue usage rate → 0.7) to eliminate dimension differences.

[0062] Step S2, the target indicator values corresponding to the target indicator acquired in each statistical time window are taken as data in the sliding prediction window, and the data in the sliding prediction window is periodically updated according to a preset time granularity.

[0063] Step S3, a data pair set of time and indicator values is generated based on the data in the sliding prediction window. For example, when steps S2 and S3 are performed, the data in the sliding prediction window can be updated every 30 seconds, the data corresponding to the oldest 1 statistical time window in the window is removed, the data corresponding to the latest 1 statistical time window is included, and a time-indicator value data pair set is generated, such as the time-indicator value data pair set corresponding to the target indicator "thread pool rejection rate" in the past 10 minutes: [(t20, 8%), (t19, 7%), …, (t1, 4%)].

[0064] Step S4, the trend of the target indicator changing over time is determined based on the data pair set.

[0065] Step S5, the predicted value of the target indicator corresponding to the future time T is determined according to the trend.

[0066] In some embodiments, the above data pair set corresponding to the target indicator can be subjected to exponential smoothing processing to fit a trend equation of the target indicator changing over time. The purpose of exponential smoothing processing is to eliminate data fluctuations and focus on the core trend. Exponential smoothing processing can be performed according to the following formula: .

[0067] wherein, represents the smoothed value of the i-th statistical time window, a is a weight coefficient, for example, 0.7, is the original value of the i-th statistical time window, is the smoothed value of the previous statistical time window of the i-th statistical time window. Still taking the above example of the time-indicator value data pair set corresponding to the target indicator "thread pool rejection rate" in the past 10 minutes: [(t20, 8%), (t19, 7%), …, (t1, 4%)]), there are a total of 20 time windows, and assuming i is 20, then = 8%, The smoothed value of the 19th statistical time window.

[0068] Further, the data pair set after exponential smoothing can be linearly fitted with time as the independent variable and the smoothed value as the dependent variable to obtain a trend equation y = kx + b. x is time, y is the smoothed value, and k and b are known. The predicted value of the target indicator in each statistical time window can be predicted based on the trend equation at a future time T. For example, T is 10 minutes in the future, and the predicted value of the target indicator in the next 20 statistical time windows can be calculated based on the trend equation.

[0069] Step S6: Determine the risk level and the target processing strategy matching the risk level according to the predicted value, and execute the target processing strategy.

[0070] In some embodiments, the risk level determination criteria and the processing strategy corresponding to each risk level are set in advance. When step S6 is executed, the risk level can be determined in combination with the risk level determination criteria and the predicted value, and the processing strategy corresponding to the risk level can be determined and executed based on the previously set correspondence between the risk level and the processing strategy. For example, the risk level determination criteria include: if the predicted value of the thread pool rejection rate is less than 3%, it is determined to be low risk, and the target processing strategy corresponding to low risk is to maintain the current strategy. In this case, when step S6 is executed, if it is determined that the predicted value of the thread pool rejection rate is less than 3%, the risk level is determined to be low risk, and the target processing strategy corresponding to low risk is to maintain the current strategy. Or if it is medium risk, the target processing strategy corresponding to medium risk is mainly to fine-tune the current strategy and enhance defense. For example, if the current strategy is a flow limiting strategy, the target processing strategy corresponding to high risk can be to reduce the QPS threshold by 10%. Or if it is high risk, the target processing strategy corresponding to high risk is mainly to upgrade the strategy level of the current strategy. For example, if the current strategy is a flow limiting strategy, the target processing strategy corresponding to high risk can be to directly upgrade the flow limiting to level 1 fuse.

[0071] In some embodiments, the cluster state feedback module is further configured to, if it is determined that the exception is alleviated based on the second multi-dimensional data currently acquired, execute a first rollback strategy corresponding to the first processing strategy, the first rollback strategy being to gradually restore a default configuration of an index involved in adjustment during execution of the first processing strategy, release a redundant resource involved in adjustment, and avoid resource waste. The exception alleviation refers to that the multi-dimensional data meets the above threshold condition, such as that the queue usage rate needs to be reduced from 90% to 60%. The gradual restoration of the index involved in adjustment during execution of the first processing strategy to the default configuration may, for example, be: 1. cancel the expanded IP flow limiting range, such as reducing from banning 30 IPs to only keeping the top 5 high-risk IPs; 2. relax the non-core interface QPS threshold, such as restoring the _ / bulk interface from 1500 QPS to 2000 QPS; and 3. cancel the non-core index reading limitation, and only keep a slight throttling of high-overhead queries, such as increasing the wildcard query release ratio from 0 to 50%.

[0072] In some embodiments, the release of the redundant resource involved in adjustment may, for example, be: 1. removing the low-priority label of the thread pool, and recycling the idle threads, such as the low-priority threads occupied by the preheating task; and 2. releasing the heap memory temporarily expanded during execution of the strategy, such as restoring from 32 GB to 16 GB.

[0073] In some embodiments, the cluster state feedback module is further configured to, if it is determined that the strategy is over-defended based on the second multi-dimensional data currently acquired, shrink the number of threads and the queue capacity according to a second rollback strategy. The specific process of determining that the strategy is over-defended may be: calculating the following redundant dimensions based on the second multi-dimensional data currently acquired: redundant thread number = current core thread number - (average active thread number in a preset time window × preset active coefficient); and redundant queue capacity = current queue capacity - (average queue length in a preset time window × preset queue coefficient). The preset time window may be, for example, 10 minutes, the preset active coefficient may be, for example, 1.2, and the preset queue coefficient may be, for example, 1.5. When the redundant thread number or the redundant queue capacity is greater than 0, it is determined that there is a redundant resource that can be shrunk in the thread pool, and it is accordingly determined that the strategy is over-defended.

[0074] Further, after determining that the strategy is over-defended, the number of threads and the queue capacity are shrunk according to the second rollback strategy, which may include a step-by-step shrinking strategy in a preset number of shrinking times and a preset time interval, and the process of shrinking the number of threads and the queue capacity according to the second rollback strategy may be: performing a shrinking operation on the determined redundant resource by using the step-by-step shrinking strategy in the preset number of shrinking times and the preset time interval.

[0075] Exemplarily, the process of performing the shrink operation in the step-by-step shrink strategy according to the preset shrink times and the preset time intervals can include: step 1, first shrink: performing the first shrink on the core thread number and the queue capacity of the thread pool based on the preset proportion of the redundant resources, and monitoring the running state of the thread pool after the shrink; step 2, after the preset time interval, verifying that the first shrink is normal, and performing the second shrink on the remaining redundant resources. Further, after completing all batches of shrink, verifying whether the active thread number, the queue length, the task rejection rate and the like of the thread pool are in the normal range, and confirming that the shrink operation does not cause the cluster load to be abnormal.

[0076] For example, the preset shrink times are 2, and the preset time interval is 5 minutes, and the process of performing the shrink operation in the step-by-step shrink strategy according to the preset shrink times and the preset time intervals can be as follows: First shrink (0-5 minutes): Thread number: shrink by 50% of the redundant thread number, for example, if there are 8 redundant threads, then 4 threads are shrunk, and the core thread number is from 20 to 16; Queue capacity: shrink by 60% of the redundant capacity, for example, if there are 400 redundant capacities, then 240 capacities are shrunk, and the queue capacity is from 1200 to 960; Priority adjustment: the core request queue proportion is maintained at 30% to avoid the influence of shrink on the core business.

[0077] Second shrink (5-10 minutes): If there are still redundant resources, the remaining redundant resources are shrunk, for example, the core thread number is from 16 to 14, and the queue capacity is from 960 to 900; The dynamic expansion function of the thread pool is closed to avoid generating redundant threads automatically.

[0078] In some embodiments, the cluster state feedback module 13 is further configured to, if it is determined based on the second multi-dimensional data currently acquired that the periodic traffic falls back (for example, the network traffic in the early morning falls to 50% of the baseline), retreat to a low-load configuration according to a third retreat strategy, and reduce the energy consumption of the cluster.

[0079] The low-load configuration includes: 1, thread pool configuration: the core thread number is reduced to 80% of the baseline, for example, from 16 to 12, and the maximum thread number is reduced to 70% of the baseline, for example, from 32 to 22; the idle thread keep-alive function of the thread pool is closed, and the idle threads are automatically recycled after more than 1 minute; 2, index configuration: reducing the shard refresh frequency of non-core indexes, for example, from 1 second to 5 seconds, and reducing the disk IO energy consumption; closing the query cache of non-core indexes to release memory resources; 3, network configuration: reducing the heartbeat detection frequency between nodes, for example, from 1 second to 3 seconds, and reducing the network interaction energy consumption.

[0080] As a feasible way, the cluster state feedback module 13 is further used to perform the following energy consumption optimization reinforcement operation after determining the periodic traffic fallback: closing real-time monitoring sampling of the cluster, such as from 1 second / time to 10 seconds / time, reducing CPU computing amount; performing "fragment merging + hibernation" on cold index, reducing disk seek times; limiting CPU main frequency of non-core nodes, such as from 2.8GHz to 2.0GHz, directly reducing hardware energy consumption.

[0081] As a feasible way, the cluster state feedback module 13 is further used to automatically trigger thread pool parameter recovery to the baseline 90% and restart query cache, recover fragment flushing frequency at a preset time point (such as 5:03 in the morning) before the next periodic traffic fallback after determining the periodic traffic fallback, to avoid triggering index reconstruction when the traffic rises.

[0082] In some embodiments, the cluster state feedback module 13 further includes a strategy experience library module, which is mainly used for establishment of a strategy experience library, to improve subsequent abnormal response efficiency, establish an optimal configuration pre-strategy experience library, and provide learning troubleshooting needs for subsequent related personnel. The strategy experience library module is specifically used for recording and storing related information of the current strategy: 1, the abnormal type, occurrence time and duration of the current strategy corresponding to the abnormality; 2, the strategy or combination of strategies that take effect this time, such as the combination of queue expansion strategy, high-overhead query strategy and priority queue strategy; 3, the change of the key index this time, such as the change of queue usage rate from 95% to 40% and the change of response time from 1000ms to 300ms.

[0083] The thread pool protection system provided in the application realizes the following through the functions of each module in the system: 1, multi-dimensional correlation detection: fusion of traffic characteristics, thread pool state, system resource indicators, accurate abnormality identification through composite rules; 2, hierarchical protection strategy: construction of a protection mechanism of "early warning → flow limiting → fuse → recovery", gradual enhancement of defense according to the abnormality level, and guarantee of key business continuity. Compared with the traditional thread pool protection mode, the following advantages exist: 1. The time for fault handling is greatly shortened Traditional mode pain point: when an abnormality occurs, the operation and maintenance personnel need to go through the process of "log troubleshooting → indicator analysis → thread pool parameter tuning → service restart", and the average time for single fault handling is 4-6 hours. Through the "real-time monitoring-automatic detection-strategy execution-effect verification" closed loop, the processing period is compressed to 5-10 minutes.

[0084] 2. The cost of manual intervention is greatly reduced

[0085] Through the application, thread pool parameter adjustment is automatically completed without the need for manual execution of commands.

[0086] It should be noted that although this application describes operations in a specific order, this should not be construed as requiring these operations to be performed in the specific order or sequence shown, or to perform all of the shown operations to obtain the desired result. In some cases, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the foregoing discussion, these details should not be construed as limiting the scope of this disclosure, but rather as descriptions of features specific to particular embodiments. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0087] Although this disclosure has been described in language specific to structural features and / or methodological behavior, it should be understood that this disclosure as defined in the appended claims is not necessarily limited to the specific features or behaviors described above. Rather, the specific features and actions described above are disclosed as exemplary forms for implementing the claims.

Claims

1. A thread pool protection system for an Elasticsearch cluster, characterized in that, The thread pool protection system includes: The traffic monitoring module is used to collect the first multi-dimensional data of the Elasticsearch cluster, which includes any one or more of the following: network traffic, thread pool status data, and host system resources. Anomaly detection module is used to identify the anomaly type corresponding to the first multi-dimensional data according to anomaly detection strategy; The strategy decision execution module is used to acquire and execute a first processing strategy that matches the exception type. The cluster status feedback module is used to continuously acquire the second multi-dimensional data of the Elasticsearch cluster after the first processing strategy is executed. If it is determined based on the currently acquired second multi-dimensional data that the execution result of the first processing strategy has not met the standard, the strategy is retried by expanding the control scope on the basis of the first processing strategy and keeping the preset core indicators in the first processing strategy unchanged. The preset core indicators are indicators related to the business. The cluster status feedback module is also used to upgrade the policy level of the first processing policy to obtain a second processing policy and execute the second processing policy when the execution result after the policy retry fails to meet the standard.

2. The thread pool protection system as described in claim 1, characterized in that, The cluster status feedback module is also used for: The target metrics associated with thread pool pressure are continuously collected within at least one statistical time window; the statistical duration of each statistical time window is the same, and the statistical time points are continuous and non-overlapping. The index value corresponding to the target index obtained under each statistical time window is used as the data in the sliding prediction window, and the sliding prediction window updates the data in the sliding prediction window periodically according to a preset time granularity. A set of data pairs between data generation time and indicator values ​​is generated based on the data in the sliding prediction window; Based on the data set, determine the trend of the target indicator over time; Based on the trend, determine the predicted value of the target indicator corresponding to the future time T; The risk level and the target treatment strategy matching the risk level are determined based on the predicted value, and the target treatment strategy is executed.

3. The thread pool protection system as described in claim 1, characterized in that, The cluster status feedback module is further configured to execute a first rollback strategy corresponding to the first processing strategy if the anomaly mitigation is determined based on the currently acquired second multi-dimensional data. The first rollback strategy is to gradually restore the indicators involved in the adjustment during the execution of the first processing strategy to their default configurations and release the redundant resources involved in the adjustment.

4. The thread pool protection system as described in claim 1, characterized in that, The cluster status feedback module is also used to reduce the number of threads and queue capacity according to the second rollback strategy if it is determined that the strategy is over-defended based on the currently acquired second multi-dimensional data.

5. The thread pool protection system as described in claim 1, characterized in that, The cluster status feedback module is also used to roll back to a low-load configuration according to the third rollback strategy if it is determined that periodic traffic drop is occurring based on the currently acquired second multi-dimensional data.

6. The thread pool protection system as described in claim 1, characterized in that, The strategy decision execution module is also used for: When the detected anomaly type is a target anomaly type related to traffic overload, Identify the hot indexes of the Elasticsearch cluster and the cold shards within those hot indexes; A preheating operation is performed on the cold-sliced ​​section.

7. The thread pool protection system as described in claim 6, characterized in that, The preheating operation for the cold slices includes: Configure the preheating request parameters for the cold shard, and transmit the preheating request carrying the preheating request parameters to the cluster index shard preheating interface. The preheating request is marked as low priority. The preheating request is used to trigger the Elasticsearch cluster to perform metadata loading and inverted index loading operations from disk to memory on the cold shard, and return the execution results. When an execution result indicating successful execution is detected, the preheating of the cold slice is confirmed to be complete.

8. The thread pool protection system as described in claim 6, characterized in that, The cold-slicing unit comprises multiple slices, and the preheating operation on the cold-slicing unit includes: A batch-based, low-priority strategy is adopted to perform preheating operations on the multiple cold shards by calling the cluster index shard preheating interface. The number of cold shards preheated in each batch does not exceed a preset threshold, a preset time interval is set between batches, and the preheating tasks are marked as low priority to avoid preempting core business thread resources.

9. The thread pool protection system as described in claim 7, characterized in that, The strategy decision execution module is also used to: after the preheating of the cold shard is completed, call the cluster status monitoring interface to verify the memory loading status and thread pool running status of the cold shard, and confirm that the preheating is effective and has not caused any thread pool abnormalities.

10. The thread pool protection system as described in claim 9, characterized in that, The memory loading status of the cold partition includes the memory usage of the inverted index of the cold partition. The strategy decision execution module is also used to confirm that the preheating is effective when the memory usage of the inverted index of the cold partition is updated from 0 to a stable value.

Images