Log processing method and system based on elk and readable storage medium

Abstract

The present application relates to the technical field of big data, and particularly relates to a log processing method and system based on ELK and a readable storage medium, the method comprising: acquiring a response duration of each ES node for log processing; if the response duration of any ES node exceeds a set first duration threshold, then adjusting the number of logs processed by each ES node so as to balance the response duration of each ES node; if it is determined that the response duration of any balanced ES node exceeds a set second duration threshold, then determining a first capacity difference value based on the difference between the balanced response duration of each ES node and the second duration threshold; determining a newly added ES node based on the first capacity difference value, and adding the newly added ES node; the present application can meet the response duration requirement and provide high-concurrency data processing capability on the premise of fully utilizing the processing performance of each ES node.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of big data processing, and particularly relates to an ELK-based log processing method and system and a readable storage medium. BACKGROUND

[0002] Enterprise log data is a valuable asset that can help enterprises discover business anomalies, ensure data security, and improve system performance and user experience. However, due to the large amount of data, there are high requirements for processing and analysis capabilities. In response to high-concurrency processing of large amounts of data, response latency directly affects user experience. Although a large number of servers can be configured to improve processing performance, this will result in increased costs.

[0003] Therefore, how to fully exploit the processing performance of servers to meet the application scenarios of high concurrency and low latency has become a problem to be solved. SUMMARY

[0004] The application aims to provide an ELK-based log processing method and system and a readable storage medium, which can meet the response time requirement and provide high-concurrency data processing capability under the premise of fully utilizing the processing performance of each ES node.

[0005] To achieve the above-mentioned purpose, the application provides the following technical solutions:

[0006] In one aspect, the application provides an ELK-based log processing method, which comprises the following steps:

[0007] S100, acquiring the response time of each ES node in an ElasticSearch cluster for log processing;

[0008] S200, if the response time of any ES node exceeds a set first time threshold, then the number of logs processed by each ES node in the ElasticSearch cluster is adjusted to balance the response time of each ES node;

[0009] S300, if it is determined that the response time of any balanced ES node exceeds a set second time threshold, then a first capacity difference value is determined based on the difference between the balanced response time of each ES node and the second time threshold; wherein the second time threshold is less than the first time threshold;

[0010] S400, determining a new ES node based on the first capacity difference value, and adding the new ES node to the ElasticSearch cluster.

[0011] Further, after S200, the method further comprises:

[0012] S500, determining whether the response duration of the balanced ES nodes is lower than a third duration threshold, and if so, determining a second capacity difference value based on a difference between the response duration of each balanced ES node and the third duration threshold; wherein the third duration threshold is less than the second duration threshold;

[0013] S600, determining a capacity-reduced ES node based on the second capacity difference value, and removing the capacity-reduced ES node from the ElasticSearch cluster.

[0014] Further, in S200, if the response duration of any ES node exceeds a first duration threshold, the number of logs processed by each ES node in the ElasticSearch cluster is adjusted to balance the response duration of each ES node, including:

[0015] S201, obtaining the response duration of each ES node in a current unit of time;

[0016] S202, determining whether the variance of the response duration of each ES node in the current unit of time exceeds a first variance threshold; if so, performing S203, otherwise performing S208;

[0017] S203, determining the duration mean of the response duration of all ES nodes in the current unit of time, and taking the ES nodes in the ElasticSearch cluster that are less than the duration mean as capacity-increased nodes and taking the ES nodes that are greater than the duration mean as capacity-reduced nodes;

[0018] S204, taking the capacity-reduced nodes and the capacity-increased nodes as to-be-allocated nodes, and determining the weight of each to-be-allocated node based on its response duration in a plurality of recent units of time; wherein the weight of the capacity-reduced node is positively correlated with the response duration, and the weight of the capacity-increased node is inversely correlated with the response duration;

[0019] S205, obtaining the processing performance of the to-be-allocated nodes, multiplying the difference between the response duration of the to-be-allocated node in the current unit of time and the duration mean by the processing performance as the maximum capacity of the to-be-allocated node, and determining the allocation capacity of the to-be-allocated node based on the maximum capacity and the weight of the to-be-allocated node; wherein the processing performance is the number of logs that can be processed in a unit of time; the allocation capacity of the capacity-reduced node is a capacity-reduced capacity, and the allocation capacity of the capacity-increased node is a capacity-increased capacity;

[0020] S206, determining the capacity-increased node corresponding to each capacity-reduced node based on the capacity-reduced capacity of each capacity-reduced node and the capacity-increased capacity of each capacity-increased node;

[0021] S207, determining a to-be-allocated log based on the decompression capacity of the decompression node, distributing the to-be-allocated log to the corresponding compression node, and performing S201 after delaying at least one unit time;

[0022] S208, determining to complete balancing of the number of logs processed by each ES node in the ElasticSearch cluster.

[0023] Further, in S204, the weight of each to-be-allocated node is determined based on the response duration of the to-be-allocated node in the recent plurality of unit times, comprising:

[0024] S241, obtaining the response duration of the to-be-allocated node in the recent plurality of unit times;

[0025] S242, determining a change trend parameter of the to-be-allocated node in the current unit time based on the response duration of the to-be-allocated node in the recent plurality of unit times; wherein the change trend parameter is used to represent the change trend of the response duration of the to-be-allocated node in the recent plurality of unit times, the change trend parameter of the decompression node and the change trend of the response duration are positively correlated, and the change trend parameter of the compression node and the change trend of the response duration are inversely correlated;

[0026] S243, taking the ratio of the response duration of the to-be-allocated node in the current unit time to the sum of the response durations of all to-be-allocated nodes in the current unit time, taking the logarithm, and multiplying the change trend parameter of the to-be-allocated node in the current unit time to obtain the initial weight of the to-be-allocated node in the current unit time;

[0027] S244, multiplying the ratio of the initial weight of the to-be-allocated node in the current unit time to the initial weight of all to-be-allocated nodes in the current unit time by a normalization coefficient to obtain the weight of the to-be-allocated node in the current unit time.

[0028] Further, in S242, the change trend parameter of the to-be-allocated node in the current unit time is determined based on the response duration of the to-be-allocated node in the recent plurality of unit times, comprising:

[0029] Obtaining the response duration of the to-be-allocated node in the recent K unit times;

[0030] Taking the response duration in K unit times as a sliding window with a duration of k unit times to obtain the last two sliding windows; wherein k

[0031] Determining the average duration of the last two sliding windows based on the k response durations contained in the sliding window;

[0032] The change trend parameter of the to-be-allocated node in the current unit time is determined based on an average time length of the to-be-allocated node in the last two sliding windows.

[0033] Further, in S206, the adding capacity node corresponding to each reducing capacity node is determined based on the reducing capacity of each reducing capacity node and the adding capacity of each adding capacity node, and the determining includes:

[0034] S261, each reducing capacity node is sorted in descending order of the reducing capacity, and each adding capacity node is sorted in descending order of the adding capacity;

[0035] S262, one reducing capacity node is taken out in sequence, each adding capacity node is polled in sequence, the adding capacity with the minimum deviation from the reducing capacity of the reducing capacity node is determined, the reducing capacity is less than the maximum capacity corresponding to the adding capacity, the adding capacity node corresponding to the adding capacity is matched with the reducing capacity node, and the reducing capacity of the reducing capacity node is updated by subtracting the corresponding adding capacity, and the adding capacity of the corresponding adding capacity node is updated by subtracting the corresponding reducing capacity;

[0036] S263, it is determined whether there is an unmatched reducing capacity node, if yes, S262 is executed, otherwise, S264 is executed;

[0037] S264, it is determined whether the variance of the reducing capacity of each reducing capacity node and the adding capacity of each adding capacity node is greater than a set capacity deviation threshold, if yes, S262 is executed, otherwise, the process is ended.

[0038] Further, in S300, the first capacity difference is determined based on the difference between the balanced response time length of each ES node and the second time length threshold, and the determining includes:

[0039] The processing performance of each ES node is obtained, the difference between the balanced response time length of the ES node and the second time length threshold is multiplied by the corresponding processing performance as the maximum capacity of the ES node, and the sum of the maximum capacities of each ES node is taken as the first capacity difference.

[0040] In a second aspect, an embodiment of the present application provides a log processing system based on ELK, and the system includes:

[0041] At least one processor;

[0042] At least one memory for storing at least one program;

[0043] When the at least one program is executed by the at least one processor, the at least one processor implements the log processing method based on ELK as any one of the first aspect.

[0044] In a third aspect, the embodiments of the present application further provide a computer readable storage medium storing computer executable instructions for executing the ELK-based log processing method according to any one of the first aspect.

[0045] The present application has the beneficial effects that: the present application discloses an ELK-based log processing method, system and readable storage medium, starting from the result of the response time length of the ES node processing abnormal behavior detection, determining the equalization degree of each ES node based on the response time length, so as to more truly, accurately and directly reflect the processing performance of each ES node, and then based on the response time length, undoing the number of logs newly assigned to each ES node, and constantly correcting the number of logs assigned to each ES node, so as to balance the response time length of each ES node, improve the processing efficiency of the ElasticSearch cluster, and shorten the response time length. In the case that the response time length of the balanced ES node still exceeds the second time threshold, the first capacity difference is made up by the newly added ES node; under the premise of fully utilizing the processing performance of each ES node, the response time length is ensured to have a certain redundancy, thereby providing high-concurrent data processing capability. BRIEF DESCRIPTION OF DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0047] Figure 1 is a flow diagram of the ELK-based log processing method in the embodiments of the present application;

[0048] Figure 2 is a structural diagram of the ELK-based log processing system in the embodiments of the present application. DETAILED DESCRIPTION

[0049] The concept, specific structure and generated technical effects of the present application will be described clearly and completely in combination with the embodiments and drawings, so as to fully understand the purpose, scheme and effects of the present application. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0050] In the related art, the way of load balancing of the ES node is based on the characteristics of the ES node itself, such as balancing from the storage space and fragmentation degree of the ES node itself, however, since the hardware performance of each ES node (i.e. server) is not the same, even if some parameters of the ES node itself are balanced, since the response time of the ES node for processing abnormal behavior detection is affected by various factors, it is also difficult to ensure that the final response time reaches balance.

[0051] It should be noted that the ELK system provided by the present application supports distributed collection and storage of operation behavior logs, device running logs, Internet access behavior logs, maintenance record logs and other logs of each business system. In the ELK system, the Logstash cluster is mainly responsible for log collection, data formatting and conversion, the Kibana cluster mainly provides data visualization and interactive analysis functions, and the ElasticSearch cluster is a distributed document database, which provides data warehouse and index services.

[0052] Referring to Figure 1 , Figure 1 is a flowchart of a log processing method based on ELK provided by the present application, the method comprising the following steps:

[0053] S100, obtaining the response time of each ES node in the ElasticSearch cluster for log processing;

[0054] It should be noted that the log processing includes but is not limited to log query, abnormal behavior detection based on log, etc., since the ElasticSearch cluster performs log processing, it is generally realized through the cooperation of multiple ES nodes. In an ElasticSearch cluster, data is distributed and stored on multiple ES nodes, each ES node processes the part of data it is responsible for and returns the result to the ES node for data aggregation. When performing log processing, multiple ES nodes also need to cooperate for processing, and an ES node can only access part of the data, so it is necessary to summarize the results processed by each ES node to obtain the final log processing result.

[0055] Among them, the response time is the time length from receiving the log processing request to feeding back the log processing result of the ES node; for example, the ES node in the ElasticSearch cluster responds to the received data processing request for abnormal behavior detection, performs clustering analysis based on the original log stored in the ES node, and feeds back the obtained clustering analysis result; the task processing time of the ES node for processing the data processing request is obtained by recording the time of receiving the data processing request and the time of feeding back the clustering analysis result of the ES node, and is recorded as the response time.

[0056] It should be noted that the ES nodes in the ElasticSearch cluster have multiple role attributes, and the ES nodes in the embodiment are data nodes responsible for log storage and query service.

[0057] In S200, if the response time of any ES node exceeds the set first time threshold, the number of logs processed by each ES node in the ElasticSearch cluster is adjusted to balance the response time of each ES node.

[0058] It should be noted that the first time threshold is the maximum response delay that the user can tolerate minus the time required for data merging, wherein the response delay is the total time from the user triggering the request to receiving the feedback result. The response time cannot exceed the first time threshold, otherwise it will cause the response delay to be too long, which will affect the business efficiency of the user and further affect the user experience. The first time threshold in the embodiment is pre-set, for example, 1 second to 6 seconds.

[0059] In the embodiment, since different ES nodes have different response times, if the response time of some ES nodes exceeds the first time threshold, the number of logs received by each ES node is first balanced to reduce the number of logs received by the ES node whose response time exceeds the first time threshold, so that the response time of the ES node is reduced. The response times of the ES nodes are balanced in order to balance the number of logs and make the response times of the ES nodes meet the requirements of the first time threshold. Even if the response times of all ES nodes exceed the first time threshold, the overall response time of the ElasticSearch cluster can be reduced.

[0060] In S300, if it is determined that the response time of any balanced ES node exceeds the set second time threshold, a first capacity difference value is determined based on the difference between the balanced response time of each ES node and the second time threshold, wherein the second time threshold is less than the first time threshold.

[0061] It should be noted that after balancing the number of logs received by each ES node, in addition to ensuring that the response time of the ES node is less than the first time threshold, the response time of the ES node is also ensured to be less than the second time threshold, so as to leave a certain redundancy space for the response time of the ES node, so as to avoid the case that the ES node exceeds the first time threshold again after running for a period of time. Therefore, in the embodiment, if the response time of the ES node still exceeds the second time threshold, the ElasticSearch cluster needs to be expanded, so as to avoid frequent balancing of the ES node and occupation of computing resources. In the embodiment, the first capacity difference value is reflected by the difference between the response time and the second time threshold, and the response time of the ES node can be reduced by compensating for the first capacity difference value.

[0062] S400, based on the first capacity difference value, determine a new ES node, and add the new ES node to the ElasticSearch cluster.

[0063] In this step, the ES node adapted to the first capacity difference value is determined as a new ES node, and the first capacity difference value is compensated by the new ES node. The number of new ES nodes is determined according to the first capacity difference value and the processing performance of the ES node, so as to ensure that the sum of the processing performance of the new ES nodes meets the first capacity difference value.

[0064] The present application adopts a technical idea opposite to the related art, starts from the response time of the ES node processing abnormal behavior detection, determines the balancing degree of each ES node based on the response time, so as to more truly, accurately and directly reflect the processing performance of each ES node, and then inversely allocates the number of logs to each ES node based on the response time, and continuously corrects the number of logs allocated to each ES node, so as to balance the response time of each ES node; improve the processing efficiency of the ElasticSearch cluster and shorten the response time. In the case that the response time of the balanced ES node still exceeds the second time threshold, the first capacity difference value is compensated by the new ES node; under the premise of fully utilizing the processing performance of each ES node, the response time has a certain redundancy, so as to provide high-concurrent data processing capability.

[0065] In an embodiment, after S200, the method further comprises:

[0066] S500, determine whether the response time of the balanced ES node is lower than a third time threshold, if yes, determine a second capacity difference value based on the difference between the response time of each balanced ES node and the third time threshold; wherein the third time threshold is less than the second time threshold;

[0067] S600, determining a capacity-reduced ES node based on the second capacity difference value, and removing the capacity-reduced ES node from the ElasticSearch cluster.

[0068] It should be noted that if the response time is less than the third time threshold, it means that the response time is extremely short, and there is sufficient time redundancy, so it is inferred that the idle degree of the ES node is too high, and the problem of excessively long response time will not occur in the short term. The ES node can be removed from the ElasticSearch cluster for use by other business systems. By flexibly allocating the ES node, unnecessary hardware resource idling is reduced, and the utilization rate of the ES node is improved.

[0069] The embodiments provided by the application can expand or reduce the ES nodes of the ElasticSearch cluster as needed, and improve the utilization rate of the ES nodes by flexible scheduling on the premise that the response time meets the requirements.

[0070] In an embodiment, in S200, if the response time of any ES node exceeds the first time threshold, the number of logs processed by each ES node in the ElasticSearch cluster is adjusted to balance the response time of each ES node, including:

[0071] S201, obtaining the response time of each ES node in a current unit time;

[0072] In this embodiment, a standard task processing time is determined, i.e., how long a single task needs to be processed on average, as a unit time. The length of the unit time is set according to actual conditions, and the ES node can complete a log processing in a unit time; the response time of the ES node is queried every unit time, and the response time of the ES node in the current unit time obtained can reflect the latest situation of the ES node.

[0073] S202, determining whether the variance of the response time of each ES node in the current unit time exceeds the first variance threshold; if yes, performing S203, otherwise performing S208;

[0074] In this embodiment, after obtaining the response time of each ES node in the current unit time, the variance of the response time of each ES node is first calculated. If the variance exceeds the first variance threshold, it means that the response time of each ES node is unbalanced, and the subsequent steps are performed to change the number of newly added logs allocated to each ES node, so as to balance the response time. Otherwise, it means that the response time of each ES node is balanced, and there is no need to change the number of newly added logs allocated to each ES node. It should be noted that the first variance threshold is set in advance according to actual conditions, for example, the value range of the first variance threshold is set to 1 millisecond to 2 seconds.

[0075] S203, determine the mean value of the response time length of all ES nodes in the current unit time, take the ES nodes less than the mean value as the capacity-increasing nodes, and take the ES nodes greater than the mean value as the capacity-decreasing nodes;

[0076] In this embodiment, the mean value is the average or median of the response time length of all ES nodes. If the response time length of a certain ES node is exactly equal to the mean value, it is temporarily not processed.

[0077] S204, take the capacity-decreasing nodes and the capacity-increasing nodes as to-be-allocated nodes, and determine the weight of each to-be-allocated node based on the response time length of the to-be-allocated node in the recent multiple unit times; wherein the weight of the capacity-decreasing node is positively correlated with the response time length, and the weight of the capacity-increasing node is inversely correlated with the response time length;

[0078] It should be noted that if the response time length of the capacity-decreasing node increases with the recent multiple unit times, it indicates that the number of logs processed by the capacity-decreasing node in the past shows a growing trend. By dynamically adjusting the decompression weight of the capacity-decreasing node, the decompression weight will be higher and higher, so that the logs allocated from the capacity-decreasing node to each capacity-increasing node will be continuously increased in proportion to the decompression weight. By increasing the logs allocated from the capacity-decreasing node to the capacity-increasing node, the logs processed by the capacity-decreasing node are balanced. Similarly, if the response time length of the capacity-increasing node increases with the recent multiple unit times, it indicates that the number of logs processed by the capacity-increasing node in the past shows a growing trend. By dynamically adjusting the compression weight of the capacity-increasing node, the compression weight will be lower and lower, so that the logs allocated from the capacity-decreasing node to the capacity-increasing node will be continuously reduced in proportion to the compression weight. By reducing the logs allocated from the capacity-decreasing node to the capacity-increasing node, the logs processed by the capacity-increasing node are balanced. In this embodiment, the logs processed by the to-be-allocated nodes are reasonably allocated according to the change trend, and the actual number of logs processed by each to-be-allocated node is flexibly adjusted, so that each to-be-allocated node smoothly reaches load balancing.

[0079] S205, obtain the processing performance of the to-be-allocated node, multiply the difference between the response time length of the to-be-allocated node in the current unit time and the mean value by the processing performance to obtain the maximum capacity of the to-be-allocated node, and determine the allocation capacity of the to-be-allocated node based on the maximum capacity and the weight of the to-be-allocated node; wherein the processing performance is the number of logs that can be processed in a unit time; the allocation capacity of the capacity-decreasing node is the decompression capacity, and the allocation capacity of the capacity-increasing node is the compression capacity;

[0080] Specifically, the processing performance of the capacity-decreasing node is obtained, and the allocation capacity of the capacity-decreasing node is determined based on the formula cap i,t =(x i,t -x avg,t )P i w i,tcalculating the decompression capacity of the i th decompression node in the current unit time; wherein, cap i,t is the decompression capacity of the i th decompression node in the current unit time, x i, is the response duration of the i th decompression node in the current unit time, x avg, is the average duration of the current unit time, P i is the processing performance of the i th decompression node, w i,t is the decompression weight of the i th decompression node in the current unit time;

[0081] obtaining the processing performance of the expansion node, based on the formula cap j,t = (x j,t -x avg,t ) P j w j,t calculating the expansion capacity of the j th expansion node in the current unit time; wherein, cap j,t is the expansion capacity of the j th expansion node in the current unit time, x j, is the response duration of the j th expansion node in the current unit time, P i is the processing performance of the j th expansion node, w j,t is the expansion weight of the j th expansion node in the current unit time;

[0082] In some embodiments, the processing performance of the ES node, i.e. the number of logs that the ES node can process per unit time, such as the number of logs that can be processed per second, is estimated by obtaining the historical performance data of the ES node. In this embodiment, the decompression capacity reflects the amount of logs that exceeds the processing performance of the ES node, and after the decompression node is selected, the corresponding number of logs is taken out to decompress the decompression node.

[0083] The log scheduling in the related art usually adopts a simple peak clipping and valley flattening manner, directly spreading the logs of the ES node with the maximum load and the ES node with the minimum load, but this manner does not consider the trend change, and in the case of a large number of logs, it is easy to cause the ES node to fluctuate sharply, and the running risk is too high; this embodiment is not directly (x i,t -x avg,t ) P iAs the decompression capacity is fully allocated, but the trend change is considered, even if a certain ES node has a smaller current load, but due to the higher speed of adding in the subsequent log, it will cause the log processing load of itself to increase faster, therefore, the embodiment adopts a dynamic way for load balancing, which is not simply limited to the response time of the current unit time, but also considers the time change of the subsequent ES node itself, combined with the actual situation of business processing, the log in the decompression node is proportionally decompressed and scheduled according to the decompression weight of the decompression node, the decompression weight can make the number of logs taken out of the decompression node more reasonable, based on the constantly updated weight, the number of logs received by each capacity node is determined, and the to-be-allocated logs are allocated to each capacity node, and the actual number of logs processed by each ES node is flexibly adjusted according to the actual situation, so as to avoid large business fluctuations; similarly, through the decompression weight, the number of logs allocated to the capacity node is more reasonable, so as to more targetedly balance the load of each ES node, and finally achieve the effect of accelerating the balancing efficiency of the ES node under the premise of avoiding large fluctuations of the ES node. Realize the smooth and efficient load balancing of ElasticSearch cluster.

[0084] S206, determining the capacity node corresponding to each decompression node based on the decompression capacity of each decompression node and the decompression capacity of each capacity node;

[0085] S207, determining the to-be-allocated log based on the decompression capacity of the decompression node, distributing the to-be-allocated log to the corresponding capacity node, and executing S201 after delaying at least one unit time;

[0086] In the embodiment, a corresponding number of logs are first taken out of the decompression node, and then the taken logs are allocated to the corresponding capacity node, the decompression and decompression are combined to improve the efficiency of log allocation, and the decompression weight and the decompression weight are used to reflect the response time of the ES node in the past multiple unit times, so as to finely adjust the number of allocated logs, more accurately and efficiently adapt to the balancing of the ES node, and improve the balancing efficiency of the ES node.

[0087] It should be noted that the Logstash cluster collects original logs, and after data formatting and conversion and other preprocessing of the original logs, the preprocessed logs are sent to the ElasticSearch cluster. After the ElasticSearch cluster obtains the logs sent by the Logstash cluster, the logs are sent to the corresponding ES node, and the logs taken out of the decompression node are added, so that the number of logs processed by the ES node can be changed. In response to S203, for the ES node whose response time is equal to the average time, when the Logstash cluster sends the log, it should also be considered. Therefore, the number of logs processed by the ES node will be updated in real time.

[0088] In this embodiment, after delaying for one unit time, the ES node can complete a log processing, and after delaying for at least one unit time, the ES node updated response duration can be obtained, and the balancing effect after the above steps is performed can be reflected.

[0089] S208, determining to complete balancing of the number of logs processed by each ES node in the ElasticSearch cluster.

[0090] In a preferred embodiment, in S204, the weight of each to-be-allocated node is determined based on the response duration of the to-be-allocated node in the recent plurality of unit times, and the weight of each to-be-allocated node is determined based on the change trend parameter of the to-be-allocated node in the current unit time.

[0091] S241, obtaining the response duration of the to-be-allocated node in the recent plurality of unit times;

[0092] It should be noted that the recent plurality of unit times includes the current unit time and the unit time before the current unit time.

[0093] S242, determining the change trend parameter of the to-be-allocated node in the current unit time based on the response duration of the to-be-allocated node in the recent plurality of unit times; wherein the change trend parameter is used to represent the change trend of the response duration of the to-be-allocated node in the recent plurality of unit times, the change trend parameter of the to-be-allocated node is positively correlated with the change trend of the response duration, and the change trend parameter of the to-be-allocated node is negatively correlated with the change trend of the response duration.

[0094] S243, multiplying the ratio of the response duration of the to-be-allocated node in the current unit time to the sum of the response durations of all to-be-allocated nodes in the current unit time by the change trend parameter of the to-be-allocated node in the current unit time to obtain the initial weight of the to-be-allocated node in the current unit time.

[0095] S244, multiplying the ratio of the initial weight of the to-be-allocated node in the current unit time to the sum of the initial weights of all to-be-allocated nodes in the current unit time by a normalization coefficient to obtain the weight of the to-be-allocated node in the current unit time.

[0096] Specifically, the weight of the i-th to-be-allocated node in the current unit time t is calculated according to the following formula:

[0097]

[0098] wherein w i, represents the weight of the i-th to-be-allocated node in the current unit time t, N represents the total number of to-be-allocated nodes, x i, represents the response duration of the i-th to-be-allocated node in the current unit time t; x sum,trepresents the sum of the response durations of all the nodes to be allocated at the current unit time t, c is a normalization coefficient, and a i is the change trend parameter of the i-th node to be allocated at the current unit time, which is used to represent the change trend of the response duration of the i-th node to be allocated in the last plurality of unit times.

[0099] For the capacity-reduced node, when the change trend of the response duration of the i-th node to be allocated in the last plurality of unit times is increasing, a i > 1; when the change trend of the response duration of the i-th node to be allocated in the last plurality of unit times is decreasing, 0 < a i < 1; and when the change trend of the response duration of the i-th node to be allocated in the last plurality of unit times is constant, a i = 1.

[0100] For the capacity-increased node, when the change trend of the response duration of the i-th node to be allocated in the last plurality of unit times is increasing, 0 < a i < 1; when the change trend of the response duration of the i-th node to be allocated in the last plurality of unit times is decreasing, a i > 1; when the change trend of the response duration of the i-th node to be allocated in the last plurality of unit times is constant, a i = 1.

[0101] In this embodiment, by adding the change trend parameter, the weight of the node to be allocated changes with the change of the processing capacity. For example, for the capacity-increased node, assuming that the response duration of the capacity-increased node changes over time, when the response duration of the capacity-increased node rises, it is hoped that the capacity-increased node can contribute a smaller weight, so as to reduce the log processing amount allocated from the capacity-reduced node, so as to avoid that the processing capacity is insufficient to cause the response duration to be too long. By balancing the allocation of tasks, the response durations of the various nodes to be allocated tend to be close, thereby reducing the overall response duration of the ElasticSearch cluster. The change trend of the response duration of the capacity-increased node in the unit time interval is increasing, that is, a i is a value less than 1, so as to make the weight of the capacity-increased node slowly decrease with the rise of the response duration; conversely, when the response duration of the capacity-increased node decreases, it is hoped that the capacity-increased node can contribute a larger weight, so as to avoid that the processing capacity utilization is not enough to cause the task allocation to be unbalanced, thereby causing the response duration of other nodes to be allocated to increase. If the change trend of the response duration of the capacity-increased node in the unit time interval is decreasing, a i is a value greater than 1, so as to make the weight of the capacity-increased node slowly increase with the decrease of the response duration. The capacity-reduced node is opposite to the capacity-increased node, and the change trend of the response duration is positively correlated with the log processing amount allocated, which will not be described herein.

[0102] It should be noted that since the sum of the weights must be 1, it is necessary to divide the initial weight of each node to be allocated by the sum of the initial weights of all nodes to be allocated and multiply by a constant c to normalize, and the sum of the weights of all nodes to be allocated is ensured to be 1 by the constant c.

[0103] In a preferred embodiment, in S242, the change trend parameter of the node to be allocated in the current unit time is determined based on the response duration of the node to be allocated in the last plurality of unit times.

[0104] S2421, obtaining the response duration of the node to be allocated in the last K unit times;

[0105] S2422, obtaining the last two sliding windows from the response duration of the K unit times with a sliding window of duration k unit times; wherein k

[0106] S2423, determining the average duration of the last two sliding windows based on the k response durations contained in the sliding window;

[0107] It can be understood that each sliding window contains k response durations, and in some embodiments, the response durations of the last K unit times are recorded as tK={t1, t2,..., tK}, then the k response durations contained in the current sliding window are {tK-k+1,..., tK}, and the k response durations contained in the last sliding window are {tK-k,..., tK};

[0108] S2424, determining the change trend parameter of the node to be allocated in the current unit time based on the average duration of the last two sliding windows.

[0109] Specifically, for the capacity-reduced node, the change trend parameter of the i-th capacity-reduced node in the current unit time t is calculated according to the following formula:

[0110]

[0111] wherein, represents the average duration of the i-th capacity-reduced node in the current sliding window, represents the average duration of the i-th capacity-reduced node in the last sliding window, and i is the change trend parameter of the i-th capacity-reduced node in the current unit time.

[0112] For the capacity-increased node, the change trend parameter of the j-th capacity-increased node in the current unit time t is calculated according to the following formula:

[0113]

[0114] wherein, represents the average duration of the jth capacity-increasing node in the current sliding window, represents the average duration of the jth capacity-increasing node in the previous sliding window, and a j is a change trend parameter of the jth capacity-increasing node in the current unit time.

[0115] The embodiment can not only reflect the trend change, but also smooth the change trend, consider the long-term trend, and avoid the fluctuation of the to-be-allocated node caused by the large fluctuation of the response duration in adjacent unit times.

[0116] In a preferred embodiment, in S206, the determination of the capacity-increasing node corresponding to each capacity-decreasing node based on the capacity-decreasing capacity of each capacity-decreasing node and the capacity-increasing capacity of each capacity-increasing node comprises:

[0117] S261, each capacity-decreasing node is sorted in descending order of the capacity-decreasing capacity, and each capacity-increasing node is sorted in descending order of the capacity-increasing capacity;

[0118] S262, a capacity-decreasing node is taken out in order, each capacity-increasing node is polled in order, the capacity-increasing capacity with the smallest deviation from the capacity-decreasing capacity of the capacity-decreasing node is determined, the capacity-decreasing capacity is smaller than the maximum capacity corresponding to the capacity-increasing capacity, the capacity-increasing node corresponding to the capacity-increasing capacity is matched with the capacity-decreasing node, the capacity-decreasing capacity of the capacity-decreasing node is updated by subtracting the corresponding capacity-increasing capacity, and the capacity-increasing capacity of the corresponding capacity-increasing node is updated by subtracting the corresponding capacity-decreasing capacity;

[0119] It should be noted that since the capacity-decreasing capacity and the capacity-increasing capacity are sorted in descending order, if the response durations of the ES nodes are arranged in equal difference, the first capacity-decreasing node can be matched with the first capacity-increasing node. However, in actual situation, the capacity-decreasing capacity of the first capacity-decreasing node and the capacity-increasing capacity of the first capacity-increasing node have a large deviation, and the capacity-decreasing capacity of the first capacity-decreasing node is matched with the capacity-increasing capacity of a subsequent capacity-increasing node. The embodiment does not simply match in order, but adapts according to the capacity size, thereby improving the log migration efficiency.

[0120] In addition, for the capacity-increasing capacity with the smallest deviation from the capacity-decreasing capacity, even if the capacity-increasing capacity is smaller than the capacity-decreasing capacity, the capacity-decreasing capacity is smaller than the maximum capacity corresponding to the capacity-increasing capacity, and the matching is performed, thereby ensuring the log migration efficiency.

[0121] S263, it is determined whether there is an unmatched capacity-decreasing node, if yes, S262 is performed, otherwise, S264 is performed;

[0122] It should be noted that since the capacity reduction nodes and the capacity increase nodes are divided according to the time average, the overall pressure reduction capacity and the overall pressure increase capacity tend to be balanced. If a capacity reduction node has not been matched to a capacity increase node, it means that there is still spare pressure increase capacity in the updated capacity increase node. The pressure reduction capacity of each capacity reduction node can be distributed to each capacity increase node in turn by performing S262 to traverse all the capacity reduction nodes based on the updated capacity after adaptation.

[0123] S264, determining whether the variance of the pressure reduction capacity of each capacity reduction node and the pressure increase capacity of each capacity increase node is greater than a set capacity deviation threshold. If yes, performing S262, otherwise ending.

[0124] It should be noted that after traversing all the capacity reduction nodes, the capacity reduction nodes can still have certain pressure reduction capacity, and the capacity increase nodes can still have certain pressure increase capacity, and absolute equality is not pursued. It is only necessary to ensure that the variance is less than or equal to the set capacity deviation threshold. If the variance is greater than the capacity deviation threshold, it means that there is still capacity imbalance between the ES nodes. The capacity reduction nodes are traversed again based on the updated pressure reduction capacity, and the updated pressure reduction capacity is distributed to the corresponding capacity increase nodes until the capacity of each ES node is balanced. In some embodiments, the capacity deviation threshold is pre-set according to actual conditions. By using the multi-round matching mode, the pressure reduction capacity of each capacity reduction node is gradually distributed to each capacity increase node, thereby reducing the fluctuation of the ES nodes and balancing the logs of each ES node while minimizing the number of log migration times of the capacity reduction nodes, reducing the migration time cost, and improving the balancing efficiency.

[0125] In an embodiment, in S300, the first capacity difference is determined based on the difference between the response time of each balanced ES node and the second time threshold, comprising:

[0126] S310, obtaining the processing performance of each ES node, and multiplying the difference between the response time of the balanced ES node and the second time threshold by the corresponding processing performance as the maximum capacity of the ES node;

[0127] S320, determining whether the sum of the absolute values of the maximum capacities of each ES node is greater than the minimum capacity threshold. If yes, the sum of the maximum capacities of each ES node is taken as the first capacity difference; otherwise, the minimum capacity threshold is taken as the first capacity difference.

[0128] It should be noted that if the response time of a certain ES node after balancing is greater than the second time length threshold, the obtained difference value is positive; if the response time of a certain ES node after balancing is less than the second time length threshold, the obtained difference value is negative, and the sum of the maximum capacities of all ES nodes is negative. In the embodiment, the minimum capacity threshold is a preset positive number. Even if the sum of the maximum capacities of all ES nodes is negative, since there are some ES nodes exceeding the second time length threshold, the minimum capacity threshold is still used as the first capacity difference value for capacity expansion.

[0129] In an embodiment, in S400, the newly added ES node is determined based on the first capacity difference value, and the newly added ES node is added to the ElasticSearch cluster, including:

[0130] The crisis degree of the ElasticSearch cluster is calculated according to the following formula:

[0131] Risk=1+ln|maxt dif / avgt dif |

[0132] Wherein, maxt dif is the difference between the maximum response time of the ES node after balancing and the second time length threshold, and avgt dif is the difference between the average response time of the ES node after balancing and the second time length threshold.

[0133] The crisis degree and the first capacity difference value are multiplied to obtain a required capacity;

[0134] An ES node matched with the required capacity is determined as a newly added ES node and added to the ElasticSearch cluster.

[0135] In the embodiment, the idle resources and the response time are calculated to determine the crisis degree. As the deviation of maxt dif and avgt dif increases, it indicates that the balancing degree also increases, which indirectly affects the overall response time of the ElasticSearch cluster. By increasing the crisis degree accordingly, the required capacity is determined based on the crisis degree, and the required capacity leaves a certain redundant space for the ElasticSearch cluster, avoiding frequent addition of ES nodes.

[0136] In addition, referring to Figure 2 , one embodiment of the present application also provides an ELK-based log processing system, which comprises:

[0137] At least one processor;

[0138] at least one memory for storing at least one program;

[0139] When the at least one program is executed by the at least one processor, the at least one processor implements the above method.

[0140] It can be seen that the contents in the above method embodiments are all applicable to the present system embodiment, the present system embodiment specifically implements the same functions as the above method embodiments, and achieves the same beneficial effects as the above method embodiments.

[0141] In addition, one embodiment of the present application further provides a computer readable storage medium, which stores computer executable instructions, and the computer executable instructions are executed by a processor or a controller, for example, are executed by a processor in the above electronic device embodiment, so that the above processor executes the ELK-based log processing method in the above embodiments.

[0142] Similarly, the contents in the above method embodiments are all applicable to the present storage medium embodiment, the present storage medium embodiment specifically implements the same functions as the above method embodiments, and achieves the same beneficial effects as the above method embodiments.

[0143] Those skilled in the art can understand that all or some steps in the above disclosed method and system can be implemented as software, firmware, hardware and appropriate combinations thereof. Some or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software can be distributed on a computer readable medium, which can include computer storage media (or non-transitory media) and communication media (or transitory media). As known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. In addition, as known to those skilled in the art, communication media generally includes computer readable instructions, data structures, program modules or other data in modulated data signals such as carrier waves or other transmission mechanisms, and can include any information delivery medium.

[0144] The above is a specific description of the preferred implementation of the present disclosure, but the present disclosure is not limited to the above implementation, and those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the present disclosure, and these equivalent modifications or replacements are all included in the scope defined by the claims of the present disclosure.

Claims

1. An ELK-based log processing method, characterized in that, The method comprises the following steps: S100, acquiring response time lengths of each ES node in an ElasticSearch cluster for log processing; S200, if the response time length of any ES node exceeds a set first time length threshold, adjusting the number of logs processed by each ES node in the ElasticSearch cluster to balance the response time lengths of the ES nodes; S300, if it is determined that the response time length of any balanced ES node exceeds a set second time length threshold, determining a first capacity difference value based on the difference between the balanced response time length of each ES node and the second time length threshold; wherein the second time length threshold is less than the first time length threshold; S400, determining a newly added ES node based on the first capacity difference value, and adding the newly added ES node to the ElasticSearch cluster; S200 specifically comprises: S201, acquiring the response time length of each ES node in a current unit of time; S202, determining whether the variance of the response time length of each ES node in the current unit of time exceeds a set first variance threshold; if yes, performing S203, otherwise performing S208; S203, determining the time length mean of the response time length of all ES nodes in the current unit of time, taking the ES nodes less than the time length mean in the ElasticSearch cluster as capacity-increasing nodes, and taking the ES nodes greater than the time length mean as capacity-decreasing nodes; S204, taking the capacity-decreasing nodes and the capacity-increasing nodes as to-be-allocated nodes, determining the weight of each to-be-allocated node based on its response time length in a plurality of recent units of time; wherein the weight of the capacity-decreasing node is positively correlated with the response time length, and the weight of the capacity-increasing node is inversely correlated with the response time length; S205, acquiring the processing performance of the to-be-allocated node, multiplying the difference between the response time length of the to-be-allocated node in the current unit of time and the time length mean by the processing performance as the maximum capacity of the to-be-allocated node, and determining the allocation capacity of the to-be-allocated node based on the maximum capacity and the weight of the to-be-allocated node; wherein the processing performance is the number of logs that can be processed in a unit of time; the allocation capacity of the capacity-decreasing node is a decompression capacity, and the allocation capacity of the capacity-increasing node is a compression capacity; S206, determining the capacity-increasing node corresponding to each capacity-decreasing node based on the decompression capacity of each capacity-decreasing node and the compression capacity of each capacity-increasing node; S207, determining to-be-allocated logs based on the decompression capacity of the capacity-decreasing node, distributing the to-be-allocated logs to the corresponding capacity-increasing node, and performing S201 after delaying at least one unit of time; S208, determining to complete the balancing of the number of logs processed by each ES node in the ElasticSearch cluster.

2. The ELK-based log processing method of claim 1, wherein, After S200, the method further comprises: S500, determine whether the response duration of the balanced ES node is lower than a third duration threshold, if yes, determine a second capacity difference value based on the difference between the response duration of each balanced ES node and the third duration threshold; wherein the third duration threshold is less than the second duration threshold; S600, determine the capacity-reduced ES node based on the second capacity difference value, and remove the capacity-reduced ES node from the ElasticSearch cluster.

3. The ELK-based log processing method of claim 1, wherein, In S204, the weight of each to-be-assigned node is determined based on the response duration of the to-be-assigned node in the recent plurality of unit times, including: S241, obtaining the response duration of the to-be-assigned node in the recent plurality of unit times; S242, determining a change trend parameter of the to-be-assigned node in the current unit time based on the response duration of the to-be-assigned node in the recent plurality of unit times; wherein the change trend parameter is used to represent the change trend of the response duration of the to-be-assigned node in the recent plurality of unit times, the change trend parameter of the capacity-reduced node and the change trend of the response duration are positively correlated, and the change trend parameter of the capacity-increased node and the change trend of the response duration are negatively correlated; S243, taking the logarithm of the ratio of the response duration of the to-be-assigned node in the current unit time to the sum of the response durations of all to-be-assigned nodes in the current unit time, and multiplying the change trend parameter of the to-be-assigned node in the current unit time to obtain the initial weight of the to-be-assigned node in the current unit time; S244, multiplying the ratio of the initial weight of the to-be-assigned node in the current unit time to the initial weight of all to-be-assigned nodes in the current unit time by a normalization coefficient to obtain the weight of the to-be-assigned node in the current unit time.

4. The ELK-based log processing method of claim 3, wherein, In S242, the change trend parameter of the to-be-assigned node in the current unit time is determined based on the response duration of the to-be-assigned node in the recent plurality of unit times, including: obtaining the response duration of the to-be-assigned node in the recent K unit times; obtaining the recent two sliding windows from the K unit time response durations with a sliding window of duration k unit times; wherein k determining the average duration of the recent two sliding windows based on the k response durations contained in the sliding window; determining the change trend parameter of the to-be-assigned node in the current unit time based on the average duration of the recent two sliding windows of the to-be-assigned node.

5. The ELK-based log processing method of claim 1, wherein, In S206, the capacity-increased node corresponding to each capacity-reduced node is determined based on the capacity reduction of each capacity-reduced node and the capacity increase of each capacity-increased node, including: S261, sorting each capacity-reduced node in descending order of capacity reduction, and sorting each capacity-increased node in descending order of capacity increase; S262, sequentially taking out a capacity reducing node, sequentially polling each capacity increasing node in turn, determining the minimum capacity increasing capacity deviating from the capacity reducing capacity of the capacity reducing node, and the capacity reducing capacity is less than the maximum capacity corresponding to the capacity increasing capacity, matching the capacity increasing node corresponding to the capacity increasing capacity with the capacity reducing node, updating the capacity reducing capacity of the capacity reducing node by subtracting the corresponding capacity increasing capacity, and updating the capacity increasing capacity of the corresponding capacity increasing node by subtracting the corresponding capacity reducing capacity; S263, determining whether there is an unmatched capacity reducing node, if yes, executing S262, otherwise executing S264; S264, determining whether the variance of the capacity reducing capacity of each capacity reducing node and the capacity increasing capacity of each capacity increasing node is greater than the set capacity deviation threshold, if yes, executing S262, otherwise ending.

6. The ELK-based log processing method of claim 1, wherein, In S300, the first capacity difference is determined based on the difference between the balanced response time of each ES node and the second time threshold, and the first capacity difference includes: S310, obtaining the processing performance of each ES node, multiplying the difference between the balanced response time of the ES node and the second time threshold by the corresponding processing performance as the maximum capacity of the ES node; S320, determining whether the sum of the maximum capacities of each ES node is greater than the minimum capacity threshold after taking the absolute value, if yes, taking the sum of the maximum capacities of each ES node as the first capacity difference; otherwise, taking the minimum capacity threshold as the first capacity difference.

7. An ELK-based log processing system, characterized by, The system includes: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the ELK-based log processing method according to any one of claims 1 to 6.

8. A computer-readable storage medium storing computer-executable instructions, characterized in that, The computer executable instructions are used to execute the ELK-based log processing method according to any one of claims 1 to 6.

Images